Opened 2 years ago

Last modified 3 months ago

#30124 new enhancement

System package information, spkg-configure for Jupyter "notebook" package, "rst2ipynb", and dependencies

Reported by: Matthias Köppe Owned by:
Priority: major Milestone: sage-9.8
Component: build: configure Keywords: sd111
Cc: Dima Pasechnik, Nils Bruin, Samuel Lelièvre, Paul Masson, François Bissey, Emmanuel Charpentier, John Palmieri, Thierry Thomas, Andrew, Nicolas M. Thiéry Merged in:
Authors: Reviewers:
Report Upstream: N/A Work issues:
Branch: u/mkoeppe/system_information__spkg_configure_for_jupyter__notebook__package_and_dependencies (Commits, GitHub, GitLab) Commit: b8164c135422731f6319d85eb1a52f37440f1b6b
Dependencies: #28197, #30299 Stopgaps:

Status badges

Description (last modified by Matthias Köppe)

(From dimpase https://groups.google.com/d/msg/sage-devel/mWTD0_iBwKc/4eKjmINXAQAJ)

We currently cannot use Python packages from the system (see #29023) - and spkg-configure.m4 is not the right tool for changing that.

However, the situation is different for the Jupyter notebook package (and its dependencies):

We do not really need to install it in our venv if a suitable version is available in the system. Rather, we should install our kernel into the system/user's jupyter installation (see https://groups.google.com/forum/#!topic/sage-devel/wH7_bru_Q0s/discussion).

Likewise, rst2ipynb is a Python package but we only provide it to make a script available, not a Python module. So again, if the system provides a suitable version, we do not need to install it in our venv.

notebook's exclusive dependencies (see #26919, https://github.com/jupyter/notebook/blob/master/setup.py):

  • terminado, prometheus_client, send2trash
  • argon2-cffi (new dependency)

Shared between notebook and optional package rst2ipynb (via nbconvert):

  • nbconvert, nbformat, jsonschema, mistune, entrypoints, bleach, pandocfilters, testpath, defusedxml
  • pyrsistent (new dependency of jsonschema)
  • pandoc, pandoc_attributes (?)

The following dependencies are shared between notebook and sagelib/ipython/ipykernel. So we need to install them even if they are present in the system python:

  • pyzmq, dateutil, tornado, jupyter-client, jupyter-core, jinja2, pygments, ipython_genutils

Packages depending on notebook:

See also:

  • #30120 Sage's jupyter doesn't "talk" to a systemwide IRkernel R package
  • #26919 Upgrade Jupyter notebook to latest (6.1.1) and its dependencies to latest
  • #30246 Support interacts in JupyterLab

Change History (26)

comment:1 Changed 2 years ago by Matthias Köppe

Cc: Samuel Lelièvre added

comment:2 Changed 2 years ago by Matthias Köppe

Cc: Paul Masson added

This code from src/sage/repl/ipython_kernel/install.py of course needs updating so that we can install the kernel outside of the Sage venv:

        if prefix is None:
            from sys import prefix
        jupyter_dir = os.path.join(prefix, "share", "jupyter")
        self.nbextensions_dir = os.path.join(jupyter_dir, "nbextensions")
        self.kernel_dir = os.path.join(jupyter_dir, "kernels", self.identifier())
        self._mkdirs()

comment:3 Changed 2 years ago by Matthias Köppe

Cc: François Bissey added

comment:4 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:5 Changed 2 years ago by Matthias Köppe

Cc: Emmanuel Charpentier John Palmieri Thierry Thomas Andrew added
Description: modified (diff)
Summary: spkg-configure for "jupyter"System information, spkg-configure for Jupyter "notebook" package and dependencies

comment:6 Changed 2 years ago by François Bissey

That implies that we will have to ask a jupyter that is in the PATH to tell us about its installation PATH.

If we just call jupyter kernelspec install then it will install in a user location for a system wide jupyter (~/.local/share/jupyter/kernels on linux), for a sage jupyter it will probably go to the sage install ({sys.prefix}/share/jupyter/kernels on linux/OS X). But for distro it will become problematic straight away since you install in a stage before merging in the system tree. It is possible to pass a --prefix but this is not the same as DESTDIR which would be the useful thing for distros.

comment:7 in reply to:  6 ; Changed 2 years ago by Matthias Köppe

I think we will only need the following changes to sagelib:

  • remove the call to sage.repl.ipython_kernel.install from the installation steps done by sagelib's setup.py -- it is not compatible with modern python packaging methods anyway
  • add a console_script that installs a kernel spec (using sage.repl.ipython_kernel.install), call it sage-kernel (interface similar to: https://ipython.readthedocs.io/en/stable/install/kernel_install.html)
  • Change sage-notebook (which implements sage -n) to a shell script that invokes the correct jupyter (from system if installed in system, from SAGE_LOCAL if installed there)

Calling sage-kernel would be done by sage-the-distribution (perhaps in build/pkgs/sagelib/spkg-install), and likewise by distribution packagers' scripts.

comment:8 in reply to:  7 Changed 2 years ago by Matthias Köppe

Replying to mkoeppe:

I think we will only need the following changes to sagelib: ... Calling sage-kernel would be done by sage-the-distribution (perhaps in build/pkgs/sagelib/spkg-install)

I'll do the first two items in #30298

Last edited 2 years ago by Matthias Köppe (previous) (diff)

comment:9 Changed 2 years ago by Matthias Köppe

I have changed the plan a bit, see #30298 and #30299.

comment:10 Changed 2 years ago by Matthias Köppe

Dependencies: #28197, #30299

comment:11 Changed 2 years ago by Matthias Köppe

Branch: u/mkoeppe/system_information__spkg_configure_for_jupyter__notebook__package_and_dependencies

comment:12 Changed 2 years ago by Matthias Köppe

Commit: b8164c135422731f6319d85eb1a52f37440f1b6b
Summary: System information, spkg-configure for Jupyter "notebook" package and dependenciesSystem package information, spkg-configure for Jupyter "notebook" package and dependencies

New commits:

b8164c1build/pkgs/notebook/spkg-configure.m4: New

comment:13 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:14 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:15 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:16 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:17 Changed 2 years ago by Matthias Köppe

Cc: Nicolas M. Thiéry added
Description: modified (diff)
Summary: System package information, spkg-configure for Jupyter "notebook" package and dependenciesSystem package information, spkg-configure for Jupyter "notebook" package, "rst2ipynb", and dependencies

comment:18 Changed 2 years ago by Matthias Köppe

Milestone: sage-9.2sage-9.3

comment:19 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:20 Changed 2 years ago by Matthias Köppe

If we attempt to automatically install the Sage jupyter kernel into the system jupyter, for example using jupyter kernelspec install --user as explained in #30476, we should think about how to disambiguate the kernels when working with multiple Sage installations.

comment:21 Changed 2 years ago by Matthias Köppe

Keywords: sd111 added

Hoping we can make progress on this ticket this week - https://wiki.sagemath.org/days111

comment:22 Changed 22 months ago by Matthias Köppe

Milestone: sage-9.3sage-9.4

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

comment:23 Changed 16 months ago by Matthias Köppe

Milestone: sage-9.4sage-9.5

comment:24 Changed 12 months ago by Matthias Köppe

Milestone: sage-9.5sage-9.6

comment:25 Changed 9 months ago by Matthias Köppe

Milestone: sage-9.6sage-9.7

comment:26 Changed 3 months ago by Matthias Köppe

Milestone: sage-9.7sage-9.8
Note: See TracTickets for help on using tickets.