Upgrade jupyterlab to 3.3.x, add "sage -n nbclassic" and "sage -n retrolab"

[mkoeppe]

#30246 added jupyterlab 2.2.x as an optional package as part of Meta-ticket #30399.

The 3.0 series has been out since December 2020. ​https://pypi.org/project/jupyterlab/#history

Current as of Mar 2022 is 3.3.0. It still supports Python 3.7, as needed for Sage 9.6.

It is now also possible to run the classic notebook on top of jupyterlab - ​https://pypi.org/project/nbclassic/ (See also ​https://github.com/jupyterlab/retrolab#relation-to-other-jupyter-frontends)

[mkoeppe]

Last 10 new commits:

​2694cbf	build/pkgs/requests/checksums.ini: Add upstream_url
​62769f9	build/pkgs/rst2ipynb/checksums.ini: Add upstream_url
​eb6d3d0	build/pkgs/scandir/checksums.ini: Add upstream_url
​e2cada9	build/pkgs/snowballstemmer/checksums.ini: Add upstream_url
​d83e2f8	build/pkgs/wcwidth/checksums.ini: Add upstream_url
​c943913	build/pkgs/{backcall,ptyprocess}/dependencies: flit_core switched from pytoml to toml; remove pytoml package
​6225684	Merge #31280
​0075b33	build/pkgs/{scipy,sphinx,sympy}/install-requires.txt: Update version ranges to include the version in package-version.txt
​5116399	Merge #32071
​62e5a88	build/pkgs/jupyterlab: Update to 3.0.x

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​b348305	Merge tag '9.4.beta4' into t/31280/jupyter_upgrade
​2582707	build/pkgs/ptyprocess: Revert upgrade to 0.7.0; pin version to 0.5.1
​45cf013	build/pkgs/rst2ipynb/patches: Remove
​7762e01	Merge #31280

[mkoeppe]

Setting a new milestone for this ticket based on a cursory review.

[slelievre]

JupyterLab 3.1.0 was released.

• ​https://github.com/jupyterlab/jupyterlab/releases/tag/v3.1.0

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​0632776	build/pkgs/jupyterlab: Update to 3.1.x
​00f06a6	build/pkgs/{bleach,html5lib,importlib_resources,sagenb_export}/dependencies: Add missing deps
​ac8bce8	Merge #31280
​a15e3eb	build/pkgs/{flit_core,jupyterlab_pygments,matplotlib_inline,nbclient,nbconvert}/dependencies: Add missing deps
​dc094c0	build/pkgs/{ipykernel,nbconvert}/dependencies: Add missing deps
​c929321	build/pkgs/nbclient/dependencies: fixup
​41224af	build/pkgs/nbconvert/dependencies: Add missing dep
​0fc01d3	build/pkgs/zipp: Update to 3.5.0
​8a314c2	build/pkgs/bleach/dependencies: Add six
​7984ef6	Merge #32493

[mkoeppe]

New commits:

​0632776	build/pkgs/jupyterlab: Update to 3.1.x
​00f06a6	build/pkgs/{bleach,html5lib,importlib_resources,sagenb_export}/dependencies: Add missing deps
​ac8bce8	Merge #31280
​a15e3eb	build/pkgs/{flit_core,jupyterlab_pygments,matplotlib_inline,nbclient,nbconvert}/dependencies: Add missing deps
​dc094c0	build/pkgs/{ipykernel,nbconvert}/dependencies: Add missing deps
​c929321	build/pkgs/nbclient/dependencies: fixup
​41224af	build/pkgs/nbconvert/dependencies: Add missing dep
​0fc01d3	build/pkgs/zipp: Update to 3.5.0
​8a314c2	build/pkgs/bleach/dependencies: Add six
​7984ef6	Merge #32493

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​59132ab	build/pkgs/tomli: New, dependency of setuptools_scm
​7ebcb6c	build/pkgs/setuptools_scm/dependencies: Add packaging
​2def651	build/pkgs/{packaging,pyparsing,setuptools_scm}/dependencies: Avoid circular deps
​3e2fb34	Merge #31280
​aceeeeb	Merge #32493
​bf15a7b	build/pkgs/tomli/dependencies: Add flit_core
​432699d	build/pkgs/flit_core/dependencies: Remove dep on toml again, replace PYTHON_TOOLCHAIN by pip only, to break circular dep
​f10ea0b	build/pkgs/tomli/dependencies: Replace PYTHON_TOOLCHAIN by pip only, to break circular dep
​4bdbddd	build/pkgs/tomli/dependencies: Add toml because flit_core needs it at its runtime
​1fc1ba3	Merge #32482

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​be4c7c8

build/pkgs/jupyterlab/dependencies: Add missing deps

[mkoeppe]

On debian-bullseye (​https://github.com/mkoeppe/sage/runs/3594020610?check_suite_focus=true)

Error:  ==== ERROR IN LOG FILE artifacts/logs-commit-be4c7c8fe60812ff2bb6a0725d9041b2c7687980-tox-docker-debian-bullseye-maximal/logs/pkgs/jupyterlab_widgets-2.0.log ====
-\|/-\|/-An error occurred.
ValueError: 
"@jupyter-widgets/jupyterlab-manager@2.0.0" is not compatible with the current JupyterLab
Conflicting Dependencies:
JupyterLab                        Extension       Package
>=3.1.11 <3.2.0                   >=2.0.0 <3.0.0  @jupyterlab/application
>=3.1.11 <3.2.0                   >=2.0.0 <3.0.0  @jupyterlab/logconsole
>=3.1.11 <3.2.0                   >=2.0.0 <3.0.0  @jupyterlab/mainmenu
>=3.1.11 <3.2.0                   >=2.0.0 <3.0.0  @jupyterlab/notebook
>=3.1.11 <3.2.0                   >=2.0.0 <3.0.0  @jupyterlab/rendermime
>=3.1.11 <3.2.0                   >=2.0.0 <3.0.0  @jupyterlab/rendermime-interfaces
>=6.1.11 <6.2.0                   >=5.0.0 <6.0.0  @jupyterlab/services
>=3.1.11 <3.2.0                   >=2.0.0 <3.0.0  @jupyterlab/settingregistry
See the log file for details:  /tmp/jupyterlab-debug-lp7xrnhx.log
Error installing jupyter-widgets extension into jupyterlab ... exiting

[dimpase]

apparently we should aim for version 3.2.4, as my experiment shows.

[egourgoulhon]

I've just made a few experiments with jupyterlab 3.3.10; it seems to work well with Sage 9.5. Basically, here is what I did in a Sage 9.5 root directory on Ubuntu 20.04:

./sage -i install jupyterlab_widgets         # => installed jupyterlab 2.2.10
./sage -sh
(sage-sh) pip install --upgrade jupyterlab   # => installed jupyterlab 3.3.10
(sage-sh) pip install --upgrade jupyterlab_widgets
(sage-sh) pip install ipympl                 # required for %matplotlib widget
(sage-sh) exit
./sage -n jupyterlab

Then jupyterlab 3.3.10 opens in the browser and everything seems to work well. In particular, @interact works, as well as threejs animations, both tested via ​https://github.com/egourgoulhon/SageMathTest/blob/master/Notebooks/test_interact.ipynb. The interactive Matplotlib display has been tested with the notebook ​https://github.com/egourgoulhon/SageMathTest/blob/master/Notebooks/test_display_latex.ipynb, where %matplotlib notebook had to be replaced with %matplotlib widget, outside the scope of %display latex.

Is the issue with Debian bullseye mentioned in comment:18 still there with jupyterlab 3.3 ? Otherwise, what prevents jupyterlab 3.3 to go into Sage 9.6?

[mkoeppe]

Replying to egourgoulhon:

Is the issue with Debian bullseye mentioned in comment:18 still there with jupyterlab 3.3?

Likely outdated, we'll have to test this.

Otherwise, what prevents jupyterlab 3.3 to go into Sage 9.6?

Nothing really; if this does not require updates of packages that regular ipython/jupyter uses, then this is fine.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​b189f7e	build/pkgs/jupyterlab: Update to 3.1.x
​625afa6	build/pkgs/jupyterlab/dependencies: Add missing deps
​fd8efe0	build/pkgs/jupyterlab: Update to ~= 3.3
​3ae6292	build/pkgs/jupyterlab_widgets: Switch it to a pip package
​ba44b3e	build/pkgs/jupyter_packaging: Update to 0.11.1
​cd70e9b	build/pkgs/ipympl: Update to 0.8.8
​180993c	build/pkgs/jupyterlab_widgets/dependencies: Add ipympl

[mkoeppe]

Here's an attempt to update it (untested)

[mkoeppe]

Tested very briefly on macOS, seems to work

[egourgoulhon]

After pulling the ticket branch in Sage 9.6.beta3 and running make, the command

./sage -i jupyterlab_widgets

failed, telling that jupyter-packaging-0.11.1.tar.gz is not found on the mirrors:

[jupyter_packaging-0.11.1] Fastest mirror: http://www-ftp.lip6.fr/pub/math/sagemath/
[jupyter_packaging-0.11.1] http://www-ftp.lip6.fr/pub/math/sagemath/spkg/upstream/jupyter_packaging/jupyter-packaging-0.11.1.tar.gz
[jupyter_packaging-0.11.1] [xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
[jupyter_packaging-0.11.1] ERROR [transfer|run:135]: [Errno socket error] [Errno 404] Not Found: '//www-ftp.lip6.fr/pub/math/sagemath/spkg/upstream/jupyter_packaging/jupyter-packaging-0.11.1.tar.gz'
[jupyter_packaging-0.11.1] https://mirror.lyrahosting.com/sagemath/spkg/upstream/jupyter_packaging/jupyter-packaging-0.11.1.tar.gz
[jupyter_packaging-0.11.1] [xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
[jupyter_packaging-0.11.1] ERROR [transfer|run:135]: [Errno socket error] [Errno 404] Not Found: '//mirror.lyrahosting.com/sagemath/spkg/upstream/jupyter_packaging/jupyter-packaging-0.11.1.tar.gz'
[jupyter_packaging-0.11.1] http://ftp.rediris.es/mirror/sagemath/spkg/upstream/jupyter_packaging/jupyter-packaging-0.11.1.tar.gz
...
[jupyter_packaging-0.11.1] sage_bootstrap.tarball.FileNotMirroredError: tarball does not exist on mirror network
[jupyter_packaging-0.11.1] ************************************************************************
[jupyter_packaging-0.11.1] ************************************************************************
[jupyter_packaging-0.11.1] Error downloading jupyter-packaging-0.11.1.tar.gz
[jupyter_packaging-0.11.1] ************************************************************************
[jupyter_packaging-0.11.1] Please email sage-devel (http://groups.google.com/group/sage-devel)
[jupyter_packaging-0.11.1] explaining the problem and including the log files
[jupyter_packaging-0.11.1]   /home/eric/sage/9.6.develop/logs/pkgs/jupyter_packaging-0.11.1.log
[jupyter_packaging-0.11.1] and
[jupyter_packaging-0.11.1]   /home/eric/sage/9.6.develop/config.log
[jupyter_packaging-0.11.1] Describe your computer, operating system, etc.
[jupyter_packaging-0.11.1] ************************************************************************
make[2]: *** [Makefile:2765 : jupyter_packaging-SAGE_VENV-no-deps] Erreur 1
make[1]: *** [Makefile:2765 : /home/eric/sage/9.6.develop/local/var/lib/sage/venv-python3.8/var/lib/sage/installed/jupyter_packaging-0.11.1] Erreur 2
make[1] : on quitte le répertoire « /home/eric/sage/9.6.develop/build/make »

real	0m40.555s
user	0m0.367s
sys	0m0.104s
***************************************************************
Error building Sage.

[dimpase]

run

./configure --enable-download-from-upstream-url

then it will be picked up by make etc.

This is the standard way to review package updates - unless you want to download package tarballs manually.

[egourgoulhon]

Replying to dimpase:

run

./configure --enable-download-from-upstream-url

then it will be picked up by make etc.

Thanks! I did it, then run make and again ./sage -i jupyterlab_widgets, but still get the same error...

[egourgoulhon]

More precisely, the option --enable-download-from-upstream-url is taken into account but the download from upstream fails:

[jupyter_packaging-0.11.1] Attempting to download from https://pypi.io/packages/source/j/jupyter_packaging/jupyter-packaging-0.11.1.tar.gz
[jupyter_packaging-0.11.1] [xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
[jupyter_packaging-0.11.1] ERROR [transfer|run:135]: [Errno socket error] [Errno socket error] [Errno socket error] [Errno 404] Not Found: '//files.pythonhosted.org/packages/source/j/jupyter_packaging/jupyter-packaging-0.11.1.tar.gz'

[mkoeppe]

Thanks for testing! Looks like the tarball filename pattern has changed, I'll fix it

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d86c38a

build/pkgs/jupyter_packaging/checksums.ini: Fix tarball name

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​780516f

sage -n nbclassic: New

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​5426888	build/pkgs/retrolab: New
​efc92cd	sage -n retrolab: New

[mkoeppe]

Added some more variants.

[egourgoulhon]

Thanks for the update. There is no longer any issue with jupyter_packaging, but ./sage -i jupyterlab_widgets fails now in the build of ipyml:

[ipympl-0.8.8] Attempting to download from https://pypi.io/packages/source/i/ipympl/ipympl-0.8.8.tar.gz
[ipympl-0.8.8] [......................................................................]
[ipympl-0.8.8] ipympl-0.8.8
[ipympl-0.8.8] ====================================================
[ipympl-0.8.8] Setting up build directory for ipympl-0.8.8
[ipympl-0.8.8] Finished extraction
[ipympl-0.8.8] No patch files found in ../patches
[ipympl-0.8.8] ****************************************************
[ipympl-0.8.8] Host system:
[ipympl-0.8.8] Linux cartan 5.13.0-30-generic #33~20.04.1-Ubuntu SMP Mon Feb 7 14:25:10 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
[ipympl-0.8.8] ****************************************************
[ipympl-0.8.8] C compiler: gcc
[ipympl-0.8.8] C compiler version:
[ipympl-0.8.8] Using built-in specs.
[ipympl-0.8.8] COLLECT_GCC=gcc
[ipympl-0.8.8] COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/9/lto-wrapper
[ipympl-0.8.8] OFFLOAD_TARGET_NAMES=nvptx-none:hsa
[ipympl-0.8.8] OFFLOAD_TARGET_DEFAULT=1
[ipympl-0.8.8] Target: x86_64-linux-gnu
[ipympl-0.8.8] Configured with: ../src/configure -v --with-pkgversion='Ubuntu 9.3.0-17ubuntu1~20.04' --with-bugurl=file:///usr/share/doc/gcc-9/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,gm2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-9 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-9-HskZEa/gcc-9-9.3.0/debian/tmp-nvptx/usr,hsa --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
[ipympl-0.8.8] Thread model: posix
[ipympl-0.8.8] gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04) 
[ipympl-0.8.8] ****************************************************
[ipympl-0.8.8] Package 'ipympl' is currently not installed
[ipympl-0.8.8] No legacy uninstaller found for 'ipympl'; nothing to do
[ipympl-0.8.8] Installing ipympl-0.8.8
[ipympl-0.8.8] Processing /home/eric/sage/9.6.develop/local/var/lib/sage/venv-python3.8/var/tmp/sage/build/ipympl-0.8.8/src
[ipympl-0.8.8]   Preparing metadata (pyproject.toml): started
[ipympl-0.8.8]   Running command Preparing metadata (pyproject.toml)
[ipympl-0.8.8]   Traceback (most recent call last):
[ipympl-0.8.8]     File "/home/eric/sage/9.6.develop/local/var/lib/sage/venv-python3.8/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
[ipympl-0.8.8]       main()
[ipympl-0.8.8]     File "/home/eric/sage/9.6.develop/local/var/lib/sage/venv-python3.8/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
[ipympl-0.8.8]       json_out['return_val'] = hook(**hook_input['kwargs'])
[ipympl-0.8.8]     File "/home/eric/sage/9.6.develop/local/var/lib/sage/venv-python3.8/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 164, in prepare_metadata_for_build_wheel
[ipympl-0.8.8]       return hook(metadata_directory, config_settings)
[ipympl-0.8.8]     File "/home/eric/sage/9.6.develop/local/var/lib/sage/venv-python3.8/lib/python3.8/site-packages/setuptools/build_meta.py", line 174, in prepare_metadata_for_build_wheel
[ipympl-0.8.8]       self.run_setup()
[ipympl-0.8.8]     File "/home/eric/sage/9.6.develop/local/var/lib/sage/venv-python3.8/lib/python3.8/site-packages/setuptools/build_meta.py", line 158, in run_setup
[ipympl-0.8.8]       exec(compile(code, __file__, 'exec'), locals())
[ipympl-0.8.8]     File "setup.py", line 41, in <module>
[ipympl-0.8.8]       cmdclass = create_cmdclass('jsdeps', data_files_spec=data_files_spec)
[ipympl-0.8.8]   TypeError: 'NoneType' object is not callable
[ipympl-0.8.8]   error: subprocess-exited-with-error

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​1a3c7df

build/pkgs/{jupyter_packaging,ipympl}: Reduce to pip packages

[mkoeppe]

Thanks for testing! Here's a new version

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​9b30976

build/pkgs/jupyterlab_widgets/package-version.txt: Remove

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​7b277c9

build/pkgs/{jupyter_packaging,ipympl}: Reduce to pip packages

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​9514d82

build/pkgs/{jupyter_packaging,ipympl}: Reduce to pip packages

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​1243bd8

build/pkgs/{jupyter_packaging,ipympl}: Reduce to pip packages

[egourgoulhon]

Works like a charm now. Thank you! I've also tested the new notebook interfaces nbclassic and retrolab. Thanks for having added them!

[egourgoulhon]

The issue of %matplotlib widget being inoperative within the scope of %display latex is tracked in #33469. This is not specific to jupyterlab: it also happens with jupyter.

[mkoeppe]

Thanks for testing!