Opened 10 months ago

# Repackage Sage's cropped threejs as a pip-installable package (nbextension) jupyter-threejs-sage

Reported by: Owned by: mkoeppe major sage-9.4 user interface sd111 dimpase, gh-kliem, paulmasson, gh-jcamp0x2a, nbruin, slelievre, gh-mjungmath, mjo, fbissey, arojas, gh-mwageringel Matthias Koeppe N/A u/mkoeppe/repackage_sage_s_cropped_threejs_as_a_pip_installable_package__nbextension__jupyter_threejs_sage 74ade58efbe5f07f683e622928c45d2c12bda87b #30972

As a follow-up to #30972, we provide a pip-installable package (nbextension) that makes (all supported versions of) the threejs-sage offline script available to a Jupyter notebook:

This allows users to set up a system Jupyter notebook in which Sage is fully functional (including offline 3d graphics), without having to manually create symlinks from a Sage installation to the system's nbextensions directory.

### comment:1 Changed 10 months ago by mkoeppe

• Priority changed from major to blocker

### comment:2 Changed 10 months ago by mkoeppe

My guess is that the new install script build/pkgs/sagelib/spkg-install poisons environment variables too aggressively.

The code that installs the Jupyter kernel has moved recently to src/sage_setup/command/sage_install.py.

This should be investigated by someone who knows about the notebook.

### comment:3 Changed 10 months ago by charpent

According to Dima Pasechnik, this can be fixed by sage -b

Indeed...

That is neverthelass a bug that should be fixed.

### comment:5 follow-up: ↓ 6 Changed 10 months ago by jhpalmieri

Note that sage -b doesn't work any more: it tries to do cd "$SAGE_SRC" &&$MAKE, but SAGE_SRC/Makefile.in was removed in #29411. (If you have done an incremental build, the Makefile is still there.)

### comment:6 in reply to: ↑ 5 ; follow-up: ↓ 7 Changed 10 months ago by mkoeppe

Note that sage -b doesn't work any more: it tries to do cd "$SAGE_SRC" &&$MAKE, but SAGE_SRC/Makefile.in was removed in #29411. (If you have done an incremental build, the Makefile is still there.)

Let's fix that in #30153.

### comment:7 in reply to: ↑ 6 Changed 9 months ago by egourgoulhon

Note that sage -b doesn't work any more: it tries to do cd "$SAGE_SRC" &&$MAKE, but SAGE_SRC/Makefile.in was removed in #29411. (If you have done an incremental build, the Makefile is still there.)

Let's fix that in #30153.

As of Sage 9.2.beta7 (which includes #30153), sage -b does no longer work to fix the Jupyter issue.

### comment:8 Changed 9 months ago by egourgoulhon

FWIW, here is the content of SAGE_ROOT/local/share/jupyter/nbextensions with Sage 9.2.beta7:

lrwxrwxrwx 1 eric eric   19 août   3 15:12 jsmol -> /doesnotexist/jsmol
drwxr-xr-x 2 eric eric 4096 mai   30 17:52 jupyter-js-widgets
lrwxrwxrwx 1 eric eric   21 août   3 15:12 mathjax -> /doesnotexist/mathjax
lrwxrwxrwx 1 eric eric   21 août   3 15:12 threejs -> /doesnotexist/threejs


The doesnotexist directory does not look good... Changing manually the symbolic links to

lrwxrwxrwx 1 eric eric   46 août   3 17:40 jsmol -> /home/eric/sage/9.2.develop/local/share/jsmol/
drwxr-xr-x 2 eric eric 4096 mai   30 17:52 jupyter-js-widgets
lrwxrwxrwx 1 eric eric   48 août   3 17:40 mathjax -> /home/eric/sage/9.2.develop/local/share/mathjax/
lrwxrwxrwx 1 eric eric   48 août   3 17:40 threejs -> /home/eric/sage/9.2.develop/local/share/threejs/


fixes the Jupyter notebook issue reported in this ticket.

### comment:9 Changed 9 months ago by mkoeppe

Thanks for investigating this.

This is, as I thought, due to the poisoning of environment variables in the script build/pkgs/sagelib/spkg-install.

The links that created here seem to assume that jsmol, mathjax, and threejs are installed in SAGE_LOCAL.

The correct fix is to create spkg-configure.m4 scripts for these packages so that equivalent system packages can be used -- and to use specific directory variables for determining the targets for these symlinks.

### comment:10 Changed 9 months ago by mkoeppe

• Description modified (diff)
• Summary changed from Jupyter notebook broken in 9.2.beta4 to spkg-configure.m4 for jsmol, mathjax, threejs to fix broken Jupyter notebook

In fact, it seems strange that the sagelib installation script would even install these links. Shouldn't the install script of mathjax install its symlink?

### comment:11 Changed 9 months ago by mkoeppe

Hoping that someone who knows about these javascript things will work on this ticket.

### comment:12 Changed 9 months ago by dimpase

for some reason ./sage -i mathjax rebuilds sagelib.

### comment:13 Changed 9 months ago by paulmasson

I think this ticket is moving in the wrong direction. The issue is not with the installation of JavaScript packages, it is that they all expect a link to nbextensions to exist. There is an entire install file for Jupyter that sets up the environment for JavaScript packages by a call to its update method. Is this method no longer being called after installing the notebook?

### comment:14 Changed 9 months ago by mkoeppe

As you can see in comment 8, it is called but the paths are configured wrong.

### comment:15 follow-up: ↓ 16 Changed 9 months ago by paulmasson

So the notebook doesn't work because of an explicit mangling of environment variables. Why can't you just unmangle the appropriate directory so that notebook setup works? Oddly enough all of the tests in src/sage/repl/ipython_kernel/install.py pass despite the mangling.

As for using a system Three.js I would definitely advise against that. Mr.doob very often makes changes that are NOT backward compatible. The template that renders Three.js is version dependent: you can't just switch to a newer version and expect it to work. That's why I continue to extract the core of the library and package it here separately from the upstream repository.

### comment:16 in reply to: ↑ 15 ; follow-ups: ↓ 17 ↓ 19 Changed 9 months ago by mkoeppe

Oddly enough all of the tests in src/sage/repl/ipython_kernel/install.py pass despite the mangling.

The mangling happens during install, not during testing.

As for using a system Three.js I would definitely advise against that. Mr.doob very often makes changes that are NOT backward compatible. The template that renders Three.js is version dependent: you can't just switch to a newer version and expect it to work.

That's an important bit of information.

That's why I continue to extract the core of the library and package it here separately from the upstream repository.

You package it where?

### comment:17 in reply to: ↑ 16 ; follow-up: ↓ 18 Changed 9 months ago by paulmasson

That's why I continue to extract the core of the library and package it here separately from the upstream repository.

You package it where?

If you look at the upgrade tickets, like #29809, I always attach a zip file containing a small portion of the full library. I've been doing that since I started the viewer, at first just to avoid downloading the full library. There is however a change coming in the new year, in which one of the two critical files we use will officially be removed from the library. That will necessitate either continuing the current process or modifying a local copy of the full library, a process over which we would not be guaranteed control.

Volker has been moving my zip files to the mirrors manually. Do you know if that is also the process for libraries with upstream links?

### comment:18 in reply to: ↑ 17 ; follow-up: ↓ 21 Changed 9 months ago by mkoeppe

If you look at the upgrade tickets, like #29809, I always attach a zip file containing a small portion of the full library. [...]

Volker has been moving my zip files to the mirrors manually. Do you know if that is also the process for libraries with upstream links?

Thanks. I am looking at build/pkgs/threejs. Is the spkg-src script current? Is this what you actually use to generate archives?

Since the 9.1 cycle we have a better mechanism for tickets that upgrade spkgs. See https://wiki.sagemath.org/ReleaseTours/sage-9.1#For_developers-1

### comment:19 in reply to: ↑ 16 ; follow-up: ↓ 22 Changed 9 months ago by mkoeppe

As for using a system Three.js I would definitely advise against that. Mr.doob very often makes changes that are NOT backward compatible. The template that renders Three.js is version dependent: you can't just switch to a newer version and expect it to work.

OK, but is it possible to reliably test for presence of a specific version in the system?

Do you know how distribution packagers handle this? (Debian, Gentoo, ...)

### comment:21 in reply to: ↑ 18 Changed 9 months ago by paulmasson

If you look at the upgrade tickets, like #29809, I always attach a zip file containing a small portion of the full library. [...]

Volker has been moving my zip files to the mirrors manually. Do you know if that is also the process for libraries with upstream links?

Thanks. I am looking at build/pkgs/threejs. Is the spkg-src script current? Is this what you actually use to generate archives?

That's precisely what I used for the last update.

### comment:22 in reply to: ↑ 19 ; follow-up: ↓ 23 Changed 9 months ago by paulmasson

As for using a system Three.js I would definitely advise against that. Mr.doob very often makes changes that are NOT backward compatible. The template that renders Three.js is version dependent: you can't just switch to a newer version and expect it to work.

OK, but is it possible to reliably test for presence of a specific version in the system?

Do you know how distribution packagers handle this? (Debian, Gentoo, ...)

You can search for a REVISION string as is done here. This code was written to ditch installed_packages to help package managers.

On a more general note, what if a user installs a newer package of any sort that creates conflicts with Sage? Do you then force installation of the version needed by Sage?

### comment:23 in reply to: ↑ 22 ; follow-up: ↓ 26 Changed 9 months ago by mkoeppe

You can search for a REVISION string as is done here. This code was written to ditch installed_packages to help package managers.

Thanks. So this looks at a local installation (located via the THREEJS_DIR variable) and then picks specific versions of remote URLs.

When you make updates, do you make simultaneous changes to both sagelib and to the threejs spkg?

On a more general note, what if a user installs a newer package of any sort that creates conflicts with Sage? Do you then force installation of the version needed by Sage?

The spkg-configure mechanism of the Sage distribution inspects installed system packages. We either test for particular features of a library/program, or for a range of versions. For example, currently we accept python3 version 3.6.x and 3.7.x but nothing older nor newer. Whenever a package is found to be installed but not suitable, the Sage distribution installs its own copy.

### comment:24 follow-up: ↓ 25 Changed 9 months ago by mkoeppe

The standard way to install notebook extensions seems to be using jupyter nbextension. I think we may want to use this (instead of the custom code in sage.repl.ipython_kernel.install).

Last edited 9 months ago by mkoeppe (previous) (diff)

### comment:25 in reply to: ↑ 24 ; follow-up: ↓ 28 Changed 9 months ago by fbissey

The standard way to install notebook extensions seems to be using jupyter nbextension. I think we may want to use this (instead of the custom code in sage.repl.ipython_kernel.install).

In many ways jupyter nbextension is for personal install - which I guess is OK for vanilla sage. But I don't really know if it is designed well enough to be handled properly by a bona fide package management system.

The only real example I have on top of my head is widgetsnbextension and it uses regular distutils to install the extension.

### comment:26 in reply to: ↑ 23 ; follow-up: ↓ 27 Changed 9 months ago by paulmasson

You can search for a REVISION string as is done here. This code was written to ditch installed_packages to help package managers.

Thanks. So this looks at a local installation (located via the THREEJS_DIR variable) and then picks specific versions of remote URLs.

When you make updates, do you make simultaneous changes to both sagelib and to the threejs spkg?

I only change threejs.

### comment:27 in reply to: ↑ 26 ; follow-up: ↓ 29 Changed 9 months ago by mkoeppe

You can search for a REVISION string as is done here. This code was written to ditch installed_packages to help package managers.

Thanks. So this looks at a local installation (located via the THREEJS_DIR variable) and then picks specific versions of remote URLs.

When you make updates, do you make simultaneous changes to both sagelib and to the threejs spkg?

I only change threejs.

OK. To summarize, THREEJS_DIR is used at least twice: (1) during installation of sagelib, to determine symlinks that will be placed in the nbextensions folder. (2) during runtime of sagelib, to determine the threejs version to get certain remote URLs right. Do both of them have to point to the same installation (same version) for things to work, or are they two separate, independent uses?

Also, suppose we develop a way so that users can install sagelib using pip, instead of going through the sage distribution. Would it make sense to provide a copy of whatever parts of threejs that sagelib needs as a pip-installable Python package? Is https://github.com/jupyter-widgets/pythreejs of any relevance in this direction?

### comment:28 in reply to: ↑ 25 Changed 9 months ago by mkoeppe

The standard way to install notebook extensions seems to be using jupyter nbextension. I think we may want to use this (instead of the custom code in sage.repl.ipython_kernel.install).

[...] I don't really know if it is designed well enough to be handled properly by a bona fide package management system.

In gentoo, are you able to use the sage.repl.ipython_kernel.install during packaging, or are you using a custom script for this?

### comment:29 in reply to: ↑ 27 ; follow-up: ↓ 32 Changed 9 months ago by paulmasson

You can search for a REVISION string as is done here. This code was written to ditch installed_packages to help package managers.

Thanks. So this looks at a local installation (located via the THREEJS_DIR variable) and then picks specific versions of remote URLs.

When you make updates, do you make simultaneous changes to both sagelib and to the threejs spkg?

I only change threejs.

OK. To summarize, THREEJS_DIR is used at least twice: (1) during installation of sagelib, to determine symlinks that will be placed in the nbextensions folder. (2) during runtime of sagelib, to determine the threejs version to get certain remote URLs right. Do both of them have to point to the same installation (same version) for things to work, or are they two separate, independent uses?

Since there is only one Three.js template, they must both point to the same version.

Also, suppose we develop a way so that users can install sagelib using pip, instead of going through the sage distribution. Would it make sense to provide a copy of whatever parts of threejs that sagelib needs as a pip-installable Python package? Is https://github.com/jupyter-widgets/pythreejs of any relevance in this direction?

pythreejs is just a port of Three.js to python, so not relevant: one needs to get the actual JavaScript files. I don't know much about pip, but if you just mean taking my small portion of the Three.js library and delivering it some way other than through Sage, then that is certainly possible.

### comment:30 Changed 9 months ago by fbissey

In Gentoo I disable instance._symlink_resources() in the update method (this is all in sage/repl/ipython_kernel/install.py so it is not run. I do these manually from the ebuild.

This bit only creates links to three things, two icons and the documentation. I mainly do them manually because I want relative links instead of absolute links and it is quite possible that at the time it is normally executed things would end up pointing to the wrong places - which would be the real killer out of the two.

It is quite an achievement that we have come down to these three little things, it used to be much more complex.

### comment:31 Changed 9 months ago by fbissey

And I should add those three links are going in the sage kernel location not in nbextensions. The nbextensions are put in the exact right place (absolute links but we can't always have everything).

### comment:32 in reply to: ↑ 29 Changed 9 months ago by mkoeppe

I don't know much about pip, but if you just mean taking my small portion of the Three.js library and delivering it some way other than through Sage, then that is certainly possible.

Yes, that's what I mean. Basically making it available on https://pypi.org/ so that users can install it with pip. (The Sage distribution would still ship a copy of it.)

I'll be happy to help with the packaging.

This would be a good step in the direction of plugging Sage (which is installed into its own venv) into a system Jupyter installation.

### comment:33 Changed 9 months ago by mkoeppe

The package could be called jupyter-threejs-minimal or something like this. See https://pypi.org/search/?q=jupyter

### comment:34 Changed 9 months ago by mkoeppe

• Description modified (diff)

### comment:35 follow-up: ↓ 48 Changed 9 months ago by mkoeppe

What's the situation with jsmol?

### comment:36 Changed 9 months ago by mkoeppe

• Description modified (diff)

### comment:37 Changed 9 months ago by mkoeppe

• Summary changed from spkg-configure.m4 for jsmol, mathjax, threejs to fix broken Jupyter notebook to Repackage Sage's cropped threejs as a pip-installable package jupyter-threejs-minimal

### comment:38 Changed 9 months ago by paulmasson

Since the repackage still needs to ship with Sage to make the viewer function, this isn’t going to solve the notebook problem.

### comment:39 Changed 9 months ago by mkoeppe

By itself, no, but it's part of the solution. I cc'd you on #30124, which is another part of the puzzle.

### comment:40 Changed 9 months ago by mkoeppe

• Priority changed from blocker to major

I'll fix the notebook in #30298, but this ticket here remains relevant.

### comment:41 Changed 9 months ago by klee

I don't really understand the discussion here. But I remember the comment I made before in a sage-devel thread: https://groups.google.com/g/sage-devel/c/RGwGeJ5Tlso/m/EjRmq6gfAQAJ

.... One cause of constant confusion was that these jsmol, threejs, mathjax are served at /nbextensions. These are not proper nbextensions in Jupyter notebook ecosystem, but just static files. I suppose the proper place to serve them is /static, like /static/jsmol, ....

Fortunately we can use a Jupyter notebook option to serve /static from arbitrary directory that contains jsmol, threejs, and mathjax:

c.NotebookApp.extra_static_paths = ["/path/to/extra/static/files"]

I guess that with some changes mainly replacing "/nbextensions" to "/static", sage 3d and interact outputs all would work fine. ....

### comment:42 follow-up: ↓ 43 Changed 9 months ago by mkoeppe

But would that work for users who want to run Sage as a remote kernel?

### comment:43 in reply to: ↑ 42 Changed 9 months ago by klee

But would that work for users who want to run Sage as a remote kernel?

I have no precise idea about "Sage remote kernel". I just might say that the static files(jsmol, threejs, mathjax) should be served at the right url (/static) by the Jupyter front end server.

### comment:44 follow-up: ↓ 45 Changed 9 months ago by mkoeppe

The thing is that Jupyter has a clean mechanism to install and inspect nbextensions:

$jupyter nbextension list Known nbextensions: config dir: /Users/mkoeppe/s/sage/sage-rebasing/local/etc/jupyter/nbconfig notebook section jupyter-js-widgets/extension enabled - Validating: OK  It does not have such a facility for random static files. ### comment:45 in reply to: ↑ 44 Changed 9 months ago by klee Replying to mkoeppe: The thing is that Jupyter has a clean mechanism to install and inspect nbextensions: $ jupyter nbextension list
Known nbextensions:
config dir: /Users/mkoeppe/s/sage/sage-rebasing/local/etc/jupyter/nbconfig
notebook section
jupyter-js-widgets/extension  enabled
- Validating: OK


It does not have such a facility for random static files.

Do you have jsmol, theejs, mathjax in the list above? I guess not. It seems that jsmol, theejs, mathjax are not (and should not be) installed as proper nbextensions, but they reside in /nbextensions folder (by the symlinks) just to be served as static files for Sage graphics.

### comment:46 follow-up: ↓ 47 Changed 9 months ago by mkoeppe

That's right, they are not properly installed as nbextensions. My point is that they should.

### comment:47 in reply to: ↑ 46 Changed 9 months ago by klee

That's right, they are not properly installed as nbextensions. My point is that they should.

and my point is that they should not :-) but I am not sure...

### comment:48 in reply to: ↑ 35 Changed 9 months ago by mkoeppe

What's the situation with jsmol?

I have opened #30315 for jsmol

### comment:49 Changed 9 months ago by mkoeppe

https://github.com/jupyter-widgets/widget-ts-cookiecutter could be used for creating the package, but it has more complexity than what we need

### comment:50 Changed 9 months ago by mkoeppe

• Summary changed from Repackage Sage's cropped threejs as a pip-installable package jupyter-threejs-minimal to Repackage Sage's cropped threejs as a pip-installable package (nbextension) jupyter-threejs-minimal

### comment:51 Changed 8 months ago by mkoeppe

• Milestone changed from sage-9.2 to sage-9.3

### comment:52 Changed 6 months ago by mkoeppe

• Summary changed from Repackage Sage's cropped threejs as a pip-installable package (nbextension) jupyter-threejs-minimal to Repackage Sage's cropped threejs as a pip-installable package (nbextension) jupyter-threejs-sage

### comment:53 Changed 6 months ago by mkoeppe

• Description modified (diff)

### comment:55 Changed 6 months ago by mkoeppe

• Authors set to Matthias Koeppe
• Dependencies set to #30972

### comment:56 Changed 5 months ago by mkoeppe

• Branch set to u/mkoeppe/repackage_sage_s_cropped_threejs_as_a_pip_installable_package__nbextension__jupyter_threejs_sage

### comment:57 Changed 5 months ago by mkoeppe

• Commit set to 500de1a67c7c7f3fc9464113d1ba43fbb2d976cd
• Status changed from new to needs_review

Last 10 new commits:

 ​5e1ca4e src/sage/repl/ipython_kernel/install.py, src/sage/env.py, build/pkgs/sage_conf: Change from threejs to threejs-sage ​938bff4 Read required threejs version from ext_data ​f847738 src/sage/repl/rich_output/backend_ipython.py: Another change threejs -> threejs-sage ​d13177d src/sage/repl/rich_output/display_manager.py: Fix up imports ​47e795b Fix 404 fetching three.min.js from Jupyter/JupyterLab ​f74a7d8 Merge branch 't/30972/versioned_installation_of_threejs' into t/30123/repackage_sage_s_cropped_threejs_as_a_pip_installable_package__nbextension__jupyter_threejs_sage ​11693be build/pkgs/threejs: Switch to pip-installable package jupyter-threejs-sage ​7faf40d build/pkgs/sage_conf: No longer define THREEJS_DIR ​d08dbf2 sage.env: Get THREEJS_DIR from jupyter_threejs_sage ​500de1a sage.repl.ipython_kernel.install: No longer symlink THREEJS_DIR into nbextensions

### comment:58 Changed 5 months ago by mkoeppe

• Description modified (diff)

### comment:59 Changed 5 months ago by git

• Commit changed from 500de1a67c7c7f3fc9464113d1ba43fbb2d976cd to 74ade58efbe5f07f683e622928c45d2c12bda87b

Branch pushed to git repo; I updated commit sha1. New commits:

 ​74ade58 build/pkgs/threejs/install-requires.txt: New

### comment:60 follow-up: ↓ 66 Changed 5 months ago by mkoeppe

Input from downstream packagers would be very welcome

### comment:61 follow-up: ↓ 63 Changed 5 months ago by fbissey

Can we avoid the b1 stuff in the future? What is it even indicating.

### comment:62 follow-up: ↓ 64 Changed 5 months ago by fbissey

So it puts a copy in site-package that is "importable" and a copy in share/jupyter/nbextensions. Seems to behave decently at install.

The branch otherwise look sane but I haven't fully tested it. I just focused on looking at the new packaging. If 'b1' was for after 'b', using 'c' would have been ok :)

### comment:63 in reply to: ↑ 61 ; follow-up: ↓ 65 Changed 5 months ago by mkoeppe

Can we avoid the b1 stuff in the future? What is it even indicating.

That's "beta" 1 (because some metadata may need polishing). This is just the standard Python normalization of version numbers

### comment:64 in reply to: ↑ 62 Changed 5 months ago by mkoeppe

So it puts a copy in site-package that is "importable" and a copy in share/jupyter/nbextensions.

That's right, same behavior as widgetsnbextension

### comment:65 in reply to: ↑ 63 ; follow-up: ↓ 67 Changed 5 months ago by fbissey

Can we avoid the b1 stuff in the future? What is it even indicating.

That's "beta" 1 (because some metadata may need polishing). This is just the standard Python normalization of version numbers

Can't quite believe how that little bit just destroy the ebuild file format. Just 'b' would pass, but 'b1' with a meaning of 'beta 1', I technically should transform that into _beta1 and then retransform it inside the ebuild. If it is standard python versioning someone needs to do something Gentoo side to allow those names (other than argue that they are wrong or stupid).

### comment:66 in reply to: ↑ 60 Changed 5 months ago by arojas

Input from downstream packagers would be very welcome

Tested with the Arch package, everything works. +1 from me.

### comment:67 in reply to: ↑ 65 ; follow-up: ↓ 68 Changed 5 months ago by mkoeppe

Can we avoid the b1 stuff in the future? What is it even indicating.

That's "beta" 1 (because some metadata may need polishing). This is just the standard Python normalization of version numbers

Can't quite believe how that little bit just destroy the ebuild file format. Just 'b' would pass, but 'b1' with a meaning of 'beta 1', I technically should transform that into _beta1 and then retransform it inside the ebuild. If it is standard python versioning someone needs to do something Gentoo side to allow those names (other than argue that they are wrong or stupid).

Hopefully we can quickly proceed to an actual release so that this problem will go away.

### comment:68 in reply to: ↑ 67 Changed 5 months ago by fbissey

Can we avoid the b1 stuff in the future? What is it even indicating.

That's "beta" 1 (because some metadata may need polishing). This is just the standard Python normalization of version numbers

Can't quite believe how that little bit just destroy the ebuild file format. Just 'b' would pass, but 'b1' with a meaning of 'beta 1', I technically should transform that into _beta1 and then retransform it inside the ebuild. If it is standard python versioning someone needs to do something Gentoo side to allow those names (other than argue that they are wrong or stupid).

Hopefully we can quickly proceed to an actual release so that this problem will go away.

That's just venting. I can, and did, work around that.