Repackage Sage's cropped threejs as a pip-installable package (nbextension) jupyter-threejs-sage

[mkoeppe]

As a follow-up to #30972, we provide a pip-installable package (nbextension) that makes (all supported versions of) the threejs-sage offline script available to a Jupyter notebook:

• ​https://pypi.org/project/jupyter-threejs-sage/#description
• ​https://github.com/sagemath/threejs-sage/pull/1

This allows users to set up a system Jupyter notebook in which Sage is fully functional (including offline 3d graphics), without having to manually create symlinks from a Sage installation to the system's nbextensions directory.

see also:

• ​https://groups.google.com/d/msg/sage-devel/mWTD0_iBwKc/wvhk-VtKBgAJ
• ​https://groups.google.com/d/msg/sage-support/Jaiqx-KTwM0/BVxEp1gxAwAJ
• #30296 System package information and spkg-configure for mathjax

User reports:

• ​https://groups.google.com/g/sage-support/c/732Dcc7gcvI/m/XzbR_y6wBwAJ

[mkoeppe]

My guess is that the new install script build/pkgs/sagelib/spkg-install poisons environment variables too aggressively.

The code that installs the Jupyter kernel has moved recently to src/sage_setup/command/sage_install.py.

This should be investigated by someone who knows about the notebook.

[charpent]

According to ​Dima Pasechnik, this can be fixed by sage -b

Indeed...

That is neverthelass a bug that should be fixed.

[jhpalmieri]

Note that sage -b doesn't work any more: it tries to do cd "$SAGE_SRC" && $MAKE, but SAGE_SRC/Makefile.in was removed in #29411. (If you have done an incremental build, the Makefile is still there.)

[mkoeppe]

Replying to jhpalmieri:

Note that sage -b doesn't work any more: it tries to do cd "$SAGE_SRC" && $MAKE, but SAGE_SRC/Makefile.in was removed in #29411. (If you have done an incremental build, the Makefile is still there.)

Let's fix that in #30153.

[egourgoulhon]

Replying to mkoeppe:

Replying to jhpalmieri:

Note that sage -b doesn't work any more: it tries to do cd "$SAGE_SRC" && $MAKE, but SAGE_SRC/Makefile.in was removed in #29411. (If you have done an incremental build, the Makefile is still there.)

Let's fix that in #30153.

As of Sage 9.2.beta7 (which includes #30153), sage -b does no longer work to fix the Jupyter issue.

[egourgoulhon]

FWIW, here is the content of SAGE_ROOT/local/share/jupyter/nbextensions with Sage 9.2.beta7:

lrwxrwxrwx 1 eric eric   19 août   3 15:12 jsmol -> /doesnotexist/jsmol
drwxr-xr-x 2 eric eric 4096 mai   30 17:52 jupyter-js-widgets
lrwxrwxrwx 1 eric eric   21 août   3 15:12 mathjax -> /doesnotexist/mathjax
lrwxrwxrwx 1 eric eric   21 août   3 15:12 threejs -> /doesnotexist/threejs

The doesnotexist directory does not look good... Changing manually the symbolic links to

lrwxrwxrwx 1 eric eric   46 août   3 17:40 jsmol -> /home/eric/sage/9.2.develop/local/share/jsmol/
drwxr-xr-x 2 eric eric 4096 mai   30 17:52 jupyter-js-widgets
lrwxrwxrwx 1 eric eric   48 août   3 17:40 mathjax -> /home/eric/sage/9.2.develop/local/share/mathjax/
lrwxrwxrwx 1 eric eric   48 août   3 17:40 threejs -> /home/eric/sage/9.2.develop/local/share/threejs/

fixes the Jupyter notebook issue reported in this ticket.

[mkoeppe]

Thanks for investigating this.

This is, as I thought, due to the poisoning of environment variables in the script build/pkgs/sagelib/spkg-install.

The links that created here seem to assume that jsmol, mathjax, and threejs are installed in SAGE_LOCAL.

The correct fix is to create spkg-configure.m4 scripts for these packages so that equivalent system packages can be used -- and to use specific directory variables for determining the targets for these symlinks.

[mkoeppe]

In fact, it seems strange that the sagelib installation script would even install these links. Shouldn't the install script of mathjax install its symlink?

[mkoeppe]

Hoping that someone who knows about these javascript things will work on this ticket.

[dimpase]

for some reason ./sage -i mathjax rebuilds sagelib.

[paulmasson]

I think this ticket is moving in the wrong direction. The issue is not with the installation of JavaScript packages, it is that they all expect a link to nbextensions to exist. There is an entire ​install file for Jupyter that sets up the environment for JavaScript packages by a call to its update method. Is this method no longer being called after installing the notebook?

[mkoeppe]

As you can see in comment 8, it is called but the paths are configured wrong.

[paulmasson]

So the notebook doesn't work because of an explicit mangling of environment variables. Why can't you just unmangle the appropriate directory so that notebook setup works? Oddly enough all of the tests in src/sage/repl/ipython_kernel/install.py pass despite the mangling.

As for using a system Three.js I would definitely advise against that. Mr.doob very often makes changes that are NOT backward compatible. The template that renders Three.js is version dependent: you can't just switch to a newer version and expect it to work. That's why I continue to extract the core of the library and package it here separately from the upstream repository.

[mkoeppe]

Replying to paulmasson:

Oddly enough all of the tests in src/sage/repl/ipython_kernel/install.py pass despite the mangling.

The mangling happens during install, not during testing.

As for using a system Three.js I would definitely advise against that. Mr.doob very often makes changes that are NOT backward compatible. The template that renders Three.js is version dependent: you can't just switch to a newer version and expect it to work.

That's an important bit of information.

That's why I continue to extract the core of the library and package it here separately from the upstream repository.

You package it where?

[paulmasson]

Replying to mkoeppe:

Replying to paulmasson:

That's why I continue to extract the core of the library and package it here separately from the upstream repository.

You package it where?

If you look at the upgrade tickets, like #29809, I always attach a zip file containing a small portion of the full library. I've been doing that since I started the viewer, at first just to avoid downloading the full library. There is however a change coming in the new year, in which one of the two critical files we use will officially be removed from the library. That will necessitate either continuing the current process or modifying a local copy of the full library, a process over which we would not be guaranteed control.

Volker has been moving my zip files to the mirrors manually. Do you know if that is also the process for libraries with upstream links?

[mkoeppe]

Replying to paulmasson:

If you look at the upgrade tickets, like #29809, I always attach a zip file containing a small portion of the full library. [...]

Volker has been moving my zip files to the mirrors manually. Do you know if that is also the process for libraries with upstream links?

Thanks. I am looking at build/pkgs/threejs. Is the spkg-src script current? Is this what you actually use to generate archives?

Since the 9.1 cycle we have a better mechanism for tickets that upgrade spkgs. See ​https://wiki.sagemath.org/ReleaseTours/sage-9.1#For_developers-1

[mkoeppe]

Replying to mkoeppe:

Replying to paulmasson:

As for using a system Three.js I would definitely advise against that. Mr.doob very often makes changes that are NOT backward compatible. The template that renders Three.js is version dependent: you can't just switch to a newer version and expect it to work.

OK, but is it possible to reliably test for presence of a specific version in the system?

Do you know how distribution packagers handle this? (Debian, Gentoo, ...)

[paulmasson]

Replying to mkoeppe:

Replying to paulmasson:

If you look at the upgrade tickets, like #29809, I always attach a zip file containing a small portion of the full library. [...]

Volker has been moving my zip files to the mirrors manually. Do you know if that is also the process for libraries with upstream links?

Thanks. I am looking at build/pkgs/threejs. Is the spkg-src script current? Is this what you actually use to generate archives?

That's precisely what I used for the last update.

[paulmasson]

Replying to mkoeppe:

Replying to mkoeppe:

Replying to paulmasson:

As for using a system Three.js I would definitely advise against that. Mr.doob very often makes changes that are NOT backward compatible. The template that renders Three.js is version dependent: you can't just switch to a newer version and expect it to work.

OK, but is it possible to reliably test for presence of a specific version in the system?

Do you know how distribution packagers handle this? (Debian, Gentoo, ...)

You can search for a REVISION string as is done ​here. This code was written to ditch installed_packages to help package managers.

On a more general note, what if a user installs a newer package of any sort that creates conflicts with Sage? Do you then force installation of the version needed by Sage?

[mkoeppe]

Replying to paulmasson:

You can search for a REVISION string as is done ​here. This code was written to ditch installed_packages to help package managers.

Thanks. So this looks at a local installation (located via the THREEJS_DIR variable) and then picks specific versions of remote URLs.

When you make updates, do you make simultaneous changes to both sagelib and to the threejs spkg?

On a more general note, what if a user installs a newer package of any sort that creates conflicts with Sage? Do you then force installation of the version needed by Sage?

The spkg-configure mechanism of the Sage distribution inspects installed system packages. We either test for particular features of a library/program, or for a range of versions. For example, currently we accept python3 version 3.6.x and 3.7.x but nothing older nor newer. Whenever a package is found to be installed but not suitable, the Sage distribution installs its own copy.

[mkoeppe]

The standard way to install notebook extensions seems to be using jupyter nbextension. I think we may want to use this (instead of the custom code in sage.repl.ipython_kernel.install).

[fbissey]

Replying to mkoeppe:

The standard way to install notebook extensions seems to be using jupyter nbextension. I think we may want to use this (instead of the custom code in sage.repl.ipython_kernel.install).

In many ways jupyter nbextension is for personal install - which I guess is OK for vanilla sage. But I don't really know if it is designed well enough to be handled properly by a bona fide package management system.

The only real example I have on top of my head is widgetsnbextension and it uses regular distutils to install the extension.

[paulmasson]

Replying to mkoeppe:

Replying to paulmasson:

You can search for a REVISION string as is done ​here. This code was written to ditch installed_packages to help package managers.

Thanks. So this looks at a local installation (located via the THREEJS_DIR variable) and then picks specific versions of remote URLs.

When you make updates, do you make simultaneous changes to both sagelib and to the threejs spkg?

I only change threejs.

[mkoeppe]

Replying to paulmasson:

Replying to mkoeppe:

Replying to paulmasson:

You can search for a REVISION string as is done ​here. This code was written to ditch installed_packages to help package managers.

Thanks. So this looks at a local installation (located via the THREEJS_DIR variable) and then picks specific versions of remote URLs.

When you make updates, do you make simultaneous changes to both sagelib and to the threejs spkg?

I only change threejs.

OK. To summarize, THREEJS_DIR is used at least twice: (1) during installation of sagelib, to determine symlinks that will be placed in the nbextensions folder. (2) during runtime of sagelib, to determine the threejs version to get certain remote URLs right. Do both of them have to point to the same installation (same version) for things to work, or are they two separate, independent uses?

Also, suppose we develop a way so that users can install sagelib using pip, instead of going through the sage distribution. Would it make sense to provide a copy of whatever parts of threejs that sagelib needs as a pip-installable Python package? Is ​https://github.com/jupyter-widgets/pythreejs of any relevance in this direction?

[mkoeppe]

Replying to fbissey:

Replying to mkoeppe:

The standard way to install notebook extensions seems to be using jupyter nbextension. I think we may want to use this (instead of the custom code in sage.repl.ipython_kernel.install).

[...] I don't really know if it is designed well enough to be handled properly by a bona fide package management system.

In gentoo, are you able to use the sage.repl.ipython_kernel.install during packaging, or are you using a custom script for this?

[paulmasson]

Replying to mkoeppe:

Replying to paulmasson:

Replying to mkoeppe:

Replying to paulmasson:

You can search for a REVISION string as is done ​here. This code was written to ditch installed_packages to help package managers.

Thanks. So this looks at a local installation (located via the THREEJS_DIR variable) and then picks specific versions of remote URLs.

When you make updates, do you make simultaneous changes to both sagelib and to the threejs spkg?

I only change threejs.

OK. To summarize, THREEJS_DIR is used at least twice: (1) during installation of sagelib, to determine symlinks that will be placed in the nbextensions folder. (2) during runtime of sagelib, to determine the threejs version to get certain remote URLs right. Do both of them have to point to the same installation (same version) for things to work, or are they two separate, independent uses?

Since there is only one Three.js template, they must both point to the same version.

Also, suppose we develop a way so that users can install sagelib using pip, instead of going through the sage distribution. Would it make sense to provide a copy of whatever parts of threejs that sagelib needs as a pip-installable Python package? Is ​https://github.com/jupyter-widgets/pythreejs of any relevance in this direction?

pythreejs is just a port of Three.js to python, so not relevant: one needs to get the actual JavaScript files. I don't know much about pip, but if you just mean taking my small portion of the Three.js library and delivering it some way other than through Sage, then that is certainly possible.

[fbissey]

In Gentoo I disable instance._symlink_resources() in the update method (this is all in sage/repl/ipython_kernel/install.py so it is not run. I do these manually from the ebuild.

This bit only creates links to three things, two icons and the documentation. I mainly do them manually because I want relative links instead of absolute links and it is quite possible that at the time it is normally executed things would end up pointing to the wrong places - which would be the real killer out of the two.

It is quite an achievement that we have come down to these three little things, it used to be much more complex.

[fbissey]

And I should add those three links are going in the sage kernel location not in nbextensions. The nbextensions are put in the exact right place (absolute links but we can't always have everything).

[mkoeppe]

Replying to paulmasson:

I don't know much about pip, but if you just mean taking my small portion of the Three.js library and delivering it some way other than through Sage, then that is certainly possible.

Yes, that's what I mean. Basically making it available on ​https://pypi.org/ so that users can install it with pip. (The Sage distribution would still ship a copy of it.)

​https://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/Distributing%20Jupyter%20Extensions%20as%20Python%20Packages.html

I'll be happy to help with the packaging.

This would be a good step in the direction of plugging Sage (which is installed into its own venv) into a system Jupyter installation.

[mkoeppe]

The package could be called jupyter-threejs-minimal or something like this. See ​https://pypi.org/search/?q=jupyter

[mkoeppe]

What's the situation with jsmol?

[paulmasson]

Since the repackage still needs to ship with Sage to make the viewer function, this isn’t going to solve the notebook problem.

[mkoeppe]

By itself, no, but it's part of the solution. I cc'd you on #30124, which is another part of the puzzle.

[mkoeppe]

I'll fix the notebook in #30298, but this ticket here remains relevant.

[klee]

I don't really understand the discussion here. But I remember the comment I made before in a sage-devel thread: ​https://groups.google.com/g/sage-devel/c/RGwGeJ5Tlso/m/EjRmq6gfAQAJ

---

.... One cause of constant confusion was that these jsmol, threejs, mathjax are served at /nbextensions. These are not proper nbextensions in Jupyter notebook ecosystem, but just static files. I suppose the proper place to serve them is /static, like /static/jsmol, ....

Fortunately we can use a Jupyter notebook option to serve /static from arbitrary directory that contains jsmol, threejs, and mathjax:

c.NotebookApp.extra_static_paths = ["/path/to/extra/static/files"]

I guess that with some changes mainly replacing "/nbextensions" to "/static", sage 3d and interact outputs all would work fine. ....

[mkoeppe]

But would that work for users who want to run Sage as a remote kernel?

[klee]

Replying to mkoeppe:

But would that work for users who want to run Sage as a remote kernel?

I have no precise idea about "Sage remote kernel". I just might say that the static files(jsmol, threejs, mathjax) should be served at the right url (/static) by the Jupyter front end server.

[mkoeppe]

The thing is that Jupyter has a clean mechanism to install and inspect nbextensions:

$ jupyter nbextension list
Known nbextensions:
  config dir: /Users/mkoeppe/s/sage/sage-rebasing/local/etc/jupyter/nbconfig
    notebook section
      jupyter-js-widgets/extension  enabled 
      - Validating: OK

It does not have such a facility for random static files.

[klee]

Replying to mkoeppe:

The thing is that Jupyter has a clean mechanism to install and inspect nbextensions:

$ jupyter nbextension list
Known nbextensions:
  config dir: /Users/mkoeppe/s/sage/sage-rebasing/local/etc/jupyter/nbconfig
    notebook section
      jupyter-js-widgets/extension  enabled 
      - Validating: OK

It does not have such a facility for random static files.

Do you have jsmol, theejs, mathjax in the list above? I guess not. It seems that jsmol, theejs, mathjax are not (and should not be) installed as proper nbextensions, but they reside in /nbextensions folder (by the symlinks) just to be served as static files for Sage graphics.

[mkoeppe]

That's right, they are not properly installed as nbextensions. My point is that they should.

[klee]

Replying to mkoeppe:

That's right, they are not properly installed as nbextensions. My point is that they should.

and my point is that they should not :-) but I am not sure...

[mkoeppe]

Replying to mkoeppe:

What's the situation with jsmol?

I have opened #30315 for jsmol

[mkoeppe]

​https://github.com/jupyter-widgets/widget-ts-cookiecutter could be used for creating the package, but it has more complexity than what we need

[mkoeppe]

This is now ​https://github.com/sagemath/threejs-sage/pull/1/files

[mkoeppe]

Last 10 new commits:

​5e1ca4e	src/sage/repl/ipython_kernel/install.py, src/sage/env.py, build/pkgs/sage_conf: Change from threejs to threejs-sage
​938bff4	Read required threejs version from ext_data
​f847738	src/sage/repl/rich_output/backend_ipython.py: Another change threejs -> threejs-sage
​d13177d	src/sage/repl/rich_output/display_manager.py: Fix up imports
​47e795b	Fix 404 fetching three.min.js from Jupyter/JupyterLab
​f74a7d8	Merge branch 't/30972/versioned_installation_of_threejs' into t/30123/repackage_sage_s_cropped_threejs_as_a_pip_installable_package__nbextension__jupyter_threejs_sage
​11693be	build/pkgs/threejs: Switch to pip-installable package jupyter-threejs-sage
​7faf40d	build/pkgs/sage_conf: No longer define THREEJS_DIR
​d08dbf2	sage.env: Get THREEJS_DIR from jupyter_threejs_sage
​500de1a	sage.repl.ipython_kernel.install: No longer symlink THREEJS_DIR into nbextensions

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​74ade58

build/pkgs/threejs/install-requires.txt: New

[mkoeppe]

Input from downstream packagers would be very welcome

[fbissey]

Can we avoid the b1 stuff in the future? What is it even indicating.

[fbissey]

So it puts a copy in site-package that is "importable" and a copy in share/jupyter/nbextensions. Seems to behave decently at install.

The branch otherwise look sane but I haven't fully tested it. I just focused on looking at the new packaging. If 'b1' was for after 'b', using 'c' would have been ok :)

[mkoeppe]

Replying to fbissey:

Can we avoid the b1 stuff in the future? What is it even indicating.

That's "beta" 1 (because some metadata may need polishing). This is just the standard Python normalization of version numbers

[mkoeppe]

Replying to fbissey:

So it puts a copy in site-package that is "importable" and a copy in share/jupyter/nbextensions.

That's right, same behavior as widgetsnbextension

[fbissey]

Replying to mkoeppe:

Replying to fbissey:

Can we avoid the b1 stuff in the future? What is it even indicating.

That's "beta" 1 (because some metadata may need polishing). This is just the standard Python normalization of version numbers

Can't quite believe how that little bit just destroy the ebuild file format. Just 'b' would pass, but 'b1' with a meaning of 'beta 1', I technically should transform that into _beta1 and then retransform it inside the ebuild. If it is standard python versioning someone needs to do something Gentoo side to allow those names (other than argue that they are wrong or stupid).

[arojas]

Replying to mkoeppe:

Input from downstream packagers would be very welcome

Tested with the Arch package, everything works. +1 from me.

[mkoeppe]

Replying to fbissey:

Replying to mkoeppe:

Replying to fbissey:

Can we avoid the b1 stuff in the future? What is it even indicating.

That's "beta" 1 (because some metadata may need polishing). This is just the standard Python normalization of version numbers

Can't quite believe how that little bit just destroy the ebuild file format. Just 'b' would pass, but 'b1' with a meaning of 'beta 1', I technically should transform that into _beta1 and then retransform it inside the ebuild. If it is standard python versioning someone needs to do something Gentoo side to allow those names (other than argue that they are wrong or stupid).

Hopefully we can quickly proceed to an actual release so that this problem will go away.

[fbissey]

Replying to mkoeppe:

Replying to fbissey:

Replying to mkoeppe:

Replying to fbissey:

Can we avoid the b1 stuff in the future? What is it even indicating.

That's "beta" 1 (because some metadata may need polishing). This is just the standard Python normalization of version numbers

Can't quite believe how that little bit just destroy the ebuild file format. Just 'b' would pass, but 'b1' with a meaning of 'beta 1', I technically should transform that into _beta1 and then retransform it inside the ebuild. If it is standard python versioning someone needs to do something Gentoo side to allow those names (other than argue that they are wrong or stupid).

Hopefully we can quickly proceed to an actual release so that this problem will go away.

That's just venting. I can, and did, work around that.

[mkoeppe]

Hoping we can make progress on this ticket this week - ​https://wiki.sagemath.org/days111

[dimpase]

please rebase

[mkoeppe]

Replying to mkoeppe:

Also, suppose we develop a way so that users can install sagelib using pip, instead of going through the sage distribution. Would it make sense to provide a copy of whatever parts of threejs that sagelib needs as a pip-installable Python package? Is ​https://github.com/jupyter-widgets/pythreejs of any relevance in this direction?

As it turns out, it is very relevant; and it would probably be best to redo the Python packaging (jupyter-threejs-sage) as a fork of pythreejs. They already have support for JupyterLab 3

[mkoeppe]

Opened #32070 (Add package pythreejs 2.3.0).

[mkoeppe]

Setting a new milestone for this ticket based on a cursory review.