Opened 22 months ago

Closed 2 months ago

#26919 closed enhancement (fixed)

Upgrade Jupyter notebook to latest (6.1.1) and its dependencies to latest

Reported by: slelievre Owned by:
Priority: critical Milestone: sage-9.2
Component: packages: standard Keywords: upgrade, jupyter, notebook
Cc: arojas, fbissey, gh-bryangingechen, gh-timokau, infinity0, jdemeyer, slelievre, thansen, egourgoulhon Merged in:
Authors: Matthias Koeppe, Joshua Campbell Reviewers: Joshua Campbell, Matthias Koeppe, Dima Pasechnik
Report Upstream: N/A Work issues:
Branch: 445c1d7 (Commits) Commit: 445c1d7a5f90ddb6af5b33e1d03697cfc3eefc54
Dependencies: #28197, #30299, #25363 Stopgaps:

Description (last modified by mkoeppe)

Upgrade:

New dependencies:

  • argon2-cffi (?)
  • pyrsistent, attrs, importlib_metadata, zipp

Upgrade of its dependencies:

Recursive dependencies:

Upgrade of a reverse dependency:

(tarball URLs: see checksums.ini; use ./configure --enable-download-from-upstream-url to enable automatic downloads of the new versions)

Previous upgrades:

  • notebook 5.7.6 in #27463
  • notebook 5.7.1, in #26709, merged in Sage 8.5.beta5.

See also:

  • #30124 System package information, spkg-configure for Jupyter "notebook" package and dependencies
  • #30306 Meta-ticket: Use system Jupyter

Change History (58)

comment:1 Changed 21 months ago by embray

  • Milestone changed from sage-8.6 to sage-8.7

Retarging tickets optimistically to the next milestone. If you are responsible for this ticket (either its reporter or owner) and don't believe you are likely to complete this ticket before the next release (8.7) please retarget this ticket's milestone to sage-pending or sage-wishlist.

comment:2 Changed 19 months ago by embray

  • Milestone changed from sage-8.7 to sage-8.8

Moving all blocker/critical issues from 8.7 to 8.8.

comment:3 Changed 19 months ago by slelievre

  • Description modified (diff)
  • Summary changed from Upgrade to Jupyter notebook 5.7.4 to Upgrade to Jupyter notebook 5.7.7

comment:4 Changed 19 months ago by slelievre

  • Description modified (diff)
  • Summary changed from Upgrade to Jupyter notebook 5.7.7 to Upgrade to Jupyter notebook 5.7.8

comment:5 Changed 19 months ago by slelievre

  • Branch set to u/slelievre/notebook-5.7.8

comment:6 Changed 19 months ago by slelievre

  • Commit set to b1a4b7fd4a215fb1340956b6a9e0c15d8ea0c760
  • Status changed from new to needs_review

New commits:

b1a4b7fUpgrade to notebook 5.7.8

comment:7 Changed 18 months ago by embray

Note, this would conflict slightly with #27715. As this ticket came first I'd be happy to rebase onto it so long as do intend to move forward on this ticket (I don't see why we shouldn't, though I haven't tested it yet...)

comment:8 Changed 16 months ago by embray

  • Milestone changed from sage-8.8 to sage-8.9

Moving tickets from the Sage 8.8 milestone that have been actively worked on in the last six months to the next release milestone (optimistically).

comment:9 Changed 14 months ago by chapoton

branch is red

comment:10 Changed 14 months ago by chapoton

  • Status changed from needs_review to needs_work

comment:11 Changed 10 months ago by embray

  • Milestone changed from sage-8.9 to sage-9.1

Ticket retargeted after milestone closed

comment:12 Changed 6 months ago by mkoeppe

  • Milestone changed from sage-9.1 to sage-9.2

comment:13 Changed 3 months ago by slelievre

Notebook 6.1.0 RC1 is out:

Notebook 5.7.10 would be the last py2-compatible release.

comment:14 Changed 3 months ago by mkoeppe

py2 compatibility is irrelevant now. Please see #29141 for discussion what we need to be compatible with

comment:15 Changed 3 months ago by slelievre

comment:16 Changed 3 months ago by mkoeppe

  • Summary changed from Upgrade to Jupyter notebook 5.7.8 to Upgrade to Jupyter notebook 5.7.10 or 6.1.0 ....

comment:17 Changed 3 months ago by mkoeppe

See also: #28313 "notebook-5.7.6 contains local files"

comment:18 Changed 3 months ago by mkoeppe

  • Cc egourgoulhon added
  • Description modified (diff)

comment:19 Changed 2 months ago by mkoeppe

  • Dependencies set to #28197
  • Description modified (diff)
  • Summary changed from Upgrade to Jupyter notebook 5.7.10 or 6.1.0 .... to Upgrade Jupyter notebook to latest (6.1.1) and its dependencies to latest

comment:20 Changed 2 months ago by mkoeppe

  • Branch changed from u/slelievre/notebook-5.7.8 to u/mkoeppe/notebook-5.7.8

comment:21 Changed 2 months ago by mkoeppe

  • Commit changed from b1a4b7fd4a215fb1340956b6a9e0c15d8ea0c760 to 6ad6e9b81d1e35aa982ab024a5bf8f7b9ed8d2f0
  [notebook-6.1.1]   Applying patches from ../patches...
  [notebook-6.1.1]   Applying ../patches/0001-Workaround-for-wrong-errno-on-socket-bind-permission.patch
  [notebook-6.1.1]   patching file notebook/notebookapp.py
  [notebook-6.1.1]   Hunk #1 FAILED at 1406.

It should be checked if this patch has been upstreamed


Last 10 new commits:

3205576build/pkgs/notebook: Update to 6.1.1
ef9a90dbuild/pkgs/nbformat: Update to 5.0.7
072ea38build/pkgs/nbconvert: Update to 5.6.1
712a5b6build/pkgs/jupyter_client: Update to 6.1.6
00373c6build/pkgs/jinja2: Update to 2.11.2
7e1b5c5build/pkgs/tornado: Update to 6.1.1
7d73f70build/pkgs/terminado: Update to 0.8.3
c4b19e8build/pkgs/prometheus_client: Update to 0.8.0
9075a16build/pkgs/jupyter_core: Update to 4.6.3
6ad6e9bbuild/pkgs/pyzmq: Update to 19.0.2

comment:22 Changed 2 months ago by mkoeppe

The patch was added for Cygwin in #27715 (Jupyter notebook can crash during startup if default port cannot be used)

comment:23 Changed 2 months ago by git

  • Commit changed from 6ad6e9b81d1e35aa982ab024a5bf8f7b9ed8d2f0 to 26691af0ce7bf0f96a12bc676037e2bda4616f97

Branch pushed to git repo; I updated commit sha1. New commits:

26691afbuild/pkgs/notebook: Remove upstreamed patch (PR 4584)

comment:24 Changed 2 months ago by mkoeppe

  • Status changed from needs_work to needs_review

Builds OK for me on macOS. I have not tested anything else because I do not use the jupyter notebook and cannot distinguish a working and a broken installation.

comment:25 Changed 2 months ago by mkoeppe

  • Description modified (diff)

comment:26 Changed 2 months ago by mkoeppe

  • Authors set to Matthias Koeppe

comment:27 Changed 2 months ago by git

  • Commit changed from 26691af0ce7bf0f96a12bc676037e2bda4616f97 to 4762704eb5a089237b18948b85e3171208ee23fd

Branch pushed to git repo; I updated commit sha1. New commits:

4762704build/pkgs/widgetsnbextension: Update to 3.5.1

comment:28 Changed 2 months ago by mkoeppe

  • Description modified (diff)

comment:29 Changed 2 months ago by mkoeppe

  • Description modified (diff)

comment:30 Changed 2 months ago by git

  • Commit changed from 4762704eb5a089237b18948b85e3171208ee23fd to 0e1c3258bdb73cdd403daabb0c52d4bb5861269c

Branch pushed to git repo; I updated commit sha1. New commits:

b97ebb2build/pkgs/jsonschema: Update to 3.2.0
4b98e39build/pkgs/mistune: Add upstream_url
4c8e3f6build/pkgs/bleach: Update to 3.1.5
0e1c325build/pkgs/testpath: Update to 0.4.4

comment:31 Changed 2 months ago by mkoeppe

  • Description modified (diff)

comment:32 Changed 2 months ago by mkoeppe

  • Description modified (diff)

comment:33 Changed 2 months ago by mkoeppe

  • Description modified (diff)

comment:34 Changed 2 months ago by mkoeppe

  • Description modified (diff)

comment:35 Changed 2 months ago by mkoeppe

  • Description modified (diff)

comment:36 Changed 2 months ago by mkoeppe

  • Dependencies changed from #28197 to #28197, #30299

comment:37 follow-up: Changed 2 months ago by gh-jcamp0x2a

Hello! Saw your post on sage-devel and figured I'd take a shot at testing this.

I was able to get this branch to build just fine, but I ran into some missing run-time dependencies for jsonschema, namely: attrs, importlib_metadata, and pyrsistent (which then required zipp).

I've pushed a commit with these packages added. This is the first time I've done any work with the SPKGs, and I'm not sure if I've followed the conventions correctly, so I didn't want to overwrite your branch in this ticket. For example, I wasn't sure how/whether to include these run-time dependencies in the build/pkgs/{jsonschema|pyrsistent}/dependencies files.

With these packages present, the Jupyter server starts up and notebooks can be opened, edited, and ran just fine. Threejs plots are working including the new animation stuff, but JSmol plots are not. Nor does MathJax function correctly. I see 404 status codes for /nbextensions/mathjax/MathJax.js?config=TeX-AMS-MML_HTMLorMML-full,Safe&delayStartupUntil=configured and /nbextensions/jsmol/JSmol.min.js, which are perhaps related to some of the other outstanding Jupyter tickets.

comment:38 Changed 2 months ago by mkoeppe

Thanks a lot! Please do update the branch, it will make it easier

comment:39 follow-up: Changed 2 months ago by mkoeppe

Yes, runtime dependencies should be added in the dependencies file. We do not really distinguish build dependencies and runtime dependencies in the Sage distribution. (There is a notion of order-only dependencies, which are listed after the vertical-bar symbol (|), but it's OK to ignore.)

comment:40 in reply to: ↑ 37 ; follow-up: Changed 2 months ago by mkoeppe

Replying to gh-jcamp0x2a:

I see 404 status codes for /nbextensions/mathjax/MathJax.js?config=TeX-AMS-MML_HTMLorMML-full,Safe&delayStartupUntil=configured and /nbextensions/jsmol/JSmol.min.js, which are perhaps related to some of the other outstanding Jupyter tickets.

OK, I forgot to push the merge with #30299 -- sorry about that

comment:41 Changed 2 months ago by git

  • Commit changed from 0e1c3258bdb73cdd403daabb0c52d4bb5861269c to 0a6487cd2eaac6c5b57792835a9899bbb437eb56

Branch pushed to git repo; I updated commit sha1. New commits:

113625bbuild/pkgs/ipywidgets: Update to 7.5.1
79317b8build/pkgs/sage_conf/src/sage_conf.py.in: Add JSMOL_DIR, MATHJAX_DIR, THREEJS_DIR
0a6487cMerge branch 't/30299/minimal_fix_for_broken_jupyter_notebook' into t/26919/jupyter-notebook-6.1.1

comment:42 Changed 2 months ago by git

  • Commit changed from 0a6487cd2eaac6c5b57792835a9899bbb437eb56 to af85282a8b93938208c8148d371522a5a4202530

Branch pushed to git repo; I updated commit sha1. New commits:

1414916Add additional run-time dependencies of jsonschema
af85282Merge commit '14149164c43758b8f2845630df96b89dbe1f1c79' of git://trac.sagemath.org/sage into t/26919/jupyter-notebook-6.1.1

comment:43 Changed 2 months ago by mkoeppe

I've merged your commit. Please test again...

comment:44 Changed 2 months ago by mkoeppe

  • Description modified (diff)

comment:46 in reply to: ↑ 39 Changed 2 months ago by gh-jcamp0x2a

Replying to mkoeppe:

Yes, runtime dependencies should be added in the dependencies file. We do not really distinguish build dependencies and runtime dependencies in the Sage distribution. (There is a notion of order-only dependencies, which are listed after the vertical-bar symbol (|), but it's OK to ignore.)

Ok, I've added dependencies on attrs, importlib_metadata, and pyrsistent to jsonschema and a dependency on zipp to importlib_metadata. (I mispoke earlier that it was required by pyrsistent.) I will update the ticket with the branch.

Also, I want to mention that the latest version of zipp (3.1.0) did not build correctly in the Sage environment, and I didn't do much troubleshooting on it, falling back instead to the specific major and minor version called out by importlib_metadata (0.5, so I used 0.5.2). Just in case that comes up in the future.

comment:47 in reply to: ↑ 40 Changed 2 months ago by gh-jcamp0x2a

Replying to mkoeppe:

Replying to gh-jcamp0x2a:

I see 404 status codes for /nbextensions/mathjax/MathJax.js?config=TeX-AMS-MML_HTMLorMML-full,Safe&delayStartupUntil=configured and /nbextensions/jsmol/JSmol.min.js, which are perhaps related to some of the other outstanding Jupyter tickets.

OK, I forgot to push the merge with #30299 -- sorry about that

I've tested this again following your merge and both JSmol and MathJax are now working correctly.

comment:48 Changed 2 months ago by gh-jcamp0x2a

  • Branch changed from u/mkoeppe/notebook-5.7.8 to u/gh-jcamp0x2a/notebook-5.7.8
  • Commit changed from af85282a8b93938208c8148d371522a5a4202530 to d7d995ddc8cf9a281b4cf22839986330ab95eea2

comment:49 Changed 2 months ago by mkoeppe

  • Authors changed from Matthias Koeppe to Matthias Koeppe, Joshua Campbell

Thanks very much! Looking great.

I have restarted the build tests, now at https://github.com/mkoeppe/sage/actions/runs/201611495

comment:50 Changed 2 months ago by mkoeppe

  • Reviewers set to ​https://github.com/mkoeppe/sage/actions/runs/201611495

comment:51 Changed 2 months ago by mkoeppe

  • Reviewers changed from ​https://github.com/mkoeppe/sage/actions/runs/201611495 to Joshua Campbell, ​https://github.com/mkoeppe/sage/actions/runs/201611495

comment:52 Changed 2 months ago by mkoeppe

  • Reviewers changed from Joshua Campbell, ​https://github.com/mkoeppe/sage/actions/runs/201611495 to Joshua Campbell, ​github.com/mkoeppe/sage/actions/runs/201611495, ...

Clean tests for slackware-14.2-standard (https://github.com/mkoeppe/sage/runs/964353866), gentoo-standard (https://github.com/mkoeppe/sage/runs/964353835), fedora-32-standard (https://github.com/mkoeppe/sage/runs/964353804). (Tests on many other platforms did not go through because of current reliability issues of GH Action - unrelated to this ticket.)

comment:53 Changed 2 months ago by mkoeppe

  • Reviewers changed from Joshua Campbell, ​github.com/mkoeppe/sage/actions/runs/201611495, ... to Joshua Campbell, Matthias Koeppe, ​github.com/mkoeppe/sage/actions/runs/201611495, ...

comment:54 Changed 2 months ago by mkoeppe

  • Branch changed from u/gh-jcamp0x2a/notebook-5.7.8 to u/mkoeppe/notebook-5.7.8

comment:55 Changed 2 months ago by mkoeppe

  • Commit changed from d7d995ddc8cf9a281b4cf22839986330ab95eea2 to 445c1d7a5f90ddb6af5b33e1d03697cfc3eefc54
  • Dependencies changed from #28197, #30299 to #28197, #30299, #25363

Merged #25363 (and 9.2.beta8) to facilitate testing


New commits:

3b8b2bdtrac 25363: implement "sage --simple-prompt"
445c1d7Merge branch 'u/jhpalmieri/ticket-25363' of git://trac.sagemath.org/sage into t/26919/jupyter-notebook-6.1.1

comment:56 Changed 2 months ago by dimpase

  • Reviewers changed from Joshua Campbell, Matthias Koeppe, ​github.com/mkoeppe/sage/actions/runs/201611495, ... to Joshua Campbell, Matthias Koeppe, ​github.com/mkoeppe/sage/actions/runs/201611495, Dima Pasechnik
  • Status changed from needs_review to positive_review

Great!

comment:57 Changed 2 months ago by mkoeppe

  • Reviewers changed from Joshua Campbell, Matthias Koeppe, ​github.com/mkoeppe/sage/actions/runs/201611495, Dima Pasechnik to Joshua Campbell, Matthias Koeppe, Dima Pasechnik

Thank you!

comment:58 Changed 2 months ago by vbraun

  • Branch changed from u/mkoeppe/notebook-5.7.8 to 445c1d7a5f90ddb6af5b33e1d03697cfc3eefc54
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.