Opened 22 months ago
Closed 2 months ago
#26919 closed enhancement (fixed)
Upgrade Jupyter notebook to latest (6.1.1) and its dependencies to latest
Reported by:  slelievre  Owned by:  

Priority:  critical  Milestone:  sage9.2 
Component:  packages: standard  Keywords:  upgrade, jupyter, notebook 
Cc:  arojas, fbissey, ghbryangingechen, ghtimokau, infinity0, jdemeyer, slelievre, thansen, egourgoulhon  Merged in:  
Authors:  Matthias Koeppe, Joshua Campbell  Reviewers:  Joshua Campbell, Matthias Koeppe, Dima Pasechnik 
Report Upstream:  N/A  Work issues:  
Branch:  445c1d7 (Commits)  Commit:  445c1d7a5f90ddb6af5b33e1d03697cfc3eefc54 
Dependencies:  #28197, #30299, #25363  Stopgaps: 
Description (last modified by )
Upgrade:
 https://pypi.org/project/notebook/ 6.1.1 (current as of 20200807, supports python >= 3.5)
New dependencies:
 argon2cffi (?)
 pyrsistent, attrs, importlib_metadata, zipp
Upgrade of its dependencies:
 https://pypi.org/project/nbformat/ 5.0.7 (released 20200609, current as of 20200807, supports python >= 3.5)
 https://pypi.org/project/nbconvert/ 5.6.1 (released 20191024, current as of 20200807, supports python >= 3.5)
 https://pypi.org/project/jupyterclient/ 6.1.6 (released 20200713, current as of 20200807, supports python >= 3.5)
 https://pypi.org/project/jupytercore/ 4.6.3 (released 20200218, current as of 20200807, supports python >= 3.5)
 https://pypi.org/project/Jinja2/ 2.11.2 (released 20200413, current as of 20200807, supports python >= 3.5)
 https://pypi.org/project/tornado/ 6.0.4 (released 20200303, current as of 20200807, supports python >= 3.5)
 https://pypi.org/project/terminado/ 0.8.3 (released 20191112, current as of 20200807, supports python >= 3.4)
 https://pypi.org/project/prometheusclient/ 0.8.0 (released 20200525, current as of 20200807, supports python >= 3.4)
 https://pypi.org/project/pyzmq/ 19.0.2 (released 20200731, current as of 20200807, supports python >= 3.3)
Recursive dependencies:
 https://pypi.org/project/jsonschema/ 3.2.0 (released 20191118, current as of 20200807, supports python >= 2.7)
 bleach 3.1.5
 testpath 0.4.4
Upgrade of a reverse dependency:
 https://pypi.org/project/widgetsnbextension/ 3.5.1 (released 20190729, supports python >= 2.7)
 ipywidgets 7.5.1
(tarball URLs: see checksums.ini
; use ./configure enabledownloadfromupstreamurl
to enable automatic downloads of the new versions)
Previous upgrades:
See also:
Change History (58)
comment:1 Changed 21 months ago by
 Milestone changed from sage8.6 to sage8.7
comment:2 Changed 19 months ago by
 Milestone changed from sage8.7 to sage8.8
Moving all blocker/critical issues from 8.7 to 8.8.
comment:3 Changed 19 months ago by
 Description modified (diff)
 Summary changed from Upgrade to Jupyter notebook 5.7.4 to Upgrade to Jupyter notebook 5.7.7
comment:4 Changed 19 months ago by
 Description modified (diff)
 Summary changed from Upgrade to Jupyter notebook 5.7.7 to Upgrade to Jupyter notebook 5.7.8
comment:5 Changed 19 months ago by
 Branch set to u/slelievre/notebook5.7.8
comment:6 Changed 19 months ago by
 Commit set to b1a4b7fd4a215fb1340956b6a9e0c15d8ea0c760
 Status changed from new to needs_review
New commits:
b1a4b7f  Upgrade to notebook 5.7.8

comment:7 Changed 18 months ago by
Note, this would conflict slightly with #27715. As this ticket came first I'd be happy to rebase onto it so long as do intend to move forward on this ticket (I don't see why we shouldn't, though I haven't tested it yet...)
comment:8 Changed 16 months ago by
 Milestone changed from sage8.8 to sage8.9
Moving tickets from the Sage 8.8 milestone that have been actively worked on in the last six months to the next release milestone (optimistically).
comment:9 Changed 14 months ago by
branch is red
comment:10 Changed 14 months ago by
 Status changed from needs_review to needs_work
comment:11 Changed 10 months ago by
 Milestone changed from sage8.9 to sage9.1
Ticket retargeted after milestone closed
comment:12 Changed 6 months ago by
 Milestone changed from sage9.1 to sage9.2
comment:13 Changed 3 months ago by
Notebook 6.1.0 RC1 is out:
Notebook 5.7.10 would be the last py2compatible release.
comment:14 Changed 3 months ago by
py2 compatibility is irrelevant now. Please see #29141 for discussion what we need to be compatible with
comment:15 Changed 3 months ago by
Notebook 6.1.0 released.
comment:16 Changed 3 months ago by
 Summary changed from Upgrade to Jupyter notebook 5.7.8 to Upgrade to Jupyter notebook 5.7.10 or 6.1.0 ....
comment:17 Changed 3 months ago by
See also: #28313 "notebook5.7.6 contains local files"
comment:18 Changed 3 months ago by
 Cc egourgoulhon added
 Description modified (diff)
comment:19 Changed 2 months ago by
 Dependencies set to #28197
 Description modified (diff)
 Summary changed from Upgrade to Jupyter notebook 5.7.10 or 6.1.0 .... to Upgrade Jupyter notebook to latest (6.1.1) and its dependencies to latest
comment:20 Changed 2 months ago by
 Branch changed from u/slelievre/notebook5.7.8 to u/mkoeppe/notebook5.7.8
comment:21 Changed 2 months ago by
 Commit changed from b1a4b7fd4a215fb1340956b6a9e0c15d8ea0c760 to 6ad6e9b81d1e35aa982ab024a5bf8f7b9ed8d2f0
[notebook6.1.1] Applying patches from ../patches... [notebook6.1.1] Applying ../patches/0001Workaroundforwrongerrnoonsocketbindpermission.patch [notebook6.1.1] patching file notebook/notebookapp.py [notebook6.1.1] Hunk #1 FAILED at 1406.
It should be checked if this patch has been upstreamed
Last 10 new commits:
3205576  build/pkgs/notebook: Update to 6.1.1

ef9a90d  build/pkgs/nbformat: Update to 5.0.7

072ea38  build/pkgs/nbconvert: Update to 5.6.1

712a5b6  build/pkgs/jupyter_client: Update to 6.1.6

00373c6  build/pkgs/jinja2: Update to 2.11.2

7e1b5c5  build/pkgs/tornado: Update to 6.1.1

7d73f70  build/pkgs/terminado: Update to 0.8.3

c4b19e8  build/pkgs/prometheus_client: Update to 0.8.0

9075a16  build/pkgs/jupyter_core: Update to 4.6.3

6ad6e9b  build/pkgs/pyzmq: Update to 19.0.2

comment:22 Changed 2 months ago by
The patch was added for Cygwin in #27715 (Jupyter notebook can crash during startup if default port cannot be used)
comment:23 Changed 2 months ago by
 Commit changed from 6ad6e9b81d1e35aa982ab024a5bf8f7b9ed8d2f0 to 26691af0ce7bf0f96a12bc676037e2bda4616f97
Branch pushed to git repo; I updated commit sha1. New commits:
26691af  build/pkgs/notebook: Remove upstreamed patch (PR 4584)

comment:24 Changed 2 months ago by
 Status changed from needs_work to needs_review
Builds OK for me on macOS. I have not tested anything else because I do not use the jupyter notebook and cannot distinguish a working and a broken installation.
comment:25 Changed 2 months ago by
 Description modified (diff)
comment:26 Changed 2 months ago by
comment:27 Changed 2 months ago by
 Commit changed from 26691af0ce7bf0f96a12bc676037e2bda4616f97 to 4762704eb5a089237b18948b85e3171208ee23fd
Branch pushed to git repo; I updated commit sha1. New commits:
4762704  build/pkgs/widgetsnbextension: Update to 3.5.1

comment:28 Changed 2 months ago by
 Description modified (diff)
comment:29 Changed 2 months ago by
 Description modified (diff)
comment:30 Changed 2 months ago by
 Commit changed from 4762704eb5a089237b18948b85e3171208ee23fd to 0e1c3258bdb73cdd403daabb0c52d4bb5861269c
comment:31 Changed 2 months ago by
 Description modified (diff)
comment:32 Changed 2 months ago by
 Description modified (diff)
comment:33 Changed 2 months ago by
 Description modified (diff)
comment:34 Changed 2 months ago by
 Description modified (diff)
comment:35 Changed 2 months ago by
 Description modified (diff)
comment:36 Changed 2 months ago by
 Dependencies changed from #28197 to #28197, #30299
comment:37 followup: ↓ 40 Changed 2 months ago by
Hello! Saw your post on sagedevel and figured I'd take a shot at testing this.
I was able to get this branch to build just fine, but I ran into some missing runtime dependencies for jsonschema
, namely: attrs
, importlib_metadata
, and pyrsistent
(which then required zipp
).
I've pushed a commit with these packages added. This is the first time I've done any work with the SPKGs, and I'm not sure if I've followed the conventions correctly, so I didn't want to overwrite your branch in this ticket. For example, I wasn't sure how/whether to include these runtime dependencies in the build/pkgs/{jsonschemapyrsistent}/dependencies
files.
With these packages present, the Jupyter server starts up and notebooks can be opened, edited, and ran just fine. Threejs plots are working including the new animation stuff, but JSmol plots are not. Nor does MathJax function correctly. I see 404 status codes for /nbextensions/mathjax/MathJax.js?config=TeXAMSMML_HTMLorMMLfull,Safe&delayStartupUntil=configured
and /nbextensions/jsmol/JSmol.min.js
, which are perhaps related to some of the other outstanding Jupyter tickets.
comment:38 Changed 2 months ago by
Thanks a lot! Please do update the branch, it will make it easier
comment:39 followup: ↓ 46 Changed 2 months ago by
Yes, runtime dependencies should be added in the dependencies
file. We do not really distinguish build dependencies and runtime dependencies in the Sage distribution. (There is a notion of orderonly dependencies, which are listed after the verticalbar symbol (), but it's OK to ignore.)
comment:40 in reply to: ↑ 37 ; followup: ↓ 47 Changed 2 months ago by
Replying to ghjcamp0x2a:
I see 404 status codes for
/nbextensions/mathjax/MathJax.js?config=TeXAMSMML_HTMLorMMLfull,Safe&delayStartupUntil=configured
and/nbextensions/jsmol/JSmol.min.js
, which are perhaps related to some of the other outstanding Jupyter tickets.
OK, I forgot to push the merge with #30299  sorry about that
comment:41 Changed 2 months ago by
 Commit changed from 0e1c3258bdb73cdd403daabb0c52d4bb5861269c to 0a6487cd2eaac6c5b57792835a9899bbb437eb56
Branch pushed to git repo; I updated commit sha1. New commits:
113625b  build/pkgs/ipywidgets: Update to 7.5.1

79317b8  build/pkgs/sage_conf/src/sage_conf.py.in: Add JSMOL_DIR, MATHJAX_DIR, THREEJS_DIR

0a6487c  Merge branch 't/30299/minimal_fix_for_broken_jupyter_notebook' into t/26919/jupyternotebook6.1.1

comment:42 Changed 2 months ago by
 Commit changed from 0a6487cd2eaac6c5b57792835a9899bbb437eb56 to af85282a8b93938208c8148d371522a5a4202530
comment:43 Changed 2 months ago by
I've merged your commit. Please test again...
comment:44 Changed 2 months ago by
 Description modified (diff)
comment:45 Changed 2 months ago by
A build test is running at https://github.com/mkoeppe/sage/actions/runs/201573039
comment:46 in reply to: ↑ 39 Changed 2 months ago by
Replying to mkoeppe:
Yes, runtime dependencies should be added in the
dependencies
file. We do not really distinguish build dependencies and runtime dependencies in the Sage distribution. (There is a notion of orderonly dependencies, which are listed after the verticalbar symbol (), but it's OK to ignore.)
Ok, I've added dependencies on attrs
, importlib_metadata
, and pyrsistent
to jsonschema
and a dependency on zipp
to importlib_metadata
. (I mispoke earlier that it was required by pyrsistent
.) I will update the ticket with the branch.
Also, I want to mention that the latest version of zipp
(3.1.0) did not build correctly in the Sage environment, and I didn't do much troubleshooting on it, falling back instead to the specific major and minor version called out by importlib_metadata
(0.5, so I used 0.5.2). Just in case that comes up in the future.
comment:47 in reply to: ↑ 40 Changed 2 months ago by
Replying to mkoeppe:
Replying to ghjcamp0x2a:
I see 404 status codes for
/nbextensions/mathjax/MathJax.js?config=TeXAMSMML_HTMLorMMLfull,Safe&delayStartupUntil=configured
and/nbextensions/jsmol/JSmol.min.js
, which are perhaps related to some of the other outstanding Jupyter tickets.OK, I forgot to push the merge with #30299  sorry about that
I've tested this again following your merge and both JSmol and MathJax are now working correctly.
comment:48 Changed 2 months ago by
 Branch changed from u/mkoeppe/notebook5.7.8 to u/ghjcamp0x2a/notebook5.7.8
 Commit changed from af85282a8b93938208c8148d371522a5a4202530 to d7d995ddc8cf9a281b4cf22839986330ab95eea2
comment:49 Changed 2 months ago by
Thanks very much! Looking great.
I have restarted the build tests, now at https://github.com/mkoeppe/sage/actions/runs/201611495
comment:50 Changed 2 months ago by
 Reviewers set to https://github.com/mkoeppe/sage/actions/runs/201611495
comment:51 Changed 2 months ago by
 Reviewers changed from https://github.com/mkoeppe/sage/actions/runs/201611495 to Joshua Campbell, https://github.com/mkoeppe/sage/actions/runs/201611495
comment:52 Changed 2 months ago by
 Reviewers changed from Joshua Campbell, https://github.com/mkoeppe/sage/actions/runs/201611495 to Joshua Campbell, github.com/mkoeppe/sage/actions/runs/201611495, ...
Clean tests for slackware14.2standard
(https://github.com/mkoeppe/sage/runs/964353866), gentoostandard
(https://github.com/mkoeppe/sage/runs/964353835), fedora32standard
(https://github.com/mkoeppe/sage/runs/964353804). (Tests on many other platforms did not go through because of current reliability issues of GH Action  unrelated to this ticket.)
comment:53 Changed 2 months ago by
 Reviewers changed from Joshua Campbell, github.com/mkoeppe/sage/actions/runs/201611495, ... to Joshua Campbell, Matthias Koeppe, github.com/mkoeppe/sage/actions/runs/201611495, ...
comment:54 Changed 2 months ago by
 Branch changed from u/ghjcamp0x2a/notebook5.7.8 to u/mkoeppe/notebook5.7.8
comment:55 Changed 2 months ago by
 Commit changed from d7d995ddc8cf9a281b4cf22839986330ab95eea2 to 445c1d7a5f90ddb6af5b33e1d03697cfc3eefc54
 Dependencies changed from #28197, #30299 to #28197, #30299, #25363
comment:56 Changed 2 months ago by
 Reviewers changed from Joshua Campbell, Matthias Koeppe, github.com/mkoeppe/sage/actions/runs/201611495, ... to Joshua Campbell, Matthias Koeppe, github.com/mkoeppe/sage/actions/runs/201611495, Dima Pasechnik
 Status changed from needs_review to positive_review
Great!
comment:57 Changed 2 months ago by
 Reviewers changed from Joshua Campbell, Matthias Koeppe, github.com/mkoeppe/sage/actions/runs/201611495, Dima Pasechnik to Joshua Campbell, Matthias Koeppe, Dima Pasechnik
Thank you!
comment:58 Changed 2 months ago by
 Branch changed from u/mkoeppe/notebook5.7.8 to 445c1d7a5f90ddb6af5b33e1d03697cfc3eefc54
 Resolution set to fixed
 Status changed from positive_review to closed
Retarging tickets optimistically to the next milestone. If you are responsible for this ticket (either its reporter or owner) and don't believe you are likely to complete this ticket before the next release (8.7) please retarget this ticket's milestone to sagepending or sagewishlist.