Opened 4 years ago
Closed 2 years ago
#26919 closed enhancement (fixed)
Upgrade Jupyter notebook to latest (6.1.1) and its dependencies to latest
Reported by:  Samuel Lelièvre  Owned by:  

Priority:  critical  Milestone:  sage9.2 
Component:  packages: standard  Keywords:  upgrade, jupyter, notebook 
Cc:  Antonio Rojas, François Bissey, Bryan Ginge Chen, Timo Kaufmann, Ximin Luo, Jeroen Demeyer, Samuel Lelièvre, Tobias Hansen, Eric Gourgoulhon  Merged in:  
Authors:  Matthias Koeppe, Joshua Campbell  Reviewers:  Joshua Campbell, Matthias Koeppe, Dima Pasechnik 
Report Upstream:  N/A  Work issues:  
Branch:  445c1d7 (Commits, GitHub, GitLab)  Commit:  445c1d7a5f90ddb6af5b33e1d03697cfc3eefc54 
Dependencies:  #28197, #30299, #25363  Stopgaps: 
Description (last modified by )
Upgrade:
 https://pypi.org/project/notebook/ 6.1.1 (current as of 20200807, supports python >= 3.5)
New dependencies:
 argon2cffi (?)
 pyrsistent, attrs, importlib_metadata, zipp
Upgrade of its dependencies:
 https://pypi.org/project/nbformat/ 5.0.7 (released 20200609, current as of 20200807, supports python >= 3.5)
 https://pypi.org/project/nbconvert/ 5.6.1 (released 20191024, current as of 20200807, supports python >= 3.5)
 https://pypi.org/project/jupyterclient/ 6.1.6 (released 20200713, current as of 20200807, supports python >= 3.5)
 https://pypi.org/project/jupytercore/ 4.6.3 (released 20200218, current as of 20200807, supports python >= 3.5)
 https://pypi.org/project/Jinja2/ 2.11.2 (released 20200413, current as of 20200807, supports python >= 3.5)
 https://pypi.org/project/tornado/ 6.0.4 (released 20200303, current as of 20200807, supports python >= 3.5)
 https://pypi.org/project/terminado/ 0.8.3 (released 20191112, current as of 20200807, supports python >= 3.4)
 https://pypi.org/project/prometheusclient/ 0.8.0 (released 20200525, current as of 20200807, supports python >= 3.4)
 https://pypi.org/project/pyzmq/ 19.0.2 (released 20200731, current as of 20200807, supports python >= 3.3)
Recursive dependencies:
 https://pypi.org/project/jsonschema/ 3.2.0 (released 20191118, current as of 20200807, supports python >= 2.7)
 bleach 3.1.5
 testpath 0.4.4
Upgrade of a reverse dependency:
 https://pypi.org/project/widgetsnbextension/ 3.5.1 (released 20190729, supports python >= 2.7)
 ipywidgets 7.5.1
(tarball URLs: see checksums.ini
; use ./configure enabledownloadfromupstreamurl
to enable automatic downloads of the new versions)
Previous upgrades:
See also:
Change History (58)
comment:1 Changed 4 years ago by
Milestone:  sage8.6 → sage8.7 

comment:2 Changed 4 years ago by
Milestone:  sage8.7 → sage8.8 

Moving all blocker/critical issues from 8.7 to 8.8.
comment:3 Changed 4 years ago by
Description:  modified (diff) 

Summary:  Upgrade to Jupyter notebook 5.7.4 → Upgrade to Jupyter notebook 5.7.7 
comment:4 Changed 4 years ago by
Description:  modified (diff) 

Summary:  Upgrade to Jupyter notebook 5.7.7 → Upgrade to Jupyter notebook 5.7.8 
comment:5 Changed 4 years ago by
Branch:  → u/slelievre/notebook5.7.8 

comment:6 Changed 4 years ago by
Commit:  → b1a4b7fd4a215fb1340956b6a9e0c15d8ea0c760 

Status:  new → needs_review 
New commits:
b1a4b7f  Upgrade to notebook 5.7.8

comment:7 Changed 4 years ago by
Note, this would conflict slightly with #27715. As this ticket came first I'd be happy to rebase onto it so long as do intend to move forward on this ticket (I don't see why we shouldn't, though I haven't tested it yet...)
comment:8 Changed 3 years ago by
Milestone:  sage8.8 → sage8.9 

Moving tickets from the Sage 8.8 milestone that have been actively worked on in the last six months to the next release milestone (optimistically).
comment:10 Changed 3 years ago by
Status:  needs_review → needs_work 

comment:11 Changed 3 years ago by
Milestone:  sage8.9 → sage9.1 

Ticket retargeted after milestone closed
comment:12 Changed 3 years ago by
Milestone:  sage9.1 → sage9.2 

comment:13 Changed 2 years ago by
Notebook 6.1.0 RC1 is out:
Notebook 5.7.10 would be the last py2compatible release.
comment:14 Changed 2 years ago by
py2 compatibility is irrelevant now. Please see #29141 for discussion what we need to be compatible with
comment:16 Changed 2 years ago by
Summary:  Upgrade to Jupyter notebook 5.7.8 → Upgrade to Jupyter notebook 5.7.10 or 6.1.0 .... 

comment:18 Changed 2 years ago by
Cc:  Eric Gourgoulhon added 

Description:  modified (diff) 
comment:19 Changed 2 years ago by
Dependencies:  → #28197 

Description:  modified (diff) 
Summary:  Upgrade to Jupyter notebook 5.7.10 or 6.1.0 .... → Upgrade Jupyter notebook to latest (6.1.1) and its dependencies to latest 
comment:20 Changed 2 years ago by
Branch:  u/slelievre/notebook5.7.8 → u/mkoeppe/notebook5.7.8 

comment:21 Changed 2 years ago by
Commit:  b1a4b7fd4a215fb1340956b6a9e0c15d8ea0c760 → 6ad6e9b81d1e35aa982ab024a5bf8f7b9ed8d2f0 

[notebook6.1.1] Applying patches from ../patches... [notebook6.1.1] Applying ../patches/0001Workaroundforwrongerrnoonsocketbindpermission.patch [notebook6.1.1] patching file notebook/notebookapp.py [notebook6.1.1] Hunk #1 FAILED at 1406.
It should be checked if this patch has been upstreamed
Last 10 new commits:
3205576  build/pkgs/notebook: Update to 6.1.1

ef9a90d  build/pkgs/nbformat: Update to 5.0.7

072ea38  build/pkgs/nbconvert: Update to 5.6.1

712a5b6  build/pkgs/jupyter_client: Update to 6.1.6

00373c6  build/pkgs/jinja2: Update to 2.11.2

7e1b5c5  build/pkgs/tornado: Update to 6.1.1

7d73f70  build/pkgs/terminado: Update to 0.8.3

c4b19e8  build/pkgs/prometheus_client: Update to 0.8.0

9075a16  build/pkgs/jupyter_core: Update to 4.6.3

6ad6e9b  build/pkgs/pyzmq: Update to 19.0.2

comment:22 Changed 2 years ago by
The patch was added for Cygwin in #27715 (Jupyter notebook can crash during startup if default port cannot be used)
comment:23 Changed 2 years ago by
Commit:  6ad6e9b81d1e35aa982ab024a5bf8f7b9ed8d2f0 → 26691af0ce7bf0f96a12bc676037e2bda4616f97 

Branch pushed to git repo; I updated commit sha1. New commits:
26691af  build/pkgs/notebook: Remove upstreamed patch (PR 4584)

comment:24 Changed 2 years ago by
Status:  needs_work → needs_review 

Builds OK for me on macOS. I have not tested anything else because I do not use the jupyter notebook and cannot distinguish a working and a broken installation.
comment:25 Changed 2 years ago by
Description:  modified (diff) 

comment:26 Changed 2 years ago by
Authors:  → Matthias Koeppe 

comment:27 Changed 2 years ago by
Commit:  26691af0ce7bf0f96a12bc676037e2bda4616f97 → 4762704eb5a089237b18948b85e3171208ee23fd 

Branch pushed to git repo; I updated commit sha1. New commits:
4762704  build/pkgs/widgetsnbextension: Update to 3.5.1

comment:28 Changed 2 years ago by
Description:  modified (diff) 

comment:29 Changed 2 years ago by
Description:  modified (diff) 

comment:30 Changed 2 years ago by
Commit:  4762704eb5a089237b18948b85e3171208ee23fd → 0e1c3258bdb73cdd403daabb0c52d4bb5861269c 

comment:31 Changed 2 years ago by
Description:  modified (diff) 

comment:32 Changed 2 years ago by
Description:  modified (diff) 

comment:33 Changed 2 years ago by
Description:  modified (diff) 

comment:34 Changed 2 years ago by
Description:  modified (diff) 

comment:35 Changed 2 years ago by
Description:  modified (diff) 

comment:36 Changed 2 years ago by
Dependencies:  #28197 → #28197, #30299 

comment:37 followup: 40 Changed 2 years ago by
Hello! Saw your post on sagedevel and figured I'd take a shot at testing this.
I was able to get this branch to build just fine, but I ran into some missing runtime dependencies for jsonschema
, namely: attrs
, importlib_metadata
, and pyrsistent
(which then required zipp
).
I've pushed a commit with these packages added. This is the first time I've done any work with the SPKGs, and I'm not sure if I've followed the conventions correctly, so I didn't want to overwrite your branch in this ticket. For example, I wasn't sure how/whether to include these runtime dependencies in the build/pkgs/{jsonschemapyrsistent}/dependencies
files.
With these packages present, the Jupyter server starts up and notebooks can be opened, edited, and ran just fine. Threejs plots are working including the new animation stuff, but JSmol plots are not. Nor does MathJax function correctly. I see 404 status codes for /nbextensions/mathjax/MathJax.js?config=TeXAMSMML_HTMLorMMLfull,Safe&delayStartupUntil=configured
and /nbextensions/jsmol/JSmol.min.js
, which are perhaps related to some of the other outstanding Jupyter tickets.
comment:39 followup: 46 Changed 2 years ago by
Yes, runtime dependencies should be added in the dependencies
file. We do not really distinguish build dependencies and runtime dependencies in the Sage distribution. (There is a notion of orderonly dependencies, which are listed after the verticalbar symbol (), but it's OK to ignore.)
comment:40 followup: 47 Changed 2 years ago by
Replying to ghjcamp0x2a:
I see 404 status codes for
/nbextensions/mathjax/MathJax.js?config=TeXAMSMML_HTMLorMMLfull,Safe&delayStartupUntil=configured
and/nbextensions/jsmol/JSmol.min.js
, which are perhaps related to some of the other outstanding Jupyter tickets.
OK, I forgot to push the merge with #30299  sorry about that
comment:41 Changed 2 years ago by
Commit:  0e1c3258bdb73cdd403daabb0c52d4bb5861269c → 0a6487cd2eaac6c5b57792835a9899bbb437eb56 

Branch pushed to git repo; I updated commit sha1. New commits:
113625b  build/pkgs/ipywidgets: Update to 7.5.1

79317b8  build/pkgs/sage_conf/src/sage_conf.py.in: Add JSMOL_DIR, MATHJAX_DIR, THREEJS_DIR

0a6487c  Merge branch 't/30299/minimal_fix_for_broken_jupyter_notebook' into t/26919/jupyternotebook6.1.1

comment:42 Changed 2 years ago by
Commit:  0a6487cd2eaac6c5b57792835a9899bbb437eb56 → af85282a8b93938208c8148d371522a5a4202530 

comment:44 Changed 2 years ago by
Description:  modified (diff) 

comment:45 Changed 2 years ago by
A build test is running at https://github.com/mkoeppe/sage/actions/runs/201573039
comment:46 Changed 2 years ago by
Replying to mkoeppe:
Yes, runtime dependencies should be added in the
dependencies
file. We do not really distinguish build dependencies and runtime dependencies in the Sage distribution. (There is a notion of orderonly dependencies, which are listed after the verticalbar symbol (), but it's OK to ignore.)
Ok, I've added dependencies on attrs
, importlib_metadata
, and pyrsistent
to jsonschema
and a dependency on zipp
to importlib_metadata
. (I mispoke earlier that it was required by pyrsistent
.) I will update the ticket with the branch.
Also, I want to mention that the latest version of zipp
(3.1.0) did not build correctly in the Sage environment, and I didn't do much troubleshooting on it, falling back instead to the specific major and minor version called out by importlib_metadata
(0.5, so I used 0.5.2). Just in case that comes up in the future.
comment:47 Changed 2 years ago by
Replying to mkoeppe:
Replying to ghjcamp0x2a:
I see 404 status codes for
/nbextensions/mathjax/MathJax.js?config=TeXAMSMML_HTMLorMMLfull,Safe&delayStartupUntil=configured
and/nbextensions/jsmol/JSmol.min.js
, which are perhaps related to some of the other outstanding Jupyter tickets.OK, I forgot to push the merge with #30299  sorry about that
I've tested this again following your merge and both JSmol and MathJax are now working correctly.
comment:48 Changed 2 years ago by
Branch:  u/mkoeppe/notebook5.7.8 → u/ghjcamp0x2a/notebook5.7.8 

Commit:  af85282a8b93938208c8148d371522a5a4202530 → d7d995ddc8cf9a281b4cf22839986330ab95eea2 
comment:49 Changed 2 years ago by
Authors:  Matthias Koeppe → Matthias Koeppe, Joshua Campbell 

Thanks very much! Looking great.
I have restarted the build tests, now at https://github.com/mkoeppe/sage/actions/runs/201611495
comment:50 Changed 2 years ago by
Reviewers:  → https://github.com/mkoeppe/sage/actions/runs/201611495 

comment:51 Changed 2 years ago by
Reviewers:  https://github.com/mkoeppe/sage/actions/runs/201611495 → Joshua Campbell, https://github.com/mkoeppe/sage/actions/runs/201611495 

comment:52 Changed 2 years ago by
Reviewers:  Joshua Campbell, https://github.com/mkoeppe/sage/actions/runs/201611495 → Joshua Campbell, github.com/mkoeppe/sage/actions/runs/201611495, ... 

Clean tests for slackware14.2standard
(https://github.com/mkoeppe/sage/runs/964353866), gentoostandard
(https://github.com/mkoeppe/sage/runs/964353835), fedora32standard
(https://github.com/mkoeppe/sage/runs/964353804). (Tests on many other platforms did not go through because of current reliability issues of GH Action  unrelated to this ticket.)
comment:53 Changed 2 years ago by
Reviewers:  Joshua Campbell, github.com/mkoeppe/sage/actions/runs/201611495, ... → Joshua Campbell, Matthias Koeppe, github.com/mkoeppe/sage/actions/runs/201611495, ... 

comment:54 Changed 2 years ago by
Branch:  u/ghjcamp0x2a/notebook5.7.8 → u/mkoeppe/notebook5.7.8 

comment:55 Changed 2 years ago by
Commit:  d7d995ddc8cf9a281b4cf22839986330ab95eea2 → 445c1d7a5f90ddb6af5b33e1d03697cfc3eefc54 

Dependencies:  #28197, #30299 → #28197, #30299, #25363 
comment:56 Changed 2 years ago by
Reviewers:  Joshua Campbell, Matthias Koeppe, github.com/mkoeppe/sage/actions/runs/201611495, ... → Joshua Campbell, Matthias Koeppe, github.com/mkoeppe/sage/actions/runs/201611495, Dima Pasechnik 

Status:  needs_review → positive_review 
Great!
comment:57 Changed 2 years ago by
Reviewers:  Joshua Campbell, Matthias Koeppe, github.com/mkoeppe/sage/actions/runs/201611495, Dima Pasechnik → Joshua Campbell, Matthias Koeppe, Dima Pasechnik 

Thank you!
comment:58 Changed 2 years ago by
Branch:  u/mkoeppe/notebook5.7.8 → 445c1d7a5f90ddb6af5b33e1d03697cfc3eefc54 

Resolution:  → fixed 
Status:  positive_review → closed 
Retarging tickets optimistically to the next milestone. If you are responsible for this ticket (either its reporter or owner) and don't believe you are likely to complete this ticket before the next release (8.7) please retarget this ticket's milestone to sagepending or sagewishlist.