Opened 2 years ago

Closed 15 months ago

Last modified 14 months ago

#22431 closed enhancement (fixed)

upgrade sagenb and build sagenb in sage/python3

Reported by: chapoton Owned by:
Priority: major Milestone: sage-8.3
Component: python3 Keywords:
Cc: dimpase, vbraun, jdemeyer, kcrisman, embray Merged in:
Authors: Dima Pasechnik Reviewers: Frédéric Chapoton, Dima Pasechnik
Report Upstream: N/A Work issues:
Branch: ef854c1 (Commits) Commit:
Dependencies: Stopgaps:

Description (last modified by chapoton)

sagenb is not compatible with python3, at least in the current version 1.0.1

The latest release 1.0.2 is certainly slightly better in this respect, but maybe still not fully py3 compliant.

Let us do the upgrade to 1.0.2. We also re-activate the build of sagenb in sagePy3, to help building the doc of sagePy3.

Side remark: to cut the ties between sagenb and sage, we need to get rid:

  • of the cluster interact #24760 and #25074
  • of the mandelbrot interact #24994
  • of the graph editor #?????
  • of the graph database navigator #?????
  • of the interactive debugger #?????
  • of calls to some tab completion function #24998

tarball with sagenb update: https://github.com/sagemath/sagenb/releases/download/1.0.2/sagenb-1.0.2.tar.bz2

Change History (63)

comment:1 Changed 2 years ago by chapoton

  • Cc dimpase vbraun jdemeyer added
  • Description modified (diff)

comment:2 Changed 2 years ago by chapoton

  • Cc kcrisman added

comment:3 Changed 2 years ago by jdemeyer

Given that sagenb is essentially a dead project, I don't think that it will ever be Python3 compliant.

comment:4 Changed 2 years ago by chapoton

I did a lot of work on python3 compliance of sagenb since the last release. At least this should prevent the build to stop on syntax errors.

Last edited 2 years ago by chapoton (previous) (diff)

comment:5 Changed 2 years ago by dimpase

IMHO one does not need supernatural resurrection powers to port a dead python2 project of sagenb size to python3---given that all of its python components are python3 compatible (which was not true for Twisted until a couple of years back).

comment:6 Changed 2 years ago by chapoton

The update of sagenb is probably dependent of #20922.

comment:7 Changed 2 years ago by chapoton

  • Milestone changed from sage-7.6 to sage-8.0

comment:8 Changed 2 years ago by chapoton

can we hope to make a (maybe last) release of the legacy notebook ?

comment:9 Changed 2 years ago by kcrisman

At the least https://github.com/sagemath/sagenb/pull/416 and your own https://github.com/sagemath/sagenb/pull/423 would need to be merged first, right?

I don't really see any reason to not keep the legacy notebook around for quite some time, granting that it will not receive much attention. We just got a support request from someone using Sage 4.6 (!) who really didn't want to upgrade; this is positively new compared to that.

But I definitely agree that whatever the absolute minimum needed to get sagenb to work minimally with python3 is fine.

comment:10 Changed 2 years ago by chapoton

comment:11 follow-up: Changed 2 years ago by chapoton

Would it be possible to make a new release of sagenb, please ?

comment:12 in reply to: ↑ 11 Changed 2 years ago by dimpase

Replying to chapoton:

Would it be possible to make a new release of sagenb, please ?

I'll try.

comment:13 Changed 2 years ago by dimpase

Please test https://github.com/sagemath/sagenb/tree/1.0.rc0 (copy of the current master) before I go ahead with making a new Sage package. It works for me following the instructions in HACKING (updated, to take care of changed names and of the need to deal with mathjax).

Report issues on github, please

Last edited 2 years ago by dimpase (previous) (diff)

comment:14 follow-up: Changed 2 years ago by dimpase

the new sagenb is on #23066 and needs review.

comment:15 in reply to: ↑ 14 ; follow-up: Changed 2 years ago by kcrisman

the new sagenb is on #23066 and needs review.

Reading the comments there and at #22787 make it unclear to me whether this one is, in fact, py3 compatible.

What about comment:6 about #20922 - still valid?

comment:16 in reply to: ↑ 15 ; follow-up: Changed 2 years ago by dimpase

Replying to kcrisman:

the new sagenb is on #23066 and needs review.

Reading the comments there and at #22787 make it unclear to me whether this one is, in fact, py3 compatible.

Given that Sage does not run with Python 3 yet, it's a bit early to say, no?

What about comment:6 about #20922 - still valid?

Probably not really, for it does work without #20922 - I don't get doctest failures mentoined in comment 4 on #20922.

$ sage -t --long local/lib/python2.7/site-packages/sagenb/notebook/challenge.py
Running doctests with ID 2017-05-25-05-57-19-d9e71fa3.
Git branch: sagenb10
Using --optional=ccache,database_gap,gap_packages,mpir,python2,sage
Doctesting 1 file.
sage -t --long --warn-long 117.7 local/lib/python2.7/site-packages/sagenb/notebook/challenge.py
    [113 tests, 0.38 s]
----------------------------------------------------------------------
All tests passed!

I also do not see any mention of click package in sagenb's installation log.

comment:17 in reply to: ↑ 16 Changed 2 years ago by kcrisman

the new sagenb is on #23066 and needs review.

Reading the comments there and at #22787 make it unclear to me whether this one is, in fact, py3 compatible.

Given that Sage does not run with Python 3 yet, it's a bit early to say, no?

Haha, I thought we were closer to that now. No worries, just checking in.

comment:18 Changed 16 months ago by chapoton

  • Description modified (diff)
  • Milestone changed from sage-8.0 to sage-8.3

comment:19 Changed 16 months ago by dimpase

see also https://github.com/sagemath/sagenb/issues/440

  • some of these dependencies should go, indeed.

comment:20 Changed 15 months ago by chapoton

We should also get rid of the sagenb documentation building, which does not work in the python3 sage..

See

https://patchbot.sagemath.org/log/0/Ubuntu/18.04/x86_64/4.15.0-20-generic/petitbonum/2018-05-15%2012:37:45?plugin=docbuild

comment:21 Changed 15 months ago by chapoton

see #25382

comment:22 Changed 15 months ago by kcrisman

What is the current status of this ticket (as opposed to the documentation build)? I thought at least some of these issues were dealt with. If it's just the graph functionality those can certainly be disabled, as I don't think they have worked properly for some time. If it's certain dependencies that is required for py3 that is somewhat different.

comment:23 Changed 15 months ago by chapoton

trying to install sagenb, one gets

[sagenb-1.0.1]   Removing source in /tmp/pip-pzy8td0f-build
[sagenb-1.0.1] Successfully installed sagenb-1.0.1
[sagenb-1.0.1] Cleaning up...
[sagenb-1.0.1] ./spkg-install: line 56: cd: /home/chapoton/sage3/local/lib/python2.7/site-packages/sagenb/data: No such file or directory
[sagenb-1.0.1] Error: Cannot find SageNB data directory.
[sagenb-1.0.1] 

because it is looking for sageNB data in the python2 library..

comment:24 Changed 15 months ago by chapoton

Dima, could you please make a new release of sagenb ?

Currently (apart from the mathjax issue above in spkg-install) sagenb 1.0.1 installs with sage3, but fails to run because of

Traceback (most recent call last):
  File "/home/chapoton/sage3/src/bin/sage-notebook", line 268, in <module>
    launcher(unknown)
  File "/home/chapoton/sage3/src/bin/sage-notebook", line 69, in __init__
    from sagenb.notebook.notebook_object import notebook
  File "/home/chapoton/sage3/local/lib/python3.6/site-packages/sagenb/notebook/notebook_object.py", line 23, in <module>
    from . import run_notebook
  File "/home/chapoton/sage3/local/lib/python3.6/site-packages/sagenb/notebook/run_notebook.py", line 21, in <module>
    from exceptions import SystemExit
ModuleNotFoundError: No module named 'exceptions'

which has been fixed after the release of 1.0.1

comment:25 Changed 15 months ago by dimpase

OK, I'll try today. Sorry for sitting on this...

comment:26 Changed 15 months ago by chapoton

see also #25394

comment:27 Changed 15 months ago by chapoton

  • Description modified (diff)

comment:28 Changed 15 months ago by dimpase

shouldn't the description say "current 1.0.1" ? (sorry, I'm still not done with a new release)

comment:29 Changed 15 months ago by dimpase

  • Description modified (diff)

comment:30 Changed 15 months ago by dimpase

  • Authors set to Dima Pasechnik
  • Branch set to u/dimpase/sagenb/1.0.2
  • Commit set to 64a1b0c301469e36e8c29266b06fcdcb16b51d5f
  • Status changed from new to needs_review

here is the latest master from sagenb repo


New commits:

64a1b0cupdate sagenb to 1.0.2

comment:31 Changed 15 months ago by chapoton

Thanks for the release, Dima. It will not yet be fully py3-compatible, but this should not prevent us from being able to install sagenb in py3-sage.

This second part means : first remove the conditional statement on PYTHON3 in spkg_install (on top of the changes in #25394) and do something for the lines about mathjax in this spkg-install. I managed to install sagenb in py3 sage by commenting out these mathjax lines, but this is obviously not a correct solution.

comment:32 follow-up: Changed 15 months ago by chapoton

I have put a tentative branch at public/ticket/22431 for the changes in spkg-install (it needs work).

comment:33 follow-up: Changed 15 months ago by kcrisman

What would "full py3 compatibility" entail? Just curious. THANK YOU for this work.

comment:34 Changed 15 months ago by git

  • Commit changed from 64a1b0c301469e36e8c29266b06fcdcb16b51d5f to 48976b514838a6d7ebac2c06c839a1f367b76652

Branch pushed to git repo; I updated commit sha1. New commits:

392a2cefor notebooks on python3
855c35fchange spkg-install for sagenb
19a9bedMerge branch 'public/25394' in 8.3.b2
e831b39python3: install sagenb !
48976b5Merge branch 'public/ticket/22431' of trac.sagemath.org:sage into nb102

comment:35 in reply to: ↑ 32 Changed 15 months ago by dimpase

Replying to chapoton:

I have put a tentative branch at public/ticket/22431 for the changes in spkg-install (it needs work).

OK, I've merged this branch into the branch on the ticket.

comment:36 follow-up: Changed 15 months ago by chapoton

  • Cc embray added

Then there probably just remains to understand what to do with the line

ln -s ../../../../../share/mathjax/ mathjax

This way to find the mathjax package seem to be very ugly... Maybe Erik would have an idea for a better replacement that would also work for python3 ?

comment:37 in reply to: ↑ 33 Changed 15 months ago by chapoton

Replying to kcrisman:

What would "full py3 compatibility" entail? Just curious. THANK YOU for this work.

Well, this just means that somebody has to install sagenb on py3-sage (using the branch here and also the hacking instructions for sagenb to work with a git branch for sagenb) and try running "sage -n=sagenb", fixing step by step all errors (in sagenb code) that occurs until the notebook finally starts. This probably implies caring about unicode (str versus bytes). But there could also be other bad surprises as well. I have started to do that, but have very urgent other matters to handle.

comment:38 in reply to: ↑ 36 Changed 15 months ago by jhpalmieri

Replying to chapoton:

Then there probably just remains to understand what to do with the line

ln -s ../../../../../share/mathjax/ mathjax

This way to find the mathjax package seem to be very ugly... Maybe Erik would have an idea for a better replacement that would also work for python3 ?

Why don't we use ln -s $SAGE_LOCAL/share/mathjax mathjax? Or use $SAGE_SHARE/mathjax? And are those any better?

Last edited 15 months ago by jhpalmieri (previous) (diff)

comment:39 Changed 15 months ago by kcrisman

Probably that's what should have been done in the first place, but removing MathJax from sagenb proper was done pretty quickly and this "just worked"? That's my guess.

comment:40 Changed 15 months ago by jhpalmieri

Also, on OS X at least, the command should be ln -s -h ... – as it is, if the mathjax link already exists, then the current command creates another link within that linked folder. It is okay if the sagenb/data directory contains

mathjax -> ../../../../../share/mathjax/

but it doesn't make sense for $SAGE_LOCAL/share/mathjax to contain this.

(On OS X, the documentation for the -h flag says "If the target_file or target_dir is a symbolic link, do not follow it." There is also a -n flag: "Same as -h, for compatibility with other ln implementations.")

comment:41 Changed 15 months ago by chapoton

  • Branch changed from u/dimpase/sagenb/1.0.2 to public/ticket/22431
  • Commit changed from 48976b514838a6d7ebac2c06c839a1f367b76652 to ef854c1f5b263acf2bab8939370f830314e6c3c4

I have modified the ln to mathjax. This seems to work well with python2. Not yet tested with python3.

We need people that test if the new proposed release of sagenb works well enough. Not much has changed since the last release.


New commits:

ef854c1change the ln command to mathjax in sagenb spkg

comment:42 Changed 15 months ago by chapoton

I have checked that with this branch, the install of sagenb is successful with python3 on sage 8.3.b2.

comment:43 Changed 15 months ago by chapoton

ping ?

comment:44 Changed 15 months ago by dimpase

I can try building the latest beta with python3,what should I pull in, besides the branch on this ticket?

comment:45 Changed 15 months ago by chapoton

Nothing else to pull, the develop branch itself build fine with python3.

comment:46 Changed 15 months ago by dimpase

and how do I start sagenb so that the underlying python is python 3?

comment:47 Changed 15 months ago by chapoton

Well, just sage -n=sagenb. But it will fail to start, because of twisted or some of the latest commited changes, not yet in the release here.

We only need to make sure that sage3 builds and starts with the branch here.

We also need to make sure that for sage2, the upgrade of sagenb does not break something inside the notebook.

Last edited 15 months ago by chapoton (previous) (diff)

comment:48 Changed 15 months ago by chapoton

I propose to rename this ticket to "upgrade sagenb and build sagenb in sage/python3"

and keep for another ticket the goal of being able to run sagenb in sage/python3

comment:49 Changed 15 months ago by kcrisman

I was going to try this today on py2 to test the notebook but had some troubles upgrading Sage, see sage-devel.

comment:50 Changed 15 months ago by dimpase

With python3 sage, I get

$ ./sage -n sagenb
SageMath version 8.3.beta3, Release Date: 2018-05-27
Please wait while the old SageNB Notebook server starts...
/home/dima/sagepy3/local/lib/python3.6/site-packages/sage/misc/inline_fortran.py:8: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
Traceback (most recent call last):
  File "/home/dima/sagepy3/src/bin/sage-notebook", line 268, in <module>
    launcher(unknown)
  File "/home/dima/sagepy3/src/bin/sage-notebook", line 74, in __init__
    notebook(*self.args, **self.kwds)
  File "/home/dima/sagepy3/local/lib/python3.6/site-packages/sagenb/notebook/notebook_object.py", line 243, in __call__
    return self.notebook(*args, **kwds)
  File "/home/dima/sagepy3/local/lib/python3.6/site-packages/sagenb/notebook/run_notebook.py", line 535, in notebook_run
    nb = notebook.load_notebook(directory)
  File "/home/dima/sagepy3/local/lib/python3.6/site-packages/sagenb/notebook/notebook.py", line 1839, in load_notebook
    nb = Notebook(dir)
  File "/home/dima/sagepy3/local/lib/python3.6/site-packages/sagenb/notebook/notebook.py", line 151, in __init__
    self.__conf = S.load_server_conf()
  File "/home/dima/sagepy3/local/lib/python3.6/site-packages/sagenb/storage/filesystem_storage.py", line 265, in load_server_conf
    return self._basic_to_server_conf(self._load('conf.pickle'))
  File "/home/dima/sagepy3/local/lib/python3.6/site-packages/sagenb/storage/filesystem_storage.py", line 187, in _load
    result = pickle.load(f)
TypeError: a bytes-like object is required, not 'str'

comment:51 Changed 15 months ago by chapoton

Thanks Dima. It seems that you got a hard time doing that..

Failing is as expected. Maybe this specific issue is already fixed by the pull request done after you made the release. But this is not so important : let us keep that for another ticket.

Do you confirm that sage3 builds and starts with the branch here ? I think I did check that also. So it seems to be good to go from this point of view.

So now, there only remains to see if the branch here is not going to break anything in the sage2 sagenb.

comment:52 Changed 15 months ago by chapoton

I have just checked again with sage3 release 8.3.b4 that sage3 builds and starts.

comment:53 Changed 15 months ago by dimpase

also, for me with beta4. You can give it a positive review, I think.

comment:54 Changed 15 months ago by chapoton

  • Description modified (diff)
  • Reviewers set to Frédéric Chapoton, Dima Pasechnik
  • Status changed from needs_review to positive_review
  • Summary changed from py3: sagenb is not python3 compatible to upgrade sagenb and build sagenb in sage/python3

ok, let it be. I change the title and description to be more adequate to what is done here.

comment:55 Changed 15 months ago by vbraun

  • Branch changed from public/ticket/22431 to ef854c1f5b263acf2bab8939370f830314e6c3c4
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:56 follow-up: Changed 14 months ago by kcrisman

  • Commit ef854c1f5b263acf2bab8939370f830314e6c3c4 deleted

As a followup, in 8.3.beta6 I am having some troubles - see this post for details. I am pretty sure this isn't due to some stale process.

comment:57 in reply to: ↑ 56 ; follow-up: Changed 14 months ago by dimpase

Replying to kcrisman:

As a followup, in 8.3.beta6 I am having some troubles - see this post for details. I am pretty sure this isn't due to some stale process.

Does sagenb still open for you directly, via sage -n sagenb ?

comment:58 in reply to: ↑ 57 Changed 14 months ago by kcrisman

As a followup, in 8.3.beta6 I am having some troubles - see this post for details. I am pretty sure this isn't due to some stale process.

Does sagenb still open for you directly, via sage -n sagenb ?

Yes, or at least with sage -notebook=sagenb it does. I think there is something with the browser + token not opening.

comment:59 follow-up: Changed 14 months ago by chapoton

could you try when undoing #25548 ?

comment:60 in reply to: ↑ 59 ; follow-up: Changed 14 months ago by kcrisman

could you try when undoing #25548 ?

You are oracular.

It would be interesting to trace where exactly quit_sage is even called that it would be necessary ...

comment:61 in reply to: ↑ 60 Changed 14 months ago by kcrisman

could you try when undoing #25548 ?

You are oracular.

Maybe I wasn't clear; that fixes it. Shouldn't we have a blocker for 8.3 that includes that?

comment:62 Changed 14 months ago by dimpase

I can only say that the description of the issue you give it too vague. Please feel free to open a ticket with details on how to reproduce this.

comment:63 Changed 14 months ago by kcrisman

Ok; see #25667.

Note: See TracTickets for help on using tickets.