Opened 12 years ago

# move notebook to flask/wsgi-based notebook — at Version 76

Reported by: Owned by: Jason Grout jason, mpatel, was blocker sage-5.2 notebook sd31 sd35.5 Dan Drake, Jason Grout, Karl-Dieter Crisman, Mike Hansen, Radoslav Kirov, Tim Dumol, Ivan Andrus, Steven Trogdon Mike Hansen, Rado Kirov, William Stein, Jason Grout Rado Kirov, Dan Drake, Jason Grout N/A #11078, #11874, #12229, #11503

This ticket tracks the progress to move the notebook to a flask/wsgi-based notebook.

Relevant code repository: https://github.com/sagemath/sagenb

Explicit instructions including dependencies (replace $SAGE_ROOT with the root sage directory): cd$SAGE_ROOT
./sage -f http://boxen.math.washington.edu/home/jdemeyer/spkg/jmol-12.0.45.p3.spkg
./sage -f http://sage.math.washington.edu/home/jason/sagenb-0.9.0.spkg

./sage -hg qimport http://trac.sagemath.org/sage_trac/raw-attachment/ticket/11874/trac-11874-remove-twisted.patch
./sage -hg qpush

./sage -hg qimport http://trac.sagemath.org/sage_trac/raw-attachment/ticket/11503/trac-11503-jmol-spkg.patch
./sage -hg qpush

cd devel/sage
../../sage -hg qimport http://trac.sagemath.org/sage_trac/raw-attachment/ticket/11078/trac_11078.patch
../../sage -hg qpush

../../sage -hg qimport http://trac.sagemath.org/sage_trac/raw-attachment/ticket/12229/trac-12229-sagenb-developer-doc.2.patch
../../sage -hg qpush

../../sage -hg qimport http://trac.sagemath.org/sage_trac/raw-attachment/ticket/12229/trac-12229-manifest.patch
../../sage -hg qpush

cd ../../
./sage -br


spkg: http://sage.math.washington.edu/home/jason/sagenb-0.9.0.spkg (this includes all dependencies not already in Sage)

### comment:2 Changed 11 years ago by Jonathan Gutow

Description: modified (diff)

### comment:4 Changed 11 years ago by Jonathan Gutow

Description: modified (diff)

### comment:5 Changed 11 years ago by Jonathan Gutow

Description: modified (diff)

### comment:6 Changed 11 years ago by Jonathan Gutow

Description: modified (diff)

removed #11078 from list as the fix is included in the 11503 command line patch.

### comment:7 Changed 11 years ago by Jason Grout

Authors: Mike Hansen, Rado Kirov → Mike Hansen, Rado Kirov, William Stein, Jason Grout → #11078 modified (diff)

### comment:8 Changed 11 years ago by Jason Grout

Description: modified (diff)

### comment:9 Changed 11 years ago by Jason Grout

Description: modified (diff)

### comment:10 Changed 11 years ago by Jason Grout

Dependencies: #11078 → #11078, #11874

### comment:11 Changed 11 years ago by Jason Grout

Status: new → needs_review

### comment:12 Changed 11 years ago by Jason Grout

(for future reference, after this is merged, jmol is a high priority. These two tickets deal with these changes:

• #11496 (update jmol script in local/bin to reflect move of jmol directory to local)
• #11503 (updated spkg and paths for move of jmol directory)
• #9238 (for reference -original Jmol enhancement ticket but heavily laden with pre-flask notebook stuff)

Again, this is only FYI and for further reference.

### comment:13 Changed 11 years ago by Jason Grout

Description: modified (diff)

### comment:14 Changed 11 years ago by Jonathan Gutow

I get the following error when I try to apply the spkg dependencies patch. Everything else seems to work. I'm still checking functionality...

applying /home/jonathan/.sage/temp/VB_Ubuntu_11/26614/tmp_0.patch
unable to find 'spkg/install' for patching
1 out of 1 hunks FAILED -- saving rejects to file spkg/install.rej
unable to find 'spkg/standard/deps' for patching
4 out of 4 hunks FAILED -- saving rejects to file spkg/standard/deps.rej
spkg/install: No such file or directory
spkg/standard/deps: No such file or directory
abort: patch failed to apply


### comment:15 Changed 11 years ago by Jonathan Gutow

With 4.8.alpha4, what is described here does not work. I redid everything. The .spkgs seem to install without error. The patch for this ticket does not work and the notebook fails to launch with the error:

/home/jonathan/Documents/sage-4.8.alpha4/<ipython console> in <module>()

/home/jonathan/Documents/sage-4.8.alpha4/devel/sagenb/sagenb/notebook/notebook_object.py in __call__(self, *args, **kwds)
202     """
203     def __call__(self, *args, **kwds):
--> 204         return self.notebook(*args, **kwds)
205
206     notebook = run_notebook.notebook_twisted

/home/jonathan/Documents/sage-4.8.alpha4/devel/sagenb/sagenb/notebook/run_notebook.py in notebook_twisted(self, directory, port, interface, address, port_tries, secure, reset, require_login, accounts, openid, server_pool, ulimit, timeout, open_viewer, sagetex_path, start_path, fork, quiet, subnets)
442     if open_viewer:
--> 444     return run(port)
445

/home/jonathan/Documents/sage-4.8.alpha4/devel/sagenb/sagenb/notebook/run_notebook.py in run(port)
427         os.chdir(cwd)
428         if e == 256:
--> 429             raise socket.error
430
431         return True



### comment:16 Changed 11 years ago by Jason Grout

It's possible that the SAGE_ROOT patch needs to be rebased to 4.8.alpha4. Let me check. Thanks for the test

### comment:17 Changed 11 years ago by Jason Grout

Description: modified (diff)

Jonathan,

I think the problem is that you didn't apply the patch here to the SAGE_ROOT repository. I've updated the instructions above with explicit commands.

### comment:18 Changed 11 years ago by Jason Grout

I'm checking now into the error you posted in your second. It appears that there is indeed a problem with the sagenb spkg not overwriting the old notebook.

### comment:19 follow-up:  21 Changed 11 years ago by Jason Grout

Description: modified (diff)

For future reference, here are the instructions if we had a sagenb_dependencies spkg. DON'T USE THESE--use the instructions in the description of the ticket.

cd $SAGE_ROOT ./sage -hg qimport http://trac.sagemath.org/sage_trac/raw-attachment/ticket/11080/trac-11080-sagenb_dependencies-spkg.patch ./sage -hg qpush ./sage -f http://sage.math.washington.edu/home/jason/sagenb_dependencies-20111216.spkg ./sage -f http://sage.math.washington.edu/home/jason/twisted-11.0.0.spkg ./sage -f http://sage.math.washington.edu/home/jason/sagenb-0.9.0.spkg cd devel/sage ../../sage -hg qimport http://trac.sagemath.org/sage_trac/raw-attachment/ticket/11078/trac_11078.patch ../../sage -hg qpush cd ../../ ./sage -br  ### comment:20 Changed 11 years ago by Jason Grout Description: modified (diff) ### comment:21 in reply to: 19 Changed 11 years ago by Jonathan Gutow OK, with the changed instructions and this explicit instruction set, I get a running sage-4.8.alpha4. The notebook launches and everything I tried other than 3-D plotting worked in the notebook. 3-D plotting does not work because Jmol seems to be gone. Is #11503 supposed to be included as well? That would install the new Jmol in the$SAGE_ROOT/local/shared directory rather than in the sagenb/data directory. I note that there is no sagenb/data/jmol directory in the notebook created by this ticket, while there is in the old 4.8.alpha4 notebook.

I think it's almost there. Jonathan Replying to jason:

For future reference, here are the instructions if we had a sagenb_dependencies spkg. DON'T USE THESE--use the instructions in the description of the ticket.

cd \$SAGE_ROOT
./sage -hg qimport http://trac.sagemath.org/sage_trac/raw-attachment/ticket/11080/trac-11080-sagenb_dependencies-spkg.patch
./sage -hg qpush
./sage -f http://sage.math.washington.edu/home/jason/sagenb_dependencies-20111216.spkg
./sage -f http://sage.math.washington.edu/home/jason/twisted-11.0.0.spkg
./sage -f http://sage.math.washington.edu/home/jason/sagenb-0.9.0.spkg
cd devel/sage
../../sage -hg qimport http://trac.sagemath.org/sage_trac/raw-attachment/ticket/11078/trac_11078.patch
../../sage -hg qpush
cd ../../
./sage -br


### comment:22 Changed 11 years ago by Jason Grout

Jonathan: the problem was that my spkg-making sagenb install had a symbolic link to jmol, rather than the actual directory, and the symbolic link was not followed when making the spkg.

I've updated the 0.9.0 spkg (tagged as 0.9.0.rc2 in the repository google code). Can you try again? I just tried installing in a clean 4.8.alpha4, and 3d plots worked fine.

Thanks, Jason

### comment:23 Changed 11 years ago by Jonathan Gutow

Jason:  I think everything is OK.  It all seemed to work on my Ubuntu Linux distro.  I would vote for a positive review, if we can get one other person to verify.  Comment...anybody using Chrome on Linux, OpenJDK/IceTea don't play very well with Chrome and this old version of Jmol.

### comment:24 Changed 11 years ago by Jason Grout

Thanks! Jmol is certainly a top priority, along with mathjax, after we get this merged into sage. Lucky for us, most of the work has already been done, so hopefully it's just rebasing the patches that are already there.

### comment:25 Changed 11 years ago by Jason Grout

Description: modified (diff)

Deleting the dependency spkg instructions, since we aren't going that route.

### comment:26 Changed 11 years ago by Dima Pasechnik

Status: needs_review → needs_info

I checked that all this appears to work on MacOSX 10.6 just fine, including Jmol and text fields with LaTeX.

I do not like that this build does not incorporate the sass update by Tim Dumol in Sept. 2011 Without it, hacking the CSS stuff won't be possible on some platforms, at least not on MacOSX 10.6.

Is it possible to merge this, and the related readme update, into the sagenb spkg?

### comment:27 Changed 11 years ago by Jason Grout

It should be easy to incorporate that. Let me try. I don't think I edited anything in those directories.

### comment:28 follow-up:  29 Changed 11 years ago by Jason Grout

I've incorporated the sass patches, but there are some very small changes in the css, which worries me. How much have these patches been tested?

### comment:29 in reply to:  28 Changed 11 years ago by Dima Pasechnik

I've incorporated the sass patches, but there are some very small changes in the css, which worries me. How much have these patches been tested?

These are changes in the generated CSS files. The only changes in main.css are in the comments; these certainly can be ignored.

There are few other changes in the other css, called test_report.css. (I don't even know if this file is used at all, is it?) But these changes are indeed very, very minor, like a different way to encode the colour, or a change of a constant from 2.0em to 2em. I worked a bit with these modified CSS files, no issues noted. IMHO it is totally safe to update.

I cc this to Tim Dumol, just in case.

### comment:30 Changed 11 years ago by Tim Dumol

The changes made were all automated, via the sass-convert tool made by the developers of SASS/SCSS themselves, with the exception of the moving of the Accounts page scss to a separate partial, which was a cut-paste into a new file; thus, there shouldn't be any regressions.

### comment:31 follow-ups:  32  33 Changed 11 years ago by Jason Grout

For example, line 614 of ../../../../sass/src/jquery-plugins/_ui.achtung.scss now generates this line in main.css:

.achtung .ui-icon.achtung-close-button { overflow: hidden; float: right; position: relative; top: -8px; right: -8px; cursor: pointer; background-image: url(/javascript/jqueryui/css/sage/images/ui-icons_cccccc_256x240.png); }


However, that background image URL doesn't exist (but the old background image URL does exist).

### comment:32 in reply to:  31 Changed 11 years ago by Tim Dumol

For example, line 614 of ../../../../sass/src/jquery-plugins/_ui.achtung.scss now generates this line in main.css:

.achtung .ui-icon.achtung-close-button { overflow: hidden; float: right; position: relative; top: -8px; right: -8px; cursor: pointer; background-image: url(/javascript/jqueryui/css/sage/images/ui-icons_cccccc_256x240.png); }


However, that background image URL doesn't exist (but the old background image URL does exist).

Sorry about that. From what I can tell, I must have used another copy of Achtung to update the Achtung CSS, which was configured to another jQuery UI theme. I'll push a fix in a minute.

### comment:33 in reply to:  31 Changed 11 years ago by Dima Pasechnik

For example, line 614 of ../../../../sass/src/jquery-plugins/_ui.achtung.scss now generates this line in main.css:

.achtung .ui-icon.achtung-close-button { overflow: hidden; float: right; position: relative; top: -8px; right: -8px; cursor: pointer; background-image: url(/javascript/jqueryui/css/sage/images/ui-icons_cccccc_256x240.png); }


However, that background image URL doesn't exist (but the old background image URL does exist).

my fault that I didn't notice these on the diff. I just looked again, and this, and one more, images at the very bottom had changed filenames. The rest are completely innocent changes (comments, #d vs #D, "..." vs '...').

### comment:35 in reply to:  34 Changed 11 years ago by Tim Dumol

I think I just fixed it: https://github.com/sagemath/sagenb/commit/624dddb228bf3ebfdc02e161d2a5646b8c32a31f

Yep, that should do the trick. Once again, sorry.

### comment:36 Changed 11 years ago by Jason Grout

No problem. Things have certainly slipped by me often enough. Okay, positive review to these patches from me too.

### comment:37 follow-up:  38 Changed 11 years ago by Jason Grout

Status: needs_info → needs_review

### comment:38 in reply to:  37 Changed 11 years ago by Dima Pasechnik

Status: needs_review → positive_review

OK, with these changes in the spkg, all good to go!

### comment:39 Changed 11 years ago by Jason Grout

Status: positive_review → needs_work

I've updated the spkg. Can someone confirm that it still installs and has these last changes? I also added a few changes to deal with using hg-git to mirror a git repository.

I'm setting this back to needs-review so someone can give it one final +1, after all has been said and done.

### comment:40 follow-up:  41 Changed 11 years ago by Jason Grout

Status: needs_work → needs_review

### comment:41 in reply to:  40 Changed 11 years ago by Dima Pasechnik

Status: needs_review → positive_review

the updated sagenb-0.9.0 installs and works on MacOSX 10.6. And the sass changes are there. I do not know how to test the CSS properly, though, i.e. when these Achtung images come into play, but perhaps it's a good day and I should give it a positive review, still.

### comment:42 Changed 11 years ago by Jeroen Demeyer

Milestone: sage-4.8 → sage-5.0

### comment:43 Changed 11 years ago by Jeroen Demeyer

Description: modified (diff)

### comment:44 follow-up:  45 Changed 11 years ago by Jason Grout

I'm curious: what is the plan for 5.0? Can I guess: finish 4.8, then merge this and the python update in 5.0alpha0?

### comment:45 in reply to:  44 Changed 11 years ago by Jeroen Demeyer

I'm curious: what is the plan for 5.0? Can I guess: finish 4.8, then merge this and the python update in 5.0alpha0?

Essentially yes. But the dependencies of this ticket and also Python (#9958) still need review.

### comment:46 Changed 11 years ago by Jeroen Demeyer

Status: positive_review → needs_work → Mercurial

SPKG.txt and spkg-install should be under Mercurial control.

### comment:47 follow-up:  52 Changed 11 years ago by Jeroen Demeyer

Work issues: Mercurial → Mercurial, rebase?

Please double-check that following tickets really have been merged in the new sagenb:

### comment:48 follow-up:  51 Changed 11 years ago by Jason Grout

SPKG.txt and spkg-install *are* under version control, in the original sagenb repository. The problem is that we then custom-make an spkg and copy those files into the spkg directory. In other words, they are under version control in the src/sagenb/ directory.

I'm not quite sure how to handle this case. Since the spkg is created fresh every time make spkg is called, a repository for the SPKG.txt and spkg-install files at the top level makes no sense---they would always be just the initial commits.

Ideas?

### comment:49 Changed 11 years ago by Jason Grout

Thanks for the list of tickets to double-check.

### comment:50 Changed 11 years ago by Jeroen Demeyer

Dependencies: #11078, #11874 → #11078, #11874, #12229

### comment:51 in reply to:  48 Changed 11 years ago by Jeroen Demeyer

SPKG.txt and spkg-install *are* under version control, in the original sagenb repository. The problem is that we then custom-make an spkg and copy those files into the spkg directory. In other words, they are under version control in the src/sagenb/ directory.

I'm not quite sure how to handle this case. Since the spkg is created fresh every time make spkg is called, a repository for the SPKG.txt and spkg-install files at the top level makes no sense---they would always be just the initial commits.

I would say, never mind for now.

### comment:52 in reply to:  47 ; follow-up:  54 Changed 11 years ago by Keshav Kini

Please double-check that following tickets really have been merged in the new sagenb:

#11732 needs rebasing. #10052 and #11106 are already merged. I'm not sure what do with #11343 given the existence of #11470. I've submitted pull requests on github to get the remaining four tickets into the flask notebook (awaiting review).

### comment:53 Changed 11 years ago by Jason Grout

I've updated the spkg above to include commits up through https://github.com/sagemath/sagenb/commit/6b06baed92d1ab0657c3f2af2c153a5958eae8e4

### comment:54 in reply to:  52 Changed 11 years ago by Keshav Kini

#11732 needs rebasing. #10052 and #11106 are already merged. I'm not sure what do with #11343 given the existence of #11470. I've submitted pull requests on github to get the remaining four tickets into the flask notebook (awaiting review).

Update: Only #11732 remains, everything else is merged. Thanks again, Jeroen, for the list.

### comment:55 Changed 11 years ago by Jason Grout

@kini: The tinymce pull request isn't merged yet (but is awaiting review! :) https://github.com/sagemath/sagenb/pull/11

### comment:56 Changed 11 years ago by Jason Grout

Okay, the tinymce pull request is now merged.

### comment:57 Changed 11 years ago by William Stein

REFEREE REMARKS 1:

• The spkg is not in the proper format. However, this is complicated to fix, and will be fixed in a *future* ticket: #12268
• notebook(secure=True) is currently totally broken, due to the twisted spkg not supporting our tls extension, since I guess the patch didn't get applied. This needs to get fixed.
• All doctests pass.

• Rado has done some work on trying to get Selenium testing to work (the main issue just being installing everything on the same computer!). The Selenium tests need to get run (and any issues fixed) before we make the release.
• Jason has simplified and streamlined the notebook developer instructions in response to discussions with Rado and I.

### comment:58 Changed 11 years ago by William Stein

(Also, let's not forget to fix up and review #11409, though a not dependency for this!.)

### comment:59 follow-up:  60 Changed 11 years ago by William Stein

To summarize, what *needs* to be done to finish this ticket completely is:

• tls (secure=True) support for Twisted

• Selenium

• Ensure that sagenb built fresh as *part of* a clean new Sage sdist works, and in particular that the jmol command line program works, e.g, sage: sphere() pops up jmol. We are worried that it won't work since we don't see how that script ever gets copied to local/bin/.

### comment:60 in reply to:  59 ; follow-up:  61 Changed 11 years ago by Dima Pasechnik

• tls (secure=True) support for Twisted

you mention a patch that didn't get applied to twisted. Any more details on this patch?

Dima

### comment:61 in reply to:  60 Changed 11 years ago by Dima Pasechnik

• tls (secure=True) support for Twisted

you mention a patch that didn't get applied to twisted. Any more details on this patch?

upon reading the thread here started by jason-sage (we know this guy, don't we) http://twistedmatrix.com/pipermail/twisted-python/2010-October/022989.html they basically say that one needs to do

"ssl:443:sslmethod=TLSv1_METHOD"


somewhere in the configs.

Dima

### comment:62 Changed 11 years ago by Jason Grout

I uploaded a new spkg with pyOpenSSL included, as well as a small fix for selenium tests from Rado.

### comment:63 Changed 11 years ago by Jeroen Demeyer

Priority: major → blocker Mercurial, rebase?

### comment:64 Changed 11 years ago by Ian Stokes-Rees

Pushed sagenb patch to github: https://github.com/sagemath/sagenb/commit/e930db178c468c488f86465cbc51c3ef4df83b70

This takes care of the blocker problem of not being able to use secure=True for HTTPS (SSL) Flask-based Sage Notebook server. It strips out all reference to GnuTLS and relies on PyOpenSSL.

For a few more details, GnuTLS support in Twisted was going to be pretty hard to get, and maintaining a sage-specific patch not a great proposition. William also suggested that GnuTLS support be removed.

### comment:65 Changed 11 years ago by Keshav Kini

Work issues: → rebase

(#11732 still needs rebasing)

### comment:66 Changed 11 years ago by Jason Grout

Keywords: sd31, sd35.5 → sd31 sd35.5

### comment:67 Changed 11 years ago by Jason Grout

Status: needs_work → needs_review

I updated the spkg to rc6, which is this commit: https://github.com/sagemath/sagenb/commit/84d442d5ac83f9484c12071e36dd98b3b896ea91

I think we've addressed William's points above:

• SSL: We now don't need GnuTLS, but instead require that the user has openSSL installed to use secure=True as well as openID (I think this is adequately noted in the README.rst).
• Selenium: we incorporated the patch from Rado which fixes some selenium tests. Rado: is there anything else we need to do on that?
• jmol is now correctly copied over on a fresh sagenb install.

### comment:68 Changed 11 years ago by Jeroen Demeyer

What's the status of #11732?

### comment:70 Changed 11 years ago by Jeroen Demeyer

Work issues: rebase

Okay, great!

### comment:71 Changed 11 years ago by Karl-Dieter Crisman

I'm asking the dumb question. Suppose someone has an old notebook server or old worksheets on a local installation. This will behave correctly without any user intervention, correct? I'm pretty sure this is true, since sagenb.org was switched without much complaint along these lines, but just wanted to ask one last time :)

### comment:72 Changed 11 years ago by Jason Grout

Yes, it should work without any user intervention. I will be testing this by upgrading the MAA server soon, when things settle down here.

### comment:73 Changed 11 years ago by Jason Grout

Dependencies: #11078, #11874, #12229 → #11078, #11874, #12229, #11503

Added #11503 as a dependency, since it has positive review and the necessary patch has been merged into sagenb master.

### comment:74 Changed 11 years ago by Jason Grout

#11503 was set back to "needs work", but as it is almost there, I'll still leave the change in sagenb master.

### comment:75 Changed 11 years ago by Jason Grout

Description: modified (diff)

I've updated the instructions and the sagenb spkg to include the most recent fixes and changes for the new jmol spkg and the twisted dependency. The sagenb spkg also includes the published jmol fix at https://github.com/sagemath/sagenb/pull/26

### comment:76 Changed 11 years ago by Jason Grout

Description: modified (diff)
Note: See TracTickets for help on using tickets.