Opened 10 years ago

Last modified 5 years ago

#11080 closed enhancement

move notebook to flask/wsgi-based notebook — at Version 173

Reported by: jason Owned by: jason, mpatel, was
Priority: blocker Milestone: sage-5.2
Component: notebook Keywords: sd31 sd35.5
Cc: ddrake, jason, kcrisman, mhansen, rkirov, timdumol, iandrus, strogdon Merged in:
Authors: Mike Hansen, Rado Kirov, William Stein, Jason Grout Reviewers: Rado Kirov, Dan Drake, Jason Grout
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: #11078, #11874, #12229, #11503, #12327 Stopgaps:

Status badges

Description (last modified by SimonKing)

This ticket tracks the progress to move the notebook to a flask/wsgi-based notebook.

Relevant code repository: https://github.com/sagemath/sagenb

Explicit instructions starting with sage-5.0.beta13 including dependencies (replace $SAGE_ROOT with the root sage directory):

cd $SAGE_ROOT
./sage -f http://boxen.math.washington.edu/home/keshav/files/sagenb-0.9.0.spkg
./sage -f http://www.uwosh.edu/faculty_staff/gutow/jmol-12.2.21.p0.spkg


./sage -hg qimport http://trac.sagemath.org/sage_trac/raw-attachment/ticket/11874/trac-11874-remove-twisted.patch
./sage -hg qpush

./sage -hg qimport http://trac.sagemath.org/sage_trac/raw-attachment/ticket/11503/trac-11503-jmol-spkg.2.patch
./sage -hg qpush

cd devel/sage
../../sage -hg qimport http://trac.sagemath.org/sage_trac/raw-attachment/ticket/11874/sage-spkg-11874.patch
../../sage -hg qpush

../../sage -hg qimport http://trac.sagemath.org/sage_trac/raw-attachment/ticket/11078/trac_11078.patch
../../sage -hg qpush


../../sage -hg qimport http://trac.sagemath.org/sage_trac/raw-attachment/ticket/11503/trac-11503-jmol-commandline.patch
../../sage -hg qpush

../../sage -hg qimport http://trac.sagemath.org/sage_trac/raw-attachment/ticket/12229/trac-12229-sagenb-developer-doc.2.patch
../../sage -hg qpush

../../sage -hg qimport http://trac.sagemath.org/sage_trac/raw-attachment/ticket/12229/trac-12229-manifest.patch
../../sage -hg qpush

../../sage -hg qimport http://trac.sagemath.org/sage_trac/raw-attachment/ticket/11080/trac-11080-notebook-docs.patch
../../sage -hg qpush

cd ../../
./sage -br

spkg: http://boxen.math.washington.edu/home/keshav/files/sagenb-0.9.0.spkg (this includes all dependencies not already in Sage)

apply: trac-11080-notebook-docs.patch to sage repository

This is a review version of the spkg. When we have finalized things, we'll make a finished spkg that has the appropriate tags in the revision history, no unnecessary revision branches, etc.

Testing release: http://boxen.math.washington.edu/home/release/sage-5.1.notebook

Change History (174)

comment:1 Changed 10 years ago by jason

  • Cc ddrake added

comment:2 Changed 10 years ago by gutow

  • Description modified (diff)

comment:3 Changed 10 years ago by gutow

  • Description modified (diff)
  • Keywords sd31 added

comment:4 Changed 10 years ago by gutow

  • Description modified (diff)

comment:5 Changed 10 years ago by gutow

  • Description modified (diff)

comment:6 Changed 10 years ago by gutow

  • Description modified (diff)

removed #11078 from list as the fix is included in the 11503 command line patch.

comment:7 Changed 10 years ago by jason

  • Authors changed from Mike Hansen, Rado Kirov to Mike Hansen, Rado Kirov, William Stein, Jason Grout
  • Dependencies set to #11078
  • Description modified (diff)

comment:8 Changed 10 years ago by jason

  • Description modified (diff)

comment:9 Changed 10 years ago by jason

  • Description modified (diff)

comment:10 Changed 10 years ago by jason

  • Dependencies changed from #11078 to #11078, #11874

comment:11 Changed 10 years ago by jason

  • Status changed from new to needs_review

comment:12 Changed 10 years ago by jason

(for future reference, after this is merged, jmol is a high priority. These two tickets deal with these changes:

  • #11496 (update jmol script in local/bin to reflect move of jmol directory to local)
  • #11503 (updated spkg and paths for move of jmol directory)
  • #9238 (for reference -original Jmol enhancement ticket but heavily laden with pre-flask notebook stuff)

Again, this is only FYI and for further reference.

comment:13 Changed 10 years ago by jason

  • Description modified (diff)

comment:14 Changed 10 years ago by gutow

I get the following error when I try to apply the spkg dependencies patch. Everything else seems to work. I'm still checking functionality...

applying /home/jonathan/.sage/temp/VB_Ubuntu_11/26614/tmp_0.patch
unable to find 'spkg/install' for patching
1 out of 1 hunks FAILED -- saving rejects to file spkg/install.rej
unable to find 'spkg/standard/deps' for patching
4 out of 4 hunks FAILED -- saving rejects to file spkg/standard/deps.rej
spkg/install: No such file or directory
spkg/standard/deps: No such file or directory
abort: patch failed to apply

comment:15 Changed 10 years ago by gutow

With 4.8.alpha4, what is described here does not work. I redid everything. The .spkgs seem to install without error. The patch for this ticket does not work and the notebook fails to launch with the error:

/home/jonathan/Documents/sage-4.8.alpha4/<ipython console> in <module>()

/home/jonathan/Documents/sage-4.8.alpha4/devel/sagenb/sagenb/notebook/notebook_object.py in __call__(self, *args, **kwds)
    202     """
    203     def __call__(self, *args, **kwds):
--> 204         return self.notebook(*args, **kwds)
    205 
    206     notebook = run_notebook.notebook_twisted

/home/jonathan/Documents/sage-4.8.alpha4/devel/sagenb/sagenb/notebook/run_notebook.py in notebook_twisted(self, directory, port, interface, address, port_tries, secure, reset, require_login, accounts, openid, server_pool, ulimit, timeout, open_viewer, sagetex_path, start_path, fork, quiet, subnets)
    442     if open_viewer:
    443         "Open viewer automatically isn't fully implemented.  You have to manually open your web browser to the above URL."
--> 444     return run(port)
    445 
    446 def get_admin_passwd():

/home/jonathan/Documents/sage-4.8.alpha4/devel/sagenb/sagenb/notebook/run_notebook.py in run(port)
    427         os.chdir(cwd)
    428         if e == 256:
--> 429             raise socket.error
    430 
    431         return True

comment:16 Changed 10 years ago by jason

It's possible that the SAGE_ROOT patch needs to be rebased to 4.8.alpha4. Let me check. Thanks for the test

comment:17 Changed 10 years ago by jason

  • Description modified (diff)

Jonathan,

I think the problem is that you didn't apply the patch here to the SAGE_ROOT repository. I've updated the instructions above with explicit commands.

comment:18 Changed 10 years ago by jason

I'm checking now into the error you posted in your second. It appears that there is indeed a problem with the sagenb spkg not overwriting the old notebook.

comment:19 follow-up: Changed 10 years ago by jason

  • Description modified (diff)

For future reference, here are the instructions if we had a sagenb_dependencies spkg. DON'T USE THESE--use the instructions in the description of the ticket.

cd $SAGE_ROOT
./sage -hg qimport http://trac.sagemath.org/sage_trac/raw-attachment/ticket/11080/trac-11080-sagenb_dependencies-spkg.patch
./sage -hg qpush
./sage -f http://sage.math.washington.edu/home/jason/sagenb_dependencies-20111216.spkg
./sage -f http://sage.math.washington.edu/home/jason/twisted-11.0.0.spkg
./sage -f http://sage.math.washington.edu/home/jason/sagenb-0.9.0.spkg
cd devel/sage
../../sage -hg qimport http://trac.sagemath.org/sage_trac/raw-attachment/ticket/11078/trac_11078.patch
../../sage -hg qpush
cd ../../
./sage -br

comment:20 Changed 10 years ago by jason

  • Description modified (diff)

comment:21 in reply to: ↑ 19 Changed 10 years ago by gutow

OK, with the changed instructions and this explicit instruction set, I get a running sage-4.8.alpha4. The notebook launches and everything I tried other than 3-D plotting worked in the notebook. 3-D plotting does not work because Jmol seems to be gone. Is #11503 supposed to be included as well? That would install the new Jmol in the $SAGE_ROOT/local/shared directory rather than in the sagenb/data directory. I note that there is no sagenb/data/jmol directory in the notebook created by this ticket, while there is in the old 4.8.alpha4 notebook.

I think it's almost there. Jonathan Replying to jason:

For future reference, here are the instructions if we had a sagenb_dependencies spkg. DON'T USE THESE--use the instructions in the description of the ticket.

cd $SAGE_ROOT
./sage -hg qimport http://trac.sagemath.org/sage_trac/raw-attachment/ticket/11080/trac-11080-sagenb_dependencies-spkg.patch
./sage -hg qpush
./sage -f http://sage.math.washington.edu/home/jason/sagenb_dependencies-20111216.spkg
./sage -f http://sage.math.washington.edu/home/jason/twisted-11.0.0.spkg
./sage -f http://sage.math.washington.edu/home/jason/sagenb-0.9.0.spkg
cd devel/sage
../../sage -hg qimport http://trac.sagemath.org/sage_trac/raw-attachment/ticket/11078/trac_11078.patch
../../sage -hg qpush
cd ../../
./sage -br

comment:22 Changed 10 years ago by jason

Jonathan: the problem was that my spkg-making sagenb install had a symbolic link to jmol, rather than the actual directory, and the symbolic link was not followed when making the spkg.

I've updated the 0.9.0 spkg (tagged as 0.9.0.rc2 in the repository google code). Can you try again? I just tried installing in a clean 4.8.alpha4, and 3d plots worked fine.

Thanks, Jason

comment:23 Changed 10 years ago by gutow

Jason:  I think everything is OK.  It all seemed to work on my Ubuntu Linux distro.  I would vote for a positive review, if we can get one other person to verify.  Comment...anybody using Chrome on Linux, OpenJDK/IceTea don't play very well with Chrome and this old version of Jmol.

comment:24 Changed 10 years ago by jason

Thanks! Jmol is certainly a top priority, along with mathjax, after we get this merged into sage. Lucky for us, most of the work has already been done, so hopefully it's just rebasing the patches that are already there.

comment:25 Changed 10 years ago by jason

  • Description modified (diff)

Deleting the dependency spkg instructions, since we aren't going that route.

comment:26 Changed 10 years ago by dimpase

  • Status changed from needs_review to needs_info

I checked that all this appears to work on MacOSX 10.6 just fine, including Jmol and text fields with LaTeX.

I do not like that this build does not incorporate the sass update by Tim Dumol in Sept. 2011 Without it, hacking the CSS stuff won't be possible on some platforms, at least not on MacOSX 10.6.

Is it possible to merge this, and the related readme update, into the sagenb spkg?

comment:27 Changed 10 years ago by jason

It should be easy to incorporate that. Let me try. I don't think I edited anything in those directories.

comment:28 follow-up: Changed 10 years ago by jason

I've incorporated the sass patches, but there are some very small changes in the css, which worries me. How much have these patches been tested?

comment:29 in reply to: ↑ 28 Changed 10 years ago by dimpase

  • Cc timdumol added

Replying to jason:

I've incorporated the sass patches, but there are some very small changes in the css, which worries me. How much have these patches been tested?

These are changes in the generated CSS files. The only changes in main.css are in the comments; these certainly can be ignored.

There are few other changes in the other css, called test_report.css. (I don't even know if this file is used at all, is it?) But these changes are indeed very, very minor, like a different way to encode the colour, or a change of a constant from 2.0em to 2em. I worked a bit with these modified CSS files, no issues noted. IMHO it is totally safe to update.

I cc this to Tim Dumol, just in case.

comment:30 Changed 10 years ago by timdumol

The changes made were all automated, via the sass-convert tool made by the developers of SASS/SCSS themselves, with the exception of the moving of the Accounts page scss to a separate partial, which was a cut-paste into a new file; thus, there shouldn't be any regressions.

comment:31 follow-ups: Changed 10 years ago by jason

For example, line 614 of ../../../../sass/src/jquery-plugins/_ui.achtung.scss now generates this line in main.css:

.achtung .ui-icon.achtung-close-button { overflow: hidden; float: right; position: relative; top: -8px; right: -8px; cursor: pointer; background-image: url(/javascript/jqueryui/css/sage/images/ui-icons_cccccc_256x240.png); }

However, that background image URL doesn't exist (but the old background image URL does exist).

comment:32 in reply to: ↑ 31 Changed 10 years ago by timdumol

Replying to jason:

For example, line 614 of ../../../../sass/src/jquery-plugins/_ui.achtung.scss now generates this line in main.css:

.achtung .ui-icon.achtung-close-button { overflow: hidden; float: right; position: relative; top: -8px; right: -8px; cursor: pointer; background-image: url(/javascript/jqueryui/css/sage/images/ui-icons_cccccc_256x240.png); }

However, that background image URL doesn't exist (but the old background image URL does exist).

Sorry about that. From what I can tell, I must have used another copy of Achtung to update the Achtung CSS, which was configured to another jQuery UI theme. I'll push a fix in a minute.

comment:33 in reply to: ↑ 31 Changed 10 years ago by dimpase

Replying to jason:

For example, line 614 of ../../../../sass/src/jquery-plugins/_ui.achtung.scss now generates this line in main.css:

.achtung .ui-icon.achtung-close-button { overflow: hidden; float: right; position: relative; top: -8px; right: -8px; cursor: pointer; background-image: url(/javascript/jqueryui/css/sage/images/ui-icons_cccccc_256x240.png); }

However, that background image URL doesn't exist (but the old background image URL does exist).

my fault that I didn't notice these on the diff. I just looked again, and this, and one more, images at the very bottom had changed filenames. The rest are completely innocent changes (comments, #d vs #D, "..." vs '...').

comment:35 in reply to: ↑ 34 Changed 10 years ago by timdumol

Replying to jason:

I think I just fixed it: https://github.com/sagemath/sagenb/commit/624dddb228bf3ebfdc02e161d2a5646b8c32a31f

Yep, that should do the trick. Once again, sorry.

comment:36 Changed 10 years ago by jason

No problem. Things have certainly slipped by me often enough. Okay, positive review to these patches from me too.

comment:37 follow-up: Changed 10 years ago by jason

  • Status changed from needs_info to needs_review

comment:38 in reply to: ↑ 37 Changed 10 years ago by dimpase

  • Status changed from needs_review to positive_review

Replying to jason:

OK, with these changes in the spkg, all good to go!

comment:39 Changed 10 years ago by jason

  • Status changed from positive_review to needs_work

I've updated the spkg. Can someone confirm that it still installs and has these last changes? I also added a few changes to deal with using hg-git to mirror a git repository.

I'm setting this back to needs-review so someone can give it one final +1, after all has been said and done.

comment:40 follow-up: Changed 10 years ago by jason

  • Status changed from needs_work to needs_review

comment:41 in reply to: ↑ 40 Changed 10 years ago by dimpase

  • Status changed from needs_review to positive_review

Replying to jason:

the updated sagenb-0.9.0 installs and works on MacOSX 10.6. And the sass changes are there. I do not know how to test the CSS properly, though, i.e. when these Achtung images come into play, but perhaps it's a good day and I should give it a positive review, still.

comment:42 Changed 10 years ago by jdemeyer

  • Milestone changed from sage-4.8 to sage-5.0

comment:43 Changed 10 years ago by jdemeyer

  • Description modified (diff)

comment:44 follow-up: Changed 10 years ago by jason

I'm curious: what is the plan for 5.0? Can I guess: finish 4.8, then merge this and the python update in 5.0alpha0?

comment:45 in reply to: ↑ 44 Changed 10 years ago by jdemeyer

Replying to jason:

I'm curious: what is the plan for 5.0? Can I guess: finish 4.8, then merge this and the python update in 5.0alpha0?

Essentially yes. But the dependencies of this ticket and also Python (#9958) still need review.

comment:46 Changed 10 years ago by jdemeyer

  • Status changed from positive_review to needs_work
  • Work issues set to Mercurial

SPKG.txt and spkg-install should be under Mercurial control.

comment:47 follow-up: Changed 10 years ago by jdemeyer

  • Work issues changed from Mercurial to Mercurial, rebase?

Please double-check that following tickets really have been merged in the new sagenb:

comment:48 follow-up: Changed 10 years ago by jason

SPKG.txt and spkg-install *are* under version control, in the original sagenb repository. The problem is that we then custom-make an spkg and copy those files into the spkg directory. In other words, they are under version control in the src/sagenb/ directory.

I'm not quite sure how to handle this case. Since the spkg is created fresh every time make spkg is called, a repository for the SPKG.txt and spkg-install files at the top level makes no sense---they would always be just the initial commits.

Ideas?

comment:49 Changed 10 years ago by jason

Thanks for the list of tickets to double-check.

comment:50 Changed 10 years ago by jdemeyer

  • Dependencies changed from #11078, #11874 to #11078, #11874, #12229

comment:51 in reply to: ↑ 48 Changed 10 years ago by jdemeyer

Replying to jason:

SPKG.txt and spkg-install *are* under version control, in the original sagenb repository. The problem is that we then custom-make an spkg and copy those files into the spkg directory. In other words, they are under version control in the src/sagenb/ directory.

I'm not quite sure how to handle this case. Since the spkg is created fresh every time make spkg is called, a repository for the SPKG.txt and spkg-install files at the top level makes no sense---they would always be just the initial commits.

I would say, never mind for now.

comment:52 in reply to: ↑ 47 ; follow-up: Changed 9 years ago by kini

Replying to jdemeyer:

Please double-check that following tickets really have been merged in the new sagenb:

#11732 needs rebasing. #10052 and #11106 are already merged. I'm not sure what do with #11343 given the existence of #11470. I've submitted pull requests on github to get the remaining four tickets into the flask notebook (awaiting review).

comment:53 Changed 9 years ago by jason

I've updated the spkg above to include commits up through https://github.com/sagemath/sagenb/commit/6b06baed92d1ab0657c3f2af2c153a5958eae8e4

comment:54 in reply to: ↑ 52 Changed 9 years ago by kini

Replying to kini:

#11732 needs rebasing. #10052 and #11106 are already merged. I'm not sure what do with #11343 given the existence of #11470. I've submitted pull requests on github to get the remaining four tickets into the flask notebook (awaiting review).

Update: Only #11732 remains, everything else is merged. Thanks again, Jeroen, for the list.

comment:55 Changed 9 years ago by jason

@kini: The tinymce pull request isn't merged yet (but is awaiting review! :) https://github.com/sagemath/sagenb/pull/11

comment:56 Changed 9 years ago by jason

Okay, the tinymce pull request is now merged.

comment:57 Changed 9 years ago by was

REFEREE REMARKS 1:

  • The spkg is not in the proper format. However, this is complicated to fix, and will be fixed in a *future* ticket: #12268
  • notebook(secure=True) is currently totally broken, due to the twisted spkg not supporting our tls extension, since I guess the patch didn't get applied. This needs to get fixed.
  • All doctests pass.

  • Rado has done some work on trying to get Selenium testing to work (the main issue just being installing everything on the same computer!). The Selenium tests need to get run (and any issues fixed) before we make the release.
  • Jason has simplified and streamlined the notebook developer instructions in response to discussions with Rado and I.

comment:58 Changed 9 years ago by was

(Also, let's not forget to fix up and review #11409, though a not dependency for this!.)

comment:59 follow-up: Changed 9 years ago by was

To summarize, what *needs* to be done to finish this ticket completely is:

  • tls (secure=True) support for Twisted

  • Selenium

  • Ensure that sagenb built fresh as *part of* a clean new Sage sdist works, and in particular that the jmol command line program works, e.g, sage: sphere() pops up jmol. We are worried that it won't work since we don't see how that script ever gets copied to local/bin/.

comment:60 in reply to: ↑ 59 ; follow-up: Changed 9 years ago by dimpase

Replying to was:

  • tls (secure=True) support for Twisted

you mention a patch that didn't get applied to twisted. Any more details on this patch?

Dima

comment:61 in reply to: ↑ 60 Changed 9 years ago by dimpase

Replying to dimpase:

Replying to was:

  • tls (secure=True) support for Twisted

you mention a patch that didn't get applied to twisted. Any more details on this patch?

upon reading the thread here started by jason-sage (we know this guy, don't we) http://twistedmatrix.com/pipermail/twisted-python/2010-October/022989.html they basically say that one needs to do

"ssl:443:sslmethod=TLSv1_METHOD"

somewhere in the configs.

Dima

comment:62 Changed 9 years ago by jason

I uploaded a new spkg with pyOpenSSL included, as well as a small fix for selenium tests from Rado.

comment:63 Changed 9 years ago by jdemeyer

  • Priority changed from major to blocker
  • Work issues Mercurial, rebase? deleted

comment:64 Changed 9 years ago by ijstokes

  • Keywords sd35.5 added

Pushed sagenb patch to github: https://github.com/sagemath/sagenb/commit/e930db178c468c488f86465cbc51c3ef4df83b70

This takes care of the blocker problem of not being able to use secure=True for HTTPS (SSL) Flask-based Sage Notebook server. It strips out all reference to GnuTLS and relies on PyOpenSSL.

For a few more details, GnuTLS support in Twisted was going to be pretty hard to get, and maintaining a sage-specific patch not a great proposition. William also suggested that GnuTLS support be removed.

comment:65 Changed 9 years ago by kini

  • Work issues set to rebase

(#11732 still needs rebasing)

comment:66 Changed 9 years ago by jason

  • Keywords changed from sd31, sd35.5 to sd31 sd35.5

comment:67 Changed 9 years ago by jason

  • Status changed from needs_work to needs_review

I updated the spkg to rc6, which is this commit: https://github.com/sagemath/sagenb/commit/84d442d5ac83f9484c12071e36dd98b3b896ea91

I think we've addressed William's points above:

  • SSL: We now don't need GnuTLS, but instead require that the user has openSSL installed to use secure=True as well as openID (I think this is adequately noted in the README.rst).
  • Selenium: we incorporated the patch from Rado which fixes some selenium tests. Rado: is there anything else we need to do on that?
  • jmol is now correctly copied over on a fresh sagenb install.

comment:68 Changed 9 years ago by jdemeyer

What's the status of #11732?

comment:70 Changed 9 years ago by jdemeyer

  • Work issues rebase deleted

Okay, great!

comment:71 Changed 9 years ago by kcrisman

I'm asking the dumb question. Suppose someone has an old notebook server or old worksheets on a local installation. This will behave correctly without any user intervention, correct? I'm pretty sure this is true, since sagenb.org was switched without much complaint along these lines, but just wanted to ask one last time :)

comment:72 Changed 9 years ago by jason

Yes, it should work without any user intervention. I will be testing this by upgrading the MAA server soon, when things settle down here.

comment:73 Changed 9 years ago by jason

  • Dependencies changed from #11078, #11874, #12229 to #11078, #11874, #12229, #11503

Added #11503 as a dependency, since it has positive review and the necessary patch has been merged into sagenb master.

comment:74 Changed 9 years ago by jason

#11503 was set back to "needs work", but as it is almost there, I'll still leave the change in sagenb master.

comment:75 Changed 9 years ago by jason

  • Description modified (diff)

I've updated the instructions and the sagenb spkg to include the most recent fixes and changes for the new jmol spkg and the twisted dependency. The sagenb spkg also includes the published jmol fix at https://github.com/sagemath/sagenb/pull/26

comment:76 Changed 9 years ago by jason

  • Description modified (diff)

comment:77 Changed 9 years ago by jason

  • Description modified (diff)

comment:78 Changed 9 years ago by jason

  • Description modified (diff)

The sagenb spkg should be installed before the jmol spkg.

comment:79 follow-ups: Changed 9 years ago by jdemeyer

The new flask-sagenb will likely require changes to the Mac App data/extcode/sage/ext/mac-app also. It seems to have port 8000 hardcoded in a few places.

comment:80 in reply to: ↑ 79 Changed 9 years ago by iandrus

Replying to jdemeyer:

The new flask-sagenb will likely require changes to the Mac App data/extcode/sage/ext/mac-app also. It seems to have port 8000 hardcoded in a few places.

I only know of one place, namely in loading-page.html (grep gives some false positives). I have been meaning to improve that page for a long time and just created #12327 to that end. Regardless, the link on that page is non-essential and so there is no reason IMHO to delay the review of this ticket.

comment:81 Changed 9 years ago by jdemeyer

  • Dependencies changed from #11078, #11874, #12229, #11503 to #11078, #11874, #12229, #11503, #12327

comment:82 Changed 9 years ago by jason

  • Description modified (diff)

Added the new #11503 command line patch.

comment:83 Changed 9 years ago by jdemeyer

  • Description modified (diff)

comment:84 in reply to: ↑ 79 ; follow-up: Changed 9 years ago by iandrus

  • Cc iandrus added

Replying to jdemeyer:

The new flask-sagenb will likely require changes to the Mac App data/extcode/sage/ext/mac-app also. It seems to have port 8000 hardcoded in a few places.

What is more likely to need changes is the script sage-is-running-on-port.sh which uses the contents of ~/.sage/sage_notebook.sagenb/twistd.pid and ~/.sage/sage_notebook.sagenb/twistedconf.tac to return the port the server is running on.

comment:85 Changed 9 years ago by jdemeyer

  • Status changed from needs_review to needs_work

I get Sphinx warnings/errors when building the documentation:

dochtml.log:/mnt/usb1/scratch/jdemeyer/merger/sage-5.0.beta2/devel/sagenb/sagenb/notebook/cell.py:docstring of sagenb.notebook.cell.Cell:4: WARNING: Block quote ends without a blank line; unexpected unindent.
dochtml.log:/mnt/usb1/scratch/jdemeyer/merger/sage-5.0.beta2/devel/sagenb/sagenb/notebook/interact.py:docstring of sagenb.notebook.interact.slider_generic.values:11: WARNING: Literal block expected; none found.
dochtml.log:/mnt/usb1/scratch/jdemeyer/merger/sage-5.0.beta2/devel/sage/doc/en/reference/sagenb/notebook/twist.rst:7: WARNING: autodoc can't import/find module 'sagenb.notebook.twist', it reported error: "No module named twist", please check your spelling and sys.path
dochtml.log:/mnt/usb1/scratch/jdemeyer/merger/sage-5.0.beta2/devel/sage/doc/en/reference/sagenb/simple/twist.rst:7: WARNING: autodoc can't import/find module 'sagenb.simple.twist', it reported error: "No module named web2", please check your spelling and sys.path
dochtml.log:/mnt/usb1/scratch/jdemeyer/merger/sage-5.0.beta2/devel/sagenb/sagenb/notebook/cell.py:docstring of sagenb.notebook.cell.Cell:14: ERROR: Unexpected indentation.

comment:86 Changed 9 years ago by jhpalmieri

Jason has already posted this to sage-devel, but a one character fix for the two problems in cell.py:

  • sagenb/notebook/cell.py

    diff --git a/sagenb/notebook/cell.py b/sagenb/notebook/cell.py
    a b class Cell(Cell_generic): 
    11611161        return u'{{{id=%s|\n%s\n}}}' % (self.id(), s)
    11621162
    11631163    def next_compute_id(self):
    1164         """
     1164        r"""
    11651165        Returns the ID of the next compute cell in this compute cell's
    11661166        worksheet object.  If this cell is *not* in the worksheet, it
    11671167        returns the ID of the worksheet's *first* compute cell.  If

Changed 9 years ago by jason

comment:87 Changed 9 years ago by jason

  • Description modified (diff)

comment:88 Changed 9 years ago by jason

  • Description modified (diff)

comment:89 Changed 9 years ago by jason

  • Status changed from needs_work to needs_review

I updated the sagenb spkg to include up through commit b4cd1d20fa617008e8f12d9da684a508bafa7891, which resolves the doc building issues.

comment:90 in reply to: ↑ 84 Changed 9 years ago by iandrus

Replying to iandrus:

Replying to jdemeyer:

The new flask-sagenb will likely require changes to the Mac App data/extcode/sage/ext/mac-app also. It seems to have port 8000 hardcoded in a few places.

What is more likely to need changes is the script sage-is-running-on-port.sh which uses the contents of ~/.sage/sage_notebook.sagenb/twistd.pid and ~/.sage/sage_notebook.sagenb/twistedconf.tac to return the port the server is running on.

I tried this with the new notebook server and it seems to work fine.

comment:91 Changed 9 years ago by strogdon

  • Cc strogdon added

comment:92 Changed 9 years ago by jason

  • Description modified (diff)

comment:93 Changed 9 years ago by ppurka

  • Status changed from needs_review to needs_work

Applying this to sage-4.8, I ran into one major problem. And I suggest some other minor changes:

  1. There is a leftover file apply-hg.py that comes up when running hg status:
    ...ons/sage-4.8/devel/sagenb> hg status
    *** failed to import extension hggit: No module named hggit
    ! util/apply-hg.py
    
  1. It is not possible to login as user or admin. The notebook returns with a "Wrong password" message. I tried to look at user_manager to see if the problem is there but it isn't there. It's only from the web interface that the problem arises. There are two aspects of this: i) I can login with the password of a new user once I create a new user and use the randomly generated password given by Sage. ii) As soon as I change the randomly generated password to my own password using the web interface, I am unable to log in anymore.
  1. Using user_manager.py, I can see that the passwords are correctly retrieved and verified. So, I am unsure where exactly the problem is. Here is an excerpt:
    sage: nb = Notebook('/home/punarbasu/tmp/a.sagenb') 
    sage: um = nb.user_manager()
    sage: um.valid_login_names()             
    ['admin', u'test']
    sage: um.check_password('admin', 'admin')          
    True
    sage: um.check_password('test', 'test')  
    True
    sage: 
    
  1. [Minor change] In devel/sagenb/sagenb/notebook/run_notebook.py the following two changes should be made:
    • sagenb/notebook/run_notebook.py

      diff --git a/sagenb/notebook/run_notebook.py b/sagenb/notebook/run_notebook.py
      a b  
      438438    print "Please choose a new password for the Sage Notebook 'admin' user."
      439439    print "Do _not_ choose a stupid password, since anybody who could guess your password"
      440440    print "and connect to your machine could access or delete your files."
      441     print "NOTE: Only the md5 hash of the password you type is stored by Sage."
       441    print "NOTE: Only the sha256 hash of the password you type is stored by Sage."
      442442    print "You can change your password by typing notebook(reset=True)."
      443443    print "\n" * 2
      444444    while True:
      445445        passwd = getpass.getpass("Enter new password: ")
      446446        from sagenb.misc.misc import min_password_length
      447447        if len(passwd) < min_password_length:
      448             print "That password is way too short. Enter a password with at least 6 characters."
       448            print "That password is way too short. Enter a password with at least %d characters."%min_password_length
      449449            continue
      450450        passwd2 = getpass.getpass("Retype new password: ")
      451451        if passwd != passwd2:

comment:94 follow-up: Changed 9 years ago by jason

  • Status changed from needs_work to needs_info

I fixed (4) in commit https://github.com/sagemath/sagenb/commit/3b24bf8f85f4850eae194b1249179d9c638150b9

(1) This will be fixed in the next version of the spkg.

(2)-(3): I couldn't replicate this. Can you give a precise sequence of steps to replicate this? Here's what I did to test it:

  1. Log in as admin
  2. Create a new user under the Settings | Manage Users
  3. Log out and then log in as the new user
  4. Change the user's password
  5. Log in again with the new password

Everything seemed to work fine.

comment:95 in reply to: ↑ 94 Changed 9 years ago by dimpase

Replying to jason:

I fixed (4) in commit https://github.com/sagemath/sagenb/commit/3b24bf8f85f4850eae194b1249179d9c638150b9

(1) This will be fixed in the next version of the spkg.

(2)-(3): I couldn't replicate this. Can you give a precise sequence of steps to replicate this? Here's what I did to test it:

  1. Log in as admin
  2. Create a new user under the Settings | Manage Users
  3. Log out and then log in as the new user
  4. Change the user's password
  5. Log in again with the new password

Everything seemed to work fine.

I guess it's about upgrading from an existing flask nb installation, from some time ago, and on Sage 4.7.2.

comment:96 follow-up: Changed 9 years ago by jason

So does that mean it's a problem, or not?

comment:97 Changed 9 years ago by jason

  • Status changed from needs_info to needs_review

I upgraded the sagenb spkg to take care of issues (1) and (4) above.

comment:98 in reply to: ↑ 96 ; follow-up: Changed 9 years ago by dimpase

Replying to jason:

So does that mean it's a problem, or not?

I mean that ppurka reports on his attempt of upgrading from an existing flask nb installation, from some time ago, that used to be running Sage 4.7.2. Might be nontrivial to reproduce...

comment:99 in reply to: ↑ 98 Changed 9 years ago by jdemeyer

Upgrading is certainly something which should be tested. The current policy in Sage is that source upgrades from sage-4.5 (released July 2010) or later should work. This isn't a very official policy and can be changed to a later Sage version if needed. But certainly upgrading from the pretty recent sage-4.7.2 should work.

Backwards-compatibility is even more important for the saved worksheets in DOT_SAGE. The older worksheets you can support, the better.

comment:100 Changed 9 years ago by ppurka

Oh no no! This is on a fresh sagenb directory! Do the following (tried on sage-4.8):

sage -n directory=new.sagenb
# Give admin password
# In web interface logout and try to log back in
# You will get error
# Restart sage using above command
# In web interface, createa  new user and then logout and login with the random password of new user
# It works. Now, go to settings of new user and change the password and logout
# Try to login as the new user with new password. It doesn't work.

comment:101 follow-ups: Changed 9 years ago by jason

I explicitly tried each of those steps, and things worked flawlessly. I'm trying this on a 5.0 beta (which is where the notebook should be tested). I wonder if that makes a difference (it shouldn't, but who knows?).

Just to make sure: did you explicitly do each of the steps in the description of this ticket?

Also, is there any error message printed out for you in the terminal?

comment:102 in reply to: ↑ 101 Changed 9 years ago by ppurka

Replying to jason:

I explicitly tried each of those steps, and things worked flawlessly. I'm trying this on a 5.0 beta (which is where the notebook should be tested). I wonder if that makes a difference (it shouldn't, but who knows?).

I believe the only difference is coming from this. I tried this on 4.8, where it doesn't work. I haven't tried on 5.0.

Just to make sure: did you explicitly do each of the steps in the description of this ticket?

Yes. Just copy pasted the whole thing.

Also, is there any error message printed out for you in the terminal?

No error messages are output to the terminal. I can consistently reproduce this on two Linux systems. One is an Ubuntu-11.10 and the other is a Gentoo system. Both 64bits. Logins clearly also doesn't work on existing sagenb directories (but that will probably fix itself if this gets fixed for new sagenb directories).

comment:103 follow-up: Changed 9 years ago by ppurka

Ok. I get the exact same behavior on a sage-5.0-beta2. Passwords don't work for admin and for user if user changes the random password. But the passwords work when tested in command line a.k.a. comment 93, no. 3. All on a fresh sagenb directory. Can anyone else confirm this behavior?

comment:104 in reply to: ↑ 103 Changed 9 years ago by dimpase

  • Status changed from needs_review to needs_info

Replying to ppurka:

Ok. I get the exact same behavior on a sage-5.0-beta2. Passwords don't work for admin and for user if user changes the random password. But the passwords work when tested in command line a.k.a. comment 93, no. 3. All on a fresh sagenb directory. Can anyone else confirm this behavior?

Is it browser-specific?

comment:105 Changed 9 years ago by ppurka

No. It's not. I get same behavior on firefox-10 and opera-11.61.

comment:106 Changed 9 years ago by gutow

Why is #12327 a dependency here? If I try to apply the patch from that ticket to 5.0 beta2 after applying everything described above, I get an error stating that the $SAGE_ROOT/ext directory is missing...otherwise so far so good...

comment:107 follow-up: Changed 9 years ago by gutow

Arrrgh...after rebuild (./sage -b). The notebook launches with an error

ImportError                               Traceback (most recent call last)

/home/jonathan/.sage/<ipython console> in <module>()

/home/jonathan/Documents/sage-5.0.beta2/devel/sagenb-main/sagenb/notebook/notebook_object.pyc in __call__(self, *args, **kwds)
    215     """
    216     def __call__(self, *args, **kwds):
--> 217         return self.notebook(*args, **kwds)
    218 
    219     notebook = run_notebook.notebook_twisted

/home/jonathan/Documents/sage-5.0.beta2/devel/sagenb-main/sagenb/notebook/run_notebook.pyc in notebook_twisted(self, directory, port, interface, address, port_tries, secure, reset, accounts, require_login, server_pool, ulimit, timeout, open_viewer, sagetex_path, start_path, fork, quiet, subnets)
    200         print '*' * 70
    201 
--> 202     nb = notebook.load_notebook(directory)
    203 
    204     directory = nb._dir

/home/jonathan/Documents/sage-5.0.beta2/devel/sagenb-main/sagenb/notebook/notebook.pyc in load_notebook(dir, interface, port, secure)
   1807     # mainly to avoid circular references, etc.  This also means
   1808     # only one notebook can actually be used at any point.
-> 1809     import sagenb.notebook.twist
   1810     sagenb.notebook.twist.notebook = nb
   1811 

/home/jonathan/Documents/sage-5.0.beta2/devel/sagenb-main/sagenb/notebook/twist.py in <module>()
     40 
     41 from twisted.python import log
---> 42 from twisted.web2 import server, http, resource, channel
     43 from twisted.web2 import static, http_headers, responsecode
     44 from twisted.web2.filter import gzip

ImportError: No module named web2

Did I miss something or is there a new problem?

comment:108 in reply to: ↑ 107 Changed 9 years ago by dimpase

ImportError?: No module named web2 }}} Did I miss something or is there a new problem?

I bet you did, as there is no web2 in the new twisted (version 11).

comment:109 Changed 9 years ago by jason

And there is no twist.py in the new notebook... Are you *sure* that you installed the sagenb spkg above? Does devel/sagenb-main/flask_version exist?

You may need to do sage -f instead of sage -i when you install the spkgs above so they overwrite old versions of the same spkg.

comment:110 Changed 9 years ago by gutow

Yep...the sagenb spkg install failed on a missing header for ssl. Somehow I missed the error message. I have the ssl binaries but not the header files on my systems. I guess ssl.h be part of the package or they need to be listed as prerequisite for the sagenb spkg. I'm running on the same Ubuntu 11.04 that I successfully installed this on through 4.8.

Here's the relevant part of the log file.

Processing pyOpenSSL-0.13.tar.gz
Running pyOpenSSL-0.13/setup.py -q bdist_egg --dist-dir /tmp/easy_install-pWFAnl/pyOpenSSL-0.13/egg-dist-tmp-4xiQaN
warning: no previously-included files matching '*.pyc' found anywhere in distribution
In file included from OpenSSL/crypto/crypto.h:30:0,
                 from OpenSSL/crypto/crypto.c:16:
OpenSSL/crypto/x509.h:17:25: fatal error: openssl/ssl.h: No such file or directory
compilation terminated.
error: Setup script exited with error: command 'gcc' failed with exit status 1
Error installing pyOpenSSL-0.13.tar.gz.

comment:111 Changed 9 years ago by gutow

sagenb spkg requires openssl and openssl-dev. After adding openssl-dev to my system the spkg installs fine. If I missed this dependency sorry. If not it needs to be clear somewhere.

comment:112 Changed 9 years ago by jason

Should we put an explicit check for that file into the spkg-install? If so, what should that check look like? We should also clearly list this somewhere.

comment:113 in reply to: ↑ 101 Changed 9 years ago by ppurka

Replying to jason:

I explicitly tried each of those steps, and things worked flawlessly. I'm trying this on a 5.0 beta (which is where the notebook should be tested). I wonder if that makes a difference (it shouldn't, but who knows?).

Just to make sure: did you explicitly do each of the steps in the description of this ticket?

Also, is there any error message printed out for you in the terminal?

Thanks to kini we have the answer to this. It seems the error is triggered only when the password length is less than 6 characters. This means that the min password length of 6 is hard-coded in some file. The new git revisions of the nb which has set the variable min_password_length to 6 effectively masks the problem without fixing it.

comment:114 Changed 9 years ago by strogdon

I have a question about the following chunk of code

try: 
    # simplejson is faster, so try to import it first
    import simplejson as json
except ImportError: 
    import json

in sagenb/notebook/misc.py. It would seem that Sage is not shipped with simplejson. In testing the flask notebook on sage-on-gentoo the sagenb/notebook/misc.py test is the only failing notebook test because simplejson, which is pulled in by werkzeug as a runtime dependency, is found. The failure, roughly, is

sage -t -long  -force_lib /storage/strogdon/gentoo/usr/lib/python2.7/site-packages/sagenb/notebook/misc.py
**********************************************************************
File "/storage/strogdon/gentoo/usr/lib/python2.7/site-packages/sagenb/notebook/misc.py", line 221:
    sage: print encode_response(d, separators = (', ', ': '), indent = 4)
Exception raised:
    Traceback (most recent call last):
...
      File "element.pyx", line 1305, in sage.structure.element.RingElement.__add__ (sage/structure/element.c:11569)
      File "coerce.pyx", line 797, in sage.structure.coerce.CoercionModel_cache_maps.bin_op (sage/structure/coerce.c:7467)
    TypeError: unsupported operand parent(s) for '+': '<type 'str'>' and 'Integer Ring'
**********************************************************************

If I remove the conditional the test passes. So, the question is, is the conditional reference to simplejson needed? Of course, the s-o-g flask notebook can always be patched.

comment:115 Changed 9 years ago by kini

Patching the sage-on-gentoo flask notebook is not the solution. Test failures should not occur no matter what python modules, or indeed other programs, the user has installed into Sage's directory.

But there's actually a new failure, on 5.0.beta4:

sage -t  misc/sageinspect.py
**********************************************************************
File "/home/fs/src/sagenb/sagenb/misc/sageinspect.py", line 653:
    sage: sage_getargspec(Poset)
Expected:
    (['data', 'element_labels', 'cover_relations'], None, None, (None, None, False))
Got:
    (['data', 'element_labels', 'cover_relations', 'category', 'facade', 'key'], None, None, (None, None, False, None, None, None))
**********************************************************************
1 items had failures:
   1 of   7 in __main__.example_16
***Test Failed*** 1 failures.

I presume this is caused by #10998.

comment:116 Changed 9 years ago by kini

(Why the heck is sageinspect in sagenb anyway...)

comment:117 Changed 9 years ago by kini

I submitted a pull request to update the failing doctest. Looking at the simple_json one now...

comment:118 Changed 9 years ago by kini

The problem is as follows. In the json module, we have

If ``indent`` is a non-negative integer, then JSON array elements and
object members will be pretty-printed with that indent level. An indent
level of 0 will only insert newlines. ``None`` is the most compact
representation.

On the other hand, in the simple_json module, we have

If ``indent`` is a string, then JSON array elements and object members
will be pretty-printed with a newline followed by that string repeated
for each level of nesting. ``None`` (the default) selects the most compact
representation without any newlines. For backwards compatibility with
versions of simplejson earlier than 2.1.0, an integer is also accepted
and is converted to a string with that many spaces.

Sage integers do not trigger the following lines in simple_json (simplejson/encoder.py:173):

        if isinstance(indent, (int, long)):
            indent = ' ' * indent
        self.indent = indent

Unfortunately it doesn't seem like the json module supports this new string form of the indent argument, which is why it works with our code - it just does ' ' * indent without caring that it's a Sage integer. Apparently the json module is just an old version of the simplejson module which is bundled with Python, so it will get the new string-based indent parameter eventually, but for now we'll either have to require an up-to-date simplejson for the new notebook, or remember/detect which of the two modules we imported and call the function accordingly.

comment:119 Changed 9 years ago by kini

FWIW I made a pull request on simplejson addressing this: https://github.com/simplejson/simplejson/pull/29

comment:120 Changed 9 years ago by kini

The pull request has been merged into simplejson, so the test will no longer fail once the next version of simplejson is released.

comment:121 follow-up: Changed 9 years ago by kini

The new version of simplejson has been released. strogdon, if you upgrade simplejson to 2.3.3 the test failure should be gone :)

comment:122 in reply to: ↑ 121 Changed 9 years ago by strogdon

Replying to kini:

The new version of simplejson has been released. strogdon, if you upgrade simplejson to 2.3.3 the test failure should be gone :)

Good deal. I just upgraded to 2.3.3 and the test passes. Thanks.

comment:123 follow-up: Changed 9 years ago by strogdon

It's not clear from this ticket which of the twisted patches from #11874 should be applied. I believe it should be trac-11874-remove-twisted.3.patch . At least that one as well as the indicated jmol-spkg patch apply on top of 5.0.beta6 and I'm able to proceed to the "sage -br" step to get the new notebook working.

comment:124 in reply to: ↑ 123 Changed 9 years ago by jdemeyer

Replying to strogdon:

It's not clear from this ticket which of the twisted patches from #11874 should be applied.

Considering that ticket needs_work, the "correct" patch for #11874 still needs to be written.

comment:125 Changed 9 years ago by jason

Current status:

#11874 needs to be fixed and reviewed. There is a failing doctest, but I think the failing doctest may be resolved by using the sage.misc.fpickle module (I'm not sure, though)

#12229 needs review (and possibly a fix for the git_links)

#11503 needs review (and looks like it is ready to go...)

Merge pull request https://github.com/sagemath/sagenb/pull/40

comment:126 follow-up: Changed 9 years ago by dimpase

Please check out http://trac.sagemath.org/sage_trac/ticket/11874#comment:44

It appears that when we build Python 2.7.2 we automatically pull the latest Twisted (currently version 12) from the net and install it into SAGE_LOCAL.

We definitely, definitely do NOT want TWO twisted frameworks installed!!!

comment:127 Changed 9 years ago by jason

  • Description modified (diff)

Change urls to boxen instead of sage.math since sage.math is down.

comment:128 in reply to: ↑ 126 Changed 9 years ago by dimpase

Replying to dimpase:

Please check out http://trac.sagemath.org/sage_trac/ticket/11874#comment:44

It appears that when we build Python 2.7.2 we automatically pull the latest Twisted (currently version 12) from the net and install it into SAGE_LOCAL.

Please ignore this comment. It was somehow due to my confusion when I recently looked at #11874.

Please also note that #11874 needs review now.

comment:129 Changed 9 years ago by dimpase

running Explicit instructions from the ticket description gives one Twisted 12, not (as intended) Twisted 11.1. This needs to be fixed: please see http://trac.sagemath.org/sage_trac/ticket/11874#comment:62

comment:130 Changed 9 years ago by jdemeyer

  • Status changed from needs_info to needs_review

There is a doctest error:

sage -t  -force_lib devel/sagenb-main/sagenb/misc/sageinspect.py
**********************************************************************
File "/padic/scratch/jdemeyer/merger/sage-5.0.notebook/devel/sagenb-main/sagenb/misc/sageinspect.py", line 653:
    sage: sage_getargspec(Poset)
Expected:
    (['data', 'element_labels', 'cover_relations'], None, None, (None, None, False))
Got:
    (['data', 'element_labels', 'cover_relations', 'category', 'facade', 'key'], None, None, (None, None, False, None, None, None))
**********************************************************************

comment:131 follow-up: Changed 9 years ago by jdemeyer

  • Status changed from needs_review to needs_work

comment:132 in reply to: ↑ 131 ; follow-up: Changed 9 years ago by dimpase

Replying to jdemeyer: as mentioned here one just needs to update the spkg using this Git commit

comment:133 Changed 9 years ago by kini

Well, you need to update the spkg to the current master branch, which happens to contain that commit in its ancestry, and thus contains the code change. (Commits are not something you apply individually like patches.)

comment:134 Changed 9 years ago by kini

  • Description modified (diff)
  • Status changed from needs_work to needs_review

Here's a new spkg which should pass doctests: http://boxen.math.washington.edu/home/keshav/files/sagenb-0.9.0.spkg

Are there any other work issues?

comment:135 in reply to: ↑ 132 Changed 9 years ago by kini

  • Status changed from needs_review to needs_work

Hmm, there is something weird when I install the spkg and run sage -t $SAGE_ROOT/devel/sagenb-main: 1) README.rst is getting doctested, which throws up a password prompt; 2) there's a tab character in flask_version/base.py .

Why were these not showing up in earlier doctests? They're not new problems, as far as I can tell.

comment:136 Changed 9 years ago by kini

  • Status changed from needs_work to needs_review

OK, uploaded a new SPKG to the same URL. Now it definitely does pass all its doctests.

comment:137 Changed 9 years ago by jason

Just curious, the new spkg probably has Twisted 12.0, right? Will people at #11874 complain that Twisted was updated in the spkg?

comment:138 follow-up: Changed 9 years ago by kini

Yes, it contains Twisted 12. Yes, they might complain... feel free to create your own spkg with Twisted 11 instead - I might have messed something up anyway :)

comment:139 follow-up: Changed 9 years ago by jason

I won't have very much time this week to work on this, so I'm happy letting you make an spkg.

comment:140 in reply to: ↑ 138 Changed 9 years ago by dimpase

Replying to kini:

Yes, it contains Twisted 12. Yes, they might complain... feel free to create your own spkg with Twisted 11 instead - I might have messed something up anyway :)

No complaints from me. Actually, with the old sagenb 0.9.0 (ie with Twisted 11), when I replaced sagenb spkg in Sage 5.0.beta8 with 0.9.0, and then followed the procedure on #11874, I got Sage built with Twisted 11.1, instead of Twisted 12. So Twisted 12 was sneaking in as it was getting installed before the installation of the old sagenb 0.9.0.

comment:141 in reply to: ↑ 139 Changed 9 years ago by kini

Replying to jason:

I won't have very much time this week to work on this, so I'm happy letting you make an spkg.

OK, just checking that I did everything right... first I cloned the git repo to another git repo (to just get master and no other commits), then used hg-git to pull that clone into a newly-initialized hg repo. Then I ran ./spkg-dist to create the spkg. Did I do anything wrong?

comment:142 Changed 9 years ago by jdemeyer

  • Status changed from needs_review to needs_work

I again get

sage -t  -force_lib devel/sagenb-main/sagenb/misc/sageinspect.py
**********************************************************************
File "/scratch/jdemeyer/merger/sage-5.0.notebook/devel/sagenb-main/sagenb/misc/sageinspect.py", line 653:
    sage: sage_getargspec(Poset)
Expected:
    (['data', 'element_labels', 'cover_relations'], None, None, (None, None, False))
Got:
    (['data', 'element_labels', 'cover_relations', 'category', 'facade', 'key'], None, None, (None, None, False, None, None, None))
**********************************************************************

comment:143 Changed 9 years ago by jason

  • Description modified (diff)

Update instructions to reflect another patch on #11874 (I didn't test the instructions, though!)

comment:144 Changed 9 years ago by ppurka

  • Description modified (diff)

Update to new jmol patch for SAGE_ROOT.


Also, after applying the changes from #11874, #11503 and this ticket, the doctest of devel/sagenb-main/sagenb passes.

After these changes however, there is a problem with typesetting maths and I get this notice on evaluation of every cell, whether it is plot or maths.

Error typesetting mathematics.

And sure enough, something as simple as view(2+2) shows the latex code instead of the maths. jsmath is loaded and the button is present on the right hand corner.

UPDATE: I have same problem with sagenb from github with mathjax instead of jsmath. Seems like something is broken in sage and not sagenb.

Last edited 9 years ago by ppurka (previous) (diff)

comment:145 follow-up: Changed 9 years ago by jdemeyer

  • Milestone changed from sage-5.0 to sage-5.1

We would like to release sage-5.0 reasonably soon, and it's too late to get this still in sage-5.0.

comment:146 follow-up: Changed 9 years ago by kini

  • Description modified (diff)
  • Status changed from needs_work to needs_review

Jeroen: Either the version you installed was from the URL in the ticket description (the one Jason uploaded a while ago) and not the SPKG I posted in a comment recently, or I screwed up making my SPKG and mistakenly packaged an old version of the sagenb code. So to cover both possibilities, I changed the link in the ticket description to point to the SPKG I'm hosting, and I also repacked the SPKG and reuploaded it. It passes doctests on my machine. Can you try it again?

comment:147 in reply to: ↑ 146 Changed 9 years ago by jdemeyer

Replying to kini:

Jeroen: Either the version you installed was from the URL in the ticket description (the one Jason uploaded a while ago)

of course the version I installed was from the URL in the ticket description. I'm not going to read 146 comments to find out the URL of the spkg to test.

comment:148 follow-up: Changed 9 years ago by kini

Right, sorry about that. From your statement that you "again" saw the doctest errors, I assumed that at some point you had not seen the errors, which would have meant that you had installed my SPKG at some point. But I guess you just meant that you were continuing to see the doctest errors. My mistake :)

comment:149 in reply to: ↑ 148 Changed 9 years ago by jdemeyer

Replying to kini:

Right, sorry about that. From your statement that you "again" saw the doctest errors, I assumed that at some point you had not seen the errors, which would have meant that you had installed my SPKG at some point.

Maybe I had installed your SPKG at some point, then later forgot about your SPKG and installed Jason's SPKG again. I don't remember...

comment:150 Changed 9 years ago by kini

Fair enough - anyway it's my fault for not putting the correct info in the ticket description.

comment:151 Changed 9 years ago by gutow

  • Description modified (diff)

update jmol spkg to match #11503

comment:152 in reply to: ↑ 145 ; follow-up: Changed 9 years ago by was

Replying to jdemeyer:

We would like to release sage-5.0 reasonably soon, and it's too late to get this still in sage-5.0.

I agree with this decision.

comment:153 in reply to: ↑ 152 Changed 9 years ago by jason

Replying to was:

Replying to jdemeyer:

We would like to release sage-5.0 reasonably soon, and it's too late to get this still in sage-5.0.

I agree with this decision.

Same here. I had to put the proverbial rallying flag down due to work, but no one has really picked it back up again and finished the job. I'll be able to work on this more in a few weeks after our classes get out, if someone else hasn't started pushing it again by then.

comment:154 Changed 9 years ago by kini

What exactly remains to be done, other than the minor hiccup on #11503?

comment:155 follow-up: Changed 9 years ago by jason

What I would do to answer that question is go through each ticket dependency and see if it is positive-reviewed. If not, then that ticket needs to be done :). I don't think there is much left to be done.

comment:156 Changed 9 years ago by gutow

  • Description modified (diff)

comment:157 in reply to: ↑ 155 ; follow-up: Changed 9 years ago by dimpase

Replying to jason:

What I would do to answer that question is go through each ticket dependency and see if it is positive-reviewed. If not, then that ticket needs to be done :). I don't think there is much left to be done.

I think one should provide a recipe for preparing a sage distribution with all these patches applied to be built from scratch, rather than patching an already built distribution. (Assuming one has hg (or a previous version of Sage) installed system-wide). That is, most of these patches are to be applied to sage-??? spkg.

It makes more sense this way, as the changes are so extensive that it's time-consuming, and more error-prone, I think, to apply these to an already built installation of Sage.

comment:158 in reply to: ↑ 157 Changed 9 years ago by jdemeyer

Replying to dimpase:

I think one should provide a recipe for preparing a sage distribution with all these patches applied to be built from scratch, rather than patching an already built distribution. (Assuming one has hg (or a previous version of Sage) installed system-wide). That is, most of these patches are to be applied to sage-??? spkg.

I have made http://boxen.math.washington.edu/home/release/sage-5.0.notebook/sage-5.0.notebook.tar to test this ticket and its dependencies.

comment:159 Changed 9 years ago by jdemeyer

  • Status changed from needs_review to needs_work

This fails to install on OS X 10.4 PPC:

Processing pyOpenSSL-0.13.tar.gz
Running pyOpenSSL-0.13/setup.py -q bdist_egg --dist-dir /tmp/easy_install-h8pJu5/pyOpenSSL-0.13/egg-dist-tmp-Awaj3a
warning: no previously-included files matching '*.pyc' found anywhere in distribution
OpenSSL/ssl/connection.c: In function 'ssl_Connection_set_context':
OpenSSL/ssl/connection.c:289:5: warning: implicit declaration of function 'SSL_set_SSL_CTX' [-Wimplicit-function-declaration]
OpenSSL/ssl/connection.c: In function 'ssl_Connection_get_servername':
OpenSSL/ssl/connection.c:313:16: error: 'TLSEXT_NAMETYPE_host_name' undeclared (first use in this function)
OpenSSL/ssl/connection.c:313:16: note: each undeclared identifier is reported only once for each function it appears in
OpenSSL/ssl/connection.c:320:5: warning: implicit declaration of function 'SSL_get_servername' [-Wimplicit-function-declaration]
OpenSSL/ssl/connection.c:320:10: warning: assignment makes pointer from integer without a cast [enabled by default]
OpenSSL/ssl/connection.c: In function 'ssl_Connection_set_tlsext_host_name':
OpenSSL/ssl/connection.c:346:5: warning: implicit declaration of function 'SSL_set_tlsext_host_name' [-Wimplicit-function-declaration]
error: Setup script exited with error: command 'gcc' failed with exit status 1
Error installing pyOpenSSL-0.13.tar.gz.

comment:160 Changed 9 years ago by jdemeyer

This also fails to install on the Skynet machine silius (SUSE ES 11 SP1 ppc64):

Processing pyOpenSSL-0.13.tar.gz
Running pyOpenSSL-0.13/setup.py -q bdist_egg --dist-dir /tmp/easy_install-_3ma5D/pyOpenSSL-0.13/egg-dist-tmp-efVFsy
warning: no previously-included files matching '*.pyc' found anywhere in distribution
In file included from OpenSSL/crypto/crypto.h:30:0,
                 from OpenSSL/crypto/crypto.c:16:
OpenSSL/crypto/x509.h:17:25: fatal error: openssl/ssl.h: No such file or directory
compilation terminated.
error: Setup script exited with error: command 'gcc' failed with exit status 1
Error installing pyOpenSSL-0.13.tar.gz.

comment:161 Changed 9 years ago by kini

According to http://stackoverflow.com/questions/7340784/easy-install-pyopenssl-error we should be able to fix this by requiring pyOpenSSL 0.12 or lower.

Edit: er, I meant the problem on OS X - dunno about this silius one.

Last edited 9 years ago by kini (previous) (diff)

comment:162 follow-up: Changed 9 years ago by jdemeyer

What's the situation with OpenSSL support anyway? I believe it was decided not to require OpenSSL as dependency? But there have been many discussions about this, I might have lost track.

comment:163 Changed 9 years ago by jdemeyer

Also on the Skynet machine cleo (RHEL 5.3 ia64):

Processing pyOpenSSL-0.13.tar.gz
Running pyOpenSSL-0.13/setup.py -q bdist_egg --dist-dir /tmp/easy_install-PQFiHf/pyOpenSSL-0.13/egg-dist-tmp-XsFtPR
warning: no previously-included files matching '*.pyc' found anywhere in distribution
OpenSSL/ssl/connection.c: In function ssl_Connection_set_context:
OpenSSL/ssl/connection.c:289:5: warning: implicit declaration of function SSL_set_SSL_CTX [-Wimplicit-function-declaration]
OpenSSL/ssl/connection.c: In function ssl_Connection_get_servername:
OpenSSL/ssl/connection.c:313:16: error: TLSEXT_NAMETYPE_host_name undeclared (first use in this function)
OpenSSL/ssl/connection.c:313:16: note: each undeclared identifier is reported only once for each function it appears in
OpenSSL/ssl/connection.c:320:5: warning: implicit declaration of function SSL_get_servername [-Wimplicit-function-declaration]
OpenSSL/ssl/connection.c:320:10: warning: assignment makes pointer from integer without a cast [enabled by default]
OpenSSL/ssl/connection.c: In function ssl_Connection_set_tlsext_host_name:
OpenSSL/ssl/connection.c:346:5: warning: implicit declaration of function SSL_set_tlsext_host_name [-Wimplicit-function-declaration]
error: Setup script exited with error: command 'gcc' failed with exit status 1
Error installing pyOpenSSL-0.13.tar.gz.

comment:164 Changed 9 years ago by jdemeyer

On the Skynet machine iras (SUSE ES10 ia64):

Processing pyOpenSSL-0.13.tar.gz
Running pyOpenSSL-0.13/setup.py -q bdist_egg --dist-dir /tmp/easy_install-9gn9yE/pyOpenSSL-0.13/egg-dist-tmp-4CCNMQ
warning: no previously-included files matching '*.pyc' found anywhere in distribution
In file included from OpenSSL/crypto/crypto.h:30:0,
                 from OpenSSL/crypto/crypto.c:16:
OpenSSL/crypto/x509.h:17:25: fatal error: openssl/ssl.h: No such file or directory
compilation terminated.
error: Setup script exited with error: command 'gcc' failed with exit status 1
Error installing pyOpenSSL-0.13.tar.gz.

comment:165 Changed 9 years ago by jdemeyer

On UGA rosemary (RHEL 5.6 x86_64):

Processing pyOpenSSL-0.13.tar.gz
Running pyOpenSSL-0.13/setup.py -q bdist_egg --dist-dir /tmp/easy_install-urCg6W/pyOpenSSL-0.13/egg-dist-tmp-fw0DPw
warning: no previously-included files matching '*.pyc' found anywhere in distribution
OpenSSL/ssl/connection.c: In function ‘ssl_Connection_set_context’:
OpenSSL/ssl/connection.c:289:5: warning: implicit declaration of function ‘SSL_set_SSL_CTX’ [-Wimplicit-function-declaration]
OpenSSL/ssl/connection.c: In function ‘ssl_Connection_get_servername’:
OpenSSL/ssl/connection.c:313:16: error: ‘TLSEXT_NAMETYPE_host_name’ undeclared (first use in this function)
OpenSSL/ssl/connection.c:313:16: note: each undeclared identifier is reported only once for each function it appears in
OpenSSL/ssl/connection.c:320:5: warning: implicit declaration of function ‘SSL_get_servername’ [-Wimplicit-function-declaration]
OpenSSL/ssl/connection.c:320:10: warning: assignment makes pointer from integer without a cast [enabled by default]
OpenSSL/ssl/connection.c: In function ‘ssl_Connection_set_tlsext_host_name’:
OpenSSL/ssl/connection.c:346:5: warning: implicit declaration of function ‘SSL_set_tlsext_host_name’ [-Wimplicit-function-declaration]
error: Setup script exited with error: command 'gcc' failed with exit status 1
Error installing pyOpenSSL-0.13.tar.gz.

comment:166 in reply to: ↑ 162 Changed 9 years ago by dimpase

Replying to jdemeyer:

What's the situation with OpenSSL support anyway? I believe it was decided not to require OpenSSL as dependency? But there have been many discussions about this, I might have lost track.

See https://groups.google.com/d/msg/sage-devel/iwrF8_kGLzM/32HIts7EPKQJ

so we'll require openssl, but we need to sort out the versions issue, requiring pyOpenSSL 0.12 or lower, as Keshav suggests.

comment:167 Changed 9 years ago by kini

New SPKG with pyOpenSSL 0.12 uploaded. All tests pass on sage.math. Jeroen, can you test it on the buildbots again? It seems like all the errors you reported are coming from pyOpenSSL so I hope downgrading it will solve them...

comment:168 Changed 9 years ago by kini

  • Status changed from needs_work to needs_review

Sorry, I guess I should mark this needs_review again.

comment:169 Changed 9 years ago by jdemeyer

  • Description modified (diff)

comment:170 Changed 9 years ago by SimonKing

I tried to follow the instructions, but the patch from #11874 fails to apply. Interesting that it was possible to create a testing release, then...

comment:171 Changed 9 years ago by kini

Huh, really? For me, the only dependency that didn't apply was #12327. #11874 worked fine for me.

comment:172 Changed 9 years ago by SimonKing

I think I start to understand. It does not apply with sage-5.0.prealpha0. But in sage-5.0.beta13, the files look totally different.

So, I may (at some point) try again with the latest beta.

comment:173 Changed 9 years ago by SimonKing

  • Description modified (diff)

The instructions did not tell that one is supposed to start with the latest beta version. Starting with 5.0.prealpha0, things fail to apply.

Note: See TracTickets for help on using tickets.