Opened 9 years ago

Closed 9 years ago

## #13121 closed enhancement (fixed)

Reported by: Owned by: kini jason, mpatel, was major sage-5.4 notebook sagenb jhpalmieri, gutow, jyeo, rbeezer sage-5.4.beta0 Keshav Kini John Palmieri, Jonathan Gutow N/A #11080, #9774, #11913, #12299, #13384

This is a followup to #11080. During the review process of #11080, which stretched for quite a while, at some point we decided to have a feature freeze on the version of sagenb 0.9.x associated with #11080, just to simplify the review process. The purpose of this ticket is to incorporate some new features, bugfixes, code refactoring, etc. into the version of sagenb shipped with Sage, beyond what was reviewed in #11080. sagenb 0.10 contains the following (among other things):

• refactoring of the startup code, uWSGI and Tornado added as possible WSGI servers
• code allowing .sws files to be uploaded by supplying a URL pointing to the .sws as an argument to the upload page's URL
• new dependency on webassets for js/css autominification (not currently used in the master branch)
• the mathjax branch, and a doctest fix in sphinxify.py (#9774 depends on and contributed to this)
• support for uploading .sws files from the local disk via the command line (#8473 basically is this)
• refactoring of sageinspect to not have a duplication of code between the notebook and the Sage library (#11913 depends on this)
• Upgrading Jmol again and allowing for advanced interactive features more easily, solve some memory issues (#12299)
• Remove strict dependency on pyOpenSSL; allow for graceful failure of HTTPS functionality when pyOpenSSL is not installed (#13384 and #13385 coordinate with this)

Changes not in sagenb itself but in how sagenb is incorporated into sage:

• No repository will be installed with sagenb - developers will need to clone sagenb themselves and install the new clone into their Sage installation manually (which is not difficult).

Instructions for installing this and its dependencies into >=sage-5.2.beta0 :



needs to take into account that $SAGE64 may be set to "no". According to SAGE_ROOT/spkg/bin/sage-arch-env, it should be unset, "no", or "yes", so you should test that it's set to "yes". ### comment:8 Changed 9 years ago by jhpalmieri You should also take this possible change to CPPFLAGS into account. For example: • ## spkg-install diff --git a/spkg-install b/spkg-install  a die () { export CPATH="$CPATH""$SAGE_LOCAL"/include export LIBRARY_PATH="$LIBRARY_PATH""$SAGE_LOCAL"/lib export CPPFLAGS="-I$SAGE_LOCAL/include $CPPFLAGS" export CPPFLAGS="-I$SAGE_LOCAL/include $CPPFLAGS -D_XOPEN_SOURCE=500" export LDFLAGS="-L$SAGE_LOCAL/lib $LDFLAGS" if [ -n$SAGE64 ]; then if [ $SAGE64 = "yes" ]; then echo "Building with extra 64-bit flags for MacOS X and Open Solaris." if [ -z$CFLAG64 ]; then CFLAG64=-m64

(even better, with a comment explaining why CPPFLAGS has been changed)

Last edited 9 years ago by jhpalmieri (previous) (diff)

### comment:9 Changed 9 years ago by jhpalmieri

With these changes, this spkg builds on OS X Lion, sage.math, hawk, and skynet machines mark (after installing the openssl spkg from #13126), cicero, iras (with openssl), silius (with openssl), and taurus. So it's looking good to me.

### comment:10 Changed 9 years ago by jason

• Description modified (diff)

### comment:11 Changed 9 years ago by kini

• Status changed from needs_work to needs_review

SPKG updated with your changes, more or less:

• ## spkg-install

diff --git a/spkg-install b/spkg-install
 a export CPATH="$CPATH""$SAGE_LOCAL"/include export LIBRARY_PATH="$LIBRARY_PATH""$SAGE_LOCAL"/lib export CPPFLAGS="-I$SAGE_LOCAL/include$CPPFLAGS" # note: the -D_XOPEN_SOURCE=500 is to fix build errors in Twisted 12.1 on # Solaris. See http://trac.sagemath.org/sage_trac/ticket/11080#comment:314 and # the subsequent few comments. export CPPFLAGS="-I$SAGE_LOCAL/include -D_XOPEN_SOURCE=500$CPPFLAGS" export LDFLAGS="-L$SAGE_LOCAL/lib$LDFLAGS" if [ -n $SAGE64 ]; then if [$SAGE64 = "yes" ]; then echo "Building with extra 64-bit flags for MacOS X and Open Solaris." if [ -z $CFLAG64 ]; then CFLAG64=-m64 ### comment:12 Changed 9 years ago by jhpalmieri On my Mac (OS X 10.7), I can't get worksheets to actually evaluate anything. With Chrome, Firefox, and Safari, I see an evaluate button, but clicking on it does nothing. Hitting 'shift-enter' either does nothing (Firefox) or gives me a new line in the current cell (Chrome and Safari), and does not evaluate anything. Hitting 'Save & quit' does nothing (Chrome and Safari) or actually quits the current worksheet (Firefox). I think I see some of the same behavior with 0.9.0, but not as severe: it works in Firefox, but not in Chrome or Safari. I'm not sure about that, though; I'm not sure my 0.9.0 install was clean, or whether I corrupted it with 0.9.1 related things. Does it matter that there is no directory for this version of sagenb in SAGE_LOCAL/lib/python/site-packages/? After installing 0.9.0 and then 0.9.1, I see  drwxr-xr-x 4 palmieri admin 136 Jun 19 11:13 sagenb-0.8.29-py2.7.egg drwxr-xr-x 4 palmieri admin 136 Jun 21 16:47 sagenb-0.9.0-py2.7.egg -rw-r--r-- 1 palmieri admin 26 Jun 21 16:59 sagenb.egg-link  ### comment:13 Changed 9 years ago by kini I was able to duplicate this on linux. The problem went away when I did a hard refresh of the page, forcing my browser to reload all javascript. The problem is probably caused by this or this. Does a hard refresh solve the problem on your end too? ### comment:14 Changed 9 years ago by jhpalmieri A hard refresh helps to get 0.9.0 to work, but not 0.9.1. By the way, when the notebook is running, how can I tell which version is running? I would suggest having a variable, or displaying it somewhere when the notebook opens. ### comment:15 Changed 9 years ago by kini • Description modified (diff) • Status changed from needs_review to needs_work • Summary changed from Upgrade sagenb to 0.9.1 to Upgrade sagenb to 0.10 • Work issues set to fix js typo, make new spkg Um, wow. Silly typo in this commit. I'll fix this, and actually I think I'm going to call this 0.10. Our version numbers are moving too slowly. Adding mathjax is a lot more substantial of a change than a bump to the patch level should cause a user to expect. Last edited 9 years ago by kini (previous) (diff) ### comment:16 Changed 9 years ago by kini • Summary changed from Upgrade sagenb to 0.10 to Upgrade sagenb to 0.10.0 ### comment:17 Changed 9 years ago by kini • Authors set to Keshav kini • Description modified (diff) • Report Upstream changed from Fixed upstream, but not in a stable release. to N/A • Status changed from needs_work to needs_review • Work issues fix js typo, make new spkg deleted OK, new SPKG is up. By the way, when the notebook is running, how can I tell which version is running? I would suggest having a variable, or displaying it somewhere when the notebook opens. sagenb.misc.misc.SAGENB_VERSION should tell you that, but it's sort of a hack. I probably could have changed that in the packaging rewrite... ### comment:18 Changed 9 years ago by jhpalmieri Okay, now a hard refresh is getting everything to work, on all three browsers. Now, is there any way to set it up to avoid the hard refresh? Otherwise users may be frustrated when they first switch to the new notebook. ### Changed 9 years ago by jhpalmieri scripts repo ### comment:19 Changed 9 years ago by jhpalmieri • Description modified (diff) I've attached a patch to put "webassets" into the .hgignore file. ### comment:20 Changed 9 years ago by kini Thanks for the .hgignore patch. As for avoiding hard refreshes, it might be possible to make Flask set some HTTP headers to the date/time when the server was started, to advise client-side caching. I'm not sure of the details, but it's probably not a one-liner like the typo fix I just made was; I'd also consider it more of a feature than a bugfix, so probably it won't go into 0.10. ### comment:21 Changed 9 years ago by kini • Authors changed from Keshav kini to Keshav Kini ### comment:22 Changed 9 years ago by kini (I've set myself as the author since this ticket is just about making an SPKG - there are many more important contributors to the new version of sagenb than me, for the record.) ### comment:23 follow-up: ↓ 28 Changed 9 years ago by jhpalmieri This is mostly working for me on OS X Lion, except issues with #8473. Otherwise, the spkg looks good to me. People should test it on other platforms. ### comment:24 Changed 9 years ago by jdemeyer • Priority changed from major to blocker Any objections to making this a blocker for sage-5.2? It feels to me like sage-0.9.0 is a kind of "beta version" for sage-0.10.0, so I would really prefer to have sage-0.10.0 in the final release. ### comment:25 Changed 9 years ago by jason Actually, I think of 0.9.0 as sort of the stable, LTS release where we had a *long* feature freeze and concentrated on stability. 0.10.0 is like the release just after LTS stable release, where we merged in a bunch of new functionality that was put off during the long feature freeze. There are some important bugfixes in 0.10.0 (IIRC), but there is a lot of new functionality in 0.10 too. ### comment:26 follow-up: ↓ 27 Changed 9 years ago by jason That said, I would prefer 0.10 to be in 5.2, but I think it's probably more important to have 0.9 out by the time school starts. There is still one or two rather frustrating known bugs in 0.9 (and 0.10): some sort of leak where worker processes don't get killed, and sometimes creating a new worksheet actually just gets an old worksheet. If we solve those in 0.10.1 or something, it would be great to have that as our before-semester release. ### comment:27 in reply to: ↑ 26 Changed 9 years ago by kcrisman That said, I would prefer 0.10 to be in 5.2, but I think it's probably more important to have 0.9 out by the time school starts. Correct. ### comment:28 in reply to: ↑ 23 ; follow-up: ↓ 29 Changed 9 years ago by kcrisman Replying to jhpalmieri: This is mostly working for me on OS X Lion, except issues with #8473. Otherwise, the spkg looks good to me. People should test it on other platforms. Just for reference, this was (or will be, when it's merged) dealt with - see this pull request on sagenb. ### comment:29 in reply to: ↑ 28 Changed 9 years ago by jhpalmieri Replying to kcrisman: Replying to jhpalmieri: This is mostly working for me on OS X Lion, except issues with #8473. Otherwise, the spkg looks good to me. People should test it on other platforms. Just for reference, this was (or will be, when it's merged) dealt with - see this pull request on sagenb. Right. Keshav, when you have a chance, could you make a new 0.10.0 spkg for us to test? Any notebook fans: can you positively review the last change at #11080 (the one which allows building on Solaris)? ### comment:30 Changed 9 years ago by kcrisman To clarify for whoever makes the "final" spkg here - to make sure #8473 is a-ok, we would need this pull request and this one to fix the minor issues John mentioned. Last edited 9 years ago by kcrisman (previous) (diff) ### comment:31 Changed 9 years ago by jhpalmieri I'm confused about the location of JMOL. It used to be (before #11080) part of sagenb. Now it isn't (or at least I don't see it anywhere). Does it need to be distributed as a separate spkg, and thus included in spkg/standard/deps? Or should the new JMOL spkg from #12299 be part of this spkg? Oh, I see: it is now in spkg/standard/deps, and it needs to be included as a separate spkg. Last edited 9 years ago by jhpalmieri (previous) (diff) ### comment:32 follow-up: ↓ 34 Changed 9 years ago by kini See #11503, where the deps file is patched to require the new JMOL spkg from #11503. For some reason #12299 seems to require a newer version of the SPKG, a version which is not on #11503 or anywhere else. I posted a comment on #11503 about it. ### comment:33 Changed 9 years ago by jhpalmieri The spkg here needs work: when it gets installed, SAGE_ROOT/devel may not yet exist. So something like this: • ## spkg-install diff --git a/spkg-install b/spkg-install  a BASENAME=${BASENAME%.tar.gz} #easy_install -H None -f src "sagenb" || die "Error installing sagenb !" # Extract sagenb into $SAGE_ROOT/devel and set up the symlink mkdir -p "$SAGE_ROOT/devel" cd "$SAGE_ROOT/devel" if [ -d "$SAGE_ROOT/devel/sagenb-main" ]; then echo "Moving old SageNB package to '$SAGE_ROOT/devel/sagenb-main-old'..." ### comment:34 in reply to: ↑ 32 Changed 9 years ago by kcrisman See #11503, where the deps file is patched to require the new JMOL spkg from #11503. For some reason #12299 seems to require a newer version of the SPKG, a version which is not on #11503 or anywhere else. I posted a comment on #11503 about it. As far as I know, this is on purpose, because the newest Jmol at #12299 needs the many updates to Jmol there. In that sense #11503 is a 'hard' prereq for #11080 and hence this ticket. I am pretty sure Jonathan was careful about exactly which version of Jmol needed to be with which other patches. ### comment:35 Changed 9 years ago by kini OK, I see, thanks. ### comment:36 Changed 9 years ago by kcrisman jhpalmieri or others: When you ran into the problem with the evaluate button not working, did you find that other buttons also stopped working? For instance, the Stop and Archive buttons on the main worksheet list page. I must have done something wrong because even a hard refresh isn't getting things to work for me, but I wanted to ask if anyone had tried those. ### comment:37 follow-up: ↓ 38 Changed 9 years ago by jhpalmieri For me, on some browsers, everything stopped working. A hard refresh did fix it (Firefox or Chrome: cmd-shift-r. Safari: hold the option key down while hitting the little "refresh" symbol at the right end of the address box). ### comment:38 in reply to: ↑ 37 Changed 9 years ago by kcrisman For me, on some browsers, everything stopped working. Ok. A hard refresh did fix it (Firefox or Chrome: cmd-shift-r. Safari: hold the option key down while hitting the little "refresh" symbol at the right end of the address box). Yup, that's what I tried, and that didn't work. But like I said on those commits in sagenb, maybe I just destroyed my installation anyway. Edit: I did finally manage to get the upstream commit of the parenthesis fix in to make it work with the "hard refresh". That definitely needs some big warning somewhere... Strange that doing it "by hand" didn't work, but getting from upstream did. Last edited 9 years ago by kcrisman (previous) (diff) ### comment:39 Changed 9 years ago by kini • Description modified (diff) ### comment:40 Changed 9 years ago by jhpalmieri • Status changed from needs_review to needs_work • Work issues set to create SAGE_ROOT/devel ### comment:41 Changed 9 years ago by kini • Description modified (diff) • Summary changed from Upgrade sagenb to 0.10.0 to Upgrade sagenb to 0.10.x • Work issues changed from create SAGE_ROOT/devel to wait for 0.10.1 Sorry for the delay, John. I'm going to release a 0.10.1 with the fixes for #8473 and also hopefully a fix for Jason Yeo's javascript event shuffling (see comments at the bottom of PR #69 on github), so I'm leaving this ticket as needs_work. ### comment:42 Changed 9 years ago by jdemeyer • Priority changed from blocker to major ### comment:43 Changed 9 years ago by kini • Description modified (diff) • Status changed from needs_work to needs_review • Work issues wait for 0.10.1 deleted Anything else? ### comment:44 Changed 9 years ago by jhpalmieri • Reviewers set to John Palmieri • Status changed from needs_review to positive_review I think this looks good now. The main parts have already been positively reviewed (as parts of the prerequisite tickets). The spkg-install script is in good shape, as is SPKG.txt. Let's merge it to get MathJax into Sage. ### comment:45 Changed 9 years ago by jdemeyer • Dependencies changed from #11080, #9774, #8473, #11913, #12299 to #11080, #9774, #11913, #12299 ### comment:46 Changed 9 years ago by kcrisman Probably meaningless, but I do get the following weirdness on OS X PPC when upgrading from 5.2.alpha0 (not beta0!) by using just the latest 0.10.1 spkg. Dasher-03:~/Desktop/sage-5.2.alpha0 student$ ./sage -i http://wstein.org/home/keshav/files/sagenb-0.10.1.spkg
Calling sage-spkg on 'http://wstein.org/home/keshav/files/sagenb-0.10.1.spkg'
http://wstein.org/home/keshav/files/sagenb-0.10.1.spkg --> sagenb-0.10.1.spkg
[..................................................]
sagenb-0.10.1
====================================================
Extracting package /Users/student/Desktop/sage-5.2.alpha0/spkg/optional/sagenb-0.10.1.spkg
-rw-r--r--   1 student  student  13506560 Jul 10 13:59 /Users/student/Desktop/sage-5.2.alpha0/spkg/optional/sagenb-0.10.1.spkg
Finished extraction
****************************************************
Host system:
Darwin Dasher-03.local 8.11.0 Darwin Kernel Version 8.11.0: Wed Oct 10 18:26:00 PDT 2007; root:xnu-792.24.17~1/RELEASE_PPC Power Macintosh powerpc
****************************************************
C compiler: gcc
C compiler version:
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/Users/student/Desktop/sage-5.2.alpha0/local/libexec/gcc/powerpc-apple-darwin8.11.0/4.6.3/lto-wrapper
Target: powerpc-apple-darwin8.11.0
Configured with: ../src/configure --prefix=/Users/student/Desktop/sage-5.2.alpha0/local --with-local-prefix=/Users/student/Desktop/sage-5.2.alpha0/local --with-gmp=/Users/student/Desktop/sage-5.2.alpha0/local --with-mpfr=/Users/student/Desktop/sage-5.2.alpha0/local --with-mpc=/Users/student/Desktop/sage-5.2.alpha0/local --with-system-zlib --disable-multilib
gcc version 4.6.3 (GCC)
****************************************************
Processing Twisted-12.1.0.tar.bz2
Running Twisted-12.1.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-UaYpDr/Twisted-12.1.0/egg-dist-tmp-ywlzEJ
conftest.c:1:23: fatal error: sys/epoll.h: No such file or directory
compilation terminated.
twisted/python/_initgroups.c: In function 'initgroups_initgroups':
twisted/python/_initgroups.c:39:2: warning: implicit declaration of function 'initgroups' [-Wimplicit-function-declaration]
twisted/python/sendmsg.c: In function 'sendmsg_sendmsg':
twisted/python/sendmsg.c:243:13: warning: implicit declaration of function 'CMSG_SPACE' [-Wimplicit-function-declaration]
twisted/python/sendmsg.c:306:13: warning: implicit declaration of function 'CMSG_LEN' [-Wimplicit-function-declaration]
twisted/internet/_sigchld.c: In function 'got_signal':
twisted/internet/_sigchld.c:15:9: warning: variable 'ignored_result' set but not used [-Wunused-but-set-variable]
Removing Twisted 12.0.0 from easy-install.pth file
Adding Twisted 12.1.0 to easy-install.pth file


after which all proceeds normally. It just looks a lot like comment:314:ticket:11080 so I wondered.

Edit:

./spkg-install: line 76: cd: sagenb: No such file or directory
python: can't open file 'setup.py': [Errno 2] No such file or directory
Error installing sagenb via setup.py develop

real    11m4.240s
user    3m0.371s
sys     2m27.994s
************************************************************************
Error installing package sagenb-0.10.1
************************************************************************


which is weird enough that I must have done something wrong. I'll try beta0 instead, sorry for the noise.

Last edited 9 years ago by kcrisman (previous) (diff)

### comment:47 Changed 9 years ago by kcrisman

Incidentally, are the instructions in the description correct for 5.2.beta0? I assume not, based on the presence of 0.10.0 and not 0.10.1. Thanks.

Last edited 9 years ago by kcrisman (previous) (diff)

### comment:48 Changed 9 years ago by kini

Oops, sorry. Indeed, the instructions are not correct for 5.2.beta0, but not just because of 0.10.0 vs. 0.10.1 - I need to remove the parts about adding patches that are already merged in 5.2.beta0.

### comment:49 Changed 9 years ago by kini

• Description modified (diff)

OK, the only remaining instructions should now be correct for 5.2.beta0... can you try again, kcrisman?

### comment:50 Changed 9 years ago by jdemeyer

• Milestone changed from sage-5.2 to sage-pending

### comment:51 Changed 9 years ago by kini

• Description modified (diff)

Added a missing patch from #12299.

### comment:53 Changed 9 years ago by gutow

• Description modified (diff)

The Jmol package has been updated to include more extensive version checking for the JVM.

### comment:54 Changed 9 years ago by ppurka

• Description modified (diff)

Update mathjax patch

### comment:55 Changed 9 years ago by gutow

• Description modified (diff)

update jmol package to match latest on #12299

### comment:57 Changed 9 years ago by novoselt

I've installed this version on Sage-5.2.beta1 and it is still not possible to sign out.

Moreover, when I removed all cookies of the site and got to the login page, I was able to register as a new user and I am pretty sure I have never enabled it (I did add one user manually as admin). I can't reproduce it, however, so maybe I've done something wrong.

### comment:58 Changed 9 years ago by kini

Jeroen: since 5.2.rc0 has been released, should I set this to the 5.3 milestone?

### comment:59 Changed 9 years ago by kini

Never mind. For some reason I thought 5.2.rc0 had been released, but apparently it has not. Sorry for the noise.

### comment:60 Changed 9 years ago by jdemeyer

sage-5.2.rc0 is as good as released (in the sense that I have decided which patches it will contain). So yes, this is going to be sage-5.3 or later.

### comment:61 Changed 9 years ago by gutow

• Description modified (diff)

Adding patch from #12299 to make java dependent doctests in jmoldata.py optional.

Last edited 9 years ago by gutow (previous) (diff)

### comment:62 Changed 9 years ago by gutow

• Status changed from positive_review to needs_work
• Work issues set to evaluate button focus,openssl

Two issues:

1. Evaluate button does not work for a cell until it loses focus to another (pull request 85 on git).
2. Secure mode does not work even with system wide ssl-dev headers ( I have libcurl4-openssl-dev and libssl-dev installed). Either the specifications or install checking need to be more specific. I suggest that the spkg check for what is needed and prompts for a local install of openssl if what is needed is not available.

Error on launch of notebook in secure mode (this goes away after local openssl install):

/home/sage/sage-5.2.beta1/<ipython console> in <module>()

/home/sage/sage-5.2.beta1/devel/sagenb/sagenb/notebook/notebook_object.py in __call__(self, *args, **kwds)
217     """
218     def __call__(self, *args, **kwds):
--> 219         return self.notebook(*args, **kwds)
220
221     notebook = run_notebook.notebook_run

/home/sage/sage-5.2.beta1/devel/sagenb/sagenb/notebook/run_notebook.py in notebook_run(self, directory, port, interface, port_tries, secure, reset, accounts, openid, server_pool, ulimit, timeout, upload, automatic_login, start_path, fork, quiet, server, profile, subnets, require_login, open_viewer, address)
614     os.chdir(cwd)
615     if e == 256:
--> 616         raise socket.error
617


Last edited 9 years ago by gutow (previous) (diff)

### comment:63 Changed 9 years ago by gutow

• Work issues changed from evaluate button focus,openssl to new worksheets should be empty,evaluate button focus,openssl

Another issue: New worksheets do not open empty. They now copy their content from one of the other worksheets.

### comment:64 Changed 9 years ago by kcrisman

Thank you for all this careful testing, Jonathan.

### comment:65 Changed 9 years ago by jason

Do new worksheets *always* copy their content? They only sporadically do that for me; maybe one in ten or twenty. If I open several new worksheets in succession, it eventually gets to a real blank worksheet and then things work okay from there.

I commented on https://github.com/sagemath/sagenb/issues/87

### comment:66 follow-up: ↓ 67 Changed 9 years ago by kini

Thank you for all this careful testing, Jonathan.

+1, yes, thank you :)

About the SSL problem, is there any further output than what you've given? Can you in fact just run sage -bn on the command line and paste everything you get from then until the next command prompt?

### comment:67 in reply to: ↑ 66 Changed 9 years ago by gutow

Thank you for all this careful testing, Jonathan.

+1, yes, thank you :)

You're welcome. I was just updating my server so thought I would try to pile in all the notebook stuff we thought was working.

About the SSL problem, is there any further output than what you've given?

That's all the error output I get when I try to start a secure notebook on my system, without doing a local install of openssl. Note, that this doesn't happen on all my systems. I suspect that the way openssl is searched for is not working. Ubuntu 12.04 are the systems I'm having trouble with.

Can you in fact just run sage -bn on the command line and paste everything you get from then until the next command prompt?

I'll see what I can do. I think I saved the .tar so can make a clean copy to work with.

Last edited 9 years ago by gutow (previous) (diff)

### comment:68 Changed 9 years ago by gutow

This time I trapped a more complete error message when I started the server in secure mode. This is on Ubuntu 12.04 with the following openssl related pacakges installed: openssl,python-openssl, libssl0.9.8,ssl-cert,libssl-doc,libssl-dev.

----------------------------------------------------------------------
| Sage Version 5.2.rc0, Release Date: 2012-07-17                     |
| Type "notebook()" for the browser-based notebook interface.        |
| Type "help()" for help.                                            |
----------------------------------------------------------------------
**********************************************************************
*                                                                    *
* Warning: this is a prerelease version, and it may be unstable.     *
*                                                                    *
**********************************************************************
The notebook files are stored in: sage_notebook.sagenb
****************************************************
*                                                  *
*                                                  *
****************************************************
There is an admin account.  If you do not remember the password,
quit the notebook and type notebook(reset=True).
Executing twistd  --pidfile="sage_notebook.sagenb/sagenb.pid" -ny "sage_notebook.sagenb/twistedconf.tac"
Unhandled Error
Traceback (most recent call last):
File "/media/Extra_Space/Sage_installs/sage-5.2.rc0/local/lib/python2.7/site-packages/Twisted-12.1.0-py2.7-linux-i686.egg/twisted/application/app.py", line 652, in run
runApp(config)
File "/media/Extra_Space/Sage_installs/sage-5.2.rc0/local/lib/python2.7/site-packages/Twisted-12.1.0-py2.7-linux-i686.egg/twisted/scripts/twistd.py", line 23, in runApp
_SomeApplicationRunner(config).run()
File "/media/Extra_Space/Sage_installs/sage-5.2.rc0/local/lib/python2.7/site-packages/Twisted-12.1.0-py2.7-linux-i686.egg/twisted/application/app.py", line 386, in run
self.application = self.createOrGetApplication()
File "/media/Extra_Space/Sage_installs/sage-5.2.rc0/local/lib/python2.7/site-packages/Twisted-12.1.0-py2.7-linux-i686.egg/twisted/application/app.py", line 451, in createOrGetApplication
application = getApplication(self.config, passphrase)
--- <exception caught here> ---
File "/media/Extra_Space/Sage_installs/sage-5.2.rc0/local/lib/python2.7/site-packages/Twisted-12.1.0-py2.7-linux-i686.egg/twisted/application/app.py", line 462, in getApplication
File "/media/Extra_Space/Sage_installs/sage-5.2.rc0/local/lib/python2.7/site-packages/Twisted-12.1.0-py2.7-linux-i686.egg/twisted/application/service.py", line 405, in loadApplication
File "/media/Extra_Space/Sage_installs/sage-5.2.rc0/local/lib/python2.7/site-packages/Twisted-12.1.0-py2.7-linux-i686.egg/twisted/persisted/sob.py", line 210, in loadValueFromFile
exec fileObj in d, d
File "sage_notebook.sagenb/twistedconf.tac", line 83, in <module>
s = strports.service('ssl:8080:interface=localhost:privateKey=/home/jonathan/.sage/notebook/private.pem:certKey=/home/jonathan/.sage/notebook/public.pem', site)
File "/media/Extra_Space/Sage_installs/sage-5.2.rc0/local/lib/python2.7/site-packages/Twisted-12.1.0-py2.7-linux-i686.egg/twisted/application/strports.py", line 77, in service
endpoints._serverFromStringLegacy(reactor, description, default),
File "/media/Extra_Space/Sage_installs/sage-5.2.rc0/local/lib/python2.7/site-packages/Twisted-12.1.0-py2.7-linux-i686.egg/twisted/internet/endpoints.py", line 873, in _serverFromStringLegacy
nameOrPlugin, args, kw = _parseServer(description, None, default)
File "/media/Extra_Space/Sage_installs/sage-5.2.rc0/local/lib/python2.7/site-packages/Twisted-12.1.0-py2.7-linux-i686.egg/twisted/internet/endpoints.py", line 864, in _parseServer
return (endpointType.upper(),) + parser(factory, *args[1:], **kw)
File "/media/Extra_Space/Sage_installs/sage-5.2.rc0/local/lib/python2.7/site-packages/Twisted-12.1.0-py2.7-linux-i686.egg/twisted/internet/endpoints.py", line 671, in _parseSSL
from twisted.internet import ssl
File "/media/Extra_Space/Sage_installs/sage-5.2.rc0/local/lib/python2.7/site-packages/Twisted-12.1.0-py2.7-linux-i686.egg/twisted/internet/ssl.py", line 23, in <module>
from OpenSSL import SSL
File "/media/Extra_Space/Sage_installs/sage-5.2.rc0/local/lib/python2.7/site-packages/pyOpenSSL-0.12-py2.7-linux-i686.egg/OpenSSL/__init__.py", line 36, in <module>
from OpenSSL import crypto
exceptions.ImportError: cannot import name crypto

Failed to load application: cannot import name crypto

---------------------------------------------------------------------------
error                                     Traceback (most recent call last)

/media/Extra_Space/Sage_installs/sage-5.2.rc0/<ipython console> in <module>()

/media/Extra_Space/Sage_installs/sage-5.2.rc0/devel/sagenb/sagenb/notebook/notebook_object.pyc in __call__(self, *args, **kwds)
217     """
218     def __call__(self, *args, **kwds):
--> 219         return self.notebook(*args, **kwds)
220
221     notebook = run_notebook.notebook_run

/media/Extra_Space/Sage_installs/sage-5.2.rc0/devel/sagenb/sagenb/notebook/run_notebook.pyc in notebook_run(self, directory, port, interface, port_tries, secure, reset, accounts, openid, server_pool, ulimit, timeout, upload, automatic_login, start_path, fork, quiet, server, profile, subnets, require_login, open_viewer, address)
614     os.chdir(cwd)
615     if e == 256:
--> 616         raise socket.error
617

error:
sage:


### comment:69 Changed 9 years ago by kini

OK, just checking, but did you install from source? If so, can you upload the output of easy_install pyopenssl inside a sage -sh session?

### comment:70 Changed 9 years ago by vbraun

I tried this updated to see if it fixes #13318. I see that require_login is now renamed to automatic_login, but I still end up with the login screen.

### comment:71 Changed 9 years ago by daniels

I have the same issue with PyOpenSSL on Debian/sid, already with sagenb 0.9.1:

daniel-mobile% sage -python
Python 2.7.3 (default, Aug  1 2012, 19:24:06)
[GCC 4.7.1] on linux2
>>> import OpenSSL
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/sage-5.2.rc0/local/lib/python2.7/site-packages/pyOpenSSL-0.12-py2.7-linux-x86_64.egg/OpenSSL/__init__.py", line 41, in <module>
from OpenSSL import rand, SSL
ImportError: /opt/sage-5.2.rc0/local/lib/python2.7/site-packages/pyOpenSSL-0.12-py2.7-linux-x86_64.egg/OpenSSL/SSL.so: undefined symbol: SSLv2_method


libssl1.0.0 is built with no-ssl2 on Debian/Ubuntu? for security reasons. Therefore the symbol SSLv2_method is not defined. Older PyOpenSSL versions don't check for this, but >=0.13 should be fine.

(Aside: This also seems to cause funky segmentation faults on Debian/Ubuntu?: https://groups.google.com/d/msg/sage-release/5TCe2JnV56o/QME2igZSF8YJ )

### comment:72 Changed 9 years ago by kini

IIRC we made a conscious decision to use PyOpenSSL <0.13 because 0.13 breaks with libssl <1.0.0 ... (?) Obviously "<"-based version dependencies are not viable in the long run though so I'd like to use new PyOpenSSL in the notebook if possible, certainly.

### comment:74 Changed 9 years ago by daniels

Thanks, I did not realize that. It seems unfortunate that our PyOpenSSL is broken out of the box on recent Ubuntu / Debian installs (and messes with sys.setdlopenflags in the notebook process). I don't really know how this should be fixed. Is it reasonable, for example, to include 0.13 and not install it on systems with old OpenSSL (<0.9.8f) or use a patched version of either 0.12 or 0.13?

### comment:75 Changed 9 years ago by kini

The best thing would be if pyOpenSSL 0.14 could be released to handle old OpenSSL versions...

Jeroen, can you try directly building pyOpenSSL on the OS X 10.4 PPC machine which caused the failure I linked a couple comments ago? That way we could submit a proper bug report to the pyOpenSSL devs.

### comment:76 Changed 9 years ago by daniels

Not much progress though.

### comment:77 Changed 9 years ago by kini

Thanks for finding that.

### comment:78 Changed 9 years ago by kini

I tested the patch there on a CentOS 5 machine I have access to. Hopefully the developer will consider that good enough to release the fix :)

### comment:80 Changed 9 years ago by kini

• Dependencies changed from #11080, #9774, #11913, #12299 to #11080, #9774, #11913, #12299, #13384

I've made a pyOpenSSL SPKG so that we can use pyOpenSSL 0.13 + the patch daniels found. See #13384.

### comment:81 Changed 9 years ago by kini

• Work issues changed from new worksheets should be empty,evaluate button focus,openssl to new worksheets should be empty, wrap openssl imports

Actually, we might not even need OpenSSL at all. See #13385.

### comment:82 Changed 9 years ago by kini

• Description modified (diff)

### comment:83 follow-up: ↓ 84 Changed 9 years ago by kini

Jonathan, do you see the non-empty new worksheets problem in the version of the Flask notebook in 5.2?

Other than that, the only remaining work issue before this can be set to needs_review again is for someone to review this pull request which makes pyOpenSSL an optional dependency of sagenb. Once that's in, I'll release 0.10.2 and make a new SPKG, since I have no idea where this non-empty new worksheets problem is coming from, nor can I seem to replicate it...

### comment:84 in reply to: ↑ 83 Changed 9 years ago by gutow

• Work issues changed from new worksheets should be empty, wrap openssl imports to wrap openssl imports

Jonathan, do you see the non-empty new worksheets problem in the version of the Flask notebook in 5.2?

I haven't seen the problem again since I manually reset my last worksheet id. However, unless someone added code to check the worksheet directory on launch, I don't think it is fixed. I only think the problem occurs if the server shuts down inappropriately; thus this should not be a blocker. I vote for removing it as an issue from this ticket and setting up a separate ticket on the sagenb site.

I like the make pyOpenSSL optional solution.

I'm in my last week of ramp-up to classes, but I will see if I can do a test of things ASAP.

### comment:85 Changed 9 years ago by kini

Just to clarify, the version of the Flask notebook that came out in 5.2 is older than the one you tested on this ticket (which makes sense since this ticket is still open! :) ). I was asking because if you didn't see that behavior on 5.2, it would mean this problem is a recent regression, not an age-old bug.

But if you can't even replicate it on this ticket again, or say that it's a corner case, then I agree that it shouldn't be a blocker for this ticket (thankfully, since I have no idea what causes it).

### comment:86 Changed 9 years ago by jason

I'm pretty sure it's an age-old bug. I've been seeing it for at least 6-9 months on sagenb.org, on occasion.

### comment:87 Changed 9 years ago by jhpalmieri

So except for updating README.rst at the pull request, is this done?

### comment:89 Changed 9 years ago by kini

• Description modified (diff)
• Status changed from needs_work to needs_review

OK, new SPKG is finally complete (I hope). Please test!

### comment:90 Changed 9 years ago by kini

• Description modified (diff)

oops

### comment:91 Changed 9 years ago by kini

• Milestone changed from sage-pending to sage-5.3

Setting milestone away from sage-pending again since I don't think anything is pending anymore (is it?).

### comment:92 follow-up: ↓ 99 Changed 9 years ago by gutow

• Reviewers changed from John Palmieri to John Palmieri, Jonathan Gutow
• Status changed from needs_review to positive_review

I'm giving this a positive review. Tested on 5.2. The evaluate button now works, and openssl is not necessary. Warning message when you try to launch a secure notebook without installing pyOpenSSL is reasonable. If I missed checking something, please set this back to needs_review and remind us what else needs to be checked.

### comment:93 Changed 9 years ago by kini

Thanks, Jonathan!

### comment:94 Changed 9 years ago by jdemeyer

• Milestone changed from sage-5.3 to sage-5.4

### comment:95 Changed 9 years ago by jdemeyer

• Description modified (diff)

### comment:96 Changed 9 years ago by jdemeyer

• Status changed from positive_review to needs_work
• Work issues changed from wrap openssl imports to wrap openssl imports, SPKG.txt

One detail for SPKG.txt: you should replace

=== 0.10.2 (Keshav Kini, 2012-09-01) ===


by

=== sagenb-0.10.2 (Keshav Kini, 2012-09-01) ===


or even

=== sagenb-0.10.2 (Keshav Kini, 1 September 2012) ===


(and analogous on the other lines)

### comment:97 Changed 9 years ago by gutow

Tests OK on 5.3.rc0 as well.

### comment:98 Changed 9 years ago by kini

• Status changed from needs_work to needs_review
• Work issues wrap openssl imports, SPKG.txt deleted

Jeroen: I changed the version strings to include "sagenb", but I'm keeping the dates as they are. See http://en.wikipedia.org/wiki/ISO_8601 .

### comment:99 in reply to: ↑ 92 ; follow-up: ↓ 105 Changed 9 years ago by kini

Warning message when you try to launch a secure notebook without installing pyOpenSSL is reasonable.

Just checking, but this should cause an error, not a warning. If you don't have pyOpenSSL installed and sagenb is unable to host the server over HTTPS, it should fail to start at all - see this commit. Is that indeed what happens? If not, something's wrong.

### comment:100 follow-up: ↓ 101 Changed 9 years ago by jdemeyer

• Status changed from needs_review to positive_review

Packaging of the spkg is fine now.

PS: when are we going to have sagenb-1.0?

### comment:101 in reply to: ↑ 100 Changed 9 years ago by ppurka

Packaging of the spkg is fine now.

great. Finally, this can go in! :)

PS: when are we going to have sagenb-1.0?

after samuela's rewrite? That would be a good time to launch 1.0, I suppose. :)

### comment:102 Changed 9 years ago by jason

Good idea. 0.9 was about rewriting the backend, 1.0 is about rewriting the frontend?

### comment:103 Changed 9 years ago by kini

Going by semver.org, I guess there should have been a 1.0 ages ago, the flask notebook should have been 2.0, and Samuel's UI rewrite should either have been 2.1 or 3.0, depending on what you consider a "public API". IMO we can just make Samuel's UI rewrite 0.11.

### comment:104 Changed 9 years ago by jason

You sound like Linus!

### comment:105 in reply to: ↑ 99 Changed 9 years ago by gutow

Warning message when you try to launch a secure notebook without installing pyOpenSSL is reasonable.

Just checking, but this should cause an error, not a warning. If you don't have pyOpenSSL installed and sagenb is unable to host the server over HTTPS, it should fail to start at all - see this commit. Is that indeed what happens? If not, something's wrong.

I wasn't specific enough. It does, indeed, throw and error with a good explanation. I concur with the positive review.

### comment:106 Changed 9 years ago by jdemeyer

• Status changed from positive_review to needs_work

There is a easy to fix bug in spkg-install: the command

tar xf "$CUR/src/$BASENAME.tar.gz"


should be

tar xzf "$CUR/src/$BASENAME.tar.gz" || die "Error: cannot extract \$BASENAME.tar.gz"


or similar.

### comment:107 Changed 9 years ago by jdemeyer

• Status changed from needs_work to positive_review

### comment:108 Changed 9 years ago by novoselt

I have just noticed that I cannot stay connected to two notebook servers: both are on the same machine on different ports (usernames and passwords are also different). One is 5.2, another 5.3.rc1+this notebook. When I login to one, another one logs out. This is a bit annoying and such interaction between servers may be dangerous, i.e. now they drop connection because of the other - can there be a situation when logging into one server will trigger logging into another?

### comment:109 Changed 9 years ago by jason

Huh. This sounds like a cookie problem. The cookie should be tied to the port.

### comment:110 Changed 9 years ago by novoselt

Cookie names seem to include the port number.

Also, while it is a bug, I don't think that it should delay this and related tickets: switch to MathJax is awesome for using classroom podium computers, as I have just confirmed ;-)

### comment:111 follow-up: ↓ 127 Changed 9 years ago by jason

I should mention on this ticket, as I mentioned on the mailing list (https://groups.google.com/forum/?fromgroups=#!topic/sage-notebook/sQBBiHygG9E) that a design issue in this version of the notebook really bothers me (and others that have emailed me). Basically, 3d graphs do not show up as jmol instances initially, but always require an extra click to "activate". See the mailing list thread above for discussion, but the end result is that I apply these patches: https://gist.github.com/3527075 to *.sagenb.org and my personal Sage server.

### comment:112 Changed 9 years ago by kini

I fixed the issue Jeroen pointed out, but forgot to mention I had uploaded a new SPKG. Looks like Jeroen noticed it anyway, though :)

### comment:113 Changed 9 years ago by jdemeyer

The current plan is to merge this in sage-5.4.beta0, so it's better not to do any changes anymore (unless serious problems are discovered of course).

### comment:114 follow-up: ↓ 115 Changed 9 years ago by jason

I'm not sure if jdemeyer's comment was answering mine or kini's. As for my comment, I'm happy to just apply the patches I mentioned to *.sagenb.org and my personal servers; I personally don't need them to be in the official notebook. However, I think it's a big regression for those that don't have the patches applied.

### comment:115 in reply to: ↑ 114 ; follow-up: ↓ 116 Changed 9 years ago by jdemeyer

I'm not sure if jdemeyer's comment was answering mine or kini's.

None, I was just making a statement.

However, I think it's a big regression for those that don't have the patches applied.

Then I guess we need to decide whether that issue is serious enough not to merge this and related tickets (my gut feeling is: no, it's not that serious).

### comment:116 in reply to: ↑ 115 Changed 9 years ago by jason

However, I think it's a big regression for those that don't have the patches applied.

Then I guess we need to decide whether that issue is serious enough not to merge this and related tickets (my gut feeling is: no, it's not that serious).

I'll leave that vote to others, as I'm happy to apply the patches reverting the behavior to installs that I administer. I'd suggest, for example, that kini and others try doing an interact that has a slider and a 3d graph and see how smooth of an experience it is. Imagine trying to present something to a class, or trying to change the slider to investigate a property. I think the regression basically makes this use-case unusable.

### comment:117 Changed 9 years ago by kcrisman

I would love the previous behavior as well, but I just want to make sure that Jonathan's concerns are at least partially addressed. Maybe we could raise some kind of warning?

I wish some other people would have commented at the time; we raised this on sage-devel as well as on the relevant ticket, if I recall correctly, and there was little response. I felt like my voice was the only one for the original behavior, and I was very hesitant to override a change due to a technical failing on the vote of one person.

### comment:118 Changed 9 years ago by jason

kcrisman: I realize that I didn't have time to participate as much as I would have liked in the original discussion.

As for a warning---we're just reverting back to original behavior, so I don't think it's any worse than it was before. I agree something needs to be done moving forward, but I don't think we need to raise a warning for continuing the same behavior as before.

### comment:119 Changed 9 years ago by kcrisman

Sorry, by "warning" I meant a warning that using an interact with Jmol over the web (not on a local computer) might lead to horrible massive desynchronization (if I understand the issue correctly).

On the other hand, I haven't used that in production environments yet; maybe it turns out not to be a problem using e.g. *.sagenb.org or your Drake server etc.

### comment:120 Changed 9 years ago by ppurka

The problem with having jmol come up by default was faced by us when we had many jmols in a single page. I had to grade some assignments last year, and it wasn't a pleasant experience. You could be sitting there twiddling your fingers all the while the browser seems completely frozen.

### comment:121 Changed 9 years ago by jason

I have not noticed any problems.

### comment:122 Changed 9 years ago by jason

But the new jmol implementation *will* automatically sleep instances when you reach a certain number, right? So "many jmols on a page" should be fixed, right?

Can you try it on sagenb.org (which has my patch) and see if it is a problem still?

### comment:123 Changed 9 years ago by ppurka

Hmm.. it froze firefox once, but it is otherwise quite smooth now. I forgot about the fact that not all jmols are loaded now. This definitely helps.

### comment:124 Changed 9 years ago by gutow

The old behavior hangs most of the time in interacts for me using anything but a local instance. Thus I still consider the old behavior a bug. For me interacts work faster and more smoothly using the on server image generation. Only if I want to rotate the image do I need to load the applet. The problem seems to be browser/OS/JVM independent so I think it is a network timing issue.

That said, I certainly think it is reasonable for people to want access to the old behavior. I suggest we start a new ticket and develop a good patch that provides a per-worksheet switch that allows one to turn on live Jmol loading in a worksheet. However, I strongly recommend against defaulting to the old behavior, which is broken except under special circumstances on devices that have a reasonably up-to-date JVM installed.

In the meantime, I suggest people who want the old behavior use Jason's patch.

### comment:125 follow-up: ↓ 128 Changed 9 years ago by jhpalmieri

Yes, definitely open up a new ticket. Once this is merged, we will get lots of questions about how to get the old behavior back, and so implementing a solution should have a very high priority.

### comment:126 Changed 9 years ago by jdemeyer

• Merged in set to sage-5.4.beta0
• Resolution set to fixed
• Status changed from positive_review to closed

OK, so this is ready to go?

### comment:127 in reply to: ↑ 111 Changed 9 years ago by kcrisman

OK, so this is ready to go?

As long as Jason and Jonathan can commit to figuring out what to do about this issue, then yes. One would hope that the 5.4 beta sequence would be long enough to accommodate this. The Jmol upgrade is otherwise really worth the effort.

I should mention on this ticket, as I mentioned on the mailing list (https://groups.google.com/forum/?fromgroups=#!topic/sage-notebook/sQBBiHygG9E) that a design issue in this version of the notebook really bothers me (and others that have emailed me). Basically, 3d graphs do not show up as jmol instances initially, but always require an extra click to "activate". See the mailing list thread above for discussion, but the end result is that I apply these patches: https://gist.github.com/3527075 to *.sagenb.org and my personal Sage server.

So *.sagenb.org is running Sage 5.2, but with sagenb 0.10.x? See also 192:ticket:12299 where I predicted this discussion :) On the plus side, this hasn't been merged into a released Sage yet, right?

### comment:128 in reply to: ↑ 125 ; follow-up: ↓ 134 Changed 9 years ago by gutow

Yes, definitely open up a new ticket. Once this is merged, we will get lots of questions about how to get the old behavior back, and so implementing a solution should have a very high priority.

OK, Jason please open a ticket in trac (I'm still getting my feet wet with GIT). Please describe the options you think would be useful to you, remembering that I have not found a way to make live Jmols in interacts reliable over the network without a major rewrite of how interacts work (something I'm not in a position to commit to doing, but think should be done). Personally, I vote for some kind of a switch. Please make sure to cc me on the new ticket.

### comment:129 Changed 9 years ago by novoselt

What will be the behaviour of Sage Cell Server? Will interacts also require clicking a button for interaction every time? Or is it possible perhaps to have a switch between 2D/3D modes which persists over changes of other parameters?

### comment:130 Changed 9 years ago by jason

The cell server will default to live jmol (in fact, that's the only option right now). It doesn't include the advanced functionality Jonathan has put into sagenb.

Last edited 9 years ago by jason (previous) (diff)

### comment:131 follow-up: ↓ 132 Changed 9 years ago by mmarco

When you try to recover a forgotten password in the new notebook, you are asked for the username, and then get the response "invalid username" even if the username is valid, and the email has been confirmed.

I think that the problem is in the file authentication.py

The function forgot_pass() has the following lines:

def forgot_pass():
if not g.notebook.conf()['email']:
return current_app.message('The account recovery system is not active.')

return render_template(os.path.join('html', 'accounts', 'account_recovery.html'))

def error(msg):
return current_app.message(msg, url_for('forgot_pass'))

try:
except KeyError:

if not user.is_email_confirmed():
return error("The e-mail address hasn't been confirmed.")

#XXX: some of this could be factored out into a random passowrd
#function.  There are a few places in admin.py that also use it.
from random import choice
import string
chara = string.letters + string.digits
password = ''.join([choice(chara) for i in range(8)])

from sagenb.notebook.smtpsend import send_mail
# TODO: make this come from the server settings

port = g.notebook.port
try:
except ValueError:
# the email address is invalid



But

    try:


   username = request.values.get('username', '').strip()


### comment:132 in reply to: ↑ 131 Changed 9 years ago by kcrisman

When you try to recover a forgotten password in the new notebook, you are asked for the username, and then get the response "invalid username" even if the username is valid, and the email has been confirmed.

Just fyi to the authors, mmarco has also already opened #13536 to track this, but put this here for more visibility. See this pull request for a pull version of his patch. (@kini - ok, I do like being able to just edit on the fly! Not a convert to github yet, though :) )

### comment:133 Changed 9 years ago by kini

• Description modified (diff)