Opened 8 years ago

Closed 7 years ago

Last modified 3 years ago

#11080 closed enhancement (fixed)

move notebook to flask/wsgi-based notebook

Reported by: jason Owned by:
Priority: blocker Milestone: sage-5.2
Component: notebook Keywords: sd31 sd35.5
Cc: ddrake, jason, kcrisman, mhansen, rkirov, timdumol, iandrus, strogdon Merged in: sage-5.2.beta0
Authors: Mike Hansen, Radoslav Kirov, William Stein, Jason Grout, Jeroen Demeyer Reviewers: Radoslav Kirov, Dan Drake, Jason Grout, Simon King, Dmitrii Pasechnik, John Palmieri, Punarbasu Purkayastha
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: #13126, #11078, #11874, #12229, #11503, #12327, #12899, #13270 Stopgaps:

Description (last modified by jdemeyer)

This ticket tracks the progress to move the notebook to a flask/wsgi-based notebook.

Relevant code repository: https://github.com/sagemath/sagenb

Explicit instructions to apply relevant patches for this whole group of tickets, starting from a working installation of sage-5.0.beta13 or later (replace $SAGE_ROOT with the root sage directory):

cd $SAGE_ROOT
./sage -i openssl # optional if you have systemwide SSL dev headers installed
./sage -f http://boxen.math.washington.edu/home/keshav/files/sagenb-0.9.0.spkg
./sage -f http://www.uwosh.edu/faculty_staff/gutow/jmol-12.2.21.p0.spkg

./sage -hg qimport -P http://trac.sagemath.org/sage_trac/raw-attachment/ticket/11874/trac-11874-remove-twisted.patch
./sage -hg qimport -P http://trac.sagemath.org/sage_trac/raw-attachment/ticket/11503/trac-11503-jmol-spkg.2.patch

cd local/bin
../../sage -hg qimport -P http://trac.sagemath.org/sage_trac/raw-attachment/ticket/11080/trac_11080-scripts-hgignore-dulwich.patch

cd ../../devel/sage
../../sage -hg qimport -P http://trac.sagemath.org/sage_trac/raw-attachment/ticket/11874/sage-spkg-11874.patch
../../sage -hg qimport -P http://trac.sagemath.org/sage_trac/raw-attachment/ticket/11078/trac_11078.patch
../../sage -hg qimport -P http://trac.sagemath.org/sage_trac/raw-attachment/ticket/11503/trac-11503-jmol-commandline.patch
../../sage -hg qimport -P http://trac.sagemath.org/sage_trac/raw-attachment/ticket/12229/trac_12229-sagenb-developer-doc.3.patch
../../sage -hg qimport -P http://trac.sagemath.org/sage_trac/raw-attachment/ticket/12229/trac-12229-manifest.patch
../../sage -hg qimport -P http://trac.sagemath.org/sage_trac/raw-attachment/ticket/11080/trac-11080-notebook-docs.patch

### NOTE: Skip the following one line if you are using Sage 5.0.rc1 or later
../../sage -hg qimport -P http://trac.sagemath.org/sage_trac/raw-attachment/ticket/12899/12899_install_guide.patch

../../sage -hg qimport -P http://trac.sagemath.org/sage_trac/raw-attachment/ticket/11080/trac_11080-openssl-in-installation-docs.patch

cd ../../
./sage -b

Alternatively, use the testing release http://boxen.math.washington.edu/home/release/sage-5.2.alpha0/sage-5.2.alpha0.tar (which includes this tickets and its dependencies but not #12229).

spkg: http://boxen.math.washington.edu/home/keshav/files/sagenb-0.9.0.spkg includes all dependencies not already in Sage. Please install it as part of the process described above, as it depends on a bunch of patches.

apply:

  1. trac-11080-notebook-docs.patch to the Sage library
  2. trac_11080-openssl-in-installation-docs.patch to the Sage library
  3. trac_11080-scripts-hgignore-dulwich.patch to sage_scripts.

This is a review version of the spkg. When we have finalized things, we'll make a finished spkg that has the appropriate tags in the revision history, no unnecessary revision branches, etc.

Attachments (5)

trac-11080-notebook-docs.patch (785 bytes) - added by jason 8 years ago.
trac_11080-openssl-in-installation-docs.patch (1.4 KB) - added by kini 7 years ago.
apply to $SAGE_ROOT/devel/sage (on top of #12899)
top-original-bottom-with11080.png (110.6 KB) - added by ppurka 7 years ago.
Original on top, with this ticket on bottom
trac_11080-scripts-hgignore-dulwich.patch (412 bytes) - added by jdemeyer 7 years ago.
apply to $SAGE_LOCAL/bin
trac_11080-user_registration.patch (1.2 KB) - added by ppurka 7 years ago.
(temporary fix) apply to devel/sagenb

Download all attachments as: .zip

Change History (353)

comment:1 Changed 8 years ago by jason

  • Cc ddrake added

comment:2 Changed 8 years ago by gutow

  • Description modified (diff)

comment:3 Changed 8 years ago by gutow

  • Description modified (diff)
  • Keywords sd31 added

comment:4 Changed 8 years ago by gutow

  • Description modified (diff)

comment:5 Changed 8 years ago by gutow

  • Description modified (diff)

comment:6 Changed 8 years ago by gutow

  • Description modified (diff)

removed #11078 from list as the fix is included in the 11503 command line patch.

comment:7 Changed 8 years ago by jason

  • Authors changed from Mike Hansen, Rado Kirov to Mike Hansen, Rado Kirov, William Stein, Jason Grout
  • Dependencies set to #11078
  • Description modified (diff)

comment:8 Changed 8 years ago by jason

  • Description modified (diff)

comment:9 Changed 8 years ago by jason

  • Description modified (diff)

comment:10 Changed 8 years ago by jason

  • Dependencies changed from #11078 to #11078, #11874

comment:11 Changed 8 years ago by jason

  • Status changed from new to needs_review

comment:12 Changed 8 years ago by jason

(for future reference, after this is merged, jmol is a high priority. These two tickets deal with these changes:

  • #11496 (update jmol script in local/bin to reflect move of jmol directory to local)
  • #11503 (updated spkg and paths for move of jmol directory)
  • #9238 (for reference -original Jmol enhancement ticket but heavily laden with pre-flask notebook stuff)

Again, this is only FYI and for further reference.

comment:13 Changed 8 years ago by jason

  • Description modified (diff)

comment:14 Changed 8 years ago by gutow

I get the following error when I try to apply the spkg dependencies patch. Everything else seems to work. I'm still checking functionality...

applying /home/jonathan/.sage/temp/VB_Ubuntu_11/26614/tmp_0.patch
unable to find 'spkg/install' for patching
1 out of 1 hunks FAILED -- saving rejects to file spkg/install.rej
unable to find 'spkg/standard/deps' for patching
4 out of 4 hunks FAILED -- saving rejects to file spkg/standard/deps.rej
spkg/install: No such file or directory
spkg/standard/deps: No such file or directory
abort: patch failed to apply

comment:15 Changed 8 years ago by gutow

With 4.8.alpha4, what is described here does not work. I redid everything. The .spkgs seem to install without error. The patch for this ticket does not work and the notebook fails to launch with the error:

/home/jonathan/Documents/sage-4.8.alpha4/<ipython console> in <module>()

/home/jonathan/Documents/sage-4.8.alpha4/devel/sagenb/sagenb/notebook/notebook_object.py in __call__(self, *args, **kwds)
    202     """
    203     def __call__(self, *args, **kwds):
--> 204         return self.notebook(*args, **kwds)
    205 
    206     notebook = run_notebook.notebook_twisted

/home/jonathan/Documents/sage-4.8.alpha4/devel/sagenb/sagenb/notebook/run_notebook.py in notebook_twisted(self, directory, port, interface, address, port_tries, secure, reset, require_login, accounts, openid, server_pool, ulimit, timeout, open_viewer, sagetex_path, start_path, fork, quiet, subnets)
    442     if open_viewer:
    443         "Open viewer automatically isn't fully implemented.  You have to manually open your web browser to the above URL."
--> 444     return run(port)
    445 
    446 def get_admin_passwd():

/home/jonathan/Documents/sage-4.8.alpha4/devel/sagenb/sagenb/notebook/run_notebook.py in run(port)
    427         os.chdir(cwd)
    428         if e == 256:
--> 429             raise socket.error
    430 
    431         return True

comment:16 Changed 8 years ago by jason

It's possible that the SAGE_ROOT patch needs to be rebased to 4.8.alpha4. Let me check. Thanks for the test

comment:17 Changed 8 years ago by jason

  • Description modified (diff)

Jonathan,

I think the problem is that you didn't apply the patch here to the SAGE_ROOT repository. I've updated the instructions above with explicit commands.

comment:18 Changed 8 years ago by jason

I'm checking now into the error you posted in your second. It appears that there is indeed a problem with the sagenb spkg not overwriting the old notebook.

comment:19 follow-up: Changed 8 years ago by jason

  • Description modified (diff)

For future reference, here are the instructions if we had a sagenb_dependencies spkg. DON'T USE THESE--use the instructions in the description of the ticket.

cd $SAGE_ROOT
./sage -hg qimport http://trac.sagemath.org/sage_trac/raw-attachment/ticket/11080/trac-11080-sagenb_dependencies-spkg.patch
./sage -hg qpush
./sage -f http://sage.math.washington.edu/home/jason/sagenb_dependencies-20111216.spkg
./sage -f http://sage.math.washington.edu/home/jason/twisted-11.0.0.spkg
./sage -f http://sage.math.washington.edu/home/jason/sagenb-0.9.0.spkg
cd devel/sage
../../sage -hg qimport http://trac.sagemath.org/sage_trac/raw-attachment/ticket/11078/trac_11078.patch
../../sage -hg qpush
cd ../../
./sage -br

comment:20 Changed 8 years ago by jason

  • Description modified (diff)

comment:21 in reply to: ↑ 19 Changed 8 years ago by gutow

OK, with the changed instructions and this explicit instruction set, I get a running sage-4.8.alpha4. The notebook launches and everything I tried other than 3-D plotting worked in the notebook. 3-D plotting does not work because Jmol seems to be gone. Is #11503 supposed to be included as well? That would install the new Jmol in the $SAGE_ROOT/local/shared directory rather than in the sagenb/data directory. I note that there is no sagenb/data/jmol directory in the notebook created by this ticket, while there is in the old 4.8.alpha4 notebook.

I think it's almost there. Jonathan Replying to jason:

For future reference, here are the instructions if we had a sagenb_dependencies spkg. DON'T USE THESE--use the instructions in the description of the ticket.

cd $SAGE_ROOT
./sage -hg qimport http://trac.sagemath.org/sage_trac/raw-attachment/ticket/11080/trac-11080-sagenb_dependencies-spkg.patch
./sage -hg qpush
./sage -f http://sage.math.washington.edu/home/jason/sagenb_dependencies-20111216.spkg
./sage -f http://sage.math.washington.edu/home/jason/twisted-11.0.0.spkg
./sage -f http://sage.math.washington.edu/home/jason/sagenb-0.9.0.spkg
cd devel/sage
../../sage -hg qimport http://trac.sagemath.org/sage_trac/raw-attachment/ticket/11078/trac_11078.patch
../../sage -hg qpush
cd ../../
./sage -br

comment:22 Changed 8 years ago by jason

Jonathan: the problem was that my spkg-making sagenb install had a symbolic link to jmol, rather than the actual directory, and the symbolic link was not followed when making the spkg.

I've updated the 0.9.0 spkg (tagged as 0.9.0.rc2 in the repository google code). Can you try again? I just tried installing in a clean 4.8.alpha4, and 3d plots worked fine.

Thanks, Jason

comment:23 Changed 8 years ago by gutow

Jason:  I think everything is OK.  It all seemed to work on my Ubuntu Linux distro.  I would vote for a positive review, if we can get one other person to verify.  Comment...anybody using Chrome on Linux, OpenJDK/IceTea don't play very well with Chrome and this old version of Jmol.

comment:24 Changed 8 years ago by jason

Thanks! Jmol is certainly a top priority, along with mathjax, after we get this merged into sage. Lucky for us, most of the work has already been done, so hopefully it's just rebasing the patches that are already there.

comment:25 Changed 8 years ago by jason

  • Description modified (diff)

Deleting the dependency spkg instructions, since we aren't going that route.

comment:26 Changed 8 years ago by dimpase

  • Status changed from needs_review to needs_info

I checked that all this appears to work on MacOSX 10.6 just fine, including Jmol and text fields with LaTeX.

I do not like that this build does not incorporate the sass update by Tim Dumol in Sept. 2011 Without it, hacking the CSS stuff won't be possible on some platforms, at least not on MacOSX 10.6.

Is it possible to merge this, and the related readme update, into the sagenb spkg?

comment:27 Changed 8 years ago by jason

It should be easy to incorporate that. Let me try. I don't think I edited anything in those directories.

comment:28 follow-up: Changed 8 years ago by jason

I've incorporated the sass patches, but there are some very small changes in the css, which worries me. How much have these patches been tested?

comment:29 in reply to: ↑ 28 Changed 8 years ago by dimpase

  • Cc timdumol added

Replying to jason:

I've incorporated the sass patches, but there are some very small changes in the css, which worries me. How much have these patches been tested?

These are changes in the generated CSS files. The only changes in main.css are in the comments; these certainly can be ignored.

There are few other changes in the other css, called test_report.css. (I don't even know if this file is used at all, is it?) But these changes are indeed very, very minor, like a different way to encode the colour, or a change of a constant from 2.0em to 2em. I worked a bit with these modified CSS files, no issues noted. IMHO it is totally safe to update.

I cc this to Tim Dumol, just in case.

comment:30 Changed 8 years ago by timdumol

The changes made were all automated, via the sass-convert tool made by the developers of SASS/SCSS themselves, with the exception of the moving of the Accounts page scss to a separate partial, which was a cut-paste into a new file; thus, there shouldn't be any regressions.

comment:31 follow-ups: Changed 8 years ago by jason

For example, line 614 of ../../../../sass/src/jquery-plugins/_ui.achtung.scss now generates this line in main.css:

.achtung .ui-icon.achtung-close-button { overflow: hidden; float: right; position: relative; top: -8px; right: -8px; cursor: pointer; background-image: url(/javascript/jqueryui/css/sage/images/ui-icons_cccccc_256x240.png); }

However, that background image URL doesn't exist (but the old background image URL does exist).

comment:32 in reply to: ↑ 31 Changed 8 years ago by timdumol

Replying to jason:

For example, line 614 of ../../../../sass/src/jquery-plugins/_ui.achtung.scss now generates this line in main.css:

.achtung .ui-icon.achtung-close-button { overflow: hidden; float: right; position: relative; top: -8px; right: -8px; cursor: pointer; background-image: url(/javascript/jqueryui/css/sage/images/ui-icons_cccccc_256x240.png); }

However, that background image URL doesn't exist (but the old background image URL does exist).

Sorry about that. From what I can tell, I must have used another copy of Achtung to update the Achtung CSS, which was configured to another jQuery UI theme. I'll push a fix in a minute.

comment:33 in reply to: ↑ 31 Changed 8 years ago by dimpase

Replying to jason:

For example, line 614 of ../../../../sass/src/jquery-plugins/_ui.achtung.scss now generates this line in main.css:

.achtung .ui-icon.achtung-close-button { overflow: hidden; float: right; position: relative; top: -8px; right: -8px; cursor: pointer; background-image: url(/javascript/jqueryui/css/sage/images/ui-icons_cccccc_256x240.png); }

However, that background image URL doesn't exist (but the old background image URL does exist).

my fault that I didn't notice these on the diff. I just looked again, and this, and one more, images at the very bottom had changed filenames. The rest are completely innocent changes (comments, #d vs #D, "..." vs '...').

comment:35 in reply to: ↑ 34 Changed 8 years ago by timdumol

Replying to jason:

I think I just fixed it: https://github.com/sagemath/sagenb/commit/624dddb228bf3ebfdc02e161d2a5646b8c32a31f

Yep, that should do the trick. Once again, sorry.

comment:36 Changed 8 years ago by jason

No problem. Things have certainly slipped by me often enough. Okay, positive review to these patches from me too.

comment:37 follow-up: Changed 8 years ago by jason

  • Status changed from needs_info to needs_review

comment:38 in reply to: ↑ 37 Changed 8 years ago by dimpase

  • Status changed from needs_review to positive_review

Replying to jason:

OK, with these changes in the spkg, all good to go!

comment:39 Changed 8 years ago by jason

  • Status changed from positive_review to needs_work

I've updated the spkg. Can someone confirm that it still installs and has these last changes? I also added a few changes to deal with using hg-git to mirror a git repository.

I'm setting this back to needs-review so someone can give it one final +1, after all has been said and done.

comment:40 follow-up: Changed 8 years ago by jason

  • Status changed from needs_work to needs_review

comment:41 in reply to: ↑ 40 Changed 8 years ago by dimpase

  • Status changed from needs_review to positive_review

Replying to jason:

the updated sagenb-0.9.0 installs and works on MacOSX 10.6. And the sass changes are there. I do not know how to test the CSS properly, though, i.e. when these Achtung images come into play, but perhaps it's a good day and I should give it a positive review, still.

comment:42 Changed 8 years ago by jdemeyer

  • Milestone changed from sage-4.8 to sage-5.0

comment:43 Changed 8 years ago by jdemeyer

  • Description modified (diff)

comment:44 follow-up: Changed 8 years ago by jason

I'm curious: what is the plan for 5.0? Can I guess: finish 4.8, then merge this and the python update in 5.0alpha0?

comment:45 in reply to: ↑ 44 Changed 8 years ago by jdemeyer

Replying to jason:

I'm curious: what is the plan for 5.0? Can I guess: finish 4.8, then merge this and the python update in 5.0alpha0?

Essentially yes. But the dependencies of this ticket and also Python (#9958) still need review.

comment:46 Changed 8 years ago by jdemeyer

  • Status changed from positive_review to needs_work
  • Work issues set to Mercurial

SPKG.txt and spkg-install should be under Mercurial control.

comment:47 follow-up: Changed 8 years ago by jdemeyer

  • Work issues changed from Mercurial to Mercurial, rebase?

Please double-check that following tickets really have been merged in the new sagenb:

comment:48 follow-up: Changed 8 years ago by jason

SPKG.txt and spkg-install *are* under version control, in the original sagenb repository. The problem is that we then custom-make an spkg and copy those files into the spkg directory. In other words, they are under version control in the src/sagenb/ directory.

I'm not quite sure how to handle this case. Since the spkg is created fresh every time make spkg is called, a repository for the SPKG.txt and spkg-install files at the top level makes no sense---they would always be just the initial commits.

Ideas?

comment:49 Changed 8 years ago by jason

Thanks for the list of tickets to double-check.

comment:50 Changed 8 years ago by jdemeyer

  • Dependencies changed from #11078, #11874 to #11078, #11874, #12229

comment:51 in reply to: ↑ 48 Changed 8 years ago by jdemeyer

Replying to jason:

SPKG.txt and spkg-install *are* under version control, in the original sagenb repository. The problem is that we then custom-make an spkg and copy those files into the spkg directory. In other words, they are under version control in the src/sagenb/ directory.

I'm not quite sure how to handle this case. Since the spkg is created fresh every time make spkg is called, a repository for the SPKG.txt and spkg-install files at the top level makes no sense---they would always be just the initial commits.

I would say, never mind for now.

comment:52 in reply to: ↑ 47 ; follow-up: Changed 8 years ago by kini

Replying to jdemeyer:

Please double-check that following tickets really have been merged in the new sagenb:

#11732 needs rebasing. #10052 and #11106 are already merged. I'm not sure what do with #11343 given the existence of #11470. I've submitted pull requests on github to get the remaining four tickets into the flask notebook (awaiting review).

comment:53 Changed 8 years ago by jason

I've updated the spkg above to include commits up through https://github.com/sagemath/sagenb/commit/6b06baed92d1ab0657c3f2af2c153a5958eae8e4

comment:54 in reply to: ↑ 52 Changed 8 years ago by kini

Replying to kini:

#11732 needs rebasing. #10052 and #11106 are already merged. I'm not sure what do with #11343 given the existence of #11470. I've submitted pull requests on github to get the remaining four tickets into the flask notebook (awaiting review).

Update: Only #11732 remains, everything else is merged. Thanks again, Jeroen, for the list.

comment:55 Changed 8 years ago by jason

@kini: The tinymce pull request isn't merged yet (but is awaiting review! :) https://github.com/sagemath/sagenb/pull/11

comment:56 Changed 8 years ago by jason

Okay, the tinymce pull request is now merged.

comment:57 Changed 8 years ago by was

REFEREE REMARKS 1:

  • The spkg is not in the proper format. However, this is complicated to fix, and will be fixed in a *future* ticket: #12268
  • notebook(secure=True) is currently totally broken, due to the twisted spkg not supporting our tls extension, since I guess the patch didn't get applied. This needs to get fixed.
  • All doctests pass.

  • Rado has done some work on trying to get Selenium testing to work (the main issue just being installing everything on the same computer!). The Selenium tests need to get run (and any issues fixed) before we make the release.
  • Jason has simplified and streamlined the notebook developer instructions in response to discussions with Rado and I.

comment:58 Changed 8 years ago by was

(Also, let's not forget to fix up and review #11409, though a not dependency for this!.)

comment:59 follow-up: Changed 8 years ago by was

To summarize, what *needs* to be done to finish this ticket completely is:

  • tls (secure=True) support for Twisted

  • Selenium

  • Ensure that sagenb built fresh as *part of* a clean new Sage sdist works, and in particular that the jmol command line program works, e.g, sage: sphere() pops up jmol. We are worried that it won't work since we don't see how that script ever gets copied to local/bin/.

comment:60 in reply to: ↑ 59 ; follow-up: Changed 8 years ago by dimpase

Replying to was:

  • tls (secure=True) support for Twisted

you mention a patch that didn't get applied to twisted. Any more details on this patch?

Dima

comment:61 in reply to: ↑ 60 Changed 8 years ago by dimpase

Replying to dimpase:

Replying to was:

  • tls (secure=True) support for Twisted

you mention a patch that didn't get applied to twisted. Any more details on this patch?

upon reading the thread here started by jason-sage (we know this guy, don't we) http://twistedmatrix.com/pipermail/twisted-python/2010-October/022989.html they basically say that one needs to do

"ssl:443:sslmethod=TLSv1_METHOD"

somewhere in the configs.

Dima

comment:62 Changed 8 years ago by jason

I uploaded a new spkg with pyOpenSSL included, as well as a small fix for selenium tests from Rado.

comment:63 Changed 8 years ago by jdemeyer

  • Priority changed from major to blocker
  • Work issues Mercurial, rebase? deleted

comment:64 Changed 8 years ago by ijstokes

  • Keywords sd35.5 added

Pushed sagenb patch to github: https://github.com/sagemath/sagenb/commit/e930db178c468c488f86465cbc51c3ef4df83b70

This takes care of the blocker problem of not being able to use secure=True for HTTPS (SSL) Flask-based Sage Notebook server. It strips out all reference to GnuTLS and relies on PyOpenSSL.

For a few more details, GnuTLS support in Twisted was going to be pretty hard to get, and maintaining a sage-specific patch not a great proposition. William also suggested that GnuTLS support be removed.

comment:65 Changed 8 years ago by kini

  • Work issues set to rebase

(#11732 still needs rebasing)

comment:66 Changed 8 years ago by jason

  • Keywords changed from sd31, sd35.5 to sd31 sd35.5

comment:67 Changed 8 years ago by jason

  • Status changed from needs_work to needs_review

I updated the spkg to rc6, which is this commit: https://github.com/sagemath/sagenb/commit/84d442d5ac83f9484c12071e36dd98b3b896ea91

I think we've addressed William's points above:

  • SSL: We now don't need GnuTLS, but instead require that the user has openSSL installed to use secure=True as well as openID (I think this is adequately noted in the README.rst).
  • Selenium: we incorporated the patch from Rado which fixes some selenium tests. Rado: is there anything else we need to do on that?
  • jmol is now correctly copied over on a fresh sagenb install.

comment:68 Changed 8 years ago by jdemeyer

What's the status of #11732?

comment:70 Changed 8 years ago by jdemeyer

  • Work issues rebase deleted

Okay, great!

comment:71 Changed 8 years ago by kcrisman

I'm asking the dumb question. Suppose someone has an old notebook server or old worksheets on a local installation. This will behave correctly without any user intervention, correct? I'm pretty sure this is true, since sagenb.org was switched without much complaint along these lines, but just wanted to ask one last time :)

comment:72 Changed 8 years ago by jason

Yes, it should work without any user intervention. I will be testing this by upgrading the MAA server soon, when things settle down here.

comment:73 Changed 8 years ago by jason

  • Dependencies changed from #11078, #11874, #12229 to #11078, #11874, #12229, #11503

Added #11503 as a dependency, since it has positive review and the necessary patch has been merged into sagenb master.

comment:74 Changed 8 years ago by jason

#11503 was set back to "needs work", but as it is almost there, I'll still leave the change in sagenb master.

comment:75 Changed 8 years ago by jason

  • Description modified (diff)

I've updated the instructions and the sagenb spkg to include the most recent fixes and changes for the new jmol spkg and the twisted dependency. The sagenb spkg also includes the published jmol fix at https://github.com/sagemath/sagenb/pull/26

comment:76 Changed 8 years ago by jason

  • Description modified (diff)

comment:77 Changed 8 years ago by jason

  • Description modified (diff)

comment:78 Changed 8 years ago by jason

  • Description modified (diff)

The sagenb spkg should be installed before the jmol spkg.

comment:79 follow-ups: Changed 8 years ago by jdemeyer

The new flask-sagenb will likely require changes to the Mac App data/extcode/sage/ext/mac-app also. It seems to have port 8000 hardcoded in a few places.

comment:80 in reply to: ↑ 79 Changed 8 years ago by iandrus

Replying to jdemeyer:

The new flask-sagenb will likely require changes to the Mac App data/extcode/sage/ext/mac-app also. It seems to have port 8000 hardcoded in a few places.

I only know of one place, namely in loading-page.html (grep gives some false positives). I have been meaning to improve that page for a long time and just created #12327 to that end. Regardless, the link on that page is non-essential and so there is no reason IMHO to delay the review of this ticket.

comment:81 Changed 8 years ago by jdemeyer

  • Dependencies changed from #11078, #11874, #12229, #11503 to #11078, #11874, #12229, #11503, #12327

comment:82 Changed 8 years ago by jason

  • Description modified (diff)

Added the new #11503 command line patch.

comment:83 Changed 8 years ago by jdemeyer

  • Description modified (diff)

comment:84 in reply to: ↑ 79 ; follow-up: Changed 8 years ago by iandrus

  • Cc iandrus added

Replying to jdemeyer:

The new flask-sagenb will likely require changes to the Mac App data/extcode/sage/ext/mac-app also. It seems to have port 8000 hardcoded in a few places.

What is more likely to need changes is the script sage-is-running-on-port.sh which uses the contents of ~/.sage/sage_notebook.sagenb/twistd.pid and ~/.sage/sage_notebook.sagenb/twistedconf.tac to return the port the server is running on.

comment:85 Changed 8 years ago by jdemeyer

  • Status changed from needs_review to needs_work

I get Sphinx warnings/errors when building the documentation:

dochtml.log:/mnt/usb1/scratch/jdemeyer/merger/sage-5.0.beta2/devel/sagenb/sagenb/notebook/cell.py:docstring of sagenb.notebook.cell.Cell:4: WARNING: Block quote ends without a blank line; unexpected unindent.
dochtml.log:/mnt/usb1/scratch/jdemeyer/merger/sage-5.0.beta2/devel/sagenb/sagenb/notebook/interact.py:docstring of sagenb.notebook.interact.slider_generic.values:11: WARNING: Literal block expected; none found.
dochtml.log:/mnt/usb1/scratch/jdemeyer/merger/sage-5.0.beta2/devel/sage/doc/en/reference/sagenb/notebook/twist.rst:7: WARNING: autodoc can't import/find module 'sagenb.notebook.twist', it reported error: "No module named twist", please check your spelling and sys.path
dochtml.log:/mnt/usb1/scratch/jdemeyer/merger/sage-5.0.beta2/devel/sage/doc/en/reference/sagenb/simple/twist.rst:7: WARNING: autodoc can't import/find module 'sagenb.simple.twist', it reported error: "No module named web2", please check your spelling and sys.path
dochtml.log:/mnt/usb1/scratch/jdemeyer/merger/sage-5.0.beta2/devel/sagenb/sagenb/notebook/cell.py:docstring of sagenb.notebook.cell.Cell:14: ERROR: Unexpected indentation.

comment:86 Changed 8 years ago by jhpalmieri

Jason has already posted this to sage-devel, but a one character fix for the two problems in cell.py:

  • sagenb/notebook/cell.py

    diff --git a/sagenb/notebook/cell.py b/sagenb/notebook/cell.py
    a b class Cell(Cell_generic): 
    11611161        return u'{{{id=%s|\n%s\n}}}' % (self.id(), s)
    11621162
    11631163    def next_compute_id(self):
    1164         """
     1164        r"""
    11651165        Returns the ID of the next compute cell in this compute cell's
    11661166        worksheet object.  If this cell is *not* in the worksheet, it
    11671167        returns the ID of the worksheet's *first* compute cell.  If

Changed 8 years ago by jason

comment:87 Changed 8 years ago by jason

  • Description modified (diff)

comment:88 Changed 8 years ago by jason

  • Description modified (diff)

comment:89 Changed 8 years ago by jason

  • Status changed from needs_work to needs_review

I updated the sagenb spkg to include up through commit b4cd1d20fa617008e8f12d9da684a508bafa7891, which resolves the doc building issues.

comment:90 in reply to: ↑ 84 Changed 8 years ago by iandrus

Replying to iandrus:

Replying to jdemeyer:

The new flask-sagenb will likely require changes to the Mac App data/extcode/sage/ext/mac-app also. It seems to have port 8000 hardcoded in a few places.

What is more likely to need changes is the script sage-is-running-on-port.sh which uses the contents of ~/.sage/sage_notebook.sagenb/twistd.pid and ~/.sage/sage_notebook.sagenb/twistedconf.tac to return the port the server is running on.

I tried this with the new notebook server and it seems to work fine.

comment:91 Changed 8 years ago by strogdon

  • Cc strogdon added

comment:92 Changed 8 years ago by jason

  • Description modified (diff)

comment:93 Changed 8 years ago by ppurka

  • Status changed from needs_review to needs_work

Applying this to sage-4.8, I ran into one major problem. And I suggest some other minor changes:

  1. There is a leftover file apply-hg.py that comes up when running hg status:
    ...ons/sage-4.8/devel/sagenb> hg status
    *** failed to import extension hggit: No module named hggit
    ! util/apply-hg.py
    
  1. It is not possible to login as user or admin. The notebook returns with a "Wrong password" message. I tried to look at user_manager to see if the problem is there but it isn't there. It's only from the web interface that the problem arises. There are two aspects of this: i) I can login with the password of a new user once I create a new user and use the randomly generated password given by Sage. ii) As soon as I change the randomly generated password to my own password using the web interface, I am unable to log in anymore.
  1. Using user_manager.py, I can see that the passwords are correctly retrieved and verified. So, I am unsure where exactly the problem is. Here is an excerpt:
    sage: nb = Notebook('/home/punarbasu/tmp/a.sagenb') 
    sage: um = nb.user_manager()
    sage: um.valid_login_names()             
    ['admin', u'test']
    sage: um.check_password('admin', 'admin')          
    True
    sage: um.check_password('test', 'test')  
    True
    sage: 
    
  1. [Minor change] In devel/sagenb/sagenb/notebook/run_notebook.py the following two changes should be made:
    • sagenb/notebook/run_notebook.py

      diff --git a/sagenb/notebook/run_notebook.py b/sagenb/notebook/run_notebook.py
      a b  
      438438    print "Please choose a new password for the Sage Notebook 'admin' user."
      439439    print "Do _not_ choose a stupid password, since anybody who could guess your password"
      440440    print "and connect to your machine could access or delete your files."
      441     print "NOTE: Only the md5 hash of the password you type is stored by Sage."
       441    print "NOTE: Only the sha256 hash of the password you type is stored by Sage."
      442442    print "You can change your password by typing notebook(reset=True)."
      443443    print "\n" * 2
      444444    while True:
      445445        passwd = getpass.getpass("Enter new password: ")
      446446        from sagenb.misc.misc import min_password_length
      447447        if len(passwd) < min_password_length:
      448             print "That password is way too short. Enter a password with at least 6 characters."
       448            print "That password is way too short. Enter a password with at least %d characters."%min_password_length
      449449            continue
      450450        passwd2 = getpass.getpass("Retype new password: ")
      451451        if passwd != passwd2:

comment:94 follow-up: Changed 8 years ago by jason

  • Status changed from needs_work to needs_info

I fixed (4) in commit https://github.com/sagemath/sagenb/commit/3b24bf8f85f4850eae194b1249179d9c638150b9

(1) This will be fixed in the next version of the spkg.

(2)-(3): I couldn't replicate this. Can you give a precise sequence of steps to replicate this? Here's what I did to test it:

  1. Log in as admin
  2. Create a new user under the Settings | Manage Users
  3. Log out and then log in as the new user
  4. Change the user's password
  5. Log in again with the new password

Everything seemed to work fine.

comment:95 in reply to: ↑ 94 Changed 8 years ago by dimpase

Replying to jason:

I fixed (4) in commit https://github.com/sagemath/sagenb/commit/3b24bf8f85f4850eae194b1249179d9c638150b9

(1) This will be fixed in the next version of the spkg.

(2)-(3): I couldn't replicate this. Can you give a precise sequence of steps to replicate this? Here's what I did to test it:

  1. Log in as admin
  2. Create a new user under the Settings | Manage Users
  3. Log out and then log in as the new user
  4. Change the user's password
  5. Log in again with the new password

Everything seemed to work fine.

I guess it's about upgrading from an existing flask nb installation, from some time ago, and on Sage 4.7.2.

comment:96 follow-up: Changed 8 years ago by jason

So does that mean it's a problem, or not?

comment:97 Changed 8 years ago by jason

  • Status changed from needs_info to needs_review

I upgraded the sagenb spkg to take care of issues (1) and (4) above.

comment:98 in reply to: ↑ 96 ; follow-up: Changed 8 years ago by dimpase

Replying to jason:

So does that mean it's a problem, or not?

I mean that ppurka reports on his attempt of upgrading from an existing flask nb installation, from some time ago, that used to be running Sage 4.7.2. Might be nontrivial to reproduce...

comment:99 in reply to: ↑ 98 Changed 8 years ago by jdemeyer

Upgrading is certainly something which should be tested. The current policy in Sage is that source upgrades from sage-4.5 (released July 2010) or later should work. This isn't a very official policy and can be changed to a later Sage version if needed. But certainly upgrading from the pretty recent sage-4.7.2 should work.

Backwards-compatibility is even more important for the saved worksheets in DOT_SAGE. The older worksheets you can support, the better.

comment:100 Changed 8 years ago by ppurka

Oh no no! This is on a fresh sagenb directory! Do the following (tried on sage-4.8):

sage -n directory=new.sagenb
# Give admin password
# In web interface logout and try to log back in
# You will get error
# Restart sage using above command
# In web interface, createa  new user and then logout and login with the random password of new user
# It works. Now, go to settings of new user and change the password and logout
# Try to login as the new user with new password. It doesn't work.

comment:101 follow-ups: Changed 8 years ago by jason

I explicitly tried each of those steps, and things worked flawlessly. I'm trying this on a 5.0 beta (which is where the notebook should be tested). I wonder if that makes a difference (it shouldn't, but who knows?).

Just to make sure: did you explicitly do each of the steps in the description of this ticket?

Also, is there any error message printed out for you in the terminal?

comment:102 in reply to: ↑ 101 Changed 8 years ago by ppurka

Replying to jason:

I explicitly tried each of those steps, and things worked flawlessly. I'm trying this on a 5.0 beta (which is where the notebook should be tested). I wonder if that makes a difference (it shouldn't, but who knows?).

I believe the only difference is coming from this. I tried this on 4.8, where it doesn't work. I haven't tried on 5.0.

Just to make sure: did you explicitly do each of the steps in the description of this ticket?

Yes. Just copy pasted the whole thing.

Also, is there any error message printed out for you in the terminal?

No error messages are output to the terminal. I can consistently reproduce this on two Linux systems. One is an Ubuntu-11.10 and the other is a Gentoo system. Both 64bits. Logins clearly also doesn't work on existing sagenb directories (but that will probably fix itself if this gets fixed for new sagenb directories).

comment:103 follow-up: Changed 8 years ago by ppurka

Ok. I get the exact same behavior on a sage-5.0-beta2. Passwords don't work for admin and for user if user changes the random password. But the passwords work when tested in command line a.k.a. comment 93, no. 3. All on a fresh sagenb directory. Can anyone else confirm this behavior?

comment:104 in reply to: ↑ 103 Changed 8 years ago by dimpase

  • Status changed from needs_review to needs_info

Replying to ppurka:

Ok. I get the exact same behavior on a sage-5.0-beta2. Passwords don't work for admin and for user if user changes the random password. But the passwords work when tested in command line a.k.a. comment 93, no. 3. All on a fresh sagenb directory. Can anyone else confirm this behavior?

Is it browser-specific?

comment:105 Changed 8 years ago by ppurka

No. It's not. I get same behavior on firefox-10 and opera-11.61.

comment:106 Changed 8 years ago by gutow

Why is #12327 a dependency here? If I try to apply the patch from that ticket to 5.0 beta2 after applying everything described above, I get an error stating that the $SAGE_ROOT/ext directory is missing...otherwise so far so good...

comment:107 follow-up: Changed 8 years ago by gutow

Arrrgh...after rebuild (./sage -b). The notebook launches with an error

ImportError                               Traceback (most recent call last)

/home/jonathan/.sage/<ipython console> in <module>()

/home/jonathan/Documents/sage-5.0.beta2/devel/sagenb-main/sagenb/notebook/notebook_object.pyc in __call__(self, *args, **kwds)
    215     """
    216     def __call__(self, *args, **kwds):
--> 217         return self.notebook(*args, **kwds)
    218 
    219     notebook = run_notebook.notebook_twisted

/home/jonathan/Documents/sage-5.0.beta2/devel/sagenb-main/sagenb/notebook/run_notebook.pyc in notebook_twisted(self, directory, port, interface, address, port_tries, secure, reset, accounts, require_login, server_pool, ulimit, timeout, open_viewer, sagetex_path, start_path, fork, quiet, subnets)
    200         print '*' * 70
    201 
--> 202     nb = notebook.load_notebook(directory)
    203 
    204     directory = nb._dir

/home/jonathan/Documents/sage-5.0.beta2/devel/sagenb-main/sagenb/notebook/notebook.pyc in load_notebook(dir, interface, port, secure)
   1807     # mainly to avoid circular references, etc.  This also means
   1808     # only one notebook can actually be used at any point.
-> 1809     import sagenb.notebook.twist
   1810     sagenb.notebook.twist.notebook = nb
   1811 

/home/jonathan/Documents/sage-5.0.beta2/devel/sagenb-main/sagenb/notebook/twist.py in <module>()
     40 
     41 from twisted.python import log
---> 42 from twisted.web2 import server, http, resource, channel
     43 from twisted.web2 import static, http_headers, responsecode
     44 from twisted.web2.filter import gzip

ImportError: No module named web2

Did I miss something or is there a new problem?

comment:108 in reply to: ↑ 107 Changed 8 years ago by dimpase

ImportError?: No module named web2 }}} Did I miss something or is there a new problem?

I bet you did, as there is no web2 in the new twisted (version 11).

comment:109 Changed 8 years ago by jason

And there is no twist.py in the new notebook... Are you *sure* that you installed the sagenb spkg above? Does devel/sagenb-main/flask_version exist?

You may need to do sage -f instead of sage -i when you install the spkgs above so they overwrite old versions of the same spkg.

comment:110 Changed 8 years ago by gutow

Yep...the sagenb spkg install failed on a missing header for ssl. Somehow I missed the error message. I have the ssl binaries but not the header files on my systems. I guess ssl.h be part of the package or they need to be listed as prerequisite for the sagenb spkg. I'm running on the same Ubuntu 11.04 that I successfully installed this on through 4.8.

Here's the relevant part of the log file.

Processing pyOpenSSL-0.13.tar.gz
Running pyOpenSSL-0.13/setup.py -q bdist_egg --dist-dir /tmp/easy_install-pWFAnl/pyOpenSSL-0.13/egg-dist-tmp-4xiQaN
warning: no previously-included files matching '*.pyc' found anywhere in distribution
In file included from OpenSSL/crypto/crypto.h:30:0,
                 from OpenSSL/crypto/crypto.c:16:
OpenSSL/crypto/x509.h:17:25: fatal error: openssl/ssl.h: No such file or directory
compilation terminated.
error: Setup script exited with error: command 'gcc' failed with exit status 1
Error installing pyOpenSSL-0.13.tar.gz.

comment:111 Changed 8 years ago by gutow

sagenb spkg requires openssl and openssl-dev. After adding openssl-dev to my system the spkg installs fine. If I missed this dependency sorry. If not it needs to be clear somewhere.

comment:112 Changed 8 years ago by jason

Should we put an explicit check for that file into the spkg-install? If so, what should that check look like? We should also clearly list this somewhere.

comment:113 in reply to: ↑ 101 Changed 8 years ago by ppurka

Replying to jason:

I explicitly tried each of those steps, and things worked flawlessly. I'm trying this on a 5.0 beta (which is where the notebook should be tested). I wonder if that makes a difference (it shouldn't, but who knows?).

Just to make sure: did you explicitly do each of the steps in the description of this ticket?

Also, is there any error message printed out for you in the terminal?

Thanks to kini we have the answer to this. It seems the error is triggered only when the password length is less than 6 characters. This means that the min password length of 6 is hard-coded in some file. The new git revisions of the nb which has set the variable min_password_length to 6 effectively masks the problem without fixing it.

comment:114 Changed 8 years ago by strogdon

I have a question about the following chunk of code

try: 
    # simplejson is faster, so try to import it first
    import simplejson as json
except ImportError: 
    import json

in sagenb/notebook/misc.py. It would seem that Sage is not shipped with simplejson. In testing the flask notebook on sage-on-gentoo the sagenb/notebook/misc.py test is the only failing notebook test because simplejson, which is pulled in by werkzeug as a runtime dependency, is found. The failure, roughly, is

sage -t -long  -force_lib /storage/strogdon/gentoo/usr/lib/python2.7/site-packages/sagenb/notebook/misc.py
**********************************************************************
File "/storage/strogdon/gentoo/usr/lib/python2.7/site-packages/sagenb/notebook/misc.py", line 221:
    sage: print encode_response(d, separators = (', ', ': '), indent = 4)
Exception raised:
    Traceback (most recent call last):
...
      File "element.pyx", line 1305, in sage.structure.element.RingElement.__add__ (sage/structure/element.c:11569)
      File "coerce.pyx", line 797, in sage.structure.coerce.CoercionModel_cache_maps.bin_op (sage/structure/coerce.c:7467)
    TypeError: unsupported operand parent(s) for '+': '<type 'str'>' and 'Integer Ring'
**********************************************************************

If I remove the conditional the test passes. So, the question is, is the conditional reference to simplejson needed? Of course, the s-o-g flask notebook can always be patched.

comment:115 Changed 7 years ago by kini

Patching the sage-on-gentoo flask notebook is not the solution. Test failures should not occur no matter what python modules, or indeed other programs, the user has installed into Sage's directory.

But there's actually a new failure, on 5.0.beta4:

sage -t  misc/sageinspect.py
**********************************************************************
File "/home/fs/src/sagenb/sagenb/misc/sageinspect.py", line 653:
    sage: sage_getargspec(Poset)
Expected:
    (['data', 'element_labels', 'cover_relations'], None, None, (None, None, False))
Got:
    (['data', 'element_labels', 'cover_relations', 'category', 'facade', 'key'], None, None, (None, None, False, None, None, None))
**********************************************************************
1 items had failures:
   1 of   7 in __main__.example_16
***Test Failed*** 1 failures.

I presume this is caused by #10998.

comment:116 Changed 7 years ago by kini

(Why the heck is sageinspect in sagenb anyway...)

comment:117 Changed 7 years ago by kini

I submitted a pull request to update the failing doctest. Looking at the simple_json one now...

comment:118 Changed 7 years ago by kini

The problem is as follows. In the json module, we have

If ``indent`` is a non-negative integer, then JSON array elements and
object members will be pretty-printed with that indent level. An indent
level of 0 will only insert newlines. ``None`` is the most compact
representation.

On the other hand, in the simple_json module, we have

If ``indent`` is a string, then JSON array elements and object members
will be pretty-printed with a newline followed by that string repeated
for each level of nesting. ``None`` (the default) selects the most compact
representation without any newlines. For backwards compatibility with
versions of simplejson earlier than 2.1.0, an integer is also accepted
and is converted to a string with that many spaces.

Sage integers do not trigger the following lines in simple_json (simplejson/encoder.py:173):

        if isinstance(indent, (int, long)):
            indent = ' ' * indent
        self.indent = indent

Unfortunately it doesn't seem like the json module supports this new string form of the indent argument, which is why it works with our code - it just does ' ' * indent without caring that it's a Sage integer. Apparently the json module is just an old version of the simplejson module which is bundled with Python, so it will get the new string-based indent parameter eventually, but for now we'll either have to require an up-to-date simplejson for the new notebook, or remember/detect which of the two modules we imported and call the function accordingly.

comment:119 Changed 7 years ago by kini

FWIW I made a pull request on simplejson addressing this: https://github.com/simplejson/simplejson/pull/29

comment:120 Changed 7 years ago by kini

The pull request has been merged into simplejson, so the test will no longer fail once the next version of simplejson is released.

comment:121 follow-up: Changed 7 years ago by kini

The new version of simplejson has been released. strogdon, if you upgrade simplejson to 2.3.3 the test failure should be gone :)

comment:122 in reply to: ↑ 121 Changed 7 years ago by strogdon

Replying to kini:

The new version of simplejson has been released. strogdon, if you upgrade simplejson to 2.3.3 the test failure should be gone :)

Good deal. I just upgraded to 2.3.3 and the test passes. Thanks.

comment:123 follow-up: Changed 7 years ago by strogdon

It's not clear from this ticket which of the twisted patches from #11874 should be applied. I believe it should be trac-11874-remove-twisted.3.patch . At least that one as well as the indicated jmol-spkg patch apply on top of 5.0.beta6 and I'm able to proceed to the "sage -br" step to get the new notebook working.

comment:124 in reply to: ↑ 123 Changed 7 years ago by jdemeyer

Replying to strogdon:

It's not clear from this ticket which of the twisted patches from #11874 should be applied.

Considering that ticket needs_work, the "correct" patch for #11874 still needs to be written.

comment:125 Changed 7 years ago by jason

Current status:

#11874 needs to be fixed and reviewed. There is a failing doctest, but I think the failing doctest may be resolved by using the sage.misc.fpickle module (I'm not sure, though)

#12229 needs review (and possibly a fix for the git_links)

#11503 needs review (and looks like it is ready to go...)

Merge pull request https://github.com/sagemath/sagenb/pull/40

comment:126 follow-up: Changed 7 years ago by dimpase

Please check out http://trac.sagemath.org/sage_trac/ticket/11874#comment:44

It appears that when we build Python 2.7.2 we automatically pull the latest Twisted (currently version 12) from the net and install it into SAGE_LOCAL.

We definitely, definitely do NOT want TWO twisted frameworks installed!!!

comment:127 Changed 7 years ago by jason

  • Description modified (diff)

Change urls to boxen instead of sage.math since sage.math is down.

comment:128 in reply to: ↑ 126 Changed 7 years ago by dimpase

Replying to dimpase:

Please check out http://trac.sagemath.org/sage_trac/ticket/11874#comment:44

It appears that when we build Python 2.7.2 we automatically pull the latest Twisted (currently version 12) from the net and install it into SAGE_LOCAL.

Please ignore this comment. It was somehow due to my confusion when I recently looked at #11874.

Please also note that #11874 needs review now.

comment:129 Changed 7 years ago by dimpase

running Explicit instructions from the ticket description gives one Twisted 12, not (as intended) Twisted 11.1. This needs to be fixed: please see http://trac.sagemath.org/sage_trac/ticket/11874#comment:62

comment:130 Changed 7 years ago by jdemeyer

  • Status changed from needs_info to needs_review

There is a doctest error:

sage -t  -force_lib devel/sagenb-main/sagenb/misc/sageinspect.py
**********************************************************************
File "/padic/scratch/jdemeyer/merger/sage-5.0.notebook/devel/sagenb-main/sagenb/misc/sageinspect.py", line 653:
    sage: sage_getargspec(Poset)
Expected:
    (['data', 'element_labels', 'cover_relations'], None, None, (None, None, False))
Got:
    (['data', 'element_labels', 'cover_relations', 'category', 'facade', 'key'], None, None, (None, None, False, None, None, None))
**********************************************************************

comment:131 follow-up: Changed 7 years ago by jdemeyer

  • Status changed from needs_review to needs_work

comment:132 in reply to: ↑ 131 ; follow-up: Changed 7 years ago by dimpase

Replying to jdemeyer: as mentioned here one just needs to update the spkg using this Git commit

comment:133 Changed 7 years ago by kini

Well, you need to update the spkg to the current master branch, which happens to contain that commit in its ancestry, and thus contains the code change. (Commits are not something you apply individually like patches.)

comment:134 Changed 7 years ago by kini

  • Description modified (diff)
  • Status changed from needs_work to needs_review

Here's a new spkg which should pass doctests: http://boxen.math.washington.edu/home/keshav/files/sagenb-0.9.0.spkg

Are there any other work issues?

comment:135 in reply to: ↑ 132 Changed 7 years ago by kini

  • Status changed from needs_review to needs_work

Hmm, there is something weird when I install the spkg and run sage -t $SAGE_ROOT/devel/sagenb-main: 1) README.rst is getting doctested, which throws up a password prompt; 2) there's a tab character in flask_version/base.py .

Why were these not showing up in earlier doctests? They're not new problems, as far as I can tell.

comment:136 Changed 7 years ago by kini

  • Status changed from needs_work to needs_review

OK, uploaded a new SPKG to the same URL. Now it definitely does pass all its doctests.

comment:137 Changed 7 years ago by jason

Just curious, the new spkg probably has Twisted 12.0, right? Will people at #11874 complain that Twisted was updated in the spkg?

comment:138 follow-up: Changed 7 years ago by kini

Yes, it contains Twisted 12. Yes, they might complain... feel free to create your own spkg with Twisted 11 instead - I might have messed something up anyway :)

comment:139 follow-up: Changed 7 years ago by jason

I won't have very much time this week to work on this, so I'm happy letting you make an spkg.

comment:140 in reply to: ↑ 138 Changed 7 years ago by dimpase

Replying to kini:

Yes, it contains Twisted 12. Yes, they might complain... feel free to create your own spkg with Twisted 11 instead - I might have messed something up anyway :)

No complaints from me. Actually, with the old sagenb 0.9.0 (ie with Twisted 11), when I replaced sagenb spkg in Sage 5.0.beta8 with 0.9.0, and then followed the procedure on #11874, I got Sage built with Twisted 11.1, instead of Twisted 12. So Twisted 12 was sneaking in as it was getting installed before the installation of the old sagenb 0.9.0.

comment:141 in reply to: ↑ 139 Changed 7 years ago by kini

Replying to jason:

I won't have very much time this week to work on this, so I'm happy letting you make an spkg.

OK, just checking that I did everything right... first I cloned the git repo to another git repo (to just get master and no other commits), then used hg-git to pull that clone into a newly-initialized hg repo. Then I ran ./spkg-dist to create the spkg. Did I do anything wrong?

comment:142 Changed 7 years ago by jdemeyer

  • Status changed from needs_review to needs_work

I again get

sage -t  -force_lib devel/sagenb-main/sagenb/misc/sageinspect.py
**********************************************************************
File "/scratch/jdemeyer/merger/sage-5.0.notebook/devel/sagenb-main/sagenb/misc/sageinspect.py", line 653:
    sage: sage_getargspec(Poset)
Expected:
    (['data', 'element_labels', 'cover_relations'], None, None, (None, None, False))
Got:
    (['data', 'element_labels', 'cover_relations', 'category', 'facade', 'key'], None, None, (None, None, False, None, None, None))
**********************************************************************

comment:143 Changed 7 years ago by jason

  • Description modified (diff)

Update instructions to reflect another patch on #11874 (I didn't test the instructions, though!)

comment:144 Changed 7 years ago by ppurka

  • Description modified (diff)

Update to new jmol patch for SAGE_ROOT.


Also, after applying the changes from #11874, #11503 and this ticket, the doctest of devel/sagenb-main/sagenb passes.

After these changes however, there is a problem with typesetting maths and I get this notice on evaluation of every cell, whether it is plot or maths.

Error typesetting mathematics.

And sure enough, something as simple as view(2+2) shows the latex code instead of the maths. jsmath is loaded and the button is present on the right hand corner.

UPDATE: I have same problem with sagenb from github with mathjax instead of jsmath. Seems like something is broken in sage and not sagenb.

Last edited 7 years ago by ppurka (previous) (diff)

comment:145 follow-up: Changed 7 years ago by jdemeyer

  • Milestone changed from sage-5.0 to sage-5.1

We would like to release sage-5.0 reasonably soon, and it's too late to get this still in sage-5.0.

comment:146 follow-up: Changed 7 years ago by kini

  • Description modified (diff)
  • Status changed from needs_work to needs_review

Jeroen: Either the version you installed was from the URL in the ticket description (the one Jason uploaded a while ago) and not the SPKG I posted in a comment recently, or I screwed up making my SPKG and mistakenly packaged an old version of the sagenb code. So to cover both possibilities, I changed the link in the ticket description to point to the SPKG I'm hosting, and I also repacked the SPKG and reuploaded it. It passes doctests on my machine. Can you try it again?

comment:147 in reply to: ↑ 146 Changed 7 years ago by jdemeyer

Replying to kini:

Jeroen: Either the version you installed was from the URL in the ticket description (the one Jason uploaded a while ago)

of course the version I installed was from the URL in the ticket description. I'm not going to read 146 comments to find out the URL of the spkg to test.

comment:148 follow-up: Changed 7 years ago by kini

Right, sorry about that. From your statement that you "again" saw the doctest errors, I assumed that at some point you had not seen the errors, which would have meant that you had installed my SPKG at some point. But I guess you just meant that you were continuing to see the doctest errors. My mistake :)

comment:149 in reply to: ↑ 148 Changed 7 years ago by jdemeyer

Replying to kini:

Right, sorry about that. From your statement that you "again" saw the doctest errors, I assumed that at some point you had not seen the errors, which would have meant that you had installed my SPKG at some point.

Maybe I had installed your SPKG at some point, then later forgot about your SPKG and installed Jason's SPKG again. I don't remember...

comment:150 Changed 7 years ago by kini

Fair enough - anyway it's my fault for not putting the correct info in the ticket description.

comment:151 Changed 7 years ago by gutow

  • Description modified (diff)

update jmol spkg to match #11503

comment:152 in reply to: ↑ 145 ; follow-up: Changed 7 years ago by was

Replying to jdemeyer:

We would like to release sage-5.0 reasonably soon, and it's too late to get this still in sage-5.0.

I agree with this decision.

comment:153 in reply to: ↑ 152 Changed 7 years ago by jason

Replying to was:

Replying to jdemeyer:

We would like to release sage-5.0 reasonably soon, and it's too late to get this still in sage-5.0.

I agree with this decision.

Same here. I had to put the proverbial rallying flag down due to work, but no one has really picked it back up again and finished the job. I'll be able to work on this more in a few weeks after our classes get out, if someone else hasn't started pushing it again by then.

comment:154 Changed 7 years ago by kini

What exactly remains to be done, other than the minor hiccup on #11503?

comment:155 follow-up: Changed 7 years ago by jason

What I would do to answer that question is go through each ticket dependency and see if it is positive-reviewed. If not, then that ticket needs to be done :). I don't think there is much left to be done.

comment:156 Changed 7 years ago by gutow

  • Description modified (diff)

comment:157 in reply to: ↑ 155 ; follow-up: Changed 7 years ago by dimpase

Replying to jason:

What I would do to answer that question is go through each ticket dependency and see if it is positive-reviewed. If not, then that ticket needs to be done :). I don't think there is much left to be done.

I think one should provide a recipe for preparing a sage distribution with all these patches applied to be built from scratch, rather than patching an already built distribution. (Assuming one has hg (or a previous version of Sage) installed system-wide). That is, most of these patches are to be applied to sage-??? spkg.

It makes more sense this way, as the changes are so extensive that it's time-consuming, and more error-prone, I think, to apply these to an already built installation of Sage.

comment:158 in reply to: ↑ 157 Changed 7 years ago by jdemeyer

Replying to dimpase:

I think one should provide a recipe for preparing a sage distribution with all these patches applied to be built from scratch, rather than patching an already built distribution. (Assuming one has hg (or a previous version of Sage) installed system-wide). That is, most of these patches are to be applied to sage-??? spkg.

I have made http://boxen.math.washington.edu/home/release/sage-5.0.notebook/sage-5.0.notebook.tar to test this ticket and its dependencies.

comment:159 Changed 7 years ago by jdemeyer

  • Status changed from needs_review to needs_work

This fails to install on OS X 10.4 PPC:

Processing pyOpenSSL-0.13.tar.gz
Running pyOpenSSL-0.13/setup.py -q bdist_egg --dist-dir /tmp/easy_install-h8pJu5/pyOpenSSL-0.13/egg-dist-tmp-Awaj3a
warning: no previously-included files matching '*.pyc' found anywhere in distribution
OpenSSL/ssl/connection.c: In function 'ssl_Connection_set_context':
OpenSSL/ssl/connection.c:289:5: warning: implicit declaration of function 'SSL_set_SSL_CTX' [-Wimplicit-function-declaration]
OpenSSL/ssl/connection.c: In function 'ssl_Connection_get_servername':
OpenSSL/ssl/connection.c:313:16: error: 'TLSEXT_NAMETYPE_host_name' undeclared (first use in this function)
OpenSSL/ssl/connection.c:313:16: note: each undeclared identifier is reported only once for each function it appears in
OpenSSL/ssl/connection.c:320:5: warning: implicit declaration of function 'SSL_get_servername' [-Wimplicit-function-declaration]
OpenSSL/ssl/connection.c:320:10: warning: assignment makes pointer from integer without a cast [enabled by default]
OpenSSL/ssl/connection.c: In function 'ssl_Connection_set_tlsext_host_name':
OpenSSL/ssl/connection.c:346:5: warning: implicit declaration of function 'SSL_set_tlsext_host_name' [-Wimplicit-function-declaration]
error: Setup script exited with error: command 'gcc' failed with exit status 1
Error installing pyOpenSSL-0.13.tar.gz.

comment:160 Changed 7 years ago by jdemeyer

This also fails to install on the Skynet machine silius (SUSE ES 11 SP1 ppc64):

Processing pyOpenSSL-0.13.tar.gz
Running pyOpenSSL-0.13/setup.py -q bdist_egg --dist-dir /tmp/easy_install-_3ma5D/pyOpenSSL-0.13/egg-dist-tmp-efVFsy
warning: no previously-included files matching '*.pyc' found anywhere in distribution
In file included from OpenSSL/crypto/crypto.h:30:0,
                 from OpenSSL/crypto/crypto.c:16:
OpenSSL/crypto/x509.h:17:25: fatal error: openssl/ssl.h: No such file or directory
compilation terminated.
error: Setup script exited with error: command 'gcc' failed with exit status 1
Error installing pyOpenSSL-0.13.tar.gz.

comment:161 Changed 7 years ago by kini

According to http://stackoverflow.com/questions/7340784/easy-install-pyopenssl-error we should be able to fix this by requiring pyOpenSSL 0.12 or lower.

Edit: er, I meant the problem on OS X - dunno about this silius one.

Last edited 7 years ago by kini (previous) (diff)

comment:162 follow-up: Changed 7 years ago by jdemeyer

What's the situation with OpenSSL support anyway? I believe it was decided not to require OpenSSL as dependency? But there have been many discussions about this, I might have lost track.

comment:163 Changed 7 years ago by jdemeyer

Also on the Skynet machine cleo (RHEL 5.3 ia64):

Processing pyOpenSSL-0.13.tar.gz
Running pyOpenSSL-0.13/setup.py -q bdist_egg --dist-dir /tmp/easy_install-PQFiHf/pyOpenSSL-0.13/egg-dist-tmp-XsFtPR
warning: no previously-included files matching '*.pyc' found anywhere in distribution
OpenSSL/ssl/connection.c: In function ssl_Connection_set_context:
OpenSSL/ssl/connection.c:289:5: warning: implicit declaration of function SSL_set_SSL_CTX [-Wimplicit-function-declaration]
OpenSSL/ssl/connection.c: In function ssl_Connection_get_servername:
OpenSSL/ssl/connection.c:313:16: error: TLSEXT_NAMETYPE_host_name undeclared (first use in this function)
OpenSSL/ssl/connection.c:313:16: note: each undeclared identifier is reported only once for each function it appears in
OpenSSL/ssl/connection.c:320:5: warning: implicit declaration of function SSL_get_servername [-Wimplicit-function-declaration]
OpenSSL/ssl/connection.c:320:10: warning: assignment makes pointer from integer without a cast [enabled by default]
OpenSSL/ssl/connection.c: In function ssl_Connection_set_tlsext_host_name:
OpenSSL/ssl/connection.c:346:5: warning: implicit declaration of function SSL_set_tlsext_host_name [-Wimplicit-function-declaration]
error: Setup script exited with error: command 'gcc' failed with exit status 1
Error installing pyOpenSSL-0.13.tar.gz.

comment:164 Changed 7 years ago by jdemeyer

On the Skynet machine iras (SUSE ES10 ia64):

Processing pyOpenSSL-0.13.tar.gz
Running pyOpenSSL-0.13/setup.py -q bdist_egg --dist-dir /tmp/easy_install-9gn9yE/pyOpenSSL-0.13/egg-dist-tmp-4CCNMQ
warning: no previously-included files matching '*.pyc' found anywhere in distribution
In file included from OpenSSL/crypto/crypto.h:30:0,
                 from OpenSSL/crypto/crypto.c:16:
OpenSSL/crypto/x509.h:17:25: fatal error: openssl/ssl.h: No such file or directory
compilation terminated.
error: Setup script exited with error: command 'gcc' failed with exit status 1
Error installing pyOpenSSL-0.13.tar.gz.

comment:165 Changed 7 years ago by jdemeyer

On UGA rosemary (RHEL 5.6 x86_64):

Processing pyOpenSSL-0.13.tar.gz
Running pyOpenSSL-0.13/setup.py -q bdist_egg --dist-dir /tmp/easy_install-urCg6W/pyOpenSSL-0.13/egg-dist-tmp-fw0DPw
warning: no previously-included files matching '*.pyc' found anywhere in distribution
OpenSSL/ssl/connection.c: In function ‘ssl_Connection_set_context’:
OpenSSL/ssl/connection.c:289:5: warning: implicit declaration of function ‘SSL_set_SSL_CTX’ [-Wimplicit-function-declaration]
OpenSSL/ssl/connection.c: In function ‘ssl_Connection_get_servername’:
OpenSSL/ssl/connection.c:313:16: error: ‘TLSEXT_NAMETYPE_host_name’ undeclared (first use in this function)
OpenSSL/ssl/connection.c:313:16: note: each undeclared identifier is reported only once for each function it appears in
OpenSSL/ssl/connection.c:320:5: warning: implicit declaration of function ‘SSL_get_servername’ [-Wimplicit-function-declaration]
OpenSSL/ssl/connection.c:320:10: warning: assignment makes pointer from integer without a cast [enabled by default]
OpenSSL/ssl/connection.c: In function ‘ssl_Connection_set_tlsext_host_name’:
OpenSSL/ssl/connection.c:346:5: warning: implicit declaration of function ‘SSL_set_tlsext_host_name’ [-Wimplicit-function-declaration]
error: Setup script exited with error: command 'gcc' failed with exit status 1
Error installing pyOpenSSL-0.13.tar.gz.

comment:166 in reply to: ↑ 162 Changed 7 years ago by dimpase

Replying to jdemeyer:

What's the situation with OpenSSL support anyway? I believe it was decided not to require OpenSSL as dependency? But there have been many discussions about this, I might have lost track.

See https://groups.google.com/d/msg/sage-devel/iwrF8_kGLzM/32HIts7EPKQJ

so we'll require openssl, but we need to sort out the versions issue, requiring pyOpenSSL 0.12 or lower, as Keshav suggests.

comment:167 Changed 7 years ago by kini

New SPKG with pyOpenSSL 0.12 uploaded. All tests pass on sage.math. Jeroen, can you test it on the buildbots again? It seems like all the errors you reported are coming from pyOpenSSL so I hope downgrading it will solve them...

comment:168 Changed 7 years ago by kini

  • Status changed from needs_work to needs_review

Sorry, I guess I should mark this needs_review again.

comment:169 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:170 Changed 7 years ago by SimonKing

I tried to follow the instructions, but the patch from #11874 fails to apply. Interesting that it was possible to create a testing release, then...

comment:171 Changed 7 years ago by kini

Huh, really? For me, the only dependency that didn't apply was #12327. #11874 worked fine for me.

comment:172 Changed 7 years ago by SimonKing

I think I start to understand. It does not apply with sage-5.0.prealpha0. But in sage-5.0.beta13, the files look totally different.

So, I may (at some point) try again with the latest beta.

comment:173 follow-up: Changed 7 years ago by SimonKing

  • Description modified (diff)

The instructions did not tell that one is supposed to start with the latest beta version. Starting with 5.0.prealpha0, things fail to apply.

comment:174 in reply to: ↑ 173 Changed 7 years ago by dimpase

Replying to SimonKing:

The instructions did not tell that one is supposed to start with the latest beta version. Starting with 5.0.prealpha0, things fail to apply.

Aren't the milestone setting to 5.1 and the fact that it was only marked as "needs review" a day or so ago indications that one has to use a very recent beta?

comment:175 Changed 7 years ago by kini

According to Jeroen's official policy, all tickets are required to have instructions that apply to the latest beta at all times. So this really should not be surprising... regardless of how recently this was marked as "needs review" or the milestone setting.

comment:176 Changed 7 years ago by jdemeyer

I think the "dependencies" field is even more important than the explicit instructions. The dependencies are an easy way to see that a ticket with positive_review cannot be merged because it depends on tickets which don't have positive_review.

comment:177 follow-up: Changed 7 years ago by jdemeyer

Installation on Skynet silius (SUSE ES 11 SP1 ppc64) still fails, presumably because OpenSSL isn't installed:

Processing pyOpenSSL-0.12.tar.gz
Running pyOpenSSL-0.12/setup.py -q bdist_egg --dist-dir /tmp/easy_install-cWTpWy/pyOpenSSL-0.12/egg-dist-tmp-iS5i6u
warning: no files found matching 'COPYING'
warning: no previously-included files matching '*.pyc' found anywhere in distribution
In file included from OpenSSL/crypto/crypto.h:17:0,
                 from OpenSSL/crypto/crypto.c:16:
OpenSSL/crypto/x509.h:17:25: fatal error: openssl/ssl.h: No such file or directory
compilation terminated.
error: Setup script exited with error: command 'gcc' failed with exit status 1
Error installing pyOpenSSL-0.12.tar.gz.

I'm going to try to install OpenSSL and try again.

comment:178 in reply to: ↑ 177 ; follow-up: Changed 7 years ago by SimonKing

Replying to jdemeyer:

Error installing pyOpenSSL-0.12.tar.gz.

When trying to build sage-5.1.notebook, I get a similar error with

king@mpc622:~$ uname -a
Linux mpc622 2.6.34.linuxpool #0 SMP PREEMPT Wed May 19 16:32:19 CEST 2010 x86_64 GNU/Linux
king@mpc622:~$ cat /etc/issue
Debian GNU/Linux 6.0 \n \l

king@mpc622:~$ cat /proc/cpuinfo 
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 37
model name	: Intel(R) Core(TM) i3 CPU         530  @ 2.93GHz
stepping	: 2
cpu MHz		: 1197.000
cache size	: 4096 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 2
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 popcnt lahf_lm arat tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5852.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 37
model name	: Intel(R) Core(TM) i3 CPU         530  @ 2.93GHz
stepping	: 2
cpu MHz		: 1197.000
cache size	: 4096 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 2
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 popcnt lahf_lm arat tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5851.87
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 37
model name	: Intel(R) Core(TM) i3 CPU         530  @ 2.93GHz
stepping	: 2
cpu MHz		: 1197.000
cache size	: 4096 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 2
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 popcnt lahf_lm arat tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5851.87
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 37
model name	: Intel(R) Core(TM) i3 CPU         530  @ 2.93GHz
stepping	: 2
cpu MHz		: 1197.000
cache size	: 4096 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 2
apicid		: 5
initial apicid	: 5
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 popcnt lahf_lm arat tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5851.86
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

I have posted the install log.

Last edited 7 years ago by SimonKing (previous) (diff)

comment:179 in reply to: ↑ 178 ; follow-up: Changed 7 years ago by dimpase

Replying to SimonKing:

Replying to jdemeyer:

Error installing pyOpenSSL-0.12.tar.gz.

When trying to build sage-5.1.notebook, I get a similar error with

it looks as if you don't have openssl installed.

comment:180 in reply to: ↑ 179 Changed 7 years ago by SimonKing

Replying to dimpase:

Replying to SimonKing:

Replying to jdemeyer:

Error installing pyOpenSSL-0.12.tar.gz.

When trying to build sage-5.1.notebook, I get a similar error with

it looks as if you don't have openssl installed.

How can I install it when I am not root? Is there an openssl spkg that I could install instead?

comment:181 Changed 7 years ago by SimonKing

Apparently there *is* an openssl spkg, I am installing it now.

The installation starts with many warnings.

Question: Shouldn't openssl be a standard spkg, since it is a dependency for the new notebook?

comment:182 Changed 7 years ago by SimonKing

I manually installed the optional openssl spkg and typed "make" again. Bug installation of sagenb-0.9.0.spkg did still not work. I have updated the install log.

comment:183 follow-up: Changed 7 years ago by jason

we can't distribute OpenSSL with Sage since it is not GPL-compatible.

comment:184 in reply to: ↑ 183 Changed 7 years ago by dimpase

Replying to jason:

we can't distribute OpenSSL with Sage since it is not GPL-compatible.

I presume one can have an optional package. Does anyone test the scenario Simon is talking about? I don't think it's easy to find a system without openssl installed in the usual way...

comment:185 Changed 7 years ago by SimonKing

By the way: I just found that openssl is installed on the machine in question!

king@mpc622:~$ which openssl
/usr/bin/openssl

And note that installing the optional openssl spkg did not help. In other words, I doubt that the installation failure of sage-5.1.notebook comes from a lack of openssl.

comment:186 follow-up: Changed 7 years ago by kini

You need libSSL development headers. On Ubuntu this is called libssl-dev, not openssl. This is a new systemwide dependency for Sage (see https://groups.google.com/d/msg/sage-devel/iwrF8_kGLzM/32HIts7EPKQJ ) and should be noted in the Sage docs - I'll upload a patch to do just this.

comment:187 Changed 7 years ago by kini

  • Status changed from needs_review to needs_work

Er, sorry, I meant on Debian, not Ubuntu. Marking as needs work until I have that patch uploaded...

comment:188 in reply to: ↑ 186 ; follow-up: Changed 7 years ago by SimonKing

Replying to kini:

You need libSSL development headers.

OK. Is there a libSSL spkg? If not, then I will not be able to build Sage any more on the machine in my office.

comment:189 Changed 7 years ago by SimonKing

I now tried the pyopenssl spkg, but this hasn't helped either...

comment:190 in reply to: ↑ 188 ; follow-up: Changed 7 years ago by dimpase

Replying to SimonKing:

Replying to kini:

You need libSSL development headers.

OK. Is there a libSSL spkg? If not, then I will not be able to build Sage any more on the machine in my office.

Why? I imagine building openssl from source is not such a big deal.

As well, why can't you ask your sysadmin to install openssl headers?

comment:191 in reply to: ↑ 190 ; follow-up: Changed 7 years ago by SimonKing

Replying to dimpase:

Why? I imagine building openssl from source is not such a big deal.

Does the optional openssl spkg contain the necessary sources? Then the openssl spkg should be modified such that the development headers are installed.

Then, I think it would be helpful if the installation script of sagenb-0.9.0.spkg would check for the presence of the headers, and would print a clear message, if there are missing. Such as:

   ****************************************************************
   ** The libSSL development headers are missing. For licensing  **
   ** reasons, we can not distribute it in Sage by default. But  **
   ** you can install the missing headers with an optional Sage  **
   ** package, by doing                                          **
   **     ./sage -i openssl                                      **
   ** in the Sage root directory.                                **
   ****************************************************************

As well, why can't you ask your sysadmin to install openssl headers?

I think that introducing yet another nontrivial dependency for Sage is a bad idea.

comment:192 in reply to: ↑ 191 ; follow-up: Changed 7 years ago by dimpase

Replying to SimonKing:

As well, why can't you ask your sysadmin to install openssl headers?

I think that introducing yet another nontrivial dependency for Sage is a bad idea.

I think this is a trivial dependency. Every modern Linux distro has openssl, and OSX's Xcode comes with openssl preinstalled. I don't see a need to jump hoops just in order to overcome a non-problem.

comment:193 in reply to: ↑ 192 ; follow-up: Changed 7 years ago by SimonKing

Replying to dimpase:

I think that introducing yet another nontrivial dependency for Sage is a bad idea.

I think this is a trivial dependency. Every modern Linux distro has openssl, ...

Sure. My computer has openssl as well. But it has not the development headers.

Meanwhile, I have

king@mpc622:/mnt/local/king/SAGE/experiment/notebook/sage-5.1.notebook$ ls local/include/openssl/
aes.h          conf_api.h     ec.h           mdc2.h         pqueue.h       ssl.h
asn1.h         conf.h         engine.h       modes.h        rand.h         stack.h
asn1_mac.h     crypto.h       e_os2.h        objects.h      rc2.h          symhacks.h
asn1t.h        des.h          err.h          obj_mac.h      rc4.h          tls1.h
bio.h          des_old.h      evp.h          ocsp.h         ripemd.h       ts.h
blowfish.h     dh.h           hmac.h         opensslconf.h  rsa.h          txt_db.h
bn.h           dsa.h          idea.h         opensslv.h     safestack.h    ui_compat.h
buffer.h       dso.h          krb5_asn.h     ossl_typ.h     seed.h         ui.h
camellia.h     dtls1.h        kssl.h         pem2.h         sha.h          whrlpool.h
cast.h         ebcdic.h       lhash.h        pem.h          ssl23.h        x509.h
cms.h          ecdh.h         md4.h          pkcs12.h       ssl2.h         x509v3.h
comp.h         ecdsa.h        md5.h          pkcs7.h        ssl3.h         x509_vfy.h

These are many headers, and they come from the openssl spkg. But apparently the development headers are still not there.

In other words, since about two hours I try to find out where I could get the development headers from, without being root. So, it is indeed a nontrivial dependency.

comment:194 in reply to: ↑ 193 Changed 7 years ago by dimpase

Replying to SimonKing:

Replying to dimpase:

I think that introducing yet another nontrivial dependency for Sage is a bad idea.

I think this is a trivial dependency. Every modern Linux distro has openssl, ...

Sure. My computer has openssl as well. But it has not the development headers.

OK, so openssl headers need to be listed as a dependency for building Sage from source. It is trivial in the sense that installing these headers is trivial (for a root). While I understand that sysadmins might be slow, or picky (at the place I did my PhD at, the sysadmin only replied to emails which had "BEER" mentioned in the subject---I'm not kidding!), I cannot imagine this being hard to overcome...

These are many headers, and they come from the openssl spkg. But apparently the development headers are still not there.

In other words, since about two hours I try to find out where I could get the development headers from, without being root. So, it is indeed a nontrivial dependency.

Looks like a bug in openssl spkg (or in whatever place that need to compile openssl-related stuff).

comment:195 Changed 7 years ago by kini

No, those are the development headers... it looks like you have some case sensitivity problem or something:

In file included from OpenSSL/crypto/crypto.h:17,
                 from OpenSSL/crypto/crypto.c:16:
OpenSSL/crypto/x509.h:17:25: error: openssl/ssl.h: Datei oder Verzeichnis nicht gefunden

Maybe the pyOpenSSL installer is picking up your systemwide openssl headers, which are in some funkily capitalized directory, rather than the ones which the ssl spkg seems to have installed into your sage directory?

comment:196 Changed 7 years ago by kini

Oh, sorry, I'm stupid - those are filenames inside the pyOpenSSL package, I guess.

comment:197 Changed 7 years ago by SimonKing

If these are the development headers, then it seems that the sagenb spkg is looking for them in the wrong place. Is SAGE_ROOT/local/include part of the include path, when sagenb is installed?

comment:198 Changed 7 years ago by SimonKing

Or more precisely: The sagenb spkg contains a file pyOpenSSL-0.12.tar.gz, and the error message indicates that the problem lies in installing that sub-package.

comment:199 follow-up: Changed 7 years ago by kini

OK, so the following text appears in the pyOpenSSL INSTALL file:

If your OpenSSL header files aren't in /usr/include, you may need to supply
the -I flag to let the setup script know where to look. The same goes for the
libraries of course, use the -L flag. Note that build won't accept these
flags, so you have to run first build_ext and then build! Example:

  $ python setup.py build_ext -I/usr/local/ssl/include -L/usr/local/ssl/lib
  $ python setup.py build

So that's why pyOpenSSL isn't seeing the dev headers that are installed by the OpenSSL package. That means we should make a separate pyOpenSSL SPKG and micromanage the build process as described above. (Judging by comments above, I guess one already exists, and so it just needs to be modified.)

By the way, can we really distribute OpenSSL as an optional SPKG? Doesn't that count as distributing it with Sage? Sure, it's not shipped literally with Sage... well, I don't know.

comment:200 Changed 7 years ago by kini

Um, OK... the pyOpenSSL SPKG already does this. So I guess the problem is that sagenb saw there was a newer version of pyOpenSSL sitting right there and decided to install it even though you already had 0.8 installed. I'll upgrade the pyOpenSSL SPKG to 0.12 and then everything should hopefully work...

comment:201 Changed 7 years ago by SimonKing

Hi Keshav,

I was just about to point to the same text.

I don't know about licensing, but I think I was told that an optional spkg is fine for non-GPL packages.

I see in spkg-install that the pyOpenSSL package is installed via easy_install --allow-hosts=None pyOpenSSL-0.12.tar.gz. Isn't there a way to tell easy_install what include directory to use?

comment:202 in reply to: ↑ 199 Changed 7 years ago by dimpase

Replying to kini:

By the way, can we really distribute OpenSSL as an optional SPKG? Doesn't that count as distributing it with Sage? Sure, it's not shipped literally with Sage... well, I don't know.

some optional packages have licenses which are incompatible with GPL, so this should not be a problem, if überhaupt we can distribute openssl...

comment:203 Changed 7 years ago by SimonKing

Replying to SimonKing:

I see in spkg-install that the pyOpenSSL package ...

I mean "spkg-install in the sagenb spkg", and pyOpenSSL should rather be named a sub-package of sagenb.

comment:204 follow-up: Changed 7 years ago by kini

Right, I get what you meant, Simon. It doesn't seem like easy_install has the functionality you mentioned, unfortunately. Not that surprising since most Python packages don't have a C component anyway...

comment:205 follow-up: Changed 7 years ago by SimonKing

By the way: pyOpenSSL has the APL2 licence. Is that compatible with GPL?

It is distributed as sub-package of a standard spkg (namely the sagenb spkg) - so there is a potential violation of GPL, isn't it?

comment:206 in reply to: ↑ 205 Changed 7 years ago by SimonKing

Replying to SimonKing:

By the way: pyOpenSSL has the APL2 licence. Is that compatible with GPL?

Google told me: "The Free Software Foundation considers the Apache License, Version 2.0 to be a free software license, compatible with version 3 of the GPL." At least one good news...

comment:207 in reply to: ↑ 204 Changed 7 years ago by dimpase

Replying to kini:

Right, I get what you meant, Simon. It doesn't seem like easy_install has the functionality you mentioned, unfortunately. Not that surprising since most Python packages don't have a C component anyway...

one would modify package's setup.py to accomplish this, looking for custom header locations. E.g. look at the one of cvxopt.

comment:208 Changed 7 years ago by kini

Supplying command line arguments to setup.py, as Simon and I discussed earlier, is less intrusive than modifying setup.py .

I am making a new pyOpenSSL SPKG.

comment:209 Changed 7 years ago by SimonKing

At stackoverflow, the variables C_INCLUDE_PATH and CPLUS_INCLUDE_PATH are mentioned as an option to make easy_install understand the location of headers.

Worth a try, I'd say!

And why should there be a new pyOpenSSL spkg? I think one should instead modify spkg-install in the sagenb spkg (using C_INCLUDE_PATH), so that the pyOpenSSL sub-package (which is different from the pyOpenSSL spkg) is correctly installed.

comment:210 Changed 7 years ago by SimonKing

  • Owner changed from jason, mpatel, was to (none)

YESSS!

I added

export C_INCLUDE_PATH="$SAGE_ROOT"/local/include

to sagenb's spkg-install script, and that seems to have solved the problem!!

I guess that what one really wants is to add "$SAGE_ROOT"/local/include to an already existing include path. But I guess you know better than I how that can be done in a script...

comment:211 Changed 7 years ago by SimonKing

My suggestions:

  • Add "$SAGE_ROOT"/local/include to C_INCLUDE_PATH, in the sagenb spkg-install
  • If installation of the notebook fails, inform the user about the optional openssl package.

comment:212 Changed 7 years ago by kini

I would like to use systemwide headers if at all possible. Otherwise we are basically requiring people to install the OpenSSL SPKG, which means that we must make the OpenSSL SPKG standard, which is not possible. I don't see a way to use $SAGE_LOCAL/include as a fallback only, other than just putting

if [ ! -d /usr/include/openssl ]; then
    python setup.py build_ext -I"$SAGE_LOCAL"/include -L"$SAGE_LOCAL"/lib
else
    python setup.py build_ext
fi

in an spkg-install script somewhere. That's why I elected to do so in the pyOpenSSL SPKG. I could as well do so in the sagenb SPKG, but that would involve unpacking the pyOpenSSL tarball that's there and messing around with it instead of using easy_install. This way we can make the pyOpenSSL SPKG standard and make the OpenSSL SPKG optional, and still tell people we prefer them to install OpenSSL dev headers systemwide.

comment:213 follow-up: Changed 7 years ago by kini

Wait, I misunderstood. I guess I could just put

if [ ! -d /usr/include/openssl ]; then
    export C_INCLUDE_PATH="$SAGE_ROOT"/local/include
fi
easy_install --allow-hosts=None pyOpenSSL-0.12.tar.gz

into sagenb's spkg_install, haha.

Last edited 7 years ago by kini (previous) (diff)

comment:214 in reply to: ↑ 213 Changed 7 years ago by SimonKing

Replying to kini:

Wait, I misunderstood. I guess I could just put

if [ ! -d /usr/include/openssl ]; then
    export C_INCLUDE_PATH="$SAGE_ROOT"/local/include
fi
easy_install --allow-hosts=None pyOpenSSL-0.12.tar.gz

into sagenb's spkg_install, haha.

Exactly. I did not intend to suggest to restrict search for headers to $SAGE_ROOT/local/include, but merely to add it to the default include path.

comment:215 follow-up: Changed 7 years ago by kini

It looks like C_INCLUDE_PATH has a priority lower than command line options to gcc but higher than the default directories. So if someone has installed the OpenSSL SPKG, its headers will take priority over the system's. Is this still OK? (I guess so...)

comment:216 Changed 7 years ago by jason

I think that's what we want, right? What if their system has an old (or newer) version of openssl? If they install the openssl spkg, we should use that.

As for a test, can't we just try to install pyOpenSSL with the C_INCLUDE_PATH set? If we have openssl installed as an spkg, it'll pick up that. (I think) it will fall through to the system headers if they aren't present in Sage. If there is an error in installing pyopenssl, then print a big warning message about installing the optional openssl spkg.

Last edited 7 years ago by jason (previous) (diff)

comment:217 in reply to: ↑ 215 ; follow-up: Changed 7 years ago by dimpase

Replying to kini:

It looks like C_INCLUDE_PATH has a priority lower than command line options to gcc but higher than the default directories. So if someone has installed the OpenSSL SPKG, its headers will take priority over the system's. Is this still OK? (I guess so...)

It's not only headers, but the runtime, too. One doesn't want to compile with headers for a wrong runtime!

comment:218 in reply to: ↑ 217 Changed 7 years ago by kini

Replying to dimpase:

Replying to kini:

It looks like C_INCLUDE_PATH has a priority lower than command line options to gcc but higher than the default directories. So if someone has installed the OpenSSL SPKG, its headers will take priority over the system's. Is this still OK? (I guess so...)

It's not only headers, but the runtime, too. One doesn't want to compile with headers for a wrong runtime!

Can you elaborate? We need to set LIBRARY_PATH to $SAGE_LOCAL/lib as well, is that what you mean?

comment:219 follow-up: Changed 7 years ago by SimonKing

If it is indeed true that setting C_INCLUDE_PATH gives priority to the given path, but uses the default patch as fallback, then I think setting C_INCLUDE_PATH for all sub-packages (not just for pyOpenSSL - in fact I added the C_INCLUDE_PATH already to the second line of the spkg-install script) would be a valid solution.

And concerning warning message: Is it really a good solution to try to install pyOpenSSL and look whether it fails? Should one not rather start by searching for the headers, and immediately raise the warning (and stop installation) if they are missing, without the attempt to install any sub-package?

comment:220 in reply to: ↑ 219 ; follow-up: Changed 7 years ago by jason

Replying to SimonKing:

And concerning warning message: Is it really a good solution to try to install pyOpenSSL and look whether it fails? Should one not rather start by searching for the headers, and immediately raise the warning (and stop installation) if they are missing, without the attempt to install any sub-package?

What if someone has openssl installed in a non-standard place? Looking for openssl in one specific place seems brittle to me.

comment:221 in reply to: ↑ 220 Changed 7 years ago by SimonKing

Replying to jason:

What if someone has openssl installed in a non-standard place? Looking for openssl in one specific place seems brittle to me.

If someone has openssl in a non-standard place, then pyOpenSSL will only pick it up if an appropriate environment variable or setup option is used. Ideally, we would respect that variable/option. This is what I meant by adding $SAGE_ROOT/local/include - adding is not replacing.

comment:222 Changed 7 years ago by kini

Simon: I think Jason was objecting to your suggestion to check for headers first and fail out if they were not found. We'd have to search in every possible place GCC might search, and that seems like too much trouble to engineer - why not just let GCC try to find the headers, I guess is what Jason is saying.

comment:223 Changed 7 years ago by jason

Kini: that's what I was thinking. The problem I had was the -d test for a specific header directory to determine whether we'd even try installing pyopenssl.

comment:224 Changed 7 years ago by SimonKing

If the attempt to build it takes not too much time, then it is ok to simply wait if installation of pyOpenSSL works or not.

By the way, meanwhile building sage-5.1.notebook has finished. I can not do make test, but that's a problem fixed elsewhere (#12568, I think).

What I did to build it boils down to the following:

  • Patch sagenb's spkg-install by adding export C_INCLUDE_PATH="$SAGE_ROOT"/local/include as the second line. Note that the sagenb spkg has plenty of uncommitted changes - that needs to be fixed before release. Also note that my fix may be too simplistic. It is just a proof of concept.
  • make, and wait until it fails
  • ./sage -i openssl
  • make

Now I am adding the patch from #12568 and try make test.

comment:225 Changed 7 years ago by jdemeyer

Testing whether openssl exists and raising an error message if not is better done in prereq, not in sagenb.

comment:226 follow-up: Changed 7 years ago by kini

Jeroen: I think we have agreed not to do such a check, and instead just try to build.

Simon: Note that the sagenb spkg doesn't even have a repository ;) Since the SPKG is generated by version-controlled code in the actual sagenb git repository, I guess someone decided that having a repository in the SPKG was not necessary...

By the way I think we should use CPATH instead of C_INCLUDE_PATH. Apparently this will allow the extra paths provided to apply to C++ and Objective-C files as well as C files. Could come in handy in the future.

comment:227 in reply to: ↑ 226 Changed 7 years ago by jdemeyer

Replying to kini:

Jeroen: I think we have agreed not to do such a check, and instead just try to build.

Why that? I much prefer to see a nice error message like Simon suggested instead of a cryptic gcc error. Also, I much prefer to see it immediatly (in prereq) instead of after hours of building. The point of prereq is precisely to check dependencies of Sage.

comment:228 Changed 7 years ago by kini

Because as Jason said we don't know where to look for SSL headers. It could be frustrating to the user if Sage refuses to build just because our code is too stupid to find the user's SSL headers, even though GCC would be able to find them. Since we are not assuming that OpenSSL headers are going to be in the Sage directory, we should wash our hands of trying to find them outside the Sage directory, where there be monsters! :)

comment:229 Changed 7 years ago by kini

Our other dependencies are all command line utilities if I'm not mistaken so a quick command -v is sufficient to find them, but that is not true for include dirs... right? Well I guess we could just write a simple C file that tries to link in openssl and attempt to compile it as our "check for SSL" in prereq...

comment:230 follow-up: Changed 7 years ago by jdemeyer

These kind of checks is precisely why autoconf was invented...

comment:231 Changed 7 years ago by SimonKing

FWIW: All tests pass!

comment:232 follow-up: Changed 7 years ago by kini

Jeroen: Well, sadly I don't know how to use autoconf :( And apparently, according to people on sage-devel, you cannot just muddle your way through writing autoconf...

Anyway, new SPKG is up. Diff at http://git.io/mM8qnQ . Simon, if you could test this SPKG it would be great. (A complete test would I guess involve putting this SPKG into a freshly extracted copy of the testing release Jeroen made, i.e. replacing the sagenb-0.9.0.spkg that is already there, and then running make and make ptestlong.) Good night for now, it's almost dawn here :) I'll work on editing the installation manual tomorrow. Er, today...

comment:233 in reply to: ↑ 232 ; follow-up: Changed 7 years ago by jdemeyer

Replying to kini:

Jeroen: Well, sadly I don't know how to use autoconf :(

I don't mind doing that. However, I'd rather wait until the whole openSSL thing in the Notebook has been settled.

comment:234 Changed 7 years ago by ppurka

IIRC, this check for a package can be done with pkgconfig. Something like

if pkg-config openssl; then
    :
else
    echo >&2 "Get some ssl!"
    exit 1
fi
Last edited 7 years ago by ppurka (previous) (diff)

comment:235 follow-up: Changed 7 years ago by jason

pkgconfig isn't installed by default on OSX, but the openssl headers are. So wouldn't the above test return a false result on stock OSX?

comment:236 in reply to: ↑ 235 Changed 7 years ago by SimonKing

Replying to jason:

pkgconfig isn't installed by default on OSX, but the openssl headers are. So wouldn't the above test return a false result on stock OSX?

If pkgconfig is nonstandard on OSX, I think we can not use it. In addition: Would pkgconfig openssl just detect openssl (which is installed on my machine), or would it detect the headers (which are not installed on my machine, by default)?

comment:237 follow-up: Changed 7 years ago by SimonKing

I just tested: pkg-config does not work in the way we need it. Namely, I did pkg-config openssl in the sage shell of sage-5.1.notebook. Apparently openssl is installed (namely, the headers are in $SAGE_ROOT/local/include), but pkg-config openssl returns nothing.

comment:238 in reply to: ↑ 230 Changed 7 years ago by dimpase

Replying to jdemeyer:

These kind of checks is precisely why autoconf was invented...

indeed. And I am also very much for this check being moved to prereq.

comment:239 Changed 7 years ago by jason

Simon, you need to use pkg-config correctly. It returns an exit code, it doesn't print out something, IIRC. Do pkg-config --help to see the options.

comment:240 Changed 7 years ago by jason

+1 to using autoconf, if someone knows how to write the necessary test. How would we then get the result when doing the sagenb spkg? We apparently want the sagenb to work without openssl too.

comment:241 in reply to: ↑ 237 Changed 7 years ago by ppurka

Replying to SimonKing:

I just tested: pkg-config does not work in the way we need it. Namely, I did pkg-config openssl in the sage shell of sage-5.1.notebook. Apparently openssl is installed (namely, the headers are in $SAGE_ROOT/local/include), but pkg-config openssl returns nothing.

pkg-config will return nothing unless you ask it to print out.

And I am surprised that it is not present in OSX. According to its homepage it runs on all *nix systems.

comment:242 Changed 7 years ago by jason

It runs just fine on OSX. It is one of the first things I install. It just doesn't come standard.

comment:243 follow-up: Changed 7 years ago by kini

Replying to jason:

We apparently want the sagenb to work without openssl too.

Er, we do? Then why is pyOpenSSL a dependency?

comment:244 in reply to: ↑ 243 Changed 7 years ago by dimpase

Replying to kini:

Replying to jason:

We apparently want the sagenb to work without openssl too.

Er, we do? Then why is pyOpenSSL a dependency?

No, I don't think we really need an openssl-less sagenb. Given the triviality of openssl installation, if needed, this looks like a makework project, to strip this functionality out of sagenb.

comment:245 follow-up: Changed 7 years ago by kini

The dev headers, unlike OpenSSL itself, are not usually installed, in binary Linux distributions. If requiring OpenSSL headers for sagenb results in a majority of Sage users installing the optional OpenSSL SPKG, then calling it "optional" starts to become a bit indefensible... so I can see why allowing sagenb to function without OpenSSL is attractive. After all, we only use it for basically openid and HTTPS connections, am I right?

comment:246 Changed 7 years ago by ppurka

If it is only for openid and https then it does make sense to have it as an optional package. Most people who download Sage, will be downloading for personal use. The few who are running servers, will have to install the extra package. This can be documented in an "upgrade guide for sagenb" perhaps.

comment:247 Changed 7 years ago by SimonKing

  • Reviewers changed from Rado Kirov, Dan Drake, Jason Grout to Rado Kirov, Dan Drake, Jason Grout, Simon King

Above, I had claimed that pkg-config did not recognise the headers installed by the optional openssl spkg. I stand corrected, since inside a sage shell I get

(sage-sh) king@mpc622:sage-5.1.notebook$ pkg-config --cflags openssl
-I/mnt/local/king/SAGE/experiment/notebook/sage-5.1.notebook/local/include  

So, testing for the presence of openssl using pkg-config during configuration (i.e., before starting the installation) makes sense.

If it is true that sagenb would - in principle - also work without openssl, I agree with ppurka that we could easily avoid openssl to become de-facto standard:

  • Make sure that sagenb spkg installs and the notebook basically works, even if openssl is not installed. Of course, the notebook will then lack some features.
  • If one tries to use the openid and https features, then a runtime error should be raised, with a clear error message along the lines
    Install openssl with development headers. If they are not available for
    your operating system, you could install the optional openssl Sage package.
    This is possible, e.g., with the command
       install_package("openssl")
    during a Sage session.
    Then, re-install the Sage notebook package, e.g., by
       install_package("sagenb", forced=True)
    during a Sage session.
    

Would that be enough, actually? Or would a recompilation of Sage be needed after re-installation of sagenb?

comment:248 in reply to: ↑ 245 Changed 7 years ago by dimpase

Replying to kini:

The dev headers, unlike OpenSSL itself, are not usually installed, in binary Linux distributions. If requiring OpenSSL headers for sagenb results in a majority of Sage users installing the optional OpenSSL SPKG, then calling it "optional" starts to become a bit indefensible... so I can see why allowing sagenb to function without OpenSSL is attractive. After all, we only use it for basically openid and HTTPS connections, am I right?

a Linux binary distro doesn't come with gcc and make either, you'd have to install it. Nothing prevents you from installing openssh-dev at this moment, too.

The only real advantage of openssl-less sagenb is inclreased modularity... But, could we postpond this to the next milestone, please?

comment:249 Changed 7 years ago by kini

Wow, really? ... It seems you are correct, Ubuntu 12.04 doesn't come with gcc preinstalled! Well then I agree that we should just encourage it as a dependency and provide an SPKG as well.

comment:250 Changed 7 years ago by SimonKing

FWIW, make test passes again. What I did:

  • Open the tar ball, cd sage-5.1.notebook; replace the existing sagenb spkg with the new one posted here.
  • make; wait until it fails.
  • ./sage -i openssl
  • make
  • make test

Can someone else test whether the new spkg is able to pick up a system wide libssl-dev?

comment:251 Changed 7 years ago by SimonKing

For the record: The problem reported at #11913 is not fixed by the new notebook. But I think, since the new notebook is probably close to being used, that #11913 should start with the new notebook.

comment:252 follow-up: Changed 7 years ago by SimonKing

And another criticism: When starting the notebook, I see

sage: notebook()
The notebook files are stored in: sage_notebook.sagenb
Upgrading model version to version 1
Done upgrading to model version 1
**************************************************
*                                                *
* Open your web browser to http://localhost:8080 *
*                                                *
**************************************************
2012-04-17 11:08:31+0200 [-] Log opened.
2012-04-17 11:08:31+0200 [-] twistd 12.0.0 (/mnt/local/king/SAGE/experiment/notebook/sage-5.1.notebook/local/bin/python 2.7.2) starting up.
2012-04-17 11:08:31+0200 [-] reactor class: twisted.internet.pollreactor.PollReactor.
2012-04-17 11:08:31+0200 [-] QuietSite starting on 8080
2012-04-17 11:08:31+0200 [-] Starting factory <__builtin__.QuietSite instance at 0x3da77e8>

It is not mentioned that one is supposed to type Ctrl-c in order to return to Sage's command line version.

I think there was a recent thread on sage-support (or sage-devel?) started by a user who was confused about that point.

William stated that there used to be a clearer instruction when starting the notebook. Hence, I believe that this ticket is a good occasion to make the notebook instructions clearer.

comment:253 in reply to: ↑ 252 Changed 7 years ago by jdemeyer

Replying to SimonKing:

I believe that this ticket is a good occasion to make the notebook instructions clearer.

I completely disagree with this. Fix it, certainly. But not on this ticket.

comment:254 Changed 7 years ago by jdemeyer

On Skynet iras (SUSE ES10 ia64), I built OpenSSL in my $HOME directory (so not in Sage) and I get the following weird error:

Processing pyOpenSSL-0.12.tar.gz
Running pyOpenSSL-0.12/setup.py -q bdist_egg --dist-dir /tmp/easy_install-GTqwq9/pyOpenSSL-0.12/egg-dist-tmp-yGKR4i
warning: no files found matching 'COPYING'
warning: no previously-included files matching '*.pyc' found anywhere in distribution
/usr/bin/ld: /home/buildbot/local/iras/lib/libcrypto.a(obj_dat.o): @gprel relocation against dynamic symbol obj_cleanup_defer
/usr/bin/ld: /home/buildbot/local/iras/lib/libcrypto.a(obj_dat.o): @gprel relocation against dynamic symbol obj_cleanup_defer
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: ld returned 1 exit status
error: Setup script exited with error: command 'gcc' failed with exit status 1
Error installing pyOpenSSL-0.12.tar.gz.

comment:255 Changed 7 years ago by jdemeyer

Installing the optional openSSL spkg did work on iras.

comment:256 Changed 7 years ago by kini

That is a strange error indeed.

Simon: I agree with Jeroen - that can be taken care of later. There are actually several things people are doing with sagenb that will not make it into 0.9.0 (the version of sagenb we are gearing up to release, as soon as this ticket has positive review). They will go into the following release of sagenb. This includes stuff like the switch from jsmath to mathjax, Ivan Andrus's code to allow opening .sws files from the command line, some LDAP implementation stuff which people are still hashing out, and a Russian translation that someone on github kindly made for us. See https://github.com/sagemath/sagenb/pulls for some stuff that people are working on but hasn't been accepted yet because we're still making 0.9.0. You can make your own edit of the startup message and add it to this list, if you wish :) Here is the relevant code: https://github.com/sagemath/sagenb/blob/master/sagenb/misc/misc.py#L26

(Of course, we should technically have a separate branch for feature freezes such as this, but we'll get around to that later.)

comment:257 Changed 7 years ago by jdemeyer

One small thing still to be fixed in sagenb is to merge the patch from #12677 (I reported this at http://code.google.com/p/sagenb/issues/detail?id=91 before being told that Google Code isn't used anymore).

comment:258 Changed 7 years ago by kini

Done. Building and uploading new SPKG now...

comment:259 Changed 7 years ago by kini

The new SPKG is up. The commit I added is here.

comment:260 in reply to: ↑ 233 ; follow-up: Changed 7 years ago by kini

So what are we going to do about SSL? It seems to me that since one cannot run sage -i openssl from a freshly unpacked tarball, we basically need to require SSL to be installed systemwide. Or we can expect the user to run make, wait for it to fail (!), then run sage -i openssl (hopefully by this time the machinery for sage -i is in place), and then run make again. Should I add a note to this effect to the "install from source" docs page?

comment:261 in reply to: ↑ 260 Changed 7 years ago by jdemeyer

Replying to kini:

hopefully by this time the machinery for sage -i is in place

If you make SageNB depend on sage_scripts, this is guaranteed.

comment:262 follow-up: Changed 7 years ago by kini

Surely sage -i must be installed by the time the sagenb SPKG fails, because the sagenb SPKG installation itself would have been started via sage -i. No?

comment:263 in reply to: ↑ 262 Changed 7 years ago by jdemeyer

Replying to kini:

Surely sage -i must be installed by the time the sagenb SPKG fails, because the sagenb SPKG installation itself would have been started via sage -i. No?

No. The spkg installation could be started by make. Moreover, the part of sage -i openssl which wouldn't work is the downloading part, not the installation part.

comment:264 Changed 7 years ago by kini

OK, I see.

comment:265 Changed 7 years ago by kini

  • Description modified (diff)
  • Status changed from needs_work to needs_review

comment:266 Changed 7 years ago by kini

I've uploaded a patch which causes the "install from source code" docs to mention OpenSSL as a requirement. Generally the whole page needs a cleanup though - see #11237. Marking as needs review again since I don't think there were any other issues (?).

comment:267 Changed 7 years ago by kini

Updated the SPKG due to merging a bugfix from Jason.

comment:268 Changed 7 years ago by jason

Don't the OpenSSL libraries need to be installed, no matter what? And the dev headers installed if you are compiling from source. Are the libraries mentioned as prerequisites somewhere for running the notebook?

comment:269 Changed 7 years ago by kini

  • Description modified (diff)

Yup, that's what the latest patch does - mentions OpenSSL as a requirement for build Sage from source. I should probably also put that in the ticket description for testers/reviewers though.

comment:270 Changed 7 years ago by jason

My point is that OpenSSL libraries are needed for *running* the notebook, are they not? If so, are they listed somewhere as runtime dependencies?

comment:271 Changed 7 years ago by kini

Oh, very good point. No, they are not.

comment:272 Changed 7 years ago by jason

They are not required? Or they are not listed as runtime dependencies?

comment:273 Changed 7 years ago by kini

They are not listed as runtime dependencies. Though I suppose it might be possible that they are not required either - and I think I'd rather not remove OpenSSL from my system to check this... :P I'll expand the latest patch to also mention this as a runtime dependency, if that's OK with you.

comment:274 Changed 7 years ago by kini

  • Description modified (diff)
  • Status changed from needs_review to needs_work

comment:275 Changed 7 years ago by kini

  • Description modified (diff)
  • Status changed from needs_work to needs_review

New patch is up.

comment:276 follow-up: Changed 7 years ago by ppurka

I don't see any problems with the set of patches and spkgs in this ticket at present.

I checked jmol on the command line and on the notebook and checked some other stuff (plot, interacts, math, etc), and they all work. I would give it positive review except that I haven't faced any openssl problems (in any of the earlier iterations of the patches).

comment:277 in reply to: ↑ 276 Changed 7 years ago by gutow

I've got a systems I can set up without OpenSSL to test.  However, I won't be able to do it until I finish with exams and grading (roughly May 12).  If we still need this part reviewed at that time, someone should remind me and I will do it.

Replying to ppurka:

I don't see any problems with the set of patches and spkgs in this ticket at present. I checked jmol on the command line and on the notebook and checked some other stuff (plot, interacts, math, etc), and they all work. I would give it positive review except that I haven't faced any openssl problems (in any of the earlier iterations of the patches).

comment:278 Changed 7 years ago by kini

  • Dependencies changed from #11078, #11874, #12229, #11503, #12327 to #11078, #11874, #12229, #11503, #12327, #12899
  • Status changed from needs_review to needs_work
  • Work issues set to rebase on #12899

Changed 7 years ago by kini

apply to $SAGE_ROOT/devel/sage (on top of #12899)

comment:279 Changed 7 years ago by kini

  • Description modified (diff)
  • Status changed from needs_work to needs_review
  • Work issues rebase on #12899 deleted

Rebased patch up.

Changed 7 years ago by ppurka

Original on top, with this ticket on bottom

comment:280 follow-up: Changed 7 years ago by ppurka

Installed it with openssl dev headers missing in Ubuntu-12.04, 64bits. The instructions provided are good.

Everything working good except for a slight glitch in live documentation - tested interacts, jsmath, plots, jmol (both in command line and in notebook).

In the live documentation, the original output does not contain newlines. See this screenshot.

comment:281 Changed 7 years ago by kini

  • Description modified (diff)

Edited the instructions since #12899 was merged in 5.0.rc1

comment:282 Changed 7 years ago by ppurka

  • Description modified (diff)

Update to refer to the new patch with fixed git links in #12229

comment:283 in reply to: ↑ 280 ; follow-up: Changed 7 years ago by dimpase

Replying to ppurka:

Installed it with openssl dev headers missing in Ubuntu-12.04, 64bits. The instructions provided are good.

Everything working good except for a slight glitch in live documentation - tested interacts, jsmath, plots, jmol (both in command line and in notebook).

In the live documentation, the original output does not contain newlines. See this screenshot.

This must be browser/OS-specific, as I can't reproduce this MacOSX.

comment:284 follow-ups: Changed 7 years ago by jason

@kini, when did you last update the spkg? The other day I fixed some serious scaling bugs that have been around for years (see the latest commits). In the process, I eliminated the need for twisted. We can merge this reviewed version in and then immediately follow up with a new version, or we can try to cram yet a few more commits in. What do you think?

comment:285 in reply to: ↑ 283 Changed 7 years ago by ppurka

Replying to dimpase:

Replying to ppurka:

Installed it with openssl dev headers missing in Ubuntu-12.04, 64bits. The instructions provided are good.

Everything working good except for a slight glitch in live documentation - tested interacts, jsmath, plots, jmol (both in command line and in notebook).

In the live documentation, the original output does not contain newlines. See this screenshot.

This must be browser/OS-specific, as I can't reproduce this MacOSX.

I had tried it on both opera and firefox on Gentoo Linux. If it is not reproducible, then it's even better. We can finally close this ticket. :)

comment:286 in reply to: ↑ 284 Changed 7 years ago by ppurka

Replying to jason:

@kini, when did you last update the spkg? The other day I fixed some serious scaling bugs that have been around for years (see the latest commits). In the process, I eliminated the need for twisted. We can merge this reviewed version in and then immediately follow up with a new version, or we can try to cram yet a few more commits in. What do you think?

I think it would be better to merge this ticket at the earliest. It has come to a stage (after so many months!) where it is finally satisfactory.

The scalability problems can be addressed in a later ticket since the problems are evident only when there are thousands of accounts. From an end-user point of view, the speedup won't be very evident. So, mathjax + uwsgi can be the target of the next big nb ticket.

comment:287 Changed 7 years ago by jason

That sounds good. I wonder if we'll reach 300 comments before this ticket is merged...

comment:288 in reply to: ↑ 284 Changed 7 years ago by dimpase

  • Status changed from needs_review to positive_review

Replying to jason:

@kini, when did you last update the spkg? The other day I fixed some serious scaling bugs that have been around for years (see the latest commits). In the process, I eliminated the need for twisted. We can merge this reviewed version in and then immediately follow up with a new version, or we can try to cram yet a few more commits in. What do you think?

boxen:/home/keshav/files$ ls -l sagenb-0.9.0.spkg 
-rw-r--r-- 1 keshav keshav 26699829 2012-05-03 23:13 sagenb-0.9.0.spkg

Anyway, IMHO it's time to close this ticket. I hereby give it a positive review...

comment:289 Changed 7 years ago by jdemeyer

  • Milestone changed from sage-5.1 to sage-pending

comment:290 follow-up: Changed 7 years ago by kcrisman

  • Reviewers changed from Rado Kirov, Dan Drake, Jason Grout, Simon King to Rado Kirov, Dan Drake, Jason Grout, Simon King, Dmitrii Pasechnik

After looking at #12229 fairly carefully, I believe that it should remain a blocker for whatever Sage release the new notebook goes in, but not for this actual ticket. The instructions are so generic and even currently inaccurate, that it would be counterproductive to keep that from allowing people to test this ticket in an early beta of some Sage version.

I know you all know I wouldn't say this lightly ;-) so as long as nobody has a problem with #12229 having that new status

  • Blocker for ... Sage 5.2, I think Jeroen was estimating?
  • No longer a prereq for this ticket being merged

then let's get this in.

comment:291 Changed 7 years ago by jdemeyer

  • Milestone changed from sage-pending to sage-5.2

comment:292 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:293 in reply to: ↑ description ; follow-up: Changed 7 years ago by jdemeyer

Replying to jason:

When we have finalized things, we'll make a finished spkg that has the appropriate tags in the revision history, no unnecessary revision branches, etc.

Maybe now would be the time to do precisely this?

comment:294 follow-up: Changed 7 years ago by jdemeyer

  • Status changed from positive_review to needs_work

The sagenb's SPKG.txt should list OpenSSL as dependency.

comment:295 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:296 in reply to: ↑ 290 ; follow-up: Changed 7 years ago by jdemeyer

Replying to kcrisman:

After looking at #12229 fairly carefully, I believe that it should remain a blocker for whatever Sage release the new notebook goes in, but not for this actual ticket.

This has a high probability of leading to the situation where #11080 gets merged but nobody looks after #12229. Then sage-5.2 (or whichever version) cannot be released. Whenever there is a group of tickets which must be released together (even if they don't strictly depend on eachother), I prefer to wait until all of them have positive review.

comment:297 in reply to: ↑ 296 Changed 7 years ago by kcrisman

After looking at #12229 fairly carefully, I believe that it should remain a blocker for whatever Sage release the new notebook goes in, but not for this actual ticket.

This has a high probability of leading to the situation where #11080 gets merged but nobody looks after #12229. Then sage-5.2 (or whichever version) cannot be released. Whenever there is a group of tickets which must be released together (even if they don't strictly depend on eachother), I prefer to wait until all of them have positive review.

In this case, because #12229 does not need that much additional work, and because we would like people to TEST the new notebook at an early stage a lot (which #12229 is totally unnecessary for), I disagree.

In any case, most of what ails #12229 is for nb.sagemath.org to be updated slightly so we can change some references there, and I've already sent Harald two emails with specific changes (which hopefully he'll get back to me on soon). So I think it would be penny-wise and pound-foolish to wait for #12229. Also, having this ticket in will put more motivation behind the people at #12229 to get it done :)

comment:298 in reply to: ↑ 293 Changed 7 years ago by kini

Replying to jdemeyer:

Replying to jason:

When we have finalized things, we'll make a finished spkg that has the appropriate tags in the revision history, no unnecessary revision branches, etc.

Maybe now would be the time to do precisely this?

I've done all of that stuff already except tagging the release. Stand by...

comment:299 Changed 7 years ago by kini

John Palmieri, who is sitting next to me at sage days 41, suggests that we just not package a repository with sagenb at all. What do you think? After all, sagenb is considered an upstream package at this point, and we don't package repositories for those. Plus it's using a revision control system that isn't included with sage anyway (yet). Also, due to various decisions by people over the lifetime of the repository, the repository is really large - 14 MB at maximum compression, at the current master. So removing it would slim down the sage source and binary distributions by a non-negligible amount. The docs added at #12229 already give instructions on how to clone the repository from github anyway. So why do we need to ship a repo?

comment:300 Changed 7 years ago by kcrisman

Please, let's just get this positively reviewed. Also, I'm not really interested in touching #12229 again, which we would have to do if you removed the hg repo this time around. "Slimming down" sounds fine after Jeroen has definitively put this in 5.2.

comment:301 Changed 7 years ago by jhpalmieri

I was really suggesting this "slimming" for the followup ticket, #13121, not for this one.

comment:302 Changed 7 years ago by kini

Sure, that's fine.

comment:303 Changed 7 years ago by kini

New SPKG is up at the same URL.

comment:304 Changed 7 years ago by kini

  • Status changed from needs_work to needs_review

comment:305 in reply to: ↑ 294 Changed 7 years ago by jhpalmieri

Replying to jdemeyer:

The sagenb's SPKG.txt should list OpenSSL as dependency.

It doesn't look like this has been addressed.

comment:306 follow-up: Changed 7 years ago by jhpalmieri

By the way, as far as openssl goes, it probably shouldn't be a dependency, since openssl (at least our current spkg) doesn't seem to build on OS X Lion. So the role of openssl should be spelled out in SPKG.txt: under what circumstances do you need to install it?

comment:307 in reply to: ↑ 306 ; follow-up: Changed 7 years ago by jdemeyer

Replying to jhpalmieri:

By the way, as far as openssl goes, it probably shouldn't be a dependency, since openssl (at least our current spkg) doesn't seem to build on OS X Lion.

Isn't OpenSSL part of the OS X operating system?

So the role of openssl should be spelled out in SPKG.txt: under what circumstances do you need to install it?

If and only if it is not already supplied on the system.

comment:308 in reply to: ↑ 307 ; follow-up: Changed 7 years ago by jhpalmieri

Replying to jdemeyer:

Replying to jhpalmieri:

By the way, as far as openssl goes, it probably shouldn't be a dependency, since openssl (at least our current spkg) doesn't seem to build on OS X Lion.

Isn't OpenSSL part of the OS X operating system?

I guess it is, but perhaps not for much longer.

So the role of openssl should be spelled out in SPKG.txt: under what circumstances do you need to install it?

If and only if it is not already supplied on the system.

Okay, sounds good, but this needs to be in SPKG.txt, in clear enough terms so someone like me reading it knows not to try to install the optional openssl spkg on OS X, rather than trying sage -i openssl and then worrying when it doesn't build.

comment:309 in reply to: ↑ 308 Changed 7 years ago by gutow

Are there any issues left other than the openssl stuff? I've been running this for almost a year on the server I use with my students and have had good luck with it (it is at least as good as the standard notebook). Additionally, I've developed #12299 on top of #11080, so have looked at some fraction of the code in #11080. I'm willing to give it a positive review once the openssl stuff is settled.

comment:310 Changed 7 years ago by jdemeyer

For me, all that is needed for positive review is to document in SPKG.txt the fact that this depends on OpenSSL.

comment:311 follow-up: Changed 7 years ago by kini

Jeroen: the SPKG.txt file here is extremely out of date and there is a lot more wrong with it than the fact that the OpenSSL dependency is not mentioned. I have more or less completely rewritten the SPKG.txt at github #63, which I hope will go into #13121 (in fact the new SPKG.txt is in the preliminary 0.9.1 SPKG I have put up on that ticket). More importantly, the OpenSSL dependency is already explained in the installation docs (by trac_11080-openssl-in-installation-docs.patch). Is it really necessary to edit this old version of SPKG.txt when OpenSSL is already stated as a dependency of Sage as a whole?

John makes a good point, though - it should be made clear that installing OpenSSL is unnecessary if OpenSSL is already on your system. The current patch to the installation docs could be updated to say that more explicitly.

comment:312 in reply to: ↑ 311 ; follow-up: Changed 7 years ago by jdemeyer

  • Status changed from needs_review to positive_review

Replying to kini:

Jeroen: the SPKG.txt file here is extremely out of date and there is a lot more wrong with it than the fact that the OpenSSL dependency is not mentioned. I have more or less completely rewritten the SPKG.txt at github #63, which I hope will go into #13121 (in fact the new SPKG.txt is in the preliminary 0.9.1 SPKG I have put up on that ticket). More importantly, the OpenSSL dependency is already explained in the installation docs (by trac_11080-openssl-in-installation-docs.patch). Is it really necessary to edit this old version of SPKG.txt when OpenSSL is already stated as a dependency of Sage as a whole?

Okay, that's good for me.

comment:313 follow-up: Changed 7 years ago by nborie

Hi everybody,

I just post this comment to tell you that my machine (core 2 duo with Ubuntu 12.04 32 bits installation) asked to install uuid-dev package when installing sagecell-0.9.0.spkg After the good apt-get install, it works now fine with the 5.0

...
checking for sem_init in -lrt... yes
checking for uuid_generate in -luuid... no
configure: error: cannot link with -luuid, install uuid-dev.
Error configuring zeromq.

The Sage Cell Server works now fine on my machine. On the same machine, I didn't manage to install the sage-5.2.alpha0 which didn't compiled.

Thanks for this nice work. As all this go too far for my computer skills, i can just test your work but no more...

comment:314 Changed 7 years ago by jdemeyer

  • Status changed from positive_review to needs_work

Unfortunately, this fails on hawk (OpenSolaris 06/2009 i86pc):

****************************************************
Host system:
SunOS hawk 5.11 snv_134 i86pc i386 i86pc
****************************************************
C compiler: gcc
C compiler version:
Using built-in specs.
Target: i386-pc-solaris2.11
Configured with: ../gcc-4.4-20100112/configure --with-as=/usr/local/binutils-2.20/bin/as --with-ld=/usr/ccs/bin/ld --with-gmp=/usr/local --with-mpfr=/usr/local
Thread model: posix
gcc version 4.4.3 20100112 (prerelease) (GCC) 
****************************************************
Processing Twisted-12.1.0.tar.bz2
Running Twisted-12.1.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-hCklWr/Twisted-12.1.0/egg-dist-tmp-5538SU
conftest.c:1:23: error: sys/epoll.h: No such file or directory
twisted/python/sendmsg.c: In function ‘sendmsg_sendmsg’:
twisted/python/sendmsg.c:198: error: ‘struct msghdr’ has no member named ‘msg_control’
twisted/python/sendmsg.c:199: error: ‘struct msghdr’ has no member named ‘msg_controllen’
twisted/python/sendmsg.c:201: error: ‘struct msghdr’ has no member named ‘msg_flags’
twisted/python/sendmsg.c:243: warning: implicit declaration of function ‘CMSG_SPACE’
twisted/python/sendmsg.c:268: error: ‘struct msghdr’ has no member named ‘msg_control’
twisted/python/sendmsg.c:269: error: ‘struct msghdr’ has no member named ‘msg_control’
twisted/python/sendmsg.c:274: error: ‘struct msghdr’ has no member named ‘msg_control’
twisted/python/sendmsg.c:276: error: ‘struct msghdr’ has no member named ‘msg_controllen’
twisted/python/sendmsg.c:285: warning: implicit declaration of function ‘CMSG_FIRSTHDR’
twisted/python/sendmsg.c:285: warning: initialization makes pointer from integer without a cast
twisted/python/sendmsg.c:306: warning: implicit declaration of function ‘CMSG_LEN’
twisted/python/sendmsg.c:316: warning: implicit declaration of function ‘CMSG_DATA’
twisted/python/sendmsg.c:316: warning: assignment makes pointer from integer without a cast
twisted/python/sendmsg.c:322: warning: implicit declaration of function ‘CMSG_NXTHDR’
twisted/python/sendmsg.c:322: warning: assignment makes pointer from integer without a cast
twisted/python/sendmsg.c:351: error: ‘struct msghdr’ has no member named ‘msg_control’
twisted/python/sendmsg.c:352: error: ‘struct msghdr’ has no member named ‘msg_control’
twisted/python/sendmsg.c:353: error: ‘struct msghdr’ has no member named ‘msg_control’
twisted/python/sendmsg.c: In function ‘sendmsg_recvmsg’:
twisted/python/sendmsg.c:414: error: ‘struct msghdr’ has no member named ‘msg_control’
twisted/python/sendmsg.c:416: error: ‘struct msghdr’ has no member named ‘msg_controllen’
twisted/python/sendmsg.c:429: warning: assignment makes pointer from integer without a cast
twisted/python/sendmsg.c:432: warning: assignment makes pointer from integer without a cast
twisted/python/sendmsg.c:477: error: ‘struct msghdr’ has no member named ‘msg_flags’
error: Setup script exited with error: command 'gcc' failed with exit status 1
Error installing Twisted-12.1.0.tar.bz2.

I seem to recall that Twisted-11 did work, but I'm not sure.

comment:315 follow-up: Changed 7 years ago by jhpalmieri

The file twisted/topfiles/setup.py has this entry in the variable extensions:

    Extension("twisted.python._epoll",
              ["twisted/python/_epoll.c"],
              condition=lambda builder: (_isCPython and _hasEpoll(builder) and
                                         sys.version_info[:2] < (2, 6))),

The function _hasEpoll (defined in twisted/python/dist.py) should return False if epoll is not present, for example on Solaris, so I don't understand what's going on. (The other parts of the code seem to have fall-backs if epoll is not present.) Anyway, maybe we should try patching this file to remove this extension, when on Solaris or OpenSolaris. I can't log in to hawk right now, so I can't see if this would help.

comment:316 in reply to: ↑ 315 Changed 7 years ago by jdemeyer

I'm not actually sure the error is related to epoll. The real errors refer to struct msghdr which should exist on OpenSolaris.

I can't log in to hawk right now, so I can't see if this would help.

Me neither, I emailed David Kirkby about this.

comment:317 in reply to: ↑ 313 Changed 7 years ago by kini

Replying to nborie:

Hi everybody,

I just post this comment to tell you that my machine (core 2 duo with Ubuntu 12.04 32 bits installation) asked to install uuid-dev package when installing sagecell-0.9.0.spkg After the good apt-get install, it works now fine with the 5.0

...
checking for sem_init in -lrt... yes
checking for uuid_generate in -luuid... no
configure: error: cannot link with -luuid, install uuid-dev.
Error configuring zeromq.

The Sage Cell Server works now fine on my machine. On the same machine, I didn't manage to install the sage-5.2.alpha0 which didn't compiled.

Thanks for this nice work. As all this go too far for my computer skills, i can just test your work but no more...

Thanks, but you should probably post this on the sage-devel mailing list, or maybe on the issue tracker for the sage cell server here. This ticket is for the Sage notebook, not the cell server. And in particular it's for version 0.9.0 of sagenb.

comment:318 Changed 7 years ago by jdemeyer

  • Dependencies changed from #11078, #11874, #12229, #11503, #12327, #12899 to #13126, #11078, #11874, #12229, #11503, #12327, #12899

Added OpenSSL fix (#13126) as dependency, as the old OpenSSL spkg doesn't build on all supported systems.

comment:319 follow-up: Changed 7 years ago by jdemeyer

On hawk, adding the compiler flag -D_XOPEN_SOURCE=500 fixes the problem. Any objections to adding this unconditionally on all architectures?

comment:320 in reply to: ↑ 312 Changed 7 years ago by kcrisman

Replying to jdemeyer:

Replying to kini:

Jeroen: the SPKG.txt file here is extremely out of date and there is a lot more wrong with it than the fact that the OpenSSL dependency is not mentioned. I have more or less completely rewritten the SPKG.txt at github #63, which I hope will go into #13121 (in fact the new SPKG.txt is in the preliminary 0.9.1 SPKG I have put up on that ticket). More importantly, the OpenSSL dependency is already explained in the installation docs (by trac_11080-openssl-in-installation-docs.patch). Is it really necessary to edit this old version of SPKG.txt when OpenSSL is already stated as a dependency of Sage as a whole?

Okay, that's good for me.

Though I'd caution - see this ask.sage.math.org post. It's not necessarily all about the notebook, but apparently people using Linux often have this problem. Do we say that this is a dependency for Sage? If this isn't relevant, my apologies.

comment:321 follow-up: Changed 7 years ago by kini

That's a problem with Python 2.6 which is fixed in 2.7 if I'm not mistaken - in Sage terms, that means that 4.x will consistently have that problem on newer distros (especially Debian/Ubuntu it seems) unless the user has purposely installed an older-than-default version of OpenSSL systemwide.

Last edited 7 years ago by kini (previous) (diff)

comment:322 in reply to: ↑ 321 Changed 7 years ago by kcrisman

Replying to kini:

That's a problem with Python 2.6 which is fixed in 2.7 if I'm not mistaken - in Sage terms, that means that 4.x will consistently have that problem on newer distros (especially Debian/Ubuntu it seems) unless the user has purposely installed an older-than-default version of OpenSSL systemwide.

Hmm, but these all refer to 5.0 and 5.0.1.

Anyway, it may have nothing to do with the notebook and something to do with us needing to have better-documented prereqs. On the other hand, most of those seem to have been using Sage for 10.04 on newer Ubuntu, maybe that's the problem. Sorry if this was noise, it just seemed possibly related because one of the posts was about the notebook.

comment:323 in reply to: ↑ 319 Changed 7 years ago by jhpalmieri

Replying to jdemeyer:

On hawk, adding the compiler flag -D_XOPEN_SOURCE=500 fixes the problem. Any objections to adding this unconditionally on all architectures?

Do you mean something like this?

  • spkg-install

    old new  
    1212[ -z "$LIBRARY_PATH" ] || LIBRARY_PATH="$LIBRARY_PATH":
    1313export CPATH="$CPATH""$SAGE_LOCAL"/include
    1414export LIBRARY_PATH="$LIBRARY_PATH""$SAGE_LOCAL"/lib
    15 
     15export CFLAGS="$CFLAGS -D_XOPEN_SOURCE=500"
    1616
    1717easy_install --allow-hosts=None Twisted-12.1.0.tar.bz2
    1818if [ $? -ne 0 ]; then

This seems to allow compilation on hawk and on mark, and compilation still works on OS X Lion and sage.math.

comment:324 Changed 7 years ago by jdemeyer

Normally such things should be added to $CPPFLAGS, not $CFLAGS.

comment:325 Changed 7 years ago by jhpalmieri

  • Status changed from needs_work to needs_review

I propose using the spkg here, which is identical to Keshav's except for this change:

  • spkg-install

    old new  
    1212[ -z "$LIBRARY_PATH" ] || LIBRARY_PATH="$LIBRARY_PATH":
    1313export CPATH="$CPATH""$SAGE_LOCAL"/include
    1414export LIBRARY_PATH="$LIBRARY_PATH""$SAGE_LOCAL"/lib
    15 
     15# Fix a build error on (Open)Solaris:
     16export CPPFLAGS="$CPPFLAGS -D_XOPEN_SOURCE=500"
    1617
    1718easy_install --allow-hosts=None Twisted-12.1.0.tar.bz2
    1819if [ $? -ne 0 ]; then

(It's a little frustrating that spkg-install is not under revision control, but this is fixed in #13121.) I think this should be easy to review positively, since the only problem in the earlier spkg was its failure to build on (Open)Solaris. Then we can move on to #13121, which we should try to get into Sage 5.2 as well.

I have not changed the link in the ticket description to point to this new spkg, but if this sounds like a good approach, please go ahead and do that.

comment:326 Changed 7 years ago by kini

Sounds good to me, except I'd call it 0.9.0, not 0.9.0.p0. Since no 0.9.0 has gone up on the Sage mirrors yet, there's no need for a patch version yet - and a patch version should technically have changes to SPKG.txt too, but as I said before, I want to avoid changing SPKG.txt since it's already been rewritten for #13121.

So I've put your SPKG at the URL in the ticket description, renaming it sagenb-0.9.0.spkg and renaming the directory inside it to sagenb-0.9.0.

comment:327 Changed 7 years ago by jhpalmieri

It's fine to incorporate my change and still call it 0.9.0; while my change was still pending, I just wanted to distinguish mine from the one you posted earlier.

Jeroen: positive review now?

comment:328 Changed 7 years ago by kcrisman

  • Authors changed from Mike Hansen, Rado Kirov, William Stein, Jason Grout to Mike Hansen, Rado Kirov, William Stein, Jason Grout, Jeroen Demeyer
  • Reviewers changed from Rado Kirov, Dan Drake, Jason Grout, Simon King, Dmitrii Pasechnik to Rado Kirov, Dan Drake, Jason Grout, Simon King, Dmitrii Pasechnik, John Palmieri, Punarbasu Purkayastha

Since jdemeyer said:

On hawk, adding the compiler flag -D_XOPEN_SOURCE=500 fixes the problem. Normally such things should be added to $CPPFLAGS, not $CFLAGS.

so the two changes are from him, and jhpalmieri said:

This seems to allow compilation on hawk and on mark, and compilation still works on OS X Lion and sage.math.

so the review of that is from him, I don't quite understand why that isn't enough for positive review. I assume that the CPPFLAGS version was also actually tested, correct?

The current version of the spkg has this in it, so this sounds good enough.

Of course there is this in the SPKG.txt, and presumably other misleading stuff. Does it matter?

== Changelog ==
     
 * For an accurate log of changes, run "hg log" in src/sagenb.

I wonder if we'll reach 300 comments before this ticket is merged...

I guess so, Jason :)

comment:329 Changed 7 years ago by jdemeyer

  • Status changed from needs_review to positive_review

Note that the OpenSSL spkg at #13126 still needs review.

comment:330 follow-up: Changed 7 years ago by kini

kcrisman: the line from SPKG.txt which you quoted remains true for the SPKG on this ticket, which does contain a mercurial repository.

Last edited 7 years ago by kini (previous) (diff)

comment:331 in reply to: ↑ 330 Changed 7 years ago by kcrisman

kcrisman: the line from SPKG which you quoted remains true for the SPKG on this ticket, which does contain a mercurial repository.

I misread it. The line I quote indeed is correct, but there is no top-level hg repository, unlike every other (well, every other recently updated) spkg.

$ cd Downloads/sagenb-0.9.0/
$ hg st
abort: There is no Mercurial repository here (.hg not found)!
$ ls -a
.		..		SPKG.txt	spkg-install	src

I see, these things are actually under revision control in src/sagenb as well. Sorry for the noise. Great work, all.

Changed 7 years ago by jdemeyer

apply to $SAGE_LOCAL/bin

comment:332 Changed 7 years ago by jdemeyer

  • Merged in set to sage-5.2.beta0
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:333 Changed 7 years ago by ddrake

...and there was much rejoicing.

comment:334 Changed 7 years ago by jdemeyer

  • Merged in sage-5.2.beta0 deleted
  • Milestone changed from sage-5.2 to sage-pending
  • Resolution fixed deleted
  • Status changed from closed to new

Unmerging this from sage-5.2 due to the serious security issue at #13270.

comment:335 Changed 7 years ago by jdemeyer

  • Status changed from new to needs_review

comment:336 Changed 7 years ago by jdemeyer

  • Dependencies changed from #13126, #11078, #11874, #12229, #11503, #12327, #12899 to #13126, #11078, #11874, #12229, #11503, #12327, #12899, #13270, to be merged with #13121
  • Status changed from needs_review to positive_review

comment:337 follow-up: Changed 7 years ago by ppurka

Can you confirm if the patch above does the job? Then we don't need to prolong the release of 5.2. You can apply it to devel/sagenb by the following command

....beta1.11080/devel/sagenb» patch --dry-run -p1 -i /path/to/trac_11080-user_registration.patch
patching file sagenb/notebook/run_notebook.py
....beta1.11080/devel/sagenb» patch -p1 -i /path/to/trac_11080-user_registration.patch
patching file sagenb/notebook/run_notebook.py

If the first command returns success, then proceed with the second command.

EDIT: Taken from git diff 0.9.0 0.10.1 in the sagenb directory. I haven't checked exactly which commit introduced this change.

Last edited 7 years ago by ppurka (previous) (diff)

Changed 7 years ago by ppurka

(temporary fix) apply to devel/sagenb

comment:338 follow-up: Changed 7 years ago by ppurka

Hmm.. the above was not complete. Typing <url>/register would still go to the registration. The new patch disables user registration. (Again, also taken from the git diff).

comment:339 in reply to: ↑ 337 Changed 7 years ago by dimpase

Replying to ppurka:

Can you confirm if the patch above does the job? Then we don't need to prolong the release of 5.2. You can apply it to devel/sagenb by the following command

....beta1.11080/devel/sagenb» patch --dry-run -p1 -i /path/to/trac_11080-user_registration.patch
patching file sagenb/notebook/run_notebook.py
....beta1.11080/devel/sagenb» patch -p1 -i /path/to/trac_11080-user_registration.patch
patching file sagenb/notebook/run_notebook.py

If the first command returns success, then proceed with the second command.

yes, it fixes it. I was actually pondering the same place here:

Last edited 7 years ago by dimpase (previous) (diff)

comment:340 in reply to: ↑ 338 Changed 7 years ago by dimpase

Last edited 7 years ago by dimpase (previous) (diff)

comment:341 follow-up: Changed 7 years ago by jason

Manage Users is an admin feature. It should *always* work, regardless of the accounts option. Normal users should NOT have the Manage Users page.

Last edited 7 years ago by jason (previous) (diff)

comment:342 in reply to: ↑ 341 ; follow-up: Changed 7 years ago by dimpase

Replying to jason:

Manage Users is an admin feature. It should *always* work, regardless of the accounts option. Normal users should NOT have the Manage Users page.

oh, OK. I'll test this again more carefully, under different users.

comment:343 in reply to: ↑ 342 Changed 7 years ago by dimpase

Replying to dimpase:

Replying to jason:

Manage Users is an admin feature. It should *always* work, regardless of the accounts option. Normal users should NOT have the Manage Users page.

oh, OK. I'll test this again more carefully, under different users.

YES, the patch works. Let's beg Jeroen to remerge the stuff...

comment:344 Changed 7 years ago by ppurka

Thanks for confirming. I have the patch in a new branch branch0.9 at github that is created from 0.9.0.

@kini: I sent you a pull request. But if you apply it, then apply to a separate branch, created from 0.9.0.

comment:345 Changed 7 years ago by jdemeyer

Please discuss #13270 on that ticket, not here.

comment:346 Changed 7 years ago by jdemeyer

  • Dependencies changed from #13126, #11078, #11874, #12229, #11503, #12327, #12899, #13270, to be merged with #13121 to #13126, #11078, #11874, #12229, #11503, #12327, #12899, #13270

comment:347 Changed 7 years ago by jdemeyer

  • Merged in set to sage-5.2.beta0
  • Milestone changed from sage-pending to sage-5.2
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:348 Changed 3 years ago by chapoton

  • Authors changed from Mike Hansen, Rado Kirov, William Stein, Jason Grout, Jeroen Demeyer to Mike Hansen, Radoslav Kirov, William Stein, Jason Grout, Jeroen Demeyer
  • Reviewers changed from Rado Kirov, Dan Drake, Jason Grout, Simon King, Dmitrii Pasechnik, John Palmieri, Punarbasu Purkayastha to Radoslav Kirov, Dan Drake, Jason Grout, Simon King, Dmitrii Pasechnik, John Palmieri, Punarbasu Purkayastha
Note: See TracTickets for help on using tickets.