Opened 11 months ago

Last modified 9 months ago

#25667 needs_info defect

Keep twisted reference in all.py

Reported by: kcrisman Owned by:
Priority: critical Milestone: sage-8.4
Component: notebook Keywords:
Cc: dimpase, chapoton, vbraun, jdemeyer Merged in:
Authors: Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description

Currently (as of Sage 8.3.beta6), for some reason #25548 keeps the various Sage notebooks from behaving as needed. We should revert the src/sage/all.py part until we get that figured out.

Details: sage -n or other launch of Sage notebook currently brings up the "SageNBExport" notebook, which allows conversion of sagenb worksheets to Jupyter. One option on that page is to simply go to the old sagenb notebook; however, with #25548, that fails to actually open a browser window with the sagenb home page. I'm not sure whether it is an authentication/token error or something else, but in any case it does not work even though the terminal indicates things are working normally behind the scenes.

Given that even if/when we switch to Jupyter for all notebook default people will still want the export functionality available, this should definitely be fixed.

Change History (25)

comment:1 Changed 11 months ago by kcrisman

  • Cc dimpase chapoton vbraun jdemeyer added

comment:2 Changed 11 months ago by dimpase

How about just removing that option on that page is to simply go to the old sagenb notebook? An explanation how one should instead explicitly start sagenb is already there.

comment:3 Changed 11 months ago by jdemeyer

I cannot reproduce this. Can you state more precisely what the problem is? What happens on the console and what happens in the browser?

Last edited 11 months ago by jdemeyer (previous) (diff)

comment:4 Changed 11 months ago by kcrisman

$ ./sage -n
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 8.3.beta6, Release Date: 2018-06-17               │
│ Type "notebook()" for the browser-based notebook interface.        │
│ Type "help()" for help.                                            │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable.     ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
Please wait while the SageNB export server starts...
[I 22:20:36.409 NotebookApp] Using MathJax: nbextensions/mathjax/MathJax.js
[I 22:20:51.609 NotebookApp] Serving notebooks from local directory: /Users/karl.crisman/Downloads/sage
[I 22:20:51.609 NotebookApp] 0 active kernels
[I 22:20:51.609 NotebookApp] The Jupyter Notebook is running at:
[I 22:20:51.609 NotebookApp] http://localhost:8888/?token=3c0b141dc2e03e462e6a4f1ee14820805e946ecd36eea489
[I 22:20:51.610 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 22:20:51.613 NotebookApp] 
    
    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://localhost:8888/?token=3c0b141dc2e03e462e6a4f1ee14820805e946ecd36eea489
[I 22:20:52.449 NotebookApp] Accepting one-time-token-authenticated connection from ::1

At this point

http://localhost:8888/sagenb?token=50867943df437ff4c97ae71d01bfed7a9f67450553b1f2f3

opens up in the browser, which is the exporter. Clicking "Run the old Sage notebook" goes to

http://localhost:8888/sagenb/www/run-sagenb.html

in the same window, and runs in terminal

('The notebook files are stored in:', 'sage_notebook.sagenb')
┌────────────────────────────────────────────────┐
│                                                │
│ Open your web browser to http://localhost:8080 │
│                                                │
└────────────────────────────────────────────────┘
Executing twistd  --pidfile="sage_notebook.sagenb/sagenb.pid" -ny "sage_notebook.sagenb/twistedconf.tac"
/Users/.../sage/local/lib/python2.7/site-packages/Crypto/Util/number.py:57: PowmInsecureWarning: Not using mpz_powm_sec.  You should rebuild using libgmp >= 5 to avoid timing attack vulnerability.
  _warn("Not using mpz_powm_sec.  You should rebuild using libgmp >= 5 to avoid timing attack vulnerability.", PowmInsecureWarning)
2018-06-27T22:22:16-0400 [-] Loading sage_notebook.sagenb/twistedconf.tac...
2018-06-27T22:22:23-0400 [-] Loaded.
2018-06-27T22:22:23-0400 [twisted.scripts._twistd_unix.UnixAppLogger#info] twistd 16.3.0 (/Users/.../sage/local/bin/python2 2.7.15) starting up.
2018-06-27T22:22:23-0400 [twisted.scripts._twistd_unix.UnixAppLogger#info] reactor class: twisted.internet.selectreactor.SelectReactor.
2018-06-27T22:22:23-0400 [-] QuietSite starting on 8080
2018-06-27T22:22:23-0400 [__builtin__.QuietSite#info] Starting factory <__builtin__.QuietSite instance at 0x1be307a70>

Note that

http://localhost:8080

does show the login screen if one opens a new tab and types it in. But ... there is no sign of the auto-login for sagenb despite the message "Please wait a few seconds..." But there should be such auto-login, and there was, and reverting that part of #25548 brings back this (correct) behavior.


For the sake of argument, if one did remove that option as suggested in comment:2 one could give very explicit instructions as currently at http://localhost:8888/sagenb/www/run-sagenb.html but I don't know that I would recommend that yet because it is the only way to access sagenb "for dummies" at this point outside the Mac app.

comment:5 Changed 11 months ago by jdemeyer

Again to clarify things, can you post the branch which fixes that problem for you? And please double-check that it doesn't work without your patch and that it works with your patch (making sure to give sagenb enough time to actually start up, the "few seconds" may be optimistic).

comment:6 Changed 11 months ago by chapoton

This seems to work again for me with 8.3.beta8, at least on one machine.

Please double check the failure with 8.3.beta8.

comment:7 Changed 11 months ago by chapoton

  • Status changed from new to needs_info

comment:8 Changed 11 months ago by kcrisman

beta8 did not work either. Reverting

-    # stop the twisted reactor
-    try:
-       from twisted.internet import reactor
-       if reactor.running:
-          reactor.callFromThread(reactor.stop)
-    except ImportError:
-       pass
-

once again fixes this, with almost instantaneous launching of sagenb once it gets going in the terminal. Based off of beta8. Yes, a few seconds is very optimistic normally, but five minutes is a bit excessive even by my patience standards :-)

Now what is particularly weird is that when I cleaned things up by reverting the reverting (i.e. going back to develop), I can no longer recreate this. So for now I propose we leave it as "needs info", because it could be even weirder than this. chapoton, any ideas on why/why not beta8 would make a difference?

I will endeavor to check this with each new beta now, though my Sage development time has been very limited for a couple years now, regrettably.

comment:9 Changed 11 months ago by jdemeyer

I am pretty sure that whatever problem you are seeing is unrelated to twisted. Especially since you cannot reproduce it after reverting.

comment:10 Changed 11 months ago by chapoton

I still meet the issue with my latest beta (on another machine).

Could be a pickle problem:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/chapoton/sage/local/lib/python2.7/site-packages/sagenb/notebook/notebook_object.py", line 243, in __call__
    return self.notebook(*args, **kwds)
  File "/home/chapoton/sage/local/lib/python2.7/site-packages/sagenb/notebook/run_notebook.py", line 535, in notebook_run
    nb = notebook.load_notebook(directory)
  File "/home/chapoton/sage/local/lib/python2.7/site-packages/sagenb/notebook/notebook.py", line 1839, in load_notebook
    nb = Notebook(dir)
  File "/home/chapoton/sage/local/lib/python2.7/site-packages/sagenb/notebook/notebook.py", line 151, in __init__
    self.__conf = S.load_server_conf()
  File "/home/chapoton/sage/local/lib/python2.7/site-packages/sagenb/storage/filesystem_storage.py", line 265, in load_server_conf
    return self._basic_to_server_conf(self._load('conf.pickle'))
  File "/home/chapoton/sage/local/lib/python2.7/site-packages/sagenb/storage/filesystem_storage.py", line 187, in _load
    result = pickle.load(f)
ValueError: unsupported pickle protocol: 3
[E 16:22:14.760 NotebookApp] {
      "Accept-Language": "fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7", 
      "Accept-Encoding": "gzip, deflate, br", 
      "Host": "localhost:8888", 
      "Accept": "*/*", 
      "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36", 
      "Connection": "keep-alive", 
      "Referer": "http://localhost:8888/sagenb/www/run-sagenb.html", 
      "Cookie": "_xsrf=2|0bf3e54e|28cdb4e20016c6f043f8849cfe61c13b|1530264556; username-localhost-8888=\"2|1:0|10:1530541329|23:username-localhost-8888|44:ZWMyNWJhOTE0YmVkNDgwM2JkMzczOWMzMDZlZGM0OWI=|90c5ea569b6ba437b5dcea7b21ad80bb83380a15fd3ebbfeec6408f6347a62ce\""
    }
[E 16:22:14.761 NotebookApp] 500 GET /sagenb/start (127.0.0.1) 2227.82ms referer=http://localhost:8888/sagenb/www/run-sagenb.html
Last edited 11 months ago by chapoton (previous) (diff)

comment:11 Changed 11 months ago by jdemeyer

Did you by chance run sagenb under Python 3? That might explain that.

comment:12 follow-up: Changed 11 months ago by chapoton

No, I am using my python2 install. And now

./sage -n=sagenb

fails with the same pickle problem.

BUT I have tried to use sagenb with my other sage3 install on the same machine. Could they interfere ?

comment:13 in reply to: ↑ 12 ; follow-up: Changed 11 months ago by jdemeyer

Replying to chapoton:

Could they interfere ?

Yes.

comment:14 Changed 11 months ago by kcrisman

  • Priority changed from blocker to critical

I don't see a pickling message, so I would say that least is unrelated.

I am pretty sure that whatever problem you are seeing is unrelated to twisted

Yet this is the first time that changing that particular stretch of code didn't reliably go back and forth on this issue. So I'm still not totally convinced - let's leave this open with "needs info" for now, though I will downgrade to "critical" for the time being.

comment:15 in reply to: ↑ 13 Changed 11 months ago by dimpase

Replying to jdemeyer:

Replying to chapoton:

Could they interfere ?

Yes.

I imagine stuff in ~/.sage/ might not be compatible, py2/3-wise.

comment:16 Changed 11 months ago by kcrisman

After upgrading to 8.3.rc0 I had this same problem; running sage --notebook=sagenb once caused it to disappear. I wonder if anyone can try to confirm that - maybe there is something going on there, race condition? Weird.

comment:17 Changed 11 months ago by chapoton

Did you try to move your .sage out of the way ?

comment:18 Changed 11 months ago by kcrisman

No, though I guess I should for the next release candidate. I don't know that this would make a big difference since I don't really have a lot of custom stuff; I'm not using py3.

comment:19 follow-up: Changed 11 months ago by kcrisman

Once again I have the behavior with rc0:

  1. Start with sage -n
  2. Click to get old notebook from the export page
  3. Goes to waiting page but never loads despite the sagenb server functioning normally
  4. Quit
  5. Do first two steps again
  6. All is well

With rc1, assuming there is one before I have worse internet for several weeks, I may try to move the .sage directory. This is weird.

comment:20 in reply to: ↑ 19 Changed 11 months ago by kcrisman

Replying to kcrisman:

Once again I have the behavior with rc0:

  1. Start with sage -n
  2. Click to get old notebook from the export page
  3. Goes to waiting page but never loads despite the sagenb server functioning normally
  4. Quit
  5. Do first two steps again
  6. All is well

Sorry, I don't know why I wrote this. The last item does NOT happen. Now I will try moving .sage. Now I get the following - still no loading, and when I do Ctrl-C:

^C2018-07-12T11:32:21-0400 [stdout#info] Quitting all running worksheets...
2018-07-12T11:32:21-0400 [stdout#info] Saving notebook...
[I 11:32:21.731 NotebookApp] interrupted
Serving notebooks from local directory: /Users/karl.crisman/Downloads/binary-pkg
0 active kernels
The Jupyter Notebook is running at:
http://localhost:8888/?token=b221b7e836980e807316ab78230331224f4273ad8cc6e9ea
Shutdown this notebook server (y/[n])? 2018-07-12T11:32:21-0400 [stdout#info] Notebook cleanly saved.
2018-07-12T11:32:21-0400 [-] (TCP Port 8080 Closed)
2018-07-12T11:32:21-0400 [__builtin__.QuietSite#info] Stopping factory <__builtin__.QuietSite instance at 0x1c6ff86c8>
2018-07-12T11:32:21-0400 [-] Main loop terminated.
2018-07-12T11:32:21-0400 [twisted.scripts._twistd_unix.UnixAppLogger#info] Server Shut Down.
No answer for 5s: resuming operation...
^C[I 11:32:39.728 NotebookApp] interrupted
Serving notebooks from local directory: /Users/karl.crisman/Downloads/binary-pkg
0 active kernels
The Jupyter Notebook is running at:
http://localhost:8888/?token=b221b7e836980e807316ab78230331224f4273ad8cc6e9ea
Shutdown this notebook server (y/[n])? No answer for 5s: resuming operation...
^C[I 11:32:51.523 NotebookApp] interrupted
Serving notebooks from local directory: /Users/karl.crisman/Downloads/binary-pkg
0 active kernels
The Jupyter Notebook is running at:
http://localhost:8888/?token=b221b7e836980e807316ab78230331224f4273ad8cc6e9ea
Shutdown this notebook server (y/[n])? y
[C 11:32:53.871 NotebookApp] Shutdown confirmed
[I 11:32:53.874 NotebookApp] Shutting down 0 kernels

Note that one has to explicitly shut down the NotebookApp server by pressing y!!! What on earth? Has anyone else ever seen that?

comment:21 Changed 11 months ago by kcrisman

And once again, using sagenb directly and then going through the steps above ends with success. So this is irrespective of .sage.

comment:22 Changed 11 months ago by kcrisman

Interestingly, when I moved .sage back to its original location, the same behavior 1.-6. came again, so there is something weird going on.

comment:23 Changed 10 months ago by vdelecroix

  • Milestone changed from sage-8.3 to sage-8.4

milestone update 8.3 -> 8.4

comment:24 Changed 9 months ago by jdemeyer

Can this problem be reproduced? If not, we should close this ticket. If yes, the description and title should be updated.

comment:25 Changed 9 months ago by kcrisman

  • Summary changed from Revert full removal of twisted to Keep twisted reference in all.py

I reproduced it most of the time, but I am only one person. I think we can keep the ticket open, and if more reports surface with 8.4 now released without this fix, we can do it quite easily. "Needs Info" for more reports, then.

Note: See TracTickets for help on using tickets.