Opened 7 years ago

Closed 6 years ago

# Add SageNB exporter, Jupyter extension

Reported by: Owned by: Volker Braun major sage-7.0 packages: standard Karl-Dieter Crisman, Vincent Delecroix Volker Braun Nicolas M. Thiéry, Eric Gourgoulhon, Benjamin Ragan-Kelley, Karl-Dieter Crisman N/A a89993a

Packaging of https://github.com/vbraun/ExportSageNB/releases, tarball is on the mirrors

sage --notebook=export


that runs the Jupyter nbextension for the export

Usage examples:

• List available legacy Sage notebooks
sage --notebook=export --list
Unique ID       | Notebook Name
-------------------------------------------------------------------------------
admin:10        | Oxford Seminar (1,1)-Calabi Yau

• Export a specific legacy Sage notebook as a Jupyter notebook
sage --notebook=export --ipynb=Output.ipynb admin:10

• Open the Jupyter notebook
sage --notebook=jupyter Output.ipynb


See #20316 for making it more user-friendly.

### comment:2 Changed 7 years ago by Volker Braun

Authors: → Volker Braun → 396488e332ef7ebc22f80a433c5fc3c047779ef6 PLEASE CHANGE → packages: standard modified (diff) new → needs_review PLEASE CHANGE → enhancement

New commits:

 ​3cbd2e6 Add the sagenb export package ​396488e Add new "sage --notebook=export" option to start the SageNB exporter

### comment:3 Changed 7 years ago by git

Commit: 396488e332ef7ebc22f80a433c5fc3c047779ef6 → 25e7fcf912c69ed994376125b3b7ddb4a33b9897

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

 ​a9d7911 Add the sagenb export package ​4705615 Add new "sage --notebook=export" option to start the SageNB exporter ​25e7fcf Add Jupyter notebook PR to obey default_url

### comment:5 Changed 7 years ago by Jeroen Demeyer

Status: needs_review → needs_work

Is the notebook patch included in the latest notebook upgrade? If yes, it should be removed. If no, then this branch here should bump the version to force a re-install of notebook.

### comment:6 Changed 7 years ago by Jeroen Demeyer

Attempting to download package sagenb_export-1.0.tar.gz from mirrors
http://www-ftp.lip6.fr/pub/math/sagemath/spkg/upstream/sagenb_export/sagenb_export-1.0.tar.gz
[......................................................................]
************************************************************************
Traceback (most recent call last):
File "/home/jdemeyer/sage/build/bin/../sage_bootstrap/cmdline.py", line 228, in run
raise ChecksumError('checksum does not match')
ChecksumError: checksum does not match
************************************************************************
************************************************************************


### comment:7 Changed 7 years ago by Volker Braun

The patch is going to be in notebook-4.2.0 which is 3h old now; In short: no

### comment:8 Changed 7 years ago by git

Commit: 25e7fcf912c69ed994376125b3b7ddb4a33b9897 → d9ce648fdb873766242a9b1bb96662331b88cac9

Branch pushed to git repo; I updated commit sha1. New commits:

 ​d9ce648 Update tarball

### comment:9 Changed 7 years ago by git

Commit: d9ce648fdb873766242a9b1bb96662331b88cac9 → a4a0b8f16ca768364c2930688e5796a5b902b40f

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

 ​133a506 Add the sagenb export package ​6a9b407 Add new "sage --notebook=export" option to start the SageNB exporter ​91e066f Add Jupyter notebook PR to obey default_url ​a4a0b8f Update tarball

### comment:10 Changed 7 years ago by git

Commit: a4a0b8f16ca768364c2930688e5796a5b902b40f → ce071e56f7263b80900dae00a5fc2ce7ee07a34d

Branch pushed to git repo; I updated commit sha1. New commits:

 ​ce071e5 New-style dependencies

### comment:11 Changed 7 years ago by Volker Braun

Status: needs_work → needs_review

Tarball is now correct (mirrors might have to catch up)

### comment:12 Changed 7 years ago by Nicolas M. Thiéry

Description: modified (diff)

### comment:13 Changed 7 years ago by Nicolas M. Thiéry

Description: modified (diff)

### comment:14 Changed 7 years ago by Jeroen Demeyer

Description: modified (diff)

### comment:15 follow-ups:  16  29 Changed 7 years ago by Eric Gourgoulhon

I've installed the package atop sage 7.3.beta2 on Ubuntu 14.04 LTS with

git pull trac u/vbraun/add_sagenb_exporter__jupyter_extension
make


Everything seemed OK, but when I run

./sage --notebook=export --list


I get the error message

[C 14:58:52.386 NotebookApp] Bad config encountered during initialization:
[C 14:58:52.386 NotebookApp] Unrecognized flag: '--list'


Maybe unrelated, but running the command without the flad --list (i.e. just ./sage --notebook=export) yields

[W 14:59:48.618 NotebookApp] Error loading server extension sagenb_export.nbextension
Traceback (most recent call last):
File "/home/eric/sage-git/beta8/local/lib/python2.7/site-packages/notebook/notebookapp.py", line 1042, in init_server_extensions
mod = importlib.import_module(modulename)
File "/home/eric/sage-git/beta8/local/lib/python/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/home/eric/sage-git/beta8/local/lib/python2.7/site-packages/sagenb_export-1.0-py2.7.egg/sagenb_export/nbextension/__init__.py", line 6, in <module>
from sagenb_export.nbextension.list_handler import ListSageNBHandler
File "/home/eric/sage-git/beta8/local/lib/python2.7/site-packages/sagenb_export-1.0-py2.7.egg/sagenb_export/nbextension/list_handler.py", line 7, in <module>
File "/home/eric/sage-git/beta8/local/lib/python2.7/site-packages/sagenb_export-1.0-py2.7.egg/sagenb_export/sagenb_reader.py", line 7, in <module>
from sagenb_export.unescape import unescape
File "/home/eric/sage-git/beta8/local/lib/python2.7/site-packages/sagenb_export-1.0-py2.7.egg/sagenb_export/unescape.py", line 16, in <module>
unescape = html.unescape
AttributeError: 'module' object has no attribute 'unescape'


### comment:16 in reply to:  15 Changed 6 years ago by Eric Gourgoulhon

I've installed the package atop sage 7.3.beta2 on Ubuntu 14.04 LTS with

git pull trac u/vbraun/add_sagenb_exporter__jupyter_extension
make


Everything seemed OK, but when I run

./sage --notebook=export --list


I get the error message

[C 14:58:52.386 NotebookApp] Bad config encountered during initialization:
[C 14:58:52.386 NotebookApp] Unrecognized flag: '--list'


Same issue after the install in a fresh 7.3.beta3 on Ubuntu 15.10.

### comment:17 Changed 6 years ago by Eric Gourgoulhon

Status: needs_review → needs_info

### comment:18 Changed 6 years ago by Nicolas M. Thiéry

Hi Volker,

I just gave it a shot with Min. Thanks for your work on this!

I have been using a lot the live documentation, and apparently this left out a whole lot of fake notebooks:

Traceback (most recent call last):
...
File "/Users/.../sage/local/lib/python2.7/site-packages/sagenb_export/actions.py", line 9, in tr
print(u'{0:<15} | {1}'.format(unique_id, name))
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in position 33: ordinal not in range(128)


I'm not sure why this would differ.

### comment:33 Changed 6 years ago by git

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

 ​b562b21 Add the sagenb export package ​2b844dd Add new "sage --notebook=export" option to start the SageNB exporter ​c0e14ea Update tarball ​1177d3e New-style dependencies ​191f5e1 Pass sage --notebook=export ... commandline options to sagenb-export

### comment:34 in reply to:  31 Changed 6 years ago by Volker Braun

Question to vbraun: is it possible to save the notebook (with some syntax, perhaps) in a different location than the cwd?

You can specify any path sage --notebook=export --ipynb=/path/to/Output.ipynb admin:10

### comment:35 in reply to:  25 Changed 6 years ago by Volker Braun

Thanks for trying this out, Nicolas! You are motivating me to do so this week too. Question before I start: did this still keep all the notebooks from sagenb in the same state? (That is, can you still use sage -n with the "original" notebook and nothing has changed?)

Yes, the old sagenb notebooks are only read and never written to.

### comment:36 in reply to:  22 ; follow-up:  43 Changed 6 years ago by Volker Braun

Minor point: in some of the worksheets, the outputs appear litteraly as:

<html><span class="math">\newcommand{\Bold}[1]{\mathbf{#1}}2</span></html>


Thats SageNB works, "typeset" output is just plain text output that happens to start with <html>, the SageNB format does not have distinguished output types.

### comment:37 Changed 6 years ago by Volker Braun

Status: needs_work → needs_review

I fixed the command line handling, should be good now.

### comment:38 Changed 6 years ago by Eric Gourgoulhon

I've pulled the latest commit and compiled it (by make). There is no error in the build, but both

./sage --notebook=export --list


and

./sage -sh
sagenb-export --list


result in the same error:

Traceback (most recent call last):
File "/home/eric/sage-git/beta8/local/bin/sagenb-export", line 9, in <module>
File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 542, in load_entry_point
File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 2569, in load_entry_point
File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 2229, in load
File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 2235, in resolve
File "/home/eric/sage-git/beta8/local/lib/python2.7/site-packages/sagenb_export-1.0-py2.7.egg/sagenb_export/cmdline.py", line 12, in <module>
File "/home/eric/sage-git/beta8/local/lib/python2.7/site-packages/sagenb_export-1.0-py2.7.egg/sagenb_export/sagenb_reader.py", line 7, in <module>
from sagenb_export.unescape import unescape
File "/home/eric/sage-git/beta8/local/lib/python2.7/site-packages/sagenb_export-1.0-py2.7.egg/sagenb_export/unescape.py", line 16, in <module>
unescape = html.unescape
AttributeError: 'module' object has no attribute 'unescape'


### comment:39 Changed 6 years ago by git

Branch pushed to git repo; I updated commit sha1. New commits:

 ​2526f08 Update to sagenb_export-2.0

### comment:40 follow-up:  41 Changed 6 years ago by Volker Braun

Yes that was fixed last week though not in the packaged tarball, I made a new release. Should work now.

### comment:41 in reply to:  40 ; follow-up:  42 Changed 6 years ago by Eric Gourgoulhon

Yes that was fixed last week though not in the packaged tarball, I made a new release. Should work now.

Indeed ./sage --notebook=export --list works now. Thanks! I will test the export on my notebooks and will report here.

### comment:42 in reply to:  41 Changed 6 years ago by Eric Gourgoulhon

I've tested the export on various notebooks (some of them quite long and rich in text). It works very well. Thank you very much for this very desired tool! I've only two minor comments:

• the displayed mathematical formulas typed between $and$ in text cells of sws notebooks are not properly rendered in the corresponding Markdown cells of jupyter notebooks: the delimiters $and$ are interpreted as ordinary brackets and not as LaTeX markers; they have to be replaced manually by  for the displayed math to appear correctly.
• for sws notebooks with the "Typeset" button turned on, it would be nice to have the jupyter notebook start by a code cell with
%display latex

so that all outputs are actually typeset, as requested in the original notebook.

### comment:43 in reply to:  36 Changed 6 years ago by Nicolas M. Thiéry

Minor point: in some of the worksheets, the outputs appear litteraly as:

<html><span class="math">\newcommand{\Bold}[1]{\mathbf{#1}}2</span></html>


Thats SageNB works, "typeset" output is just plain text output that happens to start with <html>, the SageNB format does not have distinguished output types.

Ok. No big deal, since anyway it's just about the output which can be regenerated. Still it could be an easy addition in case we could easily circumvent the lack of metadata. Would it be simple and reasonably robust to check whether the output text starts with <html>?

### comment:44 follow-up:  45 Changed 6 years ago by Karl-Dieter Crisman

Various comments. Some are specific to this and others probably are more about whether Jupyter should be default without some work. I appreciate any comments more experienced Jupyter folks can make.

In general I think this branch is a good tool for those intent upon converting (perhaps except the DATA issue). The fairly different nature of the Jupyter notebook (among other things) will note necessarily make it optimal to just convert all worksheets for many people (as opposed to new users who may wish to learn the different interaction from the start), though.

Most important: When one has a notebook that had data, one gets

NameError: name 'DATA' is not defined


I have no idea how to fix this, or if Jupyter even supports something like a data folder.

Help link is good, but unfortunately it doesn't have a local copy of some of the help files (e.g., the "Notebook help" gives this link). Since a lot of people (presumably still) occasionally need offline notebooks, that is unfortunate.

for sws notebooks with the "Typeset" button turned on, it would be nice to have the jupyter notebook start by a code cell with %display latex so that all outputs are actually typeset, as requested in the original notebook.

Or even that html output would be recognized as such... that's probably too much to ask and sed through.

No %auto - just mentioning, not a killer.

Is there a way to close a notebook without turning off the server? I didn't see something under "File".

Doing a basic interact:

NameError: name 'slider' is not defined


I can't figure out how to do interacts at all, in fact. Even

@interact
def _(n=3):
print n^2


gives just some html, which I guess makes sense since that is what it does in sagenb.

Silly but necessary requests:

./sage -h


still gives

  --notebook=[...]    -- start the Sage notebook (valid options are
'default', 'sagenb', and 'jupyter')

 ./sage -n -h


could use an example syntax and where to look for full syntax for export

When I make this change

%display latex
for n in range(1,20):
(n,sigma(n,2))


it doesn't typeset. Not sure if this is just different.

### comment:45 in reply to:  44 ; follow-up:  46 Changed 6 years ago by Eric Gourgoulhon

Is there a way to close a notebook without turning off the server? I didn't see something under "File".

There is "Close and halt" as the last item of the "File" menu (under sage 7.3.beta6).

When I make this change

%display latex
for n in range(1,20):
(n,sigma(n,2))


it doesn't typeset. Not sure if this is just different.

Even without %display latex, there is no output. To have some output in the notebook, one has to type

for n in range(1,20):
print((n,sigma(n,2)))


to get text output, and

for n in range(1,20):
show((n,sigma(n,2)))


to get displayed output. Note that one can replace show by pretty_print.

### comment:46 in reply to:  45 ; follow-up:  47 Changed 6 years ago by Karl-Dieter Crisman

Is there a way to close a notebook without turning off the server? I didn't see something under "File".

There is "Close and halt" as the last item of the "File" menu (under sage 7.3.beta6).

Yes, I used that, but what does "halt" mean? I thought that meant something more drastic than it does...

When I make this change

%display latex
for n in range(1,20):
(n,sigma(n,2))


it doesn't typeset. Not sure if this is just different.

Okay, that means the behavior here is different than in sagenb (or at least in some version of sagenb).

### comment:47 in reply to:  46 Changed 6 years ago by Eric Gourgoulhon

Is there a way to close a notebook without turning off the server? I didn't see something under "File".

There is "Close and halt" as the last item of the "File" menu (under sage 7.3.beta6).

Yes, I used that, but what does "halt" mean? I thought that meant something more drastic than it does...

"halt" simply means "halt the kernel", not the server. I agree this is not very explicit...

When I make this change

%display latex
for n in range(1,20):
(n,sigma(n,2))


it doesn't typeset. Not sure if this is just different.

Okay, that means the behavior here is different than in sagenb (or at least in some version of sagenb).

Yes indeed.

### comment:48 Changed 6 years ago by Karl-Dieter Crisman

Okay, after thinking about it, here is my list of "needs work". Some may disagree, open to various outcomes.

Most important: When one has a notebook that had data, one gets

NameError: name 'DATA' is not defined


I have no idea how to fix this, or if Jupyter even supports something like a data folder.

This really breaks some worksheets that depend on it, so there should be some way to bring the data along for the ride.

I can't figure out how to do interacts at all, in fact. Even

@interact
def _(n=3):
print n^2


gives just some html, which I guess makes sense since that is what it does in sagenb.

But unfortunately it still means they don't work. I thought someone had implemented this with Jupyter? Maybe I am remembering something else unrelated. Anyway (in my view) it sort of defeats the purpose of exporting if there isn't anything equivalent to export to.

Silly but necessary requests:

./sage -h


still gives

  --notebook=[...]    -- start the Sage notebook (valid options are
'default', 'sagenb', and 'jupyter')

 ./sage -n -h


could use an example syntax and where to look for full syntax for export

### comment:49 follow-up:  54 Changed 6 years ago by Volker Braun

Jupyter notebooks are a single file and not an opaque directory full of stuff. This is a fundamental design choice and not going to change. About the only thing we can do is prevent people from writing new SageNB notebooks so their data doesn't end up in an evolutionary dead end.

Jupyter interacts have a slightly different syntax

from ipywidgets import interact, IntSlider
@interact
def _(n=IntSlider(3)):
print n^2


One might write a compatibility layer so that the old SageNB interact syntax still works, thats not something that belongs on this ticket.

### comment:50 Changed 6 years ago by Eric Gourgoulhon

IMHO, it would be nice if the SageNB exporter is shipped with Sage 7.3. It will be useful to many users. The missing features could be postponed to another ticket, except maybe the documentation one (i.e. the export option should appear in the output of ./sage -h). Accordingly, I would suggest to set the ticket to positive review. Do you agree?

### comment:51 Changed 6 years ago by git

Branch pushed to git repo; I updated commit sha1. New commits:

 ​a89993a Mention '--notebook=export' in the sage help

Done

### comment:53 Changed 6 years ago by Vincent Delecroix

(+1 for having it in the next release. however I did not participate in any way to the review)

### comment:54 in reply to:  49 Changed 6 years ago by Karl-Dieter Crisman

Jupyter notebooks are a single file and not an opaque directory full of stuff. This is a fundamental design choice and not going to change. About the only thing we can do is prevent people from writing new SageNB notebooks so their data doesn't end up in an evolutionary dead end.

So in a Jupyter notebook one would include other data in a more 'Pythonic' way? We'd want fairly good documentation on how to do that (as well as the below).

Jupyter interacts have a slightly different syntax

from ipywidgets import interact, IntSlider
@interact
def _(n=IntSlider(3)):
print n^2


One might write a compatibility layer so that the old SageNB interact syntax still works, thats not something that belongs on this ticket.

I see. That's too bad that it's not in the global namespace for Sage Jupyter worksheets, perhaps could be addressed there. Making interacts more difficult to do is a regression (completely irrespective of the whole default or exporting question). As a different question, would Jupyter notebooks be able to support the "extended" syntax that SMC and Sage cell support (e.g. nested interacts)?

Accordingly, I would suggest to set the ticket to positive review. Do you agree?

I think I'm okay with that as long as it's not (yet) the default notebook for continuing users. We need a solution of some kind for the above two issues before that. Please cc: me on any such tickets - once created I think we can set this to positive review, yes.

### comment:55 in reply to:  52 Changed 6 years ago by Eric Gourgoulhon

Status: needs_review → positive_review

Done

Thanks!

Accordingly, I would suggest to set the ticket to positive review. Do you agree?

I think I'm okay with that as long as it's not (yet) the default notebook for continuing users.

OK, very good.

### comment:56 Changed 6 years ago by Volker Braun

Branch: u/vbraun/add_sagenb_exporter__jupyter_extension → a89993a9228ed4960dc1f972b3b195f438bf998a → fixed positive_review → closed

### comment:57 Changed 6 years ago by Karl-Dieter Crisman

Commit: a89993a9228ed4960dc1f972b3b195f438bf998a

Don't forget the followup tickets! :-)

### comment:58 Changed 6 years ago by Jeroen Demeyer

Reviewers: Nicolas Thiéry, Eric Gourgoulhon, Benjamin Ragan-Kelley, Karl-Dieter Crisman → Nicolas M. Thiéry, Eric Gourgoulhon, Benjamin Ragan-Kelley, Karl-Dieter Crisman
Note: See TracTickets for help on using tickets.