Provide a "sage -ipynb2rst" command

[tmonteil]

With jupyter as the default notebook, we need to easily transform an .ipynb notebook back into some source code. This ticket adds a sage -ipynb2rst command.

A custom template is created to write code blocks as:

   sage: # blah
   ....: # blah

and to make mathematics formulas the default role (hence avoiding the clumsy :math: notation everywhere). A postprocess script fixes some issues.

Note that reviewers have to run ./configure before make to let this ticket work in order to copy rst_sage.tpl and postprocess.py into the local/share/sage/ext/nbconvert/ directory.

[tmonteil]

New commits:

​c18f853

#23416 sage -ipynb2rst command.

[vdelecroix]

Using bash 4.4.12 on Archlinux

$ sage -ipynb2rst measure_simulation.ipynb 
/opt/sage/src/bin/sage-ipynb2rst: ligne 39: erreur de syntaxe près du symbole inattendu « ( »
/opt/sage/src/bin/sage-ipynb2rst: ligne 39: `        jupyter nbconvert --to rst --RSTExporter.template_path=[\'$SAGE_EXTCODE/nbconvert/\'] --RSTExporter.template_file='rst_sage.tpl' --stdout ${1} | sed 's/:math://g' > ${2} ( echo -e '\n' ; help ) ;;'

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2ef220d

#23416 : typo.

[tmonteil]

Hmm, i copy/pasted too fast, sorry for the typo, this should work better now.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​1edaf59	Merge branch 'u/tmonteil/provide_a__sage__ipynb2rst__command' of trac.sagemath.org:sage into HEAD
​4ecef2f	#23416 : better template (output correctly displayed)

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d22357e	#23416 : do not start with a blank line
​2af3bfc	#23416 : let nbconvert write file and save images
​aa75e08	#23416 : postprocessing

[tmonteil]

We could always add improvements, but i am fine with the current behavior for a first version.

[slabbe]

Thierry, can you add a doctest ​here in src/sage/tests/cmdline.py please?

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​03e157c	Merge branch 'u/tmonteil/provide_a__sage__ipynb2rst__command' of trac.sagemath.org:sage into HEAD
​0785497	#23416 : command line doctest

[tmonteil]

Done.

[jdemeyer]

Typo: the header that define -> the header that defines

[jdemeyer]

I generally don't like checks of the form

# tests if dependencies are available
pandoc -v > /dev/null 2>&1 || fail 'sage -ipynb2rst requires pandoc; please install it on your system.'

because it is not at all obvious that the jupyter nbconvert command will correctly execute pandoc if and only if that check above works. For example, if the pandoc command is configurable (I have no idea whether it is, but it might be), then you are doing the wrong check. Also, maybe pandoc fails to run because of a different reason than it not being installed (say, it is missing some library).

So unless there is a good reason to add that check, I would prefer to remove it.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​5ed5ab1

#23416 : remove depedency test to pandoc

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​7897736

#23416 : typo

[tmonteil]

Done. Sorry, it seems i cannot refrain to ask for permission ;)

[slabbe]

sage -sws2rst file.sws puts images into a file_media folder. Should we follow the same idea here instead of putting the images in the .?

[slabbe]

Why

        sage: L = ["sage", "--ipynb2rst", input, output]
        sage: try:
        ....:     _ = test_executable(['pandoc', '--version'])
        ....: except OSError:
        ....:     print('True')
        ....: else:
        ....:     _ = test_executable(L)
        ....:     print(open(output, 'r').read() == t)
        True

instead of

sage: L = ["sage", "--ipynb2rst", input, output]
sage: _ = test_executable(L)                     # optional: pandoc
sage: print(open(output, 'r').read() == t)       # optional: pandoc
True

?

[tmonteil]

Replying to slabbe:

Why

        sage: L = ["sage", "--ipynb2rst", input, output]
        sage: try:
        ....:     _ = test_executable(['pandoc', '--version'])
        ....: except OSError:
        ....:     print('True')
        ....: else:
        ....:     _ = test_executable(L)
        ....:     print(open(output, 'r').read() == t)
        True

instead of

sage: L = ["sage", "--ipynb2rst", input, output]
sage: _ = test_executable(L)                     # optional: pandoc
sage: print(open(output, 'r').read() == t)       # optional: pandoc
True

?

Because pandoc is not an optional package but a system dependency, so the doctest will never be run (unless someone explicitely ask for it).

[vdelecroix]

Replying to tmonteil:

Replying to slabbe:

Why

        sage: L = ["sage", "--ipynb2rst", input, output]
        sage: try:
        ....:     _ = test_executable(['pandoc', '--version'])
        ....: except OSError:
        ....:     print('True')
        ....: else:
        ....:     _ = test_executable(L)
        ....:     print(open(output, 'r').read() == t)
        True

instead of

sage: L = ["sage", "--ipynb2rst", input, output]
sage: _ = test_executable(L)                     # optional: pandoc
sage: print(open(output, 'r').read() == t)       # optional: pandoc
True

?

Because pandoc is not an optional package but a system dependency, so the doctest will never be run (unless someone explicitely ask for it).

you can add has_pandoc in sage.doctest.external so that it can be properly detected (see also #25305).

[tmonteil]

Why not. Then, i will wait the next beta to avoid useless rebasing mess.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​30806a2	Merge branch 'u/tmonteil/provide_a__sage__ipynb2rst__command' of trac.sagemath.org:sage into HEAD
​f935b04	#23416 : has_pandoc doctest function
​6e37aa2	#23416 : update cmdline doctest