Implement sage -rst2txt and sage -rst2sws

[slabbe]

This ticket concerns the following :

• Convert a ReST file into a worksheet file from the command line : sage -rst2txt.
• Convert a ReST file into a Sage worksheet from the command line : sage -rst2sws.
• Upload a ReST (.rst) file from the Notebook.
• Upload a .html file output from docutils rst2html on a ReST.
• Upload a worksheet.html file as those stored in the repository ~/.sage/sage_notebook.sagenb/home/admin without having to change the extension to .txt.
• Add the class docutilsHTMLProcessor to the file sagenb/notebook/docHTMLProcessor.py which translates HTML file generated by docutils rst2html command into a worksheet text file. This class was inspired from the class SphinxHTMLProcessor already in the same Python file.
• Add the class genericHTMLProcessor to the same file which gathers all the common methods of the other two classes.
• Improve a lot the documentation of the file sagenb/notebook/docHTMLProcessor.py : it is now 100% doctest coveraged.

My motivation is that in general I wrote my own code using the Sage Code Convention where examples are formatted in ReST. The possibility of converting rst syntax to a worksheet would be nice. Also, ReST file is more fun to edit than the html/cells worksheet text file. It is also more versatile as it can be used to generate a web page or S5 presentation, etc. using docutils.

On my web page, I created a page ​rst2sws where I put an example of ReST file. Note that I managed to use MathJax to render the HTML output of docutils rst2html command.

Here is ​my clone of the sage-flask-notebook where I my patch on the sage notebook will be available.

I am planning to make this script more clean during ​Sage Days 31. I will upload patches as it gets better.

Related to : #10637 (Implement sage -sws2rst) and #10652.

I mentionned the idea on sage-devel in December 2010 ​here.

This is based on a script called rst2notebook I wrote in January 2011 available ​on the Sage Combinat server.

---

Apply: trac_11459-root.patch​, trac_11459-root.reviewer.patch​ to $SAGE_ROOT

Apply: trac_11459-scripts.patch​, trac_11459-scripts.reviewer.patch​ to $SAGE_LOCAL/bin

[kcrisman]

Maybe this should be rst2sws rather than rst2notebook? By analogy with #10637 sws2rst.

[slabbe]

Replying to kcrisman:

Maybe this should be rst2sws rather than rst2notebook? By analogy with #10637 sws2rst.

By rst2notebook, I mean something different. I mean the worksheet txt file which is made of html code and {{{'s cells. I agree rst2notebook is confusing. Maybe sage -rst2worksheet_txt would be better?

By the way, I also implemented the sage -rst2sws. I still need to add some doctests and do some more testing before it needs review. I hope to get to that point by wednesday.

Sébastien

[kcrisman]

Replying to slabbe:

Replying to kcrisman:

Maybe this should be rst2sws rather than rst2notebook? By analogy with #10637 sws2rst.

By rst2notebook, I mean something different. I mean the worksheet txt file which is made of html code and {{{'s cells. I agree rst2notebook is confusing. Maybe sage -rst2worksheet_txt would be better?

Hmm, or something like that. That is really long. But rst2worksheet would be misleading as well, so maybe that's the best option.

What about rst2txt?

By the way, I also implemented the sage -rst2sws. I still need to add some doctests and do some more testing before it needs review. I hope to get to that point by wednesday.

Great! Except for the fact that I usually need to go the other direction - but of course those are also appearing.

[slabbe]

What about rst2txt?

Well, at first, I excluded that because txt extension are often used for writing ReST files. But, on the other hand, when one uploads a file having a .txt extension from the notebook, it is assume to be that worksheet txt file made of html code and triple braces cells. So I think you are right. rst2txt is a good suggestion. Thank you!

By the way, I also implemented the sage -rst2sws. I still need to add some doctests and do some more testing before it needs review. I hope to get to that point by wednesday.

Great! Except for the fact that I usually need to go the other direction - but of course those are also appearing.

Why BTW? I feel I am the only one that needs more the rst -> sws way?! I write all my examples in ReST syntax in my personal code, that might be why.

[slabbe]

Applies on the new flask sagenb repository

[jasonbhill]

I'm testing this, but I wanted to give others a heads-up on something. (Maybe Kini can shed some light on this from the hg side of things.)

I'm currently using the flask repo (the ~official~ one...) and have a symlink redirecting devel/sagenb->flask. The path for various diffs in this patch should be following that sagenb symlink, but they're following my sage->sage-main (or whatever clone I'm working in) repo instead.

[kini]

The root of the sagenb repository is in your devel/sagenb, i.e. already beyond the symlink. The paths should have nothing to do with that (and look fine to me fwiw). Are you applying it in the right place?

[jasonbhill]

Yup... "sagenb" and "sage" are the two symlinks in devel. All of my patches are attempting to following the latter.

I'm using hg_sage.apply("address"). My usual workflow is to create a clone, build into that clone, then apply any patches. This has always worked, but then again I am always working under the "sage" symlink. But, in this situation the patch (regardless of branch) is attempting to apply in "devel/sage->my_".

How might one apply this otherwise ... in a specific location?

[kini]

Ah. Use hg_sagenb instead of hg_sage - try tab completing hg_ in the Sage console to see the list of repositories you have access to. hg_sage is the Sage library, hg_sagenb is the notebook, hg_extcode seems to be for the interfaces to other software, hg_scripts is for the sage command line tools and related stuff, hg_root is for some miscellaneous "top-level" stuff.

I would personally recommend using Mercurial from the command line, though - it allows you to see what you are doing more precisely, where patches are going exactly, etc.

[kini]

To be precise, hg_sage points to devel/sage, hg_sagenb points to devel/sagenb, hg_extcode points to data/extcode, and hg_scripts points to local/bin, if you want to go exploring. Using sage -clone and sage -b <branchname> only apply to devel/sage - other repositories are left uncloned and unmodified.

By the way, you might consider dropping by ​IRC to talk about these things in realtime instead of posting to this trac ticket, as this is getting quite off-topic... :)

[slabbe]

Salut!

Just to give an update about the status of the patches on this ticket :

1. trac11459_rst2notebook_sagenb-sl.patch : I was not able to export the patch once I qfinish it and pushed it on my branch on the sagenb google code. Hence, the patch available here do not have the very last modification I did just before I made qfinish. So if you are testing it, it would be better if you test my sagenb branch. Anyhow, I do not intend to change my sagenb branch. I believe it is ready for review.

Question for kini : How can I export the patch once it is qfinished (commited) so that I upload here the latest version?

2. trac11459_rst2notebook_localbin-sl.patch : The script needs work. I want to implement better options about escaping backslashes. I should be able to finish this in the next days during the week. Once this is done, I will change the status of this ticket to needs review.

Sébastien

[kini]

You can turn committed changes back into patches with hg qimport -r tip if you want. If you want to leave them committed but export a patch, use hg export tip > file.patch. (don't forget to put "[diff]\ngit=yes" in your ~/.hgrc.)

[slabbe]

Hence, the patch available here do not have the very last modification I did just before I made qfinish.

It was not true finally. The patch trac11459_rst2notebook_sagenb-sl.patch available here contains the very last modifications.

[slabbe]

Applies in the local/bin repository

[slabbe]

I finished working on the local/bin scripts and just re-uploaded the patch.

Now needs review!

[kini]

apply to $SAGE_ROOT

[kini]

apply to $SAGE_LOCAL/bin

[kini]

Hi Sebastien,

I've rebased the scripts patch to Sage 5.0. Because the file formerly at $SAGE_LOCAL/bin/sage-sage is now at $SAGE_ROOT/spkg/bin/sage, it had to be split into two patches.

The notebook patch will probably need to be rebased as well, but since we're doing development of sagenb on github now, it won't go here on trac. I'll see what I can do...

[kini]

Never mind, it looks like you ​already got that code into sagenb after all.

[kini]

whoops

[kini]

So here is a reviewer patch, and positive review for this ticket. Everything looks good to me.

[kini]

apply to $SAGE_ROOT

[kini]

Whoops, forgot the other reviewer patch...

[slabbe]

Hi Kini,

Thanks for looking at this!

I remember I wanted to remove the guillemets (") in the line of documentation of both scripts:

".. escape-backslashes"

Can you update your patch to do it, if you agree? I am sorry I never did it...

Is the new notebook now part of Sage ? If I want to test this ticket, should I install the latest sage-alpha or should I install the new sage-notebook?

Sébastien

[kini]

OK, I'll do it, and I'll also modify the file sage-rst2txt which I completely overlooked...!

The new notebook is not part of Sage yet (though it finally got positive review a couple of hours ago - #11080). If you want to test the ticket, follow the instructions on #11080 to install the new notebook first.

[kini]

apply to $SAGE_LOCAL/bin

[kini]

OK, there we go.

[slabbe]

Thanks for the review Keshav!

[kcrisman]

Hi! I'm trying to get #10637 (the other direction) in shape for positive review, and I notice there are no patches here documenting in the Sage library how to use this. I guess that's because they just make text and sws. Was there any place that you would think would be natural, other than sage -advanced?

[slabbe]

Was there any place that you would think would be natural, other than sage -advanced?

Note: There is some documentation in sage -rst2sws -h as well.

It would be a good idea to add documentation somewhere in the Sage reference manual. Where are the other sage-scripts documented?

[kcrisman]

My sense is that they are not.

Okay, I'll look at your script and see how you implement that, then do that at #10637. That would suffice for the purposes there.

Do you want to open another ticket to document (at least some) of the scripts in the usual reference manual?

[kcrisman]

By the way, not that you have to deal with this now, see ​http://docs.python.org/library/optparse.html:

Deprecated since version 2.7: The optparse module is deprecated and will not be developed further; 
development will continue with the argparse module.

Oh well.

[jdemeyer]

This "depends" on #11080 but somehow got merged anyway.

[slabbe]

Replying to jdemeyer:

This "depends" on #11080 but somehow got merged anyway.

It depends on code written in the new sagenb indeed. That means the script won't work on sage-5.1 : it will get an import error I would guess. The thing is that there is no test for the script... so you could not see that the script was broken without #11080. Or worse, in the future, if a patch on sagenb breaks the script, we won't know it. How should a script be doctested?

[slabbe]

The part of the script which depends on the new sagenb is the following :

# Do the translation html -> worksheet txt 
from sagenb.notebook.docHTMLProcessor import docutilsHTMLProcessor 
translator = docutilsHTMLProcessor() 
worksheet_txt = translator.process_doc_html(html) 
 
# create a Notebook 
from sagenb.notebook.notebook import Notebook 
from sage.misc.misc import tmp_dir 
nb = Notebook(tmp_dir()+'.sagenb') 
 
# create a worksheet 
W = nb.create_new_worksheet(title, 'admin') 
W.edit_save(worksheet_txt) 
 
# export the worksheet as sws 
nb.export_worksheet(W.filename(), output, title=title) 
 
# delete the Notebook 
nb.delete() 

Is it possible that such code get broken in future version of sagenb?

[jdemeyer]

Replying to slabbe:

How should a script be doctested?

You can have a look at sage/tests/cmdline.py for many examples.

In any case, such a test should be the subject of a new ticket.

[slabbe]

In any case, such a test should be the subject of a new ticket.

This is now #13297.