Opened 10 years ago

Closed 9 years ago

Implement sage -rst2txt and sage -rst2sws

Reported by: Owned by: slabbe slabbe major sage-5.1 notebook sd31 days31 sd40.5 kcrisman, kini, jasonbhill sage-5.1.beta2 Sébastien Labbé Keshav Kini N/A #11080

This ticket concerns the following :

• Convert a ReST file into a worksheet file from the command line : sage -rst2txt.
• Convert a ReST file into a Sage worksheet from the command line : sage -rst2sws.
• Upload a ReST (.rst) file from the Notebook.
• Upload a .html file output from docutils rst2html on a ReST.
• Upload a worksheet.html file as those stored in the repository ~/.sage/sage_notebook.sagenb/home/admin without having to change the extension to .txt.
• Add the class docutilsHTMLProcessor to the file sagenb/notebook/docHTMLProcessor.py which translates HTML file generated by docutils rst2html command into a worksheet text file. This class was inspired from the class SphinxHTMLProcessor already in the same Python file.
• Add the class genericHTMLProcessor to the same file which gathers all the common methods of the other two classes.
• Improve a lot the documentation of the file sagenb/notebook/docHTMLProcessor.py : it is now 100% doctest coveraged.

My motivation is that in general I wrote my own code using the Sage Code Convention where examples are formatted in ReST. The possibility of converting rst syntax to a worksheet would be nice. Also, ReST file is more fun to edit than the html/cells worksheet text file. It is also more versatile as it can be used to generate a web page or S5 presentation, etc. using docutils.

On my web page, I created a page rst2sws where I put an example of ReST file. Note that I managed to use MathJax to render the HTML output of docutils rst2html command.

Here is my clone of the sage-flask-notebook where I my patch on the sage notebook will be available.

I am planning to make this script more clean during Sage Days 31. I will upload patches as it gets better.

Related to : #10637 (Implement sage -sws2rst) and #10652.

I mentionned the idea on sage-devel in December 2010 here.

This is based on a script called rst2notebook I wrote in January 2011 available on the Sage Combinat server.

---

Apply: trac_11459-root.patch, trac_11459-root.reviewer.patch to $SAGE_ROOT Apply: trac_11459-scripts.patch, trac_11459-scripts.reviewer.patch to $SAGE_LOCAL/bin

comment:2 follow-up: ↓ 3 Changed 10 years ago by kcrisman

Maybe this should be rst2sws rather than rst2notebook? By analogy with #10637 sws2rst.

comment:3 in reply to: ↑ 2 ; follow-up: ↓ 4 Changed 10 years ago by slabbe

Maybe this should be rst2sws rather than rst2notebook? By analogy with #10637 sws2rst.

By rst2notebook, I mean something different. I mean the worksheet txt file which is made of html code and {{{'s cells. I agree rst2notebook is confusing. Maybe sage -rst2worksheet_txt would be better?

By the way, I also implemented the sage -rst2sws. I still need to add some doctests and do some more testing before it needs review. I hope to get to that point by wednesday.

Sébastien

comment:4 in reply to: ↑ 3 Changed 10 years ago by kcrisman

Maybe this should be rst2sws rather than rst2notebook? By analogy with #10637 sws2rst.

By rst2notebook, I mean something different. I mean the worksheet txt file which is made of html code and {{{'s cells. I agree rst2notebook is confusing. Maybe sage -rst2worksheet_txt would be better?

Hmm, or something like that. That is really long. But rst2worksheet would be misleading as well, so maybe that's the best option.

By the way, I also implemented the sage -rst2sws. I still need to add some doctests and do some more testing before it needs review. I hope to get to that point by wednesday.

Great! Except for the fact that I usually need to go the other direction - but of course those are also appearing.

comment:5 Changed 10 years ago by slabbe

Well, at first, I excluded that because txt extension are often used for writing ReST files. But, on the other hand, when one uploads a file having a .txt extension from the notebook, it is assume to be that worksheet txt file made of html code and triple braces cells. So I think you are right. rst2txt is a good suggestion. Thank you!

By the way, I also implemented the sage -rst2sws. I still need to add some doctests and do some more testing before it needs review. I hope to get to that point by wednesday.

Great! Except for the fact that I usually need to go the other direction - but of course those are also appearing.

Why BTW? I feel I am the only one that needs more the rst -> sws way?! I write all my examples in ReST syntax in my personal code, that might be why.

comment:6 Changed 10 years ago by slabbe

• Description modified (diff)
• Summary changed from Implement sage -rst2notebook to Implement sage -rst2txt and sage -rst2sws

comment:9 Changed 10 years ago by slabbe

• Description modified (diff)

Changed 10 years ago by slabbe

Applies on the new flask sagenb repository

comment:10 Changed 10 years ago by slabbe

• Description modified (diff)

comment:11 Changed 10 years ago by jasonbhill

I'm testing this, but I wanted to give others a heads-up on something. (Maybe Kini can shed some light on this from the hg side of things.)

I'm currently using the flask repo (the ~official~ one...) and have a symlink redirecting devel/sagenb->flask. The path for various diffs in this patch should be following that sagenb symlink, but they're following my sage->sage-main (or whatever clone I'm working in) repo instead.

comment:12 Changed 10 years ago by kini

The root of the sagenb repository is in your devel/sagenb, i.e. already beyond the symlink. The paths should have nothing to do with that (and look fine to me fwiw). Are you applying it in the right place?

comment:13 Changed 10 years ago by jasonbhill

Yup... "sagenb" and "sage" are the two symlinks in devel. All of my patches are attempting to following the latter.

I'm using hg_sage.apply("address"). My usual workflow is to create a clone, build into that clone, then apply any patches. This has always worked, but then again I am always working under the "sage" symlink. But, in this situation the patch (regardless of branch) is attempting to apply in "devel/sage->my_".

How might one apply this otherwise ... in a specific location?

comment:14 Changed 10 years ago by kini

Ah. Use hg_sagenb instead of hg_sage - try tab completing hg_ in the Sage console to see the list of repositories you have access to. hg_sage is the Sage library, hg_sagenb is the notebook, hg_extcode seems to be for the interfaces to other software, hg_scripts is for the sage command line tools and related stuff, hg_root is for some miscellaneous "top-level" stuff.

I would personally recommend using Mercurial from the command line, though - it allows you to see what you are doing more precisely, where patches are going exactly, etc.

comment:15 Changed 10 years ago by kini

To be precise, hg_sage points to devel/sage, hg_sagenb points to devel/sagenb, hg_extcode points to data/extcode, and hg_scripts points to local/bin, if you want to go exploring. Using sage -clone and sage -b <branchname> only apply to devel/sage - other repositories are left uncloned and unmodified.

By the way, you might consider dropping by IRC to talk about these things in realtime instead of posting to this trac ticket, as this is getting quite off-topic... :)

comment:16 follow-up: ↓ 18 Changed 10 years ago by slabbe

Salut!

Just to give an update about the status of the patches on this ticket :

1. trac11459_rst2notebook_sagenb-sl.patch : I was not able to export the patch once I qfinish it and pushed it on my branch on the sagenb google code. Hence, the patch available here do not have the very last modification I did just before I made qfinish. So if you are testing it, it would be better if you test my sagenb branch. Anyhow, I do not intend to change my sagenb branch. I believe it is ready for review.

Question for kini : How can I export the patch once it is qfinished (commited) so that I upload here the latest version?

1. trac11459_rst2notebook_localbin-sl.patch : The script needs work. I want to implement better options about escaping backslashes. I should be able to finish this in the next days during the week. Once this is done, I will change the status of this ticket to needs review.

Sébastien

comment:17 Changed 10 years ago by kini

You can turn committed changes back into patches with hg qimport -r tip if you want. If you want to leave them committed but export a patch, use hg export tip > file.patch. (don't forget to put "[diff]\ngit=yes" in your ~/.hgrc.)

comment:18 in reply to: ↑ 16 Changed 10 years ago by slabbe

Hence, the patch available here do not have the very last modification I did just before I made qfinish.

It was not true finally. The patch trac11459_rst2notebook_sagenb-sl.patch available here contains the very last modifications.

Changed 10 years ago by slabbe

Applies in the local/bin repository

comment:19 Changed 10 years ago by slabbe

• Authors set to Sébastien Labbé
• Status changed from new to needs_review

I finished working on the local/bin scripts and just re-uploaded the patch.

Now needs review!

comment:20 Changed 9 years ago by slabbe

• Dependencies set to #11080

comment:21 Changed 9 years ago by kini

Hi Sebastien,

I've rebased the scripts patch to Sage 5.0. Because the file formerly at $SAGE_LOCAL/bin/sage-sage is now at $SAGE_ROOT/spkg/bin/sage, it had to be split into two patches.

The notebook patch will probably need to be rebased as well, but since we're doing development of sagenb on github now, it won't go here on trac. I'll see what I can do...

comment:22 Changed 9 years ago by kini

• Milestone changed from sage-5.1 to sage-5.0

Never mind, it looks like you already got that code into sagenb after all.

comment:23 Changed 9 years ago by kini

• Milestone changed from sage-5.0 to sage-5.1

whoops

comment:24 Changed 9 years ago by kini

• Reviewers set to Keshav Kini
• Status changed from needs_review to positive_review

So here is a reviewer patch, and positive review for this ticket. Everything looks good to me.

comment:25 Changed 9 years ago by kini

• Keywords changed from sd31, days31, sd40.5 to sd31 days31 sd40.5

comment:29 Changed 9 years ago by kini

• Status changed from needs_work to positive_review

OK, there we go.

comment:30 Changed 9 years ago by jdemeyer

• Milestone changed from sage-5.1 to sage-pending

comment:31 Changed 9 years ago by slabbe

Thanks for the review Keshav!

comment:32 Changed 9 years ago by jdemeyer

• Merged in set to sage-5.1.beta2
• Resolution set to fixed
• Status changed from positive_review to closed

comment:33 Changed 9 years ago by kcrisman

Hi! I'm trying to get #10637 (the other direction) in shape for positive review, and I notice there are no patches here documenting in the Sage library how to use this. I guess that's because they just make text and sws. Was there any place that you would think would be natural, other than sage -advanced?

comment:34 Changed 9 years ago by slabbe

Was there any place that you would think would be natural, other than sage -advanced?

Note: There is some documentation in sage -rst2sws -h as well.

It would be a good idea to add documentation somewhere in the Sage reference manual. Where are the other sage-scripts documented?

Last edited 9 years ago by slabbe (previous) (diff)

comment:35 Changed 9 years ago by kcrisman

My sense is that they are not.

Okay, I'll look at your script and see how you implement that, then do that at #10637. That would suffice for the purposes there.

Do you want to open another ticket to document (at least some) of the scripts in the usual reference manual?

comment:36 Changed 9 years ago by kcrisman

By the way, not that you have to deal with this now, see http://docs.python.org/library/optparse.html:

Deprecated since version 2.7: The optparse module is deprecated and will not be developed further;
development will continue with the argparse module.


Oh well.

comment:37 Changed 8 years ago by jdemeyer

• Milestone changed from sage-pending to sage-5.1

comment:38 follow-up: ↓ 39 Changed 8 years ago by jdemeyer

This "depends" on #11080 but somehow got merged anyway.

comment:39 in reply to: ↑ 38 ; follow-up: ↓ 41 Changed 8 years ago by slabbe

This "depends" on #11080 but somehow got merged anyway.

It depends on code written in the new sagenb indeed. That means the script won't work on sage-5.1 : it will get an import error I would guess. The thing is that there is no test for the script... so you could not see that the script was broken without #11080. Or worse, in the future, if a patch on sagenb breaks the script, we won't know it. How should a script be doctested?

comment:40 Changed 8 years ago by slabbe

The part of the script which depends on the new sagenb is the following :

# Do the translation html -> worksheet txt
from sagenb.notebook.docHTMLProcessor import docutilsHTMLProcessor
translator = docutilsHTMLProcessor()
worksheet_txt = translator.process_doc_html(html)

# create a Notebook
from sagenb.notebook.notebook import Notebook
from sage.misc.misc import tmp_dir
nb = Notebook(tmp_dir()+'.sagenb')

# create a worksheet
W.edit_save(worksheet_txt)

# export the worksheet as sws
nb.export_worksheet(W.filename(), output, title=title)

# delete the Notebook
nb.delete()


Is it possible that such code get broken in future version of sagenb?

comment:41 in reply to: ↑ 39 ; follow-up: ↓ 42 Changed 8 years ago by jdemeyer

How should a script be doctested?

You can have a look at sage/tests/cmdline.py for many examples.

In any case, such a test should be the subject of a new ticket.

comment:42 in reply to: ↑ 41 Changed 8 years ago by slabbe

In any case, such a test should be the subject of a new ticket.

This is now #13297.

Note: See TracTickets for help on using tickets.