Opened 10 years ago

Closed 9 years ago

Last modified 8 years ago

#11459 closed enhancement (fixed)

Implement sage -rst2txt and sage -rst2sws

Reported by: slabbe Owned by: slabbe
Priority: major Milestone: sage-5.1
Component: notebook Keywords: sd31 days31 sd40.5
Cc: kcrisman, kini, jasonbhill Merged in: sage-5.1.beta2
Authors: Sébastien Labbé Reviewers: Keshav Kini
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: #11080 Stopgaps:

Description (last modified by kini)

This ticket concerns the following :

  • Convert a ReST file into a worksheet file from the command line : sage -rst2txt.
  • Convert a ReST file into a Sage worksheet from the command line : sage -rst2sws.
  • Upload a ReST (.rst) file from the Notebook.
  • Upload a .html file output from docutils rst2html on a ReST.
  • Upload a worksheet.html file as those stored in the repository ~/.sage/sage_notebook.sagenb/home/admin without having to change the extension to .txt.
  • Add the class docutilsHTMLProcessor to the file sagenb/notebook/docHTMLProcessor.py which translates HTML file generated by docutils rst2html command into a worksheet text file. This class was inspired from the class SphinxHTMLProcessor already in the same Python file.
  • Add the class genericHTMLProcessor to the same file which gathers all the common methods of the other two classes.
  • Improve a lot the documentation of the file sagenb/notebook/docHTMLProcessor.py : it is now 100% doctest coveraged.

My motivation is that in general I wrote my own code using the Sage Code Convention where examples are formatted in ReST. The possibility of converting rst syntax to a worksheet would be nice. Also, ReST file is more fun to edit than the html/cells worksheet text file. It is also more versatile as it can be used to generate a web page or S5 presentation, etc. using docutils.

On my web page, I created a page rst2sws where I put an example of ReST file. Note that I managed to use MathJax to render the HTML output of docutils rst2html command.

Here is my clone of the sage-flask-notebook where I my patch on the sage notebook will be available.

I am planning to make this script more clean during Sage Days 31. I will upload patches as it gets better.

Related to : #10637 (Implement sage -sws2rst) and #10652.

I mentionned the idea on sage-devel in December 2010 here.

This is based on a script called rst2notebook I wrote in January 2011 available on the Sage Combinat server.

---

Apply: trac_11459-root.patch, trac_11459-root.reviewer.patch to $SAGE_ROOT

Apply: trac_11459-scripts.patch, trac_11459-scripts.reviewer.patch to $SAGE_LOCAL/bin

Attachments (6)

trac11459_rst2notebook_sagenb-sl.patch (59.3 KB) - added by slabbe 10 years ago.
Applies on the new flask sagenb repository
trac11459_rst2notebook_localbin-sl.patch (11.4 KB) - added by slabbe 10 years ago.
Applies in the local/bin repository
trac_11459-root.patch (1.8 KB) - added by kini 9 years ago.
apply to $SAGE_ROOT
trac_11459-scripts.patch (9.9 KB) - added by kini 9 years ago.
apply to $SAGE_LOCAL/bin
trac_11459-root.reviewer.patch (1.2 KB) - added by kini 9 years ago.
apply to $SAGE_ROOT
trac_11459-scripts.reviewer.patch (11.1 KB) - added by kini 9 years ago.
apply to $SAGE_LOCAL/bin

Download all attachments as: .zip

Change History (48)

comment:1 Changed 10 years ago by slabbe

  • Keywords sd31 days31 added

comment:2 follow-up: Changed 10 years ago by kcrisman

  • Cc kcrisman added

Maybe this should be rst2sws rather than rst2notebook? By analogy with #10637 sws2rst.

comment:3 in reply to: ↑ 2 ; follow-up: Changed 10 years ago by slabbe

Replying to kcrisman:

Maybe this should be rst2sws rather than rst2notebook? By analogy with #10637 sws2rst.

By rst2notebook, I mean something different. I mean the worksheet txt file which is made of html code and {{{'s cells. I agree rst2notebook is confusing. Maybe sage -rst2worksheet_txt would be better?

By the way, I also implemented the sage -rst2sws. I still need to add some doctests and do some more testing before it needs review. I hope to get to that point by wednesday.

Sébastien

comment:4 in reply to: ↑ 3 Changed 10 years ago by kcrisman

Replying to slabbe:

Replying to kcrisman:

Maybe this should be rst2sws rather than rst2notebook? By analogy with #10637 sws2rst.

By rst2notebook, I mean something different. I mean the worksheet txt file which is made of html code and {{{'s cells. I agree rst2notebook is confusing. Maybe sage -rst2worksheet_txt would be better?

Hmm, or something like that. That is really long. But rst2worksheet would be misleading as well, so maybe that's the best option.

What about rst2txt?

By the way, I also implemented the sage -rst2sws. I still need to add some doctests and do some more testing before it needs review. I hope to get to that point by wednesday.

Great! Except for the fact that I usually need to go the other direction - but of course those are also appearing.

comment:5 Changed 10 years ago by slabbe

What about rst2txt?

Well, at first, I excluded that because txt extension are often used for writing ReST files. But, on the other hand, when one uploads a file having a .txt extension from the notebook, it is assume to be that worksheet txt file made of html code and triple braces cells. So I think you are right. rst2txt is a good suggestion. Thank you!

By the way, I also implemented the sage -rst2sws. I still need to add some doctests and do some more testing before it needs review. I hope to get to that point by wednesday.

Great! Except for the fact that I usually need to go the other direction - but of course those are also appearing.

Why BTW? I feel I am the only one that needs more the rst -> sws way?! I write all my examples in ReST syntax in my personal code, that might be why.

comment:6 Changed 10 years ago by slabbe

  • Description modified (diff)
  • Summary changed from Implement sage -rst2notebook to Implement sage -rst2txt and sage -rst2sws

comment:7 Changed 10 years ago by kini

  • Cc kini added

comment:8 Changed 10 years ago by slabbe

  • Cc jasonbhill added

comment:9 Changed 10 years ago by slabbe

  • Description modified (diff)

Changed 10 years ago by slabbe

Applies on the new flask sagenb repository

comment:10 Changed 10 years ago by slabbe

  • Description modified (diff)

comment:11 Changed 10 years ago by jasonbhill

I'm testing this, but I wanted to give others a heads-up on something. (Maybe Kini can shed some light on this from the hg side of things.)

I'm currently using the flask repo (the ~official~ one...) and have a symlink redirecting devel/sagenb->flask. The path for various diffs in this patch should be following that sagenb symlink, but they're following my sage->sage-main (or whatever clone I'm working in) repo instead.

comment:12 Changed 10 years ago by kini

The root of the sagenb repository is in your devel/sagenb, i.e. already beyond the symlink. The paths should have nothing to do with that (and look fine to me fwiw). Are you applying it in the right place?

comment:13 Changed 10 years ago by jasonbhill

Yup... "sagenb" and "sage" are the two symlinks in devel. All of my patches are attempting to following the latter.

I'm using hg_sage.apply("address"). My usual workflow is to create a clone, build into that clone, then apply any patches. This has always worked, but then again I am always working under the "sage" symlink. But, in this situation the patch (regardless of branch) is attempting to apply in "devel/sage->my_".

How might one apply this otherwise ... in a specific location?

comment:14 Changed 10 years ago by kini

Ah. Use hg_sagenb instead of hg_sage - try tab completing hg_ in the Sage console to see the list of repositories you have access to. hg_sage is the Sage library, hg_sagenb is the notebook, hg_extcode seems to be for the interfaces to other software, hg_scripts is for the sage command line tools and related stuff, hg_root is for some miscellaneous "top-level" stuff.

I would personally recommend using Mercurial from the command line, though - it allows you to see what you are doing more precisely, where patches are going exactly, etc.

comment:15 Changed 10 years ago by kini

To be precise, hg_sage points to devel/sage, hg_sagenb points to devel/sagenb, hg_extcode points to data/extcode, and hg_scripts points to local/bin, if you want to go exploring. Using sage -clone and sage -b <branchname> only apply to devel/sage - other repositories are left uncloned and unmodified.

By the way, you might consider dropping by IRC to talk about these things in realtime instead of posting to this trac ticket, as this is getting quite off-topic... :)

comment:16 follow-up: Changed 10 years ago by slabbe

Salut!

Just to give an update about the status of the patches on this ticket :

  1. trac11459_rst2notebook_sagenb-sl.patch : I was not able to export the patch once I qfinish it and pushed it on my branch on the sagenb google code. Hence, the patch available here do not have the very last modification I did just before I made qfinish. So if you are testing it, it would be better if you test my sagenb branch. Anyhow, I do not intend to change my sagenb branch. I believe it is ready for review.

Question for kini : How can I export the patch once it is qfinished (commited) so that I upload here the latest version?

  1. trac11459_rst2notebook_localbin-sl.patch : The script needs work. I want to implement better options about escaping backslashes. I should be able to finish this in the next days during the week. Once this is done, I will change the status of this ticket to needs review.

Sébastien

comment:17 Changed 10 years ago by kini

You can turn committed changes back into patches with hg qimport -r tip if you want. If you want to leave them committed but export a patch, use hg export tip > file.patch. (don't forget to put "[diff]\ngit=yes" in your ~/.hgrc.)

comment:18 in reply to: ↑ 16 Changed 10 years ago by slabbe

Hence, the patch available here do not have the very last modification I did just before I made qfinish.

It was not true finally. The patch trac11459_rst2notebook_sagenb-sl.patch available here contains the very last modifications.

Changed 10 years ago by slabbe

Applies in the local/bin repository

comment:19 Changed 10 years ago by slabbe

  • Authors set to Sébastien Labbé
  • Status changed from new to needs_review

I finished working on the local/bin scripts and just re-uploaded the patch.

Now needs review!

comment:20 Changed 9 years ago by slabbe

  • Dependencies set to #11080

Changed 9 years ago by kini

apply to $SAGE_ROOT

Changed 9 years ago by kini

apply to $SAGE_LOCAL/bin

comment:21 Changed 9 years ago by kini

Hi Sebastien,

I've rebased the scripts patch to Sage 5.0. Because the file formerly at $SAGE_LOCAL/bin/sage-sage is now at $SAGE_ROOT/spkg/bin/sage, it had to be split into two patches.

The notebook patch will probably need to be rebased as well, but since we're doing development of sagenb on github now, it won't go here on trac. I'll see what I can do...

comment:22 Changed 9 years ago by kini

  • Milestone changed from sage-5.1 to sage-5.0

Never mind, it looks like you already got that code into sagenb after all.

comment:23 Changed 9 years ago by kini

  • Milestone changed from sage-5.0 to sage-5.1

whoops

comment:24 Changed 9 years ago by kini

  • Keywords sd40.5 added
  • Reviewers set to Keshav Kini
  • Status changed from needs_review to positive_review

So here is a reviewer patch, and positive review for this ticket. Everything looks good to me.

comment:25 Changed 9 years ago by kini

  • Keywords changed from sd31, days31, sd40.5 to sd31 days31 sd40.5

Changed 9 years ago by kini

apply to $SAGE_ROOT

comment:26 Changed 9 years ago by kini

  • Description modified (diff)

Whoops, forgot the other reviewer patch...

comment:27 Changed 9 years ago by slabbe

Hi Kini,

Thanks for looking at this!

I remember I wanted to remove the guillemets (") in the line of documentation of both scripts:

".. escape-backslashes"

Can you update your patch to do it, if you agree? I am sorry I never did it...

Is the new notebook now part of Sage ? If I want to test this ticket, should I install the latest sage-alpha or should I install the new sage-notebook?

Sébastien

comment:28 Changed 9 years ago by kini

  • Status changed from positive_review to needs_work

OK, I'll do it, and I'll also modify the file sage-rst2txt which I completely overlooked...!

The new notebook is not part of Sage yet (though it finally got positive review a couple of hours ago - #11080). If you want to test the ticket, follow the instructions on #11080 to install the new notebook first.

Changed 9 years ago by kini

apply to $SAGE_LOCAL/bin

comment:29 Changed 9 years ago by kini

  • Status changed from needs_work to positive_review

OK, there we go.

comment:30 Changed 9 years ago by jdemeyer

  • Milestone changed from sage-5.1 to sage-pending

comment:31 Changed 9 years ago by slabbe

Thanks for the review Keshav!

comment:32 Changed 9 years ago by jdemeyer

  • Merged in set to sage-5.1.beta2
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:33 Changed 9 years ago by kcrisman

Hi! I'm trying to get #10637 (the other direction) in shape for positive review, and I notice there are no patches here documenting in the Sage library how to use this. I guess that's because they just make text and sws. Was there any place that you would think would be natural, other than sage -advanced?

comment:34 Changed 9 years ago by slabbe

Was there any place that you would think would be natural, other than sage -advanced?

Note: There is some documentation in sage -rst2sws -h as well.

It would be a good idea to add documentation somewhere in the Sage reference manual. Where are the other sage-scripts documented?

Last edited 9 years ago by slabbe (previous) (diff)

comment:35 Changed 9 years ago by kcrisman

My sense is that they are not.

Okay, I'll look at your script and see how you implement that, then do that at #10637. That would suffice for the purposes there.

Do you want to open another ticket to document (at least some) of the scripts in the usual reference manual?

comment:36 Changed 9 years ago by kcrisman

By the way, not that you have to deal with this now, see http://docs.python.org/library/optparse.html:

Deprecated since version 2.7: The optparse module is deprecated and will not be developed further; 
development will continue with the argparse module.

Oh well.

comment:37 Changed 8 years ago by jdemeyer

  • Milestone changed from sage-pending to sage-5.1

comment:38 follow-up: Changed 8 years ago by jdemeyer

This "depends" on #11080 but somehow got merged anyway.

comment:39 in reply to: ↑ 38 ; follow-up: Changed 8 years ago by slabbe

Replying to jdemeyer:

This "depends" on #11080 but somehow got merged anyway.

It depends on code written in the new sagenb indeed. That means the script won't work on sage-5.1 : it will get an import error I would guess. The thing is that there is no test for the script... so you could not see that the script was broken without #11080. Or worse, in the future, if a patch on sagenb breaks the script, we won't know it. How should a script be doctested?

comment:40 Changed 8 years ago by slabbe

The part of the script which depends on the new sagenb is the following :

# Do the translation html -> worksheet txt 
from sagenb.notebook.docHTMLProcessor import docutilsHTMLProcessor 
translator = docutilsHTMLProcessor() 
worksheet_txt = translator.process_doc_html(html) 
 
# create a Notebook 
from sagenb.notebook.notebook import Notebook 
from sage.misc.misc import tmp_dir 
nb = Notebook(tmp_dir()+'.sagenb') 
 
# create a worksheet 
W = nb.create_new_worksheet(title, 'admin') 
W.edit_save(worksheet_txt) 
 
# export the worksheet as sws 
nb.export_worksheet(W.filename(), output, title=title) 
 
# delete the Notebook 
nb.delete() 

Is it possible that such code get broken in future version of sagenb?

comment:41 in reply to: ↑ 39 ; follow-up: Changed 8 years ago by jdemeyer

Replying to slabbe:

How should a script be doctested?

You can have a look at sage/tests/cmdline.py for many examples.

In any case, such a test should be the subject of a new ticket.

comment:42 in reply to: ↑ 41 Changed 8 years ago by slabbe

In any case, such a test should be the subject of a new ticket.

This is now #13297.

Note: See TracTickets for help on using tickets.