Opened 4 years ago

Closed 3 years ago

#10146 closed enhancement (fixed)

update SageTeX spkg to version 2.3.1

Reported by: ddrake Owned by: ddrake
Priority: major Milestone: sage-4.7.2
Component: packages: standard Keywords: sd31
Cc: vbraun, abmasse Merged in: sage-4.7.2.alpha0
Authors: Dan Drake Reviewers: Mariah Lenox, Andrey Novoseltsev
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description (last modified by jdemeyer)

I've just finished a new version of the SageTeX spkg, which is available from

http://sage.math.washington.edu/home/drake/code/sage/st/sagetex-2.3.1.spkg

Attachments (3)

Screenshot.png (41.4 KB) - added by novoselt 4 years ago.
Example of going over the margins
STE.pdf (56.9 KB) - added by novoselt 4 years ago.
Vertical space issues with sagecommandline
STE-new.pdf (56.9 KB) - added by novoselt 4 years ago.
Fixed vertical spacing (same input as before, with modified sagetex.py)

Download all attachments as: .zip

Change History (44)

comment:1 Changed 4 years ago by jason

What's new in this? What should we look for?

comment:3 follow-up: Changed 4 years ago by jason

  • Status changed from new to needs_review

What is the sagetex-2.3.cksum file? How come it's not in the spkg directory? I've never seen a file like this before

comment:4 in reply to: ↑ 3 Changed 4 years ago by ddrake

Replying to jason:

What is the sagetex-2.3.cksum file? How come it's not in the spkg directory? I've never seen a file like this before

See #329. It's for doing integrity checks of spkg files. The md5sum of the spkg should be a377b1b98b6e92164f4042783a949e75, if you want to be very cautious.

You could review #329 after you're done here. :)

Replying to jason:

What's new in this? What should we look for?

The sagecommandline environment is the new thing, so please look carefully at that. Look at the examples, try to make up your own examples, check over the documentation, etc.

Looking over hg log, other changes include:

  • generated files are now called .sagetex.sage, .sagetex.py, etc.
  • the warning that you need to re-run Sage that LaTeX prints is hopefully a little more obvious
  • there's a CONTRIBUTORS file.

The commandline environment is the big thing.

Changed 4 years ago by novoselt

Example of going over the margins

comment:5 Changed 4 years ago by novoselt

  • Status changed from needs_review to needs_work
  • Work issues set to Going over margins

The sagecommandline environment is going over margins on the right (perhaps it has textwidth width, but gets shifted?). I think this is a quite serious issue for any LaTeX package, so I will switch it to "needs work."

(Vertical line on the left and blue on the right on the attached screenshot are parts of window borders, the text above is within margins.)

comment:6 follow-up: Changed 4 years ago by novoselt

Another minor issue: clicking on the link in the footnote of the first page goes to

http://mathsci.kaist.ac.kr/%5Cprotect%20%5Cunhbox%20%5Cvoidb@x%20%5Cpenalty%20%5C@M%20%5C%20%7B%7Ddrake/

leading to

The requested URL /\protect \unhbox \voidb@x \penalty \@M \ {}drake/ was not found on this server.

And two comments which are more like feature requests than issues:

  • I have a document with one-and-a-half line interval and Sage examples are typeset with the same interval. I'd rather use a single one (or even less) for examples, since there are no sub- and superscripts. I can easily achieve it by wrapping an example into \begin{spacing}{0}...\end{spacing} (using setspace package), but I'd have to repeat it every time, since my attempts to define a combined environment were unsuccessful, I guess because of fragility of sagecommandline.
  • It would be nice to have an option to turn off line numbers at all. Especially if they actually were meant to be on the margins, as it seems to me after looking at the documentation more carefully.

comment:7 Changed 4 years ago by novoselt

The documentation says: "The SageTEX package defines a length \sagetexindent, which controls how much the Sage code is indented when typeset." But when I try to change it, it seems that it affects only position of line numbers, not the code indentation itself.

After reading listings documentation, I see that it is easy to turn off line numbers (or shift them off the margins), but it would be great if this was mentioned in the documentation of SageTEX - I think that users of Sage are much more likely to worry about it than about syntax highlighting and other stuff supported by listings.

comment:8 Changed 4 years ago by novoselt

  • Status changed from needs_work to needs_review
  • Work issues Going over margins deleted

OK, I'll try to make this the last comment (I wish I could edit the previous ones...)

My complains about \sagetexindent affecting line numbers seem to be wrong, I think it was due to some of the extra code I had. But perhaps it is worth mentioning in the documentation that it does not affect sagecommandline.

I managed to change the spacing in code blocks by adjusting SageInput/Output styles. However, as I understand, every line of Sage input/output is typeset in a separate listings environment. As a result, aboveskip, belowskip, and lineskip all affect the line spacing and allow making it whatever one wants, but I don't see a way to adjust before/after space for the whole sagecommandline environment. Is it possible? If yes, it would be nice to document it, if no, it would be nice to implement it.

As for line numbers on margins, I still don't like it, but it seems to be supposed to be like that, so I withdraw my request to change it (but still it would be nice to explain in SateTEX documentation, how to shift them or turn them off).

Overall, thanks for a great package and a new environment in this version!

comment:9 in reply to: ↑ 6 ; follow-up: Changed 4 years ago by ddrake

Replying to novoselt:

Another minor issue: clicking on the link in the footnote of the first page goes to

http://mathsci.kaist.ac.kr/%5Cprotect%20%5Cunhbox%20%5Cvoidb@x%20%5Cpenalty%20%5C@M%20%5C%20%7B%7Ddrake/

That is weird. I have no idea what that is happening. It's because of the tilde in the URL in the footnote. I've asked about this: http://tex.stackexchange.com/questions/5260/why-is-docstrip-putting-extra-macro-calls-in-a-hyperlink-inside-a-footnote-that-h

Changed 4 years ago by novoselt

Vertical space issues with sagecommandline

comment:10 Changed 4 years ago by novoselt

I've found a couple of issues with vertical spacing before and after sagecommandline environment and posted an example (the second page is the document that produces the first one). I have no idea how to fix it, but judging from other environments it is possible ;-)

comment:11 follow-up: Changed 4 years ago by novoselt

I think that keyword coloring should be removed for the output:

\begin{sagecommandline}
sage: 'Is this statement True?'
\end{sagecommandline}

produces the correct output line

Is this statement True?

But "True" is blue and the rest of the text is black.

comment:12 follow-up: Changed 4 years ago by novoselt

Those using SageTeX for plotting may find #7981 relevant.

Also I find the reference "Till Tantau has some good commentary on the use of graphics in section 6 of the pgf manual" a bit confusing. I suggest pointing the link at http://www.ctan.org/tex-archive/help/Catalogue/entries/pgf.html and adding the name of the section since now it is "7 Guidelines on Graphics."

comment:13 in reply to: ↑ 9 Changed 4 years ago by ddrake

Replying to ddrake:

That is weird. I have no idea what that is happening. It's because of the tilde in the URL in the footnote. I've asked about this: http://tex.stackexchange.com/questions/5260/why-is-docstrip-putting-extra-macro-calls-in-a-hyperlink-inside-a-footnote-that-h

I got an answer there and have fixed that problem.

comment:14 in reply to: ↑ 11 Changed 4 years ago by ddrake

  • Cc vbraun added

Replying to novoselt:

I think that keyword coloring should be removed for the output:

\begin{sagecommandline}
sage: 'Is this statement True?'
\end{sagecommandline}

produces the correct output line

Is this statement True?

But "True" is blue and the rest of the text is black.

I'll add Volker Braun to the CC list and see if he has a comment; he knows more about the LaTeX listings package than I do.

comment:15 in reply to: ↑ 12 Changed 4 years ago by ddrake

  • Owner changed from tbd to ddrake

Replying to novoselt:

Those using SageTeX for plotting may find #7981 relevant.

Also I find the reference "Till Tantau has some good commentary on the use of graphics in section 6 of the pgf manual" a bit confusing. I suggest pointing the link at http://www.ctan.org/tex-archive/help/Catalogue/entries/pgf.html and adding the name of the section since now it is "7 Guidelines on Graphics."

Fixed.

I'll look over the other issues raised here and see what I can do.

comment:16 Changed 4 years ago by novoselt

Thanks for fixes! It was indeed illuminating to read that section in PGF manual. And after reading it, it seems to me that there should be no default scaling of the produced graphics to 0.75\textwidth, users should control it via figsize option to matplotlib, since in this case specifying font sizes and line widths in points makes sense and works as expected. This behaviour is easy to achieve in the present version by adding empty [] after \sageplot, but I think it should be the default.

Better yet, it would be nice if it was possible to write

\sageplot[width=0.5\textwidth]{...}

and this resulted in automatic computation of the required figsize parameters (since figsize=[\textwidth, 2] does not work). I guess this is a bit tricky since figsize requires two parameters, but perhaps it would make sense to use the same width and height if only one was given.

comment:17 Changed 4 years ago by novoselt

#2100 may be relevant to the above suggestion.

Also I have another very general suggestion. I didn't succeed in installing sagetex.sty in my home folder and I don't really care what was wrong, as I just copied it directly to the folder with my main .tex file. I think that this is the best option, it should be recommended in the manual, and instead of checking that sagetex.sty version matches the installed Sage version, there should be a check that Sage installation is not older than sagetex.sty.

The reason was mentioned on sage-devel:
http://groups.google.com/group/sage-devel/msg/999f3712689df932
papers don't get updates as often as software and one of the really nice features of LaTeX is that its current versions can process old input files and produce the same output as many years ago. I think the same should be the case for SageTeX and it probably can be accomplished if its Sage module maintained backward compatibility with all previous versions of sagetex.sty. Then when one wants to write a new paper using SageTeX, s/he copies the current version of sagetex.sty into the paper directory, writes the paper and keeps that version of sagetex.sty with the paper source. Even if later sagetex.sty will be significantly altered, that paper should still work fine with the old copy, even if Sage installation is newer.

comment:18 Changed 4 years ago by novoselt

Optimization suggestion for listings-styles: basicstyle propagates to all other ones, so setting it to \ttfamily makes everything typeset with it. Also, there is no bold version of typewriter fonts (at least for standard fonts of LaTeX), so \bfseries didn't do anything. The following commands do for me the same typesetting as the original ones, but look a little cleaner and make it easier for the user to understand what should be changed. (I wanted to make everything a little smaller, and my first impression was that I have to insert \small into 10 places or so, but one turned out to be enough.)

\lstdefinestyle{DefaultSageInputOutput}{
  nolol,
  identifierstyle=,
  name=sagecommandline,
  xleftmargin=5pt,
  numbersep=5pt,
  aboveskip=0pt,
  belowskip=0pt,
  breaklines=true,
  basicstyle={\ttfamily},
  numberstyle=\footnotesize,
  numbers=right
}
\lstdefinestyle{DefaultSageInput}{
  language=Sage,
  style=DefaultSageInputOutput,
  commentstyle={\color{dgreencolor}},
  keywordstyle={\color{dbluecolor}},
  stringstyle={\color{dgraycolor}},
}
\lstdefinestyle{DefaultSageOutput}{
  language=SageOutput,
  style=DefaultSageInputOutput,
  commentstyle={\color{dgreencolor}},
  keywordstyle={\color{dbluecolor}},
  stringstyle={\color{dgraycolor}},
}

In the last definition I think all styles can go since the output should not be typeset with any kind of "syntax highlighting", in particular True/False? should not be blue.

comment:19 follow-up: Changed 4 years ago by novoselt

Some more issues:

  • If I have a sagecommandline block and then I insert $\sage{...}$ before it, I cannot compile the file. As I understand, because of the reference shift LaTeX tries to substitute the output of sagecommandline into inline sage and that does not work since the latter is in the math mode. But then rerunning Sage does not help since the source code for Sage does not get updated without a successful LaTeX run. So pretty much the only option seems to delete all (or some) autogenerated files, which is a bit annoying.
  • If I put \sage{...} inside, say, \begin{gather} ... \end{gather}, then I get warnings about multiple definition of sage-related labels. Looking at *.sagetex.py file I suppose that the issue is in lines
    try:
     _st_.inline(_sage_const_28 , latex(1+1))
    except:
     _st_.goboom(_sage_const_961 )
    try:
     _st_.inline(_sage_const_28 , latex(1+1))
    except:
     _st_.goboom(_sage_const_961 )
    
    i.e. they are exactly the same. I suppose that's just how LaTeX works, it processes the input of complicated math environments twice, but it would be nice if it was somehow taken into account in SageTeX. "Just ignoring these warnings" is not a good option, because I see only the top warnings automatically in my editor and if that space is "occupied" I'll have to scroll manually to see if my new lines have introduced any warnings that I would like to fix. It is much more pleasant when there are usually no warnings at all and when they do appear I notice it right away.

comment:20 Changed 4 years ago by novoselt

By the way, IMHO this update is very useful as is and I have an itch to switch it to positive review. I hesitate since I cannot really understand the guts of the package, but as a user I like it a lot despite of all the issues I have discovered. Perhaps this version can be merged with fixes and improvements delegated to future ones...

Changed 4 years ago by novoselt

Fixed vertical spacing (same input as before, with modified sagetex.py)

comment:21 follow-up: Changed 4 years ago by novoselt

  • Priority changed from minor to major
  • Status changed from needs_review to needs_work
  • Work issues set to spacing before/after sagecommandline

OK, I have found fixes for the vertical space issues both before and after. In the sagetex.py file (I am not sure where the original is supposed to be, I was fixing the version in sage/local/lib/python2.6/site-packages).

Lines 77-78 should be

    self.souttmp.write('\\newlabel{@sageinline' + str(counter) + '}{{%\n' +
                 s.rstrip() + '}}\n')

I have removed four pairs of {} before the newline. It may be not the proper place/way to do it, but the main idea is that these braces should not appear on the last line of every command line listing in *.sagetex.sout. This seems to fix the issue with extra space appearing after sagecommandline, if the following text starts a new paragraph or a theorem.

Line 145 should be

      latex_string = r"\unskip\unskip\vspace{\sagecommandlineskip}" + "\n"

Two unskips eliminate the space before sagecommandline which was appearing if the preceding line was "full." One unskip does not work, what is the correct number of them I don't know, but I took the repetition idea from amsthm.sty which contains

\ifhmode\unskip\unskip\par\fi

on line 140.

It is possible that \ignorespaces should be used somehow in the end of blocks for even more robust behaviour.

Please include these fixes, for these spacing issues are really annoying and it is somewhat non-trivial to figure out a workaround ;-)

comment:22 Changed 4 years ago by novoselt

#11067 (sage -t on lstlistings) may be related.

comment:23 in reply to: ↑ 21 Changed 4 years ago by ddrake

Replying to novoselt:

Lines 77-78 should be

    self.souttmp.write('\\newlabel{@sageinline' + str(counter) + '}{{%\n' +
                 s.rstrip() + '}}\n')

I have removed four pairs of {} before the newline.

Doing that breaks hyperref compatibility. We can't remove those brace pairs.

I'll look into the other spacing issues.

comment:24 follow-up: Changed 4 years ago by novoselt

As I said, it may be not the right place to fix it. Does sagecommandline need something from hyperref? It wraps each line into its own lstlisting environment and in the very end of the whole block there are these extra braces. Perhaps inline function should be left as is, but a copy of it without braces can be created specifically for sagecommandline?

comment:25 in reply to: ↑ 24 Changed 4 years ago by ddrake

Replying to novoselt:

As I said, it may be not the right place to fix it. Does sagecommandline need something from hyperref?

No, but if the user has hyperref loaded, any \newlabel must have the five {} pairs or the document can't be typeset.

It wraps each line into its own lstlisting environment and in the very end of the whole block there are these extra braces. Perhaps inline function should be left as is, but a copy of it without braces can be created specifically for sagecommandline?

That's a good idea, but in that case the copy of inline would have the same problem: with hyperref loaded, the \newlabel will fail without the extra {}s.

comment:26 Changed 4 years ago by novoselt

I have also changed line 146 to be

      bottom_skip =  r"\vspace{\sagecommandlineskip}" + "\n"

since otherwise there is no bottom skip for blocks that do not produce any output.

It may be better to use \addvspace instead of \vspace - it works fine in the end of blocks since LaTeX is in vertical mode by then, but in the beginning it is in horizontal one, \par is not acceptable inside \newlabel, and I don't know how to switch to vertical mode otherwise. Does anyone know it? Or how to allow wrapping arbitrarily complicated LaTeX?

Regarding \newlabel and hyperref - perhaps SageTeX should detect whether it is in use or not or at least have a manual switch to adjust the number of parameters according to the used definition of \newlabel. By the way - is there documentation on \newlabel command and if yes, how can I find it???

comment:27 in reply to: ↑ 19 ; follow-up: Changed 4 years ago by ddrake

Replying to novoselt:

  • If I have a sagecommandline block and then I insert $\sage{...}$ before it, I cannot compile the file. As I understand, because of the reference shift LaTeX tries to substitute the output of sagecommandline into inline sage and that does not work since the latter is in the math mode. But then rerunning Sage does not help since the source code for Sage does not get updated without a successful LaTeX run. So pretty much the only option seems to delete all (or some) autogenerated files, which is a bit annoying.

Note that it will impossible to prevent this from happening: after the .sout file is written, you can rewrite your LaTeX file in all sorts of crazy ways to get the labels pulled into strange places. I can add a bit of sanity checking to the commandline stuff so that you at least don't get bizarre errors when typesetting your document.

comment:28 in reply to: ↑ 27 ; follow-up: Changed 4 years ago by novoselt

Replying to ddrake:

Replying to novoselt:

  • If I have a sagecommandline block and then I insert $\sage{...}$ before it, I cannot compile the file. As I understand, because of the reference shift LaTeX tries to substitute the output of sagecommandline into inline sage and that does not work since the latter is in the math mode. But then rerunning Sage does not help since the source code for Sage does not get updated without a successful LaTeX run. So pretty much the only option seems to delete all (or some) autogenerated files, which is a bit annoying.

Note that it will impossible to prevent this from happening: after the .sout file is written, you can rewrite your LaTeX file in all sorts of crazy ways to get the labels pulled into strange places. I can add a bit of sanity checking to the commandline stuff so that you at least don't get bizarre errors when typesetting your document.

Would it be possible to use different label names for different SageTeX environments? If they are not mixed, then there should be no issue with using the output in a wrong mode and the user will be able to generate new input file and run Sage on it without deleting anything.

comment:29 in reply to: ↑ 28 Changed 4 years ago by ddrake

Replying to novoselt:

Would it be possible to use different label names for different SageTeX environments? If they are not mixed, then there should be no issue with using the output in a wrong mode and the user will be able to generate new input file and run Sage on it without deleting anything.

That's a much smarter idea than what I was thinking. I've implemented your idea; it's in the most recent commits on bitbucket.

comment:30 Changed 3 years ago by abmasse

  • Cc abmasse added

I just found this ticket and downloaded the latest version. This is great! I'll try testing it.

What's the current status of this ticket? I mean, could you summarize what remains to be done?

Thank you!

comment:31 follow-up: Changed 3 years ago by vbraun

  • Milestone changed from sage-4.7 to sage-4.7.1

I agree that the current state of the spkg is a big improvement, even if there are issues remaining. I'm in favor of giving it a positive review now so we can release it with Sage-4.7.1. There were some good suggestions on this ticket, maybe Dan can comment on whether he wants to make any other changes during this release?

comment:32 Changed 3 years ago by novoselt

Let's please not include the package which was made when this ticket was created, but rather a fresh version reflecting SageTeX repository. The very last issue (resolved by using different link names for different environments) was making it very annoying to modify files with extensive SageTeX usage.

I also made a bunch of suggestions in comment 18 regarding settings of listings. The most important one of them: I strongly feel that there should be no syntax highlighting of any kind in the output (currently True/False? are different).

It would be also a good idea to include some short instructions on installing luximono, which allows using bold font for listings and is quite a bit narrower than the standard typerwriter font.

Regarding bold, I also quite strongly feel that listings should not be bold by default (the output may), since "the main thing" should be typeset using "the main style" and when commandline environment is used, the emphasis is probably on the code itself, so it is likely to be the bigger part of the listing. When the output gets large, it is probably a better idea to typeset/represent it some other way.

comment:33 in reply to: ↑ 31 ; follow-up: Changed 3 years ago by ddrake

Replying to vbraun:

I agree that the current state of the spkg is a big improvement, even if there are issues remaining. I'm in favor of giving it a positive review now so we can release it with Sage-4.7.1. There were some good suggestions on this ticket, maybe Dan can comment on whether he wants to make any other changes during this release?

I'd really like to get the current version into 4.7.1. I looked over the changes since 2.3, and I see three nontrivial, user-visible things:

  • use foo.sagetex.sout, etc, instead of foo.sout. This means that if you put Sage code into important_paper.sage, it won't get clobbered when you typeset important_paper.tex.
  • handle (some) spaces in filenames.
  • use separate labels for inline and commandline (as reported above).

The issues not addressed are the spacing issues that Andrey has noted. The stuff about syntax highlighting needs to be addressed, but I think that counts as bikeshedding here.

If we merge the current version of SageTeX, we have something that definitely works better than what's currently shipping. I will likely have time over the summer to work on that, and in any case, Andrey (and Volker, and many other SageTeXers?) will be at Sage Days 31 and we can have a discussion there.

Here's my proposal: I make a spkg for the current version of SageTeX and we merge that with 4.7.1, and work on the other problems noted here over the summer. What do you think?

comment:34 in reply to: ↑ 33 Changed 3 years ago by vbraun

Replying to ddrake:

Here's my proposal: I make a spkg for the current version of SageTeX and we merge that with 4.7.1, and work on the other problems noted here over the summer. What do you think?

+1

I also don't mind if the output is too colorful. If you don't like the syntax highlighting then its easy to turn off, but if its initially turned off then its hard to discover that the feature is there.

comment:35 Changed 3 years ago by novoselt

  • Work issues changed from spacing before/after sagecommandline to update spkg to the latest version of SageTeX

As long as we merge the version with separate label names - I am fine!

I will be even happier if there are no colored keywords in the output, which is a simple change ;-) But since it does not affect how things work, we may leave it and other things to the next version, after perhaps having some live discussions.

comment:36 Changed 3 years ago by ddrake

  • Description modified (diff)
  • Status changed from needs_work to needs_review
  • Summary changed from update SageTeX spkg to version 2.3 to update SageTeX spkg to version 2.3 (now 2.3.1)

Okay, here's a new spkg: http://sage.math.washington.edu/home/drake/code/sage/st/sagetex-2.3.1.spkg

I'm switching this to "needs review"; if that spkg works, then switch this to positive review and let's get this into 4.7.1 and work on the other bits this summer.

comment:37 Changed 3 years ago by mariah

  • Reviewers set to Mariah Lenox
  • Status changed from needs_review to positive_review

I put the spkg into the source of sage-4.7.1.alpha2 and did 'make testlong'. All tests passed. Positive review!

comment:38 Changed 3 years ago by novoselt

  • Keywords sd31 added
  • Reviewers changed from Mariah Lenox to Mariah Lenox, Andrey Novoseltsev
  • Work issues update spkg to the latest version of SageTeX deleted

comment:39 Changed 3 years ago by jdemeyer

  • Description modified (diff)
  • Summary changed from update SageTeX spkg to version 2.3 (now 2.3.1) to update SageTeX spkg to version 2.3.1

comment:40 Changed 3 years ago by jdemeyer

  • Milestone changed from sage-4.7.1 to sage-4.7.2

comment:41 Changed 3 years ago by jdemeyer

  • Merged in set to sage-4.7.2.alpha0
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.