KeyError raised by GraphLatex.dot2tex_picture when edge_labels=True

[slabbe]

The following is ok :

sage: G = DiGraph()
sage: G.add_edge(3333, 88, 'my_label')
sage: G.set_latex_options(format='dot2tex')
sage: view(G)

but with edges_labels=True a KeyError is raised :

sage: G.set_latex_options(edge_labels=True)
sage: view(G)
ERROR    Failed to process input
Traceback (most recent call last):
  File "/Users/slabbe/Applications/sage-5.2/local/lib/python2.7/site-packages/dot2tex/dot2tex.py", line 2939, in main
    s =  conv.convert(dotdata)
  File "/Users/slabbe/Applications/sage-5.2/local/lib/python2.7/site-packages/dot2tex/dot2tex.py", line 794, in convert
    return self.do_preview_preproc()
  File "/Users/slabbe/Applications/sage-5.2/local/lib/python2.7/site-packages/dot2tex/dot2tex.py", line 1167, in do_preview_preproc
    hp,dp,wt = pp.texdims[name]
KeyError: '3333880'

The error is also raised by dot2tex_picture method:

sage: G = DiGraph()
sage: G.add_edge(0, 1, 'p0')
sage: G.latex_options().dot2tex_picture()
'\n\\begin{tikzpicture}[>=latex,line join=bevel,]\n%%\n\\node (1) at (6bp,7bp) [draw,draw=none] {$1$};\n  \\node (0) at (6bp,57bp) [draw,draw=none] {$0$};\n  \\draw [black,->] (0) ..controls (6bp,44.053bp) and (6bp,33.103bp)  .. (1);\n%\n\\end{tikzpicture}\n'
.dot2tex_picture()
sage: G.set_latex_options(edge_labels=True)
sage: G.latex_options().dot2tex_picture()
ERROR    Failed to process input
Traceback (most recent call last):
  File "/Users/slabbe/Applications/sage-5.2/local/lib/python2.7/site-packages/dot2tex/dot2tex.py", line 2939, in main
    s =  conv.convert(dotdata)
  File "/Users/slabbe/Applications/sage-5.2/local/lib/python2.7/site-packages/dot2tex/dot2tex.py", line 794, in convert
    return self.do_preview_preproc()
  File "/Users/slabbe/Applications/sage-5.2/local/lib/python2.7/site-packages/dot2tex/dot2tex.py", line 1167, in do_preview_preproc
    hp,dp,wt = pp.texdims[name]
KeyError: '010'
''

This is due to dot2tex currently not supporting '\verb' properly.

The attached patch works around this by replacing \verb by \text, and stripping out tricky characters like ^ and _.

[slabbe]

I managed to fix the bug with the following hack :

src/dot2tex/dot2tex.py

@@ -1153 +1153 @@
         for name,item in usededges.items():
 
             edge = item
+            break
             hp,dp,wt = pp.texdims[name]
             xmargin, ymargin = self.get_margins(edge)
             labelcode = '<<<table border="0" cellborder="0" cellpadding="0">'\

but I still don't know what was this loop for... because the output is OK (with edge labels).

[slabbe]

I just created an issue on the dot2tex google code project :

​http://code.google.com/p/dot2tex/issues/detail?id=32

[nthiery]

Hi Sébastien,

I had not noticed this ticket. A quick and dirty workaround has been in the Sage-Combinat queue for some time. I did some further cleanup, but it's still nothing but a workaround. Shall we get it into Sage? If you think the patch could use more work, do you mind taking it over?

Cheers,

Nicolas

[nthiery]

While I was at it, I folded in a couple tests I had put in a later patch in the Sage-Combinat queue.

[nthiery]

The change verb -> text in sage.misc.latex was breaking quite a few doctests. I made it a separate ticket: #14256. This one should not depend on it.

On the other hand, I added three tiny things that were in a later patch in the queue:

• shape=plain text whenever there are labels on the vertices (latex or not)
• trim "." from keys (I guess I must have had a problem with them at some point)
• a doctest fix.

[jdemeyer]

Please add your name as Author.

[nthiery]

Hi Nathann!

Would you have time to review this one?

Thanks!

Nicolas

[ncohen]

This is what happens when I install graphviz

[...]
make[3]: *** No rule to make target `../../plugin/pango/libgvplugin_pango.la', needed by `dot_builtins'.  Stop.
make[3]: *** Waiting for unfinished jobs....
mv -f .deps/no_demand_loading.Tpo .deps/no_demand_loading.Po
mv -f .deps/dot_builtins.Tpo .deps/dot_builtins.Po
mv -f .deps/dot.Tpo .deps/dot.Po
make[3]: Leaving directory `/home/ncohen/.Sage/spkg/build/graphviz-2.16.1.p0/src/cmd/dot'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/home/ncohen/.Sage/spkg/build/graphviz-2.16.1.p0/src/cmd'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/home/ncohen/.Sage/spkg/build/graphviz-2.16.1.p0/src'
make: *** [all] Error 2
Error building Graphviz

real	0m53.650s
user	0m45.475s
sys	0m8.313s
************************************************************************
Error installing package graphviz-2.16.1.p0
************************************************************************
Please email sage-devel (http://groups.google.com/group/sage-devel)
explaining the problem and including the relevant part of the log file
  /home/ncohen/.Sage/spkg/logs/graphviz-2.16.1.p0.log
Describe your computer, operating system, etc.
If you want to try to fix the problem yourself, *don't* just cd to
/home/ncohen/.Sage/spkg/build/graphviz-2.16.1.p0 and type 'make' or whatever is appropriate.
Instead, the following commands setup all environment variables
correctly and load a subshell for you to debug the error:
  (cd '/home/ncohen/.Sage/spkg/build/graphviz-2.16.1.p0' && '/home/ncohen/.Sage/sage' -sh)
When you are done debugging, you can type "exit" to leave the subshell.
************************************************************************

Nathann

[nthiery]

Hi!

Yes, the graphviz spkg is broken (and as far as I can remember has always been). But most linux distro have a graphviz package, so that's not too bad. The doc already says to install graphviz out of Sage.

We should probably remove this broken spkg from "experimental" ...

Cheers,

Nicolas

[ncohen]

Yes, the graphviz spkg is broken (and as far as I can remember has always been). But most linux distro have a graphviz package, so that's not too bad. The doc already says to install graphviz out of Sage.

......

who the hell put that in the doc without removing the package ?...

Nathann

[ncohen]

We should probably remove this broken spkg from "experimental" ...

Indeed. The only thing to do is update the doc to remove any mention of this graphviz package, and ask Jeroen in a ticket to remove it from experimental.

And it will be quickly reviewed, I swear.

Nathann

[nthiery]

Replying to ncohen:

We should probably remove this broken spkg from "experimental" ...

Indeed. The only thing to do is update the doc to remove any mention of this graphviz package, and ask Jeroen in a ticket to remove it from experimental.

I was already careful *not* to mention the spkg in the doc (but if you see how to improve the graphviz installation instructions, please go ahead in a separate ticket; I'll review it).

As for removing the spkg, this is now: #14398.

[ncohen]

I was already careful *not* to mention the spkg in the doc (but if you see how to improve the graphviz installation instructions, please go ahead in a separate ticket; I'll review it).

Is written explicitly anywhere that there is no spkg and that the users should install it by themselves ?

Nathann

[jdemeyer]

# optional - requires dot2tex and graphviz

should be

# optional - dot2tex graphviz

[ncohen]

(and while you're editing the patch, you included a http link which Sphinx does not understand as one)

Nathann

[ncohen]

5 doctests fail on my machine. The error message do not seem to say that there is something wrong with my install (dot or graphviz which I installed through my distro).

sage -t --long dot2tex_utils.py
**********************************************************************
File "dot2tex_utils.py", line 11, in sage.graphs.dot2tex_utils
Failed example:
    output = dot2tex.dot2tex(graph, format="positions", prog="neato") # optional - requires dot2tex and graphviz
Expected nothing
Got:
    ERROR    Failed to process input
    Traceback (most recent call last):
      File "/home/ncohen/.Sage/local/lib/python2.7/site-packages/dot2tex/dot2tex.py", line 2928, in main
        s =  conv.convert(dotdata)
      File "/home/ncohen/.Sage/local/lib/python2.7/site-packages/dot2tex/dot2tex.py", line 848, in convert
        return self.output()
      File "/home/ncohen/.Sage/local/lib/python2.7/site-packages/dot2tex/dot2tex.py", line 2566, in output
        positions[node.name] = map(int, pos.split(','))
    ValueError: invalid literal for int() with base 10: '87.137'
**********************************************************************

Nathann

[nthiery]

Replying to ncohen:

I was already careful *not* to mention the spkg in the doc (but if you see how to improve the graphviz installation instructions, please go ahead in a separate ticket; I'll review it).

Is written explicitly anywhere that there is no spkg and that the users should install it by themselves ?

It's written that the soft can be downloaded from the graphviz website. This seems explicit enough to me, but if you want to improve that, go ahead (but I would not speak of spkg at all).

[ncohen]

It's written that the soft can be downloaded from the graphviz website.

Where ?

Nathann

[nthiery]

Replying to ncohen:

It's written that the soft can be downloaded from the graphviz website.

Where ?

Nathann

grep is your friend :-)

See the method layout_graphviz in generic_graph.

[nthiery]

Replying to ncohen:

(and while you're editing the patch, you included a http link which Sphinx does not understand as one)

Done. And added the file to the index.rst. And misc cleanup of the docstrings.

[ncohen]

grep is your friend :-)

Cool.

See the method layout_graphviz in generic_graph.

Do we both agree that this is nowhere sufficient ?

Here is what happens when some code that requires graphviz (and which does not use layout_graphviz) is executed wihout graphviz installed

sage: sage: import dot2tex                                                    # optional - dot2tex graphviz ## line 10 ##
sage: sage: output = dot2tex.dot2tex(graph, format="positions", prog="neato") # optional - dot2tex graphviz ## line 11 ##
ERROR    Could not locate Graphviz binaries
ERROR    Failed to create xdotdata. Is Graphviz installed?
ERROR    Failed to parse the input data. Is it a valid dot file?
Try to input xdot data directly. Example:
    dot -Txdot file.dot | dot2tex > file.tex

If this does not work, check that you have an updated version of PyParsing and
Graphviz. Users have reported problems with old versions. You can also run
dot2tex in debug mode using the --debug option:
    dot2tex --debug file.dot
A file dot2tex.log will be written to the current directory with detailed
information useful for debugging.
An exception has occurred, use %tb to see the full traceback.

SystemExit: 1

To exit: use 'exit', 'quit', or Ctrl-D

The error message should redirect the user toward a way to install graphviz I guess. And I have never met "SystemExit" By the way, do we really need pyparsing ? I thought this did not belong to Sage, and I needed it a couple of days ago.

I reset this ticket to needs_work because the doctests from two messages above still do not pass, even with graphviz installed.

Nathann

[nthiery]

Replying to ncohen:

5 doctests fail on my machine. The error message do not seem to say that there is something wrong with my install (dot or graphviz which I installed through my distro).

This is a bug in dot2tex. It's fixed in the dev branch since two years. I was waiting the official release to make a new spkg, but it seems to be taking some time ...

Oh well, I uploaded my dev spkg and updated the instructions in #7004. And updated a bit the docstring in the patch as well.

Note: I had to wipe by hand the dot2tex directory before reinstalling dot2tex for things to work:

rm -rf /opt/sage-5.9.beta2/local/lib/python2.7/site-packages/dot2tex

[nthiery]

Replying to ncohen:

See the method layout_graphviz in generic_graph.

Do we both agree that this is nowhere sufficient ?

Here is what happens when some code that requires graphviz (and which does not use layout_graphviz) is executed wihout graphviz installed

sage: sage: import dot2tex                                                    # optional - dot2tex graphviz ## line 10 ##
sage: sage: output = dot2tex.dot2tex(graph, format="positions", prog="neato") # optional - dot2tex graphviz ## line 11 ##
ERROR    Could not locate Graphviz binaries
ERROR    Failed to create xdotdata. Is Graphviz installed?
ERROR    Failed to parse the input data. Is it a valid dot file?
Try to input xdot data directly. Example:
    dot -Txdot file.dot | dot2tex > file.tex

If this does not work, check that you have an updated version of PyParsing and
Graphviz. Users have reported problems with old versions. You can also run
dot2tex in debug mode using the --debug option:
    dot2tex --debug file.dot
A file dot2tex.log will be written to the current directory with detailed
information useful for debugging.
An exception has occurred, use %tb to see the full traceback.

SystemExit: 1

To exit: use 'exit', 'quit', or Ctrl-D

The error message should redirect the user toward a way to install graphviz I guess. And I have never met "SystemExit"

I agree this is ugly, but that's a problem with dot2tex, not with Sage (feel free to report!). Any Sage code using dot2tex should call assert_have_doc2tex.

By the way, do we really need pyparsing ? I thought this did not belong to Sage, and I needed it a couple of days ago.

No clue.

[ncohen]

I agree this is ugly, but that's a problem with dot2tex, not with Sage (feel free to report!). Any Sage code using dot2tex should call assert_have_doc2tex.

This one doesn't ?

Nathann

[ncohen]

Oh. dot2tex is a Python spkg. Explains a lot, sorry :-P

Nathann

[ncohen]

Anyway why do you set this ticket to needs_review. I said twice that the doctests do not pass even with graphviz installed.

Nathann

[ncohen]

And I had not noticed that you had not answered my rethorical question :

See the method layout_graphviz in generic_graph.

Do we both agree that this is nowhere sufficient ?

Nathann

[slabbe]

Nicolas, could you add the following doctest to your patch to make sure the issue do not come up again in the future?

We make sure :trac:`13624` is fixed::

    sage: G = DiGraph()
    sage: G.add_edge(3333, 88, 'my_label')
    sage: G.set_latex_options(format='dot2tex')
    sage: G.set_latex_options(edge_labels=True)
    sage: G.latex_options().dot2tex_picture()  # optional - dot2tex graphviz
    \begin{tikzpicture}
    first_line?
    ...
    last_line?
    \end{tikzpicture}

I don't really know where to put the information about where and how to download graphviz and dot2tex. In fact, there are many entry points for the user:

• G.set_latex_options
• G.latex_options
• G.latex_options().dot2tex_picture
• G.latex_options().set_option <- it might be strange, but a lot of doc is in here
• G.latex_options().set_options
• G.latex_options().latex

It is hard to know where the user will first have the idea that installing graphviz + dot2tex could be nice. Personnaly, I found once that information here :

• G.layout_graphviz?

Here is what it says :

This requires "graphviz" and the "dot2tex" spkg. Here are some    
installation tips:                                                
                                                                  
   * Install graphviz >= 2.14 so that the programs dot, neato, ...
     are in your path. The graphviz suite can be download from    
     http://graphviz.org.                                         
                                                                  
   * Download dot2tex-2.8.?.spkg from                             
     http://trac.sagemath.org/sage_trac/ticket/7004 and install it
     with "sage -i dot2tex-2.8.?.spkg"                            

This does not speak about any graphviz spkg. In fact, I did not know there was one. When I saw that doc in the past, I was then able to install graphviz properly.

The information about the dot2tex spgk is obsolete. I suggest to replace it by :

   * Get the name of the most recent version of dot2tex from
     the sage command ``optional_packages()``. Then,
     install it with ``sage -i dot2tex-2.8.?.spkg``. 

[nthiery]

Replying to ncohen:

Anyway why do you set this ticket to needs_review. I said twice that the doctests do not pass even with graphviz installed.

Have you installed the new spkg as I pointed out in my previous comment?

[nthiery]

Ok guys, I have used up my time on this patch. I need to move on. Feel free to takeover.

Nathann: if you want patches to get in Sage quickly, don't keep expanding tickets beyond their original scope.

[nthiery]

Replying to ncohen:

And I had not noticed that you had not answered my rethorical question :

See the method layout_graphviz in generic_graph.

Do we both agree that this is nowhere sufficient ?

Certainly not great. Certainly out of scope for this ticket.

[slabbe]

Replying to nthiery:

Ok guys, I have used up my time on this patch. I need to move on. Feel free to takeover.

Sorry, I will propose a patch soon which apply on yours that will implement what I suggested. I can't do it now as the version of Sage I have now (sage-5.7.beta4) is too old: I get a conflict with the patch.

Sébastien

[ncohen]

Do we both agree that this is nowhere sufficient ?

Certainly not great. Certainly out of scope for this ticket.

Then find somebody else to review it. I refuse to be part of this mess.

Nathann

[slabbe]

Ils sont fous ces romains...

Replying to nthiery:

Have you installed the new spkg as I pointed out in my previous comment?

I think it is better not to update the old ticket #7004. Here is the link to the new spkg:

sage -f http://sage.math.washington.edu/home/nthiery/dot2tex-2.8.7-dev.spkg

[slabbe]

I just added a patch which fixes the broken doctests seen by the patchbot. I also added a doctest that make sure the issue of this ticket is solved. I also fixed an # optional - dot2tex, graphviz which was not up to date.

Finally I realize that some optional test are broken. Like this one that we can see in the doc of layout_graphviz. This may be considered in another ticket.

sage: g = digraphs.ButterflyGraph(2)
sage: g.plot(layout = "graphviz", prog = "neato") # optional - dot2tex, graphviz
Traceback (most recent call last): 
...
ValueError: invalid literal for int() with base 10: '67.297'

[nthiery]

Replying to slabbe:

I just added a patch which fixes the broken doctests seen by the patchbot. I also added a doctest that make sure the issue of this ticket is solved. I also fixed an # optional - dot2tex, graphviz which was not up to date.

Thanks Sébastien! I looked at your reviewer patch, and it looks good to me. Let's see what the patchbot says. Back to needs review for the first patch.

Finally I realize that some optional test are broken. Like this one that we can see in the doc of layout_graphviz. This may be considered in another ticket.

sage: g = digraphs.ButterflyGraph(2)
sage: g.plot(layout = "graphviz", prog = "neato") # optional - dot2tex, graphviz
Traceback (most recent call last): 
...
ValueError: invalid literal for int() with base 10: '67.297'

+1 since the tests were already failing before this ticket. This is now #14408 which is about updating the dot2tex spkg.

[novoselt]

For the record, current installation instructions for dot2tex are outdated (I tried the example from the description and it failed at first).

With the current patch at #14382, the example from the description does not give errors.

[slabbe]

Adding a doctest to make sure the initial issue is fixed

[slabbe]

I just updated my patch which adds a doctest. Even if #14382 finally fix the problem, I think it could be merged anyway.

I removed the part that was fixing dot2tex optional doctests errors. I will move this part in another patch that I am going to post on #14408 where similar issues were discussed.

[novoselt]

Replying to slabbe:

Even if #14382 finally fix the problem,

Feel free to give it a try and set to positive review - it is not big ;-)

[tscrim]

Currently in 6.2.rc0, the example given in the description works (probably from #14382):

sage: G = DiGraph()
sage: G.add_edge(333, 88, "my_label")
sage: G.set_latex_options(format="dot2tex", edge_labels=True)
sage: latex(G)

\begin{tikzpicture}[>=latex,line join=bevel,]
%%
\node (node_1) at (11.000000bp,79.000000bp) [draw,draw=none] {$333$};
  \node (node_0) at (11.000000bp,7.000000bp) [draw,draw=none] {$88$};
  \draw [black,->] (node_1) ..controls (11.000000bp,61.481000bp) and (11.000000bp,39.548000bp)  .. (node_0);
  \definecolor{strokecol}{rgb}{0.0,0.0,0.0};
  \pgfsetstrokecolor{strokecol}
  \draw (38.000000bp,43.000000bp) node {$\text{\texttt{my{\char`\_}label}}$};
%
\end{tikzpicture}

So how much of this ticket should we keep or should we just close as a "works-for-me"?

[slabbe]

So how much of this ticket should we keep or should we just close as a "works-for-me"?

As I suggested in a previous comment, I think we could merge the patch trac_13624-review-sl.patch which simply adds a doctest.

Sébastien

[slabbe]

I just converted my old hg patch to a git branch. It should be easy to review as it only adds a doctest since the problem was fixed by #14382.

---

New commits:

​2634500

adding a doctest to make sure 13624 is fixed