Adding has_graphviz, has_ffmpeg, has_imagemagick to sage.doctest.external

[slabbe]

Implement this:

sage: from sage.doctest.external import has_graphviz, has_ffmpeg, has_imagemagick
sage: has_graphviz()   # random
True
sage: has_ffmpeg()   # random
True
sage: has_imagemagick()   # random
True

so that graphviz, ffmpeg, imagemagick are included in the external software to be detected:

$ sage -t --optional=external,sage src/sage/graphs/graph_latex.py src/sage/plot/animate.py
...
Using --optional=external
External software to be detected: cplex,ffmpeg,graphviz,gurobi,
imagemagick,internet,latex,macaulay2,magma,maple,mathematica,
matlab,octave,scilab
...

[slabbe]

New commits:

​99f37ac

25305: adding sage.doctest.external.has_graphviz

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​e09d9b7

25305: adding has_ffmpeg and has_imagemagick to sage.doctest.external

[slabbe]

I added has_ffmpeg and have_imagemagick in the second commit.

For references, the list tests_not_run below contains all tags of tests that are not run by default. The list was obtained from grep "not run" on the log of ./sage -t -p --all --long --show-skipped:

sage: tests_not_run = ['4ti2', 'and', 'axiom', 'beautifulsoup', 'benzene', 'bliss', 'buckygen',
....: 'cbc', 'chomp', 'coxeter3', 'cplex', 'cryptominisat', 'csdp', 'cunningham',
....: 'database', 'database_cremona_ellcurve', 'database_gap',
....: 'database_jones_numfield', 'database_kohel', 'database_mutation_class',
....: 'database_odlyzko_zeta', 'database_pari', 'database_stein_watkins',
....: 'database_symbolic_data', 'debug', 'fes', 'ffmpeg', 'fricas', 'frobby',
....: 'gambit', 'gap3', 'gap_packages', 'gdb', 'giacpy_sage', 'ginv', 'glucose',
....: 'gmp', 'gmpy2', 'gnuplot', 'gperftools', 'graphviz', 'growing', 'gurobi',
....: 'hmisc', 'imagemagick', 'internet', 'java', 'kash', 'latex', 'latte_int',
....: 'libbraiding', 'libhomfly', 'libjpeg', 'lie', 'lrs', 'lrslib', 'macaulay2',
....: 'magma', 'maple', 'mathematica', 'mathematicafrontend', 'matlab', 'mcqd',
....: 'meataxe', 'mupad', 'nonexistent_lp_solver', 'octave', 'ore_algebra', 'phc',
....: 'plantri', 'polymake', 'polytopes_db_4d', 'py3', 'pynormaliz', 'python_igraph',
....: 'qepcad', 'randomly', 'required', 'requires', 'rgraphics', 'rsat', 'runsnake',
....: 'scilab', 'sirocco', 'sloane_database', 'surf', 'tdlib', 'the', 'tides',
....: 'time', 'topcom', 'webbrowser']

And this is a tentative list of all "external" packages:

sage: S = set(package_versions('standard'))
sage: O = set(package_versions('optional'))
sage: E = set(package_versions('experimental'))
sage: set(tests_not_run) - (S|O|E)
{'and',
 'axiom',
 'beautifulsoup',
 'chomp',
 'cplex',
 'cunningham',
 'database',
 'debug',
 'fes',
 'ffmpeg',
 'ginv',
 'glucose',
 'gnuplot',
 'gperftools',
 'graphviz',
 'growing',
 'gurobi',
 'hmisc',
 'imagemagick',
 'internet',
 'java',
 'kash',
 'latex',
 'libjpeg',
 'lrs',
 'macaulay2',
 'magma',
 'maple',
 'mathematica',
 'mathematicafrontend',
 'matlab',
 'mupad',
 'nonexistent_lp_solver',
 'octave',
 'phc',
 'polytopes_db_4d',
 'py3',
 'randomly',
 'required',
 'requires',
 'rgraphics',
 'rsat',
 'runsnake',
 'scilab',
 'sloane_database',
 'the',
 'time',
 'webbrowser'}

Maybe we want to detect some more external packages...

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​fe6163f

25305: adding has_ffmpeg and has_imagemagick to sage.doctest.external

[vdelecroix]

Shouldn't this use the recently merged #20382?

[vdelecroix]

Replying to vdelecroix:

Shouldn't this use the recently merged #20382?

Actually, I think that all the has_xxx stuff would better be features.

[slabbe]

New commits:

​cb6e088

25305: adding has_ffmpeg, has_imagemagick, has_graphviz to sage.doctest.external

[slabbe]

Replying to vdelecroix:

Shouldn't this use the recently merged #20382?

Ok as you wish. I am now using find_executable from the Features stuff instead of the have_program from the misc stuff.

[slabbe]

Replying to vdelecroix:

Actually, I think that all the has_xxx stuff would better be features.

Why better? To me adding three new has_key functions to the external.py module makes things better. But changing has_maple in a way that it uses features does not make anything better.

The way the external.py module is coded right now is that the presence of an external software with name tag is detected with a method whose name is has_tag. The goal of this ticket is not to redesign all of this external.py module. So I suggest that further uses of feature inside of external.py be postpone to another ticket if this is at all pertinent.

Needs review!

[vdelecroix]

Replying to slabbe:

Replying to vdelecroix:

Actually, I think that all the has_xxx stuff would better be features.

Why better? To me adding three new has_key functions to the external.py module makes things better. But changing has_maple in a way that it uses features does not make anything better.

I just realized that it is not possible for has_maple. The maple interface allows subtle configurations such as

• an alternative name for the executable
• a remote execution through ssh

This is not the kind of things a feature is aware of.

The way the external.py module is coded right now is that the presence of an external software with name tag is detected with a method whose name is has_tag. The goal of this ticket is not to redesign all of this external.py module. So I suggest that further uses of feature inside of external.py be postpone to another ticket if this is at all pertinent.

In your situation, you are just checking for the presence of executable in the path. This is exactly what sage.features.Executable is about. The main difference between `has_XXX_ and a feature is that

• a feature is not only about doctest
• a feature can run small tests
• meaningful error messages
• this is not run twice in the same session

What I wanted to suggest with my message is to introduce sage.features.graphviz, sage.features.ffmpeg and sage.features.imagemagick and simply use

def has_ffmpeg():
    from sage.features.ffmpeg import FFmpeg
    return FFmpeg.is_present()

The software FFmpeg is indeed used in other parts of Sage (e.g. animate) and it would help to have it as a feature.

[slabbe]

I now creates the files for ffmpeg, imagemagick and graphviz in the feature module and I am using these to solve the problem described in this ticket.

I suggest that the use of the FFmpeg().require() in animate.py etc be postpone to an indenpendent ticket.

---

New commits:

​034678b

25305: adding has_ffmpeg, has_imagemagick, has_graphviz to sage.doctest.external

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​9686e97

25305: adding has_ffmpeg, has_imagemagick, has_graphviz to sage.doctest.external

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​eeaa92f

25305: adding has_ffmpeg, has_imagemagick, has_graphviz to sage.doctest.external

[slabbe]

I fixed some doctests and removed import os,subprocess in the previous two forced pushed.

[slabbe]

Needs review!

[vdelecroix]

Image magick is a suite and not a unique executable:

• magick
• magick-script
• convert
• mogrify
• identify
• composite
• montage
• compare
• etc

[slabbe]

Can you provide an example where a command from imagemagick other than convert is used in the sage source code?

$ sage -grep mogrify
$ sage -grep magick-script
$ sage -grep magick
sage/misc/latex.py:    http://www.imagemagick.org
sage/misc/latex.py:            print("http://www.imagemagick.org to download these programs.")
sage/misc/latex.py:            print("Go to http://www.imagemagick.org to download it.")
sage/misc/latex.py:            print("Go to http://www.imagemagick.org to download it.")
sage/combinat/words/paths.py:            See www.imagemagick.org, for example.
sage/graphs/graph_latex.py:- convert: http://www.imagemagick.org (the ImageMagick suite)
sage/plot/animate.py:- `ImageMagick <http://www.imagemagick.org>`_
sage/plot/animate.py:              See www.imagemagick.org and www.ffmpeg.org for more information.
sage/plot/animate.py:        - `ImageMagick <http://www.imagemagick.org>`_
sage/plot/animate.py:See www.imagemagick.org and www.ffmpeg.org for more information."""
sage/plot/animate.py:See www.imagemagick.org and www.ffmpeg.org for more information."""
sage/plot/animate.py:              See www.imagemagick.org and www.ffmpeg.org for more information.

etc.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​4a7e6b1	25305: adding has_ffmpeg, has_imagemagick, has_graphviz to sage.doctest.external
​84ded42	25305: added doc about only convert being currently checked
​cb74033	25305: fix doctests

[slabbe]

Branch rebased on top of 8.3.beta2.

Added doc about the fact that currently only the presence of convert is checked.

I also fix the doctest that was failing on the patchbot.

Needs review.

[vdelecroix]

You forgot the copyright.

The rest looks ok.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​d4c5fec	25305: adding has_ffmpeg, has_imagemagick, has_graphviz to sage.doctest.external
​8712542	25305: added doc about only convert being currently checked
​baa0bae	25305: fix doctests
​ba9cf98	25305: added licence to newly created files

[slabbe]

Rebased the branch on top of 8.3.beta4.

Added a commit which adds the licence to the newly created files.

Needs review.

[slabbe]

Thank you for the review

[slelievre]

I noticed that some of the doctests have

# optional -- graphviz

while some other doctests have

# optional: graphviz

Are both styles equivalent?