Interface to Mathics

[Sebastian Oehms]

Mathics is an open source interpreter for the Wolfram Language. From the introduction of its reference manual:

Mathics — to be pronounced like “Mathematics” without the “emat” — is a general-purpose computer algebra system (CAS). It is meant to be a free, light-weight alternative to Mathematica®. It is free both as in “free beer” and as in “freedom”. There are various online mirrors running Mathics but it is also possible to run Mathics locally. A list of mirrors can be found at the Mathics homepage, ​http://mathics.github.io.

The programming language of Mathics is meant to resemble Wolfram’s famous Mathematica® as much as possible. However, Mathics is in no way affiliated or supported by Wolfram. Mathics will probably never have the power to compete with Mathematica® in industrial applications; yet, it might be an interesting alternative for educational purposes.

The aim of this ticket is to implement an interface to Mathics. Such an interface has the advantage that functions written in Wolfram Language can be used in Sage on systems that miss a Mathematica® licence.

Moreover, there are many classical software packages written in Wolfram Language that eventually could be used inside Sage in order to participate from their functionality. Currently, for example, Mathics developers are working to get ​KnotTheory and ​FeynCalc running.

To install Mathics inside Sage just run:

./configure --enable-download-from-upstream-url
make mathics

Thanks to the developers of Mathics to support this interface!

[Sebastian Oehms]

The initial version for the interface is an adaption of the corresponding interface for Mathematica. But in contrast it directly uses the Mathics library functionality instead of Pexpect. It contains all doctests inherited from the Mathematica module but also additional ones which are more Mathics specific. Furthermore, the coverage of doctests is complete.

Two of the Mathematica doctests use FindMinimum which doesn't seem to be available in Mathics. I mark them as not tested. 25 doctests had differences to the output that once has been produced by Mathematica. Most of them are caused by a different way of formatting. I've accepted them all, since they where either minor or a task for Mathics developers to decide if they are acceptable or not (see also ​this comment in ​Mathics issue 1235). Explicitly these are:

**********************************************************************
File "src/sage/interfaces/mathics.py", line 37, in sage.interfaces.mathics
Failed example:
    mobj = mathics(x^2-1); mobj       # optional - mathics
Expected:
    x^2 - 1
Got:
    -1 + x ^ 2
**********************************************************************
File "src/sage/interfaces/mathics.py", line 39, in sage.interfaces.mathics
Failed example:
    mobj.Factor()                     # optional - mathics
Expected:
    (x + 1)*(x - 1)
Got:
    (-1 + x) (1 + x)
**********************************************************************
File "src/sage/interfaces/mathics.py", line 51, in sage.interfaces.mathics
Failed example:
    mobj2 = mobj.Factor(); mobj2          # optional - mathics
Expected:
    (-1 + x)*(1 + x)
Got:
    (-1 + x) (1 + x)
**********************************************************************
File "src/sage/interfaces/mathics.py", line 66, in sage.interfaces.mathics
Failed example:
    mathics('Factor[x^2-1]')          # optional - mathics
Expected:
    (-1 + x)*(1 + x)
Got:
    (-1 + x) (1 + x)
**********************************************************************
File "src/sage/interfaces/mathics.py", line 78, in sage.interfaces.mathics
Failed example:
    mathics.eval('x^2 - 1')           # optional - mathics
Expected:
                   2
             -1 + x
Got:
    '-1 + x ^ 2'
**********************************************************************
File "src/sage/interfaces/mathics.py", line 137, in sage.interfaces.mathics
Failed example:
    eqn                               # optional - mathics
Expected:
    5 + 3*x == 14
Got:
    5 + 3 x == 14
**********************************************************************
File "src/sage/interfaces/mathics.py", line 142, in sage.interfaces.mathics
Failed example:
    print(sys)                        # optional - mathics
Expected:
               2
             {x  - 3 y == 3, 2 x - y == 1}
Got:
    {x ^ 2 - 3 y == 3, 2 x - y == 1}
**********************************************************************
File "src/sage/interfaces/mathics.py", line 167, in sage.interfaces.mathics
Failed example:
    v = m([eq1, eq2]); v              # optional - mathics
Expected:
    {x^2 - 3*y == 3, 2*x - y == 1}
Got:
    {x ^ 2 - 3 y == 3, 2 x - y == 1}
**********************************************************************
File "src/sage/interfaces/mathics.py", line 193, in sage.interfaces.mathics
Failed example:
    eqn.FindRoot(['x', 2])            # optional - mathics
Expected:
    {x -> 1.512134551657842}
Got:
    {x -> 1.51213}
**********************************************************************
File "src/sage/interfaces/mathics.py", line 223, in sage.interfaces.mathics
Failed example:
    print(g)                           # optional - mathics
Expected:
                               2       100       101    200
             100 + 315 x - 85 x  + 25 x    + 12 x    + x
Got:
    100 + 315 x - 85 x ^ 2 + 25 x ^ 100 + 12 x ^ 101 + x ^ 200
**********************************************************************
File "src/sage/interfaces/mathics.py", line 226, in sage.interfaces.mathics
Failed example:
    g                                  # optional - mathics
Expected:
    100 + 315*x - 85*x^2 + 25*x^100 + 12*x^101 + x^200
Got:
    100 + 315 x - 85 x ^ 2 + 25 x ^ 100 + 12 x ^ 101 + x ^ 200
**********************************************************************
File "src/sage/interfaces/mathics.py", line 228, in sage.interfaces.mathics
Failed example:
    print(g.Factor())                  # optional - mathics
Expected:
                          100               100
             (20 - 5 x + x   ) (5 + 17 x + x   )
Got:
    (5 + 17 x + x ^ 100) (20 - 5 x + x ^ 100)
**********************************************************************
File "src/sage/interfaces/mathics.py", line 235, in sage.interfaces.mathics
Failed example:
    print(f.Factor())                  # optional - mathics
Expected:
              3                  2    3
             x  (x - y) (-2 x + x  + y )
Got:
    x ^ 3 (x - y) (-2 x + x ^ 2 + y ^ 3)
**********************************************************************
File "src/sage/interfaces/mathics.py", line 275, in sage.interfaces.mathics
Failed example:
    print(x)                  # optional - mathics
Expected:
             Pi
             --
             2
Got:
    Pi / 2
**********************************************************************
File "src/sage/interfaces/mathics.py", line 327, in sage.interfaces.mathics
Failed example:
    math_bessel_K(2,I)                      # optional - mathics
Expected:
    -2.59288617549119697817 + 0.18048997206696202663*I
Got:
    -2.5928861754911969782 + 0.18048997206696202663 I
**********************************************************************
File "src/sage/interfaces/mathics.py", line 341, in sage.interfaces.mathics
Failed example:
    slist3 = mlist.sage(); slist3         # optional - mathics
Expected:
    [[1, 2], 3.00000000000000, I + 4]
Got:
    [[1, 2], 3.00000000000000, 4.00000000000000 + 1.00000000000000*I]
**********************************************************************
File "src/sage/interfaces/mathics.py", line 360, in sage.interfaces.mathics
Failed example:
    mathics('Pi/2').N(10)           # optional -- mathics
Expected:
    1.5707963268
Got:
    1.570796327
**********************************************************************
File "src/sage/interfaces/mathics.py", line 362, in sage.interfaces.mathics
Failed example:
    mathics('Pi').N(10)             # optional -- mathics
Expected:
    3.1415926536
Got:
    3.141592654
**********************************************************************
File "src/sage/interfaces/mathics.py", line 364, in sage.interfaces.mathics
Failed example:
    mathics('Pi').N(50)             # optional -- mathics
Expected:
    3.14159265358979323846264338327950288419716939937511
Got:
    3.1415926535897932384626433832795028841971693993751
**********************************************************************
File "src/sage/interfaces/mathics.py", line 366, in sage.interfaces.mathics
Failed example:
    str(mathics('Pi*x^2-1/2').N())  # optional -- mathics
Expected:
                    2
    -0.5 + 3.14159 x
Got:
    '-0.5 + 3.14159 x ^ 2.'
**********************************************************************
File "src/sage/interfaces/mathics.py", line 670, in sage.interfaces.mathics.Mathics.chdir
Failed example:
    mathics('Directory[]')      # optional - mathics
Expected:
    "/"
Got:
    /
**********************************************************************
File "src/sage/interfaces/mathics.py", line 1015, in sage.interfaces.mathics.MathicsElement._sage_
Failed example:
    s = m.sage(); s       # optional - mathics
Expected:
    [[1.00000000000000, 4], pi, 3.20000000000000*e100, I]
Got:
    [[1.00000000000000, 4], pi, 3.20000000000000*e100, 1.00000000000000*I]
**********************************************************************
File "src/sage/interfaces/mathics.py", line 1019, in sage.interfaces.mathics.MathicsElement._sage_
Failed example:
    s[3]^2                # optional - mathics
Expected:
    -1
Got:
    -1.00000000000000
**********************************************************************
File "src/sage/interfaces/mathics.py", line 1155, in sage.interfaces.mathics.MathicsElement._rich_repr_
Failed example:
    P._rich_repr_(dm)                              # optional - mathics
Expected:
    OutputImagePng container
Got:
    OutputImageSvg container
**********************************************************************
File "src/sage/interfaces/mathics.py", line 1195, in sage.interfaces.mathics.MathicsElement.show
Failed example:
    show(Q)                                    # optional - mathics
Expected:
    <html><script type="math/tex">\frac{\sin (x \cos (y))}{\sqrt{1-x^2}}</script></html>
Got:
    Sin[x Cos[y]] / Sqrt[1 - x ^ 2]
**********************************************************************

Observe that the graphical support of Mathics is restricted to SVG. For the usage in Sage it would be preferable to have PNG possible, as well, or to convert it here to PNG (for example using svg2rlg introducing a new dependency to svglib).

Sage package integration is initialized using:

sage -package create Mathics3 --pypi --type optional

I rename the created package Mathics3 to mathics to match Sage package naming conventions.

---

New commits:

​ea877d9	29009: initial version
​e7d6c32	29009: add comment to doctest
​d06d3d7	Merge branch 'u/soehms/mpmath_divmod_29009' of trac.sagemath.org:sage into mathics_interface
​3e82038	31676: initial version
​507eb62	Merge branch 'u/soehms/mpmath_hash' of trac.sagemath.org:sage into mathics_interface
​e621dea	31775: initial version
​84b8c14	Merge branch 'u/soehms/complex_interfaces_31775' of trac.sagemath.org:sage into mathics_interface
​85a3a0e	31778: initial version

[R. Bernstein]

Thanks for undertaking this, Sebastian Oehms!

Know that I'll help out whereever and however I can.

[Juan Mauricio Matera]

The problem with the order of terms and factors is that usually the evaluation of algebraic expressions is delegated to sympy, so we cannot from Mathics warrant a specific order. A way to make tests compatible with both backends (WMA and Mathics) could be to write tests that check that a difference between two equivalent expressions is zero.

Regarding format, it is possible to use an init file/ set of commands to overwrite the standard behavior of MakeBoxes? over Times or Power.

About images, notice that to produce png`s, you can load one of the optional backend modules: pymathics-matplotlib or pymathics-asymptote. To load the modules, you need to install one of the following libraries:

​https://github.com/mathics3/pymathics-matplotlib ​https://github.com/mathics3/pymathics-asymptote

and then run inside Mathics either LoadModule["pymathics-matplotlib"] or LoadModule["pymathics-asymptote"]. Both are work in progress, but produces pngs for basic plots and graphics.

[R. Bernstein]

Replying to gh-mmatera:

The problem with the order of terms and factors is that usually the evaluation of algebraic expressions is delegated to sympy, so we cannot from Mathics warrant a specific order.

This is related to something I noticed a while ago and have been meaning to fix. I thought I recorded it as an issue and I think I had a local branch for it, but I can't find the issue or branch. The closest I can find is ​https://github.com/mathics/Mathics/issues/1098 from Jan 3, 2021 and I have just renamed the title to make it more in line with what I have in mind.

We should be canonicalize output along the lines of standard formatting conventions and practice: if I have an expression in a variable, in StandardOutput format the higher powers of the variable come first in order and any constant comes last. It gets a little more complicated when there are combinations of variables.

Interestingly, on parsing we do canonicalize the input seen, and it is in the opposite order! But this is the way WL does it too.

However WL seems to canonicalize its StandardOutput too so you don't notice the reversal between what is parsed in input (as opposed to written by the user) and what is formatted on output.

Were this done, we would have both more natural looking, predicatble, and more WL-compatible output.

[Sebastian Oehms]

To be more clear about the differences to the Mathematica results in the doctests: These are not a real problem for this interface. I just listed them for documentation reason before I erased them. So, please don't spend too much time with this.

Concerning the PNG format: I will check this out using your optional backend modules. But, is there something standing against just converting SVG to PNG?

[R. Bernstein]

With respect to:

But note that the current stable release contains an incompatibility to mpmath running on a Sage backend

If it will help simplify things, (and personally I think it will), we can put out another release shortly. I generally would like to do so roughly every month that there are significant changes and I think in the last month there have been.

It is better to have many smaller releases as opposed to larger releases over a longer periods of time, especially as we are in a period of great flux and we are making great strides towards reducing the overall flakiness.

[Juan Mauricio Matera]

Replying to soehms:

To be more clear about the differences to the Mathematica results in the doctests: These are not a real problem for this interface. I just listed them for documentation reason before I erased them. So, please don't spend too much time with this.

Concerning the PNG format: I will check this out using your optional backend modules. But, is there something standing against just converting SVG to PNG?

Natively, mathics is able to produce asymptote and SVG graphics, with the first having much better support than the second. Then, to get a png we need to compile the asymptote code to produce the image. But to do that, we need to install a lot of dependencies. The other way around is to use matplotlib, but this requires a new implementation. This is why we have this in external packages.

[Sebastian Oehms]

Replying to gh-rocky:

If it will help simplify things, (and personally I think it will), we can put out another release shortly. I generally would like to do so roughly every month that there are significant changes and I think in the last month there have been.

That would be great! Thanks!

[Sebastian Oehms]

Replying to gh-mmatera:

Natively, mathics is able to produce asymptote and SVG graphics, with the first having much better support than the second. Then, to get a png we need to compile the asymptote code to produce the image. But to do that, we need to install a lot of dependencies. The other way around is to use matplotlib, but this requires a new implementation. This is why we have this in external packages.

I see! But, I think we can postpone the PNG availability to a follow up ticket. This is not essential to start with the interface. At least, if you convert to Sage objects you can do plotting there.

[R. Bernstein]

Replying to soehms:

Replying to gh-rocky:

If it will help simplify things, (and personally I think it will), we can put out another release shortly. I generally would like to do so roughly every month that there are significant changes and I think in the last month there have been.

That would be great! Thanks!

Ok tentatively then we can plan on a release the weekend of May 15-16, assuming that Mauricio is on board with this.

[R. Bernstein]

Replying to gh-rocky:

Replying to soehms:

Replying to gh-rocky:

If it will help simplify things, (and personally I think it will), we can put out another release shortly. I generally would like to do so roughly every month that there are significant changes and I think in the last month there have been.

That would be great! Thanks!

​Mathics3 2.2.0 has been released. Hopefully it addresses the problems with the last release. The next release should aslo add FindMinimum.

[Sebastian Oehms]

Replying to gh-rocky:

​Mathics3 2.2.0 has been released. Hopefully it addresses the problems with the last release.

Thanks! These problems are solved. But on sytsems where lxml is not present I'm getting a startup message lxml.html is not available..., now, which distrubes the first doctest:

File "src/sage/interfaces/mathics.py", line 37, in sage.interfaces.mathics
Failed example:
    mobj = mathics(x^2-1); mobj       # optional - mathics
Expected:
    -1 + x ^ 2
Got:
    lxml.html is not available...
    -1 + x ^ 2
**********************************************************************
1 item had failures:
   1 of  87 in sage.interfaces.mathics
    [222 tests, 1 failure, 9.45 s]

Is this something essential? If not: is there a way to supress this message in MathicsSession? Or, is there a missing dependency?

[R. Bernstein]

This dependency was newly added to the last release when an HTML import feature was added.

It shouldn't be necessary. Perhaps down the line, loading various types of files will moved to packages outside of the Mathics core.

When ​https://github.com/mathics/Mathics/pull/1379 lands, it should address this by making the lxml package optional. And next release we may have a FindMinimum as well.

For now though, I don't know what to suggest. Sigh.

Also, after this gets into sage, we should figure out how to incorporate sage testing into a broader set of tests we do. (We need this for other packages as well.)

[R. Bernstein]

I just thought of some more possibilities.

1. install lxml in sage. If sage already has something for parsing HTML, then maybe we can use that in the next release of Mathics.
2. If lxml is not installed, provide your own lxml import stub with submodule html and a parse function in it.

[Sebastian Oehms]

Replying to gh-rocky:

When ​https://github.com/mathics/Mathics/pull/1379 lands, it should address this by making the lxml package optional. And next release we may have a FindMinimum? as well.

Thats fine!

install lxml in sage ...

If lxml is optional in Mathics then it doesn't make sense to do anything in Sage (especially outside this interface) just to avoid this message. If a Sage user wants to use the optional functionality of Mathics via the interface then he can install it manually by sage -pip install lxml.

If sage already has something for parsing HTML, then maybe we can use that in the next release of Mathics.

AFAIK, html parsers that can be used per default in Sage are those of the following libraries:

sage: import html
sage: import html5lib

[R. Bernstein]

Ok. We'll go with one of those, probably html, which comes built in to Python in the version ranges we support, and most likely works just fine.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​5dca2f6	Merge branch 'u/soehms/mathics_interface' of trac.sagemath.org:sage into mathics_interface_31778
​1051c6f	31778: update Mathics3 to version 2.2.0

[Sebastian Oehms]

With the current commit there are three expected doctest failures (one concerning lxml and two with respect to FindMiminum) which sould disappear with the next Mathics release.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​529a722	Merge branch 'u/soehms/mathics_interface' of trac.sagemath.org:sage into mathics_interface_31778
​cdd4328	31778: upgrade to Mathics3 3.1.0

[Sebastian Oehms]

The upgrade to version 3.1.0 fixes the lxml issue, but not the one with FindMiminum.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a30242a

31778: dependency to mathics_scanner added

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​8cac399	Merge branch 'u/soehms/mathics_interface' of trac.sagemath.org:sage into mathics_interface_31778
​1565c92	31778: update to Mathics3 4.0.0
​a95d91c	31778: ignore FindMinimum doctests
​5e68c59	31778: add dependency importlib_metadata (because of Python 3.7)

[Sebastian Oehms]

The commits above contain an update to the current release of Mathics3 introducing two new dependencies to pint and palettable. Still, the Mathematica function FindMinimum is not present in this version. But anyway, I think this is not necessary for a start with the interface. Thus, I marked the corresponding doctests to be ignored.

I've started a ​workflow which is identical to tox-optional.yml except that the matrix for the optional packages is reduced to mathics.

It seems that there are no failures which are related to the new package except on debian buster one failing doctest on mathics.py. This was caused by the missing dependency to importlib_metadata which is needed for Sage on Python 3.7 and which is added in the current commit. Rerunning the ​debian buster test looks good now, as well.

Therefore, I think the ticket is ready for review, right now!

[Dima Pasechnik]

looks good to me, thanks

[Sebastian Oehms]

Replying to dimpase:

looks good to me, thanks

Many thanks, as well!