Huge speed regression in some tests after matplotlib upgrade

[jdemeyer]

On this machine

Linux sardonis 4.4.0-57-generic #78-Ubuntu SMP Fri Dec 9 23:46:13 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux

this test now takes 926 seconds

sage -t --long --warn-long 10.0 src/sage/plot/plot3d/list_plot3d.py
**********************************************************************
File "src/sage/plot/plot3d/list_plot3d.py", line 143, in sage.plot.plot3d.list_plot3d.list_plot3d
Warning, slow doctest:
    list_plot3d(l, interpolation_type='clough', texture='yellow', num_points=100)
Test ran for 926.47 s
**********************************************************************

Before the matplotlib upgrade at #23696, this took about 3 seconds.

[jdemeyer]

OK, let's move on to do something about it. Is there any other plots that are taking an awful long time?

Yes, several others with interpolation="clough" take hundreds of seconds. I'll give you more details later today.

[fbissey]

The easiest speed up is to replace clough by linear but plot will actually lose in "quality". clough gives very nice and smooth plots. I think nn may have been caching an intermediate step that speeds things considerably when a lot of points are involved.

[jdemeyer]

These are the slow tests from running ./sage -tp --long --warn-long 10 src/sage/plot. I'm not saying that all these are because of the matplotlib upgrade.

File "src/sage/plot/plot3d/tachyon.py", line 42, in sage.plot.plot3d.tachyon
Warning, slow doctest:
    t.show() # long time
Test ran for 67.29 s

File "src/sage/plot/plot3d/plot3d.py", line 1231, in sage.plot.plot3d.plot3d.spherical_plot3d
Warning, slow doctest:
    (rea + ima).show(aspect_ratio=1)  # long time (4s on sage.math, 2011)
Test ran for 48.94 s

File "src/sage/plot/graphics.py", line 1576, in sage.plot.graphics.Graphics.show
Warning, slow doctest:
    p.show(figsize=[327,10],dpi=100)
Test ran for 11.47 s

File "src/sage/plot/plot3d/list_plot3d.py", line 143, in sage.plot.plot3d.list_plot3d.list_plot3d
Warning, slow doctest:
    list_plot3d(l, interpolation_type='clough', texture='yellow', num_points=100)
Test ran for 933.90 s

File "src/sage/plot/plot3d/list_plot3d.py", line 391, in sage.plot.plot3d.list_plot3d.list_plot3d_tuples
Warning, slow doctest:
    list_plot3d([(1, 2, 3), (0, 1, 3), (2, 1, 4), (1, 0, -2)], texture='yellow', num_points=50)
Test ran for 323.90 s

File "src/sage/plot/plot3d/transform.pyx", line 236, in sage.plot.plot3d.transform.rotate_arbitrary
Warning, slow doctest:
    m = m.parent()([x.simplify_full() for x in m._list()])
Test ran for 50.80 s

[fbissey]

OK.

Replying to jdemeyer:

These are the slow tests from running ./sage -tp --long --warn-long 10 src/sage/plot. I'm not saying that all these are because of the matplotlib upgrade.

File "src/sage/plot/plot3d/tachyon.py", line 42, in sage.plot.plot3d.tachyon
Warning, slow doctest:
    t.show() # long time
Test ran for 67.29 s

Probably not the upgrade, just tachyon being slow.

File "src/sage/plot/plot3d/plot3d.py", line 1231, in sage.plot.plot3d.plot3d.spherical_plot3d
Warning, slow doctest:
    (rea + ima).show(aspect_ratio=1)  # long time (4s on sage.math, 2011)
Test ran for 48.94 s

That warrants examination in whatever case. 4s in 2011 to 49s in 2018 with probably newer hardware.

File "src/sage/plot/graphics.py", line 1576, in sage.plot.graphics.Graphics.show
Warning, slow doctest:
    p.show(figsize=[327,10],dpi=100)
Test ran for 11.47 s

Uncertain.

File "src/sage/plot/plot3d/list_plot3d.py", line 143, in sage.plot.plot3d.list_plot3d.list_plot3d
Warning, slow doctest:
    list_plot3d(l, interpolation_type='clough', texture='yellow', num_points=100)
Test ran for 933.90 s

File "src/sage/plot/plot3d/list_plot3d.py", line 391, in sage.plot.plot3d.list_plot3d.list_plot3d_tuples
Warning, slow doctest:
    list_plot3d([(1, 2, 3), (0, 1, 3), (2, 1, 4), (1, 0, -2)], texture='yellow', num_points=50)
Test ran for 323.90 s

Those two are prime suspect as they both use clough in the second case because it is the default.

File "src/sage/plot/plot3d/transform.pyx", line 236, in sage.plot.plot3d.transform.rotate_arbitrary
Warning, slow doctest:
    m = m.parent()([x.simplify_full() for x in m._list()])
Test ran for 50.80 s

Nothing involving MPL directly, this is probably something else. That's certainly a long test.

How old is your hardware that's what I get locally

sage -t --long --warn-long 72.2 /usr/lib64/python2.7/site-packages/sage/plot/plot3d/list_plot3d.py
    [43 tests, 53.07 s]

[jdemeyer]

This is my POWER 8 ppc64le machine. So not old, but perhaps unusual.

[fbissey]

That explains things. POWER chips are notoriously bad at interpreted languages, my power7 boxes would be giving similar results.

[jdemeyer]

Tests 1, 2 and 6 in that list are also slow on Sage 8.1. So they are not caused by matplotlib.

[jdemeyer]

Regardless of hardware, it's still a regression of a factor 300.

[jdemeyer]

Do you happen to know how doctests "plot" stuff? I am asking because I'm trying to reproduce the problem outside the doctesting framework but I don't know how.

[jdemeyer]

Duplicate of #13135

[vbraun]

For normal plots you can iterate over the elements, which is usually a reasonably detailed textual representation

sage: g1 = plot([x*exp(-n*x^2)/.4 for n in [1..3]], (0, 2), color='blue', aspect_ratio=.8)
sage: list(g1)
[Line defined by 210 points,
 Line defined by 221 points,
 Line defined by 219 points]

There is no similar solution for 3d plots afair.

Reducing the output resolution of tachyon would be fine with me; we don't test anything by raytracing big images instead of small ones except possibly for our patience.