Update matplotlib to 3.4

[arojas]

Causes some deprecation warnings due to set_dpi_cor. According to ​​https://github.com/matplotlib/matplotlib/commit/1843a484194c45d4724431fe2990ad3e31d21ca5 this was a no-op, so we just remove it.

[arojas]

---

New commits:

​282ff30	Remove call to deprecated set_dpi_cor
​c2e282b	Update matplotlib to 3.4.0

[fbissey]

At this stage, I don't think that will make it in 9.3 unless there was something critical about it. That being said, would matplotlib 3.3.x work just as well with that line removed?

[arojas]

Replying to fbissey:

At this stage, I don't think that will make it in 9.3 unless there was something critical about it. That being said, would matplotlib 3.3.x work just as well with that line removed?

Sure, I didn't expect it to - just didn't change the default Milestone value. I haven't tested, but given that the line in question didn't have any effect (according to the upstream commit I linked) I don't expect there to be any issues with 3.3.x

[fbissey]

I didn't really see your link earlier. Probably a bit too early in the morning for me to process information properly. In any case I was wondering if it would be OK to just squeeze that change in since it is a no-op anyway.

[arojas]

Replying to fbissey:

I didn't really see your link earlier. Probably a bit too early in the morning for me to process information properly. In any case I was wondering if it would be OK to just squeeze that change in since it is a no-op anyway.

As usual, I wrote in the comment box instead of the description. Yes, would be nice to remove that line for 9.3 - would save me some garbage output when doctesting.

[mkoeppe]

Sage development has entered the release candidate phase for 9.3. Setting a new milestone for this ticket based on a cursory review.

[mkoeppe]

3.4.2 is out

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​426b67a

Update matplotlib to 3.4.2

[fbissey]

LGTM

[vbraun]

Build tries to download additional source files:

  copying lib/matplotlib/backends/web_backend/all_figures.html -> build/lib.linux-x86_64-3.9/matplotlib/backends/web_backend
  copying lib/matplotlib/mpl-data/fonts/pdfcorefonts/ZapfDingbats.afm -> build/lib.linux-x86_64-3.9/matplotlib/mpl-data/fonts/pdfcorefonts
  copying lib/matplotlib/mpl-data/fonts/afm/pagd8a.afm -> build/lib.linux-x86_64-3.9/matplotlib/mpl-data/fonts/afm
  copying lib/matplotlib/mpl-data/images/qt4_editor_options.svg -> build/lib.linux-x86_64-3.9/matplotlib/mpl-data/images
  copying lib/matplotlib/mpl-data/images/zoom_to_rect.png -> build/lib.linux-x86_64-3.9/matplotlib/mpl-data/images
  UPDATING build/lib.linux-x86_64-3.9/matplotlib/_version.py
  set build/lib.linux-x86_64-3.9/matplotlib/_version.py to '3.4.2'
  running build_ext
  error: Failed to download any of the following: ['http://www.qhull.org/download/qhull-2020-src-8.0.2.tgz'].  Please download one of these urls and extract it into 'build/' at the top-level of the source repository.
  Building wheel for matplotlib (setup.py): finished with status 'error'
  ERROR: Failed building wheel for matplotlib

[arojas]

heh, so it looks like it no longer bundles qhull [1] so it will finally have to be made standard. Since this may take a while, I'll split off the sagelib part.

[1] ​https://github.com/matplotlib/matplotlib/commit/d81d15d368ec8b7fbdd3c0450d7d3ee96641ffe1

[arojas]

Done in #31827

[dimpase]

New commits:

​a2db12c	Update matplotlib to 3.4.0
​59e4fa9	Update matplotlib to 3.4.2
​9879b7c	build/pkgs/cmake/distros: Add alpine, arch, gentoo, nix
​de6fde8	build/pkgs/cmake: Update to 3.21.0
​a8cbd93	build/pkgs/cmake/type: Change to standard
​0c566dd	Merge branch 'u/mkoeppe/make_cmake_a_standard_package' of trac.sagemath.org:sage into u/dimpase/packages/matplotlib/3_4
​00522ef	qhull is a dependency

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​9c80f38

qhull standard, as a dep of matplotlib

[dimpase]

tests are running on ​https://github.com/sagemath/sagetrac-mirror/actions/runs/1040329661

[mkoeppe]

I haven't looked at the details, but it looks like our qhull package is outdated - ​https://repology.org/project/qhull/versions

[dimpase]

errors about inability to build wheels for mathplotlib.

[dimpase]

one needs to unconditionally allow qhull from the system in the python script generat8ng .cfg file.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​3ff41ca	Update matplotlib to 3.4.0
​8cf68f6	Update matplotlib to 3.4.2
​f3e1b07	build/pkgs/cmake/distros: Add alpine, arch, gentoo, nix
​79cc7b9	build/pkgs/cmake: Update to 3.21.0
​94ffa8f	build/pkgs/cmake/type: Change to standard
​13ba8a8	qhull is a dependency
​6e76e02	qhull standard, as a dep of matplotlib
​c360d37	do not vendor ghull, unconditionally

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​cbf8f4b	31161: Upgrade: qhull 2020-src-8.0.2
​fa99c20	added basic spkg-configure.m4
​4ed8c39	Update matplotlib to 3.4.0
​b2a0599	Update matplotlib to 3.4.2
​a2f910d	build/pkgs/cmake/distros: Add alpine, arch, gentoo, nix
​14bcb58	build/pkgs/cmake: Update to 3.21.0
​4c91e5c	build/pkgs/cmake/type: Change to standard
​54472c1	qhull is a dependency
​189ca21	qhull standard, as a dep of matplotlib
​9f94876	do not vendor ghull, unconditionally

[dimpase]

OK, this should fix the issue. TOX tests on ​https://github.com/sagemath/sagetrac-mirror/actions/runs/1045348934

[dimpase]

There is a race condition in docbuild which leads to this GH Actions run erroring out on many systems, as follows:

  [dochtml]   [reference]     topology: 1937 js index entries
  [dochtml]   [reference]     valuations: 973 js index entries
  [dochtml]   [reference] ... done (64387 js index entries)
  [dochtml]   [reference] copying static files... failed
  [dochtml]   [reference] WARNING: cannot copy static file FileExistsError(17, 'File exists')
  [dochtml]   [reference] dumping search index in English (code: en)... done
  [dochtml]   [reference] The HTML pages are in .tox/local-homebrew-macos-standard/local/share/doc/sage/html/en/reference.
  [dochtml]   Error building the documentation.
  [dochtml]   Traceback (most recent call last):
  [dochtml]     File "/Users/runner/work/sagetrac-mirror/sagetrac-mirror/.tox/local-homebrew-macos-standard/homebrew/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
  [dochtml]       return _run_code(code, main_globals, None,
  [dochtml]     File "/Users/runner/work/sagetrac-mirror/sagetrac-mirror/.tox/local-homebrew-macos-standard/homebrew/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
  [dochtml]       exec(code, run_globals)
  [dochtml]     File "/Users/runner/work/sagetrac-mirror/sagetrac-mirror/.tox/local-homebrew-macos-standard/local/lib/python3.9/site-packages/sage_docbuild/__main__.py", line 2, in <module>
  [dochtml]       main()
  [dochtml]     File "/Users/runner/work/sagetrac-mirror/sagetrac-mirror/.tox/local-homebrew-macos-standard/local/lib/python3.9/site-packages/sage_docbuild/__init__.py", line 1813, in main
  [dochtml]       builder()
  [dochtml]     File "/Users/runner/work/sagetrac-mirror/sagetrac-mirror/.tox/local-homebrew-macos-standard/local/lib/python3.9/site-packages/sage_docbuild/__init__.py", line 133, in f
  [dochtml]       runsphinx()
  [dochtml]     File "/Users/runner/work/sagetrac-mirror/sagetrac-mirror/.tox/local-homebrew-macos-standard/local/lib/python3.9/site-packages/sage_docbuild/sphinxbuild.py", line 323, in runsphinx
  [dochtml]       sys.stderr.raise_errors()
  [dochtml]     File "/Users/runner/work/sagetrac-mirror/sagetrac-mirror/.tox/local-homebrew-macos-standard/local/lib/python3.9/site-packages/sage_docbuild/sphinxbuild.py", line 258, in raise_errors
  [dochtml]       raise OSError(self._error)
  [dochtml]   OSError: WARNING: cannot copy static file FileExistsError(17, 'File exists')
  [dochtml]   
  [dochtml]       Note: incremental documentation builds sometimes cause spurious
  [dochtml]       error messages. To be certain that these are real errors, run
  [dochtml]       "make doc-clean" first and try again.
  [dochtml]   make[3]: *** [doc-html--reference_top] Error 1
  [dochtml]   make[2]: *** [doc-html-reference] Error 2
  [dochtml]   make[2]: Target `doc-html' not remade because of errors.
  [dochtml] Full log file: /Users/runner/work/sagetrac-mirror/sagetrac-mirror/logs/dochtml.log
make[1]: *** [doc-html] Error 2

this was altready reported on ​https://groups.google.com/g/sage-devel/c/ts6u19KpapA/m/22BdxXZhBAAJ

[klee]

From #32262:

Replying to dimpase:

Replying to klee:

What is the command to reproduce the bug?

try building the branch of #31580 with make -jX (for X=8, say?) on macOS or Ubuntu.

Building fails with me:

[matplotlib-3.4.2] ERROR [transfer|run:135]: [Errno 404] Not Found: '//sagepad.org/spkg/upstream/matplotlib/matplotlib-3.4.2.tar.gz'
[matplotlib-3.4.2] ************************************************************************
[matplotlib-3.4.2] Traceback (most recent call last):
...
[matplotlib-3.4.2]     raise FileNotMirroredError('tarball does not exist on mirror network')
[matplotlib-3.4.2] FileNotMirroredError: tarball does not exist on mirror network

[dimpase]

Of course the tarballs are not on the mirrors yet, this only happens after the ticket is merged etc.

You need to

./configure --enable-download-from-upstream-url

Then the URLs in checksum.ini files will be used if nothing is found on the mirrors. Due to this feature (crucial for CI), nobody lists explicit tarballs in ticket descriptions any more. (Or look at checksum.ini, find the URL, manually upload the tarfile to upstream/)

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​98652ec	31161: Add qhull distro info
​26a5aa7	31161: Upgrade: qhull 2020-src-8.0.2
​15ee2d7	added basic spkg-configure.m4
​50748e7	Update matplotlib to 3.4.0
​20d4fc4	Update matplotlib to 3.4.2
​af27c8b	qhull is a dependency
​cd4a0f2	qhull standard, as a dep of matplotlib
​0a0856e	do not vendor ghull, unconditionally
​8295905	upstream patch to recitfy docbuild problem

[mkoeppe]

3.4.3 is out

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​c3c3108	Update matplotlib to 3.4.2
​a67736e	qhull is a dependency
​f424c41	qhull standard, as a dep of matplotlib
​936203d	do not vendor ghull, unconditionally
​13912ac	upstream patch to recitfy docbuild problem
​78c663a	update to 3.4.3
​8cdfdac	31161: Add qhull distro info
​8c5cd81	31161: Upgrade: qhull 2020-src-8.0.2
​47df756	added basic spkg-configure.m4
​f958b71	Merge branch 'public/31161' of trac.sagemath.org:sage into u/dimpase/packages/matplotlib/3_4

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​5f2eaad

Revert "upstream patch to recitfy docbuild problem"

[dimpase]

updated to 3.4.3.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​d30ca7d	31161: Upgrade: qhull 2020-src-8.0.2
​de66f70	added basic spkg-configure.m4
​2f044ee	Update matplotlib to 3.4.0
​4e9ecc4	Update matplotlib to 3.4.2
​9a3d7a3	qhull is a dependency
​00b580a	qhull standard, as a dep of matplotlib
​248bc95	do not vendor ghull, unconditionally
​97faa41	upstream patch to recitfy docbuild problem
​12fcaa7	update to 3.4.3
​df91ddc	Revert "upstream patch to recitfy docbuild problem"

[dimpase]

rebased over 9.5.beta1

[dimpase]

tests on ​https://github.com/sagemath/sagetrac-mirror/actions/runs/1233039865

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​64f613a	added basic spkg-configure.m4
​0939967	correct distros info
​78184d4	Update matplotlib to 3.4.0
​696978f	Update matplotlib to 3.4.2
​992f989	qhull is a dependency
​18fddf8	qhull standard, as a dep of matplotlib
​fa16032	do not vendor ghull, unconditionally
​d45933b	upstream patch to recitfy docbuild problem
​862e656	update to 3.4.3
​df8d17c	Revert "upstream patch to recitfy docbuild problem"

[dimpase]

Replying to dimpase:

tests on ​https://github.com/sagemath/sagetrac-mirror/actions/runs/1233039865

should be ​https://github.com/sagemath/sagetrac-mirror/actions/runs/1233224818

​https://github.com/sagemath/sagetrac-mirror/actions/runs/1235471874

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​7e7e983	correct package name on debian
​e073d08	replace tabs by spaces, adjust spacing
​094592e	Update matplotlib to 3.4.0
​33e9833	Update matplotlib to 3.4.2
​a004b06	qhull is a dependency
​6d150f1	qhull standard, as a dep of matplotlib
​fcb3fc7	do not vendor ghull, unconditionally
​70876f6	upstream patch to recitfy docbuild problem
​b879882	update to 3.4.3
​382de1b	Revert "upstream patch to recitfy docbuild problem"

[dimpase]

tests are done, please have a look

[mkoeppe]

Also the dependencies of matplotlib: cycler, kiwisolver, ... are outdated

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​6cd3723	Update matplotlib to 3.4.0
​4d742f2	Update matplotlib to 3.4.2
​e406102	qhull is a dependency
​51ba6f3	qhull standard, as a dep of matplotlib
​8669821	do not vendor ghull, unconditionally
​f999e9b	upstream patch to recitfy docbuild problem
​af761fa	update to 3.4.3
​24b7a9a	Revert "upstream patch to recitfy docbuild problem"
​42435f4	add gfortran to _prereq/distros/*
​8af511c	rebase, update deps

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​5c972e4

correct checksum

[dimpase]

and more new deps are coming, such as cppy... hold tight

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​00b677b	new dep of kiwisolver, cppy
​c7ab8fe	update matplotlib deps

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​48b7ec3

update matplotlib deps

[mkoeppe]

cppy is missing SPKG.rst. Consider using sage --package create --pypi cppy to create packages...

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​15369f9

fix SPKG.rst's for cppy and kiwisolver

[mkoeppe]

Let's get rid of these "Dependencies" sections in SPKG.rst please if they just duplicate what's in the dependencies file.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​26910e6

fix SPKG.rst's for cppy and kiwisolver

[dimpase]

New commits:

​26910e6

fix SPKG.rst's for cppy and kiwisolver

---

New commits:

​26910e6

fix SPKG.rst's for cppy and kiwisolver

[mkoeppe]

diff --git a/build/pkgs/_prereq/distros/debian.txt b/build/pkgs/_prereq/distros/debian.txt
index 50be6ac..c348ee3 100644
--- a/build/pkgs/_prereq/distros/debian.txt
+++ b/build/pkgs/_prereq/distros/debian.txt
@@ -26,5 +26,6 @@ gcc
 # configure: error: in `/sage':
 # configure: error: C++ preprocessor "/lib/cpp" fails sanity check
 g++
+gfortran
 # Needed if we download some packages from a https upstream URL
 ca-certificates

Please don't sneak these changes in.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​aedaf93

Revert "add gfortran to _prereq/distros/*"

[dimpase]

oops, sorry, should be OK now.

[mkoeppe]

--- a/build/pkgs/cycler/checksums.ini
+++ b/build/pkgs/cycler/checksums.ini
@@ -1,4 +1,5 @@
 tarball=cycler-VERSION.tar.gz
-sha1=1bcf4a985a5f414333603f3f70a7deebbaf30439
-md5=4cb42917ac5007d1cdff6cccfe2d016b
-cksum=4189438538
+sha1=ea144185cf1d54d7415e3c66d705d9a232e57da5
+md5=cf3b4846fded12fa4d2a7a291a5d6e13
+cksum=3626319704
+upstream_url=https://github.com/matplotlib/cycler/archive/refs/tags/vVERSION.tar.gz

Please use PyPI URL, not a github archive

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​62792d4

use pypi host for cycler url

[arojas]

In the meantime, 3.5 is released with new dependencies (fonttools) and new deprecations. I've opened #32909 to fix these.

[dimpase]

OK, so can this ticket get in, please?

[tornaria]

Note that matplotlib 3.5 is needed to avoid distutils deprecation warnings with python 3.10 (see ​https://trac.sagemath.org/ticket/30766#comment:68).

The actual commit fixing the deprecation warning is ​https://github.com/matplotlib/matplotlib/commit/7d77bb96707f10032d8fedf2bd225491f9a28afe

[dimpase]

Are we aiming at a full support of Python 3.10 in 9.5? If so, I guess we should go straight to matplotlib 3.5.

[mkoeppe]

Replying to dimpase:

Are we aiming at a full support of Python 3.10 in 9.5?

Not sure if we can/should. I think it's better to have a release very soon because of the glibc 2.34 issue (#32576), which lots of people are already running into.

[mkoeppe]

Replying to tornaria:

Note that matplotlib 3.5 is needed to avoid distutils deprecation warnings with python 3.10

Opened #33040 for this

[vbraun]

On OSX:

**********************************************************************
File "src/sage/plot/plot3d/list_plot3d.py", line 380, in sage.plot.plot3d.list_plot3d.list_plot3d_tuples
Failed example:
    list_plot3d(m, color='yellow', interpolation_type='linear', num_points=5) # indirect doctest
Exception raised:
    Traceback (most recent call last):
      File "/Users/buildbot-sage/worker/sage_git/build/local/var/lib/sage/venv-python3.9.9/lib/python3.9/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/buildbot-sage/worker/sage_git/build/local/var/lib/sage/venv-python3.9.9/lib/python3.9/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.plot.plot3d.list_plot3d.list_plot3d_tuples[2]>", line 1, in <module>
        list_plot3d(m, color='yellow', interpolation_type='linear', num_points=Integer(5)) # indirect doctest
      File "/Users/buildbot-sage/worker/sage_git/build/local/var/lib/sage/venv-python3.9.9/lib/python3.9/site-packages/sage/plot/plot3d/list_plot3d.py", line 188, in list_plot3d
        return list_plot3d_tuples(data, interpolation_type, **kwds)
      File "/Users/buildbot-sage/worker/sage_git/build/local/var/lib/sage/venv-python3.9.9/lib/python3.9/site-packages/sage/plot/plot3d/list_plot3d.py", line 453, in list_plot3d_tuples
        T = tri.Triangulation(x, y)
      File "/Users/buildbot-sage/worker/sage_git/build/local/var/lib/sage/venv-python3.9.9/lib/python3.9/site-packages/matplotlib/tri/triangulation.py", line 39, in __init__
        from matplotlib import _qhull
    ImportError: dlopen(/Users/buildbot-sage/worker/sage_git/build/local/var/lib/sage/venv-python3.9.9/lib/python3.9/site-packages/matplotlib/_qhull.cpython-39-darwin.so, 2): Library not loaded: lib/libqhull_r.8.0.dylib
      Referenced from: /Users/buildbot-sage/worker/sage_git/build/local/var/lib/sage/venv-python3.9.9/lib/python3.9/site-packages/matplotlib/_qhull.cpython-39-darwin.so
      Reason: image not found
**********************************************************************
1 item had failures:
   1 of   7 in sage.plot.plot3d.list_plot3d.list_plot3d_tuples
    [48 tests, 1 failure, 4.36 s]
----------------------------------------------------------------------
sage -t --long --random-seed=269285139450068045496865355624462748906 src/sage/plot/plot3d/list_plot3d.py  # 1 doctest failed
----------------------------------------------------------------------

[fbissey]

Looks like some wrong install_name was used on that qhull install. Is qhull supposed to come from conda, brew or sage internals on OSX during those testruns?

It is a bit strange it wasn't seen before.

[fbissey]

OK, qhull used to be optional and has been switched to standard with this release. Unfortunately boiler plate sdh_cmake doesn't set install_name properly for qhull. I am not sure how it all install in the proper places but to get the right rpath and install_name, -DLIB_INSTALL_DIR=... has to be passed to camke when configuring.

I'll fix that ASAP. This would affect the further upgrade to 3.5.1.

[fbissey]

Should work now.

---

New commits:

​0018e34	Merge branch 'develop' into MPL3_4
​0e5c130	Specify custom variable LIB_INSTALL_DIR for qhull, it controls rpath and install_name settings on OSX

[mkoeppe]

Also cppy is missing install-requires.txt.

[mkoeppe]

New commits:

​475a26a

build/pkgs/cppy/install-requires.txt: New