Opened 13 months ago
Closed 10 months ago
#29425 closed defect (duplicate)
Upgrade NumPy to 1.18.5, scipy to 1.4.1, add pybind11 package
Reported by:  mkoeppe  Owned by:  

Priority:  major  Milestone:  sageduplicate/invalid/wontfix 
Component:  porting  Keywords:  
Cc:  isuruf, dimpase, jdemeyer, embray, slelievre, vbraun  Merged in:  
Authors:  Matthias Koeppe  Reviewers:  Isuru Fernando, Dima Pasechnik 
Report Upstream:  N/A  Work issues:  
Branch:  u/mkoeppe/trac/local_conda_forge_macos_standard__build_error_with_numpy (Commits, GitHub, GitLab)  Commit:  56a6fe17708622cf6066629e7a5eeb2baead5508 
Dependencies:  #29480  Stopgaps: 
Description (last modified by )
Followup from the update of numpy to 1.16.6 (#29429):
We update
 NumPy to the latest release, 1.18.5
 scipy to latest release, 1.4.1.
(latest as of 20200605; both versions (and also the upcoming numpy 1.19) support python >= 3.6)
Change History (62)
comment:1 Changed 13 months ago by
comment:2 Changed 13 months ago by
This didn't make a difference
comment:3 Changed 13 months ago by
Upgrading numpy to 1.18.2 fixes it; but this will require a Cython upgrade too
comment:4 Changed 13 months ago by
 Branch set to u/mkoeppe/local_conda_forge_macos_standard__build_error_with_numpy
comment:5 Changed 13 months ago by
 Cc jdemeyer embray slelievre vbraun added
 Commit set to 6170b87af113609a593e422428ccfcdddab36c22
 Description modified (diff)
 Summary changed from localcondaforgemacosstandard: Build error with numpy to Upgrade Cython to 0.29.16, NumPy to 1.18.2 to fix numpy build error on localcondaforgemacosstandard
comment:6 Changed 13 months ago by
 Status changed from new to needs_review
comment:7 Changed 13 months ago by
comment:8 Changed 13 months ago by
Cython URL can be set to https://pypi.io/packages/source/C/Cython/CythonVERSION.tar.gz
comment:9 Changed 13 months ago by
 Commit changed from 6170b87af113609a593e422428ccfcdddab36c22 to 078a653c12a328ef2f87e1b5711ee90a082ce715
Branch pushed to git repo; I updated commit sha1. New commits:
078a653  build/pkgs/cython/checksums.ini: Use predictable URL for upstream_url pattern

comment:10 Changed 13 months ago by
Thanks!
comment:11 Changed 13 months ago by
 Reviewers set to Isuru Fernando
 Status changed from needs_review to positive_review
comment:12 Changed 13 months ago by
 Status changed from positive_review to needs_work
Breaks py2 builds
Traceback (most recent call last): File "setup.py", line 32, in <module> raise RuntimeError("Python version >= 3.5 required.") RuntimeError: Python version >= 3.5 required. }}
comment:13 Changed 13 months ago by
 Milestone changed from sage9.1 to sage9.2
 Status changed from needs_work to positive_review
Setting milestone to 9.2, which is presumably the first py3only release
comment:14 Changed 13 months ago by
I'll check whether an update to NumPy? 1.16.6 (supports py2) also fixes the problem.
comment:15 Changed 13 months ago by
That is #29429  Upgrade numpy to 1.16.6 (last release supporting py2)
comment:16 Changed 13 months ago by
 Dependencies set to dropping py2 support
 Description modified (diff)
 Priority changed from critical to major
 Summary changed from Upgrade Cython to 0.29.16, NumPy to 1.18.2 to fix numpy build error on localcondaforgemacosstandard to Upgrade Cython to 0.29.16, NumPy to 1.18.2 [py3 only!]
comment:17 Changed 12 months ago by
 Dependencies changed from dropping py2 support to dropping py2 support, #29480, #29429
 Status changed from positive_review to needs_work
 Work issues set to rebase on top of #29480, #29429
comment:18 Changed 12 months ago by
 Description modified (diff)
 Summary changed from Upgrade Cython to 0.29.16, NumPy to 1.18.2 [py3 only!] to Upgrade NumPy to 1.18.2 [py3 only!]
comment:19 Changed 11 months ago by
 Summary changed from Upgrade NumPy to 1.18.2 [py3 only!] to Upgrade NumPy to 1.18.4 [py3 only!]
comment:20 Changed 11 months ago by
 Summary changed from Upgrade NumPy to 1.18.4 [py3 only!] to Upgrade NumPy to 1.18.4, scipy to 1.4.1 [py3 only!]
comment:21 Changed 11 months ago by
 Commit changed from 078a653c12a328ef2f87e1b5711ee90a082ce715 to 204213debcb838ff4253ccd7f9ca40552a5b3bdb
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
a5c17a8  build/pkgs/cython: Update to 0.29.16

c7cc999  build/pkgs/cython/checksums.ini: Use predictable URL for upstream_url pattern

d54b715  Update Cython to 0.29.17

5958d38  build/pkgs/numpy: Update numpy to 1.18.2, back out numpy1.10.2nohardcodeblas.patch (does not apply)

0075beb  Update numpy to 1.18.4

204213d  Update scipy to 1.4.1

comment:22 Changed 11 months ago by
 Description modified (diff)
 Status changed from needs_work to needs_review
(untested)
comment:23 Changed 11 months ago by
 Dependencies changed from dropping py2 support, #29480, #29429 to dropping py2 support, #29480
comment:24 Changed 11 months ago by
 Work issues rebase on top of #29480, #29429 deleted
comment:25 Changed 11 months ago by
I am trying to use it. I had a first problem, it did not download the required version of cython, solved by downloading and putting it in upstream. The same happened for numpy, but it does not look for the package in upstream and failed. By some reason, giac also fail with a message like "undefined reference to `gmpn_add_nc'"
comment:26 Changed 11 months ago by
To enable automatic downloads, use ./configure enabledownloadfromupstreamurl
.
comment:27 followup: ↓ 28 Changed 11 months ago by
The giac build failure  is this in a build from scratch?
comment:28 in reply to: ↑ 27 Changed 11 months ago by
No, maybe it is a problem of mixing gcc9 and gcc10
Replying to mkoeppe:
The giac build failure  is this in a build from scratch?
comment:29 Changed 11 months ago by
Both numpy and scipy can be reached, but the following problem appears. If solved, I will try to build from scratch.
Installing scipy1.4.1 Download error on https://pypi.org/simple/pybind11/: [Errno 110] Connection timed out  Some packages may not be found! Couldn't find index page for 'pybind11' (maybe misspelled?)
comment:30 Changed 11 months ago by
 Status changed from needs_review to needs_work
Thanks for catching this. This is an additional dependency that we need to ship. https://github.com/scipy/scipy/blob/master/pyproject.toml
comment:31 Changed 11 months ago by
 Commit changed from 204213debcb838ff4253ccd7f9ca40552a5b3bdb to 14bcb1f25653b3baf78b1523913be0dc73257a95
Branch pushed to git repo; I updated commit sha1. New commits:
14bcb1f  build/pkgs/pybind11: New, dependency of scipy

comment:32 Changed 11 months ago by
 Summary changed from Upgrade NumPy to 1.18.4, scipy to 1.4.1 [py3 only!] to Upgrade NumPy to 1.18.4, scipy to 1.4.1, add pybind11 package [py3 only!]
comment:33 Changed 11 months ago by
 Status changed from needs_work to needs_review
comment:34 Changed 11 months ago by
I compiled from scratch using gcc10. Besides the expected failures of palp and sympow, these are the last lines of the log of scipy.
error: Command "gfortran Wall g ffixedform fnosecondunderscore fPIC fPIC O3 funrollloops Iscipy/sparse/linalg/eigen/arpack/ARPACK/SRC I/home/artal/sage/local/lib/python3.7/sitepackages/numpy/core/include c c scipy/sparse/linalg/eigen/arpack/ARPACK/SRC/dsaitr.f o build/temp.linuxx86_643.7/scipy/sparse/linalg/eigen/arpack/ARPACK/SRC/dsaitr.o" failed with exit status 1 Running setup.py install for scipy: finished with status 'error'
Cleaning up...
Removing source in /tmp/pipreqbuildyghxywzq
Removed build tracker '/tmp/pipreqtrackermaw6u14m' Command "/home/artal/sage/local/bin/python3 u c "import setuptools, tokenize;file='/tmp/pipreqbuildyghxywzq/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" nousercfg install record /tmp/piprecordaxtqqq0z/installrecord.txt singleversionexternallymanaged root /home/artal/sage/local/var/tmp/sage/build/scipy1.4.1/inst compile" failed with error code 1 in /tmp/pipreqbuildyghxywzq/ Exception information: Traceback (most recent call last):
File "/home/artal/sage/local/lib/python3.7/sitepackages/pip/_internal/cli/base_command.py", line 143, in main
status = self.run(options, args)
File "/home/artal/sage/local/lib/python3.7/sitepackages/pip/_internal/commands/install.py", line 366, in run
use_user_site=options.use_user_site,
File "/home/artal/sage/local/lib/python3.7/sitepackages/pip/_internal/req/init.py", line 49, in install_given_reqs
kwargs
File "/home/artal/sage/local/lib/python3.7/sitepackages/pip/_internal/req/req_install.py", line 791, in install
spinner=spinner,
File "/home/artal/sage/local/lib/python3.7/sitepackages/pip/_internal/utils/misc.py", line 705, in call_subprocess
% (command_desc, proc.returncode, cwd))
pip._internal.exceptions.InstallationError?: Command "/home/artal/sage/local/bin/python3 u c "import setuptools, tokenize;file='/tmp/pipreqbuildyghxywzq/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" nousercfg install record /tmp/piprecordaxtqqq0z/installrecord.txt singleversionexternallymanaged root /home/artal/sage/local/var/tmp/sage/build/scipy1.4.1/inst compile" failed with error code 1 in /tmp/pipreqbuildyghxywzq/ Error: installing with pip3 failed
comment:35 Changed 11 months ago by
what's the status of here? are we really waiting for gcc10 starting to work?
comment:36 Changed 11 months ago by
No, just waiting for review of this ticket and its dependency #29480.
comment:37 Changed 11 months ago by
I get doctest errors due to
DeprecationWarning: scipy.array is deprecated and will be removed in SciPy 2.0.0, use numpy.array instead
namely,
sage t warnlong 76.6 src/sage/graphs/generic_graph.py ********************************************************************** File "src/sage/graphs/generic_graph.py", line 9789, in sage.graphs.generic_graph.GenericGraph.? Failed example: G.pagerank(algorithm="Scipy") # abs tol 1e9 Expected: {1: 0.16112205885619563, 2: 0.1619531043247219, 3: 0.16112205885619563, 4: 0.2374999999999999, 5: 0.17775588228760858, 6: 0.100546895675278} Got: doctest:warning .... File "/home/scratch2/dimpase/sage/sage/local/lib64/python3.7/sitepackages/sage/doctest/forker.py", line 1104, in compile_and_execute exec(compiled, globs) File "<doctest sage.graphs.generic_graph.GenericGraph.?[7]>", line 1, in <module> G.pagerank(algorithm="Scipy") # abs tol 1e9 File "/home/scratch2/dimpase/sage/sage/local/lib64/python3.7/sitepackages/sage/graphs/generic_graph.py", line 9880, in pagerank dangling=dangling) File "/home/scratch2/dimpase/sage/sage/local/lib64/python3.7/sitepackages/networkx/algorithms/link_analysis/pagerank_alg.py", line 429, in pagerank_scipy S = scipy.array(M.sum(axis=1)).flatten() File "/home/scratch2/dimpase/sage/sage/local/lib64/python3.7/sitepackages/scipy/_lib/deprecation.py", line 19, in call stacklevel=stacklevel) File "/usr/lib64/python3.7/warnings.py", line 110, in _showwarnmsg msg.file, msg.line) : DeprecationWarning: scipy.array is deprecated and will be removed in SciPy 2.0.0, use numpy.array instead
it seems that these warnings are coming from scipy itself, or from networkx (which in turn calls scipy).
Let me see if upgrading networkx helps.
comment:38 Changed 11 months ago by
No, with networkx 2.4 one still gets these warnings, but they are fixed upstream, just not yet released, see https://github.com/networkx/networkx/commit/ea19baaff252468f5a911260430e057393d9637d
comment:39 Changed 11 months ago by
So we can patch networkx, or modify doctests, or suppress warning somehow (for the latter, I vaguely remember it's possible, but can't find it in docs)
comment:40 followups: ↓ 41 ↓ 45 Changed 11 months ago by
Filtering out doctest warnings was done on one of the recent glpk tickets
comment:41 in reply to: ↑ 40 Changed 11 months ago by
 Dependencies changed from dropping py2 support, #29480 to dropping py2 support, #29480, #29766
 Reviewers changed from Isuru Fernando to Isuru Fernando, Dima Pasechnik
 Status changed from needs_review to positive_review
comment:42 Changed 11 months ago by
 Dependencies changed from dropping py2 support, #29480, #29766 to #29480, #29766
comment:43 Changed 11 months ago by
 Branch changed from u/mkoeppe/local_conda_forge_macos_standard__build_error_with_numpy to u/dimpase/local_conda_forge_macos_standard__build_error_with_numpy
 Commit 14bcb1f25653b3baf78b1523913be0dc73257a95 deleted
 Dependencies changed from #29480, #29766 to #29480
drop the networkx dependency with the latest patch (and rebase over latest beta).
comment:44 Changed 11 months ago by
 Branch changed from u/dimpase/local_conda_forge_macos_standard__build_error_with_numpy to u/dimpase/trac/local_conda_forge_macos_standard__build_error_with_numpy
 Commit set to c005562fed79e91883fa808237f5d52ceb024fd3
New commits:
9a3617d  build/pkgs/cython: Update to 0.29.16

4a606b6  build/pkgs/cython/checksums.ini: Use predictable URL for upstream_url pattern

e9a7cb0  Update Cython to 0.29.17

15911cb  build/pkgs/numpy: Update numpy to 1.18.2, back out numpy1.10.2nohardcodeblas.patch (does not apply)

96e7a41  Update numpy to 1.18.4

78b4907  Update scipy to 1.4.1

54595b0  build/pkgs/pybind11: New, dependency of scipy

c005562  suppress scipy deprecation warnings

comment:45 in reply to: ↑ 40 Changed 11 months ago by
Replying to mkoeppe:
Filtering out doctest warnings was done on one of the recent glpk tickets
added the last commit to suppress the warning in the proper Pythonic way. The rest stays (mod rebase)
comment:46 Changed 10 months ago by
 Status changed from positive_review to needs_work
I'm getting various test failures
comment:47 followup: ↓ 49 Changed 10 months ago by
An upgrade ticket like this really needs to be run on the GitHub test infrastructure...
comment:48 Changed 10 months ago by
running GH Actions test here https://github.com/dimpase/sage/pull/9
comment:49 in reply to: ↑ 47 Changed 10 months ago by
Replying to mkoeppe:
An upgrade ticket like this really needs to be run on the GitHub test infrastructure...
Unfortunately there seem to be problems with runner setups, e.g. here https://github.com/dimpase/sage/runs/740349791?check_suite_focus=true
 this is
docker (archlinuxlatest, standardgcc_spkg)
I see
20200604T22:39:01.9688082Z Suggested packages: 20200604T22:39:01.9688790Z python3pytest pythonsetuptoolsdoc
and a lot of
20200605T03:08:41.2134392Z Failure: ModuleNotFoundError (No module named 'pytest') ... ERROR
which seems to suggest that python3pytest
is a requirement.
Do you know how to fix this?
(there are other errors pointing at internal problems of runners)
Can we easily extend setup to use our own runners?
comment:50 Changed 10 months ago by
comment:51 Changed 10 months ago by
In the meantime, numpy 1.18.5 has appeared
comment:52 Changed 10 months ago by
 Branch changed from u/dimpase/trac/local_conda_forge_macos_standard__build_error_with_numpy to u/mkoeppe/trac/local_conda_forge_macos_standard__build_error_with_numpy
comment:53 Changed 10 months ago by
 Commit changed from c005562fed79e91883fa808237f5d52ceb024fd3 to 56a6fe17708622cf6066629e7a5eeb2baead5508
Tests (with #29766) run at https://github.com/mkoeppe/sage/pull/38/checks
New commits:
56a6fe1  build/pkgs/numpy: Update to 1.18.5

comment:54 Changed 10 months ago by
 Description modified (diff)
 Summary changed from Upgrade NumPy to 1.18.4, scipy to 1.4.1, add pybind11 package [py3 only!] to Upgrade NumPy to 1.18.5, scipy to 1.4.1, add pybind11 package
comment:55 Changed 10 months ago by
 Status changed from needs_work to needs_review
comment:56 followup: ↓ 57 Changed 10 months ago by
comment:57 in reply to: ↑ 56 ; followup: ↓ 59 Changed 10 months ago by
comment:58 Changed 10 months ago by
OK, running a new test without #29766 at https://github.com/mkoeppe/sage/pull/39/checks
comment:59 in reply to: ↑ 57 Changed 10 months ago by
Replying to dimpase:
Replying to mkoeppe:
Can this ticket be tested successfully by itself or only as part of the
networkx
upgrade in #29766? If the latter, then let's close this ticket and merge it into #29766.no, this one does not need #29766
Actually, on ubuntuxenialstandard
(https://github.com/mkoeppe/sage/runs/745921419), I see lots of failures from the networkx
testsuite such as this one:
File "/sage/local/var/tmp/sage/build/networkx2.2/src/networkx/drawing/nx_pylab.py", line 249, in networkx.drawing.nx_pylab.draw_networkx Failed example: nx.draw(G) Exception raised: Traceback (most recent call last): File "/sage/local/lib/python3.7/doctest.py", line 1329, in __run compileflags, 1), test.globs) File "<doctest networkx.drawing.nx_pylab.draw_networkx[1]>", line 1, in <module> nx.draw(G) File "/sage/local/var/tmp/sage/build/networkx2.2/src/networkx/drawing/nx_pylab.py", line 126, in draw draw_networkx(G, pos=pos, ax=ax, **kwds) File "/sage/local/var/tmp/sage/build/networkx2.2/src/networkx/drawing/nx_pylab.py", line 278, in draw_networkx edge_collection = draw_networkx_edges(G, pos, arrows=arrows, **kwds) File "/sage/local/var/tmp/sage/build/networkx2.2/src/networkx/drawing/nx_pylab.py", line 611, in draw_networkx_edges if cb.is_numlike(alpha): AttributeError: module 'matplotlib.cbook' has no attribute 'is_numlike'
comment:60 Changed 10 months ago by
 Milestone changed from sage9.2 to sageduplicate/invalid/wontfix
Therefore, let's close this ticket and continue on #29766.
comment:61 Changed 10 months ago by
 Status changed from needs_review to positive_review
comment:62 Changed 10 months ago by
 Resolution set to duplicate
 Status changed from positive_review to closed
Can you try setting
NPY_DISTUTILS_APPEND_FLAGS=1
in the numpy build script?