#29425 closed defect (duplicate)

Upgrade NumPy to 1.18.5, scipy to 1.4.1, add pybind11 package

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-duplicate/invalid/wontfix
Component: porting Keywords:
Cc: isuruf, dimpase, jdemeyer, embray, slelievre, vbraun Merged in:
Authors: Matthias Koeppe Reviewers: Isuru Fernando, Dima Pasechnik
Report Upstream: N/A Work issues:
Branch: u/mkoeppe/trac/local_conda_forge_macos_standard__build_error_with_numpy (Commits, GitHub, GitLab) Commit: 56a6fe17708622cf6066629e7a5eeb2baead5508
Dependencies: #29480 Stopgaps:

Status badges

Description (last modified by mkoeppe)

Follow-up from the update of numpy to 1.16.6 (#29429):

We update

  • NumPy to the latest release, 1.18.5
  • scipy to latest release, 1.4.1.

(latest as of 2020-06-05; both versions (and also the upcoming numpy 1.19) support python >= 3.6)

Change History (62)

comment:1 Changed 13 months ago by isuruf

Can you try setting NPY_DISTUTILS_APPEND_FLAGS=1 in the numpy build script?

comment:2 Changed 13 months ago by mkoeppe

This didn't make a difference

comment:3 Changed 13 months ago by mkoeppe

Upgrading numpy to 1.18.2 fixes it; but this will require a Cython upgrade too

comment:4 Changed 13 months ago by mkoeppe

  • Branch set to u/mkoeppe/local_conda_forge_macos_standard__build_error_with_numpy

comment:5 Changed 13 months ago by mkoeppe

  • Authors set to Matthias Koeppe
  • Cc jdemeyer embray slelievre vbraun added
  • Commit set to 6170b87af113609a593e422428ccfcdddab36c22
  • Description modified (diff)
  • Summary changed from local-conda-forge-macos-standard: Build error with numpy to Upgrade Cython to 0.29.16, NumPy to 1.18.2 to fix numpy build error on local-conda-forge-macos-standard

New commits:

adf988dbuild/pkgs/cython: Update to 0.29.16
6170b87build/pkgs/numpy: Update numpy to 1.18.2, back out numpy-1.10.2-no-hardcode-blas.patch (does not apply)

comment:6 Changed 13 months ago by mkoeppe

  • Status changed from new to needs_review

comment:8 Changed 13 months ago by isuruf

Cython URL can be set to https://pypi.io/packages/source/C/Cython/Cython-VERSION.tar.gz

comment:9 Changed 13 months ago by git

  • Commit changed from 6170b87af113609a593e422428ccfcdddab36c22 to 078a653c12a328ef2f87e1b5711ee90a082ce715

Branch pushed to git repo; I updated commit sha1. New commits:

078a653build/pkgs/cython/checksums.ini: Use predictable URL for upstream_url pattern

comment:10 Changed 13 months ago by mkoeppe

Thanks!

comment:11 Changed 13 months ago by isuruf

  • Reviewers set to Isuru Fernando
  • Status changed from needs_review to positive_review

comment:12 Changed 13 months ago by mkoeppe

  • Status changed from positive_review to needs_work

Breaks py2 builds

Traceback (most recent call last):
  File "setup.py", line 32, in <module>
    raise RuntimeError("Python version >= 3.5 required.")
RuntimeError: Python version >= 3.5 required.
}}

comment:13 Changed 13 months ago by mkoeppe

  • Milestone changed from sage-9.1 to sage-9.2
  • Status changed from needs_work to positive_review

Setting milestone to 9.2, which is presumably the first py3-only release

comment:14 Changed 13 months ago by mkoeppe

I'll check whether an update to NumPy? 1.16.6 (supports py2) also fixes the problem.

comment:15 Changed 13 months ago by mkoeppe

That is #29429 - Upgrade numpy to 1.16.6 (last release supporting py2)

comment:16 Changed 13 months ago by mkoeppe

  • Dependencies set to dropping py2 support
  • Description modified (diff)
  • Priority changed from critical to major
  • Summary changed from Upgrade Cython to 0.29.16, NumPy to 1.18.2 to fix numpy build error on local-conda-forge-macos-standard to Upgrade Cython to 0.29.16, NumPy to 1.18.2 [py3 only!]

comment:17 Changed 12 months ago by mkoeppe

  • Dependencies changed from dropping py2 support to dropping py2 support, #29480, #29429
  • Status changed from positive_review to needs_work
  • Work issues set to rebase on top of #29480, #29429

comment:18 Changed 12 months ago by mkoeppe

  • Description modified (diff)
  • Summary changed from Upgrade Cython to 0.29.16, NumPy to 1.18.2 [py3 only!] to Upgrade NumPy to 1.18.2 [py3 only!]

comment:19 Changed 11 months ago by mkoeppe

  • Summary changed from Upgrade NumPy to 1.18.2 [py3 only!] to Upgrade NumPy to 1.18.4 [py3 only!]

comment:20 Changed 11 months ago by mkoeppe

  • Summary changed from Upgrade NumPy to 1.18.4 [py3 only!] to Upgrade NumPy to 1.18.4, scipy to 1.4.1 [py3 only!]

comment:21 Changed 11 months ago by git

  • Commit changed from 078a653c12a328ef2f87e1b5711ee90a082ce715 to 204213debcb838ff4253ccd7f9ca40552a5b3bdb

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

a5c17a8build/pkgs/cython: Update to 0.29.16
c7cc999build/pkgs/cython/checksums.ini: Use predictable URL for upstream_url pattern
d54b715Update Cython to 0.29.17
5958d38build/pkgs/numpy: Update numpy to 1.18.2, back out numpy-1.10.2-no-hardcode-blas.patch (does not apply)
0075bebUpdate numpy to 1.18.4
204213dUpdate scipy to 1.4.1

comment:22 Changed 11 months ago by mkoeppe

  • Description modified (diff)
  • Status changed from needs_work to needs_review

(untested)

comment:23 Changed 11 months ago by mkoeppe

  • Dependencies changed from dropping py2 support, #29480, #29429 to dropping py2 support, #29480

comment:24 Changed 11 months ago by mkoeppe

  • Work issues rebase on top of #29480, #29429 deleted

comment:25 Changed 11 months ago by enriqueartal

I am trying to use it. I had a first problem, it did not download the required version of cython, solved by downloading and putting it in upstream. The same happened for numpy, but it does not look for the package in upstream and failed. By some reason, giac also fail with a message like "undefined reference to `gmpn_add_nc'"

comment:26 Changed 11 months ago by mkoeppe

To enable automatic downloads, use ./configure --enable-download-from-upstream-url.

comment:27 follow-up: Changed 11 months ago by mkoeppe

The giac build failure - is this in a build from scratch?

comment:28 in reply to: ↑ 27 Changed 11 months ago by enriqueartal

No, maybe it is a problem of mixing gcc9 and gcc10

Replying to mkoeppe:

The giac build failure - is this in a build from scratch?

comment:29 Changed 11 months ago by enriqueartal

Both numpy and scipy can be reached, but the following problem appears. If solved, I will try to build from scratch.

Installing scipy-1.4.1 Download error on https://pypi.org/simple/pybind11/: [Errno 110] Connection timed out -- Some packages may not be found! Couldn't find index page for 'pybind11' (maybe misspelled?)

comment:30 Changed 11 months ago by mkoeppe

  • Status changed from needs_review to needs_work

Thanks for catching this. This is an additional dependency that we need to ship. https://github.com/scipy/scipy/blob/master/pyproject.toml

comment:31 Changed 11 months ago by git

  • Commit changed from 204213debcb838ff4253ccd7f9ca40552a5b3bdb to 14bcb1f25653b3baf78b1523913be0dc73257a95

Branch pushed to git repo; I updated commit sha1. New commits:

14bcb1fbuild/pkgs/pybind11: New, dependency of scipy

comment:32 Changed 11 months ago by mkoeppe

  • Summary changed from Upgrade NumPy to 1.18.4, scipy to 1.4.1 [py3 only!] to Upgrade NumPy to 1.18.4, scipy to 1.4.1, add pybind11 package [py3 only!]

comment:33 Changed 11 months ago by mkoeppe

  • Status changed from needs_work to needs_review

comment:34 Changed 11 months ago by enriqueartal

I compiled from scratch using gcc10. Besides the expected failures of palp and sympow, these are the last lines of the log of scipy.

error: Command "gfortran -Wall -g -ffixed-form -fno-second-underscore -fPIC -fPIC -O3 -funroll-loops -Iscipy/sparse/linalg/eigen/arpack/ARPACK/SRC -I/home/artal/sage/local/lib/python3.7/site-packages/numpy/core/include -c -c scipy/sparse/linalg/eigen/arpack/ARPACK/SRC/dsaitr.f -o build/temp.linux-x86_64-3.7/scipy/sparse/linalg/eigen/arpack/ARPACK/SRC/dsaitr.o" failed with exit status 1 Running setup.py install for scipy: finished with status 'error'

Cleaning up...

Removing source in /tmp/pip-req-build-yghxywzq

Removed build tracker '/tmp/pip-req-tracker-maw6u14m' Command "/home/artal/sage/local/bin/python3 -u -c "import setuptools, tokenize;file='/tmp/pip-req-build-yghxywzq/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" --no-user-cfg install --record /tmp/pip-record-axtqqq0z/install-record.txt --single-version-externally-managed --root /home/artal/sage/local/var/tmp/sage/build/scipy-1.4.1/inst --compile" failed with error code 1 in /tmp/pip-req-build-yghxywzq/ Exception information: Traceback (most recent call last):

File "/home/artal/sage/local/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 143, in main

status = self.run(options, args)

File "/home/artal/sage/local/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 366, in run

use_user_site=options.use_user_site,

File "/home/artal/sage/local/lib/python3.7/site-packages/pip/_internal/req/init.py", line 49, in install_given_reqs

kwargs

File "/home/artal/sage/local/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 791, in install

spinner=spinner,

File "/home/artal/sage/local/lib/python3.7/site-packages/pip/_internal/utils/misc.py", line 705, in call_subprocess

% (command_desc, proc.returncode, cwd))

pip._internal.exceptions.InstallationError?: Command "/home/artal/sage/local/bin/python3 -u -c "import setuptools, tokenize;file='/tmp/pip-req-build-yghxywzq/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" --no-user-cfg install --record /tmp/pip-record-axtqqq0z/install-record.txt --single-version-externally-managed --root /home/artal/sage/local/var/tmp/sage/build/scipy-1.4.1/inst --compile" failed with error code 1 in /tmp/pip-req-build-yghxywzq/ Error: installing with pip3 failed

comment:35 Changed 11 months ago by dimpase

what's the status of here? are we really waiting for gcc10 starting to work?

comment:36 Changed 11 months ago by mkoeppe

No, just waiting for review of this ticket and its dependency #29480.

comment:37 Changed 11 months ago by dimpase

I get doctest errors due to

DeprecationWarning: scipy.array is deprecated and will be removed in SciPy 2.0.0, use numpy.array instead

namely,

sage -t --warn-long 76.6 src/sage/graphs/generic_graph.py
**********************************************************************
File "src/sage/graphs/generic_graph.py", line 9789, in sage.graphs.generic_graph.GenericGraph.?
Failed example:
    G.pagerank(algorithm="Scipy") # abs tol 1e-9
Expected:
    {1: 0.16112205885619563,
     2: 0.1619531043247219,
     3: 0.16112205885619563,
     4: 0.2374999999999999,
     5: 0.17775588228760858,
     6: 0.100546895675278}
Got:
    doctest:warning
....
      File "/home/scratch2/dimpase/sage/sage/local/lib64/python3.7/site-packages/sage/doctest/forker.py", line 1104, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.graphs.generic_graph.GenericGraph.?[7]>", line 1, in <module>
        G.pagerank(algorithm="Scipy") # abs tol 1e-9
      File "/home/scratch2/dimpase/sage/sage/local/lib64/python3.7/site-packages/sage/graphs/generic_graph.py", line 9880, in pagerank
        dangling=dangling)
      File "/home/scratch2/dimpase/sage/sage/local/lib64/python3.7/site-packages/networkx/algorithms/link_analysis/pagerank_alg.py", line 429, in pagerank_scipy
        S = scipy.array(M.sum(axis=1)).flatten()
      File "/home/scratch2/dimpase/sage/sage/local/lib64/python3.7/site-packages/scipy/_lib/deprecation.py", line 19, in call
        stacklevel=stacklevel)
      File "/usr/lib64/python3.7/warnings.py", line 110, in _showwarnmsg
        msg.file, msg.line)
    :
    DeprecationWarning: scipy.array is deprecated and will be removed in SciPy 2.0.0, use numpy.array instead

it seems that these warnings are coming from scipy itself, or from networkx (which in turn calls scipy).

Let me see if upgrading networkx helps.

comment:38 Changed 11 months ago by dimpase

No, with networkx 2.4 one still gets these warnings, but they are fixed upstream, just not yet released, see https://github.com/networkx/networkx/commit/ea19baaff252468f5a911260430e057393d9637d

comment:39 Changed 11 months ago by dimpase

So we can patch networkx, or modify doctests, or suppress warning somehow (for the latter, I vaguely remember it's possible, but can't find it in docs)

comment:40 follow-ups: Changed 11 months ago by mkoeppe

Filtering out doctest warnings was done on one of the recent glpk tickets

comment:41 in reply to: ↑ 40 Changed 11 months ago by dimpase

  • Dependencies changed from dropping py2 support, #29480 to dropping py2 support, #29480, #29766
  • Reviewers changed from Isuru Fernando to Isuru Fernando, Dima Pasechnik
  • Status changed from needs_review to positive_review

Replying to mkoeppe:

Filtering out doctest warnings was done on one of the recent glpk tickets

I don't know how to filter multiline warnings with python backtraces.

OK, I've opened #29766 to fix this. This one is good to go, modulo the latter.

comment:42 Changed 11 months ago by vbraun

  • Dependencies changed from dropping py2 support, #29480, #29766 to #29480, #29766

comment:43 Changed 11 months ago by dimpase

  • Branch changed from u/mkoeppe/local_conda_forge_macos_standard__build_error_with_numpy to u/dimpase/local_conda_forge_macos_standard__build_error_with_numpy
  • Commit 14bcb1f25653b3baf78b1523913be0dc73257a95 deleted
  • Dependencies changed from #29480, #29766 to #29480

drop the networkx dependency with the latest patch (and rebase over latest beta).

comment:44 Changed 11 months ago by dimpase

  • Branch changed from u/dimpase/local_conda_forge_macos_standard__build_error_with_numpy to u/dimpase/trac/local_conda_forge_macos_standard__build_error_with_numpy
  • Commit set to c005562fed79e91883fa808237f5d52ceb024fd3

New commits:

9a3617dbuild/pkgs/cython: Update to 0.29.16
4a606b6build/pkgs/cython/checksums.ini: Use predictable URL for upstream_url pattern
e9a7cb0Update Cython to 0.29.17
15911cbbuild/pkgs/numpy: Update numpy to 1.18.2, back out numpy-1.10.2-no-hardcode-blas.patch (does not apply)
96e7a41Update numpy to 1.18.4
78b4907Update scipy to 1.4.1
54595b0build/pkgs/pybind11: New, dependency of scipy
c005562suppress scipy deprecation warnings

comment:45 in reply to: ↑ 40 Changed 11 months ago by dimpase

Replying to mkoeppe:

Filtering out doctest warnings was done on one of the recent glpk tickets

added the last commit to suppress the warning in the proper Pythonic way. The rest stays (mod rebase)

comment:46 Changed 10 months ago by vbraun

  • Status changed from positive_review to needs_work

I'm getting various test failures

comment:47 follow-up: Changed 10 months ago by mkoeppe

An upgrade ticket like this really needs to be run on the GitHub test infrastructure...

comment:48 Changed 10 months ago by dimpase

running GH Actions test here https://github.com/dimpase/sage/pull/9

comment:49 in reply to: ↑ 47 Changed 10 months ago by dimpase

Replying to mkoeppe:

An upgrade ticket like this really needs to be run on the GitHub test infrastructure...

Unfortunately there seem to be problems with runner setups, e.g. here https://github.com/dimpase/sage/runs/740349791?check_suite_focus=true

  • this is docker (archlinux-latest, standard-gcc_spkg)

I see

2020-06-04T22:39:01.9688082Z Suggested packages:
2020-06-04T22:39:01.9688790Z   python3-pytest python-setuptools-doc

and a lot of

2020-06-05T03:08:41.2134392Z Failure: ModuleNotFoundError (No module named 'pytest') ... ERROR

which seems to suggest that python3-pytest is a requirement.

Do you know how to fix this?

(there are other errors pointing at internal problems of runners)

Can we easily extend setup to use our own runners?

comment:50 Changed 10 months ago by mkoeppe

Apparently, running the testsuite of networkx requires pytest. Let's discuss this on #29766.

I don't think this has anything to do with GitHub.

comment:51 Changed 10 months ago by mkoeppe

In the meantime, numpy 1.18.5 has appeared

comment:52 Changed 10 months ago by mkoeppe

  • Branch changed from u/dimpase/trac/local_conda_forge_macos_standard__build_error_with_numpy to u/mkoeppe/trac/local_conda_forge_macos_standard__build_error_with_numpy

comment:53 Changed 10 months ago by mkoeppe

  • Commit changed from c005562fed79e91883fa808237f5d52ceb024fd3 to 56a6fe17708622cf6066629e7a5eeb2baead5508

Tests (with #29766) run at https://github.com/mkoeppe/sage/pull/38/checks


New commits:

56a6fe1build/pkgs/numpy: Update to 1.18.5

comment:54 Changed 10 months ago by mkoeppe

  • Description modified (diff)
  • Summary changed from Upgrade NumPy to 1.18.4, scipy to 1.4.1, add pybind11 package [py3 only!] to Upgrade NumPy to 1.18.5, scipy to 1.4.1, add pybind11 package

comment:55 Changed 10 months ago by mkoeppe

  • Status changed from needs_work to needs_review

comment:56 follow-up: Changed 10 months ago by mkoeppe

Can this ticket be tested successfully by itself or only as part of the networkx upgrade in #29766? If the latter, then let's close this ticket and merge it into #29766.

comment:57 in reply to: ↑ 56 ; follow-up: Changed 10 months ago by dimpase

Replying to mkoeppe:

Can this ticket be tested successfully by itself or only as part of the networkx upgrade in #29766? If the latter, then let's close this ticket and merge it into #29766.

no, this one does not need #29766

comment:58 Changed 10 months ago by mkoeppe

OK, running a new test without #29766 at https://github.com/mkoeppe/sage/pull/39/checks

comment:59 in reply to: ↑ 57 Changed 10 months ago by mkoeppe

Replying to dimpase:

Replying to mkoeppe:

Can this ticket be tested successfully by itself or only as part of the networkx upgrade in #29766? If the latter, then let's close this ticket and merge it into #29766.

no, this one does not need #29766

Actually, on ubuntu-xenial-standard (https://github.com/mkoeppe/sage/runs/745921419), I see lots of failures from the networkx testsuite such as this one:

File "/sage/local/var/tmp/sage/build/networkx-2.2/src/networkx/drawing/nx_pylab.py", line 249, in networkx.drawing.nx_pylab.draw_networkx
Failed example:
    nx.draw(G)
Exception raised:
    Traceback (most recent call last):
      File "/sage/local/lib/python3.7/doctest.py", line 1329, in __run
        compileflags, 1), test.globs)
      File "<doctest networkx.drawing.nx_pylab.draw_networkx[1]>", line 1, in <module>
        nx.draw(G)
      File "/sage/local/var/tmp/sage/build/networkx-2.2/src/networkx/drawing/nx_pylab.py", line 126, in draw
        draw_networkx(G, pos=pos, ax=ax, **kwds)
      File "/sage/local/var/tmp/sage/build/networkx-2.2/src/networkx/drawing/nx_pylab.py", line 278, in draw_networkx
        edge_collection = draw_networkx_edges(G, pos, arrows=arrows, **kwds)
      File "/sage/local/var/tmp/sage/build/networkx-2.2/src/networkx/drawing/nx_pylab.py", line 611, in draw_networkx_edges
        if cb.is_numlike(alpha):
    AttributeError: module 'matplotlib.cbook' has no attribute 'is_numlike'

comment:60 Changed 10 months ago by mkoeppe

  • Milestone changed from sage-9.2 to sage-duplicate/invalid/wontfix

Therefore, let's close this ticket and continue on #29766.

comment:61 Changed 10 months ago by dimpase

  • Status changed from needs_review to positive_review

comment:62 Changed 10 months ago by chapoton

  • Resolution set to duplicate
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.