Opened 3 months ago

Closed 5 weeks ago

#27000 closed enhancement (fixed)

Upgrade NumPy to 1.16.1

Reported by: thansen Owned by:
Priority: major Milestone: sage-8.7
Component: packages: standard Keywords: upgrade, numpy
Cc: arojas, fbissey, slelievre, thansen, tmonteil Merged in:
Authors: Jeroen Demeyer Reviewers: Dima Pasechnik
Report Upstream: N/A Work issues:
Branch: 0f3dab9 (Commits) Commit: 0f3dab964e4da4926cc1cb371b3b2272663301aa
Dependencies: #27223 Stopgaps:

Description (last modified by slelievre)

NumPy 1.16.0 was released on 2019-01-14, and NumPy 1.16.1 on 2019-02-01.

From the release announcement for NumPy 1.16.0:

This is the last NumPy release to support Python 2.7 and will be maintained as a long term release with bug fixes until 2020. This release has seen a lot of refactoring and features many bug fixes, improved code organization, and better cross platform compatibility. Not all of these improvements will be visible to users, but they should help make maintenance easier going forward. Highlights are

  • Experimental support for overriding numpy functions in downstream projects.
  • The matmul function is now a ufunc and can be overridden using __array_ufunc__.
  • Improved support for the ARM, POWER, and SPARC architectures.
  • Improved support for AIX and PyPy.
  • Improved interoperation with ctypes.
  • Improved support for PEP 3118.

From the Release announcement for NumPy 1.16.1:

This release fixes bugs reported against the 1.16.0 release, and also backports several enhancements from master that seem appropriate for a release series that is the last to support Python 2.7. [...]

If you are installing using pip, you may encounter a problem with older installed versions of NumPy that pip did not delete becoming mixed with the current version, resulting in an ImportError. That problem is particularly common on Debian derived distributions due to a modified pip. The fix is to make sure all previous NumPy versions installed by pip have been removed. See NumPy issue 12736 for discussion of the issue. Note that previously this problem resulted in an AttributeError.

Both release announcements indicate:

The wheels on PyPI are linked with OpenBLAS v0.3.4+, which should fix the known threading issues found in previous OpenBLAS versions. Downstream developers building this release should use Cython >= 0.29.2 and, if linking OpenBLAS, OpenBLAS > v0.3.4.

So this depends on the upgrade to OpenBLAS 0.3.5 (from 0.3.3) in #27020, and on the upgrade to Cython 0.29.2 in #27058.

Debian developers warn us that NumPy 1.16 in Debian leads to some deprecation warnings:

DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead

These are fixed by adding a patch to matplotlib in #26288.

Change History (41)

comment:1 Changed 3 months ago by fbissey

  • Cc fbissey added

comment:2 Changed 2 months ago by tmonteil

  • Cc tmonteil added

comment:3 Changed 2 months ago by slelievre

  • Cc arojas slelievre added
  • Description modified (diff)
  • Keywords upgrade numpy added

comment:4 Changed 2 months ago by slelievre

  • Description modified (diff)

comment:5 Changed 2 months ago by embray

It seems most, if not all of these DeprecationWarnings are actually coming via matplotlib. I don't even see any explict np.asscalar() calls in the Sage sources.

comment:6 Changed 2 months ago by embray

comment:7 Changed 2 months ago by arojas

I'm getting this on Arch after updating to 1.16.0

sage -t /usr/lib/python2.7/site-packages/sage/misc/inline_fortran.py
**********************************************************************
File "/usr/lib/python2.7/site-packages/sage/misc/inline_fortran.py", line 134, in sage.misc.inline_fortran._import_module_from_path._import_module_from_path_impl
Failed example:
    fortran(code, globals())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/sage/doctest/forker.py", line 671, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python2.7/site-packages/sage/doctest/forker.py", line 1086, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.misc.inline_fortran._import_module_from_path._import_module_from_path_impl[1]>", line 1, in <module>
        fortran(code, globals())
      File "sage/misc/lazy_import.pyx", line 354, in sage.misc.lazy_import.LazyImport.__call__ (build/cythonized/sage/misc/lazy_import.c:3684)
        return self.get_object()(*args, **kwds)
      File "/usr/lib/python2.7/site-packages/sage/misc/inline_fortran.py", line 96, in __call__
        return self.eval(*args, **kwds)
      File "/usr/lib/python2.7/site-packages/sage/misc/inline_fortran.py", line 191, in eval
        with open(log) as fobj:
    IOError: [Errno 2] No such file or directory: 'fortran_module.log'
**********************************************************************

comment:8 Changed 2 months ago by slelievre

  • Cc thansen added
  • Dependencies set to #27020, #27058
  • Description modified (diff)
  • Type changed from defect to enhancement

Editing ticket description now that NumPy 1.16.0 is released.

The release announcement for NumPy 1.16.0 warns:

Downstream developers building this release should use Cython >= 0.29.2 and, if linking OpenBLAS, OpenBLAS > v0.3.4.

So this depends on the upgrade to OpenBLAS 0.3.5 (from 0.3.3) in #27020, and on the upgrade to Cython 0.29.2 in #27058.

Last edited 2 months ago by slelievre (previous) (diff)

comment:9 Changed 2 months ago by embray

I also get some problems with inline_fortran on Numpy 1.16 (this has been affecting the Debian port). Haven't looked into it yet.

comment:10 Changed 2 months ago by embray

The inline_fortran problem appears to be caused by a backwards-incompatibility introduced by https://github.com/numpy/numpy/pull/11937

This interface previously was documented as calling f2py through a shell, which meant it was possible to pass constructs like shell redirections in the extra_args argument. So that PR should have passed shell=True to subprocess.check_output in order to keep that functionality, but it did not (that, or formally deprecate it and also allow a way to capture standard I/O).

comment:11 Changed 2 months ago by embray

  • Dependencies changed from #27020, #27058 to #27020, #27058, #27061

#27061 offers a fix to the problem with sage.misc.inline_fortran.

comment:12 Changed 2 months ago by embray

  • Milestone changed from sage-8.6 to sage-8.7

Retarging tickets optimistically to the next milestone. If you are responsible for this ticket (either its reporter or owner) and don't believe you are likely to complete this ticket before the next release (8.7) please retarget this ticket's milestone to sage-pending or sage-wishlist.

comment:13 Changed 7 weeks ago by jdemeyer

  • Dependencies changed from #27020, #27058, #27061 to #26288
  • Description modified (diff)

comment:14 Changed 7 weeks ago by dimpase

Is there a branch to test this?

comment:15 Changed 7 weeks ago by jdemeyer

  • Branch set to u/jdemeyer/upgrade_numpy_to_1_16

comment:16 follow-up: Changed 7 weeks ago by dimpase

  • Commit set to 4fe5680ceb4c30673f9a8c52aa9c7a6b5487f842

Is it compatible with scipy 1.2? It was released earlier, and given that scipy logs are full of Using deprecated Numpy API warnings, it seems that the real blocker to this would be a new scipy version.


New commits:

c009e77update matplotlib to version 2.2.3
7794b09Avoid numpy.isscalar()
4fe5680NumPy 1.16.0

comment:17 Changed 7 weeks ago by git

  • Commit changed from 4fe5680ceb4c30673f9a8c52aa9c7a6b5487f842 to 07ecaa914488197f485976b484fb0e61b97b32b1

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

c892da7Avoid numpy.asscalar()
07ecaa9NumPy 1.16.0

comment:18 in reply to: ↑ 16 Changed 7 weeks ago by jdemeyer

Replying to dimpase:

Is it compatible with scipy 1.2? It was released earlier, and given that scipy logs are full of Using deprecated Numpy API warnings, it seems that the real blocker to this would be a new scipy version.

There is a long discussion about that in https://github.com/cython/cython/issues/2498

comment:19 Changed 7 weeks ago by dimpase

anything stops us from going forward with this upgrade? it all seems to work, after matplotlib fix.

comment:20 Changed 7 weeks ago by jdemeyer

  • Authors set to Jeroen Demeyer
  • Status changed from new to needs_review

comment:21 Changed 7 weeks ago by dimpase

  • Reviewers set to Dima Pasechnik
  • Status changed from needs_review to positive_review

LGTM

comment:22 Changed 7 weeks ago by vbraun

  • Status changed from positive_review to needs_work

Seems to conflict with matplotlib on OSX:

Ignoring indexes: https://pypi.org/simple
Created temporary directory: /private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-ephem-wheel-cache-kl7Hms
Created temporary directory: /private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-tracker-Za__6B
Created requirements tracker '/private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-tracker-Za__6B'
Created temporary directory: /private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-install-DJEDBz
Processing /Users/buildslave-sage/slave/sage_git/build/local/var/tmp/sage/build/matplotlib-2.2.3.p0/src
  Created temporary directory: /private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-build-46rwMp
  Added file:///Users/buildslave-sage/slave/sage_git/build/local/var/tmp/sage/build/matplotlib-2.2.3.p0/src to build tracker '/private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-tracker-Za__6B'
  Running setup.py (path:/private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-build-46rwMp/setup.py) egg_info for package from file:///Users/buildslave-sage/slave/sage_git/build/local/var/tmp/sage/build/matplotlib-2.2.3.p0/src
    Running command python setup.py egg_info
    ============================================================================
    Edit setup.cfg to change the build options
    BUILDING MATPLOTLIB
                matplotlib: yes [2.2.3]
                    python: yes [2.7.15 (default, Jan 27 2019, 04:12:55)  [GCC
                            4.2.1 Compatible Apple LLVM 10.0.0
                            (clang-1000.11.45.5)]]
                  platform: yes [darwin]
    REQUIRED DEPENDENCIES AND EXTENSIONS
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-build-46rwMp/setup.py", line 172, in <module>
        result = package.check()
      File "setupext.py", line 963, in check
        import numpy
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/numpy/__init__.py", line 142, in <module>
        from . import core
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/numpy/core/__init__.py", line 59, in <module>
        from . import numeric
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/numpy/core/numeric.py", line 3093, in <module>
        from . import fromnumeric
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/numpy/core/fromnumeric.py", line 17, in <module>
        from . import _methods
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/numpy/core/_methods.py", line 158, in <module>
        _NDARRAY_ARRAY_FUNCTION = mu.ndarray.__array_function__
    AttributeError: type object 'numpy.ndarray' has no attribute '__array_function__'
Cleaning up...
  Removing source in /private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-build-46rwMp
Removed file:///Users/buildslave-sage/slave/sage_git/build/local/var/tmp/sage/build/matplotlib-2.2.3.p0/src from build tracker '/private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-tracker-Za__6B'
Removed build tracker '/private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-tracker-Za__6B'
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-build-46rwMp/
Exception information:
Traceback (most recent call last):
  File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pip/_internal/cli/base_command.py", line 143, in main
    status = self.run(options, args)
  File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 318, in run
    resolver.resolve(requirement_set)
  File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pip/_internal/resolve.py", line 102, in resolve
    self._resolve_one(requirement_set, req)
  File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pip/_internal/resolve.py", line 256, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pip/_internal/resolve.py", line 209, in _get_abstract_dist_for
    self.require_hashes
  File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pip/_internal/operations/prepare.py", line 298, in prepare_linked_requirement
    abstract_dist.prep_for_dist(finder, self.build_isolation)
  File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pip/_internal/operations/prepare.py", line 126, in prep_for_dist
    self.req.run_egg_info()
  File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pip/_internal/req/req_install.py", line 473, in run_egg_info
    command_desc='python setup.py egg_info')
  File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pip/_internal/utils/misc.py", line 705, in call_subprocess
    % (command_desc, proc.returncode, cwd))
InstallationError: Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-build-46rwMp/
Error: installing with pip2 failed
********************************************************************************
Error installing matplotlib-2.2.3.p0
********************************************************************************
real	0m6.450s
user	0m2.680s
sys	0m3.147s
************************************************************************
Error installing package matplotlib-2.2.3.p0
************************************************************************

comment:23 Changed 7 weeks ago by jdemeyer

  • Dependencies #26288 deleted

comment:24 Changed 7 weeks ago by git

  • Commit changed from 07ecaa914488197f485976b484fb0e61b97b32b1 to 14bf0f940c3844e511931727a50d9bd7b3327a91

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

14bf0f9NumPy 1.16.0

comment:25 Changed 6 weeks ago by arojas

1.16.1 is out, no test failures (on Arch)

comment:26 Changed 6 weeks ago by dimpase

What OSX version do you get that error on?

comment:27 Changed 6 weeks ago by vbraun

latest (mojave)

comment:28 Changed 6 weeks ago by dimpase

AttributeError: type object 'numpy.ndarray' has no attribute '__array_function__'
Cleaning up...

seems to tell one that the error was during the build of numpy. Bring on da log!

comment:29 Changed 6 weeks ago by jdemeyer

The scipy build fails similarly:

Installing scipy-1.2.0
Traceback (most recent call last):
  File "setup.py", line 492, in <module>
    setup_package()
  File "setup.py", line 424, in setup_package
    import numpy
  File "/Users/jdemeyer/sage/local/lib/python2.7/site-packages/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/Users/jdemeyer/sage/local/lib/python2.7/site-packages/numpy/core/__init__.py", line 59, in <module>
    from . import numeric
  File "/Users/jdemeyer/sage/local/lib/python2.7/site-packages/numpy/core/numeric.py", line 3093, in <module>
    from . import fromnumeric
  File "/Users/jdemeyer/sage/local/lib/python2.7/site-packages/numpy/core/fromnumeric.py", line 17, in <module>
    from . import _methods
  File "/Users/jdemeyer/sage/local/lib/python2.7/site-packages/numpy/core/_methods.py", line 158, in <module>
    _NDARRAY_ARRAY_FUNCTION = mu.ndarray.__array_function__
AttributeError: type object 'numpy.ndarray' has no attribute '__array_function__'
Error: could not determine package name
********************************************************************************

comment:30 Changed 6 weeks ago by dimpase

this seems to be an NumPy/pip (?) problem: https://github.com/numpy/numpy/issues/12736

comment:31 follow-up: Changed 6 weeks ago by dimpase

We should upgrade to 1.16.1, where this bug is apparently fixed by https://github.com/numpy/numpy/pull/12891

https://github.com/numpy/numpy/releases/tag/v1.16.1

Last edited 6 weeks ago by dimpase (previous) (diff)

comment:32 Changed 6 weeks ago by jdemeyer

  • Summary changed from Upgrade NumPy to 1.16 to Upgrade NumPy to 1.16.1

comment:33 in reply to: ↑ 31 Changed 6 weeks ago by jdemeyer

Replying to dimpase:

We should upgrade to 1.16.1, where this bug is apparently fixed by https://github.com/numpy/numpy/pull/12891

There is nothing "fixed". The only thing that changed is a better error message. Apparently we're not properly deleting old versions of numpy.

comment:34 Changed 6 weeks ago by jdemeyer

  • Dependencies set to #27223

comment:35 Changed 6 weeks ago by git

  • Commit changed from 14bf0f940c3844e511931727a50d9bd7b3327a91 to 0f3dab964e4da4926cc1cb371b3b2272663301aa

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

0cf4613Fix legacy uninstall
0f3dab9NumPy 1.16.1

comment:36 Changed 6 weeks ago by jdemeyer

Adding a spkg-legacy-uninstall script should fix things.

comment:37 Changed 6 weeks ago by jdemeyer

  • Status changed from needs_work to needs_review

comment:38 Changed 6 weeks ago by jdemeyer

  • Description modified (diff)

comment:39 Changed 6 weeks ago by slelievre

  • Description modified (diff)

comment:40 Changed 6 weeks ago by dimpase

  • Status changed from needs_review to positive_review

OK, looks good.

comment:41 Changed 5 weeks ago by vbraun

  • Branch changed from u/jdemeyer/upgrade_numpy_to_1_16 to 0f3dab964e4da4926cc1cb371b3b2272663301aa
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.