Opened 3 years ago
Closed 3 years ago
#27000 closed enhancement (fixed)
Upgrade NumPy to 1.16.1
Reported by: | thansen | Owned by: | |
---|---|---|---|
Priority: | major | Milestone: | sage-8.7 |
Component: | packages: standard | Keywords: | upgrade, numpy |
Cc: | arojas, fbissey, slelievre, thansen, tmonteil | Merged in: | |
Authors: | Jeroen Demeyer | Reviewers: | Dima Pasechnik |
Report Upstream: | N/A | Work issues: | |
Branch: | 0f3dab9 (Commits, GitHub, GitLab) | Commit: | 0f3dab964e4da4926cc1cb371b3b2272663301aa |
Dependencies: | #27223 | Stopgaps: |
Description (last modified by )
NumPy 1.16.0 was released on 2019-01-14, and NumPy 1.16.1 on 2019-02-01.
From the release announcement for NumPy 1.16.0:
This is the last NumPy release to support Python 2.7 and will be maintained as a long term release with bug fixes until 2020. This release has seen a lot of refactoring and features many bug fixes, improved code organization, and better cross platform compatibility. Not all of these improvements will be visible to users, but they should help make maintenance easier going forward. Highlights are
- Experimental support for overriding numpy functions in downstream projects.
- The matmul function is now a ufunc and can be overridden using
__array_ufunc__
.- Improved support for the ARM, POWER, and SPARC architectures.
- Improved support for AIX and PyPy.
- Improved interoperation with ctypes.
- Improved support for PEP 3118.
From the Release announcement for NumPy 1.16.1:
This release fixes bugs reported against the 1.16.0 release, and also backports several enhancements from master that seem appropriate for a release series that is the last to support Python 2.7. [...]
If you are installing using pip, you may encounter a problem with older installed versions of NumPy that pip did not delete becoming mixed with the current version, resulting in an
ImportError
. That problem is particularly common on Debian derived distributions due to a modified pip. The fix is to make sure all previous NumPy versions installed by pip have been removed. See NumPy issue 12736 for discussion of the issue. Note that previously this problem resulted in anAttributeError
.
Both release announcements indicate:
The wheels on PyPI are linked with OpenBLAS v0.3.4+, which should fix the known threading issues found in previous OpenBLAS versions. Downstream developers building this release should use Cython >= 0.29.2 and, if linking OpenBLAS, OpenBLAS > v0.3.4.
So this depends on the upgrade to OpenBLAS 0.3.5 (from 0.3.3) in #27020, and on the upgrade to Cython 0.29.2 in #27058.
Debian developers warn us that NumPy 1.16 in Debian leads to some deprecation warnings:
DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead
These are fixed by adding a patch to matplotlib in #26288.
Change History (41)
comment:1 Changed 3 years ago by
- Cc fbissey added
comment:2 Changed 3 years ago by
- Cc tmonteil added
comment:3 Changed 3 years ago by
- Cc arojas slelievre added
- Description modified (diff)
- Keywords upgrade numpy added
comment:4 Changed 3 years ago by
- Description modified (diff)
comment:5 Changed 3 years ago by
comment:6 Changed 3 years ago by
Upstream fix to matplotlib: https://github.com/matplotlib/matplotlib/pull/12478
comment:7 Changed 3 years ago by
I'm getting this on Arch after updating to 1.16.0
sage -t /usr/lib/python2.7/site-packages/sage/misc/inline_fortran.py ********************************************************************** File "/usr/lib/python2.7/site-packages/sage/misc/inline_fortran.py", line 134, in sage.misc.inline_fortran._import_module_from_path._import_module_from_path_impl Failed example: fortran(code, globals()) Exception raised: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/sage/doctest/forker.py", line 671, in _run self.compile_and_execute(example, compiler, test.globs) File "/usr/lib/python2.7/site-packages/sage/doctest/forker.py", line 1086, in compile_and_execute exec(compiled, globs) File "<doctest sage.misc.inline_fortran._import_module_from_path._import_module_from_path_impl[1]>", line 1, in <module> fortran(code, globals()) File "sage/misc/lazy_import.pyx", line 354, in sage.misc.lazy_import.LazyImport.__call__ (build/cythonized/sage/misc/lazy_import.c:3684) return self.get_object()(*args, **kwds) File "/usr/lib/python2.7/site-packages/sage/misc/inline_fortran.py", line 96, in __call__ return self.eval(*args, **kwds) File "/usr/lib/python2.7/site-packages/sage/misc/inline_fortran.py", line 191, in eval with open(log) as fobj: IOError: [Errno 2] No such file or directory: 'fortran_module.log' **********************************************************************
comment:8 Changed 3 years ago by
- Cc thansen added
- Dependencies set to #27020, #27058
- Description modified (diff)
- Type changed from defect to enhancement
Editing ticket description now that NumPy 1.16.0 is released.
The release announcement for NumPy 1.16.0 warns:
Downstream developers building this release should use Cython >= 0.29.2 and, if linking OpenBLAS, OpenBLAS > v0.3.4.
So this depends on the upgrade to OpenBlas? 0.3.5 (from 0.3.3) in #27020, and on the upgrade to Cython 0.29.2 in #27058.
comment:9 Changed 3 years ago by
I also get some problems with inline_fortran on Numpy 1.16 (this has been affecting the Debian port). Haven't looked into it yet.
comment:10 Changed 3 years ago by
The inline_fortran problem appears to be caused by a backwards-incompatibility introduced by https://github.com/numpy/numpy/pull/11937
This interface previously was documented as calling f2py through a shell, which meant it was possible to pass constructs like shell redirections in the extra_args
argument. So that PR should have passed shell=True
to subprocess.check_output
in order to keep that functionality, but it did not (that, or formally deprecate it and also allow a way to capture standard I/O).
comment:11 Changed 3 years ago by
- Dependencies changed from #27020, #27058 to #27020, #27058, #27061
#27061 offers a fix to the problem with sage.misc.inline_fortran
.
comment:12 Changed 3 years ago by
- Milestone changed from sage-8.6 to sage-8.7
Retarging tickets optimistically to the next milestone. If you are responsible for this ticket (either its reporter or owner) and don't believe you are likely to complete this ticket before the next release (8.7) please retarget this ticket's milestone to sage-pending or sage-wishlist.
comment:13 Changed 3 years ago by
- Dependencies changed from #27020, #27058, #27061 to #26288
- Description modified (diff)
comment:14 Changed 3 years ago by
Is there a branch to test this?
comment:15 Changed 3 years ago by
- Branch set to u/jdemeyer/upgrade_numpy_to_1_16
comment:16 follow-up: ↓ 18 Changed 3 years ago by
- Commit set to 4fe5680ceb4c30673f9a8c52aa9c7a6b5487f842
Is it compatible with scipy 1.2? It was released earlier, and given that scipy logs are full of Using deprecated Numpy API
warnings, it seems that the real blocker to this would be a new scipy version.
New commits:
c009e77 | update matplotlib to version 2.2.3
|
7794b09 | Avoid numpy.isscalar()
|
4fe5680 | NumPy 1.16.0
|
comment:17 Changed 3 years ago by
- Commit changed from 4fe5680ceb4c30673f9a8c52aa9c7a6b5487f842 to 07ecaa914488197f485976b484fb0e61b97b32b1
comment:18 in reply to: ↑ 16 Changed 3 years ago by
Replying to dimpase:
Is it compatible with scipy 1.2? It was released earlier, and given that scipy logs are full of
Using deprecated Numpy API
warnings, it seems that the real blocker to this would be a new scipy version.
There is a long discussion about that in https://github.com/cython/cython/issues/2498
comment:19 Changed 3 years ago by
anything stops us from going forward with this upgrade? it all seems to work, after matplotlib fix.
comment:20 Changed 3 years ago by
- Status changed from new to needs_review
comment:21 Changed 3 years ago by
- Reviewers set to Dima Pasechnik
- Status changed from needs_review to positive_review
LGTM
comment:22 Changed 3 years ago by
- Status changed from positive_review to needs_work
Seems to conflict with matplotlib on OSX:
Ignoring indexes: https://pypi.org/simple Created temporary directory: /private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-ephem-wheel-cache-kl7Hms Created temporary directory: /private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-tracker-Za__6B Created requirements tracker '/private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-tracker-Za__6B' Created temporary directory: /private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-install-DJEDBz Processing /Users/buildslave-sage/slave/sage_git/build/local/var/tmp/sage/build/matplotlib-2.2.3.p0/src Created temporary directory: /private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-build-46rwMp Added file:///Users/buildslave-sage/slave/sage_git/build/local/var/tmp/sage/build/matplotlib-2.2.3.p0/src to build tracker '/private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-tracker-Za__6B' Running setup.py (path:/private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-build-46rwMp/setup.py) egg_info for package from file:///Users/buildslave-sage/slave/sage_git/build/local/var/tmp/sage/build/matplotlib-2.2.3.p0/src Running command python setup.py egg_info ============================================================================ Edit setup.cfg to change the build options BUILDING MATPLOTLIB matplotlib: yes [2.2.3] python: yes [2.7.15 (default, Jan 27 2019, 04:12:55) [GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.5)]] platform: yes [darwin] REQUIRED DEPENDENCIES AND EXTENSIONS Traceback (most recent call last): File "<string>", line 1, in <module> File "/private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-build-46rwMp/setup.py", line 172, in <module> result = package.check() File "setupext.py", line 963, in check import numpy File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/numpy/__init__.py", line 142, in <module> from . import core File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/numpy/core/__init__.py", line 59, in <module> from . import numeric File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/numpy/core/numeric.py", line 3093, in <module> from . import fromnumeric File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/numpy/core/fromnumeric.py", line 17, in <module> from . import _methods File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/numpy/core/_methods.py", line 158, in <module> _NDARRAY_ARRAY_FUNCTION = mu.ndarray.__array_function__ AttributeError: type object 'numpy.ndarray' has no attribute '__array_function__' Cleaning up... Removing source in /private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-build-46rwMp Removed file:///Users/buildslave-sage/slave/sage_git/build/local/var/tmp/sage/build/matplotlib-2.2.3.p0/src from build tracker '/private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-tracker-Za__6B' Removed build tracker '/private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-tracker-Za__6B' Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-build-46rwMp/ Exception information: Traceback (most recent call last): File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pip/_internal/cli/base_command.py", line 143, in main status = self.run(options, args) File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 318, in run resolver.resolve(requirement_set) File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pip/_internal/resolve.py", line 102, in resolve self._resolve_one(requirement_set, req) File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pip/_internal/resolve.py", line 256, in _resolve_one abstract_dist = self._get_abstract_dist_for(req_to_install) File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pip/_internal/resolve.py", line 209, in _get_abstract_dist_for self.require_hashes File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pip/_internal/operations/prepare.py", line 298, in prepare_linked_requirement abstract_dist.prep_for_dist(finder, self.build_isolation) File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pip/_internal/operations/prepare.py", line 126, in prep_for_dist self.req.run_egg_info() File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pip/_internal/req/req_install.py", line 473, in run_egg_info command_desc='python setup.py egg_info') File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pip/_internal/utils/misc.py", line 705, in call_subprocess % (command_desc, proc.returncode, cwd)) InstallationError: Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-build-46rwMp/ Error: installing with pip2 failed ******************************************************************************** Error installing matplotlib-2.2.3.p0 ******************************************************************************** real 0m6.450s user 0m2.680s sys 0m3.147s ************************************************************************ Error installing package matplotlib-2.2.3.p0 ************************************************************************
comment:23 Changed 3 years ago by
- Dependencies #26288 deleted
comment:24 Changed 3 years ago by
- Commit changed from 07ecaa914488197f485976b484fb0e61b97b32b1 to 14bf0f940c3844e511931727a50d9bd7b3327a91
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
14bf0f9 | NumPy 1.16.0
|
comment:25 Changed 3 years ago by
1.16.1 is out, no test failures (on Arch)
comment:26 Changed 3 years ago by
What OSX version do you get that error on?
comment:27 Changed 3 years ago by
latest (mojave)
comment:28 Changed 3 years ago by
AttributeError: type object 'numpy.ndarray' has no attribute '__array_function__' Cleaning up...
seems to tell one that the error was during the build of numpy. Bring on da log!
comment:29 Changed 3 years ago by
The scipy build fails similarly:
Installing scipy-1.2.0 Traceback (most recent call last): File "setup.py", line 492, in <module> setup_package() File "setup.py", line 424, in setup_package import numpy File "/Users/jdemeyer/sage/local/lib/python2.7/site-packages/numpy/__init__.py", line 142, in <module> from . import core File "/Users/jdemeyer/sage/local/lib/python2.7/site-packages/numpy/core/__init__.py", line 59, in <module> from . import numeric File "/Users/jdemeyer/sage/local/lib/python2.7/site-packages/numpy/core/numeric.py", line 3093, in <module> from . import fromnumeric File "/Users/jdemeyer/sage/local/lib/python2.7/site-packages/numpy/core/fromnumeric.py", line 17, in <module> from . import _methods File "/Users/jdemeyer/sage/local/lib/python2.7/site-packages/numpy/core/_methods.py", line 158, in <module> _NDARRAY_ARRAY_FUNCTION = mu.ndarray.__array_function__ AttributeError: type object 'numpy.ndarray' has no attribute '__array_function__' Error: could not determine package name ********************************************************************************
comment:30 Changed 3 years ago by
this seems to be an NumPy/pip (?) problem: https://github.com/numpy/numpy/issues/12736
comment:31 follow-up: ↓ 33 Changed 3 years ago by
We should upgrade to 1.16.1, where this bug is apparently fixed by https://github.com/numpy/numpy/pull/12891
comment:32 Changed 3 years ago by
- Summary changed from Upgrade NumPy to 1.16 to Upgrade NumPy to 1.16.1
comment:33 in reply to: ↑ 31 Changed 3 years ago by
Replying to dimpase:
We should upgrade to 1.16.1, where this bug is apparently fixed by https://github.com/numpy/numpy/pull/12891
There is nothing "fixed". The only thing that changed is a better error message. Apparently we're not properly deleting old versions of numpy.
comment:34 Changed 3 years ago by
- Dependencies set to #27223
comment:35 Changed 3 years ago by
- Commit changed from 14bf0f940c3844e511931727a50d9bd7b3327a91 to 0f3dab964e4da4926cc1cb371b3b2272663301aa
comment:36 Changed 3 years ago by
Adding a spkg-legacy-uninstall
script should fix things.
comment:37 Changed 3 years ago by
- Status changed from needs_work to needs_review
comment:38 Changed 3 years ago by
- Description modified (diff)
comment:39 Changed 3 years ago by
- Description modified (diff)
comment:40 Changed 3 years ago by
- Status changed from needs_review to positive_review
OK, looks good.
comment:41 Changed 3 years ago by
- Branch changed from u/jdemeyer/upgrade_numpy_to_1_16 to 0f3dab964e4da4926cc1cb371b3b2272663301aa
- Resolution set to fixed
- Status changed from positive_review to closed
It seems most, if not all of these
DeprecationWarning
s are actually coming via matplotlib. I don't even see any explictnp.asscalar()
calls in the Sage sources.