Upgrade NumPy to 1.16.1

[thansen]

NumPy 1.16.0 was released on 2019-01-14, and NumPy 1.16.1 on 2019-02-01.

• Tarball: ​https://files.pythonhosted.org/packages/2b/26/07472b0de91851b6656cbc86e2f0d5d3a3128e7580f23295ef58b6862d6c/numpy-1.16.1.zip

From the ​release announcement for NumPy 1.16.0:

This is the last NumPy release to support Python 2.7 and will be maintained as a long term release with bug fixes until 2020. This release has seen a lot of refactoring and features many bug fixes, improved code organization, and better cross platform compatibility. Not all of these improvements will be visible to users, but they should help make maintenance easier going forward. Highlights are

• Experimental support for overriding numpy functions in downstream projects.
• The matmul function is now a ufunc and can be overridden using __array_ufunc__.
• Improved support for the ARM, POWER, and SPARC architectures.
• Improved support for AIX and PyPy.
• Improved interoperation with ctypes.
• Improved support for PEP 3118.

From the ​Release announcement for NumPy 1.16.1:

This release fixes bugs reported against the 1.16.0 release, and also backports several enhancements from master that seem appropriate for a release series that is the last to support Python 2.7. [...]

If you are installing using pip, you may encounter a problem with older installed versions of NumPy that pip did not delete becoming mixed with the current version, resulting in an ImportError. That problem is particularly common on Debian derived distributions due to a modified pip. The fix is to make sure all previous NumPy versions installed by pip have been removed. See ​NumPy issue 12736 for discussion of the issue. Note that previously this problem resulted in an AttributeError.

Both release announcements indicate:

The wheels on PyPI are linked with OpenBLAS v0.3.4+, which should fix the known threading issues found in previous OpenBLAS versions. Downstream developers building this release should use Cython >= 0.29.2 and, if linking OpenBLAS, OpenBLAS > v0.3.4.

So this depends on the upgrade to OpenBLAS 0.3.5 (from 0.3.3) in #27020, and on the upgrade to Cython 0.29.2 in #27058.

Debian developers warn us that NumPy 1.16 in Debian leads to some deprecation warnings:

DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead

These are fixed by adding a patch to matplotlib in #26288.

[embray]

It seems most, if not all of these DeprecationWarnings are actually coming via matplotlib. I don't even see any explict np.asscalar() calls in the Sage sources.

[embray]

Upstream fix to matplotlib: ​https://github.com/matplotlib/matplotlib/pull/12478

[arojas]

I'm getting this on Arch after updating to 1.16.0

sage -t /usr/lib/python2.7/site-packages/sage/misc/inline_fortran.py
**********************************************************************
File "/usr/lib/python2.7/site-packages/sage/misc/inline_fortran.py", line 134, in sage.misc.inline_fortran._import_module_from_path._import_module_from_path_impl
Failed example:
    fortran(code, globals())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/sage/doctest/forker.py", line 671, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python2.7/site-packages/sage/doctest/forker.py", line 1086, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.misc.inline_fortran._import_module_from_path._import_module_from_path_impl[1]>", line 1, in <module>
        fortran(code, globals())
      File "sage/misc/lazy_import.pyx", line 354, in sage.misc.lazy_import.LazyImport.__call__ (build/cythonized/sage/misc/lazy_import.c:3684)
        return self.get_object()(*args, **kwds)
      File "/usr/lib/python2.7/site-packages/sage/misc/inline_fortran.py", line 96, in __call__
        return self.eval(*args, **kwds)
      File "/usr/lib/python2.7/site-packages/sage/misc/inline_fortran.py", line 191, in eval
        with open(log) as fobj:
    IOError: [Errno 2] No such file or directory: 'fortran_module.log'
**********************************************************************

[slelievre]

Editing ticket description now that NumPy 1.16.0 is released.

The release announcement for NumPy 1.16.0 warns:

Downstream developers building this release should use Cython >= 0.29.2 and, if linking OpenBLAS, OpenBLAS > v0.3.4.

So this depends on the upgrade to OpenBlas? 0.3.5 (from 0.3.3) in #27020, and on the upgrade to Cython 0.29.2 in #27058.

[embray]

I also get some problems with inline_fortran on Numpy 1.16 (this has been affecting the Debian port). Haven't looked into it yet.

[embray]

The inline_fortran problem appears to be caused by a backwards-incompatibility introduced by ​https://github.com/numpy/numpy/pull/11937

This interface previously was ​documented as calling f2py through a shell, which meant it was possible to pass constructs like shell redirections in the extra_args argument. So that PR should have passed shell=True to subprocess.check_output in order to keep that functionality, but it did not (that, or formally deprecate it and also allow a way to capture standard I/O).

[embray]

#27061 offers a fix to the problem with sage.misc.inline_fortran.

[embray]

Retarging tickets optimistically to the next milestone. If you are responsible for this ticket (either its reporter or owner) and don't believe you are likely to complete this ticket before the next release (8.7) please retarget this ticket's milestone to sage-pending or sage-wishlist.

[dimpase]

Is there a branch to test this?

[dimpase]

Is it compatible with scipy 1.2? It was released earlier, and given that scipy logs are full of Using deprecated Numpy API warnings, it seems that the real blocker to this would be a new scipy version.

---

New commits:

​c009e77	update matplotlib to version 2.2.3
​7794b09	Avoid numpy.isscalar()
​4fe5680	NumPy 1.16.0

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​c892da7	Avoid numpy.asscalar()
​07ecaa9	NumPy 1.16.0

[jdemeyer]

Replying to dimpase:

Is it compatible with scipy 1.2? It was released earlier, and given that scipy logs are full of Using deprecated Numpy API warnings, it seems that the real blocker to this would be a new scipy version.

There is a long discussion about that in ​https://github.com/cython/cython/issues/2498

[dimpase]

anything stops us from going forward with this upgrade? it all seems to work, after matplotlib fix.

[dimpase]

LGTM

[vbraun]

Seems to conflict with matplotlib on OSX:

Ignoring indexes: https://pypi.org/simple
Created temporary directory: /private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-ephem-wheel-cache-kl7Hms
Created temporary directory: /private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-tracker-Za__6B
Created requirements tracker '/private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-tracker-Za__6B'
Created temporary directory: /private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-install-DJEDBz
Processing /Users/buildslave-sage/slave/sage_git/build/local/var/tmp/sage/build/matplotlib-2.2.3.p0/src
  Created temporary directory: /private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-build-46rwMp
  Added file:///Users/buildslave-sage/slave/sage_git/build/local/var/tmp/sage/build/matplotlib-2.2.3.p0/src to build tracker '/private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-tracker-Za__6B'
  Running setup.py (path:/private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-build-46rwMp/setup.py) egg_info for package from file:///Users/buildslave-sage/slave/sage_git/build/local/var/tmp/sage/build/matplotlib-2.2.3.p0/src
    Running command python setup.py egg_info
    ============================================================================
    Edit setup.cfg to change the build options
    BUILDING MATPLOTLIB
                matplotlib: yes [2.2.3]
                    python: yes [2.7.15 (default, Jan 27 2019, 04:12:55)  [GCC
                            4.2.1 Compatible Apple LLVM 10.0.0
                            (clang-1000.11.45.5)]]
                  platform: yes [darwin]
    REQUIRED DEPENDENCIES AND EXTENSIONS
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-build-46rwMp/setup.py", line 172, in <module>
        result = package.check()
      File "setupext.py", line 963, in check
        import numpy
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/numpy/__init__.py", line 142, in <module>
        from . import core
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/numpy/core/__init__.py", line 59, in <module>
        from . import numeric
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/numpy/core/numeric.py", line 3093, in <module>
        from . import fromnumeric
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/numpy/core/fromnumeric.py", line 17, in <module>
        from . import _methods
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/numpy/core/_methods.py", line 158, in <module>
        _NDARRAY_ARRAY_FUNCTION = mu.ndarray.__array_function__
    AttributeError: type object 'numpy.ndarray' has no attribute '__array_function__'
Cleaning up...
  Removing source in /private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-build-46rwMp
Removed file:///Users/buildslave-sage/slave/sage_git/build/local/var/tmp/sage/build/matplotlib-2.2.3.p0/src from build tracker '/private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-tracker-Za__6B'
Removed build tracker '/private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-tracker-Za__6B'
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-build-46rwMp/
Exception information:
Traceback (most recent call last):
  File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pip/_internal/cli/base_command.py", line 143, in main
    status = self.run(options, args)
  File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 318, in run
    resolver.resolve(requirement_set)
  File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pip/_internal/resolve.py", line 102, in resolve
    self._resolve_one(requirement_set, req)
  File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pip/_internal/resolve.py", line 256, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pip/_internal/resolve.py", line 209, in _get_abstract_dist_for
    self.require_hashes
  File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pip/_internal/operations/prepare.py", line 298, in prepare_linked_requirement
    abstract_dist.prep_for_dist(finder, self.build_isolation)
  File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pip/_internal/operations/prepare.py", line 126, in prep_for_dist
    self.req.run_egg_info()
  File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pip/_internal/req/req_install.py", line 473, in run_egg_info
    command_desc='python setup.py egg_info')
  File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pip/_internal/utils/misc.py", line 705, in call_subprocess
    % (command_desc, proc.returncode, cwd))
InstallationError: Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/hd/9pzxdk253fjbnwfcnj5bfmwc0000gs/T/pip-req-build-46rwMp/
Error: installing with pip2 failed
********************************************************************************
Error installing matplotlib-2.2.3.p0
********************************************************************************
real	0m6.450s
user	0m2.680s
sys	0m3.147s
************************************************************************
Error installing package matplotlib-2.2.3.p0
************************************************************************

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​14bf0f9

NumPy 1.16.0

[arojas]

1.16.1 is out, no test failures (on Arch)

[dimpase]

What OSX version do you get that error on?

[vbraun]

latest (mojave)

[dimpase]

AttributeError: type object 'numpy.ndarray' has no attribute '__array_function__'
Cleaning up...

seems to tell one that the error was during the build of numpy. Bring on da log!

[jdemeyer]

The scipy build fails similarly:

Installing scipy-1.2.0
Traceback (most recent call last):
  File "setup.py", line 492, in <module>
    setup_package()
  File "setup.py", line 424, in setup_package
    import numpy
  File "/Users/jdemeyer/sage/local/lib/python2.7/site-packages/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/Users/jdemeyer/sage/local/lib/python2.7/site-packages/numpy/core/__init__.py", line 59, in <module>
    from . import numeric
  File "/Users/jdemeyer/sage/local/lib/python2.7/site-packages/numpy/core/numeric.py", line 3093, in <module>
    from . import fromnumeric
  File "/Users/jdemeyer/sage/local/lib/python2.7/site-packages/numpy/core/fromnumeric.py", line 17, in <module>
    from . import _methods
  File "/Users/jdemeyer/sage/local/lib/python2.7/site-packages/numpy/core/_methods.py", line 158, in <module>
    _NDARRAY_ARRAY_FUNCTION = mu.ndarray.__array_function__
AttributeError: type object 'numpy.ndarray' has no attribute '__array_function__'
Error: could not determine package name
********************************************************************************

[dimpase]

this seems to be an NumPy/pip (?) problem: ​https://github.com/numpy/numpy/issues/12736

[dimpase]

We should upgrade to 1.16.1, where this bug is apparently fixed by ​https://github.com/numpy/numpy/pull/12891

​https://github.com/numpy/numpy/releases/tag/v1.16.1

[jdemeyer]

Replying to dimpase:

We should upgrade to 1.16.1, where this bug is apparently fixed by ​https://github.com/numpy/numpy/pull/12891

There is nothing "fixed". The only thing that changed is a better error message. Apparently we're not properly deleting old versions of numpy.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​0cf4613	Fix legacy uninstall
​0f3dab9	NumPy 1.16.1

[jdemeyer]

Adding a spkg-legacy-uninstall script should fix things.

[dimpase]

OK, looks good.