#25260 closed enhancement (fixed)
Upgrade to NumPy 1.15.2
Reported by:  Samuel Lelièvre  Owned by:  

Priority:  major  Milestone:  sage8.5 
Component:  packages: standard  Keywords:  upgrade, numpy 
Cc:  François Bissey, Ximin Luo, Jeroen Demeyer, Samuel Lelièvre, Elisa Palezzato  Merged in:  
Authors:  Timo Kaufmann, Julian Rüth, Bryan Ginge Chen  Reviewers:  Travis Scrimshaw 
Report Upstream:  Fixed upstream, but not in a stable release.  Work issues:  
Branch:  b151634 (Commits, GitHub, GitLab)  Commit:  
Dependencies:  Stopgaps: 
Description (last modified by )
NumPy 1.15.2 was released.
Our last upgrade was to NumPy 1.13.3 in #24063.
A bug introduced in NumPy 1.15 prevents building NumPy on machines with many CPUs.
The issue is reported and solved upstream:
Change History (81)
comment:1 Changed 4 years ago by
Description:  modified (diff) 

comment:3 Changed 4 years ago by
If we support both kinds of archive, we should probably use the smallest one.
comment:4 Changed 4 years ago by
I don't know how or if I can add changes here, but I have fixed the doctests for the numpy+scipy upgrade here.
Its mostly formatting. Interesting bits:
 here I avoid a numpy deprecation warning (empty array should not be used as false) by always making sure points is a regular array and then comparing its
len
.  the
iprint
kwarg infmin_cobyla
is deprecated and repaced bydisp
Edit: I was a bit too fast and didn't test it properly. I'll post an updated version once thats tested.
comment:6 Changed 4 years ago by
Description:  modified (diff) 

Changing target from Numpy 1.14.3 to 1.14.4.
comment:7 Changed 4 years ago by
Branch:  → public/25260 

Commit:  → 3322ff1a1446cc5d0b8b34b6f7203250df0bdd5c 
Status:  new → needs_review 
I updated the package and applied my doctest fixes. make ptestlong
passes.
New commits:
3322ff1  Upgrade numpy to 1.14.4

comment:8 Changed 4 years ago by
Description:  modified (diff) 

Milestone:  sage8.3 → sage8.4 
Status:  needs_review → needs_work 
Numpy 1.15.0 just came out.
comment:9 Changed 4 years ago by
Summary:  Upgrade to NumPy 1.14.3 → Upgrade to NumPy 1.15.0 

comment:10 Changed 4 years ago by
Dependencies:  → #25755 

comment:11 Changed 4 years ago by
Commit:  3322ff1a1446cc5d0b8b34b6f7203250df0bdd5c → aede54d13c2263b551473c15e264cf32a8b971d0 

comment:12 Changed 4 years ago by
I've done most of the upgrade, but two failures in src/sage/plot/histogram.py
remain. I think both are rooted in the following issue:
> import numpy > numpy.histogram([], range=(1.0,1.0)) TypeError Traceback (most recent call last) <ipythoninput205b803a2f80b> in <module>() > 1 numpy.histogram([], range=(RealNumber('1.0'),RealNumber('1.0'))) /home/timo/repos/sage/local/lib/python2.7/sitepackages/numpy/lib/histograms.pyc in histogram(a, bins, range, normed, weights, density) 674 a, weights = _ravel_and_check_weights(a, weights) 675 > 676 bin_edges, uniform_bins = _get_bin_edges(a, bins, range, weights) 677 678 # Histogram is an integer or a float array depending on the weights. /home/timo/repos/sage/local/lib/python2.7/sitepackages/numpy/lib/histograms.pyc in _get_bin_edges(a, bins, range, weights) 342 # shapes. To avoid this causing problems, we pick a type now and stick 343 # with it throughout. > 344 bin_type = np.result_type(first_edge, last_edge, a) 345 if np.issubdtype(bin_type, np.integer): 346 bin_type = np.result_type(bin_type, float) TypeError: data type not understood
This does not happen outside of sage. It probably has something to do with sages custom number types.
comment:13 Changed 4 years ago by
Dependencies:  #25755 → #25702 

comment:14 Changed 4 years ago by
Commit:  aede54d13c2263b551473c15e264cf32a8b971d0 → 27a7f75a33bf2ce129c01cc9b2a287472e2b0f2e 

comment:15 Changed 4 years ago by
Status:  needs_work → needs_review 

I haven't run full doctests but this fixes the issues in histogram.py
.
comment:16 Changed 4 years ago by
Status:  needs_review → needs_work 

Avoid map
, it's bad for Python 3 compatibility. Replace map(float, options['range'])
> [float(x) for x in options['range']]
(which is also more readable to nonmathematicians)
comment:17 Changed 4 years ago by
Mixing abs tol
and ...
is messy, it should be sufficient to use only abs tol
here:
sage: p = P[0]; p.ydata # abs tol 1e8 [0.0, 0.0, 0.211524990023434..., 0.211524990023434..., 0.244978663126864..., 0.244978663126864..., 0.149106180027477..., 0.149106180027477...]
comment:18 Changed 4 years ago by
Dependencies:  #25702 

Needs rebase to 8.4.beta2.
Also don't forget your name as Author.
comment:20 Changed 4 years ago by
Description:  modified (diff) 

Summary:  Upgrade to NumPy 1.15.0 → Upgrade to NumPy 1.15.1 
comment:21 Changed 4 years ago by
Commit:  27a7f75a33bf2ce129c01cc9b2a287472e2b0f2e → 8494c36f2f1a0cae559ca3600a93d58b90c11a06 

comment:22 Changed 4 years ago by
Authors:  → Timo Kaufmann, Julian Rüth, Bryan Ginge Chen 

I fixed the merge conflict (in sage/plot/histogram.py due to the upgrade to matplotlib in #25702) and I bumped the version to 1.15.1.
There's one remaining failing doctest in histogram.py: #25702 added a test for the function get_minmax_data
with the deprecated normed
option. As of 1.15.0, numpy spits out a DeprecationWarning? and I don't know the standard way to deal with this.
comment:23 followup: 24 Changed 4 years ago by
Thanks! Can't you just switch it to use the density
option?
comment:24 Changed 4 years ago by
Replying to ghtimokau:
Thanks! Can't you just switch it to use the
density
option?
The point of the test seems to be to check the normed
option; note that the first example in the docstring is essentially identical except it uses density
.
I thought about just deleting the test, but I figured I'd ask first.
comment:25 Changed 4 years ago by
Cc:  Elisa Palezzato added 

@epalezzato since you authored #25702, why did you add that test testing the normed
argument?
comment:26 Changed 4 years ago by
The new way is to use the density
option, but Sage wants to give its users some time
to adapt, so keeps normed
working but with a deprecation warning.
The test should check that the deprecation warning works. Please adapt it to do that.
For examples of deprecation warnings in doctests, see:
https://github.com/sagemath/sage/search?q=deprecationwarning
After a year, the deprecated stuff can be removed.
comment:27 Changed 4 years ago by
I ran into an annoying interaction between rel tol
and ...
in the doctesting framework while trying to fix up the test. Reported as #26183.
For now I will try to work around it by making the test a little cruder.
comment:28 Changed 4 years ago by
Commit:  8494c36f2f1a0cae559ca3600a93d58b90c11a06 → 2d605673654c77aa14456a040bcb22947de8bc7f 

comment:29 Changed 4 years ago by
Status:  needs_work → needs_review 

comment:30 Changed 4 years ago by
Status:  needs_review → needs_work 

Oops, I haven't addressed comments 16 and 17 yet.
comment:31 Changed 4 years ago by
Commit:  2d605673654c77aa14456a040bcb22947de8bc7f → 164bfdc4239d849e8bbf561fa271288e56920739 

Branch pushed to git repo; I updated commit sha1. New commits:
164bfdc  Reviewer comments

comment:32 Changed 4 years ago by
Status:  needs_work → needs_review 

comment:33 Changed 4 years ago by
Status:  needs_review → needs_work 

if options.get('range', None)
: do you mean if 'range' in options
?
comment:34 followup: 37 Changed 4 years ago by
I'm not convinced about
if 'normed' in options: from sage.misc.superseded import deprecation deprecation(25260, "the 'normed' option is deprecated. Use 'density' instead.")
because it's essentially copying a deprecation warning that numpy already gives.
comment:35 Changed 4 years ago by
And I'm also not convinced about
points = list(points) # make sure points is a python list
Why was that added?
comment:36 Changed 4 years ago by
Commit:  164bfdc4239d849e8bbf561fa271288e56920739 → 1743f709cfb55db4fd6bc14c71406fdcc74354e6 

Branch pushed to git repo; I updated commit sha1. New commits:
1743f70  'normed' is not quite an alias for 'density'

comment:37 followup: 38 Changed 4 years ago by
Thanks for the quick feedback!
Replying to jdemeyer:
if
options.get('range', None)
: do you meanif 'range' in options
?
if 'range' in options:
does not work, since that also triggers upon passing range=None
(which is the default).
Replying to jdemeyer:
I'm not convinced about
if 'normed' in options: from sage.misc.superseded import deprecation deprecation(25260, "the 'normed' option is deprecated. Use 'density' instead.")because it's essentially copying a deprecation warning that numpy already gives.
What do you suggest? numpy / matplot (which do indeed also throw deprecation warnings) aren't called until either _render_on_subplot
or get_minmax_data
are run. I wanted to give the user a notice when they construct the initial Histogram
object.
Replying to jdemeyer:
And I'm also not convinced about
points = list(points) # make sure points is a python listWhy was that added?
points = list(points)
is necessary to properly treat the case when points
is a generator or other nonlist iterable. Note the test line(enumerate(range(2)), marker = 'o', legend_label='a')
at line 498 which was apparently added for unrelated reasons in #13690.
I made a change to a string which stated that normed
was a deprecated alias for density
. Strictly speaking, normed
has different (incorrect) behavior from density
when constructing a histogram with nonuniform bars.
New commits:
1743f70  'normed' is not quite an alias for 'density'

New commits:
1743f70  'normed' is not quite an alias for 'density'

comment:38 Changed 4 years ago by
Replying to ghbryangingechen:
Replying to jdemeyer:
I'm not convinced about
if 'normed' in options: from sage.misc.superseded import deprecation deprecation(25260, "the 'normed' option is deprecated. Use 'density' instead.")because it's essentially copying a deprecation warning that numpy already gives.
What do you suggest? numpy / matplot (which do indeed also throw deprecation warnings) aren't called until either
_render_on_subplot
orget_minmax_data
are run. I wanted to give the user a notice when they construct the initialHistogram
object.
I'm also in favor of throwing our own deprecation warning. That we're using numpy under the hood is an implementation detail. This way we have more control.
comment:39 Changed 4 years ago by
This is positive_review from me, pending Jeroen's judgement on the deprecation warning. Jeroen, would you mind commenting on that?
comment:40 Changed 4 years ago by
I don't have a strong opinion on the deprecation warning, I was mostly just commenting.
However, points = list(points)
is still wrong! The docs for line2d
state
 ``points``  either a single point (as a tuple), a list of points, a single complex number, or a list of complex numbers.
It's not at all clear to me that calling list()
on a single point or a single complex number is the right thing to do.
comment:41 followup: 42 Changed 4 years ago by
If it helps "to properly treat the case when points is a generator or other nonlist iterable" (quote ghbryangingechen) and doesn't make a difference in the case of a single point, why not?
comment:42 Changed 4 years ago by
Replying to ghtimokau:
doesn't make a difference in the case of a single point
Are you sure about this part?
comment:44 Changed 4 years ago by
Hi, thanks for pushing back on this. The code from this ticket indeed works fine with a single point (entered as a tuple, list, or a list containing a single tuple or single list, etc.).
However, in the course of testing this, I realized that I did not completely understand the addition of points = list(points)
in line2d()
and my answer above to Jeroen was incorrect. Note that this line was actually added by ghtimokau here, so I should probably have deferred to him to explain it.
Anyways, here's what I think now. The real reason this line was added was to properly treat line(numpy.array([]))
(which is one of the doctests). The previous version of line2d
used the more pythoniclooking if not points:
to detect "no points". As of numpy 1.15, the truth value of numpy.array([])
is still False
, but it returns a DeprecationWarning? that this behavior will be removed.
So to get around this, ghtimokau replaced if not points:
with the points = list(points)
and if len(points) == 0
.
Two observations:
 The test
if points in CC or points in CDF:
now always fails.
xydata_from_point_list()
already has this:if not isinstance(points, (list, tuple)): points = list(points)
For the first point, I think we should just remove this test altogether, since it looks like it was originally there to catch some edge cases for if not points:
(e.g. points=CC(0)
). One somewhat related thing that I noticed was that symbolic complex input is not handled well: line([1+I,I])
doesn't work but line([CC(1+I),CC(I)])
does. Maybe this should be dealt with in another ticket for xydata_from_point_list
though.
The second point gives me pause. Is there a cleaner way to detect both an empty list and an empty numpy array in line2d
? I'd appreciate suggestions, as I don't see anything nice at the moment. It does seem like dealing with empty numpy arrays is a common pain point.
The current code does work so we could also just go with what we have.
comment:45 Changed 4 years ago by
Oh I forgot I added that, thanks for investigating :)
I think if not <something that is not a bool>
is an antipattern anyways, even if it still worked. If we don't want to change points
to a list permanently, maybe we could just do if len(list(points)) == 0
instead.
comment:46 Changed 4 years ago by
Commit:  1743f709cfb55db4fd6bc14c71406fdcc74354e6 → 99512a8d5081a009e010cb18b926c469102e2244 

Branch pushed to git repo; I updated commit sha1. New commits:
99512a8  Remove obsolete check for CC or CDF in line2d()

comment:47 Changed 4 years ago by
Unfortunately, if len(list(points)) == 0
breaks the line(enumerate(range(2)), marker = 'o', legend_label='a')
test, since after spitting out all the values once, future calls to list(points)
just return []
.
The commit I've just pushed simply removes the dead code and leaves points = list(points)
and if len(points) == 0:
alone. Since this does work and no better alternatives are coming to mind, I propose we just go with this.
comment:48 Changed 4 years ago by
Status:  needs_work → needs_review 

comment:49 Changed 4 years ago by
Ah right. I now remember that I had the same problem.
Since this does work and no better alternatives are coming to mind, I propose we just go with this.
+1 from me. Jeroen?
comment:50 Changed 4 years ago by
Commit:  99512a8d5081a009e010cb18b926c469102e2244 → 45fb0bd5fd9e25ad46102d9428566b6455ca4d33 

comment:51 followup: 53 Changed 4 years ago by
Description:  modified (diff) 

Status:  needs_review → needs_info 
Summary:  Upgrade to NumPy 1.15.1 → Upgrade to NumPy 1.15.2 
I've changed the target to 1.15.2, as that's just been released. Does anyone understand the numpy patches we have? They seem not to apply to this version and the build currently fails:
[numpy1.15.2] Found local metadata for numpy1.15.2 [numpy1.15.2] Using cached file /Applications/SageMath/upstream/numpy1.15.2.tar.gz [numpy1.15.2] numpy1.15.2 [numpy1.15.2] ==================================================== [numpy1.15.2] Setting up build directory for numpy1.15.2 [numpy1.15.2] Finished extraction [numpy1.15.2] Applying patches from ../patches... [numpy1.15.2] Applying ../patches/fixnumpy1.13.3forpython3.7.patch [numpy1.15.2] patching file setup.py [numpy1.15.2] Hunk #1 succeeded at 381 (offset 11 lines). [numpy1.15.2] The next patch would create the file tools/cythonize.py, [numpy1.15.2] which already exists! Assume R? [n] [numpy1.15.2] Apply anyway? [n] [numpy1.15.2] Skipping patch. [numpy1.15.2] 1 out of 1 hunk ignored [numpy1.15.2] Error applying '../patches/fixnumpy1.13.3forpython3.7.patch' [numpy1.15.2] ************************************************************************ [numpy1.15.2] Error applying patches [numpy1.15.2] ************************************************************************ [numpy1.15.2] Please email sagedevel (http://groups.google.com/group/sagedevel) [numpy1.15.2] explaining the problem and including the log file [numpy1.15.2] /Applications/SageMath/logs/pkgs/numpy1.15.2.log [numpy1.15.2] Describe your computer, operating system, etc. [numpy1.15.2] ************************************************************************ make[3]: *** [/Applications/SageMath/local/var/lib/sage/installed/numpy1.15.2] Error 1 make[2]: *** [allbuild] Error 2
I have no experience with patching so if someone wants fix this themselves rather than explain to me how to do it they're also welcome to do so.
comment:52 Changed 4 years ago by
Commit:  45fb0bd5fd9e25ad46102d9428566b6455ca4d33 → 839e504978764dcd8062ac40ecc811c5c1013d2d 

Branch pushed to git repo; I updated commit sha1. New commits:
839e504  #25260 remove fixnumpy1.13.3forpython3.7.patch

comment:53 Changed 4 years ago by
Replying to ghbryangingechen:
I have no experience with patching so if someone wants fix this themselves rather than explain to me how to do it they're also welcome to do so.
A you can see in the first line of $SAGE_ROOT/build/pkgs/numpy/patches/fixnumpy1.13.3forpython3.7.patch
, there is a link to https://github.com/numpy/numpy/issues/10500 which is now fixed (the patch or a similar one was applied upstream), so it sufice to remove that patch. I just did that and recompiled numpy and scipy, but did not run all doctests.
comment:54 Changed 4 years ago by
By the way, both scipy and numpy offer selftests using pytest), so we should add some spkgcheck
for them (not in that ticket).
comment:55 Changed 4 years ago by
Status:  needs_info → needs_review 

Thanks! For what it's worth, after your patch, all the tests seem to pass for me.
comment:56 Changed 4 years ago by
Reviewers:  → Travis Scrimshaw 

Status:  needs_review → positive_review 
They also pass for me. So I am setting this to a positive review.
comment:57 Changed 4 years ago by
This fails on kucalc (48core linux 64bit):
********************************************************************** File "src/sage/misc/inline_fortran.py", line 134, in sage.misc.inline_fortran._import_module_from_path._import_module_from_path_impl Failed example: fortran(code, globals()) Exception raised: Traceback (most recent call last): File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 659, in _run self.compile_and_execute(example, compiler, test.globs) File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 1070, in compile_and_execute exec(compiled, globs) File "<doctest sage.misc.inline_fortran._import_module_from_path._import_module_from_path_impl[1]>", line 1, in <module> fortran(code, globals()) File "sage/misc/lazy_import.pyx", line 354, in sage.misc.lazy_import.LazyImport.__call__ (build/cythonized/sage/misc/lazy_import.c:3759) return self.get_object()(*args, **kwds) File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/misc/inline_fortran.py", line 96, in __call__ return self.eval(*args, **kwds) File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/misc/inline_fortran.py", line 200, in eval log_string) RuntimeError: failed to compile Fortran code: Traceback (most recent call last): File "<string>", line 1, in <module> File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/sitepackages/numpy/f2py/f2py2e.py", line 648, in main run_compile() File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/sitepackages/numpy/f2py/f2py2e.py", line 633, in run_compile setup(ext_modules=[ext]) File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/sitepackages/numpy/distutils/core.py", line 169, in setup return old_setup(**new_attr) File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/distutils/core.py", line 151, in setup dist.run_commands() File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/sitepackages/numpy/distutils/command/build.py", line 47, in run old_build.run(self) File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/distutils/command/build.py", line 127, in run self.run_command(cmd_name) File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/distutils/cmd.py", line 326, in run_command self.distribution.run_command(command) File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/sitepackages/numpy/distutils/command/build_ext.py", line 262, in run self.build_extensions() File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/distutils/command/build_ext.py", line 449, in build_extensions self.build_extension(ext) File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/sitepackages/numpy/distutils/command/build_ext.py", line 380, in build_extension **kws) File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/sitepackages/numpy/distutils/ccompiler.py", line 337, in CCompiler_compile pool = multiprocessing.pool.ThreadPool(jobs) File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/multiprocessing/pool.py", line 732, in __init__ Pool.__init__(self, processes, initializer, initargs) File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/multiprocessing/pool.py", line 187, in __init__ self._result_handler.start() File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/threading.py", line 736, in start _start_new_thread(self.__bootstrap, ()) thread.error: can't start new thread
Something is wrong on highcpu count machines, I think. Works on others.
comment:58 Changed 4 years ago by
Status:  positive_review → needs_work 

comment:59 Changed 4 years ago by
Any ideas how we could debug this? k8s has 48 cores so we could try to debug this there.
I'll probably start by building there and running just the tests in inline_fortran.py
. Maybe that's already enough to trigger the error.
comment:60 Changed 4 years ago by
Description:  modified (diff) 

For Python packages, the sources on PyPI should always be considered the official sources (unless the project specifically points to GitHub for their sources, which is not the case here).
comment:61 Changed 4 years ago by
Commit:  839e504978764dcd8062ac40ecc811c5c1013d2d → 78057db66f6ecaf6c44b98b3ed682aae480d07e9 

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
78057db  Upgrade numpy to v1.15.2

comment:62 Changed 4 years ago by
I'm getting a timeout instead on that test:
Doctesting 1 file using 8 threads. sage t long src/sage/misc/inline_fortran.py Timed out ********************************************************************** Tests run before process (pid=16924) timed out: sage: from sage.misc.inline_fortran import _import_module_from_path ## line 27 ## sage: modname = '___test__import_module_from_path' ## line 28 ## sage: tmpdir = tmp_dir() ## line 29 ## sage: filename = os.path.join(tmpdir, modname + '.py') ## line 30 ## sage: with open(filename, 'w') as fobj: _ = fobj.write('foo = "bar"') ## line 31 ## sage: mod = _import_module_from_path(modname) ## line 33 ## sage: mod = _import_module_from_path('DoEsNoTeXiSt', path=[tmpdir]) ## line 37 ## sage: mod = _import_module_from_path(modname, path=[tmpdir]) ## line 41 ## sage: mod.foo ## line 42 ## 'bar' sage: sig_on_count() # check sig_on/off pairings (virtual doctest) ## line 44 ## 0 sage: code = ''' C FILE: FIB1.F SUBROUTINE FIB(A,N) C C CALCULATE FIRST N FIBONACCI NUMBERS C INTEGER N REAL*8 A(N) DO I=1,N IF (I.EQ.1) THEN A(I) = 0.0D0 ELSEIF (I.EQ.2) THEN A(I) = 1.0D0 ELSE A(I) = A(I1) + A(I2) ENDIF ENDDO END C END FILE FIB1.F ''' ## line 114 ## sage: fortran(code, globals()) ## line 134 ##  /home/jdemeyer/sagegit/local/lib/python2.7/sitepackages/cysignals/signals.so(+0x851c)[0x3fff7ea0851c] /home/jdemeyer/sagegit/local/lib/python2.7/sitepackages/cysignals/signals.so(+0x8638)[0x3fff7ea08638] /home/jdemeyer/sagegit/local/lib/python2.7/sitepackages/cysignals/signals.so(+0xd370)[0x3fff7ea0d370] [0x3fff7ffa04d8] /lib/powerpc64lelinuxgnu/libc.so.6(__read+0x58)[0x3fff7fc0e02c] [0x3fffd1c122b0] /lib/powerpc64lelinuxgnu/libc.so.6(_IO_file_read+0x64)[0x3fff7fb88674] /lib/powerpc64lelinuxgnu/libc.so.6(+0x8839c)[0x3fff7fb8839c] /lib/powerpc64lelinuxgnu/libc.so.6(_IO_sgetn+0x28)[0x3fff7fb8bac8] /lib/powerpc64lelinuxgnu/libc.so.6(fread+0xb4)[0x3fff7fb78a84] /home/jdemeyer/sagegit/local/lib/libpython2.7.so.1.0(Py_UniversalNewlineFread+0x1e8)[0x3fff7fdc2668] [...]  sage t long src/sage/misc/inline_fortran.py # Timed out  Total time for all tests: 1800.1 seconds cpu time: 0.0 seconds cumulative wall time: 0.0 seconds
comment:63 Changed 4 years ago by
Running this under strace
shows indeed that f2py
is creating a huge number of threads which at some points leads to outofmemory problems. For some reason, this is causing a hang for me instead of an exception.
comment:64 Changed 4 years ago by
I reported the thread issue upstream: https://github.com/numpy/numpy/issues/12087
comment:65 Changed 4 years ago by
Commit:  78057db66f6ecaf6c44b98b3ed682aae480d07e9 → 40f43182129eb1d593cfa68e438515fca5499e95 

Branch pushed to git repo; I updated commit sha1. New commits:
40f4318  Use just 1 thread for compiling to prevent bombing the system

comment:66 Changed 4 years ago by
Commit:  40f43182129eb1d593cfa68e438515fca5499e95 → 3508a0515bd49f08d2fbc58953a8075dcc25c8c4 

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
3508a05  Limit number of threads to prevent bombing the system

comment:67 Changed 4 years ago by
Commit:  3508a0515bd49f08d2fbc58953a8075dcc25c8c4 → b1516346ac15a136cf8ccdf99429feb799f4754f 

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
b151634  Limit number of threads to prevent bombing the system

comment:68 Changed 4 years ago by
Status:  needs_work → needs_review 

comment:69 Changed 4 years ago by
Just for the record I would prefer to monkeypatch instead (or maybe additionally). It seems to be a clear bug in this case and downstream distributions should probably patch, but I still think sage should do what it can to prevent errors even on an unpatched system.
I know your views on this but still wanted to make that point.
comment:70 Changed 4 years ago by
Status:  needs_review → positive_review 

ghtimokau, I would usually agree that we should monkeypatch. However, in this case, there seems to be nothing Sage specific going on so I do not see a strong reason to monkeypatch. In particular, this does not completely break Sage but just breaks things in some rare cases. As this is a numpy bug, distributions should backport it in their numpy if they care.
Probably more importantly, this would be somewhat hard to monkeypatch. Numpy is not imported in a central spot in Sage as the import takes a while, so it is only loaded on demand. Therefore, there is not a single spot where the monkeypatching could happen.
comment:71 Changed 4 years ago by
Yes, I agree that the argument is a bit weak in this case. If it was easy I would still prefer it, but since its not its not that bad.
comment:72 Changed 4 years ago by
Report Upstream:  N/A → Fixed upstream, but not in a stable release. 

The patch was accepted upstream.
comment:73 Changed 4 years ago by
Description:  modified (diff) 

Should we then upgrade to NumPy 1.14.6 now, or wait for NumPy 1.15.3 to be released?
comment:74 Changed 4 years ago by
From a packaging perspective its preferable to upgrade to the latest version now. The latest version just requires a patch for some edge case. Without the upgrade sage is completely incompatible with newer numpy versions (which distros probably have by now).
Thank you for your efforts in fixing this upstream Jeroen!
comment:75 Changed 4 years ago by
Branch:  public/25260 → b1516346ac15a136cf8ccdf99429feb799f4754f 

Resolution:  → fixed 
Status:  positive_review → closed 
comment:76 Changed 4 years ago by
Milestone:  sage8.4 → sage8.5 

This should be retargeted for 8.5.
comment:77 Changed 4 years ago by
Commit:  b1516346ac15a136cf8ccdf99429feb799f4754f 

Next NumPy upgrade at #26643.
comment:78 Changed 4 years ago by
I get two doctest failures on OSX with SageMath version 8.5.beta2, which I believe are related to this ticket, please let me know what other information is helpful (or if I should ignore this).
File "src/sage/schemes/elliptic_curves/height.py", line 1629, in sage.schemes.elliptic_curves.height.EllipticCurveCanonicalHeight.wp_on_grid Failed example: H.wp_on_grid(v,4) Expected: array([[25.43920182, 5.28760943, 5.28760943, 25.43920182], [ 6.05099485, 1.83757786, 1.83757786, 6.05099485], [ 6.05099485, 1.83757786, 1.83757786, 6.05099485], [25.43920182, 5.28760943, 5.28760943, 25.43920182]]) Got: array([[ 25.43920182, 5.28760943, 5.28760943, 25.43920182], [ 6.05099485, 1.83757786, 1.83757786, 6.05099485], [ 6.05099485, 1.83757786, 1.83757786, 6.05099485], [ 25.43920182, 5.28760943, 5.28760943, 25.43920182]]) ********************************************************************** File "src/sage/schemes/elliptic_curves/height.py", line 1637, in sage.schemes.elliptic_curves.height.EllipticCurveCanonicalHeight.wp_on_grid Failed example: H.wp_on_grid(v,4,True) Expected: array([[25.43920182, 5.28760943], [ 6.05099485, 1.83757786], [ 6.05099485, 1.83757786], [25.43920182, 5.28760943]]) Got: array([[ 25.43920182, 5.28760943], [ 6.05099485, 1.83757786], [ 6.05099485, 1.83757786], [ 25.43920182, 5.28760943]]) **********************************************************************
comment:79 Changed 4 years ago by
I don't think it is the first time we have a formatting issue of that kind but I am not sure how to deal with it. The first thing that comes to mind is: are you using OS X' clang or sage's gcc?
comment:80 Changed 4 years ago by
I'm not 100% sure I'm looking in the right place but looks like homebrew gcc, version 8.2.0. As far as I know I haven't tried to change to clang for my sage install.
I have this line in config.log Configured with: ../configure build=x86_64appledarwin17.7.0 prefix=/usr/local/Cellar/gcc/8.2.0 libdir=/usr/local/Cellar/gcc/8.2.0/lib/gcc/8 enablelanguages=c,c++,objc,objc++,fortran programsuffix=8 withgmp=/usr/local/opt/gmp withmpfr=/usr/local/opt/mpfr withmpc=/usr/local/opt/libmpc withisl=/usr/local/opt/isl withsystemzlib enablechecking=release withpkgversion='Homebrew GCC 8.2.0' withbugurl=https://github.com/Homebrew/homebrewcore/issues disablenls
comment:81 Changed 4 years ago by
Hum... Not sure. You should open a new ticket in any case. While numpy could be your culprit if numpy arrays are used it could be something else entirely.
For NumPy 1.13.3 we have the tarball in .zip, should we also use the .zip this time or the .tar.gz?