Opened 11 years ago
Last modified 11 years ago
#9808 closed task
Upgrade numpy to 1.5.0 and scipy to 0.8 — at Version 287
Reported by:  maldun  Owned by:  maldun 

Priority:  major  Milestone:  sage4.6.1 
Component:  packages: standard  Keywords:  numpy, scipy 
Cc:  Merged in:  
Authors:  Stefan Reiterer, Francois Bissey, John Palmieri, David Kirkby, KarlDieter Crisman  Reviewers:  KarlDieter Crisman, David Kirkby, Leif Leonhardy, Francois Bissey 
Report Upstream:  Fixed upstream, but not in a stable release.  Work issues:  
Branch:  Commit:  
Dependencies:  Stopgaps: 
Description (last modified by )
This ticket updates two packages, which must be updated together,
http://sage.math.washington.edu/home/kcrisman/numpy1.5.0.spkg
http://sage.math.washington.edu/home/palmieri/SPKG/scipy0.8.spkg
If you are applying these to any version before Sage 4.6.alpha3, then you also have to update the scipy_sandbox package too (#10092). This has been merged in 4.6.alpha3, though.
http://boxen.math.washington.edu/home/kirkby/patches/scipy_sandbox20071020.p7.spkg
After installing Numpy, one needs to execute sage ba, or do
touch $SAGE_LOCAL/lib/python/sitepackages/Cython/Includes/numpy.pxd
or else one will get runtime warnings. (Or if someone wants less hassle, they can patch sage4.6.alpha3.spkg before building Sage).
trac_9808_numpy_doctest_change.patch in the attachment has to be applied, in order to get all doctests running because some of the output has changed.
For reviewers: changes.txt holds a summary of all changes with reference to the diffs, and links to other tickets
Change History (304)
comment:1 Changed 11 years ago by
 Status changed from new to needs_work
comment:2 Changed 11 years ago by
 Summary changed from Upgrade numpy to 1.5b and scipy to 0.8 to Upgrade numpy to 1.5.0rc1 and scipy to 0.8
numpy 1.5.0rc1: http://www.megaupload.com/?d=KK31RSZV
comment:3 followup: ↓ 4 Changed 11 years ago by
I don't think that we should upgrade to 1.5.0rc1  we should do 1.4.1 for now and wait until 1.5 is released.
comment:4 in reply to: ↑ 3 ; followup: ↓ 16 Changed 11 years ago by
Replying to mhansen:
I don't think that we should upgrade to 1.5.0rc1  we should do 1.4.1 for now and wait until 1.5 is released.
That has to be double checked but maldun says he needs features in 1.5. It may be that the features he wants are 1.4.x and he doesn't know. Do we know roughly when 1.5 will be released? They are at 1.5rc1 9 days after beta2 so it could very well be upon us in a very short time.
comment:5 Changed 11 years ago by
The warnings go away after rebuilding the source. And the doctests only throw warnings but there don't seem to be errors, and the output is correct
@mhansen, fbissey I think numpy has the same issues since version 1.4. and I'm quite sure that resolving it in 1.5.0rc1 would, solve the problem with 1.4.1. too. (and perhaps also with 1.5. later) So I suggest the following procedure: I solve the issues with the latest. Dobule check if 1.4.1 also works, and we decide then which of the versions we keep, or for example keep 1.4.1 in standard and 1.5.0rc1 as experimental
comment:6 followup: ↓ 7 Changed 11 years ago by
If 1.5.0rc1 is merged, #7166 can be closed. I don't know about 1.4.1, but in any case that is not a critical bug.
Some time back I made a Solarisspecific change to Numpy, as I wanted to get it reviewed with the least hassle  that generally means making it Solaris specific, as reviewers are easier to please if it only effects a rarer platform.
But I think the change should be implemented on OS X too. Currently there's a really nasty hack on OS X to build Numpy, that involves a script called fake_gcc
, copying that to $SAGE/LOCAL/bin/gcc
, then using the fake gcc to build Numpy. Finally this fake gcc file gets deleted.
The far neater option is the way I did it on Solaris. I suggest the changes I made for Solaris are implemented whenever SAGE64
is set to "yes", irrespective of whatever platform it may be. Then all this fake gcc rubbish can be removed.
If you want, I can create a patch.
comment:7 in reply to: ↑ 6 Changed 11 years ago by
Replying to drkirkby:
If you want, I can create a patch.
would be nice! But first I have to sort some things out, I hope it will get ready soon...
comment:8 followup: ↓ 9 Changed 11 years ago by
Looking at all that stuff in the spkg and comparing to Gentoo. Not very pretty. Do we really still need to use sage_fortran? On most platform we now use a native fortran compiler rather than a sage shipped one  I don't know if we still do that for OS X. In the patch to gnu.py there is:
+ # Note that sse flags and so on lead to gfortran code that segfaults, so disable arch flags + return opt +
An extract of the Gentoo set up:
export NUMPY_FCONFIG="config_fc noopt noarch"
Actually here is the full set up that you might find interesting:
# See progress in http://projects.scipy.org/scipy/numpy/ticket/573 # with the subtle difference that we don't want to break Darwin where # shared is not a valid linker argument if [[ ${CHOST} != *darwin* ]] ; then appendldflags shared fi # only one fortran to link with: # linking with cblas and lapack library will force # autodetecting and linking to all available fortran compilers use lapack  return [[ z ${FC} ]] && FC=$(tcgetFC) # when fortran flags are set, pic is removed. FFLAGS="${FFLAGS} fPIC" export NUMPY_FCONFIG="config_fc noopt noarch"
The other patches we have are relatively minor, a fix to the f2py man page, a patch for freebsd  that may be usefull, a patch for interix(!). I cannot comment on the cygwin patches but they look very small. We are are a bit more anal with site.cfg, I don't think it is useful in the context of sage  we just have more complex possible combinations of blas/lapack.
The NUMPY_FCONFIG is passed to distutils as an argument of
python setup.py install
i.e. in the end what we do boils down to "python setup.py install ${NUMPY_FCONFIG}".
Can you point me to your solaris fix Dave? I'd like to see if I can tidy all that up in something that requires less patching and is more based on FLAGS settings.
comment:9 in reply to: ↑ 8 Changed 11 years ago by
Replying to fbissey:
Looking at all that stuff in the spkg and comparing to Gentoo. Not very pretty.
What a surprise.
Do we really still need to use sage_fortran? On most platform we now use a native fortran compiler rather than a sage shipped one  I don't know if we still do that for OS X.
I've never understood the need for this SAGE_FORTRAN
. I've tried arguing for it to be removed and use FC
instead, but I had no luck.
A Fortran compiler is still shipped on OS X, but I don't see why the variable FC
can't be made to point to that, rather than have the variable SAGE_FORTRAN
.
Can you point me to your solaris fix Dave? I'd like to see if I can tidy all that up in something that requires less patching and is more based on FLAGS settings.
On Solaris, and some OS X versions, if you want a 64bit build, you must add the compiler flag m64
with gcc. Usually, putting that in CFLAGS
is enough. However, for Numpy that does not work.
Someone came up with a fix for this which was implemented only on OS X, that involved creating a wrapper script called gcc_fake
which basically calls gcc with the m64
option. You can see the script yourself in the top level directory.
Since they had done this only on OS X, it did not work on Solaris. So I came up with a solution for Solaris, but I avoided the wrapper script. Instead I set the variable to CC=gcc m64
I'm attaching a patch, which basically uses the Solaris on any system, including OS X. I think this is the sensible way to do it, not have a wrapper script.
I've not tested the attached patch  not even on Solaris!! But I think you can see what I am trying to do. I was going to try to explain it in words, but a bit of code, even if untested, should be more sensible.
Changed 11 years ago by
An untested patch, which makes Numpy build the same was on OS X as it does on Solaris or other platforms where SAGE64=yes. It removes the stupid wrapper script.
comment:10 followup: ↓ 11 Changed 11 years ago by
Ok  so we still use g95 on some targets. So we need to keep some patches just for these  bother.
comment:11 in reply to: ↑ 10 ; followup: ↓ 12 Changed 11 years ago by
Replying to fbissey:
Ok  so we still use g95 on some targets. So we need to keep some patches just for these  bother.
I don't believe g95 is used anywhere. There are g95 binaries in the Fortran package in Sage, but William said they can be removed. There is a gfortran binary. So as far as I'm aware, all g95 stuff can be removed, but SAGE_FORTRAN
can't be removed.
Dave
comment:12 in reply to: ↑ 11 ; followup: ↓ 14 Changed 11 years ago by
Replying to drkirkby:
Replying to fbissey:
Ok  so we still use g95 on some targets. So we need to keep some patches just for these  bother.
I don't believe g95 is used anywhere. There are g95 binaries in the Fortran package in Sage, but William said they can be removed. There is a gfortran binary. So as far as I'm aware, all g95 stuff can be removed, but
SAGE_FORTRAN
can't be removed.
That's good! That means we probably can give the shove to the gnu.py and init.py patches. I wouldn't be sorry to see the back of these.
There is a comment in SPKG.txt:
Special Update/Build Instructions: * The file $SAGE_ROOT/devel/sage/sage/ext/numpy.pxi comes from this file and must be updated if/when the file src/numpy/doc/cython/numpy.pxi is updated.
I cannot find the file in question in that location. There is however a numpy.pxi under src/numpy/random/mtrand but I am not sure that's the file in question. Furthermore I don't appear to have a numpy.pxi in $SAGE_ROOT/devel/sage/sage/ext/numpy.pxi . Does anyone know if these instructions are obsolete?
comment:13 followup: ↓ 19 Changed 11 years ago by
FYI: Numpy 1.5 has been officially released now.
comment:14 in reply to: ↑ 12 ; followup: ↓ 15 Changed 11 years ago by
Replying to fbissey:
Replying to drkirkby:
Replying to fbissey:
Ok  so we still use g95 on some targets. So we need to keep some patches just for these  bother.
I don't believe g95 is used anywhere. There are g95 binaries in the Fortran package in Sage, but William said they can be removed. There is a gfortran binary. So as far as I'm aware, all g95 stuff can be removed, but
SAGE_FORTRAN
can't be removed.That's good! That means we probably can give the shove to the gnu.py and init.py patches. I wouldn't be sorry to see the back of these.
There is a comment in SPKG.txt:
Special Update/Build Instructions: * The file $SAGE_ROOT/devel/sage/sage/ext/numpy.pxi comes from this file and must be updated if/when the file src/numpy/doc/cython/numpy.pxi is updated.I cannot find the file in question in that location. There is however a numpy.pxi under src/numpy/random/mtrand but I am not sure that's the file in question. Furthermore I don't appear to have a numpy.pxi in $SAGE_ROOT/devel/sage/sage/ext/numpy.pxi . Does anyone know if these instructions are obsolete?
I think I was the one that added those instructions, and I'm pretty sure they're obsolete instructions now. I believe we took care of merging the differences between the two numpy.pxi/pxd files a while ago.
comment:15 in reply to: ↑ 14 Changed 11 years ago by
Replying to jason:
I think I was the one that added those instructions, and I'm pretty sure they're obsolete instructions now. I believe we took care of merging the differences between the two numpy.pxi/pxd files a while ago.
Thank you for the confirmation. I have done some cleaning to numpy's spkginstall and it seems to work as intended on my machine. I guess we'll update to 1.5 and give it a spin.
comment:16 in reply to: ↑ 4 ; followups: ↓ 17 ↓ 18 Changed 11 years ago by
Replying to fbissey:
Replying to mhansen:
I don't think that we should upgrade to 1.5.0rc1  we should do 1.4.1 for now and wait until 1.5 is released.
That has to be double checked but maldun says he needs features in 1.5. It may be that the features he wants are 1.4.x and he doesn't know. Do we know roughly when 1.5 will be released? They are at 1.5rc1 9 days after beta2 so it could very well be upon us in a very short time.
In general though, we should not be upgrading to prerelease versions just because one person needs a feature that's only available in a prerelease. Everyone should not have to suffer the extra risks a prerelease gives unless there are compelling arguments for it.
I realise in this case 1.5 has since been released, so it's immaterial now.
Dave
comment:17 in reply to: ↑ 16 Changed 11 years ago by
Replying to drkirkby:
In general though, we should not be upgrading to prerelease versions just because one person needs a feature that's only available in a prerelease. Everyone should not have to suffer the extra risks a prerelease gives unless there are compelling arguments for it.
+1
comment:18 in reply to: ↑ 16 Changed 11 years ago by
Replying to drkirkby:
In general though, we should not be upgrading to prerelease versions just because one person needs a feature that's only available in a prerelease. Everyone should not have to suffer the extra risks a prerelease gives unless there are compelling arguments for it.
I realise in this case 1.5 has since been released, so it's immaterial now.
Yes, I didn't think it would be worth working on that particular version if it wasn't for the real proximity of the release (curses pari svn upgrade). I would probably have put pressure for 1.4.1 otherwise (very keen to see numpy upgraded).
comment:19 in reply to: ↑ 13 Changed 11 years ago by
Replying to jason:
FYI: Numpy 1.5 has been officially released now.
So I was right =) Til I'm done with 1.5.0rc1 1.5 is out...
So changed so far:
The following doctest had to be rewritten:
File "/home/maldun/sage/sage4.5.2/devel/sage/doc/en/faq/faqusage.rst", line 341: sage: stats.ttest_ind(list([1,2,3,4,5]),list([2,3,4,5,.6])) Expected: doctest...DeprecationWarning... (0.076752955645333687, 0.94070490247380478) Got: (0.076752955645333687, 0.94070490247380478)
That was easy =)
The next prob was:
File "/home/maldun/sage/sage4.5.2/devel/sage/sage/graphs/graph.py", line 615: sage: Graph(A) Exception raised: Traceback (most recent call last): File "/home/maldun/sage/sage4.5.2/local/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/home/maldun/sage/sage4.5.2/local/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/home/maldun/sage/sage4.5.2/local/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_1[22]>", line 1, in <module> Graph(A)###line 615: sage: Graph(A) File "/home/maldun/sage/sage4.5.2/local/lib/python/sitepackages/sage/graphs/graph.py", line 846, in __init__ data = networkx.MultiGraph(data) File "/home/maldun/sage/sage4.5.2/local/lib/python2.6/networkx/classes/graph.py", line 220, in __init__ convert.from_whatever(data,create_using=self) File "/home/maldun/sage/sage4.5.2/local/lib/python2.6/networkx/convert.py", line 157, in from_whatever if isinstance(thing,numpy.core.defmatrix.matrix) or \ AttributeError: 'module' object has no attribute 'defmatrix'
that was also easy. defmatrix just changed from numpy.core to numpy.matrix to numpy.matrixlib Only changed that line in /local/lib/python2.6/networkx/classes/graph.py
Here comes now a trickier one:
sage t valgrind "devel/sage/sage/rings/polynomial/polynomial_element.pyx" Total time for all tests: 716.4 seconds maldun@zauberbuch:~/sage/sage4.5.2$ sage t valgrind "devel/sage/sage/rings/polynomial/real_roots.pyx > " ERROR: File ./devel/sage/sage/rings/polynomial/real_roots.pyx is missing  The following tests failed: ./devel/sage/sage/rings/polynomial/real_roots.pyx # File not found Total time for all tests: 0.0 seconds maldun@zauberbuch:~/sage/sage4.5.2$ sage t valgrind "devel/sage/sage/rings/polynomial/real_roots.pyx" sage t valgrind "devel/sage/sage/rings/polynomial/real_roots.pyx" ********************************************************************** File "/home/maldun/sage/sage4.5.2/devel/sage/sage/rings/polynomial/real_roots.pyx", line 1819, in __main__.example_76 Failed example: oc.find_roots()###line 3064:_sage_ >>> oc.find_roots() Expected nothing Got: doctest:1: RuntimeWarning: tp_compare didn't return 1 or 2 for exception ********************************************************************** File "/home/maldun/sage/sage4.5.2/devel/sage/sage/rings/polynomial/real_roots.pyx", line 1840, in __main__.example_77 Failed example: oc.find_roots()###line 3085:_sage_ >>> oc.find_roots() Expected nothing Got: doctest:1: RuntimeWarning: tp_compare didn't return 1 or 2 for exception ********************************************************************** File "/home/maldun/sage/sage4.5.2/devel/sage/sage/rings/polynomial/real_roots.pyx", line 1934, in __main__.example_80 Failed example: oc.find_roots()###line 3157:_sage_ >>> oc.find_roots() Expected nothing Got: doctest:1: RuntimeWarning: tp_compare didn't return 1 or 2 for exception ********************************************************************** File "/home/maldun/sage/sage4.5.2/devel/sage/sage/rings/polynomial/real_roots.pyx", line 2320, in __main__.example_98 Failed example: real_roots(x**Integer(5) * (x**Integer(2)  Integer(9999))**Integer(2)  Integer(1))###line 3870:_sage_ >>> real_roots(x^5 * (x^2  9999)^2  1) Expected: [((29274496381311/9007199254740992, 419601125186091/2251799813685248), 1), ((2126658450145849453951061654415153249597/21267647932558653966460912964485513216, 4253316902721330018853696359533061621799/42535295865117307932921825928971026432), 1), ((1063329226287740282451317352558954186101/10633823966279326983230456482242756608, 531664614358685696701445201630854654353/5316911983139663491615228241121378304), 1)] Got: doctest:1: RuntimeWarning: tp_compare didn't return 1 or 2 for exception [((29274496381311/9007199254740992, 419601125186091/2251799813685248), 1), ((2126658450145849453951061654415153249597/21267647932558653966460912964485513216, 4253316902721330018853696359533061621799/42535295865117307932921825928971026432), 1), ((1063329226287740282451317352558954186101/10633823966279326983230456482242756608, 531664614358685696701445201630854654353/5316911983139663491615228241121378304), 1)] ********************************************************************** 4 items had failures: 1 of 10 in __main__.example_76 1 of 9 in __main__.example_77 1 of 10 in __main__.example_80 1 of 44 in __main__.example_98 ***Test Failed*** 4 failures. [227.6 s]  The following tests failed: sage t valgrind "devel/sage/sage/rings/polynomial/real_roots.pyx" Total time for all tests: 227.6 seconds
This is not the only one, but the root of evil seems to be in real_roots (how ironic...) more precisly: I tracked it down, with help of valgrind, to the class ocean. Has anyone an Idea??
comment:20 Changed 11 years ago by
 Summary changed from Upgrade numpy to 1.5.0rc1 and scipy to 0.8 to Upgrade numpy to 1.5 and scipy to 0.8
comment:21 Changed 11 years ago by
Update: Found the problem. see: http://groups.google.com/group/cythonusers/browse_thread/thread/624c696293b7fe44
It seems that all versions 1.5.x hold this bug.... Tried downgrading to 1.4.1. and all corrections I have done so far worked.
I will now apply the patch from drkirkby, pack it again, test it overnight and hopefully we are done with 1.4.1, and hopefully they get it right in the next time. perhaps I should send the numpy/scipy guys the doctests I've done so far
scipy 0.8. don't seem to make any problems so far
comment:22 Changed 11 years ago by
 Summary changed from Upgrade numpy to 1.5 and scipy to 0.8 to Upgrade numpy to 1.4.1 and scipy to 0.8
comment:23 followup: ↓ 24 Changed 11 years ago by
an doctest I oversaw:
Type ``numpy.typecodes`` for a list of the possible typecodes:: sage: import numpy sage: sorted(numpy.typecodes.items()) [('All', '?bhilqpBHILQPfdgFDGSUVOMm'), ('AllFloat', 'fdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Datetime', 'Mm'), ('Float', 'fdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')]
The output is now:
[('All', '?bhilqpBHILQPfdgFDGSUVOMm'), ('AllFloat', 'fdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Datetime', 'Mm'), ('Float', 'fdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')]
But since it's an extension, and no downgrade it is also no prob
comment:24 in reply to: ↑ 23 ; followup: ↓ 25 Changed 11 years ago by
Replying to maldun:
an doctest I oversaw:
Type ``numpy.typecodes`` for a list of the possible typecodes:: sage: import numpy sage: sorted(numpy.typecodes.items()) [('All', '?bhilqpBHILQPfdgFDGSUVOMm'), ('AllFloat', 'fdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Datetime', 'Mm'), ('Float', 'fdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')]The output is now:
[('All', '?bhilqpBHILQPfdgFDGSUVOMm'), ('AllFloat', 'fdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Datetime', 'Mm'), ('Float', 'fdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')]But since it's an extension, and no downgrade it is also no prob
the doctest should be fixed as well. I have attached a new version of spkginstall for numpy if you cared to give it a spin. It completely drop the noncygwin patches. It may need a little bit of work as I haven't looked yet at how you included Dave's fix.
comment:25 in reply to: ↑ 24 Changed 11 years ago by
 Status changed from needs_work to needs_review
Replying to fbissey:
the doctest should be fixed as well. I have attached a new version of spkginstall for numpy if you cared to give it a spin. It completely drop the noncygwin patches. It may need a little bit of work as I haven't looked yet at how you included Dave's fix.
Your install file worked well for me. I have merged it, such that the patches are getting applied. All doctests work now. You can download the packages now at: http://code.google.com/p/spkgupload/downloads/detail?name=numpy1.4.1.spkg and http://code.google.com/p/spkgupload/downloads/detail?name=scipy0.8.spkg
but you also have to install a modified version of networkx. I have opened a ticket for this, and this also includes a package: http://trac.sagemath.org/sage_trac/ticket/9854
next I will add a patch which includes the changed doctests
comment:26 Changed 11 years ago by
One important note: after updating numpy one has to rebuild the source with sage ba, or else you get runtime warnings.
The reason is the size change in some of the numpy functions, and then the .pyx files have to be rebuild
comment:27 followup: ↓ 28 Changed 11 years ago by
Thanks for the patches. I will probably introduce these in sageongentoo in short order. About networkx, you realize that sage4.5.3 will use networkx1.2? see ticket #9567 So have you tried to see if networkx1.2 needs patching (my guess is no, otherwise I would know by now from gentoo).
comment:28 in reply to: ↑ 27 ; followup: ↓ 29 Changed 11 years ago by
Replying to fbissey:
Thanks for the patches. I will probably introduce these in sageongentoo in short order. About networkx, you realize that sage4.5.3 will use networkx1.2? see ticket #9567 So have you tried to see if networkx1.2 needs patching (my guess is no, otherwise I would know by now from gentoo).
I've seen that networkx 1.2. is out but it doesn't seem to work with my sage4.2 with numpy 1.4.1 nor with my sage4.1 with the old numpy so I created this spkg in order that it can be used with old versions of sage too
But it seems the line where the problems come from doesn't exist in the new version of networkx anymore. Could you give the numpy packages a spin with the new versions of numpy/scipy? It is very time consuming for me to rebuild every time the source. Thanx in advance!
comment:29 in reply to: ↑ 28 Changed 11 years ago by
ith the new versions of numpy/scipy? It is very time consuming for me to rebuild every time the source. Thanx in advance!
sorry I meant new versions of networkx...
comment:30 Changed 11 years ago by
Understood. We already ship networkx1.2 with sage4.5.2 on sageongentoo [because networkx1.0.1 has been removed from Gentoo] so I can test your patches along with networkx. It may take 1 or 2 days for me to fit it in my schedule.
comment:31 Changed 11 years ago by
ok. Since I had some time this afternoon I build a version of sage1.4.3.alpha1 (which also has networkx1.2.) on my machine, applied the packages, the patch, rebuild the source with sage ba and everything worked fine!
./sage tp 4 long devel/sagenumpy .... sage t long devel/sagenumpy/doc/en/a_tour_of_sage/index.rst [23.1 s] sage t long devel/sagenumpy/doc/en/thematic_tutorials/group_theory.rst [247.1 s]  All tests passed! Timings have been updated. Total time for all tests: 7320.5 seconds
So networkx1.2 works!
For Info: I use Ubuntu 10.04 on my machine. So I think it would still be good if someone else would test it
comment:32 Changed 11 years ago by
 Milestone changed from sage4.6 to sage4.5.3
comment:33 followup: ↓ 34 Changed 11 years ago by
It all works in sageongentoo.
However I think there are a few points that should be taken into consideration for both numpy and scipy. Just before posting my spkginstall I edited it to remove a change that I now think is probably necessary. I had set FC to SAGE_FORTRAN... Why? Because numpy tries to autodetect a fortran compiler and will take a gnu compiler first. Unless you set FC. Which means if Dave tries to compile sage with sunstudio on a machine that has also gfortran he is in for a world of hurt. So I think we should either set FC to SAGE_FORTRAN in numpy and possibly scipy. The other option is to set it globally but that may cause problems on OSX.
Thoughts? Francois
comment:34 in reply to: ↑ 33 Changed 11 years ago by
Replying to fbissey:
It all works in sageongentoo.
However I think there are a few points that should be taken into consideration for both numpy and scipy. Just before posting my spkginstall I edited it to remove a change that I now think is probably necessary. I had set FC to SAGE_FORTRAN... Why? Because numpy tries to autodetect a fortran compiler and will take a gnu compiler first. Unless you set FC. Which means if Dave tries to compile sage with sunstudio on a machine that has also gfortran he is in for a world of hurt. So I think we should either set FC to SAGE_FORTRAN in numpy and possibly scipy. The other option is to set it globally but that may cause problems on OSX.
Thoughts? Francois
A very pragmatic thougt: If it doesn"t hurt to set FC to SAGE_FORTRAN, why not? But I think it"s better not to do it globally. Because then we could cause more problems then we solve. Can you give a modified spkginstall? I can try it out over night then.
But I think it would also be good to get some feedback for Solaris and OSX for the current versions. Then we could decide to keep the current, or to take the other.
comment:35 followup: ↓ 36 Changed 11 years ago by
Good news: It seems that the problem from numpy1.5.0 has been resolved http://projects.scipy.org/numpy/ticket/1605 I don't think it's a big deal. Shall I give it a try, or should we stick to 1.4.1 for now?
comment:36 in reply to: ↑ 35 Changed 11 years ago by
Replying to maldun:
Good news: It seems that the problem from numpy1.5.0 has been resolved http://projects.scipy.org/numpy/ticket/1605 I don't think it's a big deal. Shall I give it a try, or should we stick to 1.4.1 for now?
...or wait til 1.5.1 is out?
comment:37 Changed 11 years ago by
May be wait until numpy1.5.1. it is not a big deal right now I guess. May be we should discuss it on sagedevel. At the moment we don't really have much testing apart from the fact I unleashed the upgrade to numpy1.4.1/scipy0.8 on sageongentoo users. I am not sure we can believably merge this in 4.5.3 unless it takes more than one week before it is released. If we target 4.5.3 I say we stay with what we have now, if we target 4.6 which should be a little further down the track we go for 1.5.x.
comment:38 Changed 11 years ago by
I updated my spkg_install as you requested. The other thing about using sage_fortran, that I had forgotten to change back when I removed it, is that it includes "fPIC". I didn't check the fortran spkg but hopefully the work done by Dave to set the correct pic flag is picked up in sage_fortran.
If we don't adopt this, FFLAG="fPIC" (or whatever mechanism Dave came up with) should be added in my previous version.
comment:39 Changed 11 years ago by
Replying to fbissey:
May be wait until numpy1.5.1. it is not a big deal right now I guess. May be we should discuss it on sagedevel. At the moment we don't really have much testing apart from the fact I unleashed the upgrade to numpy1.4.1/scipy0.8 on sageongentoo users. I am not sure we can believably merge this in 4.5.3 unless it takes more than one week before it is released. If we target 4.5.3 I say we stay with what we have now, if we target 4.6 which should be a little further down the track we go for 1.5.x.
Ok I think we should do the following: Let's stick to 1.4.1 since it is necessary for scipy 0.8. and quite well tested yet, in contrary to 1.5.x and it seems to work. Especially since the update from scipy 0.7 to 0.8 holds a lot of changes, and we don't know if they build some more bugs into numpy 1.5.1... If it turns out that updating from 1.4.1 to 1.5.1 is no problem, then well... packing a new package would'nt be the prob I think.
@new spkg: Ok I will give it a try tonight!
comment:40 followup: ↓ 41 Changed 11 years ago by
There's a lot of packages etc. on this ticket. Can someone provide a concise list of what would be needed to apply on a given platform? (For instance, in a very cursory glance, the networkx package being here mystifies.)
comment:41 in reply to: ↑ 40 Changed 11 years ago by
Replying to kcrisman:
There's a lot of packages etc. on this ticket. Can someone provide a concise list of what would be needed to apply on a given platform? (For instance, in a very cursory glance, the networkx package being here mystifies.)
Needed are: http://code.google.com/p/spkgupload/downloads/detail?name=numpy1.4.1.spkg http://code.google.com/p/spkgupload/downloads/detail?name=scipy0.8.spkg
after installing numpy, one needs so execute sage ba, or else one get's runtime warnings.
You also need networkx1.2. (the other networkx is just a small hack for testing, because 1.2. didn't build correctly on my machine, but this is obsolete now, since networkx1.2 is merged into sage 4.3.alpha1)
comment:42 Changed 11 years ago by
 Description modified (diff)
comment:43 Changed 11 years ago by
 Description modified (diff)
comment:44 Changed 11 years ago by
I gave the links and (current) build instructions into the discription, so that people can find the latest versions more quickly
comment:45 Changed 11 years ago by
 Description modified (diff)
comment:46 Changed 11 years ago by
 Description modified (diff)
Included direct links to files in description.
comment:47 Changed 11 years ago by
 Milestone changed from sage4.5.3 to sage4.6
comment:48 Changed 11 years ago by
 Description modified (diff)
comment:49 Changed 11 years ago by
This seems to work fine on Mac OS X 10.6.4 Intel. I will try to test it tomorrow on a PowerPC machine. Note: I haven't looked at the spkg installs or anything, this is just a data point with regard to whether it works, not a review.
comment:50 followup: ↓ 51 Changed 11 years ago by
This fails very early on in Cygwin with
building library "npymath" sources creating build/src.cygwin1.7.5i6862.6/src conv_template:> build/src.cygwin1.7.5i6862.6/src/npy_math.c error: src/npy_math.c.src: No such file or directory Error building numpy.
comment:51 in reply to: ↑ 50 Changed 11 years ago by
Replying to mhansen:
This fails very early on in Cygwin with
building library "npymath" sources creating build/src.cygwin1.7.5i6862.6/src conv_template:> build/src.cygwin1.7.5i6862.6/src/npy_math.c error: src/npy_math.c.src: No such file or directory Error building numpy.
Ok I think I have the picture: some files from /src/numpy/core/src moved to subfolders this have to be changed in the cygwincoresetup.py
I'm installing already cygwin on my laptop and look if I get it up and running, but if anyone could help out I would be thankful!
comment:52 followup: ↓ 53 Changed 11 years ago by
I've tested it out, and everything works if you just remove the cygwincoresetup.py patch.
comment:53 in reply to: ↑ 52 ; followup: ↓ 54 Changed 11 years ago by
Replying to mhansen:
I've tested it out, and everything works if you just remove the cygwincoresetup.py patch.
I was just about to upload a patch... But great this is even better than my solution! (and simpler) The reason why it fails is just that cygwincoresetup.py is outdated.
one question: (because I'm quite the noob...) If I want to upload a modified version of numpy1.4.1 do I have to overwrite the old version, or shall I rename it with numpy1.4.1.p0.spkg?
comment:54 in reply to: ↑ 53 Changed 11 years ago by
Replying to maldun:
one question: (because I'm quite the noob...) If I want to upload a modified version of numpy1.4.1 do I have to overwrite the old version, or shall I rename it with numpy1.4.1.p0.spkg?
Overwrite the old one.
comment:55 Changed 11 years ago by
Thanx! Numpy is now updated with
 changes from fbissey to the installer (it worked for me)
 removed the cygwincoresetup.py patch
comment:56 Changed 11 years ago by
The SciPy? spkg works fine in Cygwin.
comment:57 followup: ↓ 58 Changed 11 years ago by
I'm getting a corrupted package message with the packages right now while trying it on a different system. Did something get changed with respect to the file with this latest update?
comment:58 in reply to: ↑ 57 ; followup: ↓ 59 Changed 11 years ago by
Replying to kcrisman:
I'm getting a corrupted package message with the packages right now while trying it on a different system. Did something get changed with respect to the file with this latest update?
Yes we did some changes to the spkginstall. But I downloaded and installed the package right now again. So I don't know wehre the problem is lying?
comment:59 in reply to: ↑ 58 ; followup: ↓ 60 Changed 11 years ago by
Replying to maldun:
Replying to kcrisman:
I'm getting a corrupted package message with the packages right now while trying it on a different system. Did something get changed with respect to the file with this latest update?
Yes we did some changes to the spkginstall. But I downloaded and installed the package right now again. So I don't know wehre the problem is lying?
Did you use sage pkg
to create it or just do some kind of compression? I get a message about tar: this does not look like a file archive
and tar: Archive contains obsolescent base64 headers
. I just installed the optional biopython package on this machine to test things, so the machine shouldn't be the problem (and it built Sage 4.5.2 just fine).
comment:60 in reply to: ↑ 59 ; followup: ↓ 61 Changed 11 years ago by
Replying to kcrisman:
Replying to maldun:
Replying to kcrisman:
I'm getting a corrupted package message with the packages right now while trying it on a different system. Did something get changed with respect to the file with this latest update?
Yes we did some changes to the spkginstall. But I downloaded and installed the package right now again. So I don't know wehre the problem is lying?
Did you use
sage pkg
to create it or just do some kind of compression? I get a message abouttar: this does not look like a file archive
andtar: Archive contains obsolescent base64 headers
. I just installed the optional biopython package on this machine to test things, so the machine shouldn't be the problem (and it built Sage 4.5.2 just fine).
Ok I repacked it now with pkg and uploaded it. Does it work now? And sorry for the newby question: what is the difference between pkg and just compress it? Is sage using a different version of tar? Because in fact a spkg is noting else then a tar.gz with different ending.
comment:61 in reply to: ↑ 60 Changed 11 years ago by
Ok I repacked it now with pkg and uploaded it. Does it work now? And sorry for the newby question: what is the difference between pkg and just compress it? Is sage using a different version of tar? Because in fact a spkg is noting else then a tar.gz with different ending.
No, it's a little more than that  it has an SPKG.txt, it has Mercurial repositories, etc. True that the file type is the same. But sage pkg tests several things, and a successful review would check all that. See here for more info.
comment:62 Changed 11 years ago by
 Reviewers set to KarlDieter Crisman
Hmm, when I open it manually it seems fine. Though when I run sage pkg
on that new folder, I get
HG REPO: Unchecked in changes
So you'll need to fix that. Putting myself as a reviewer now.
Calling sage f
on this new spkg I made *does* seem to work, for whatever reason. This doesn't make sense to me  why is tar
complaining about yours? Hopefully someone who knows about file systems will test this soon and explain why this causes a problem on Mac OS X 10.4 Tiger.
comment:63 followup: ↓ 65 Changed 11 years ago by
Okay, that worked. Scipy seems to be installing fine directly from the website, I'm not sure why this happened.
comment:64 Changed 11 years ago by
Thanx for the hints! I updated the mercurial entries now to the latest version. Uploaded changed packages. Everything should be fine now
comment:65 in reply to: ↑ 63 Changed 11 years ago by
Replying to kcrisman:
Okay, that worked. Scipy seems to be installing fine directly from the website, I'm not sure why this happened.
I have an Idea: different versions of .tar may cause problems (so far as I know) If you look at the file sizes only it seems that sage does a different kind of compression.
comment:66 followup: ↓ 67 Changed 11 years ago by
I get the following error with the Scipy for some reason.
creating build/temp.macosx10.4ppc2.6/scipy/special creating build/temp.macosx10.4ppc2.6/scipy/special/c_misc compile options: 'I/Users/student/Desktop/sage4.5.2/local/lib/python2.6/sitepackages/numpy/core/include c' gcc: scipy/special/c_misc/fsolve.c scipy/special/c_misc/fsolve.c: In function 'false_position': scipy/special/c_misc/fsolve.c:58: warning: 'f3' may be used uninitialized in this function scipy/special/c_misc/fsolve.c:58: warning: 'x3' may be used uninitialized in this function gcc: scipy/special/c_misc/gammaincinv.c scipy/special/c_misc/gammaincinv.c:1:20: error: Python.h: No such file or directory In file included from /Users/student/Desktop/sage4.5.2/local/lib/python2.6/sitepackages/numpy/core/include/numpy/npy_math.h:5, from scipy/special/c_misc/gammaincinv.c:2: /Users/student/Desktop/sage4.5.2/local/lib/python2.6/sitepackages/numpy/core/include/numpy/npy_common.h:79:2: error: #error Must use Python with unicode enabled. scipy/special/c_misc/gammaincinv.c:1:20: error: Python.h: No such file or directory In file included from /Users/student/Desktop/sage4.5.2/local/lib/python2.6/sitepackages/numpy/core/include/numpy/npy_math.h:5, from scipy/special/c_misc/gammaincinv.c:2: /Users/student/Desktop/sage4.5.2/local/lib/python2.6/sitepackages/numpy/core/include/numpy/npy_common.h:79:2: error: #error Must use Python with unicode enabled. error: Command "gcc fnostrictaliasing g O2 DNDEBUG g fwrapv O3 Wall Wstrictprototypes I/Users/student/Desktop/sage4.5.2/local/lib/python2.6/sitepackages/numpy/core/include c scipy/special/c_misc/gammaincinv.c o build/temp.macosx10.4ppc2.6/scipy/special/c_misc/gammaincinv.o" failed with exit status 1 Error building scipy.
Did you use some special encoding for some of your stuff?
comment:67 in reply to: ↑ 66 Changed 11 years ago by
Replying to kcrisman:
I get the following error with the Scipy for some reason.
creating build/temp.macosx10.4ppc2.6/scipy/special creating build/temp.macosx10.4ppc2.6/scipy/special/c_misc compile options: 'I/Users/student/Desktop/sage4.5.2/local/lib/python2.6/sitepackages/numpy/core/include c' gcc: scipy/special/c_misc/fsolve.c scipy/special/c_misc/fsolve.c: In function 'false_position': scipy/special/c_misc/fsolve.c:58: warning: 'f3' may be used uninitialized in this function scipy/special/c_misc/fsolve.c:58: warning: 'x3' may be used uninitialized in this function gcc: scipy/special/c_misc/gammaincinv.c scipy/special/c_misc/gammaincinv.c:1:20: error: Python.h: No such file or directory In file included from /Users/student/Desktop/sage4.5.2/local/lib/python2.6/sitepackages/numpy core/include/numpy/npy_math.h:5, from scipy/special/c_misc/gammaincinv.c:2: /Users/student/Desktop/sage4.5.2/local/lib/python2.6/sitepackages/numpy/core/include/numpy/npy_common.h:79:2: error: #error Must use Python with unicode enabled. scipy/special/c_misc/gammaincinv.c:1:20: error: Python.h: No such file or directory In file included from /Users/student/Desktop/sage4.5.2/local/lib/python2.6/sitepackages/numpy/core/include/numpy/npy_math.h:5, from scipy/special/c_misc/gammaincinv.c:2: /Users/student/Desktop/sage4.5.2/local/lib/python2.6/sitepackages/numpy/core/include/numpy/npy_common.h:79:2: error: #error Must use Python with unicode enabled. error: Command "gcc fnostrictaliasing g O2 DNDEBUG g fwrapv O3 Wall Wstrictprototypes I/Users/student/Desktop/sage4.5.2/local/lib/python2.6/sitepackages/numpy/core/include c scipy/special/c_misc/gammaincinv.c o build/temp.macosx10.4ppc2.6/scipy/special/c_misc/gammaincinv.o" failed with exit status 1 Error building scipy.
Did you use some special encoding for some of your stuff?
Nope. Perhaps something else did go wrong. I repacked it now on a different machine with a newer version of ubuntu and uploaded it. Hope this works. The first install worked for me.
comment:68 followup: ↓ 69 Changed 11 years ago by
Sadly, same error. This is probably on my end, but this is a supported architecture so it's important to know what went wrong. I don't think that the ./sage ba
would have anything to do with it.
So I suspect that the "missing: Python.h
is the problem, as I've seen a few other things about this online (including ones like this one, where /include/Python.h definitely exists, here it's within the $SAGE_ROOT
directory). I wonder why it's not finding it this time?
comment:69 in reply to: ↑ 68 ; followup: ↓ 70 Changed 11 years ago by
Replying to kcrisman:
Sadly, same error. This is probably on my end, but this is a supported architecture so it's important to know what went wrong. I don't think that the
./sage ba
would have anything to do with it.So I suspect that the "missing:
Python.h
is the problem, as I've seen a few other things about this online (including ones like this one, where /include/Python.h definitely exists, here it's within the$SAGE_ROOT
directory). I wonder why it's not finding it this time?
Question: Have you tried to give the direct path to the compiler? And does in OS X gcc points to /include/Python.h because in sage it's /include/python2.6/Python.h ?
comment:70 in reply to: ↑ 69 Changed 11 years ago by
Replying to maldun:
Replying to kcrisman:
Sadly, same error. This is probably on my end, but this is a supported architecture so it's important to know what went wrong. I don't think that the
./sage ba
would have anything to do with it.So I suspect that the "missing:
Python.h
is the problem, as I've seen a few other things about this online (including ones like this one, where /include/Python.h definitely exists, here it's within the$SAGE_ROOT
directory). I wonder why it's not finding it this time?Question: Have you tried to give the direct path to the compiler?
No  how would I do that?
And does in OS X gcc points to /include/Python.h because in sage it's /include/python2.6/Python.h ?
I find that very unlikely, since everything else works fine and in general Sage builds fine on OS X 10.410.6. But sometimes things get mixed up, I'm sure. There aren't any weird env variables that would do that here, though, I don't think.
comment:71 followup: ↓ 72 Changed 11 years ago by
A question: do we impose python to be built with or without unicode support? If not, is the support enabled by default depending on the platform? To me it looks like numpy ships headers that are encoded with unicode and that your sage's python chock on them, that's the first and foremost error. distutils seem to be unable to work things properly after that.
Two things could be tried: 1) have python built with unicode support on OSX. 2) "vet" numpy to convert the headers in /Users/student/Desktop/sage4.5.2/local/lib/python2.6/sitepackages/numpy/core/include/numpy/ to plain text. Actually you probably can do that easily and manually right now on the installed headers.
Give it a spin and see if it works. If it does we'll have to do something about that piece of unicode one way or another.
comment:72 in reply to: ↑ 71 Changed 11 years ago by
Replying to fbissey:
A question: do we impose python to be built with or without unicode support? If not, is the support enabled by default depending on the platform? To me it looks like numpy ships headers that are encoded with unicode and that your sage's python chock on them, that's the first and foremost error. distutils seem to be unable to work things properly after that.
Two things could be tried: 1) have python built with unicode support on OSX. 2) "vet" numpy to convert the headers in /Users/student/Desktop/sage4.5.2/local/lib/python2.6/sitepackages/numpy/core/include/numpy/ to plain text. Actually you probably can do that easily and manually right now on the installed headers.
Umm... except that I have no idea how I might do that. What exact changes would I need to make to Python.h (and/or something else)? It seems unwise to make yet another python spkg change to deal with this; did numpy only recently (between whatever Sage has now and 1.4.1) start making its headers unicode? I am not exactly a C expert.
That said, 1) seems to be more canonical, though I guess if it makes it work, 2) could be an option. This is the sort of thing Leif and/or drkirkby usually have an informed opinion on...
Give it a spin and see if it works. If it does we'll have to do something about that piece of unicode one way or another.
comment:73 Changed 11 years ago by
2) would be good to find if this is indeed the problem. Doing it is another story. Do you have iconv on OSX? I would think so as it is needed by R amongst other.
so go into the right folder and try:
iconv f UTF8 t ISO88591 *.h
comment:74 followup: ↓ 80 Changed 11 years ago by
Same error.
comment:75 followups: ↓ 76 ↓ 77 Changed 11 years ago by
In the spkginstall, there is the line
unset CFLAGS LDFLAGS CXXFLAGS SHAREDFLAGS
could this be somehow connected to our problem, because if we unset the flags, gcc doesn't point at Python.h in OS X?
comment:76 in reply to: ↑ 75 Changed 11 years ago by
Replying to maldun:
In the spkginstall, there is the line
unset CFLAGS LDFLAGS CXXFLAGS SHAREDFLAGScould this be somehow connected to our problem, because if we unset the flags, gcc doesn't point at Python.h in OS X?
Ok I reformulate it: Can we use the CFLAGS and LDFLAGS to give to tell gcc to link Python.h directly?
comment:77 in reply to: ↑ 75 Changed 11 years ago by
Replying to maldun:
In the spkginstall, there is the line
unset CFLAGS LDFLAGS CXXFLAGS SHAREDFLAGScould this be somehow connected to our problem, because if we unset the flags, gcc doesn't point at Python.h in OS X?
But this is not a change in the spkginstall script. Sage has built fine on this computer in the very recent past. The new spkg also works fine on other (newer) Macs.
From the numpy website
Reusing npymath In 1.3.0, we started putting portable C math routines in npymath library, so that people can use those to write portable extensions. Unfortunately, it was not possible to easily link against this library: in 1.4.0, support has been added to numpy.distutils so that 3rd party can reuse this library. See coremath documentation for more information.
This seems quite possibly relevant (given the error message). After all, if Python.h
isn't imported, then maybe the first error seen is that PY_UNICODE_DEF
or whatever it's called wouldn't be set... maybe?
I'm just trying to throw out an idea here, I am woefully uniformed when it comes to headers and things like that. But I feel like it has to be a change in Scipy or Numpy, since the previous Sage built fine on this machine.
comment:78 followup: ↓ 79 Changed 11 years ago by
I have another wild theory: Haven't you said, that the package build one time? And then not. Is this correct?
The only thing I changed was updating the mercurial files, and then commiting. Therefore I had to write the commit messages, and I do this in kate. kate's standard encoding is utf8. ( My computer's language is german, so everything is utf8) This (or just packing it, because the encoding on my machine is not unicode) did change something to the encoding. What happens if you untar the package use the iconv from up there, and repack it on your machine?
comment:79 in reply to: ↑ 78 Changed 11 years ago by
Replying to maldun:
I have another wild theory: Haven't you said, that the package build one time? And then not. Is this correct?
The only thing I changed was updating the mercurial files, and then commiting. Therefore I had to write the commit messages, and I do this in kate. kate's standard encoding is utf8. ( My computer's language is german, so everything is utf8) This (or just packing it, because the encoding on my machine is not unicode) did change something to the encoding. What happens if you untar the package use the iconv from up there, and repack it on your machine?
I don't think so but it can be tested I guess. I think Karl is on something, I am having a look at scipy's spkginstall to see if there is something that should be done there.
I have an endianess problem with numpy1.4.1 on linux ppc, funny it works on ppc OSX: http://projects.scipy.org/numpy/ticket/1403 and http://bugs.gentoo.org/show_bug.cgi?id=306237 on the other hand numpy1.5.0 built on my test machine. I am trying to find the changeset that made it possible to see if it is worth backporting.
comment:80 in reply to: ↑ 74 Changed 11 years ago by
Replying to kcrisman:
Same error.
It may sound unrelated but what fortran compiler are you using in your sage install? g95 or gfortran?
In any case I think some of the patches in scipy would need rebasing. But the fact of the matter is that they are g95 dependent and we dropped g95  I already did so in numpy so we should do it as well in scipy.
comment:81 Changed 11 years ago by
I don't know exactly, but recall that Sage still includes Fortran compilers for Mac (see here). This is an spkg (sagefortran...). From the spkg readme:
= gFortran = == Description == G95 is a stable, production Fortran 95 compiler available for multiple CPU architectures and operating systems. == Upstream contacts == URL: http://ftp.g95.org http://www.g95.org
So I don't know how to answer your question  apparently it's both gfortran and g95 :) The spkginstall has more details on this, but it apparently depends on the version (there are different .bz2 files for different Macs); my old PPC Mac would have used g95, I think. Is there some technical reason we should drop this?
Responding to the other comment, sadly, although I thought Scipy installed correctly at first, my machine is REALLY slow, so eventually it turned out that it didn't. I succeeded on a newer Intel Mac, but that is unrelated.
comment:82 followup: ↓ 83 Changed 11 years ago by
 Status changed from needs_review to needs_work
See comment 10 by Dave. g95 can be removed. Unless they forgot about OSX ppc. For my question, what does
$SAGE_LOCAL/bin/sage_fortran version
says? I am guessing we have gfortran binaries for all platforms. Dropping g95 means that we can remove some (ugly) patches for numpy/scipy there are probably other valid reasons to do so but generally speaking that's less work.
I removed all patches that were applied for g95 in numpy, so if you have g95 the problems may come from there.
I share your pain about building on ppc (although I dropped OS X sometimes ago, this is now linux only and gentoo so everything or almost install from sources. openoffice3.1 clocks in at 27 hours to build). Did you mean that numpy failed or restating that scipy failed in your latest comment?
If numpy fails for you as well on OSX ppc we may have a strong motivation to move to 1.5.0 again.
I am changing this to need_works until we work out what's wrong in your case.
comment:83 in reply to: ↑ 82 Changed 11 years ago by
Replying to fbissey:
If numpy fails for you as well on OSX ppc we may have a strong motivation to move to 1.5.0 again.
If you want I can prepare a package for 1.5.0 with the patch from the numpy trac, and run the doctests, in the evening.
comment:84 Changed 11 years ago by
Numpy was fine in all cases; Scipy was what didn't build. drkirkby's comment "There are g95 binaries in the Fortran package in Sage, but William said they can be removed." may not be true.
The command that supposedly checks which version I have returns with an error:
local/bin/sage_fortran: line 3: sage_fortran.bin: command not found
This also happens on my (successful) build of 4.6.prealpha4, so maybe the command is wrong?
Incidentally, does this ticket do anything with respect to #7831 and #8010? Just curious.
comment:85 Changed 11 years ago by
Okay, I finally figured out how to check this  I had to run the binary directly, the scripts didn't work.
G95 (GCC 4.0.3 (g95 0.91!) Jun 4 2007)
etcetera. On my Macintel, I get
GNU Fortran (GCC) 4.2.3 Copyright (C) 2007 Free Software Foundation, Inc.
So yes, it is using G95 on Tiger.
comment:86 Changed 11 years ago by
It would be great if this is the problem and it can somehow be solved!
Another thing: I builded numpy1.5.0 with the new patch now. There only failed some doctests because numpy is throwing some new warnings. Want to give it a try? Perhaps the problem is then obsolete with scipy 0.8. also? I personally would prefer to stick to 1.4.1, but well it is like it is...
comment:87 followup: ↓ 88 Changed 11 years ago by
If you post it, I can try it  though it might be after the weekend, depending on my physical access to that particular box.
fbissey  can you explain (in layman's terms) how having the wrong fortran compiler might affect things? It does confuse me :)
comment:88 in reply to: ↑ 87 ; followup: ↓ 91 Changed 11 years ago by
Replying to kcrisman:
If you post it, I can try it  though it might be after the weekend, depending on my physical access to that particular box.
fbissey  can you explain (in layman's terms) how having the wrong fortran compiler might affect things? It does confuse me :)
Up and ready!
Yes this would be interesting! I don't really get it either because it looks more like a C problem to me?
comment:89 Changed 11 years ago by
comment:90 Changed 11 years ago by
It's more of a hunch than anything special I'll admit to that. The numpy/scipy connection seems to be fragile so I am not discounting it. In actual fact the fortran compiler is used only to compile one file in the whole of numpy (lapack_lite.so). I am more worried that it mixes up the toolchain it prepares for scipy.
One thing I'd like to see however is a complete build log for scipy not just the failing bit. There may be clues earlier in the process.
comment:91 in reply to: ↑ 88 ; followup: ↓ 92 Changed 11 years ago by
Replying to maldun:
Replying to kcrisman:
If you post it, I can try it  though it might be after the weekend, depending on my physical access to that particular box.
fbissey  can you explain (in layman's terms) how having the wrong fortran compiler might affect things? It does confuse me :)
Up and ready!
Sorry, fails with exactly the same error (on a different box with similar specs, same G95 in its Sage). So for now probably stick with the bettertested 1.4.1 for numpy. Hopefully we can figure out what's going on here.
The logs for the numpy (1.5.0) and scipy are in this directory, from the second computer. Happy hunting! By the way, Python.h is clearly working fine elsewhere in these logs, and sage_fortran compiles all kinds of neat stuff up to that point.
comment:92 in reply to: ↑ 91 ; followup: ↓ 93 Changed 11 years ago by
Replying to kcrisman:
Replying to maldun:
Replying to kcrisman:
If you post it, I can try it  though it might be after the weekend, depending on my physical access to that particular box.
fbissey  can you explain (in layman's terms) how having the wrong fortran compiler might affect things? It does confuse me :)
Up and ready!
Sorry, fails with exactly the same error (on a different box with similar specs, same G95 in its Sage). So for now probably stick with the bettertested 1.4.1 for numpy. Hopefully we can figure out what's going on here.
The logs for the numpy (1.5.0) and scipy are in this directory, from the second computer. Happy hunting! By the way, Python.h is clearly working fine elsewhere in these logs, and sage_fortran compiles all kinds of neat stuff up to that point.
Thanks for the logs. Sorry to be a bother but would you have old build logs for scipy0.7 as well? Since I don't have a ppc OSX setup that would be very useful to have a successful log even from an older version of scipy.
comment:93 in reply to: ↑ 92 ; followup: ↓ 94 Changed 11 years ago by
Sorry, fails with exactly the same error (on a different box with similar specs, same G95 in its Sage). So for now probably stick with the bettertested 1.4.1 for numpy. Hopefully we can figure out what's going on here.
The logs for the numpy (1.5.0) and scipy are in this directory, from the second computer. Happy hunting! By the way, Python.h is clearly working fine elsewhere in these logs, and sage_fortran compiles all kinds of neat stuff up to that point.
Thanks for the logs. Sorry to be a bother but would you have old build logs for scipy0.7 as well? Since I don't have a ppc OSX setup that would be very useful to have a successful log even from an older version of scipy.
Look in the same place, just posted it. So nice to be operating at 1.25 GHz instead of 700 MHz...
Also, interestingly, I now have a "mixed" installation on this computer:
sage: import numpy sage: numpy.version.version '1.5.0' sage: import scipy sage: scipy.version.version '0.7.0'
At least some of the relevant tests seem to pass with this, though of course I get the RunTimeErrors? mentioned above and did Inf
and inf
change places? I know little about numpy and scipy, though. Anyway, that's better than on the other box, whose issues with bad tarballs seem to not exist on this box.
comment:94 in reply to: ↑ 93 ; followup: ↓ 95 Changed 11 years ago by
Replying to kcrisman:
Look in the same place, just posted it. So nice to be operating at 1.25 GHz instead of 700 MHz...
Also, interestingly, I now have a "mixed" installation on this computer:
sage: import numpy sage: numpy.version.version '1.5.0' sage: import scipy sage: scipy.version.version '0.7.0'At least some of the relevant tests seem to pass with this, though of course I get the RunTimeErrors? mentioned above and did
Inf
andinf
change places? I know little about numpy and scipy, though. Anyway, that's better than on the other box, whose issues with bad tarballs seem to not exist on this box.
Found the logs, I may have a theory and the logs may help to (in)validate it. Not sure about Inf/inf either. scipy should work fine with numpy1.5  once rebuilt, and sage needs rebuilding too. That may get rid of some of these runtime errors. scipy0.8 needs numpy1.4 minimum on the other hand.
comment:95 in reply to: ↑ 94 Changed 11 years ago by
Also, interestingly, I now have a "mixed" installation on this computer:
sage: import numpy sage: numpy.version.version '1.5.0' sage: import scipy sage: scipy.version.version '0.7.0'At least some of the relevant tests seem to pass with this, though of course I get the RunTimeErrors? mentioned above and did
Inf
andinf
change places? I know little about numpy and scipy, though. Anyway, that's better than on the other box, whose issues with bad tarballs seem to not exist on this box.Found the logs, I may have a theory and the logs may help to (in)validate it. Not sure about Inf/inf either. scipy should work fine with numpy1.5  once rebuilt, and sage needs rebuilding too. That may get rid of some of these runtime errors. scipy0.8 needs numpy1.4 minimum on the other hand.
Yup, tests in relevant modules are passing, with only the RuntimeWarning
s (not errors, sorry for that) in general marring things (as noted, one or two other very minor things). Anyway, none of this not compiling nonsense, so something must have changed in Scipy 0.8 from 0.7  maybe something that used that change mentioned above about reusing npymath
. There were indeed a couple changes to the gammaincinv
function in Scipy, if you look at their changesets, but as a nonexpert how those could have resulted in this Python.h/unicode error is totally beyond me.
comment:96 Changed 11 years ago by
Interestingly enough on Gentoo we have I/usr/include/python2.6 added to the compilation flags everywhere. I wonder if just finding a way of adding I${SAGE_LOCAL}/include/python2.6 in sage would help at all. The mystery then would be "how did it work before?" Furthermore why is it not needed for the earlier C compilations.... or on other platforms? May be we are using system python headers without knowing it?
I will work on that over the week end. Hopefully by Monday I will have something you can test.
comment:97 Changed 11 years ago by
It turns out to be more subtle than I thought and I don't know yet why.
It is not easy to add I${SAGE_LOCAL}/include/python2.6 to compile options, in any case many other compilation lines actually have it already. After careful checking, I found that this particular option is not used in any scipy0.7.x log I had on sage or Gentoo for these particular objects. But I see it in scipy0.8.0 in Gentoo.
I am messing up with some of my system at the moment so I cannot generate a successful build log of scipy0.8.0 from sage at the moment. But it would be interesting to know if this particular compilation option has been added in the successful builds.
comment:98 Changed 11 years ago by
OK  so there is a fair difference on that file between 0.7.2 and 0.8.0.
headers in gammaincinv.c in 0.7.2
#include <stdio.h> #include <math.h> #include "../cephes.h" #undef fabs #include "misc.h"
in 0.8.0
#include <Python.h> #include <numpy/npy_math.h> #include <stdio.h> #include <math.h> #include "../cephes.h" #undef fabs #include "misc.h"
So it will be very interesting to get a successful build log of scipy0.8.0 in sage for inspection. A quick and dirty fix would be to change <Python.h> to <python2.6/Python.h> and I am not completely sure it would work either.
There are a few more Python.h header in that folder so just fixing that one may not work:
grep r "Python\.h" * amos_wrappers.h:#include "Python.h" cdf_wrappers.h:#include "Python.h" cephes/sici.c:#include <Python.h> cephes/mconf.h:#include <Python.h> _cephesmodule.c:#include "Python.h" c_misc/gammaincinv.c:#include <Python.h> lambertw.c:#include "Python.h" orthogonal_eval.c:#include "Python.h" specfun_wrappers.h:#include "Python.h" toms_wrappers.h:#include "Python.h" ufunc_extras.h:#include "Python.h"
considering the SConscript the files in the cephes subfolder will probably be trouble too.
comment:99 Changed 11 years ago by
But it would nice to know anyway. If it doesn't work with gammaincinv.c, at least the error message would change. Providing a patch would then be easy for me.
Ironically I already had asked if this had could been the problem... The only thing what me disturbs is the this entry in the error log of kcrisman:
...y/npy_common.h:79:2: error: #error Must use Python with unicode enabled.
This sounds more like that unicode isn't enabled, and perhaps this is related to native python installation on the system, and the gcc uses this header instead that from sage. I think to be absolutely sure, we should try also to set the path really to Python.h in the sage install in the header, if setting to <python2.6/Python.h> doesn't work.
@fbissey Another question: would it help you, if you see my install log from ubuntu?
comment:100 Changed 11 years ago by
No need for extra logs. I found the culprit and why it affects only Karl's ppc machine. I point the finger to g95! or rather the fact that the patches that are installed if g95 is found should have been rebased.
Basically the file special/setup.py is replaced by a version from scipy0.7.0 which doesn't need the python header. Could you please use spkginstall.2 attached in scipy0.8.0 it doesn't use the patches for g95. It should work without special patch in my opinion anyway.
comment:101 Changed 11 years ago by
So I've packed the scipy with your spkginstall, but this time it doesn't work for me. I get this error:
Host system uname a: Linux math181 2.6.3122generic #63Ubuntu SMP Thu Aug 19 00:23:50 UTC 2010 x86_64 GNU/Linux **************************************************** **************************************************** CC Version gcc v Using builtin specs. Target: x86_64linuxgnu Configured with: ../src/configure v withpkgversion='Ubuntu 4.4.14ubuntu9' withbugurl=file:///usr/share/doc/gcc4.4/README.Bugs enablelanguages=c,c++,fortran,objc,objc++ prefix=/usr enableshared enablemultiarch enablelinkerbuildid withsystemzlib libexecdir=/usr/lib withoutincludedgettext enablethreads=posix withgxxincludedir=/usr/include/c++/4.4 programsuffix=4.4 enablenls enableclocale=gnu enablelibstdcxxdebug enableobjcgc disablewerror witharch32=i486 withtune=generic enablechecking=release build=x86_64linuxgnu host=x86_64linuxgnu target=x86_64linuxgnu Thread model: posix gcc version 4.4.1 (Ubuntu 4.4.14ubuntu9) **************************************************** invalid command name 'config_fc noopt noarch' Error building scipy. real 0m2.296s
comment:102 Changed 11 years ago by
I think I have gone overboard with double quotes, attaching a new version shortly.
comment:103 Changed 11 years ago by
Ok. I deleted all outdated patches like you suggested (initially I let them in, because they didn't any harm, and since I didn't know much about every detail of the patches, I first tried out if it works, but of course it's better to throw them out) and it builds without problem. But with the old spkg. the spkginstall.2 you attached still, with the same error, because my machine doesn't know the command config_fc noopt noarch:
invalid command name 'config_fc noopt noarch'
could it be that I have to install something on my system? Or should we solve this with an If clause?
comment:104 followup: ↓ 106 Changed 11 years ago by
ok I solved it: You forgot the "" before the config =)
comment:105 Changed 11 years ago by
I think I found the problem, I think the shell was not set up properly.
I mean, these are options passed to setup.py not separate commands. They shouldn't be interpreted as such and aren't in the numpy spkginstall, so there are no reason why they should be in scipy. But the shells for the two spkginstall were different so I think that may be a point of bash/shell semantic. In which case I am curious to find out what I am supposed to do in sh. There is a new spkginstall.2 for you to try.
comment:106 in reply to: ↑ 104 Changed 11 years ago by
Replying to maldun:
ok I solved it: You forgot the "" before the config =)
Ok you posted while I was answering. That may pass but I am not sure that does what it's supposed to do. There shouldn't be a "" in front. I suspect it is now just ignored.
comment:107 Changed 11 years ago by
ok I was to quick... indeed it builded, but not for too long: it took it a c"onfig_fc"... so it didn't ignore, but missinterpreted it.
But the error remains. I don't really get it is there something different with ubuntu's gcc?
comment:108 Changed 11 years ago by
Ah okay: I looked into the scipy docu: This option seems to be for numpy, not for scipy. http://projects.scipy.org/numpy/wiki/DistutilsDoc#specifingconfigfcoptionsforlibrariesinsetuppyscript
Because numpy builded with that command. Does it build on your system?
comment:109 Changed 11 years ago by
OK. I thought it was used in scipy as well. There seems to be 2 files with it, and it shouldn't have an impact. You can remove it then. Strange it doesn't generate errors in Gentoo.
comment:110 Changed 11 years ago by
Ok updatet scipy package, with deleted patches
comment:111 Changed 11 years ago by
Help! I couldn't work on this over the weekend, and now I'm not quite sure what I should do to test :) Should I just try installing the new Scipy 0.8, or what? I am very glad that it seems you've figured out what G95 didn't like, though. It would also be nice to see a diff between the spkginstall
scripts.
comment:112 Changed 11 years ago by
Ok short summary: 1) Try the new scipy 2) If this doesn't work try to change the header in gammaincinv.c in the scipy sources from <Python.h> to <python2.6/Python.h> 3) If this results in no change, rewrite the header that it takes the sage header directly means change <Python.h> to "whateverpathitmaybe/python2.6/Python.h" and look if the error message changes. (because there are other files affected, but this should be no prob to patch)
If it's really only the location of the header patching wouldn't be a problem. And Thanx for all the trouble!
I have the diffs made for on my other machine, I can post them tomorrow!
Changed 11 years ago by
Diff of the spkg installs; The lines wich concern config fc are removed due problems in ubuntu
comment:113 followup: ↓ 118 Changed 11 years ago by
Well, that seems to have helped...
gcc: scipy/special/c_misc/gammaincinv.c
and it keeps going!
So it was the patches for G95 that were the problem. Which one in particular (which command/line) caused the header confusion, do you think? I'm just curious, and don't know enough about this to say.
By the way, I'm seeing things like
compile options: 'I/Users/student/Desktop/sage4.5.2/local/lib/python2.6/sitepackages/numpy/core/include I/Users/student/Desktop/sage4.5.2/local/include/python2.6 c'
so hopefully that is also a positive sign.
comment:114 Changed 11 years ago by
Success! Now I'll set SAGE_CHECK=yes
, retry numpy, do ./sage ba
, retry scipy, and test the Sage library. It will take a LONG time but hopefully tomorrow I'll have a report for you.
comment:115 Changed 11 years ago by
That are great news! Hopefully everything works!
comment:116 Changed 11 years ago by
Ah and don't forget the patch for the doctests in the attachement. Else there would 2 doctests fail due to output changes in numpy.
comment:117 followups: ↓ 119 ↓ 120 Changed 11 years ago by
I'm not worried about that, but I'll do it when I get there. Numpy just finished installing successfully.
Question:
# Program around a bug in SciPY's distutils. unset CFLAGS python setup.py install ${NUMPY_FCONFIG}
I assume this is still needed in the numpy spkginstall
. Just checking.
Question: I know drkirkby is cc:ed on this ticket. Numpy seems to have a very easy to run test suite  see here. However, it requires the Nose package. Maybe this should be a separate ticket, but it would seem reasonable to include Nose, which is about 250 KB, so that we can run the numpy tests. I am curious as to how we run the scipy tests without it!
comment:118 in reply to: ↑ 113 Changed 11 years ago by
Replying to kcrisman:
Well, that seems to have helped...
gcc: scipy/special/c_misc/gammaincinv.cand it keeps going!
So it was the patches for G95 that were the problem. Which one in particular (which command/line) caused the header confusion, do you think? I'm just curious, and don't know enough about this to say.
Well, there are 4 patches that were applied in case g95 was found as the fortran compiler. The faulty one is special.setup.py (or possibly setup.py.special) which replaces the file setup.py in the folder scipy/special.
This setup.py file has changed quite a lot between version 0.7.0 and 0.8.0. The lines at fault:
# C libraries config.add_library('sc_c_misc',sources=[join('c_misc','*.c')]) config.add_library('sc_cephes',sources=[join('cephes','*.c')], include_dirs=[get_python_inc(), get_numpy_include_dirs()], macros=define_macros)
in version 0.7.0 and this is the code found in the "patch" file and
# C libraries config.add_library('sc_c_misc',sources=[join('c_misc','*.c')], include_dirs=[get_python_inc(), get_numpy_include_dirs()], macros=define_macros) config.add_library('sc_cephes',sources=[join('cephes','*.c')], include_dirs=[get_python_inc(), get_numpy_include_dirs()], macros=define_macros)
in version 0.8.0. This is a case were using patches rather copying files wholesale would have prevented the problem. Either the patch wouldn't have worked with the newer version and we would have known straight away or it would have applied with some fuzz without dommaging this particular section.
comment:119 in reply to: ↑ 117 Changed 11 years ago by
Replying to kcrisman:
I'm not worried about that, but I'll do it when I get there. Numpy just finished installing successfully.
Question:
# Program around a bug in SciPY's distutils. unset CFLAGS python setup.py install ${NUMPY_FCONFIG}I assume this is still needed in the numpy
spkginstall
. Just checking.Question: I know drkirkby is cc:ed on this ticket. Numpy seems to have a very easy to run test suite  see here. However, it requires the Nose package. Maybe this should be a separate ticket, but it would seem reasonable to include Nose, which is about 250 KB, so that we can run the numpy tests. I am curious as to how we run the scipy tests without it!
Actually Dave has started a thread about this a few weeks ago on the mailing list, you are welcome to add your opinion and revive the thread.
comment:120 in reply to: ↑ 117 Changed 11 years ago by
Replying to kcrisman:
I'm not worried about that, but I'll do it when I get there. Numpy just finished installing successfully.
Question:
# Program around a bug in SciPY's distutils. unset CFLAGS python setup.py install ${NUMPY_FCONFIG}I assume this is still needed in the numpy
spkginstall
. Just checking.
I thought I had asked maldun to remove this. It's working fine on Gentoo as it is, I am assuming this has been fixed upstream (possibly a while ago).
comment:121 followup: ↓ 122 Changed 11 years ago by
I added the
# Program around a bug in SciPY's distutils. unset CFLAGS
into the numpy spkginstall from the old one, just to be sure that we don't run into troubles, and actually it doesn't do any harm... but if you want I can remove it as well.
But didn't we agree that the
setup.py install ${NUMPY_FCONFIG
}
is necessary in numpy? I had to delete it from scipy or else it doesn't build anymore, but in numpy it's correct.
comment:122 in reply to: ↑ 121 Changed 11 years ago by
Replying to maldun:
I added the
# Program around a bug in SciPY's distutils. unset CFLAGSinto the numpy spkginstall from the old one, just to be sure that we don't run into troubles, and actually it doesn't do any harm... but if you want I can remove it as well.
We should remove it. If there is a problem  which I doubt, we can put it back.
But didn't we agree that the
setup.py install ${NUMPY_FCONFIG
} is necessary in numpy? I had to delete it from scipy or else it doesn't build anymore, but in numpy it's correct.
Yes! Leave it here.
comment:123 Changed 11 years ago by
Ok I removed the line.
I also thougth before posting I should check if there are some old rests in there, and I removed some patches that are possible(?) outdated. I think this could affect the cygwin installation, so it would be important to see if it still builds on all machines, or else I have to carefully merge the changed lines, from the old patches with the new versions of the files.
It works for me, so I hope I didn't cause some new problems
comment:124 Changed 11 years ago by
I have a different problem: When I upload the new Version, and try to download it I get the old one. Is there some confusion on the server?
comment:125 Changed 11 years ago by
Ok worked. Don't ask why...
comment:126 followups: ↓ 127 ↓ 129 Changed 11 years ago by
 Status changed from needs_work to needs_review
Ok  now this needs review!
I looked at the cygwinlapack_litesetup.py patch. The file patched hasn't changed between numpy1.3.x and 1.4.1, so it can still be applied safely if needed.
In summary the only problems overall are:
 numpy 1.4.1 is broken on linux ppc (on the official list of supported OS  so bad) and alpha (not on the supported list  so OK).
 numpy1.5.0 has an issues with cython that will be solved in 1.5.1  shame because it also fixes linux ppc and alpha.
Anything to add?
comment:127 in reply to: ↑ 126 Changed 11 years ago by
Replying to fbissey:
Anything to add?
as mentioned above: The version of numpy1.5.0 with the patch applied is ok. The only trouble I have are some doctests, which fails because numpy throws a "Division by zero encountered!" Exception, which makes them fail, although the results are correct. But I still don't know how to catch the exceptions.
comment:128 Changed 11 years ago by
I should say that while breaking on linux ppc is not good, I only know of one other person building sage on that platform. So it could take the backseat while problems with the 1.5 series are ironed out.
comment:129 in reply to: ↑ 126 ; followup: ↓ 133 Changed 11 years ago by
 numpy1.5.0 has an issues with cython that will be solved in 1.5.1  shame because it also fixes linux ppc and alpha.
Hmm, dropping an official thing is not so good. I think that one would want to post to the list about this  more likely wait for 1.5.1? Or is the fix for this easily backported?
Also, although I've been going for over 24 hours now on the tests (and several timeouts! so I'll have to run those with SAGE_TIMEOUT_LONG>3600
), I have only seen two errors, both like this:
sage t long "devel/sage/sage/graphs/digraph.py" ********************************************************************** File "/Users/student/Desktop/sage4.5.2/devel/sage/sage/graphs/digraph.py", line 204: sage: DiGraph(A) Exception raised: Traceback (most recent call last): File "/Users/student/Desktop/sage4.5.2/local/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/Users/student/Desktop/sage4.5.2/local/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/Users/student/Desktop/sage4.5.2/local/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_1[17]>", line 1, in <module> DiGraph(A)###line 204: sage: DiGraph(A) File "/Users/student/Desktop/sage4.5.2/local/lib/python/sitepackages/sage/graphs/digraph.py", line 364, in __init__ data = networkx.MultiDiGraph(data) File "/Users/student/Desktop/sage4.5.2/local/lib/python2.6/networkx/classes/digraph.py", line 217, in __init__ convert.from_whatever(data,create_using=self) File "/Users/student/Desktop/sage4.5.2/local/lib/python2.6/networkx/convert.py", line 157, in from_whatever if isinstance(thing,numpy.core.defmatrix.matrix) or \ AttributeError: 'module' object has no attribute 'defmatrix' **********************************************************************
Have you seen this? Clearly it's something to do with numpy, from the end  but it's not clear to me why it would be asking whether a module has defmatrix attribute.
comment:130 followup: ↓ 135 Changed 11 years ago by
I know this error. Since you use sage 4.5.2 you will also use networkx1.0.1. See comments 24 and following or ticket #9854.
You have to either use Sage 4.5.3 which holds networkx1.2, upgrade networkx1.0.1 in your current version, or apply the networkx1.0.1.p0.spkg in the attachement, which holds a simple patch.
The reason is that numpy.core.defmatrix moved to numpy.matrixlib.defmatrix.
comment:131 Changed 11 years ago by
@linux ppc I could work over the weekend the solve the failed doctests in numpy1.5.0, because with the patch from the numpy developers the biggest problem i.e. the cython problem is solved. The only thing remains are some "Division by Zero Warnings" as mentioned above.
comment:132 Changed 11 years ago by
And another thing: If we aren't able to merge numpy and scipy to 4.6. in time, why don't we provide the packages as optional or experimental? Because they work on most systems, and it would be a waste to just throw them away.
comment:133 in reply to: ↑ 129 ; followup: ↓ 134 Changed 11 years ago by
Replying to kcrisman:
 numpy1.5.0 has an issues with cython that will be solved in 1.5.1  shame because it also fixes linux ppc and alpha.
Hmm, dropping an official thing is not so good. I think that one would want to post to the list about this  more likely wait for 1.5.1? Or is the fix for this easily backported?
Depending on what list you read, Linux on PPC may or may not be supported. I've tried to get an agreement before on what we do support and what we don't, but never got anywhere.
I'm personally in favor of keeping Sage building on less regular systems, as it often indicates problems. The Solaris port has found endless bugs in Sage  some of which have even according to William been serious.
I have an old IBM RS/6000 7025 F50 (Power PC) which I was setting up AIX on the other day. I contemlated whether I should put Linux on it too for testing Sage. But when I read Sage's README.txt, PowerPC was only supported on OS X, not Linux. So I did not bother. But the machine has space for 18 disk drives, so I could always add it!!
Dave
comment:134 in reply to: ↑ 133 Changed 11 years ago by
I forgot about the networkx change, thanks for the reminder.
 numpy1.5.0 has an issues with cython that will be solved in 1.5.1  shame because it also fixes linux ppc and alpha.
Hmm, dropping an official thing is not so good. I think that one would want to post to the list about this  more likely wait for 1.5.1? Or is the fix for this easily backported?
I'm personally in favor of keeping Sage building on less regular systems, as it often indicates problems. The Solaris port has found endless bugs in Sage  some of which have even according to William been serious.
I have an old IBM RS/6000 7025 F50 (Power PC) which I was setting up AIX on the other day. I contemlated whether I should put Linux on it too for testing Sage. But when I read Sage's README.txt, PowerPC was only supported on OS X, not Linux. So I did not bother. But the machine has space for 18 disk drives, so I could always add it!!
Why not? I've been trying to figure out how to get a thumb drive to boot my PPC machine on Ubuntu (and possibly other distros) but it turns out it's devilishly hard to get that to work, and I haven't had time to do it. I can't find an easy virtualization option either (neither xen nor anything else seem to really be particularly easy, if they even still work with such an old machine...)
And another thing: If we aren't able to merge numpy and scipy to 4.6. in time, why don't we provide the packages as optional or experimental? Because they work on most systems, and it would be a waste to just throw them away.
Certainly one could do this, but even that would require a positive review from someone. You might have to ask on the list about this  I've never heard of having an upgrade to a normal package as only optional.
comment:135 in reply to: ↑ 130 Changed 11 years ago by
You have to either use Sage 4.5.3 which holds networkx1.2, upgrade networkx1.0.1 in your current version, or apply the networkx1.0.1.p0.spkg in the attachement, which holds a simple patch.
Incidentally, I also had to repackage that spkg, because you didn't use ./sage pkg
for it either :) and it had unchecked in changes :) But clearly that is irrelevant for testing, since the upgrade is already in a later Sage.
Okay, with this upgrade those two tests pass. I am currently testing the other four failures, all of which were timeouts, all of which are in files I personally know to have VERY long long
doctests, and all but one of which hopefully will pass with SAGE_TIMEOUT_LONG=10000
(there is one in interfaces/maxima.py
which always fails on OS X 10.4, as far as I know, which is due to tabcompletion testing of Maxima never finishing, totally unrelated to this). I'll let you know if something *doesn't* work.
So I think that my job is done here (pointing out that I was testing the version from before the weekend. fbissey or drkirkby should comment on the spkg itself.
comment:136 followup: ↓ 137 Changed 11 years ago by
Well thanks Karl! Your testing was very useful. Since I have contributed to bits and pieces I think I should had myself as an author. But most of the heavy lifting as been done by maldun.
So what's left is the question of whether we go for numpy1.4.1 and leave minor archs unsupported for a little bit. Or we wait for 1.5.1, which works on the minor archs in question and will play well with cython.
As an aside I have already pushed the upgrade in sageongentoo (to avoid tree rote, we already have to keep two old packages that are otherwise removed from Gentoo) and we want to avoid that kind of stuff as much as possible. So the current code is out there and used by a few people.
After that there are question of details. In spkginstall I have set FC to ${SAGE_LOCAL}/bin/sage_fortran, with the idea that it was basically calling "gfortran fpic" or the g95 equivalent. We have been talking about that very subject on sagedevel recently. Is it the best way to go? If one uses sunstudio (and I am planning to give a go) the correct flag would be Kpic but would it be set up properly in sage_fortran? I doubt it.
So what would be the best course of action? Using the variable SAGE_FORTRAN for now and ask it to be set with the proper pic flag and hopefully drop it later on when FC is the mainstay?
comment:137 in reply to: ↑ 136 Changed 11 years ago by
Replying to fbissey:
Well thanks Karl! Your testing was very useful. Since I have contributed to bits and pieces I think I should had myself as an author. But most of the heavy lifting as been done by maldun.
I think you are right. So I allowed myself to set you as coauthor, since this whole thing wouldn't work without your help =)
So what's left is the question of whether we go for numpy1.4.1 and leave minor archs unsupported for a little bit. Or we wait for 1.5.1, which works on the minor archs in question and will play well with cython.
I would suggest the following: If numpy 1.5.1 doesn't come out before Sage 4.6. let's take 1.4.1 (since ppc linux is not on the current supported list; see readme.tex of sage) I will start this weekend with correcting things of 1.5.0, since the current errors are new warnings, and I'm quite sure numpy 1.5.1 will throw those as well. When numpy 1.5.1 is out before 4.6. I will try to pack it, and then hopefully it is done in no time, since it shared the same probs.
As an aside I have already pushed the upgrade in sageongentoo (to avoid tree rote, we already have to keep two old packages that are otherwise removed from Gentoo) and we want to avoid that kind of stuff as much as possible. So the current code is out there and used by a few people.
After that there are question of details. In spkginstall I have set FC to ${SAGE_LOCAL}/bin/sage_fortran, with the idea that it was basically calling "gfortran fpic" or the g95 equivalent. We have been talking about that very subject on sagedevel recently. Is it the best way to go? If one uses sunstudio (and I am planning to give a go) the correct flag would be Kpic but would it be set up properly in sage_fortran? I doubt it.
So what would be the best course of action? Using the variable SAGE_FORTRAN for now and ask it to be set with the proper pic flag and hopefully drop it later on when FC is the mainstay?
The question is how fast this change with the fortran compiler is done. I think we should wait a little bit now.
My question is: should this given a positive review, or is something still missing?
comment:138 followup: ↓ 139 Changed 11 years ago by
I'd like to check this out on sage4.6.alpha1, but its unclear to me exactly what to do. Can someone summarize the current procedure for incorporating this ticket?
comment:139 in reply to: ↑ 138 Changed 11 years ago by
Replying to mhampton:
I'd like to check this out on sage4.6.alpha1, but its unclear to me exactly what to do. Can someone summarize the current procedure for incorporating this ticket?
Install notes and links to the packages are in the discription of this ticket!
Quick and dirty: Install the packagages, do sage ba to compile the whole source, apply the patch in the attachment for the doctests.
greez maldun
comment:140 Changed 11 years ago by
Just a quick note that I used this package in sage4.6alpha1 and it didn't break any tests on linuxx86, I haven't heard of any problems on linuxamd64 from a friend who was also testing this. So it applies cleanly on 4.6alpha1.
comment:141 followup: ↓ 142 Changed 11 years ago by
Dave pointed me to this ticket the day before yesterday (and I read it from the beginning). ;)
Despite that, it's unclear to me which diffs are currrent, and the description should IMHO be updated wrt. what's to be done to review this.
./sage ba
is also in my opinion not a solution to get dependent extension modules updated, i.e. if ./sage b
isn't sufficient, dependencies should be added to module_list.py
.
comment:142 in reply to: ↑ 141 ; followup: ↓ 162 Changed 11 years ago by
 Description modified (diff)
Replying to leif:
Dave pointed me to this ticket the day before yesterday (and I read it from the beginning). ;)
Despite that, it's unclear to me which diffs are currrent, and the description should IMHO be updated wrt. what's to be done to review this.
./sage ba
is also in my opinion not a solution to get dependent extension modules updated, i.e. if./sage b
isn't sufficient, dependencies should be added tomodule_list.py
.
I cleaned up the discription a little bit, and deleted the obsolete remark about networkx1.2
I don't know if this is a good Idea because this is a whole bunch of files we are talking about, which have to be added. Perhaps you won't save much time either when compiling. I first tried this too, but after about 2 hours searching, I got somehow tired of this... And after merging numpy > 1.3.x into Sage this has not to be done anymore. All versions of 1.5.x builded without problems, and didn't need ba anymore. You only have to do this when big changes are happening, and this is rather occassionally.
package with a linux ppc patch is out: http://code.google.com/p/spkgupload/downloads/detail?name=numpy1.4.1.p0.spkg hope it works this time!
comment:143 followup: ↓ 145 Changed 11 years ago by
Ah before I forget it, and you think I'm just lazy: I just looked up again in an non compiled version which files are concerned, and I remembered again what was really the problem Most of them are already in the module_list.py.
However the problem is, that there are actually no changes to that files, so sage b won't do nothing, because it does'nt recognice any changes!
only ba force sage to do that and link the whole c files to the new version of numpy.
You would have to change the behavior of b, not add something to the module_list.py
comment:144 Changed 11 years ago by
Just for the record, gcc
has the following option:
mabi=ieeelongdouble
Change the current ABI to use IEEE extended precision long double. This is a PowerPC 32bit Linux ABI option.
I don't know if this requires a different (or differently built) libc
however.
comment:145 in reply to: ↑ 143 ; followup: ↓ 146 Changed 11 years ago by
Replying to maldun:
Ah before I forget it, and you think I'm just lazy: I just looked up again in an non compiled version which files are concerned, and I remembered again what was really the problem Most of them are already in the module_list.py.
However the problem is, that there are actually no changes to that files, so sage b won't do nothing, because it does'nt recognice any changes!
So we have to add (at least some of) those files to depends
that actually have changed.
only ba force sage to do that and link the whole c files to the new version of numpy.
You would have to change the behavior of b, not add something to the module_list.py
That's IMHO not an option, since missing dependencies will break any Sage upgrade anyway.
P.S.: The attached diffs are still confusing. (One should at least change/update their descriptions.) For review, complete diffs between the old and new spkgs (of course excluding src/
) would be helpful.
comment:146 in reply to: ↑ 145 ; followups: ↓ 147 ↓ 148 Changed 11 years ago by
Replying to leif:
So we have to add (at least some of) those files to
depends
that actually have changed.
And that's the problem: There are NO files which would have changed! Not within sage/devel/sagewhateverbranch. And that's also the reason for the warnings: numpy wonders why still the old size of flatiter is in the .c files. The only changes are in local/LIB/python/sitepackages/numpy, and b don't care about those. The only way to trick b was to put in a comment to the certain files, delete them again and save it, that sage b recognices the changes.
I when kcrisman gives me feedback if numpy works now on ppc, I will put a complete changelog here on trac. Perhaps over the weekend I find some time for that.
comment:147 in reply to: ↑ 146 ; followup: ↓ 149 Changed 11 years ago by
 Reviewers changed from KarlDieter Crisman to KarlDieter Crisman, David Kirkby, Leif Leonhardy, Francois Bissey
I when kcrisman gives me feedback if numpy works now on ppc, I will put a complete changelog here on trac. Perhaps over the weekend I find some time for that.
You mean on OS X, not Linux, right?
Numpy installed fine, and Scipy got past the bad spot fine, so I don't anticipate any problems. Then I'll run tests.
Leif, is this really a problem? (Asked as a query, not a complaint.) Presumably someone will only use these packages via sageupgrade, and then the whole library rebuilds (because sagex.y.z is new), right? Or is that not how sageupgrade works?
I'm adding a few reviewers based on how things have gone over these ~150 comments.
comment:148 in reply to: ↑ 146 Changed 11 years ago by
Replying to maldun:
Replying to leif:
So we have to add (at least some of) those files to
depends
that actually have changed.And that's the problem: There are NO files which would have changed! Not within sage/devel/sagewhateverbranch. And that's also the reason for the warnings: numpy wonders why still the old size of flatiter is in the .c files.
So what? Then just make them also depend on e.g. $SAGE_LOCAL/lib/python/sitepackages/numpy/core/include/numpy/numpyconfig.h
(if that has changed).
comment:149 in reply to: ↑ 147 Changed 11 years ago by
Replying to kcrisman:
Leif, is this really a problem? (Asked as a query, not a complaint.) Presumably someone will only use these packages via sageupgrade, and then the whole library rebuilds (because sagex.y.z is new), right? Or is that not how sageupgrade works?
The Sage library is "new" in every Sage version (even if there were no real changes to it), but does not get rebuilt from scratch (since there's no reason to do so if the dependencies are correct [or at least close to complete].)
comment:150 Changed 11 years ago by
P.S.: $SAGE_LOCAL/lib/python/sitepackages/Cython/Includes/numpy.pxd
could be an option, too.
comment:151 Changed 11 years ago by
I can do that, but this still won't solve the problem. Perhaps you misunderstand me, so I explain it again in more detail.
After installing numpy the only thing that has to be done is to recompile the .pyx files (which don't have changed), so that the old .c files (which haven't changed either) have be builded with the new version of numpy, that actually the change happens. I'm not a pro to sage, but as far I understood that is, that b only recompile changed files. No changes, no compilation. Even if I add now the changed files (the package) sage only would recompile those, and leaves out a whole lot of files which also have to be recompiled. The only way to get now sage to recompile those .pyx which is to change the files after installation for example by putting a comment into them, or force sage to recompile them, that is that what ba does. since there is no option that I know to force recompilation, I don't have an Idea how else I could get it working.
If you have a better way please let me know. I will apply a patch as soon as possible!
@kcrisman: didn't we talk about linux ppc? You said it builds on OS X ppc, or am I wrong?
comment:152 Changed 11 years ago by
No, I'm sorry if you misread my post about that. I would *like* to do so but there are still a number of technical hurdles to me running Linux PPC on my box (I think this should be clear in my sagedevel post). I am testing whether the change you made to support Linux PPC doesn't break anything. Sorry if this is not useful :( and I wish I did have access to Linux PPC more easily; unfortunately, this requires either having access to an already existing Linux PPC machine OR trying to run an image of Linux on a CD even with so little RAM there is no point OR using a portable hard drive, which is a more significant expense.
comment:153 Changed 11 years ago by
Apparently we have to add Carl Friedrich Gauss to the authors... ;)
comment:154 Changed 11 years ago by
As I feared: Making them dependent didn't force compilation either.
comment:155 Changed 11 years ago by
Update: If you are changing the numpy.pxd file (for example insert a blank line and delete it) sage b compile the dependent sources, but this file is in cython not in numpy.
manual changes on the numpy header files didn't do anything.
@Leif I know what you are trying to tell me. I would have expected that making a header dependent would get sage to resolve the dependencies and recompile the .pyx files also.
But even little changes to the header files do nothing even if they are dependent, or I forced dependency.
I didn't meant this as offense or something, I just told you what I had experienced in the early stages of this ticket. If you have a solution for example a diff with an entry that forces dependency, (plot3d for example) I could write the others in no time.
Perhaps I do something wrong. If you know what the trouble is please tell me.
comment:156 followup: ↓ 157 Changed 11 years ago by
There are only a few headers that actually have been updated, e.g. _numpyconfig.h
.
A trivial solution is to just touch
one header file after installation in spkginstall
, and make the relevant extension modules (also) depend on that one.
Touching $SAGE_LOCAL/lib/python/sitepackages/Cython/Includes/numpy.pxd
could also help.
comment:157 in reply to: ↑ 156 ; followup: ↓ 159 Changed 11 years ago by
Replying to leif:
Touching
$SAGE_LOCAL/lib/python/sitepackages/Cython/Includes/numpy.pxd
could also help.
Doing so triggers at least recompilation of the 13 extension modules that add the numpy include dir to their include_dirs
.
comment:158 Changed 11 years ago by
Lots happens when I am asleep and lecturing first thing in the morning :)
I cannot comment on the sagerelease thread from work. I am one of the few people running sage on linux ppc  more as hobby to check that it works than real production. The problem for numpy on ppc is fixed in numpy1.5.
cannot comment on the rest right now I have a truckload of exams to prepare.
comment:159 in reply to: ↑ 157 ; followup: ↓ 160 Changed 11 years ago by
Replying to leif:
Replying to leif:
Touching
$SAGE_LOCAL/lib/python/sitepackages/Cython/Includes/numpy.pxd
could also help.Doing so triggers at least recompilation of the 13 extension modules that add the numpy include dir to their
include_dirs
.
... and caused ptestlong
to pass without any errors. So touching just that file in numpy's spkginstall
should fix all dependency issues, without touching module_list.py
at all (which is advantageous, too).
I just wonder that (or if) that file is still compatible with the new numpy.
Regarding the discussion on sagerelease, who prepares a numpy 1.5.0 (or later) spkg? ;)
The 1.4.1 seems obsolete now...
comment:160 in reply to: ↑ 159 Changed 11 years ago by
Replying to leif:
So touching just that file in numpy's
spkginstall
should fix all dependency issues, without touchingmodule_list.py
at all (which is advantageous, too).
Unfortunately this will only work if Cython is not updated/rebuilt (after numpy has been installed), since none of these packages depends on the other, and Cython might still ship one with an old time stamp (which is preserved on installation).
comment:161 Changed 11 years ago by
Regarding the discussion on sagerelease, who prepares a numpy 1.5.0 (or later) spkg? ;)
The 1.4.1 seems obsolete now...
I do. There are only 2 doctests that fails, due to output changes inf > Inf I hope I can complete it over the weekend.
Actually there is another problem with the headers: You would have to make ALL haeaders dependent on the concerning entries in the module_list.py, because in every version of numpy there can be an other headers that changes... and of course that is a matter of taste, but wouldn't get this the module_list.py a little bit messy, with dependencies that actually aren't needed?
just my 2 cents
comment:162 in reply to: ↑ 142 Changed 11 years ago by
Replying to maldun:
Replying to leif:
Dave pointed me to this ticket the day before yesterday (and I read it from the beginning). ;)
Despite that, it's unclear to me which diffs are currrent, and the description should IMHO be updated wrt. what's to be done to review this.
./sage ba
is also in my opinion not a solution to get dependent extension modules updated, i.e. if./sage b
isn't sufficient, dependencies should be added tomodule_list.py
.I cleaned up the discription a little bit, and deleted the obsolete remark about networkx1.2
I don't know if this is a good Idea because this is a whole bunch of files we are talking about, which have to be added. Perhaps you won't save much time either when compiling. I first tried this too, but after about 2 hours searching, I got somehow tired of this... And after merging numpy > 1.3.x into Sage this has not to be done anymore. All versions of 1.5.x builded without problems, and didn't need ba anymore. You only have to do this when big changes are happening, and this is rather occassionally.
package with a linux ppc patch is out: http://code.google.com/p/spkgupload/downloads/detail?name=numpy1.4.1.p0.spkg hope it works this time!
Ok so I tried to build numpy1.4.1 with our patch on ppc. Didn't work. It may be a good start but it fails in the same way it did before:
[39mcompile options: 'Inumpy/core/src/private Inumpy/core/src Inumpy/core Inumpy/core/src/npymath Inumpy/core/src/multiarray Inumpy/core/src/umath Inumpy/core/include I/usr/include/python2.6 c'[0m [39mpowerpcunknownlinuxgnugcc: _configtest.c[0m [39mremoving: _configtest.c _configtest.o[0m Traceback (most recent call last): File "setup.py", line 187, in <module> setup_package() File "setup.py", line 180, in setup_package configuration=configuration ) File "/scratch/portage/portage/devpython/numpy1.4.1/work/numpy1.4.1/numpy/distutils/core.py", line 186, in setup return old_setup(**new_attr) File "/usr/lib/python2.6/distutils/core.py", line 152, in setup dist.run_commands() File "/usr/lib/python2.6/distutils/dist.py", line 975, in run_commands self.run_command(cmd) File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command cmd_obj.run() File "/scratch/portage/portage/devpython/numpy1.4.1/work/numpy1.4.1/numpy/distutils/command/build.py", line 37, in run old_build.run(self) File "/usr/lib/python2.6/distutils/command/build.py", line 134, in run self.run_command(cmd_name) File "/usr/lib/python2.6/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command cmd_obj.run() File "/scratch/portage/portage/devpython/numpy1.4.1/work/numpy1.4.1/numpy/distutils/command/build_src.py", line 152, in run self.build_sources() File "/scratch/portage/portage/devpython/numpy1.4.1/work/numpy1.4.1/numpy/distutils/command/build_src.py", line 169, in build_sources self.build_extension_sources(ext) File "/scratch/portage/portage/devpython/numpy1.4.1/work/numpy1.4.1/numpy/distutils/command/build_src.py", line 328, in build_extension_sources sources = self.generate_sources(sources, ext) File "/scratch/portage/portage/devpython/numpy1.4.1/work/numpy1.4.1/numpy/distutils/command/build_src.py", line 385, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 416, in generate_config_h rep = check_long_double_representation(config_cmd) File "numpy/core/setup_common.py", line 136, in check_long_double_representation type = long_double_representation(pyod(object)) File "numpy/core/setup_common.py", line 244, in long_double_representation raise ValueError("Unrecognized format (%s)" % saw) ValueError: Unrecognized format (['001', '043', '105', '147', '211', '253', '315', '357', '301', '235', '157', '064', '124', '000', '000', '000', '000', '000', '000', '000', '000', '000', '000', '000', '376', '334', '272', '230', '166', '124', '062', '020']) [31;01m*[0m ERROR: devpython/numpy1.4.1 failed:
The expectation from numpy/core/setup_common.py is:
LONG_DOUBLE_REPRESENTATION_SRC = r""" /* "before" is 16 bytes to ensure there's no padding between it and "x". * We're not expecting any "long double" bigger than 16 bytes or with * alignment requirements stricter than 16 bytes. */ typedef %(type)s test_type; struct { char before[16]; test_type x; char after[8]; } foo = { { '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\001', '\043', '\105', '\147', '\211', '\253', '\315', '\357' }, 123456789.0, { '\376', '\334', '\272', '\230', '\166', '\124', '\062', '\020' } }; """
And when you compare the two you can see that it is an endianess problem.
comment:163 Changed 11 years ago by
 Description modified (diff)
 Summary changed from Upgrade numpy to 1.4.1 and scipy to 0.8 to Upgrade numpy to 1.5.0 and scipy to 0.8
Changed 11 years ago by
Changes of the spkginstall from the upgrade to 1.3.x to 1.5.0 (changes that were applied to 1.4.1 are also contained)
comment:164 Changed 11 years ago by
 Description modified (diff)
comment:165 Changed 11 years ago by
I am incorporating that in my 4.6alpha2 branch on Gentoo right now.
I will unfortunately be unable to test it on linux ppc before Wednesday 6th because of limited access to the hardware. So it may be too short for me to do a comprehensive review for the alpha3 deadline. I already know for sure that numpy1.5.0/scipy0.8.0 will build it's just the testing that will be lacking.
comment:166 Changed 11 years ago by
Build on linux x86 and tests pass here on 4.6alpha2.
comment:167 Changed 11 years ago by
What do I need to do to install these? I did this: unpacked a sage4.6.alpha2 tarball, downloaded the new numpy and scipy spkg files, applied the file "trac_9808_numpy_doctest_change.patch" to the main Sage library spkg, and built Sage from scratch (with SAGE_CHECK=yes, except when installing the python spkg, which always fails selftests). On taurus (a skynet linux machine, x86_64Linuxnehalemfc), I'm getting doctest failures:
The following tests failed: sage t long devel/sage/sage/plot/plot.py # 12 doctests failed sage t long devel/sage/sage/misc/citation.pyx # 2 doctests failed sage t long devel/sage/sage/plot/misc.py # 4 doctests failed
For example:
sage t long devel/sage/sage/misc/citation.pyx ********************************************************************** File "/home/palmieri/taurus/numpy/sage4.6.alpha2/devel/sagemain/sage/misc/citation.pyx", line 60\ : sage: get_systems('integrate(x^2, x)') Expected: ['ginac', 'Maxima'] Got: ['numpy', 'ginac', 'Maxima'] **********************************************************************
and
sage t long devel/sage/sage/plot/plot.py ********************************************************************** File "/home/palmieri/taurus/numpy/sage4.6.alpha2/devel/sagemain/sage/plot/plot.py", line 207: sage: (g1+g2).show(ticks=pi/6, tick_formatter=pi) # show their sum, nicely formatted Exception raised: Traceback (most recent call last): File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_0[49]>", line 1, in <module> (g1+g2).show(ticks=pi/Integer(6), tick_formatter=pi) # show their sum, nicely formatted###line 207: sage: (g1+g2).show(ticks=pi/6, tick_formatter=pi) # show their sum, nicely formatted File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/sage/plot/plot.py", line 1546, in show self.save(DOCTEST_MODE_FILE, **options) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/sage/plot/plot.py", line 2202, in save figure.savefig(filename,dpi=dpi,bbox_inches='tight',**options) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/figure.py", line 1032, in savefig self.canvas.print_figure(*args, **kwargs) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/backend_bases.py", line 1455, in print_figure **kwargs) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/backends/backend_agg.py", line 358, in print_png FigureCanvasAgg.draw(self) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/backends/backend_agg.py", line 314, in draw self.figure.draw(self.renderer) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/artist.py", line 46, in draw_wrapper draw(artist, renderer, *args, **kwargs) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/figure.py", line 773, in draw for a in self.axes: a.draw(renderer) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/artist.py", line 46, in draw_wrapper draw(artist, renderer, *args, **kwargs) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/axes.py", line 1735, in draw a.draw(renderer) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/artist.py", line 46, in draw_wrapper draw(artist, renderer, *args, **kwargs) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/axis.py", line 742, in draw tick.draw(renderer) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/artist.py", line 46, in draw_wrapper draw(artist, renderer, *args, **kwargs) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/axis.py", line 196, in draw self.label1.draw(renderer) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/text.py", line 518, in draw bbox, info = self._get_layout(renderer) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/text.py", line 280, in _get_layout clean_line, self._fontproperties, ismath=ismath) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/backends/backend_agg.py", line 156, in get_text_width_height_descent self.mathtext_parser.parse(s, self.dpi, prop) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/mathtext.py", line 2797, in parse font_output = fontset_class(prop, backend) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/mathtext.py", line 658, in __init__ self._stix_fallback = StixFonts(*args, **kwargs) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/mathtext.py", line 900, in __init__ fullpath = findfont(name) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/font_manager.py", line 1306, in findfont if not os.path.exists(font): File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python2.6/genericpath.py", line 18, in exists st = os.stat(path) TypeError: coercing to Unicode: need string or buffer, dict found
comment:168 Changed 11 years ago by
Another report: on the skynet machine fulvia (Solaris on x86), matplotlib fails to find numpy, so it doesn't install:
============================================================================ BUILDING MATPLOTLIB matplotlib: 0.99.3 python: 2.6.4 (r264:75706, Oct 4 2010, 18:22:25) [GCC 4.5.1] platform: sunos5 REQUIRED DEPENDENCIES numpy: no * You must install numpy 1.1 or later to build * matplotlib. Error building matplotlib package. real 0m1.344s user 0m0.062s sys 0m0.093s sage: An error occurred while installing matplotlib0.99.3
The new numpy package has already been installed at this point in the build, so I don't know what's going on here. On the same machine, scipy fails to install:
gcc version 4.5.1 (GCC) **************************************************** ./spkginstall: LDFLAGS=shared: is not an identifier
Perhaps you need something like LDFLAGS="shared"
? I'm not really sure.
comment:169 followup: ↓ 170 Changed 11 years ago by
Can you try the new matplotlib spkg on #9221, if you think it is a matplotlib problem?
comment:170 in reply to: ↑ 169 Changed 11 years ago by
Replying to jason:
Can you try the new matplotlib spkg on #9221, if you think it is a matplotlib problem?
For fulvia, it doesn't help: I still get the same error about numpy not being installed. If I run ./sage python
and then import numpy
, I get an error, so although the numpy spkg claims to install correctly, there is some kind of problem:
ImportError: ld.so.1: python: fatal: relocation error: file /home/palmieri/fulvia/numpy/sage4.6.alpha2/local/lib/python2.6/sitepackages/numpy/core/multiarray.so: symbol isfinite: referenced symbol not found
For taurus, using the new matplotlib spkg (and applying the patch from #9221) does seem to fix most of the problems:
 The following tests failed: sage t long devel/sage/sage/misc/citation.pyx # 2 doctests failed 
comment:171 followup: ↓ 172 Changed 11 years ago by
Dear me!
And I was going to report numpy1.5/scipy0.8 build on ppc (should be running testall now, I'd do long but on that hardware we would still be there next week).
OK there are several issues to be solved: first "shared" is a good idea. That could be it.
Failing to build matplotlib with an installed numpy :) where did I see something like that... Why, I reported a bug on that on gentoo bugzilla a while ago: http://bugs.gentoo.org/show_bug.cgi?id=320669.
Long story short: that means your lapack (and possibly ATLAS) install is hosed. Probably related to problems producing .so libraries.
I don't get any of the failures you report. A quick package check on Gentoo shows that: matplotlib and rpy depend on numpy and it would be a good idea to rebuild them.
I am a bit worried about the unicode mention in plot.py, that may mean a mismatch between various packages configuration regarding unicode (python/numpy/matplotlib).
comment:172 in reply to: ↑ 171 ; followups: ↓ 173 ↓ 174 Changed 11 years ago by
Replying to fbissey:
Dear me!
And I was going to report numpy1.5/scipy0.8 build on ppc (should be running testall now, I'd do long but on that hardware we would still be there next week).
OK there are several issues to be solved: first "shared" is a good idea. That could be it.
Oops, it looks like it's already quoted in the spkginstall file. I changed the first line from
#!/bin/sh
to
#!/usr/bin/env bash
I don't know if this was a good idea, but it did get past that error, only to run into the same import error as when trying to install matplotlib.
Failing to build matplotlib with an installed numpy :) where did I see something like that... Why, I reported a bug on that on gentoo bugzilla a while ago: http://bugs.gentoo.org/show_bug.cgi?id=320669.
Long story short: that means your lapack (and possibly ATLAS) install is hosed. Probably related to problems producing .so libraries.
Hmm. We've had problems with shared libraries with ATLAS before on Solaris machines, among others, but it's interesting that this particular problem wouldn't have shown up before this.
I don't get any of the failures you report. A quick package check on Gentoo shows that: matplotlib and rpy depend on numpy and it would be a good idea to rebuild them.
Since I'm building from scratch, all of the dependencies should work themselves out.
I am a bit worried about the unicode mention in plot.py, that may mean a mismatch between various packages configuration regarding unicode (python/numpy/matplotlib).
As Jason suggested, the new matplotlib spkg seems to fix this, at least on taurus. Since that package has been merged as of 4.6.alpha3, it should be taken care of.
comment:173 in reply to: ↑ 172 Changed 11 years ago by
Replying to jhpalmieri:
Replying to fbissey:
Dear me!
And I was going to report numpy1.5/scipy0.8 build on ppc (should be running testall now, I'd do long but on that hardware we would still be there next week).
OK there are several issues to be solved: first "shared" is a good idea. That could be it.
Oops, it looks like it's already quoted in the spkginstall file. I changed the first line from
#!/bin/shto
#!/usr/bin/env bashI don't know if this was a good idea, but it did get past that error, only to run into the same import error as when trying to install matplotlib.
The old version of the spkg was doing that as well. I don't think we should have dropped it. There may be a more elegant solution but in the meantime that should do.
Failing to build matplotlib with an installed numpy :) where did I see something like that... Why, I reported a bug on that on gentoo bugzilla a while ago: http://bugs.gentoo.org/show_bug.cgi?id=320669.
Long story short: that means your lapack (and possibly ATLAS) install is hosed. Probably related to problems producing .so libraries.
Hmm. We've had problems with shared libraries with ATLAS before on Solaris machines, among others, but it's interesting that this particular problem wouldn't have shown up before this.
I cannot comment on that, I don't know the package history of that machine. But searching for numpy in the closed bugs from gentoo's bugzilla show that I haven't been the only victim of that and the answer is always the same: for a reason or another lapack or blas/cblas is foobar.
I don't get any of the failures you report. A quick package check on Gentoo shows that: matplotlib and rpy depend on numpy and it would be a good idea to rebuild them.
Since I'm building from scratch, all of the dependencies should work themselves out.
I am a bit worried about the unicode mention in plot.py, that may mean a mismatch between various packages configuration regarding unicode (python/numpy/matplotlib).
As Jason suggested, the new matplotlib spkg seems to fix this, at least on taurus. Since that package has been merged as of 4.6.alpha3, it should be taken care of.
Ah! I already have matplotlib1.0 here that's probably why I never saw any of that.
comment:174 in reply to: ↑ 172 Changed 11 years ago by
 Status changed from needs_review to needs_work
Replying to jhpalmieri:
Replying to fbissey:
Long story short: that means your lapack (and possibly ATLAS) install is hosed. Probably related to problems producing .so libraries.
Hmm. We've had problems with shared libraries with ATLAS before on Solaris machines, among others, but it's interesting that this particular problem wouldn't have shown up before this.
The upstream ATLAS program never builds shared libraries. Mathematica only ships with static libraries for ATLAS. It's not entirely clear to me why we bother building shared libraries for ATLAS, given they have tended to cause problems on several systems.
However, I'm not convinced that's the problem here. The ATLAS package has remained unchanged for a long time.
Dave
comment:175 followup: ↓ 176 Changed 11 years ago by
I don't have time to look at this now, but is this being built as C99? If not, that would explain it, since isfinite
was not defined until the C99 standard.
Dave
comment:176 in reply to: ↑ 175 ; followup: ↓ 177 Changed 11 years ago by
Replying to drkirkby:
I don't have time to look at this now, but is this being built as C99? If not, that would explain it, since
isfinite
was not defined until the C99 standard.
Why would fluvia not compile stuff in C99 by default? Compared to the other platforms? As for ATLAS it was a suggestion. I just know that the problem is linked to something broken in lapack and because of linking to libf77blas/libcblas it can come down to a problem with ATLAS. I don't know what exactly is happening with that particular machine.
comment:177 in reply to: ↑ 176 Changed 11 years ago by
Replying to fbissey:
Replying to drkirkby:
I don't have time to look at this now, but is this being built as C99? If not, that would explain it, since
isfinite
was not defined until the C99 standard.Why would fluvia not compile stuff in C99 by default? Compared to the other platforms? As for ATLAS it was a suggestion. I just know that the problem is linked to something broken in lapack and because of linking to libf77blas/libcblas it can come down to a problem with ATLAS. I don't know what exactly is happening with that particular machine.
gcc does not compile C99 by default.
http://gcc.gnu.org/onlinedocs/gcc4.5.1/gcc/CDialectOptions.html#CDialectOptions
says:
std= Determine the language standard. <snip> `gnu89' GNU dialect of ISO C90 (including some C99 features). This is the default for C code.
But isfinite
was not introduced until C99. The Sun headers tend not to define everything under the sun.
I'm not saying that's the issue. But it could be. I've seen a similar issue before.
Dave
comment:178 Changed 11 years ago by
For what it's worth, I redid the build on fulvia with SAGE_CHECK=yes, and ATLAS passed its selftests. The matplotlib package still fails to install, with the import error from numpy.
comment:179 followup: ↓ 180 Changed 11 years ago by
Does matplotlib generate any sort of config.log or similar? That might indicate what the problem is.
I know at one point I had issues on 64bit builds, when Numpy thought the static library was 32bit.
http://projects.scipy.org/numpy/ticket/1525 http://trac.sagemath.org/sage_trac/ticket/8086
comment:180 in reply to: ↑ 179 Changed 11 years ago by
By the way, I'm getting the same error on mark2 (Solaris on sparc) as on fulvia (Solaris on x86). Cicero (another linux box) doesn't have this problem with building matplotlib: it's built successfully and is now doctesting.
Replying to drkirkby:
Does matplotlib generate any sort of config.log or similar? That might indicate what the problem is.
I don't see any log file in spkg/build/matplotlib on the machines where this is failing. I think the problem is what I indicated above (comment:170): matplotlib tests whether numpy is installed by running python and trying "import numpy". In this case, that import fails, so matplotlib deduces that numpy is not installed.
Here's the relevant code (from setupext.py):
def check_for_numpy(): try: import numpy except ImportError: print_status("numpy", "no") print_message("You must install numpy 1.1 or later to build matplotlib.") return False
comment:181 Changed 11 years ago by
Here's a recursive grep for isfinite
in /usr/include
on my OpenSolaris machine. I've not tested the package on my OpenSolaris machine, but given the results on Solaris 10, I doubt it would build.
drkirkby@hawk:~$ ggrep R isfinite /usr/include /usr/include/python2.6/pyconfig.h:/* Define to 1 if you have the declaration of `isfinite', and to 0 if you /usr/include/python2.6/pymath.h:#define Py_IS_FINITE(X) isfinite(X) ggrep: warning: /usr/include/gphoto2/gphoto2: recursive directory loop /usr/include/iso/math_c99.h:#undef isfinite /usr/include/iso/math_c99.h:#define isfinite(x) __extension__( \ /usr/include/iso/math_c99.h: { __typeof(x) __x_r = (x); isfinite(__x_r) && \ /usr/include/iso/math_c99.h:#undef isfinite /usr/include/iso/math_c99.h:#define isfinite(x) __builtin_isfinite(x)
Note, there's no reference at all in math.h
to isfinite
. But when the conditions are right, the contents of /usr/include/iso/math_c99.h
will get included, and so the macro isfinite
will be defined.
IMHO, the error message John got:
ImportError: ld.so.1: python: fatal: relocation error: file /home/palmieri/fulvia/numpy/sage4.6.alpha2/local/lib/python2.6/sitepackages/numpy/core/multiarray.so: symbol isfinite: referenced symbol not found
is related to the fact that for whatever reason, the C99 headers are not being included properly.
Unless this is compiled with std=c99
, I doubt this will work.
This contrasts to a Linux box (sage.math) where it looks to me that isfinite
will be defined whenever math.h
is included. That is just an error in my opinion, as isfinite
was not defined until the C99 standard. So Linux is wrong to define it, but some software is assuming it's defined, and so fails when it is not.
IMHO, the bug is in unlikely to be LAPACK or ATLAS.
comment:182 Changed 11 years ago by
The more I look at this, the more I think it's a Numpy bug.
Python is not built with C99 on my machine. The config.log
of Python shows:
ac_cv_have_decl_isfinite=no
which is to be expected since its not built C99. So Numpy is at fault here I think, since that's making a reference to isfinite
despite Python is not built with C99, so it's not defined, as evidenced by Python's config.log
BTW John, do you want an account on my OpenSolaris machine? It can build Sage in under an hour, so is a bit quicker than fulvia.
Dave
comment:183 Changed 11 years ago by
IMHO, we need to see if we can reproduce this with the latest cvs/svn/whatever snapshot of Numpy, and if so, report it as a bug to
http://projects.scipy.org/numpy
If it's been fixed, then perhaps we can add a patch to the stable version, to just correct this bug.
Dave
comment:184 Changed 11 years ago by
I just downloaded the latest code (as listed here), replaced the "src" directory with it, and installed it. Running "./sage python" and then "import numpy" results in the same error about isfinite.
comment:185 Changed 11 years ago by
I reported this as a Numpy bug:
comment:186 Changed 11 years ago by
 Report Upstream changed from N/A to Reported upstream. Developers acknowledge bug.
I see this problem on my OpenSolaris machine too, which I expected would be the case.
On my OpenSolaris machine, Python's header file pyconfig.h has in it:
/* Define to 1 if you have the declaration of `isfinite', and to 0 if you don't. */ #define HAVE_DECL_ISFINITE 0
but Numpy's config.h has in it:
#define HAVE_DECL_ISFINITE
Someone by the usename of 'pv' has said on the Numpy bug report http://projects.scipy.org/numpy/ticket/1625
However, the code at numpy/core/setup.py:218 is wrong, since it checks for #ifdef and not #if for Python's HAVE_DECL_ISFINITE.
So it seems this is a bug, and one which should be easy to fix. Note the fix does not need to be Solarisspecific, as the code is basically wrong on any platform  you just happen to get away with it on systems where the header file defines isfinite
even though the code is not built C99.
It's gone midnight here, I'm tired and have a lot to do all this week. But if nobody else patches this, I'll be able to do it in the next day or two.
Dave
comment:187 Changed 11 years ago by
Actually a bigger fix is being proposed on the numpy list. See there  I'm going to bed!
dave
comment:188 followup: ↓ 190 Changed 11 years ago by
Dave, thanks a lot for working on this with the numpy people. I prepared a new spkg and so far, the new numpy builds on fulvia, and after "./sage python", "import numpy" works. Matplotlib and scipy have now successfully built. It will be a while before the build is complete, but things look good to me.
I'm putting the spkg up at http://sage.math.washington.edu/home/palmieri/misc/numpy1.5.0.svn20101005.spkg.
This is from a current svn snapshot based on 1.5.0. I don't know if we have a policy about names for such spkg's, so I just used the date. I also don't know if we have a policy about whether to include the repository information, so I've kept it. This makes the spkg much bigger than the old version...
Should we mark this as "needs review" again? It seems to depend on #9221.
comment:189 Changed 11 years ago by
Should we mark this as "needs review" again?
Or should we wait until the patch is positively reviewed on the numpy trac server?
comment:190 in reply to: ↑ 188 ; followup: ↓ 191 Changed 11 years ago by
Replying to jhpalmieri:
Dave, thanks a lot for working on this with the numpy people. I prepared a new spkg and so far, the new numpy builds on fulvia, and after "./sage python", "import numpy" works. Matplotlib and scipy have now successfully built. It will be a while before the build is complete, but things look good to me.
I'm putting the spkg up at http://sage.math.washington.edu/home/palmieri/misc/numpy1.5.0.svn20101005.spkg.
This is from a current svn snapshot based on 1.5.0. I don't know if we have a policy about names for such spkg's, so I just used the date. I also don't know if we have a policy about whether to include the repository information, so I've kept it. This makes the spkg much bigger than the old version...
You don't need to keep the svn repository history, but you do need to put the revision number in the spkg name (numpy1.5.0.svnXXXX).
However, I think it would be better to just include the one patch on top of 1.5.0, rather than pull an unstable release from subversion. See the comments at the top of this trac ticket about using an unstable release (like 1.5.0RC2).
comment:191 in reply to: ↑ 190 Changed 11 years ago by
However, I think it would be better to just include the one patch on top of 1.5.0, rather than pull an unstable release from subversion. See the comments at the top of this trac ticket about using an unstable release (like 1.5.0RC2).
I agree on this point, for what it's worth (since I will just be doing ./sage i
and testing).
comment:192 followup: ↓ 193 Changed 11 years ago by
Yes, but the patch was based on the svn code, not vanilla 1.5.0.
comment:193 in reply to: ↑ 192 Changed 11 years ago by
Replying to jhpalmieri:
Yes, but the patch was based on the svn code, not vanilla 1.5.0.
Are you saying that the patch does not apply to 1.5.0 directly?
comment:194 Changed 11 years ago by
The patch affects two files. One part of the patch is this:

numpy/core/setup.py
diff git a/numpy/core/setup.py b/numpy/core/setup.py index ad8d5cb..f71ec10 100644
a b def check_ieee_macros(config): 215 215 _macros = ["isnan", "isinf", "signbit", "isfinite"] 216 216 if sys.version_info[:2] >= (2, 6): 217 217 for f in _macros: 218 already_declared = config.check_decl(fname2def("decl_%s" % f), 218 py_symbol = fname2def("decl_%s" % f) 219 already_declared = config.check_decl(py_symbol, 219 220 headers=["Python.h", "math.h"]) 220 221 if already_declared: 221 pub.append('NPY_%s' % fname2def("decl_%s" % f)) 222 if config.check_macro_true(py_symbol, 223 headers=["Python.h", "math.h"]): 224 pub.append('NPY_%s' % fname2def("decl_%s" % f)) 222 225 else: 223 226 macros.append(f) 224 227 else:
and for example the line "already_declared = config.check_decl(fname2def("decl_%s" % f),
" is not present in the 1.5.0 version.
Maybe you can modify it (and maybe easily) to apply to 1.5.0, but the diff, as written, does not.
comment:195 Changed 11 years ago by
Okay, I have a new version of the spkg, based on vanilla 1.5.0. I haven't tested it because I just lost my internet connection to fulvia. See http://sage.math.washington.edu/home/palmieri/misc/numpy1.5.0.spkg. I'm also attaching the file displaying the differences between this spkg and the old 1.5.0 spkg posted here, except for two things: the new spkg includes modified copies of numpy/core/setup.py and numpy/distutils/command/config.py (in the patches directory  the src directory is unmodified), and I didn't see any reason to include them in the posted patch file. They are of course included in the repository, but to keep the patch file small, I edited them out before posting it.
Oh, the internet connection is back again. I'm rebuilding on fulvia from scratch, which will take a while, but I just rebuilt the numpy package on mark2 successfully: "import numpy" works.
comment:196 followup: ↓ 198 Changed 11 years ago by
Ah I'm only a few days away and everyones busy. Sometimes I think this ticket is cursed... How high are the bets that this will work one day before 1.5.1 is released? =D
@jhpalmiere, drkirkby, fbissey: Thanks for all the trouble!
If you want I build the package with the new patches and upload it on google code for testing. (Or do you want to upload it?) I will test it out on Ubuntu tonight for sure.
grettings, maldun
comment:197 Changed 11 years ago by
 Description modified (diff)
Ok I oversaw the link in the post... I changed the old link in the discription to your new one. The new package is doing fine on ubuntu. report for doctests will come soon!
comment:198 in reply to: ↑ 196 Changed 11 years ago by
Replying to maldun:
Sometimes I think this ticket is cursed... How high are the bets that this will work one day before 1.5.1 is released? =D
Well the Numpy now works, but Scipy is presenting a problem on OpenSolaris due to the fact that the wrong compiler is being used. The variable SAGE_FORTRAN
should specify the compiler:
drkirkby@hawk:~$ echo $SAGE_FORTRAN /usr/local/gcc4.5.0delayed/bin/gfortran
used for all Fortran bits in Sage. But this version of Scipy has now decided to go off and use /usr/bin/f90
which is the Sun Fortran compiler.
I note that Francois has added this line:
export FC="${SAGE_LOCAL}/bin/sage_fortran"
but it is not sufficient to get Scipy to use the right compiler.
f90: Internal Error, code=fwinterfacecexp277, last src=scipy/sparse/linalg/eigen/arpack/ARPACK/SRC/zneupd.f:673 error: Command "/usr/bin/f90 ftrap=%none f77 xcode=pic32 Iscipy/sparse/linalg/eigen/arpack/ARPACK/SRC I/export/home/drkirkby/sage4.6.alpha2/local/lib/python2.6/sitepackages/numpy/core/include c c scipy/sparse/linalg/eigen/arpack/ARPACK/SRC/zneupd.f o build/temp.solaris2.11i86pc2.6/scipy/sparse/linalg/eigen/arpack/ARPACK/SRC/zneupd.o" failed with exit status 1 Error building scipy. real 2m40.038s user 1m13.094s sys 0m12.185s sage: An error occurred while installing scipy0.8
I then set F90
, but found Scipy was then using the Fortran 77 compiler /usr/bin/f77
. So I had to set F77
too. I decided to set F95
just in case, there is, or the developers add some Fortran 95 code. So with all these set:
export FC="$SAGE_LOCAL/bin/sage_fortran" export F77="$SAGE_LOCAL/bin/sage_fortran" export F90="$SAGE_LOCAL/bin/sage_fortran" export F95="$SAGE_LOCAL/bin/sage_fortran"
Scipy does build on OpenSolaris OK.
But I looked at the SPKG.txt and found there are many things missing, which should be there. I will try to improve that and post a Scipy package.
Dave
comment:199 Changed 11 years ago by
 Description modified (diff)
Changed 11 years ago by
Sets F77, F90 and F95 so SciPy? really does use the Fortran compiler we want. Setting FC alone is insufficient.
comment:200 followups: ↓ 201 ↓ 204 Changed 11 years ago by
Can I still test this by upgrading the packages, doing ./sage ba
, and adding the doctest patch? By the way, with the Pynac upgrade now in 4.6.alpha3 (see #9901), it's conceivable that the functions/hyperbolic.py piece of that will have to be rebased.
comment:201 in reply to: ↑ 200 Changed 11 years ago by
Replying to kcrisman:
Can I still test this by upgrading the packages, doing
./sage ba
, and adding the doctest patch? By the way, with the Pynac upgrade now in 4.6.alpha3 (see #9901), it's conceivable that the functions/hyperbolic.py piece of that will have to be rebased.
I don't know enough about what needs to be done to test this. I've not even applied the patch myself yet. But the packages are at least building now, which is a start!
I really need to be doing other things today, so are going to leave this ticket for the rest of the day  at least I will not be adding any code.
Dave
comment:202 followup: ↓ 203 Changed 11 years ago by
I've not applied the patches, so have not checked this fully, so I am not marking for review. But the two packages do now both build on OpenSolaris 06/2009 and I assume will do on Solaris 10 both SPARC and x86. However, I have not tested these on Solaris, or any other system
I also added a lot of information which was missing from SPKG.txt
The new SciPy package is here.
http://boxen.math.washington.edu/home/kirkby/patches/scipy0.8.spkg
I don't have time to sort out how to apply the patches, fully build and doctest this. Sorry.
Dave
comment:203 in reply to: ↑ 202 Changed 11 years ago by
Replying to drkirkby:
I've not applied the patches, so have not checked this fully, so I am not marking for review. But the two packages do now both build on OpenSolaris 06/2009 and I assume will do on Solaris 10 both SPARC and x86. However, I have not tested these on Solaris, or any other system
I also added a lot of information which was missing from SPKG.txt
The new SciPy package is here.
http://boxen.math.washington.edu/home/kirkby/patches/scipy0.8.spkg
I don't have time to sort out how to apply the patches, fully build and doctest this. Sorry.
Dave
Thanks for doing this! I already applied your package on my machine, it builded fine! I hope I find some time to make the tests soon.
comment:204 in reply to: ↑ 200 ; followup: ↓ 208 Changed 11 years ago by
Replying to kcrisman:
Can I still test this by upgrading the packages, doing
./sage ba
, and adding the doctest patch? By the way, with the Pynac upgrade now in 4.6.alpha3 (see #9901), it's conceivable that the functions/hyperbolic.py piece of that will have to be rebased.
Normally it should do. If you want save some time, write something in the numpy.pxd delete it again and save the file, so that the timestamp gets refreshed. After that sage b should also do the trick.
comment:205 Changed 11 years ago by
@changed doctests: Since this ticket won't be ready before the deadline, I wait till the release comes out and change it then.
comment:206 Changed 11 years ago by
I've built and tested this, and get the following test failures on my OpenSolaris machine:
 The following tests failed: sage t long devel/sage/doc/en/bordeaux_2008/introduction.rst # 1 doctests failed sage t long devel/sage/sage/numerical/test.py # 3 doctests failed sage t long devel/sage/sage/rings/polynomial/polynomial_element.pyx # 1 doctests failed sage t long devel/sage/sage/tensor/differential_forms.py # 1 doctests failed sage t long devel/sage/sage/plot/plot3d/list_plot3d.py # 8 doctests failed  Total time for all tests: 1815.6 seconds
Prior to upgrading Numpy and !Scipy, I had only two failures, which were:
devel/sage/sage/rings/polynomial/polynomial_element.pyx
#10042
and
devel/sage/sage/tensor/differential_forms.py
#10041
So that is three new failures after the upgrade of these two packages.
sage t long devel/sage/doc/en/bordeaux_2008/introduction.rst # 1 doctests failed sage t long devel/sage/sage/numerical/test.py # 3 doctests failed sage t long devel/sage/sage/plot/plot3d/list_plot3d.py # 8 doctests failed
I've attached a log showing just the failed tests. At least some of the problems are related to:
ImportError: No module named delaunay
ImportError: No module named arpack
Dave
Changed 11 years ago by
The three new doctest failures which are almost certainly a result of upgrading Numpy and Scipy
comment:207 followup: ↓ 209 Changed 11 years ago by
In a build from scratch on fulvia (with the previous version of the scipy package), I see these failures:
The following tests failed: sage t long devel/sage/sage/tensor/differential_forms.py # 1 doctests failed sage t long devel/sage/sage/misc/citation.pyx # 2 doctests failed sage t long devel/sage/sage/rings/polynomial/polynomial_element.pyx # 1 doctests failed
The first of these is covered by #10041. The second of these I mentioned before:
sage t long devel/sage/sage/misc/citation.pyx ********************************************************************** File "/home/palmieri/fulvia/numpy/sage4.6.alpha2/devel/sagemain/sage/misc/citation.pyx", line 60: sage: get_systems('integrate(x^2, x)') Expected: ['ginac', 'Maxima'] Got: ['numpy', 'ginac', 'Maxima'] ********************************************************************** File "/home/palmieri/fulvia/numpy/sage4.6.alpha2/devel/sagemain/sage/misc/citation.pyx", line 64: sage: get_systems('I.primary_decomposition()') Expected: ['Singular'] Got: ['Singular', 'numpy'] ********************************************************************** 1 items had failures: 2 of 8 in __main__.example_0 ***Test Failed*** 2 failures.
The third of these is mostly covered by #10042, except for the first line, which somewhat concerns me:
sage t long devel/sage/sage/rings/polynomial/polynomial_element.pyx Warning: invalid value encountered in isfinite ********************************************************************** File "/home/palmieri/fulvia/numpy/sage4.6.alpha2/devel/sagemain/sage/rings/polynomial/polynomial_element.pyx", line 4316: sage: f.roots(algorithm='numpy')
Dave, do you see this?
comment:208 in reply to: ↑ 204 Changed 11 years ago by
Normally it should do. If you want save some time, write something in the numpy.pxd delete it again and save the file, so that the timestamp gets refreshed. After that sage b should also do the trick.
I think that just the command
touch .../numpy.pxd
should work, as leif pointed out. Maybe this isn't a universal command? Anyway, it did the trick on my computer  MUCH less time to build ~12 files than all the Cython files! It's still doing this, so I won't have a chance to test yet, but I hope I won't get the failures drkirkby gets.
comment:209 in reply to: ↑ 207 ; followup: ↓ 211 Changed 11 years ago by
Replying to jhpalmieri:
In a build from scratch on fulvia (with the previous version of the scipy package), I see these failures:
<snip>
Dave, do you see this?
IIRC, the old Scipy would not build at all once Numpy had been upgraded.
I originally thought it was a bit silly to create a ticket which upgraded two packages, but then I came to the conclusion one could not be done without the other. Perhaps I was mistaken.
So the short answer is no I do not see what you got, but I was unable to build the old SciPy?.
I will start another fresh build. I'll report in an hour or so.
Dave
comment:210 Changed 11 years ago by
Not, since the patch to resolve the isfinite
problem on http://projects.scipy.org/numpy/ticket/1625 worked, I commented on that fact on the ticket. That patch has now been committed to the Numpy source code, so will be in the next release, which is expected to be 1.5.1 but may be 2.0.0.
So we have that one solved!
Dave
comment:211 in reply to: ↑ 209 Changed 11 years ago by
Replying to drkirkby:
Dave, do you see this?
IIRC, the old Scipy would not build at all once Numpy had been upgraded.
I originally thought it was a bit silly to create a ticket which upgraded two packages, but then I came to the conclusion one could not be done without the other. Perhaps I was mistaken.
So the short answer is no I do not see what you got, but I was unable to build the old SciPy?.
I will start another fresh build. I'll report in an hour or so.
Dave
Well, I started another build, and whilst the new Numpy built, the old Scipy failed:
f90: Internal Error, code=fwinterfacecexp277, last src=scipy/sparse/linalg/eigen/arpack/ARPACK/SRC/zneupd.f:673 f90: Warning: The option xcode=pic32 has no effect on x86 scipy/sparse/linalg/eigen/arpack/ARPACK/SRC/zneupd.f: zneupd: f90: Internal Error, code=fwinterfacecexp277, last src=scipy/sparse/linalg/eigen/arpack/ARPACK/SRC/zneupd.f:673 error: Command "/usr/bin/f90 ftrap=%none f77 xcode=pic32 Iscipy/sparse/linalg/eigen/arpack/ARPACK/SRC I/export/home/drkirkby/half/sage4.6.alpha2/local/lib/python2.6/sitepackages/numpy/core/include c c scipy/sparse/linalg/eigen/arpack/ARPACK/SRC/zneupd.f o build/temp.solaris2.11i86pc2.6/scipy/sparse/linalg/eigen/arpack/ARPACK/SRC/zneupd.o" failed with exit status 1 Error building scipy. real 1m52.484s user 1m6.919s sys 0m8.931s sage: An error occurred while installing scipy0.7.p5
Note this is picking up the Sun compilers, not the gcc one.
That was the problem I had with the new Scipy, but never the old one. Perhaps something in the new Numpy has changed so that the old Scipy is finding the wrong compiler. I've no idea!
Maybe the Sun compilers are not in your path on fulvia.
Dave
comment:212 followup: ↓ 215 Changed 11 years ago by
I just started a fresh build with the new Numpy and the new SciPy. I assume once it builds, I need to apply the patch, rebuild the library and doctest again. I've done this once today, but I'll try again just to make sure I did not make an error.
dave
comment:213 followup: ↓ 214 Changed 11 years ago by
I don't know what's happening with me today. Things which did work no longer do!
I'm getting a failure of scipy_sandbox20071020.p5
to build now. It has the same issue SciPy had  using the wrong compiler.
I'm wondering if Numpy should in some way indicate the compiler so programs which depend on Numpy (SciPy and I assume scipy_sandbox), use the same compiler?
This is what I just see:
f90: Internal Error, code=fwinterfacecexp277, last src=./ARPACK/SRC/zneupd.f:673 error: Command "/usr/bin/f90 ftrap=%none f77 xcode=pic32 I./ARPACK/SRC I/export/home/drkirkby/new/sage4.6.alpha2/local/lib/python2.6/sitepackages/numpy/core/include c c ./ARPACK/SRC/zneupd.f o build/temp.solaris2.11i86pc2.6/ARPACK/SRC/zneupd.o" failed with exit status 1 Error building arpack real 0m11.229s user 0m4.239s sys 0m1.659s sage: An error occurred while installing scipy_sandbox20071020.p5
I could easily patch scipy_sandbox20071020.p5 so it uses the right compiler, but I'm wondering if Numpy should tell it the compiler to use, and this version is failing to do that.
Has anyone changed anything connected with the Fortran compiler in Numpy?
Dave
comment:214 in reply to: ↑ 213 Changed 11 years ago by
Replying to drkirkby:
Has anyone changed anything connected with the Fortran compiler in Numpy?
Dave
I've just checked, and see someone (I think Francois) made some changes to how Fortran is handled in Numpy. He has exported FC, but not F77, F90 or F95. I think that's having a knockon effect on anything that depends on Numpy, such as !Scipy and scipysandbox.
I'll change the Numpy package so it exports FC, F77, F90 & F95. I suspect that will fix the problem.
The doctest failures I got before, can easily be explained by this Fortran mixup.
Dave
comment:215 in reply to: ↑ 212 Changed 11 years ago by
Replying to drkirkby:
I just started a fresh build with the new Numpy and the new SciPy. I assume once it builds, I need to apply the patch, rebuild the library and doctest again.
Instead, before building, you can cd into spkg/standard, type "tar jxf sage4.6.alpha2.spkg", cd into the newly created directory "sage4.6.alpha2", and apply the patches ("hg import ..."). Then cd back to spkg/standard and type "sage pkg sage4.6.alpha2" to recreate the spkg file for the Sage library, but with the patches included. Then you can "make ptestlong".
By the way, I just noticed these lines in the new scipy spkginstall file:
if [ `uname` = "Darwin" ]; then unset ATLAS unset BLAS unset LAPACK export FC="sage_fortran" else export LDFLAGS="${LDFLAGS} shared" fi export FC="${SAGE_LOCAL}/bin/sage_fortran"
It seems to me that the last line will override the similar line in the "if" block. Could you fix that, too?
comment:216 Changed 11 years ago by
By the way, I just noticed these lines in the new scipy spkginstall file:
Sorry, I meant the numpy spkginstall file.
comment:217 followup: ↓ 218 Changed 11 years ago by
 Description modified (diff)
Dave: Is this the sort of change you had in mind?

spkginstall
diff r 1c2a7c8515fc spkginstall
a b 37 37 unset ATLAS 38 38 unset BLAS 39 39 unset LAPACK 40 export FC="sage_fortran"41 40 else 42 41 export LDFLAGS="${LDFLAGS} shared" 43 42 fi 44 43 export FC="${SAGE_LOCAL}/bin/sage_fortran" 44 export F77="${SAGE_LOCAL}/bin/sage_fortran" 45 export F90="${SAGE_LOCAL}/bin/sage_fortran" 46 export F95="${SAGE_LOCAL}/bin/sage_fortran" 45 47 export NUMPY_FCONFIG="config_fc noopt noarch" 46 48 47 49 ################################################
I've prepared a new spkg in http://sage.math.washington.edu/home/palmieri/SPKG/numpy1.5.0.spkg. (This is not the same place as the previous one. I'm updating the ticket description with this new version.)
comment:218 in reply to: ↑ 217 Changed 11 years ago by
Replying to jhpalmieri:
Dave: Is this the sort of change you had in mind?
Yes. I personally only ever use () and not {} in scripts, but I assume the latter is ok.
That probably means that SciPy and scipy_sandbox do not need similar  I suspect they pick these up from Numpy. But we will see. It might be necessary to add it to scipy_sandbox too.
Dave
comment:219 followup: ↓ 229 Changed 11 years ago by
Back from snoozeland via morning teaching.
I would have expected that the fortran settings in scipy and numpy would have to match. I am completely astonished that you had problems with scipy and not numpy. In fact on Gentoo we currently have a bug open because numpy picks up the intel fortran compiler when it shouldn't.
Other than that I haven't tested the new spkg on linux ppc, but I don't think anything that has been touched that would affect compilation . Most of the test suite passed, a lot of time out on this hardware but since we are not in a hurry anymore I will do a run of the long test suite once I have checked the latest spkg.
I am not seeing the failure on citation.pyx or polynomial_element.pyx on that hardware, I just checked specifically.
Francois
comment:220 Changed 11 years ago by
I hit the same issue with scipy_sandbox
. I've put that on another ticket, with a patch that needs review.
Dave
comment:221 Changed 11 years ago by
Oops, I forgot to say, the other ticket is #10092. Perhaps someone can review that  it should be very easy.
dave
comment:222 Changed 11 years ago by
I'm having problems with scipy on Mac OS X (Intel, 10.6):
compiling Fortran sources Fortran f77 compiler: /Applications/sage_builds/numpy/sage4.6.alpha2/local/bin/sage_fortran Wall ffixedform fnosecondunderscore arch ppc arch x86_64 arch ppc64 fPIC O3 funrollloops Fortran f90 compiler: /Applications/sage_builds/numpy/sage4.6.alpha2/local/bin/sage_fortran Wall fnosecondunderscore arch ppc arch x86_64 arch ppc64 fPIC O3 funrollloops Fortran fix compiler: /Applications/sage_builds/numpy/sage4.6.alpha2/local/bin/sage_fortran Wall ffixedform fnosecondunderscore Wall fnosecondunderscore arch ppc arch x86_64 arch ppc64 fPIC O3 funrollloops creating build/temp.macosx10.6i3862.6 creating build/temp.macosx10.6i3862.6/scipy creating build/temp.macosx10.6i3862.6/scipy/fftpack creating build/temp.macosx10.6i3862.6/scipy/fftpack/src creating build/temp.macosx10.6i3862.6/scipy/fftpack/src/dfftpack compile options: 'I/Applications/sage_builds/numpy/sage4.6.alpha2/local/lib/python2.6/sitepackages/numpy/core/include c' sage_fortran:f77: scipy/fftpack/src/dfftpack/dcosqb.f lipo: /var/folders/ks/ksTkDzjXF7KfYo6uWLAQHk+++TM/Tmp//ccMIVvsm.out and /var/folders/ks/ksTkDzjXF7KfYo6uWLAQHk+++TM/Tmp//ccmhVCg6.out have the same architectures (ppc64) and can't be in the same fat output file lipo: /var/folders/ks/ksTkDzjXF7KfYo6uWLAQHk+++TM/Tmp//ccMIVvsm.out and /var/folders/ks/ksTkDzjXF7KfYo6uWLAQHk+++TM/Tmp//ccmhVCg6.out have the same architectures (ppc64) and can't be in the same fat output file error: Command "/Applications/sage_builds/numpy/sage4.6.alpha2/local/bin/sage_fortran Wall ffixedform fnosecondunderscore arch ppc arch x86_64 arch ppc64 fPIC O3 funrollloops I/Applications/sage_builds/numpy/sage4.6.alpha2/local/lib/python2.6/sitepackages/numpy/core/include c c scipy/fftpack/src/dfftpack/dcosqb.f o build/temp.macosx10.6i3862.6/scipy/fftpack/src/dfftpack/dcosqb.o" failed with exit status 1 Error building scipy.
comment:223 Changed 11 years ago by
OK, things are a lot better now on OpenSolaris, though seem to be a problem for John on OS X!!
Using:
 The updated Numpy on this ticket
 The updated Scipy which John made to export FC, F77, F90 and F95
 An upated scipy_sandbox on #10092 which I made to export FC, F77, F90 and F95
then the three additional test failures caused by the Numpy/Scipy? changes have all been resolved. This is on a Sun Ultra 27 running OpenSolaris 06/2009.
drkirkby@hawk:~/new/sage4.6.alpha2$ ./sage t long devel/sage/doc/en/bordeaux_2008/introduction.rst sage t long "devel/sage/doc/en/bordeaux_2008/introduction.rst" [3.1 s]  All tests passed! Total time for all tests: 3.1 seconds drkirkby@hawk:~/new/sage4.6.alpha2$ ./sage t long devel/sage/sage/numerical/test.py sage t long "devel/sage/sage/numerical/test.py" [4.5 s]  All tests passed! Total time for all tests: 4.5 seconds drkirkby@hawk:~/new/sage4.6.alpha2$ ./sage t long devel/sage/sage/plot/plot3d/list_plot3d.py sage t long "devel/sage/sage/plot/plot3d/list_plot3d.py" [3.1 s]  All tests passed! Total time for all tests: 3.1 seconds
I suspect these will work on the Solaris x86 and Solaris SPARC machines, but since:
 Skynet is down for maintenance.
 t2 is down and will remain so until William is less rude towards me.
 Everything except my slowest SPARC is powered off.
I'm not able to check this on Solaris 10, either SPARC or x86.
I will run a 'ptestlong' which will take half an hour.
We will have to look at the OS X issues now!
Will this ticket ever get closed?
Dave
comment:224 followup: ↓ 227 Changed 11 years ago by
I've got no idea if this will solve it, but the sage_fortran script has in it:
/usr/local/gcc4.5.0delayed/bin/gfortran fPIC $@
but the bit at the end should be quoted:
/usr/local/gcc4.5.0delayed/bin/gfortran fPIC "$@"
I've never found that an issue with any package in Sage, but when I tried to use an identical script for gcc and g++, I found problems which got resolved when I quoted the end of the line. You could try manually editing the script, putting a couple of quotation marks and seeing if that helps.
This is very puzzling, but it is 1:25 AM here, and time I hit the bed.
John, do you think the changing of the Fortran settings in these packages has caused the problem on OS X?
Dave
comment:225 Changed 11 years ago by
Is it supposed to have all of these: arch ppc arch x86_64 arch ppc64 on OSX? Are making a fat build that could be used on all these archs somehow?
comment:226 Changed 11 years ago by
John, do you think the changing of the Fortran settings in these packages has caused the problem on OS X?
When I saw the problem, I tried rebuilding with older versions, or with versions modified to undo those changes. I had the same problem, so assuming I didn't screw anything up, it wasn't caused by changing the Fortran settings.
You could try manually editing the script, putting a couple of quotation marks and seeing if that helps.
I'll try it.
comment:227 in reply to: ↑ 224 Changed 11 years ago by
Replying to drkirkby:
I've got no idea if this will solve it, but the sage_fortran script has in it:
/usr/local/gcc4.5.0delayed/bin/gfortran fPIC $@
On my Mac, it actually looks like this:
#!/usr/bin/env bash $SAGE_LOCAL/bin/gfortran64 m64 "$@"
So I don't need to add quotes  they're already there. And so this is not the cause of the problem.
comment:228 Changed 11 years ago by
 Description modified (diff)
 Report Upstream changed from Reported upstream. Developers acknowledge bug. to Fixed upstream, but not in a stable release.
I run ptestlong after sorting out the Fortran issues, and get:
 The following tests failed: sage t long devel/sage/sage/rings/polynomial/polynomial_element.pyx # 1 doctests failed sage t long devel/sage/sage/tensor/differential_forms.py # 1 doctests failed  Total time for all tests: 1868.7 seconds
which is two failures I was aware of, which are being worked on at #10041 and #10042.
However, to have only these 2 failures, and not 5, it's essential to use an updated package at #10092, which just adds exports FC, F77, F90 and F95 from scipy_sandbox.
Of course, that still does not resolve the OS X issues which John has. If they are new to these updated packages, then we still have a problem.
Changed 11 years ago by
A summary of changes to help reviewers  this ticket has got a bit complex, as 3 packages and a patch all need to be applied.
comment:229 in reply to: ↑ 219 Changed 11 years ago by
Replying to fbissey:
Back from snoozeland via morning teaching.
I would have expected that the fortran settings in scipy and numpy would have to match.
That seems not to the be case. It's probably a good idea if they do, but it looks like setting FC in Numpy is enough. I've just verified with grep by looking at the Numpy log that it only ever uses the GNU compiler when compiling F90 code and does not attempt to use /usr/bin/f90
drkirkby@hawk:~/new/sage4.6.alpha2/spkg/logs$ grep f90 numpy1.5.0.log  grep bin drkirkby@hawk:~/new/sage4.6.alpha2/spkg/logs$ grep f95 numpy1.5.0.log  grep bin drkirkby@hawk:~/new/sage4.6.alpha2/spkg/logs$
It does not appear to have anything that's obviously using a Fortran 90 target, though it does compile files with the extension .f90.
I am completely astonished that you had problems with scipy and not numpy. In fact on Gentoo we currently have a bug open because numpy picks up the intel fortran compiler when it shouldn't.
Have you reported that bug upstream to the Numpy developers? If so, do you have the link handy? Do you have a link to the Gentoo bug? You could try exporting F77, F90 and F95 in addition to FC and see if that fixes it. It would be worth opening a bug report in Sage if this is affecting Sage too, which I expect it will. I could try creating a file 'icc' on my Sun and seeing if Numpy uses that, though since the Intel compiler does not run on Solaris, there's less chance of that being an issue. Perhaps someone with Linux can create a file icc
that simply exits with 1.
#!/bin/sh exit 1
and see if that screws up building Numpy. If so, it's a bug.
In fact, given the issues with Scipy and scipy_sandbox, it might be wise we export all these variables in Numpy.
What puzzles me most is why it is now necessary do this in scipy_sandbox, despite there have been no changes to the scipy_sandbox code at all! But checking in the sage4.6.1.alpha1 code, I see the f90 target is using sage_fortran
and not /usr/bin/f90
.
drkirkby@hawk:~/sage4.6.alpha1/spkg/logs$ grep f90 scipy_sandbox20071020.p5.log Fortran f90 compiler: sage_fortran Wall fnosecondunderscore fPIC O3 funrollloops Fortran f90 compiler: sage_fortran Wall fnosecondunderscore fPIC O3 funrollloops
Hence this Numpy or SciPy? upgrade has needed some rather surprising changes to be made to scipy_sandbox.
I know when you build gmp, mpfr will be build with the same compiler, as the location of the compiler is actually stored in the gmp header files. I thought Numpy/ScipPy? might use some similar technique, but it appears not.
comment:230 followup: ↓ 231 Changed 11 years ago by
The problem with ifc is actually more subtle than that http://bugs.gentoo.org/show_bug.cgi?id=336077 . someone has ifc installed but no license for whatever reason. Even so FC is set to gfortran, ifc will be tested if found. Without a license ifc wants to create a number of folders in the system, because the build system is sandboxed (so that no live files are touched will we are building the software) ifc throws an error with stops the build.
While sage doesn't use a sandbox the build could stop if ifc has been installed as root and a regular user is building numpy.
That's something to remember, even if a fortran compiler is selected, numpy/scipy will test for all the compilers they know off and try the one they found.
As for scipy, there were notes in the old spkg that scipy uses numpy distutils to build, that may have changed slightly in scipy0.8.0 as it wasn't necessary to set all these compilers before at least in sage. A quick look at the corresponding Gentoo ebuilds show that we have been setting FC, F77 and F90 (but not F95) for some time in scipy (OK I didn't go in the CVS attic to checkout 0.7.0 in particular, just 0.8.0 and 0.7.2).
comment:231 in reply to: ↑ 230 Changed 11 years ago by
Replying to fbissey:
The problem with ifc is actually more subtle than that http://bugs.gentoo.org/show_bug.cgi?id=336077 . someone has ifc installed but no license for whatever reason.
Probably a trial which has run out. I've had a few of them!
Even so FC is set to gfortran, ifc will be tested if found. Without a license ifc wants to create a number of folders in the system, because the build system is sandboxed (so that no live files are touched will we are building the software) ifc throws an error with stops the build.
While sage doesn't use a sandbox the build could stop if ifc has been installed as root and a regular user is building numpy.
OK, thank you. I think we need to check that. If Numpy is not using the compiler we specify, that must be a bug IMHO.
That's something to remember, even if a fortran compiler is selected, numpy/scipy will test for all the compilers they know off and try the one they found.
It seems a bit odd, given we have specified one.
As for scipy, there were notes in the old spkg that scipy uses numpy distutils to build, that may have changed slightly in scipy0.8.0 as it wasn't necessary to set all these compilers before at least in sage. A quick look at the corresponding Gentoo ebuilds show that we have been setting FC, F77 and F90 (but not F95) for some time in scipy (OK I didn't go in the CVS attic to checkout 0.7.0 in particular, just 0.8.0 and 0.7.2).
Setting F95 was a precaution.
Any chance of you taking a look at #10092, which sets these Fortran variable in scipy_sandbox? That should be quite easy to review and would be one step closer to getting this more complex ticket reviewed.
Dave
comment:232 followup: ↓ 233 Changed 11 years ago by
I've upgraded to the new ones on the Mac PPC, and the only problem I get in sage/matrix and sage/functions (other than the ones the patch fixes) is
sage t long "devel/sage/sage/functions/hyperbolic.py" Warning: divide by zero encountered in divide Warning: divide by zero encountered in divide
Any ideas? Looks like some other stuff reported here. I don't know that it would show up in doctesting.
comment:233 in reply to: ↑ 232 Changed 11 years ago by
Replying to kcrisman:
I've upgraded to the new ones on the Mac PPC, and the only problem I get in sage/matrix and sage/functions (other than the ones the patch fixes) is
sage t long "devel/sage/sage/functions/hyperbolic.py" Warning: divide by zero encountered in divide Warning: divide by zero encountered in divideAny ideas? Looks like some other stuff reported here. I don't know that it would show up in doctesting.
I don't get that on linux ppc, although I had a similar warning in sage/calculus/interpolators.pyx on one of my x86 box with sageongentoo, I don't remember it showing on any other box, and the test in question has had a failure with sageongentoo on that box only for a long time. A bit of a mistery http://github.com/cschwan/sageongentoo/issues#issue/6 .
I will review #10092 shortly, should be quick.
comment:234 followups: ↓ 235 ↓ 236 Changed 11 years ago by
How close is this ticket being to positively reviewed? I have a trial 4.6.alpha3 that's ready to release, except for #10097, which I expect we'll fix within a day. I'll release 4.6.alpha3 after that and put 4.6 in feature freeze.
comment:235 in reply to: ↑ 234 Changed 11 years ago by
Replying to mpatel:
How close is this ticket being to positively reviewed? I have a trial 4.6.alpha3 that's ready to release, except for #10097, which I expect we'll fix within a day. I'll release 4.6.alpha3 after that and put 4.6 in feature freeze.
I must say that with all that's happened to the ticket in the last few days, I and possibly maldun had lost hope of it making the deadline for the feature freeze.
I am about to review positively #10092 which would need merging at the same time. I am happy with the spkges as they are for linuxppc, so it is positive for me on that platform. I don't know about the other.
comment:236 in reply to: ↑ 234 Changed 11 years ago by
Replying to mpatel:
How close is this ticket being to positively reviewed? I have a trial 4.6.alpha3 that's ready to release, except for #10097, which I expect we'll fix within a day. I'll release 4.6.alpha3 after that and put 4.6 in feature freeze.
#10092 has been positively reviewed. That has no dependencies, so that can go in now.
These SciPy? and Numpy patches look ok to me, but John had an issue on OS X.
I think this must be very close to a positive review.
Dave
comment:237 followup: ↓ 238 Changed 11 years ago by
I get the same error as John on OS X 10.6.
comment:238 in reply to: ↑ 237 Changed 11 years ago by
Replying to kcrisman:
I get the same error as John on OS X 10.6.
In which case I think we can forget this going into this release. Numpy 1.5.1 should be released by the end of the month, so I guess we will have to wait for that.
#10092 will be needed for sure, and that is very safe and has positive review, so it would seem sensible to merge that if possible.
Dave
comment:239 Changed 11 years ago by
 Description modified (diff)
comment:240 followup: ↓ 242 Changed 11 years ago by
This seems related to m 64
and arch ppc64
 see for instance here for something like this in Boost (which it seems they solved by just removing that option! maybe no one is building Sage on PPC in 64bit mode anyway?). This discussion also seems to have relevant information about those two options; see especially 'On Mon, 7 Sep 2009'.
Another unrelated thread says 'the fact that the gcc compiler now creates x86_64
code by default so we are likely missing some instances of m32
which weren't required' in a similar context, though obviously that's not exactly what's going on here given the error messages  just that 32/64 bit and intel/ppc has something to do with it. Maybe we are missing some Mac flags in the new Scipy?
comment:241 Changed 11 years ago by
 Description modified (diff)
comment:242 in reply to: ↑ 240 ; followup: ↓ 243 Changed 11 years ago by
Replying to kcrisman:
This seems related to
m 64
andarch ppc64
 see for instance here for something like this in Boost (which it seems they solved by just removing that option! maybe no one is building Sage on PPC in 64bit mode anyway?). This discussion also seems to have relevant information about those two options; see especially 'On Mon, 7 Sep 2009'.Another unrelated thread says 'the fact that the gcc compiler now creates
x86_64
code by default so we are likely missing some instances ofm32
which weren't required' in a similar context, though obviously that's not exactly what's going on here given the error messages  just that 32/64 bit and intel/ppc has something to do with it. Maybe we are missing some Mac flags in the new Scipy?
May be. Could we have a little bit more of the log of the failure please? It could help to know what succeeded before the failure.
comment:243 in reply to: ↑ 242 Changed 11 years ago by
May be. Could we have a little bit more of the log of the failure please? It could help to know what succeeded before the failure.
Sorry, that terminal is now gone. Everything looks like jhpalmieri's  once Scipy is doing
compiling Fortran sources
the problem starts precisely there, so everything up to wherever that starts for dfftpack works.
comment:244 Changed 11 years ago by
May be. Could we have a little bit more of the log of the failure please? It could help to know what succeeded before the failure.
I've had the failure on two different OS X machines. I've posted the full scipy log here.
comment:245 Changed 11 years ago by
Just to check, I also built the older packages  the ones uploaded by maldun to code.google.com  and got the same error with scipy. So it doesn't seem to be because of the recent changes to those packages.
comment:246 Changed 11 years ago by
Here is something relevant in the latest Scipy  is it in previous version (in INSTALL.TXT)? This might obviate the need to specify all those fortran compiler options...
You can specify which Fortran compiler to use by using the following install command:: python setup.py config_fc fcompiler=<Vendor> install To see a valid list of <Vendor> names, run:: python setup.py config_fc helpfcompiler
Is there somewhere in Scipy's makefile (or whatever) that specifies the architecture stuff for the superuniversal binary? I don't think that other Sage spkgs have the ppc64 arch, and I don't see anything in spkginstall
that would do this. Anyway, the thing about not finding certain fortran compilers in John's log looks suspicious, perhaps.
comment:247 Changed 11 years ago by
I don't know the significance of this, but from the scipy0.7.5.log (the old version, which builds just fine):
copying scipy/weave/weave_version.py > build/lib.macosx10.6i3862.6/scipy/weave running build_clib customize UnixCCompiler customize UnixCCompiler using build_clib customize Sage_FCompiler_1 i686appledarwin8gfortran4.2: no input files i686appledarwin8gfortran4.2: no input files i686appledarwin8gfortran4.2: unrecognized option 'shared' i686appledarwin8gfortran4.2: no input files i686appledarwin8gfortran4.2: unrecognized option 'shared' i686appledarwin8gfortran4.2: no input files customize Sage_FCompiler_1 i686appledarwin8gfortran4.2: no input files i686appledarwin8gfortran4.2: no input files i686appledarwin8gfortran4.2: unrecognized option 'shared' i686appledarwin8gfortran4.2: no input files i686appledarwin8gfortran4.2: unrecognized option 'shared' i686appledarwin8gfortran4.2: no input files customize Sage_FCompiler_1 using build_clib building 'dfftpack' library compiling Fortran sources Fortran f77 compiler: sage_fortran Wall ffixedform fnosecondunderscore fPIC O3 funrollloops Fortran f90 compiler: sage_fortran Wall fnosecondunderscore fPIC O3 funrollloops Fortran fix compiler: sage_fortran Wall ffixedform fnosecondunderscore Wall fnosecondunderscore fPIC O3 funrollloops
From the failed scipy0.8.log:
copying scipy/weave/weave_version.py > build/lib.macosx10.6i3862.6/scipy/weave running build_clib customize UnixCCompiler customize UnixCCompiler using build_clib customize NAGFCompiler Found executable /Applications/sage_builds/numpy/sage4.6.alpha2/local/bin/sage_fortran Found executable /usr/bin/ar Found executable /usr/bin/ranlib customize AbsoftFCompiler customize IBMFCompiler Could not locate executable xlf95 customize IntelFCompiler customize GnuFCompiler gnu: no Fortran 90 compiler found Found executable /usr/bin/ld gnu: no Fortran 90 compiler found customize Gnu95FCompiler customize Gnu95FCompiler customize Gnu95FCompiler using build_clib building 'dfftpack' library compiling Fortran sources Fortran f77 compiler: /Applications/sage_builds/numpy/sage4.6.alpha2/local/bin/sage_fortran Wall ffixedform fnosecondunderscore arch ppc arch x86_64 arch ppc64 fPIC O3 funrollloops Fortran f90 compiler: /Applications/sage_builds/numpy/sage4.6.alpha2/local/bin/sage_fortran Wall fnosecondunderscore arch ppc arch x86_64 arch ppc64 fPIC O3 funrollloops Fortran fix compiler: /Applications/sage_builds/numpy/sage4.6.alpha2/local/bin/sage_fortran Wall ffixedform fnosecondunderscore Wall fnosecondunderscore arch ppc arch x86_64 arch ppc64 fPIC O3 funrollloops
Note that there are now lots of extra arguments to the fortran compiler.
comment:248 Changed 11 years ago by
This is what I was alluding to  they are now trying to build a universal binary for Mac, 32 and 64 bit, Intel and PPC. Notice that they also got rid of the shared option and probably replaced it by the dynamic lookup option or whatever works for Mac. Somehow our fortran compiler thing is not doing it right  maybe because we got rid of the whole Sage_F_compiler
thingie?
But I have no idea where this would be in the Scipy source; I'm learning on the job. I do think that getting rid of wherever the arch ppc64
is in there has a high probability of clearing up the issue, though in a hackish way (and this is what Boost did in the threads I link to in Comment 240).
comment:249 followup: ↓ 250 Changed 11 years ago by
I think the best thing is we ask on the SciPy mailing list. I assume there is one  there is one for Numpy. None of us appear to know how to fix this properly,
I've got no idea if is related to getting rid of the Sage_F_compiler thingie, but that sure is a possibility.
I'm going to try to find a suitable list, subscribe to it, then ask.
Dave
comment:250 in reply to: ↑ 249 ; followup: ↓ 253 Changed 11 years ago by
Replying to drkirkby:
I think the best thing is we ask on the SciPy mailing list. I assume there is one  there is one for Numpy. None of us appear to know how to fix this properly,
I've got no idea if is related to getting rid of the Sage_F_compiler thingie, but that sure is a possibility.
I'm going to try to find a suitable list, subscribe to it, then ask.
comment:251 Changed 11 years ago by
comment:252 Changed 11 years ago by
Almost certainly so.
comment:253 in reply to: ↑ 250 Changed 11 years ago by
Replying to jason:
Replying to drkirkby:
I think the best thing is we ask on the SciPy mailing list. I assume there is one  there is one for Numpy. None of us appear to know how to fix this properly,
I've got no idea if is related to getting rid of the Sage_F_compiler thingie, but that sure is a possibility.
I'm going to try to find a suitable list, subscribe to it, then ask.
I've subscribed to scipyuserATscipy.org and made a post with the title Problems building SciPy? on OS X due to ppc64 issues I posted a link to John's build log.
Robert Kern has replied to my post:
Are the multiple "c" options causing issues? From the build log, it looks like "c" is being added explicitly somewhere. compile options: 'I/Applications/sage_builds/numpy/sage4.6.alpha2/local/lib/python2.6/sitepackages/numpy/core/include c' Exactly where did the gfortran compiler come from? What version is it? What architecture is the machine doing the build?
It would help if someone who is experiencing the problem could subscribe to scipyuser, as otherwise I'm just a middleman!
Dave
comment:254 followup: ↓ 255 Changed 11 years ago by
There's a potentially helpful comment on the SciPy list from Ralf Gommers. It concerns boost again. It suggests we are trying to build ppc on a system which does not support PPC. In which case, removing that option for ppc is a necessarily and not a hack.
I'm puzzled how this option gets added now though  where does it come from?
Some googling turns up this which seems related to your issue:
You are using the 10.6 SDK and gcc 4.2. In the 10.6 SDK the ppc64 architecture is not supported anymore, you want to use 10.4 or 10.5 SDK. Since Python is built with gcc 4.0 you want to do the same if you want C++ support for ppc64 (which you'll need for scipy.sparsetools).
The above may be irrelevant for you though, since Sage is distributing binaries for each OS X version separately, right? 10.6 doesn't install on ppc64 machines, so no need to build that arch at all.
Dave
comment:255 in reply to: ↑ 254 ; followup: ↓ 256 Changed 11 years ago by
Replying to drkirkby:
There's a potentially helpful comment on the SciPy list from Ralf Gommers. It concerns boost again. It suggests we are trying to build ppc on a system which does not support PPC. In which case, removing that option for ppc is a necessarily and not a hack.
I'm puzzled how this option gets added now though  where does it come from?
That's what I'm trying to figure out. Usually it gets added in the spkginstall, but apparently that's not the case here  it's coming from Scipy itself, here (first noted by John).
You are using the 10.6 SDK and gcc 4.2. In the 10.6 SDK the ppc64 architecture is not supported anymore, you want to use 10.4 or 10.5 SDK. Since Python is built with gcc 4.0 you want to do the same if you want C++ support for ppc64 (which you'll need for scipy.sparsetools).
Well, not exactly. However, we always tell people that Mac Sage versions shouldn't be expected to work on older versions of Mac than they are created on. So what's going on here is that we'll have to likely add something to the skpginstall or to the numpy distutils (whatever that is) that removes the ppc64 option (perhaps both ppc options?) from the arch
flags in order to get around this only in the case that we in fact have OS X 10.6
sage: os.uname() ('Darwin', 'newhost2.home', '10.4.0', 'Darwin Kernel Version 10.4.0: Fri Apr 23 18:28:53 PDT 2010; root:xnu1504.7.4~1/RELEASE_I386', 'i386')
where in particular os.uname()[2][:2]==10
might be the thing to look at (this is OS X 10.6, confusingly) for checking for this, otherwise allowing a multiple architecture build.
The above may be irrelevant for you though, since Sage is distributing binaries for each OS X version separately, right? 10.6 doesn't install on ppc64 machines, so no need to build that arch at all.
Yeah, but apparently we have to ask Scipy specifically not to build on it, or perhaps ask Numpy not to  otherwise it's already asked for by Numpy/Scipy?, which seems confusing.
Incidentally, this exact boost thread is one of the ones I pointed out earlier, I think ;)
comment:256 in reply to: ↑ 255 ; followup: ↓ 257 Changed 11 years ago by
Replying to kcrisman:
Replying to drkirkby:
There's a potentially helpful comment on the SciPy list from Ralf Gommers. It concerns boost again. It suggests we are trying to build ppc on a system which does not support PPC. In which case, removing that option for ppc is a necessarily and not a hack.
I'm puzzled how this option gets added now though  where does it come from?
That's what I'm trying to figure out. Usually it gets added in the spkginstall, but apparently that's not the case here  it's coming from Scipy itself, here (first noted by John).
If you grep through the scipy directory, I don't think you'll find "ppc64" anywhere. I think it's coming from numpy, and in particular the file src/numpy/distutils/fcompiler/gnu.py. If you make the change

gnu.py
old new 254 254 if not sys.platform == 'darwin': 255 255 return [] 256 256 arch_flags = [] 257 for arch in ["ppc", "i686", "x86_64" , "ppc64"]:257 for arch in ["ppc", "i686", "x86_64"]: 258 258 if _can_target(cmd, arch): 259 259 arch_flags.extend(["arch", arch]) 260 260 return arch_flags
to numpy and install it, then afterwards scipy seems to install correctly. I don't have time this weekend to create a new spkg which incorporates this patch, or even to figure out under what circumstances to do it (do we have to detect DARWIN + x86? should we get rid of "ppc" also?). If someone else wants to make a new spkg, that would be fine with me.
comment:257 in reply to: ↑ 256 Changed 11 years ago by
That's what I'm trying to figure out. Usually it gets added in the spkginstall, but apparently that's not the case here  it's coming from Scipy itself, here (first noted by John).
If you grep through the scipy directory, I don't think you'll find "ppc64" anywhere. I think it's coming from numpy, and in particular the file src/numpy/distutils/fcompiler/gnu.py. If you make the change
Yes, you're right  I realized that later last night, but had been confused by the scipy.org address. And this is exactly the change mentioned in all these links being bandied about.
gnu.py
old new 254 254 if not sys.platform == 'darwin': 255 255 return [] 256 256 arch_flags = [] 257 for arch in ["ppc", "i686", "x86_64" , "ppc64"]:257 for arch in ["ppc", "i686", "x86_64"]: 258 258 if _can_target(cmd, arch): 259 259 arch_flags.extend(["arch", arch]) 260 260 return arch_flags to numpy and install it, then afterwards scipy seems to install correctly. I don't have time this weekend to create a new spkg which incorporates this patch, or even to figure out under what circumstances to do it (do we have to detect DARWIN + x86? should we get rid of "ppc" also?). If someone else wants to make a new spkg, that would be fine with me.
I can't do that this weekend, but might be able to next week. I think that detecting os.uname()[2][:2]==10
(or however that works in shell script) would be sufficient, since it seems to be a 10.6 problem with Python (namely, that the Python we use wouldn't support ppc64 or whatever with 10.6  which makes sense, since 10.6 doesn't work on ppc). It probably wouldn't hurt to remove ppc as well in that situation, but I think that Python/Apple? haven't removed that as much yet since they still want to support universal binaries on PPC Leopard (10.5).
comment:258 Changed 11 years ago by
As requested I updated the doctest changes in the patch.
I tested all recent numpy and scipy versions on ubuntu 10.04, and they built without problems on sage4.6.alpha3 and all doctests passed with the updated patch.
Hope that all bugs in numpy are killed soon. I think the whole ticket here contributed quite a bit to numpy and scipy.
comment:259 Changed 11 years ago by
Okay, I'm trying to look at this numpy package, and have some questions.
First, there are all these special instructions in the SPKG.txt that seem to be totally irrelevant now about the sage_fcompiler
etc. for gfortran and so forth. Is all that unnecessary now that we exported F*, or not? I assume drkirkby would be the most knowledgeable about this. I know we tested this a lot, but it still worries me, since I know little about such things.
Second, I think that jhpalmieri's spkg should be 1.5.0.p0, correct? And that should also be in SPKG.txt.
Next, toward the end it should be SciPy
not SciPY
.
Also, 11 hours ago someone pointed on on the Numpy trac that this might solve our issue more appropriately. I was going to try to make a numpy spkg based on the other fix, but this might be better  I have to look at it and test it. Anyway, an update.
comment:260 followup: ↓ 261 Changed 11 years ago by
First, there are all these special instructions ...
I don't know about this. I hope Dave does.
Second, I think that jhpalmieri's spkg should be 1.5.0.p0, correct? And that should also be in SPKG.txt.
I think that since it's not a vanilla 1.5.0, making it p0 is fine. Or because I patched someone else's (already patched) 1.5.0, it could be p0. Anyway, it's fine with me. (I seem to remember seeing someone express the opinion that if it's the first time we've released a package for that particular version, then we don't need a patch number, but I don't really care one way or the other.)
Next, toward the end it should be SciPy? not SciPY.
Okay.
Finally, if you want to try a different fix, I'm happy to test it out. Thanks for paying attention to the Numpy trac ticket; that's more than I was doing...
comment:261 in reply to: ↑ 260 Changed 11 years ago by
Replying to jhpalmieri:
First, there are all these special instructions ...
I don't know about this. I hope Dave does.
I think I know about them, they should be mostly removed. My fault for pushing some of the changes obsoleting these to maldun without updating the instructions at the same time.
comment:262 Changed 11 years ago by
Finally, if you want to try a different fix, I'm happy to test it out. Thanks for paying attention to the Numpy trac ticket; that's more than I was doing...
I'm trying it right now, but it's slow going because I know nothing about shell. For instance, I just learned that diff
has three different possible options, and the third one was the one that Sage patches look like  this took a half hour :(
I am trying to use uname r
to check for the Darwin version, but I don't know what would happen on (say) Solaris with this command. David, portability ideas? According to this it seems okay.
comment:263 Changed 11 years ago by
Also, because unix tutorials are bad for true beginners, I have no idea how to do something that in pseudoPython would be
output = `uname =r` if output[:2] == 10: do ...
in shell script. How do I take the output of uname r
and just get the first two characters (which would be 10 for OS X 10.6)? Maddening  and of course it's impossible to figure out how to get this from the tutorials online without reading for hours. Sorry  hopefully tomorrow I'll be able to get that figured out and test that patch, which depends on checking the CFLAGS
passed to the fcompiler/gnu.py script.
comment:264 followup: ↓ 266 Changed 11 years ago by
Better use plain 'uname' as it should return Darwin for you if I am not mistaken. I am hoping that plain uname is portable. http://www.opengroup.org/onlinepubs/009695399/utilities/uname.html It would be much easier. There is already a bunch of line for OSX:
if [ `uname` = "Darwin" ]; then unset ATLAS unset BLAS unset LAPACK else export LDFLAGS="${LDFLAGS} shared" fi
comment:265 followup: ↓ 269 Changed 11 years ago by
So I looked at the special build instructions and right now they should be amended to
Special Update/Build Instructions: * Scipy uses numpy's distutils to control its compilation of fortran code. Whenever numpy is updated it is necessary to make sure that scipy still builds ok.
And that's it. The rest is obsolete. I should review what we have done to make sure there are no other  new instructions that should be given.
comment:266 in reply to: ↑ 264 Changed 11 years ago by
Replying to fbissey:
Better use plain 'uname' as it should return Darwin for you if I am not mistaken.
No, the point is you want to test not only for Darwin, but for Darwin version 10.6, as opposed to 10.5 or 10.4. uname r
should return strings like 10.4.0, 9.3.1, and 8.8.0, respectively, for these (I think). (The last parts of the string, like 4.0 or 3.1 or 8.0, are the minor version numbers, which I don't think we care about.)
This seems to work for me, but I'm not a sed expert:
VER=`uname r  sed 's/\([09]*\)\..*/\1/'`
(This takes the output from uname r, sends it to sed, which does a regular expression match to return the digits found before the first period.) Then you do something like
if [ $VER ge 10 ]; then ... fi
(Might as well test whether VER is at least 10, rather than equal to 10 on the nose.)
Actually, do we know if their patch needs us to test for the Darwin version? Do you have access to a 10.4 machine to test on?
comment:267 Changed 11 years ago by
A few points.
 I'm going to be very busy over the next week, and particularly today, so I don't have a lot of time to put into this.
 I believe the pacakge should be called 1.5.0 and not 1.5.0.p0  this was discussed some time ago on sagedevel.
 I'm not sure that 'sed' command is doing what John wants. Instead of using 'uname' for test purposes, use 'echo'.
drkirkby@hawk:~$ echo 10.1.4  sed 's/\([09]*\)\..*/\1/' 10 drkirkby@hawk:~$ echo 10.6.0  sed 's/\([09]*\)\..*/\1/' 10 drkirkby@hawk:~$ echo 10.6.0  sed 's/\([09]*\)\..*/\1/' 10 drkirkby@hawk:~$ echo 10.5.1  sed 's/\([09]*\)\..*/\1/' 10 drkirkby@hawk:~$ echo 9.5.1  sed 's/\([09]*\)\..*/\1/' 9 drkirkby@hawk:~$ echo 10.4.1  sed 's/\([09]*\)\..*/\1/' 10
That looks to be taking only the major part, and so can't distinguish from 10.5 or 10.6, which I think is what is said is needed. If that's what's needed, my approach would be
 Simulate the output of 'uname r' (which is portable, so can be used on any platform, not just OS X), using 'echo'. That way you don't need to find different systems to test on.
 Make sure the OS is OS X. My HP C3600 runs HPUX 11.11 and I expect (though it's off so I can't check), that 'uname r' might print 11.11 which would not be any use here. So put this all in 'if' statemant that checks for OS X first.
 If you consider the output version to be x.y.z, then I'd do something like this, where I split the x.y.z into individual parts, by first replacing the dots with a space, then printing the 1st, 2nd or 3rd part of the expressions using 'awk'.
drkirkby@hawk:~$ echo 10.4.1 10.4.1 drkirkby@hawk:~$ echo 10.4.1  sed 's/\./ /g' 10 4 1 drkirkby@hawk:~$ echo 10.4.1  sed 's/\./ /g'  awk '{print $1}' 10 drkirkby@hawk:~$ echo 10.4.1  sed 's/\./ /g'  awk '{print $2}' 4 drkirkby@hawk:~$ echo 10.4.1  sed 's/\./ /g'  awk '{print $3}' 1 drkirkby@hawk:~$
comment:268 Changed 11 years ago by
Great, all this should hopefully work out fine. Most of what is said above I already had been checking anyway, but having the syntax is nice.
if [ `uname` = "Darwin"]; then VER=`uname r  sed 's/\([09]*\)\..*/\1/'` if [ $VER ge 10 ]; then cp ../patches/gnuchanges.py numpy/distutils/fcompiler/gnu.py fi
because uname r
being 10.x.y means OS X 10.6. It is VERY confusing, but that's how it is (as John points out). So John's sed command should work.
I also updated the SPKG.txt as Francois indicated. I should have an spkg ready to test sometime today with all this changed  assuming it works, that is.
The patch doesn't require us to test for the Darwin version, and I do have a 10.4 machine to test it on (though it takes ages to build Scipy). I guess I could just patch it completely. I just would rather only apply the patch if absolutely necessary, since although the code makes sense, I don't know what will happen until I try it (which, as said, will happen later today). What they do is get the arch
flags from the ambient CFLAGS
and I don't actually know what the ambient CFLAGS
will be  presumably from elsewhere in Python or whatever, but who knows?
Anyway, if we only apply it on Mac Snow Leopard, that will also make it maximally easy to review that part of the change; I haven't made any other changes to spkginstall.
Well, now it's time to sagepkg it up, and hope for the best.
comment:269 in reply to: ↑ 265 ; followup: ↓ 271 Changed 11 years ago by
Replying to fbissey:
So I looked at the special build instructions and right now they should be amended to
Special Update/Build Instructions: * Scipy uses numpy's distutils to control its compilation of fortran code. Whenever numpy is updated it is necessary to make sure that scipy still builds ok.And that's it. The rest is obsolete. I should review what we have done to make sure there are no other  new instructions that should be given.
That includes the business about
The file $SAGE_ROOT/devel/sage/sage/ext/numpy.pxi comes from this file
and so on? I just want to confirm this is also obsolete.
comment:270 Changed 11 years ago by
Doesn't work, same error. That doesn't mean it wouldn't be worth keeping, because of the other things mentioned in this thread and the scipy/boost lists, so I am keeping this in my current attempt at an spkg.
But I think that this means the only other possible problem is the
Are the multiple "c" options causing issues? From the build log, it looks like "c" is being added explicitly somewhere.
issue. From the gcc docs:
c Compile or assemble the source files, but do not link. The linking stage simply is not done. The ultimate output is in the form of an object file for each source file.
So I guess doing that twice WOULD cause the exact error message we are seeing. Maybe once we fix that, we would still have to fix the ppc64 issue, so (as I say above) I'll keep that in for now.
compile options: 'I/Applications/sage_builds/numpy/sage4.6.alpha2/local/lib/python2.6/sitepackages/numpy/core/include c'
And then the usual c
is added. So where does this one come from? In the Numpy install I see
C compiler: gcc fnostrictaliasing g O2 DNDEBUG g fwrapv O3 Wall Wstrictprototypes compile options: 'Inumpy/core/src/private Inumpy/core/src Inumpy/core Inumpy/core/src/npymath Inumpy/core/src/multiarray Inumpy/core/src/umath Inumpy/core/include I/Users/karldietercrisman/Downloads/sage4.6.alpha3/local/include/python2.6 c'
so at least sometimes it's okay to have c
in the compile options. But I haven't got a clue where that string is set.
Also, there were two syntax errors in my initial change  here is the corrected one:
if [ `uname` = "Darwin" ]; then VER=`uname r  sed 's/\([09]*\)\..*/\1/'` if [ $VER ge 10 ]; then cp ../patches/gnuchanges.py numpy/distutils/fcompiler/gnu.py fi fi
comment:271 in reply to: ↑ 269 ; followup: ↓ 278 Changed 11 years ago by
Replying to kcrisman:
Replying to fbissey:
So I looked at the special build instructions and right now they should be amended to
Special Update/Build Instructions: * Scipy uses numpy's distutils to control its compilation of fortran code. Whenever numpy is updated it is necessary to make sure that scipy still builds ok.And that's it. The rest is obsolete. I should review what we have done to make sure there are no other  new instructions that should be given.
That includes the business about
The file $SAGE_ROOT/devel/sage/sage/ext/numpy.pxi comes from this fileand so on? I just want to confirm this is also obsolete.
It was obsolete before we started working with the spkg. It should have been removed in 1.3.1 if not before.
comment:272 Changed 11 years ago by
You are OK with the bash shell, but it is very unportable to have no space before the left bracket. That will break with many shells, so is not a good habit to get into.
See example below:
bash3.00$ cat bad #!/bin/sh if [ `uname` = "Darwin"]; then echo "This is OS X" fi bash3.00$ ./bad ./bad: test: ] missing
Now just add a space:
bash3.00$ cat good #!/bin/sh if [ `uname` = "Darwin" ]; then echo "This is OS X" fi bash3.00$ ./good bash3.00$
Since this was a Solaris system, not OS X, good
did not print it was OS X.
bash3.00$ uname SunOS
Just like Python, different people have different styles, and some are more portable than others. But that lack of a space is particular troublesome.
Dave
comment:273 Changed 11 years ago by
Oops, I see you had realised this. Also I note it fails with bash too.
Oh well, I'll shut up!!!
dave
comment:274 Changed 11 years ago by
No problem :)
But if you know where that extra "c" comes from, please pipe up! I'm stumped, but then again I know nothing of where such things are originated  presumably there is a typical file where they come from.
comment:275 followup: ↓ 276 Changed 11 years ago by
 Description modified (diff)
 Status changed from needs_work to needs_review
Okay, I went back to John's initial idea and just removed the ppc64 option. I'm still totally mystified as to why that would solve it per se, if the double c
option is really a problem, but oh well  ppc64 is the problem.
So I have now changed the description yet again to point us to a numpy spkg at my sage account. Try it out. I will now also attach diffs for reference about this. Notice that I only attach the changed gnu.py file on Snow Leopard, so this should presumably only change anything on there.
comment:276 in reply to: ↑ 275 ; followup: ↓ 277 Changed 11 years ago by
Replying to kcrisman:
Okay, I went back to John's initial idea and just removed the ppc64 option. I'm still totally mystified as to why that would solve it per se, if the double
c
option is really a problem, but oh well  ppc64 is the problem.
There's no reason a double c should be a problem. There are numerous options that get passed to gcc multiple times. Here's a simple session where I list the files in my directory, and look for the most recent one.
drkirkby@hawk:~$ touch foobar drkirkby@hawk:~$ ls lrt  tail 1 rwrr 1 drkirkby other 8 Oct 14 10:37 foobar drkirkby@hawk:~$ gcc lm test.c drkirkby@hawk:~$ ls lrt  tail 1 rwxrxrx 1 drkirkby staff 8316 Oct 14 10:38 a.out drkirkby@hawk:~$ gcc lm c c test.c drkirkby@hawk:~$ ls lrt  tail 1 rwrr 1 drkirkby staff 1012 Oct 14 10:38 test.o drkirkby@hawk:~$
We can see that calling
gcc lm c c test.c
resulted in the most recent file being an object file (.o) and not an executable a.out.
The patch you attached is not a Mercurial patch, so could not easily be integrated.
comment:277 in reply to: ↑ 276 Changed 11 years ago by
There's no reason a double c should be a problem. There are numerous options that get passed to gcc multiple times. Here's a simple session where I list the files in my directory, and look for the most recent one.
This example just shows I am totally clueless. So that couldn't have caused the problem?
The patch you attached is not a Mercurial patch, so could not easily be integrated.
It wasn't intended to be integrated  it was just intended to be viewed. It is already part of the (correctly integrated, I think) patch in the actual spkg in the description, which is here.
Anyway, all long doctests pass with this on my 10.6 Snow Leopard computer. And the change to spkginstall are exactly the same, so in theory this is the only platform it should affect, meaning all other review should work. As I said, though, I don't feel comfortable giving it final positive review.
Will this ticket reach 300 comments?
comment:278 in reply to: ↑ 271 Changed 11 years ago by
Replying to fbissey:
Replying to kcrisman:
Replying to fbissey:
So I looked at the special build instructions and right now they should be amended to
Special Update/Build Instructions: * Scipy uses numpy's distutils to control its compilation of fortran code. Whenever numpy is updated it is necessary to make sure that scipy still builds ok.And that's it. The rest is obsolete. I should review what we have done to make sure there are no other  new instructions that should be given.
That includes the business about
The file $SAGE_ROOT/devel/sage/sage/ext/numpy.pxi comes from this fileand so on? I just want to confirm this is also obsolete.
It was obsolete before we started working with the spkg. It should have been removed in 1.3.1 if not before.
Sorry that's my fault. I had realized it, but then forgot to remove it...
comment:279 followups: ↓ 280 ↓ 282 Changed 11 years ago by
 Status changed from needs_review to needs_work
I built this on a unix box (eno), an OS X 10.6 box, an OpenSolaris box (hawk), and a Solaris on x86 box (fulvia). (By "built", I mean I built from scratch based on Sage 4.6.alpha3, replacing the numpy and scipy packages with the ones on this ticket, and I also applied the doctest patch.)
 On some machines, the doctest patch didn't apply cleanly: the patch to doc/en/faq/faqusage.rst didn't apply, because it seems to already have been applied. Maybe the tar file for 4.6.alpha3 changed? The patch didn't apply cleanly on the machines where I downloaded it more recently.
 On linux, all tests passed.
 On Mac and OpenSolaris, two doctest failures:
sage t long force_lib devel/sage/sage/matrix/matrix1.pyx ********************************************************************** File "/Applications/sage_builds/numpy/sage4.6.alpha3/devel/sagemain/sage/matrix/matrix1.pyx", line 448: sage: sorted(numpy.typecodes.items()) Expected: [('All', '?bhilqpBHILQPfdgFDGSUVOMm'), ('AllFloat', 'fdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Datetime', 'Mm'), ('Float', 'fdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')] Got: [('All', '?bhilqpBHILQPfdgFDGSUVO'), ('AllFloat', 'fdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Float', 'fdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')] **********************************************************************
andsage t long force_lib devel/sage/sage/misc/citation.pyx ********************************************************************** File "/Applications/sage_builds/numpy/sage4.6.alpha3/devel/sagemain/sage/misc/citation.pyx", line 60: sage: get_systems('integrate(x^2, x)') Expected: ['ginac', 'Maxima'] Got: ['numpy', 'ginac', 'Maxima'] ********************************************************************** File "/Applications/sage_builds/numpy/sage4.6.alpha3/devel/sagemain/sage/misc/citation.pyx", line 64: sage: get_systems('I.primary_decomposition()') Expected: ['Singular'] Got: ['Singular', 'numpy'] **********************************************************************
 the bad news: scipy doesn't build on fulvia. After listing all of the files extracted, this is basically all of the log file:
Finished extraction **************************************************** Host system uname a: SunOS fulvia 5.10 Generic_12712811 i86pc i386 i86pc **************************************************** **************************************************** CC Version gcc v Using builtin specs. COLLECT_GCC=gcc COLLECT_LTO_WRAPPER=/usr/local/gcc4.5.1/x86_64SunOScore2sunld/libexec/gcc/i386pcsolaris2.10\ /4.5.1/ltowrapper Target: i386pcsolaris2.10 Configured with: /usr/local/gcc4.5.1/src/gcc4.5.1/configure enablelanguages=c,c++,fortran w\ ithgnuas withas=/usr/local/binutils2.20.1/x86_64SunOScore2gcc4.4.3/bin/as withld=/usr\ /ccs/bin/ld withgmp=/usr/local/gmp5.0.1/x86_64SunOScore2gcc4.5.0abi32 withmpfr=/usr/lo\ cal/mpfr3.0.0/x86_64SunOScore2gmp5.0.1gcc4.5.0abi32 withmpc=/usr/local/mpc0.8.2/x86_64\ SunOScore2mpfr3.0.0gmp5.0.1gcc4.5.0abi32 prefix=/usr/local/gcc4.5.1/x86_64SunOScore2\ sunld Thread model: posix gcc version 4.5.1 (GCC) **************************************************** ./spkginstall: LDFLAGS=shared: is not an identifier
comment:280 in reply to: ↑ 279 ; followup: ↓ 281 Changed 11 years ago by
 On some machines, the doctest patch didn't apply cleanly: the patch to doc/en/faq/faqusage.rst didn't apply, because it seems to already have been applied. Maybe the tar file for 4.6.alpha3 changed? The patch didn't apply cleanly on the machines where I downloaded it more recently.
Yeah, I had that problem too at times.
 On Mac and OpenSolaris, two doctest failures:
Did those errors go away upon testing 'by hand'? Incidentally, I don't think I got these errors on my Mac (Snow Leopard, like yours).
 the bad news: scipy doesn't build on fulvia. After listing all of the files extracted, this is basically all of the log file:
Finished extraction **************************************************** Host system uname a: SunOS fulvia 5.10 Generic_12712811 i86pc i386 i86pc **************************************************** **************************************************** CC Version gcc v Using builtin specs. COLLECT_GCC=gcc COLLECT_LTO_WRAPPER=/usr/local/gcc4.5.1/x86_64SunOScore2sunld/libexec/gcc/i386pcsolaris2.10\ /4.5.1/ltowrapper Target: i386pcsolaris2.10 Configured with: /usr/local/gcc4.5.1/src/gcc4.5.1/configure enablelanguages=c,c++,fortran w\ ithgnuas withas=/usr/local/binutils2.20.1/x86_64SunOScore2gcc4.4.3/bin/as withld=/usr\ /ccs/bin/ld withgmp=/usr/local/gmp5.0.1/x86_64SunOScore2gcc4.5.0abi32 withmpfr=/usr/lo\ cal/mpfr3.0.0/x86_64SunOScore2gmp5.0.1gcc4.5.0abi32 withmpc=/usr/local/mpc0.8.2/x86_64\ SunOScore2mpfr3.0.0gmp5.0.1gcc4.5.0abi32 prefix=/usr/local/gcc4.5.1/x86_64SunOScore2\ sunld Thread model: posix gcc version 4.5.1 (GCC) **************************************************** ./spkginstall: LDFLAGS=shared: is not an identifier
Isn't this basically the same error that fulvia had before with this Scipy? So nothing has changed there. You suggested something about
export LDFLAGS="shared"
before, but this is now quoted in the spkginstall already, assuming you used the one on kirkby's account.
I assume fulvia is a machine which already has "official" support?
comment:281 in reply to: ↑ 280 Changed 11 years ago by
Replying to kcrisman:
 On Mac and OpenSolaris, two doctest failures:
Did those errors go away upon testing 'by hand'? Incidentally, I don't think I got these errors on my Mac (Snow Leopard, like yours).
On the Mac, the citation.pyx failure went away when done by hand, but not the matrix1.pyx failure. On OpenSolaris, both are repeatable by hand.
 the bad news: scipy doesn't build on fulvia. After listing all of the files extracted, this is basically all of the log file:
Isn't this basically the same error that fulvia had before with this Scipy? So nothing has changed there. You suggested something about
export LDFLAGS="shared"before, but this is now quoted in the spkginstall already, assuming you used the one on kirkby's account.
In fact, it was already there before I made the suggestion. I think the suggestion above about changing the first line of spkginstall from "/bin/sh" to "/usr/bin/env bash" might work, and I hope it won't break on any other platforms. I'll prepare a new spkg later today.
I assume fulvia is a machine which already has "official" support?
To the extent that this concept is defined, I would say yes. Sage should build on all of the skynet machines, and fulvia is one of those.
comment:282 in reply to: ↑ 279 Changed 11 years ago by
Replying to jhpalmieri:
I built this on a unix box (eno), an OS X 10.6 box, an OpenSolaris box (hawk), and a Solaris on x86 box (fulvia). (By "built", I mean I built from scratch based on Sage 4.6.alpha3, replacing the numpy and scipy packages with the ones on this ticket, and I also applied the doctest patch.)
 On some machines, the doctest patch didn't apply cleanly: the patch to doc/en/faq/faqusage.rst didn't apply, because it seems to already have been applied. Maybe the tar file for 4.6.alpha3 changed? The patch didn't apply cleanly on the machines where I downloaded it more recently.
Strange I tested the patch out, and it really didn't work. Then I made a new patch and it worked. So perhaps the file was corrupt. I upload a new version right now. Hope nothing gets corrupted again...
maldun
comment:283 Changed 11 years ago by
Ok if I download it again I have the same problem. It's really strange. If you look at the file sizes, the uploaded version has 2.7 kb (which the file what works also has), while the downloaded has 4.3 kb. Something strange happens here...
comment:284 Changed 11 years ago by
Did you download the raw or the html version of the patch?
comment:285 Changed 11 years ago by
The raw, and I'm already suspecting what's going on here: When I download from the wiki trac I get the old version for sage 4.5.3 instead of the new one... I had this problem frequently on several wikis now and it is starting to get on my nerves...
try this link out: http://computationalsage.googlecode.com/files/trac_9808_numpy_doctest_change.patch
It's my googlecode repository. Downloading from there works fine.
maldun
comment:286 Changed 11 years ago by
Ok better solution: new name no probs anymore =)
comment:287 Changed 11 years ago by
 Description modified (diff)
Updates: my doctest failure in citation.pyx is a "red herring". In particular, it's not because of the patches here; it's because of a flaw in how the function get_systems works. Often when I've been testing the packages and patches on this ticket, I've created a new version of sage in a directory called "numpy", and get_systems sees that string in the path name and adds "numpy" to the systems used by the particular command. If I rename the directory to something else, then the doctest passes. So let's not worry about this; at some point, someone can fix get_systems so it ignores the initial chunk of the path name (ignore the parent of SAGE_ROOT, for example).
My doctest failure in matrix1.pyx is still there. For some reason, the "Datetime" type codes are not there on some systems. Any ideas why? I glanced at the install log for numpy, but I didn't see anything suspicious, not that I really know what to look for. Here's the log from my OS X 10.6 machine, which is one place I see this problem: http://sage.math.washington.edu/home/palmieri/misc/numpy1.5.0.log.
Finally, I have a new version of the scipy spkg, which works for me on all of the machines I've tested on: linux, OpenSolaris, Solaris on x86, Mac OS X 10.6. I'm putting the link in the ticket description, and I'm posting the Mercurial patch  it's very small.
I'm leaving this as "needs work" because of the matrix1.pyx issue, but I think that's the only remaining problem. Please test the new scipy spkg on other systems to make sure, though.
I have now workable spkg files for the both packages. Scipy worked well. Numpy need some work, but it's functioning.
I host the files at megaupload, at: Numpy: http://www.megaupload.com/?d=6GCFZD65 Scipy: http://www.megaupload.com/?d=JORT40DK
I think the problem is the same as described in Trac # 7791 (http://trac.sagemath.org/sage_trac/ticket/7791 )
I get the following errors:
What's this? Has someone hardcoded the sizes of the routines?