Opened 12 years ago
Last modified 12 years ago
#9808 closed task
Upgrade numpy to 1.5.0 and scipy to 0.8 — at Version 197
Reported by:  Stefan Reiterer  Owned by:  Stefan Reiterer 

Priority:  major  Milestone:  sage4.6.1 
Component:  packages: standard  Keywords:  numpy, scipy 
Cc:  Merged in:  
Authors:  Stefan Reiterer, Francois Bissey  Reviewers:  KarlDieter Crisman, David Kirkby, Leif Leonhardy, Francois Bissey 
Report Upstream:  Fixed upstream, but not in a stable release.  Work issues:  
Branch:  Commit:  
Dependencies:  Stopgaps: 
Description (last modified by )
The packages can be found found under:
http://sage.math.washington.edu/home/palmieri/misc/numpy1.5.0.spkg http://code.google.com/p/spkgupload/downloads/detail?name=scipy0.8.spkg
(NumPy? 1.4.1 can also be found under http://code.google.com/p/spkgupload/downloads/detail?name=numpy1.4.1.spkg when needed)
Files are found with direct links:
http://spkgupload.googlecode.com/files/scipy0.8.spkg
Important notes: after installing numpy, one needs to execute sage ba, or else one get's runtime warnings.
trac_9808_numpy_doctest_change.patch in the attachment has to be applied, in order to get all doctests running because some of the output has changed.
For reviewers: changes.txt holds a summary of all changes with reference to the diffs, and links to other tickets
Change History (205)
comment:1 Changed 12 years ago by
Status:  new → needs_work 

comment:2 Changed 12 years ago by
Summary:  Upgrade numpy to 1.5b and scipy to 0.8 → Upgrade numpy to 1.5.0rc1 and scipy to 0.8 

numpy 1.5.0rc1: http://www.megaupload.com/?d=KK31RSZV
comment:3 followup: 4 Changed 12 years ago by
I don't think that we should upgrade to 1.5.0rc1  we should do 1.4.1 for now and wait until 1.5 is released.
comment:4 followup: 16 Changed 12 years ago by
Replying to mhansen:
I don't think that we should upgrade to 1.5.0rc1  we should do 1.4.1 for now and wait until 1.5 is released.
That has to be double checked but maldun says he needs features in 1.5. It may be that the features he wants are 1.4.x and he doesn't know. Do we know roughly when 1.5 will be released? They are at 1.5rc1 9 days after beta2 so it could very well be upon us in a very short time.
comment:5 Changed 12 years ago by
The warnings go away after rebuilding the source. And the doctests only throw warnings but there don't seem to be errors, and the output is correct
@mhansen, fbissey I think numpy has the same issues since version 1.4. and I'm quite sure that resolving it in 1.5.0rc1 would, solve the problem with 1.4.1. too. (and perhaps also with 1.5. later) So I suggest the following procedure: I solve the issues with the latest. Dobule check if 1.4.1 also works, and we decide then which of the versions we keep, or for example keep 1.4.1 in standard and 1.5.0rc1 as experimental
comment:6 followup: 7 Changed 12 years ago by
If 1.5.0rc1 is merged, #7166 can be closed. I don't know about 1.4.1, but in any case that is not a critical bug.
Some time back I made a Solarisspecific change to Numpy, as I wanted to get it reviewed with the least hassle  that generally means making it Solaris specific, as reviewers are easier to please if it only effects a rarer platform.
But I think the change should be implemented on OS X too. Currently there's a really nasty hack on OS X to build Numpy, that involves a script called fake_gcc
, copying that to $SAGE/LOCAL/bin/gcc
, then using the fake gcc to build Numpy. Finally this fake gcc file gets deleted.
The far neater option is the way I did it on Solaris. I suggest the changes I made for Solaris are implemented whenever SAGE64
is set to "yes", irrespective of whatever platform it may be. Then all this fake gcc rubbish can be removed.
If you want, I can create a patch.
comment:7 Changed 12 years ago by
Replying to drkirkby:
If you want, I can create a patch.
would be nice! But first I have to sort some things out, I hope it will get ready soon...
comment:8 followup: 9 Changed 12 years ago by
Looking at all that stuff in the spkg and comparing to Gentoo. Not very pretty. Do we really still need to use sage_fortran? On most platform we now use a native fortran compiler rather than a sage shipped one  I don't know if we still do that for OS X. In the patch to gnu.py there is:
+ # Note that sse flags and so on lead to gfortran code that segfaults, so disable arch flags + return opt +
An extract of the Gentoo set up:
export NUMPY_FCONFIG="config_fc noopt noarch"
Actually here is the full set up that you might find interesting:
# See progress in http://projects.scipy.org/scipy/numpy/ticket/573 # with the subtle difference that we don't want to break Darwin where # shared is not a valid linker argument if [[ ${CHOST} != *darwin* ]] ; then appendldflags shared fi # only one fortran to link with: # linking with cblas and lapack library will force # autodetecting and linking to all available fortran compilers use lapack  return [[ z ${FC} ]] && FC=$(tcgetFC) # when fortran flags are set, pic is removed. FFLAGS="${FFLAGS} fPIC" export NUMPY_FCONFIG="config_fc noopt noarch"
The other patches we have are relatively minor, a fix to the f2py man page, a patch for freebsd  that may be usefull, a patch for interix(!). I cannot comment on the cygwin patches but they look very small. We are are a bit more anal with site.cfg, I don't think it is useful in the context of sage  we just have more complex possible combinations of blas/lapack.
The NUMPY_FCONFIG is passed to distutils as an argument of
python setup.py install
i.e. in the end what we do boils down to "python setup.py install ${NUMPY_FCONFIG}".
Can you point me to your solaris fix Dave? I'd like to see if I can tidy all that up in something that requires less patching and is more based on FLAGS settings.
comment:9 Changed 12 years ago by
Replying to fbissey:
Looking at all that stuff in the spkg and comparing to Gentoo. Not very pretty.
What a surprise.
Do we really still need to use sage_fortran? On most platform we now use a native fortran compiler rather than a sage shipped one  I don't know if we still do that for OS X.
I've never understood the need for this SAGE_FORTRAN
. I've tried arguing for it to be removed and use FC
instead, but I had no luck.
A Fortran compiler is still shipped on OS X, but I don't see why the variable FC
can't be made to point to that, rather than have the variable SAGE_FORTRAN
.
Can you point me to your solaris fix Dave? I'd like to see if I can tidy all that up in something that requires less patching and is more based on FLAGS settings.
On Solaris, and some OS X versions, if you want a 64bit build, you must add the compiler flag m64
with gcc. Usually, putting that in CFLAGS
is enough. However, for Numpy that does not work.
Someone came up with a fix for this which was implemented only on OS X, that involved creating a wrapper script called gcc_fake
which basically calls gcc with the m64
option. You can see the script yourself in the top level directory.
Since they had done this only on OS X, it did not work on Solaris. So I came up with a solution for Solaris, but I avoided the wrapper script. Instead I set the variable to CC=gcc m64
I'm attaching a patch, which basically uses the Solaris on any system, including OS X. I think this is the sensible way to do it, not have a wrapper script.
I've not tested the attached patch  not even on Solaris!! But I think you can see what I am trying to do. I was going to try to explain it in words, but a bit of code, even if untested, should be more sensible.
Changed 12 years ago by
Attachment:  9808removegcc_fake.patch added 

An untested patch, which makes Numpy build the same was on OS X as it does on Solaris or other platforms where SAGE64=yes. It removes the stupid wrapper script.
comment:10 followup: 11 Changed 12 years ago by
Ok  so we still use g95 on some targets. So we need to keep some patches just for these  bother.
comment:11 followup: 12 Changed 12 years ago by
Replying to fbissey:
Ok  so we still use g95 on some targets. So we need to keep some patches just for these  bother.
I don't believe g95 is used anywhere. There are g95 binaries in the Fortran package in Sage, but William said they can be removed. There is a gfortran binary. So as far as I'm aware, all g95 stuff can be removed, but SAGE_FORTRAN
can't be removed.
Dave
comment:12 followup: 14 Changed 12 years ago by
Replying to drkirkby:
Replying to fbissey:
Ok  so we still use g95 on some targets. So we need to keep some patches just for these  bother.
I don't believe g95 is used anywhere. There are g95 binaries in the Fortran package in Sage, but William said they can be removed. There is a gfortran binary. So as far as I'm aware, all g95 stuff can be removed, but
SAGE_FORTRAN
can't be removed.
That's good! That means we probably can give the shove to the gnu.py and init.py patches. I wouldn't be sorry to see the back of these.
There is a comment in SPKG.txt:
Special Update/Build Instructions: * The file $SAGE_ROOT/devel/sage/sage/ext/numpy.pxi comes from this file and must be updated if/when the file src/numpy/doc/cython/numpy.pxi is updated.
I cannot find the file in question in that location. There is however a numpy.pxi under src/numpy/random/mtrand but I am not sure that's the file in question. Furthermore I don't appear to have a numpy.pxi in $SAGE_ROOT/devel/sage/sage/ext/numpy.pxi . Does anyone know if these instructions are obsolete?
comment:14 followup: 15 Changed 12 years ago by
Replying to fbissey:
Replying to drkirkby:
Replying to fbissey:
Ok  so we still use g95 on some targets. So we need to keep some patches just for these  bother.
I don't believe g95 is used anywhere. There are g95 binaries in the Fortran package in Sage, but William said they can be removed. There is a gfortran binary. So as far as I'm aware, all g95 stuff can be removed, but
SAGE_FORTRAN
can't be removed.That's good! That means we probably can give the shove to the gnu.py and init.py patches. I wouldn't be sorry to see the back of these.
There is a comment in SPKG.txt:
Special Update/Build Instructions: * The file $SAGE_ROOT/devel/sage/sage/ext/numpy.pxi comes from this file and must be updated if/when the file src/numpy/doc/cython/numpy.pxi is updated.I cannot find the file in question in that location. There is however a numpy.pxi under src/numpy/random/mtrand but I am not sure that's the file in question. Furthermore I don't appear to have a numpy.pxi in $SAGE_ROOT/devel/sage/sage/ext/numpy.pxi . Does anyone know if these instructions are obsolete?
I think I was the one that added those instructions, and I'm pretty sure they're obsolete instructions now. I believe we took care of merging the differences between the two numpy.pxi/pxd files a while ago.
comment:15 Changed 12 years ago by
Replying to jason:
I think I was the one that added those instructions, and I'm pretty sure they're obsolete instructions now. I believe we took care of merging the differences between the two numpy.pxi/pxd files a while ago.
Thank you for the confirmation. I have done some cleaning to numpy's spkginstall and it seems to work as intended on my machine. I guess we'll update to 1.5 and give it a spin.
comment:16 followups: 17 18 Changed 12 years ago by
Replying to fbissey:
Replying to mhansen:
I don't think that we should upgrade to 1.5.0rc1  we should do 1.4.1 for now and wait until 1.5 is released.
That has to be double checked but maldun says he needs features in 1.5. It may be that the features he wants are 1.4.x and he doesn't know. Do we know roughly when 1.5 will be released? They are at 1.5rc1 9 days after beta2 so it could very well be upon us in a very short time.
In general though, we should not be upgrading to prerelease versions just because one person needs a feature that's only available in a prerelease. Everyone should not have to suffer the extra risks a prerelease gives unless there are compelling arguments for it.
I realise in this case 1.5 has since been released, so it's immaterial now.
Dave
comment:17 Changed 12 years ago by
Replying to drkirkby:
In general though, we should not be upgrading to prerelease versions just because one person needs a feature that's only available in a prerelease. Everyone should not have to suffer the extra risks a prerelease gives unless there are compelling arguments for it.
+1
comment:18 Changed 12 years ago by
Replying to drkirkby:
In general though, we should not be upgrading to prerelease versions just because one person needs a feature that's only available in a prerelease. Everyone should not have to suffer the extra risks a prerelease gives unless there are compelling arguments for it.
I realise in this case 1.5 has since been released, so it's immaterial now.
Yes, I didn't think it would be worth working on that particular version if it wasn't for the real proximity of the release (curses pari svn upgrade). I would probably have put pressure for 1.4.1 otherwise (very keen to see numpy upgraded).
comment:19 Changed 12 years ago by
Replying to jason:
FYI: Numpy 1.5 has been officially released now.
So I was right =) Til I'm done with 1.5.0rc1 1.5 is out...
So changed so far:
The following doctest had to be rewritten:
File "/home/maldun/sage/sage4.5.2/devel/sage/doc/en/faq/faqusage.rst", line 341: sage: stats.ttest_ind(list([1,2,3,4,5]),list([2,3,4,5,.6])) Expected: doctest...DeprecationWarning... (0.076752955645333687, 0.94070490247380478) Got: (0.076752955645333687, 0.94070490247380478)
That was easy =)
The next prob was:
File "/home/maldun/sage/sage4.5.2/devel/sage/sage/graphs/graph.py", line 615: sage: Graph(A) Exception raised: Traceback (most recent call last): File "/home/maldun/sage/sage4.5.2/local/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/home/maldun/sage/sage4.5.2/local/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/home/maldun/sage/sage4.5.2/local/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_1[22]>", line 1, in <module> Graph(A)###line 615: sage: Graph(A) File "/home/maldun/sage/sage4.5.2/local/lib/python/sitepackages/sage/graphs/graph.py", line 846, in __init__ data = networkx.MultiGraph(data) File "/home/maldun/sage/sage4.5.2/local/lib/python2.6/networkx/classes/graph.py", line 220, in __init__ convert.from_whatever(data,create_using=self) File "/home/maldun/sage/sage4.5.2/local/lib/python2.6/networkx/convert.py", line 157, in from_whatever if isinstance(thing,numpy.core.defmatrix.matrix) or \ AttributeError: 'module' object has no attribute 'defmatrix'
that was also easy. defmatrix just changed from numpy.core to numpy.matrix to numpy.matrixlib Only changed that line in /local/lib/python2.6/networkx/classes/graph.py
Here comes now a trickier one:
sage t valgrind "devel/sage/sage/rings/polynomial/polynomial_element.pyx" Total time for all tests: 716.4 seconds maldun@zauberbuch:~/sage/sage4.5.2$ sage t valgrind "devel/sage/sage/rings/polynomial/real_roots.pyx > " ERROR: File ./devel/sage/sage/rings/polynomial/real_roots.pyx is missing  The following tests failed: ./devel/sage/sage/rings/polynomial/real_roots.pyx # File not found Total time for all tests: 0.0 seconds maldun@zauberbuch:~/sage/sage4.5.2$ sage t valgrind "devel/sage/sage/rings/polynomial/real_roots.pyx" sage t valgrind "devel/sage/sage/rings/polynomial/real_roots.pyx" ********************************************************************** File "/home/maldun/sage/sage4.5.2/devel/sage/sage/rings/polynomial/real_roots.pyx", line 1819, in __main__.example_76 Failed example: oc.find_roots()###line 3064:_sage_ >>> oc.find_roots() Expected nothing Got: doctest:1: RuntimeWarning: tp_compare didn't return 1 or 2 for exception ********************************************************************** File "/home/maldun/sage/sage4.5.2/devel/sage/sage/rings/polynomial/real_roots.pyx", line 1840, in __main__.example_77 Failed example: oc.find_roots()###line 3085:_sage_ >>> oc.find_roots() Expected nothing Got: doctest:1: RuntimeWarning: tp_compare didn't return 1 or 2 for exception ********************************************************************** File "/home/maldun/sage/sage4.5.2/devel/sage/sage/rings/polynomial/real_roots.pyx", line 1934, in __main__.example_80 Failed example: oc.find_roots()###line 3157:_sage_ >>> oc.find_roots() Expected nothing Got: doctest:1: RuntimeWarning: tp_compare didn't return 1 or 2 for exception ********************************************************************** File "/home/maldun/sage/sage4.5.2/devel/sage/sage/rings/polynomial/real_roots.pyx", line 2320, in __main__.example_98 Failed example: real_roots(x**Integer(5) * (x**Integer(2)  Integer(9999))**Integer(2)  Integer(1))###line 3870:_sage_ >>> real_roots(x^5 * (x^2  9999)^2  1) Expected: [((29274496381311/9007199254740992, 419601125186091/2251799813685248), 1), ((2126658450145849453951061654415153249597/21267647932558653966460912964485513216, 4253316902721330018853696359533061621799/42535295865117307932921825928971026432), 1), ((1063329226287740282451317352558954186101/10633823966279326983230456482242756608, 531664614358685696701445201630854654353/5316911983139663491615228241121378304), 1)] Got: doctest:1: RuntimeWarning: tp_compare didn't return 1 or 2 for exception [((29274496381311/9007199254740992, 419601125186091/2251799813685248), 1), ((2126658450145849453951061654415153249597/21267647932558653966460912964485513216, 4253316902721330018853696359533061621799/42535295865117307932921825928971026432), 1), ((1063329226287740282451317352558954186101/10633823966279326983230456482242756608, 531664614358685696701445201630854654353/5316911983139663491615228241121378304), 1)] ********************************************************************** 4 items had failures: 1 of 10 in __main__.example_76 1 of 9 in __main__.example_77 1 of 10 in __main__.example_80 1 of 44 in __main__.example_98 ***Test Failed*** 4 failures. [227.6 s]  The following tests failed: sage t valgrind "devel/sage/sage/rings/polynomial/real_roots.pyx" Total time for all tests: 227.6 seconds
This is not the only one, but the root of evil seems to be in real_roots (how ironic...) more precisly: I tracked it down, with help of valgrind, to the class ocean. Has anyone an Idea??
comment:20 Changed 12 years ago by
Summary:  Upgrade numpy to 1.5.0rc1 and scipy to 0.8 → Upgrade numpy to 1.5 and scipy to 0.8 

comment:21 Changed 12 years ago by
Update: Found the problem. see: http://groups.google.com/group/cythonusers/browse_thread/thread/624c696293b7fe44
It seems that all versions 1.5.x hold this bug.... Tried downgrading to 1.4.1. and all corrections I have done so far worked.
I will now apply the patch from drkirkby, pack it again, test it overnight and hopefully we are done with 1.4.1, and hopefully they get it right in the next time. perhaps I should send the numpy/scipy guys the doctests I've done so far
scipy 0.8. don't seem to make any problems so far
comment:22 Changed 12 years ago by
Summary:  Upgrade numpy to 1.5 and scipy to 0.8 → Upgrade numpy to 1.4.1 and scipy to 0.8 

comment:23 followup: 24 Changed 12 years ago by
an doctest I oversaw:
Type ``numpy.typecodes`` for a list of the possible typecodes:: sage: import numpy sage: sorted(numpy.typecodes.items()) [('All', '?bhilqpBHILQPfdgFDGSUVOMm'), ('AllFloat', 'fdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Datetime', 'Mm'), ('Float', 'fdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')]
The output is now:
[('All', '?bhilqpBHILQPfdgFDGSUVOMm'), ('AllFloat', 'fdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Datetime', 'Mm'), ('Float', 'fdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')]
But since it's an extension, and no downgrade it is also no prob
comment:24 followup: 25 Changed 12 years ago by
Replying to maldun:
an doctest I oversaw:
Type ``numpy.typecodes`` for a list of the possible typecodes:: sage: import numpy sage: sorted(numpy.typecodes.items()) [('All', '?bhilqpBHILQPfdgFDGSUVOMm'), ('AllFloat', 'fdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Datetime', 'Mm'), ('Float', 'fdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')]The output is now:
[('All', '?bhilqpBHILQPfdgFDGSUVOMm'), ('AllFloat', 'fdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Datetime', 'Mm'), ('Float', 'fdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')]But since it's an extension, and no downgrade it is also no prob
the doctest should be fixed as well. I have attached a new version of spkginstall for numpy if you cared to give it a spin. It completely drop the noncygwin patches. It may need a little bit of work as I haven't looked yet at how you included Dave's fix.
comment:25 Changed 12 years ago by
Status:  needs_work → needs_review 

Replying to fbissey:
the doctest should be fixed as well. I have attached a new version of spkginstall for numpy if you cared to give it a spin. It completely drop the noncygwin patches. It may need a little bit of work as I haven't looked yet at how you included Dave's fix.
Your install file worked well for me. I have merged it, such that the patches are getting applied. All doctests work now. You can download the packages now at: http://code.google.com/p/spkgupload/downloads/detail?name=numpy1.4.1.spkg and http://code.google.com/p/spkgupload/downloads/detail?name=scipy0.8.spkg
but you also have to install a modified version of networkx. I have opened a ticket for this, and this also includes a package: http://trac.sagemath.org/sage_trac/ticket/9854
next I will add a patch which includes the changed doctests
comment:26 Changed 12 years ago by
One important note: after updating numpy one has to rebuild the source with sage ba, or else you get runtime warnings.
The reason is the size change in some of the numpy functions, and then the .pyx files have to be rebuild
Changed 12 years ago by
Attachment:  networkx1.0.1.p0.spkg added 

modified networkx package (test version)
Changed 12 years ago by
Attachment:  convert.py.diff added 

changes to networkx, which have to be applied
comment:27 followup: 28 Changed 12 years ago by
Thanks for the patches. I will probably introduce these in sageongentoo in short order. About networkx, you realize that sage4.5.3 will use networkx1.2? see ticket #9567 So have you tried to see if networkx1.2 needs patching (my guess is no, otherwise I would know by now from gentoo).
comment:28 followup: 29 Changed 12 years ago by
Replying to fbissey:
Thanks for the patches. I will probably introduce these in sageongentoo in short order. About networkx, you realize that sage4.5.3 will use networkx1.2? see ticket #9567 So have you tried to see if networkx1.2 needs patching (my guess is no, otherwise I would know by now from gentoo).
I've seen that networkx 1.2. is out but it doesn't seem to work with my sage4.2 with numpy 1.4.1 nor with my sage4.1 with the old numpy so I created this spkg in order that it can be used with old versions of sage too
But it seems the line where the problems come from doesn't exist in the new version of networkx anymore. Could you give the numpy packages a spin with the new versions of numpy/scipy? It is very time consuming for me to rebuild every time the source. Thanx in advance!
comment:29 Changed 12 years ago by
ith the new versions of numpy/scipy? It is very time consuming for me to rebuild every time the source. Thanx in advance!
sorry I meant new versions of networkx...
comment:30 Changed 12 years ago by
Understood. We already ship networkx1.2 with sage4.5.2 on sageongentoo [because networkx1.0.1 has been removed from Gentoo] so I can test your patches along with networkx. It may take 1 or 2 days for me to fit it in my schedule.
comment:31 Changed 12 years ago by
ok. Since I had some time this afternoon I build a version of sage1.4.3.alpha1 (which also has networkx1.2.) on my machine, applied the packages, the patch, rebuild the source with sage ba and everything worked fine!
./sage tp 4 long devel/sagenumpy .... sage t long devel/sagenumpy/doc/en/a_tour_of_sage/index.rst [23.1 s] sage t long devel/sagenumpy/doc/en/thematic_tutorials/group_theory.rst [247.1 s]  All tests passed! Timings have been updated. Total time for all tests: 7320.5 seconds
So networkx1.2 works!
For Info: I use Ubuntu 10.04 on my machine. So I think it would still be good if someone else would test it
comment:32 Changed 12 years ago by
Milestone:  sage4.6 → sage4.5.3 

comment:33 followup: 34 Changed 12 years ago by
It all works in sageongentoo.
However I think there are a few points that should be taken into consideration for both numpy and scipy. Just before posting my spkginstall I edited it to remove a change that I now think is probably necessary. I had set FC to SAGE_FORTRAN... Why? Because numpy tries to autodetect a fortran compiler and will take a gnu compiler first. Unless you set FC. Which means if Dave tries to compile sage with sunstudio on a machine that has also gfortran he is in for a world of hurt. So I think we should either set FC to SAGE_FORTRAN in numpy and possibly scipy. The other option is to set it globally but that may cause problems on OSX.
Thoughts? Francois
comment:34 Changed 12 years ago by
Replying to fbissey:
It all works in sageongentoo.
However I think there are a few points that should be taken into consideration for both numpy and scipy. Just before posting my spkginstall I edited it to remove a change that I now think is probably necessary. I had set FC to SAGE_FORTRAN... Why? Because numpy tries to autodetect a fortran compiler and will take a gnu compiler first. Unless you set FC. Which means if Dave tries to compile sage with sunstudio on a machine that has also gfortran he is in for a world of hurt. So I think we should either set FC to SAGE_FORTRAN in numpy and possibly scipy. The other option is to set it globally but that may cause problems on OSX.
Thoughts? Francois
A very pragmatic thougt: If it doesn"t hurt to set FC to SAGE_FORTRAN, why not? But I think it"s better not to do it globally. Because then we could cause more problems then we solve. Can you give a modified spkginstall? I can try it out over night then.
But I think it would also be good to get some feedback for Solaris and OSX for the current versions. Then we could decide to keep the current, or to take the other.
comment:35 followup: 36 Changed 12 years ago by
Good news: It seems that the problem from numpy1.5.0 has been resolved http://projects.scipy.org/numpy/ticket/1605 I don't think it's a big deal. Shall I give it a try, or should we stick to 1.4.1 for now?
comment:36 Changed 12 years ago by
Replying to maldun:
Good news: It seems that the problem from numpy1.5.0 has been resolved http://projects.scipy.org/numpy/ticket/1605 I don't think it's a big deal. Shall I give it a try, or should we stick to 1.4.1 for now?
...or wait til 1.5.1 is out?
comment:37 Changed 12 years ago by
May be wait until numpy1.5.1. it is not a big deal right now I guess. May be we should discuss it on sagedevel. At the moment we don't really have much testing apart from the fact I unleashed the upgrade to numpy1.4.1/scipy0.8 on sageongentoo users. I am not sure we can believably merge this in 4.5.3 unless it takes more than one week before it is released. If we target 4.5.3 I say we stay with what we have now, if we target 4.6 which should be a little further down the track we go for 1.5.x.
comment:38 Changed 12 years ago by
I updated my spkg_install as you requested. The other thing about using sage_fortran, that I had forgotten to change back when I removed it, is that it includes "fPIC". I didn't check the fortran spkg but hopefully the work done by Dave to set the correct pic flag is picked up in sage_fortran.
If we don't adopt this, FFLAG="fPIC" (or whatever mechanism Dave came up with) should be added in my previous version.
comment:39 Changed 12 years ago by
Replying to fbissey:
May be wait until numpy1.5.1. it is not a big deal right now I guess. May be we should discuss it on sagedevel. At the moment we don't really have much testing apart from the fact I unleashed the upgrade to numpy1.4.1/scipy0.8 on sageongentoo users. I am not sure we can believably merge this in 4.5.3 unless it takes more than one week before it is released. If we target 4.5.3 I say we stay with what we have now, if we target 4.6 which should be a little further down the track we go for 1.5.x.
Ok I think we should do the following: Let's stick to 1.4.1 since it is necessary for scipy 0.8. and quite well tested yet, in contrary to 1.5.x and it seems to work. Especially since the update from scipy 0.7 to 0.8 holds a lot of changes, and we don't know if they build some more bugs into numpy 1.5.1... If it turns out that updating from 1.4.1 to 1.5.1 is no problem, then well... packing a new package would'nt be the prob I think.
@new spkg: Ok I will give it a try tonight!
comment:40 followup: 41 Changed 12 years ago by
There's a lot of packages etc. on this ticket. Can someone provide a concise list of what would be needed to apply on a given platform? (For instance, in a very cursory glance, the networkx package being here mystifies.)
comment:41 Changed 12 years ago by
Replying to kcrisman:
There's a lot of packages etc. on this ticket. Can someone provide a concise list of what would be needed to apply on a given platform? (For instance, in a very cursory glance, the networkx package being here mystifies.)
Needed are: http://code.google.com/p/spkgupload/downloads/detail?name=numpy1.4.1.spkg http://code.google.com/p/spkgupload/downloads/detail?name=scipy0.8.spkg
after installing numpy, one needs so execute sage ba, or else one get's runtime warnings.
You also need networkx1.2. (the other networkx is just a small hack for testing, because 1.2. didn't build correctly on my machine, but this is obsolete now, since networkx1.2 is merged into sage 4.3.alpha1)
comment:42 Changed 12 years ago by
Description:  modified (diff) 

comment:43 Changed 12 years ago by
Description:  modified (diff) 

comment:44 Changed 12 years ago by
I gave the links and (current) build instructions into the discription, so that people can find the latest versions more quickly
comment:45 Changed 12 years ago by
Description:  modified (diff) 

comment:46 Changed 12 years ago by
Description:  modified (diff) 

Included direct links to files in description.
comment:47 Changed 12 years ago by
Milestone:  sage4.5.3 → sage4.6 

comment:48 Changed 12 years ago by
Description:  modified (diff) 

comment:49 Changed 12 years ago by
This seems to work fine on Mac OS X 10.6.4 Intel. I will try to test it tomorrow on a PowerPC machine. Note: I haven't looked at the spkg installs or anything, this is just a data point with regard to whether it works, not a review.
comment:50 followup: 51 Changed 12 years ago by
This fails very early on in Cygwin with
building library "npymath" sources creating build/src.cygwin1.7.5i6862.6/src conv_template:> build/src.cygwin1.7.5i6862.6/src/npy_math.c error: src/npy_math.c.src: No such file or directory Error building numpy.
comment:51 Changed 12 years ago by
Replying to mhansen:
This fails very early on in Cygwin with
building library "npymath" sources creating build/src.cygwin1.7.5i6862.6/src conv_template:> build/src.cygwin1.7.5i6862.6/src/npy_math.c error: src/npy_math.c.src: No such file or directory Error building numpy.
Ok I think I have the picture: some files from /src/numpy/core/src moved to subfolders this have to be changed in the cygwincoresetup.py
I'm installing already cygwin on my laptop and look if I get it up and running, but if anyone could help out I would be thankful!
comment:52 followup: 53 Changed 12 years ago by
I've tested it out, and everything works if you just remove the cygwincoresetup.py patch.
comment:53 followup: 54 Changed 12 years ago by
Replying to mhansen:
I've tested it out, and everything works if you just remove the cygwincoresetup.py patch.
I was just about to upload a patch... But great this is even better than my solution! (and simpler) The reason why it fails is just that cygwincoresetup.py is outdated.
one question: (because I'm quite the noob...) If I want to upload a modified version of numpy1.4.1 do I have to overwrite the old version, or shall I rename it with numpy1.4.1.p0.spkg?
comment:54 Changed 12 years ago by
Replying to maldun:
one question: (because I'm quite the noob...) If I want to upload a modified version of numpy1.4.1 do I have to overwrite the old version, or shall I rename it with numpy1.4.1.p0.spkg?
Overwrite the old one.
comment:55 Changed 12 years ago by
Thanx! Numpy is now updated with
 changes from fbissey to the installer (it worked for me)
 removed the cygwincoresetup.py patch
comment:57 followup: 58 Changed 12 years ago by
I'm getting a corrupted package message with the packages right now while trying it on a different system. Did something get changed with respect to the file with this latest update?
comment:58 followup: 59 Changed 12 years ago by
Replying to kcrisman:
I'm getting a corrupted package message with the packages right now while trying it on a different system. Did something get changed with respect to the file with this latest update?
Yes we did some changes to the spkginstall. But I downloaded and installed the package right now again. So I don't know wehre the problem is lying?
comment:59 followup: 60 Changed 12 years ago by
Replying to maldun:
Replying to kcrisman:
I'm getting a corrupted package message with the packages right now while trying it on a different system. Did something get changed with respect to the file with this latest update?
Yes we did some changes to the spkginstall. But I downloaded and installed the package right now again. So I don't know wehre the problem is lying?
Did you use sage pkg
to create it or just do some kind of compression? I get a message about tar: this does not look like a file archive
and tar: Archive contains obsolescent base64 headers
. I just installed the optional biopython package on this machine to test things, so the machine shouldn't be the problem (and it built Sage 4.5.2 just fine).
comment:60 followup: 61 Changed 12 years ago by
Replying to kcrisman:
Replying to maldun:
Replying to kcrisman:
I'm getting a corrupted package message with the packages right now while trying it on a different system. Did something get changed with respect to the file with this latest update?
Yes we did some changes to the spkginstall. But I downloaded and installed the package right now again. So I don't know wehre the problem is lying?
Did you use
sage pkg
to create it or just do some kind of compression? I get a message abouttar: this does not look like a file archive
andtar: Archive contains obsolescent base64 headers
. I just installed the optional biopython package on this machine to test things, so the machine shouldn't be the problem (and it built Sage 4.5.2 just fine).
Ok I repacked it now with pkg and uploaded it. Does it work now? And sorry for the newby question: what is the difference between pkg and just compress it? Is sage using a different version of tar? Because in fact a spkg is noting else then a tar.gz with different ending.
comment:61 Changed 12 years ago by
Ok I repacked it now with pkg and uploaded it. Does it work now? And sorry for the newby question: what is the difference between pkg and just compress it? Is sage using a different version of tar? Because in fact a spkg is noting else then a tar.gz with different ending.
No, it's a little more than that  it has an SPKG.txt, it has Mercurial repositories, etc. True that the file type is the same. But sage pkg tests several things, and a successful review would check all that. See here for more info.
comment:62 Changed 12 years ago by
Reviewers:  → KarlDieter Crisman 

Hmm, when I open it manually it seems fine. Though when I run sage pkg
on that new folder, I get
HG REPO: Unchecked in changes
So you'll need to fix that. Putting myself as a reviewer now.
Calling sage f
on this new spkg I made *does* seem to work, for whatever reason. This doesn't make sense to me  why is tar
complaining about yours? Hopefully someone who knows about file systems will test this soon and explain why this causes a problem on Mac OS X 10.4 Tiger.
comment:63 followup: 65 Changed 12 years ago by
Okay, that worked. Scipy seems to be installing fine directly from the website, I'm not sure why this happened.
comment:64 Changed 12 years ago by
Thanx for the hints! I updated the mercurial entries now to the latest version. Uploaded changed packages. Everything should be fine now
comment:65 Changed 12 years ago by
Replying to kcrisman:
Okay, that worked. Scipy seems to be installing fine directly from the website, I'm not sure why this happened.
I have an Idea: different versions of .tar may cause problems (so far as I know) If you look at the file sizes only it seems that sage does a different kind of compression.
comment:66 followup: 67 Changed 12 years ago by
I get the following error with the Scipy for some reason.
creating build/temp.macosx10.4ppc2.6/scipy/special creating build/temp.macosx10.4ppc2.6/scipy/special/c_misc compile options: 'I/Users/student/Desktop/sage4.5.2/local/lib/python2.6/sitepackages/numpy/core/include c' gcc: scipy/special/c_misc/fsolve.c scipy/special/c_misc/fsolve.c: In function 'false_position': scipy/special/c_misc/fsolve.c:58: warning: 'f3' may be used uninitialized in this function scipy/special/c_misc/fsolve.c:58: warning: 'x3' may be used uninitialized in this function gcc: scipy/special/c_misc/gammaincinv.c scipy/special/c_misc/gammaincinv.c:1:20: error: Python.h: No such file or directory In file included from /Users/student/Desktop/sage4.5.2/local/lib/python2.6/sitepackages/numpy/core/include/numpy/npy_math.h:5, from scipy/special/c_misc/gammaincinv.c:2: /Users/student/Desktop/sage4.5.2/local/lib/python2.6/sitepackages/numpy/core/include/numpy/npy_common.h:79:2: error: #error Must use Python with unicode enabled. scipy/special/c_misc/gammaincinv.c:1:20: error: Python.h: No such file or directory In file included from /Users/student/Desktop/sage4.5.2/local/lib/python2.6/sitepackages/numpy/core/include/numpy/npy_math.h:5, from scipy/special/c_misc/gammaincinv.c:2: /Users/student/Desktop/sage4.5.2/local/lib/python2.6/sitepackages/numpy/core/include/numpy/npy_common.h:79:2: error: #error Must use Python with unicode enabled. error: Command "gcc fnostrictaliasing g O2 DNDEBUG g fwrapv O3 Wall Wstrictprototypes I/Users/student/Desktop/sage4.5.2/local/lib/python2.6/sitepackages/numpy/core/include c scipy/special/c_misc/gammaincinv.c o build/temp.macosx10.4ppc2.6/scipy/special/c_misc/gammaincinv.o" failed with exit status 1 Error building scipy.
Did you use some special encoding for some of your stuff?
comment:67 Changed 12 years ago by
Replying to kcrisman:
I get the following error with the Scipy for some reason.
creating build/temp.macosx10.4ppc2.6/scipy/special creating build/temp.macosx10.4ppc2.6/scipy/special/c_misc compile options: 'I/Users/student/Desktop/sage4.5.2/local/lib/python2.6/sitepackages/numpy/core/include c' gcc: scipy/special/c_misc/fsolve.c scipy/special/c_misc/fsolve.c: In function 'false_position': scipy/special/c_misc/fsolve.c:58: warning: 'f3' may be used uninitialized in this function scipy/special/c_misc/fsolve.c:58: warning: 'x3' may be used uninitialized in this function gcc: scipy/special/c_misc/gammaincinv.c scipy/special/c_misc/gammaincinv.c:1:20: error: Python.h: No such file or directory In file included from /Users/student/Desktop/sage4.5.2/local/lib/python2.6/sitepackages/numpy core/include/numpy/npy_math.h:5, from scipy/special/c_misc/gammaincinv.c:2: /Users/student/Desktop/sage4.5.2/local/lib/python2.6/sitepackages/numpy/core/include/numpy/npy_common.h:79:2: error: #error Must use Python with unicode enabled. scipy/special/c_misc/gammaincinv.c:1:20: error: Python.h: No such file or directory In file included from /Users/student/Desktop/sage4.5.2/local/lib/python2.6/sitepackages/numpy/core/include/numpy/npy_math.h:5, from scipy/special/c_misc/gammaincinv.c:2: /Users/student/Desktop/sage4.5.2/local/lib/python2.6/sitepackages/numpy/core/include/numpy/npy_common.h:79:2: error: #error Must use Python with unicode enabled. error: Command "gcc fnostrictaliasing g O2 DNDEBUG g fwrapv O3 Wall Wstrictprototypes I/Users/student/Desktop/sage4.5.2/local/lib/python2.6/sitepackages/numpy/core/include c scipy/special/c_misc/gammaincinv.c o build/temp.macosx10.4ppc2.6/scipy/special/c_misc/gammaincinv.o" failed with exit status 1 Error building scipy.
Did you use some special encoding for some of your stuff?
Nope. Perhaps something else did go wrong. I repacked it now on a different machine with a newer version of ubuntu and uploaded it. Hope this works. The first install worked for me.
comment:68 followup: 69 Changed 12 years ago by
Sadly, same error. This is probably on my end, but this is a supported architecture so it's important to know what went wrong. I don't think that the ./sage ba
would have anything to do with it.
So I suspect that the "missing: Python.h
is the problem, as I've seen a few other things about this online (including ones like this one, where /include/Python.h definitely exists, here it's within the $SAGE_ROOT
directory). I wonder why it's not finding it this time?
comment:69 followup: 70 Changed 12 years ago by
Replying to kcrisman:
Sadly, same error. This is probably on my end, but this is a supported architecture so it's important to know what went wrong. I don't think that the
./sage ba
would have anything to do with it.So I suspect that the "missing:
Python.h
is the problem, as I've seen a few other things about this online (including ones like this one, where /include/Python.h definitely exists, here it's within the$SAGE_ROOT
directory). I wonder why it's not finding it this time?
Question: Have you tried to give the direct path to the compiler? And does in OS X gcc points to /include/Python.h because in sage it's /include/python2.6/Python.h ?
comment:70 Changed 12 years ago by
Replying to maldun:
Replying to kcrisman:
Sadly, same error. This is probably on my end, but this is a supported architecture so it's important to know what went wrong. I don't think that the
./sage ba
would have anything to do with it.So I suspect that the "missing:
Python.h
is the problem, as I've seen a few other things about this online (including ones like this one, where /include/Python.h definitely exists, here it's within the$SAGE_ROOT
directory). I wonder why it's not finding it this time?Question: Have you tried to give the direct path to the compiler?
No  how would I do that?
And does in OS X gcc points to /include/Python.h because in sage it's /include/python2.6/Python.h ?
I find that very unlikely, since everything else works fine and in general Sage builds fine on OS X 10.410.6. But sometimes things get mixed up, I'm sure. There aren't any weird env variables that would do that here, though, I don't think.
comment:71 followup: 72 Changed 12 years ago by
A question: do we impose python to be built with or without unicode support? If not, is the support enabled by default depending on the platform? To me it looks like numpy ships headers that are encoded with unicode and that your sage's python chock on them, that's the first and foremost error. distutils seem to be unable to work things properly after that.
Two things could be tried: 1) have python built with unicode support on OSX. 2) "vet" numpy to convert the headers in /Users/student/Desktop/sage4.5.2/local/lib/python2.6/sitepackages/numpy/core/include/numpy/ to plain text. Actually you probably can do that easily and manually right now on the installed headers.
Give it a spin and see if it works. If it does we'll have to do something about that piece of unicode one way or another.
comment:72 Changed 12 years ago by
Replying to fbissey:
A question: do we impose python to be built with or without unicode support? If not, is the support enabled by default depending on the platform? To me it looks like numpy ships headers that are encoded with unicode and that your sage's python chock on them, that's the first and foremost error. distutils seem to be unable to work things properly after that.
Two things could be tried: 1) have python built with unicode support on OSX. 2) "vet" numpy to convert the headers in /Users/student/Desktop/sage4.5.2/local/lib/python2.6/sitepackages/numpy/core/include/numpy/ to plain text. Actually you probably can do that easily and manually right now on the installed headers.
Umm... except that I have no idea how I might do that. What exact changes would I need to make to Python.h (and/or something else)? It seems unwise to make yet another python spkg change to deal with this; did numpy only recently (between whatever Sage has now and 1.4.1) start making its headers unicode? I am not exactly a C expert.
That said, 1) seems to be more canonical, though I guess if it makes it work, 2) could be an option. This is the sort of thing Leif and/or drkirkby usually have an informed opinion on...
Give it a spin and see if it works. If it does we'll have to do something about that piece of unicode one way or another.
comment:73 Changed 12 years ago by
2) would be good to find if this is indeed the problem. Doing it is another story. Do you have iconv on OSX? I would think so as it is needed by R amongst other.
so go into the right folder and try:
iconv f UTF8 t ISO88591 *.h
comment:75 followups: 76 77 Changed 12 years ago by
In the spkginstall, there is the line
unset CFLAGS LDFLAGS CXXFLAGS SHAREDFLAGS
could this be somehow connected to our problem, because if we unset the flags, gcc doesn't point at Python.h in OS X?
comment:76 Changed 12 years ago by
Replying to maldun:
In the spkginstall, there is the line
unset CFLAGS LDFLAGS CXXFLAGS SHAREDFLAGScould this be somehow connected to our problem, because if we unset the flags, gcc doesn't point at Python.h in OS X?
Ok I reformulate it: Can we use the CFLAGS and LDFLAGS to give to tell gcc to link Python.h directly?
comment:77 Changed 12 years ago by
Replying to maldun:
In the spkginstall, there is the line
unset CFLAGS LDFLAGS CXXFLAGS SHAREDFLAGScould this be somehow connected to our problem, because if we unset the flags, gcc doesn't point at Python.h in OS X?
But this is not a change in the spkginstall script. Sage has built fine on this computer in the very recent past. The new spkg also works fine on other (newer) Macs.
From the numpy website
Reusing npymath In 1.3.0, we started putting portable C math routines in npymath library, so that people can use those to write portable extensions. Unfortunately, it was not possible to easily link against this library: in 1.4.0, support has been added to numpy.distutils so that 3rd party can reuse this library. See coremath documentation for more information.
This seems quite possibly relevant (given the error message). After all, if Python.h
isn't imported, then maybe the first error seen is that PY_UNICODE_DEF
or whatever it's called wouldn't be set... maybe?
I'm just trying to throw out an idea here, I am woefully uniformed when it comes to headers and things like that. But I feel like it has to be a change in Scipy or Numpy, since the previous Sage built fine on this machine.
comment:78 followup: 79 Changed 12 years ago by
I have another wild theory: Haven't you said, that the package build one time? And then not. Is this correct?
The only thing I changed was updating the mercurial files, and then commiting. Therefore I had to write the commit messages, and I do this in kate. kate's standard encoding is utf8. ( My computer's language is german, so everything is utf8) This (or just packing it, because the encoding on my machine is not unicode) did change something to the encoding. What happens if you untar the package use the iconv from up there, and repack it on your machine?
comment:79 Changed 12 years ago by
Replying to maldun:
I have another wild theory: Haven't you said, that the package build one time? And then not. Is this correct?
The only thing I changed was updating the mercurial files, and then commiting. Therefore I had to write the commit messages, and I do this in kate. kate's standard encoding is utf8. ( My computer's language is german, so everything is utf8) This (or just packing it, because the encoding on my machine is not unicode) did change something to the encoding. What happens if you untar the package use the iconv from up there, and repack it on your machine?
I don't think so but it can be tested I guess. I think Karl is on something, I am having a look at scipy's spkginstall to see if there is something that should be done there.
I have an endianess problem with numpy1.4.1 on linux ppc, funny it works on ppc OSX: http://projects.scipy.org/numpy/ticket/1403 and http://bugs.gentoo.org/show_bug.cgi?id=306237 on the other hand numpy1.5.0 built on my test machine. I am trying to find the changeset that made it possible to see if it is worth backporting.
comment:80 Changed 12 years ago by
Replying to kcrisman:
Same error.
It may sound unrelated but what fortran compiler are you using in your sage install? g95 or gfortran?
In any case I think some of the patches in scipy would need rebasing. But the fact of the matter is that they are g95 dependent and we dropped g95  I already did so in numpy so we should do it as well in scipy.
comment:81 Changed 12 years ago by
I don't know exactly, but recall that Sage still includes Fortran compilers for Mac (see here). This is an spkg (sagefortran...). From the spkg readme:
= gFortran = == Description == G95 is a stable, production Fortran 95 compiler available for multiple CPU architectures and operating systems. == Upstream contacts == URL: http://ftp.g95.org http://www.g95.org
So I don't know how to answer your question  apparently it's both gfortran and g95 :) The spkginstall has more details on this, but it apparently depends on the version (there are different .bz2 files for different Macs); my old PPC Mac would have used g95, I think. Is there some technical reason we should drop this?
Responding to the other comment, sadly, although I thought Scipy installed correctly at first, my machine is REALLY slow, so eventually it turned out that it didn't. I succeeded on a newer Intel Mac, but that is unrelated.
comment:82 followup: 83 Changed 12 years ago by
Status:  needs_review → needs_work 

See comment 10 by Dave. g95 can be removed. Unless they forgot about OSX ppc. For my question, what does
$SAGE_LOCAL/bin/sage_fortran version
says? I am guessing we have gfortran binaries for all platforms. Dropping g95 means that we can remove some (ugly) patches for numpy/scipy there are probably other valid reasons to do so but generally speaking that's less work.
I removed all patches that were applied for g95 in numpy, so if you have g95 the problems may come from there.
I share your pain about building on ppc (although I dropped OS X sometimes ago, this is now linux only and gentoo so everything or almost install from sources. openoffice3.1 clocks in at 27 hours to build). Did you mean that numpy failed or restating that scipy failed in your latest comment?
If numpy fails for you as well on OSX ppc we may have a strong motivation to move to 1.5.0 again.
I am changing this to need_works until we work out what's wrong in your case.
comment:83 Changed 12 years ago by
Replying to fbissey:
If numpy fails for you as well on OSX ppc we may have a strong motivation to move to 1.5.0 again.
If you want I can prepare a package for 1.5.0 with the patch from the numpy trac, and run the doctests, in the evening.
comment:84 Changed 12 years ago by
Numpy was fine in all cases; Scipy was what didn't build. drkirkby's comment "There are g95 binaries in the Fortran package in Sage, but William said they can be removed." may not be true.
The command that supposedly checks which version I have returns with an error:
local/bin/sage_fortran: line 3: sage_fortran.bin: command not found
This also happens on my (successful) build of 4.6.prealpha4, so maybe the command is wrong?
Incidentally, does this ticket do anything with respect to #7831 and #8010? Just curious.
comment:85 Changed 12 years ago by
Okay, I finally figured out how to check this  I had to run the binary directly, the scripts didn't work.
G95 (GCC 4.0.3 (g95 0.91!) Jun 4 2007)
etcetera. On my Macintel, I get
GNU Fortran (GCC) 4.2.3 Copyright (C) 2007 Free Software Foundation, Inc.
So yes, it is using G95 on Tiger.
comment:86 Changed 12 years ago by
It would be great if this is the problem and it can somehow be solved!
Another thing: I builded numpy1.5.0 with the new patch now. There only failed some doctests because numpy is throwing some new warnings. Want to give it a try? Perhaps the problem is then obsolete with scipy 0.8. also? I personally would prefer to stick to 1.4.1, but well it is like it is...
comment:87 followup: 88 Changed 12 years ago by
If you post it, I can try it  though it might be after the weekend, depending on my physical access to that particular box.
fbissey  can you explain (in layman's terms) how having the wrong fortran compiler might affect things? It does confuse me :)
comment:88 followup: 91 Changed 12 years ago by
Replying to kcrisman:
If you post it, I can try it  though it might be after the weekend, depending on my physical access to that particular box.
fbissey  can you explain (in layman's terms) how having the wrong fortran compiler might affect things? It does confuse me :)
Up and ready!
Yes this would be interesting! I don't really get it either because it looks more like a C problem to me?
comment:89 Changed 12 years ago by
comment:90 Changed 12 years ago by
It's more of a hunch than anything special I'll admit to that. The numpy/scipy connection seems to be fragile so I am not discounting it. In actual fact the fortran compiler is used only to compile one file in the whole of numpy (lapack_lite.so). I am more worried that it mixes up the toolchain it prepares for scipy.
One thing I'd like to see however is a complete build log for scipy not just the failing bit. There may be clues earlier in the process.
comment:91 followup: 92 Changed 12 years ago by
Replying to maldun:
Replying to kcrisman:
If you post it, I can try it  though it might be after the weekend, depending on my physical access to that particular box.
fbissey  can you explain (in layman's terms) how having the wrong fortran compiler might affect things? It does confuse me :)
Up and ready!
Sorry, fails with exactly the same error (on a different box with similar specs, same G95 in its Sage). So for now probably stick with the bettertested 1.4.1 for numpy. Hopefully we can figure out what's going on here.
The logs for the numpy (1.5.0) and scipy are in this directory, from the second computer. Happy hunting! By the way, Python.h is clearly working fine elsewhere in these logs, and sage_fortran compiles all kinds of neat stuff up to that point.
comment:92 followup: 93 Changed 12 years ago by
Replying to kcrisman:
Replying to maldun:
Replying to kcrisman:
If you post it, I can try it  though it might be after the weekend, depending on my physical access to that particular box.
fbissey  can you explain (in layman's terms) how having the wrong fortran compiler might affect things? It does confuse me :)
Up and ready!
Sorry, fails with exactly the same error (on a different box with similar specs, same G95 in its Sage). So for now probably stick with the bettertested 1.4.1 for numpy. Hopefully we can figure out what's going on here.
The logs for the numpy (1.5.0) and scipy are in this directory, from the second computer. Happy hunting! By the way, Python.h is clearly working fine elsewhere in these logs, and sage_fortran compiles all kinds of neat stuff up to that point.
Thanks for the logs. Sorry to be a bother but would you have old build logs for scipy0.7 as well? Since I don't have a ppc OSX setup that would be very useful to have a successful log even from an older version of scipy.
comment:93 followup: 94 Changed 12 years ago by
Sorry, fails with exactly the same error (on a different box with similar specs, same G95 in its Sage). So for now probably stick with the bettertested 1.4.1 for numpy. Hopefully we can figure out what's going on here.
The logs for the numpy (1.5.0) and scipy are in this directory, from the second computer. Happy hunting! By the way, Python.h is clearly working fine elsewhere in these logs, and sage_fortran compiles all kinds of neat stuff up to that point.
Thanks for the logs. Sorry to be a bother but would you have old build logs for scipy0.7 as well? Since I don't have a ppc OSX setup that would be very useful to have a successful log even from an older version of scipy.
Look in the same place, just posted it. So nice to be operating at 1.25 GHz instead of 700 MHz...
Also, interestingly, I now have a "mixed" installation on this computer:
sage: import numpy sage: numpy.version.version '1.5.0' sage: import scipy sage: scipy.version.version '0.7.0'
At least some of the relevant tests seem to pass with this, though of course I get the RunTimeErrors? mentioned above and did Inf
and inf
change places? I know little about numpy and scipy, though. Anyway, that's better than on the other box, whose issues with bad tarballs seem to not exist on this box.
comment:94 followup: 95 Changed 12 years ago by
Replying to kcrisman:
Look in the same place, just posted it. So nice to be operating at 1.25 GHz instead of 700 MHz...
Also, interestingly, I now have a "mixed" installation on this computer:
sage: import numpy sage: numpy.version.version '1.5.0' sage: import scipy sage: scipy.version.version '0.7.0'At least some of the relevant tests seem to pass with this, though of course I get the RunTimeErrors? mentioned above and did
Inf
andinf
change places? I know little about numpy and scipy, though. Anyway, that's better than on the other box, whose issues with bad tarballs seem to not exist on this box.
Found the logs, I may have a theory and the logs may help to (in)validate it. Not sure about Inf/inf either. scipy should work fine with numpy1.5  once rebuilt, and sage needs rebuilding too. That may get rid of some of these runtime errors. scipy0.8 needs numpy1.4 minimum on the other hand.
comment:95 Changed 12 years ago by
Also, interestingly, I now have a "mixed" installation on this computer:
sage: import numpy sage: numpy.version.version '1.5.0' sage: import scipy sage: scipy.version.version '0.7.0'At least some of the relevant tests seem to pass with this, though of course I get the RunTimeErrors? mentioned above and did
Inf
andinf
change places? I know little about numpy and scipy, though. Anyway, that's better than on the other box, whose issues with bad tarballs seem to not exist on this box.Found the logs, I may have a theory and the logs may help to (in)validate it. Not sure about Inf/inf either. scipy should work fine with numpy1.5  once rebuilt, and sage needs rebuilding too. That may get rid of some of these runtime errors. scipy0.8 needs numpy1.4 minimum on the other hand.
Yup, tests in relevant modules are passing, with only the RuntimeWarning
s (not errors, sorry for that) in general marring things (as noted, one or two other very minor things). Anyway, none of this not compiling nonsense, so something must have changed in Scipy 0.8 from 0.7  maybe something that used that change mentioned above about reusing npymath
. There were indeed a couple changes to the gammaincinv
function in Scipy, if you look at their changesets, but as a nonexpert how those could have resulted in this Python.h/unicode error is totally beyond me.
comment:96 Changed 12 years ago by
Interestingly enough on Gentoo we have I/usr/include/python2.6 added to the compilation flags everywhere. I wonder if just finding a way of adding I${SAGE_LOCAL}/include/python2.6 in sage would help at all. The mystery then would be "how did it work before?" Furthermore why is it not needed for the earlier C compilations.... or on other platforms? May be we are using system python headers without knowing it?
I will work on that over the week end. Hopefully by Monday I will have something you can test.
comment:97 Changed 12 years ago by
It turns out to be more subtle than I thought and I don't know yet why.
It is not easy to add I${SAGE_LOCAL}/include/python2.6 to compile options, in any case many other compilation lines actually have it already. After careful checking, I found that this particular option is not used in any scipy0.7.x log I had on sage or Gentoo for these particular objects. But I see it in scipy0.8.0 in Gentoo.
I am messing up with some of my system at the moment so I cannot generate a successful build log of scipy0.8.0 from sage at the moment. But it would be interesting to know if this particular compilation option has been added in the successful builds.
comment:98 Changed 12 years ago by
OK  so there is a fair difference on that file between 0.7.2 and 0.8.0.
headers in gammaincinv.c in 0.7.2
#include <stdio.h> #include <math.h> #include "../cephes.h" #undef fabs #include "misc.h"
in 0.8.0
#include <Python.h> #include <numpy/npy_math.h> #include <stdio.h> #include <math.h> #include "../cephes.h" #undef fabs #include "misc.h"
So it will be very interesting to get a successful build log of scipy0.8.0 in sage for inspection. A quick and dirty fix would be to change <Python.h> to <python2.6/Python.h> and I am not completely sure it would work either.
There are a few more Python.h header in that folder so just fixing that one may not work:
grep r "Python\.h" * amos_wrappers.h:#include "Python.h" cdf_wrappers.h:#include "Python.h" cephes/sici.c:#include <Python.h> cephes/mconf.h:#include <Python.h> _cephesmodule.c:#include "Python.h" c_misc/gammaincinv.c:#include <Python.h> lambertw.c:#include "Python.h" orthogonal_eval.c:#include "Python.h" specfun_wrappers.h:#include "Python.h" toms_wrappers.h:#include "Python.h" ufunc_extras.h:#include "Python.h"
considering the SConscript the files in the cephes subfolder will probably be trouble too.
comment:99 Changed 12 years ago by
But it would nice to know anyway. If it doesn't work with gammaincinv.c, at least the error message would change. Providing a patch would then be easy for me.
Ironically I already had asked if this had could been the problem... The only thing what me disturbs is the this entry in the error log of kcrisman:
...y/npy_common.h:79:2: error: #error Must use Python with unicode enabled.
This sounds more like that unicode isn't enabled, and perhaps this is related to native python installation on the system, and the gcc uses this header instead that from sage. I think to be absolutely sure, we should try also to set the path really to Python.h in the sage install in the header, if setting to <python2.6/Python.h> doesn't work.
@fbissey Another question: would it help you, if you see my install log from ubuntu?
comment:100 Changed 12 years ago by
No need for extra logs. I found the culprit and why it affects only Karl's ppc machine. I point the finger to g95! or rather the fact that the patches that are installed if g95 is found should have been rebased.
Basically the file special/setup.py is replaced by a version from scipy0.7.0 which doesn't need the python header. Could you please use spkginstall.2 attached in scipy0.8.0 it doesn't use the patches for g95. It should work without special patch in my opinion anyway.
comment:101 Changed 12 years ago by
So I've packed the scipy with your spkginstall, but this time it doesn't work for me. I get this error:
Host system uname a: Linux math181 2.6.3122generic #63Ubuntu SMP Thu Aug 19 00:23:50 UTC 2010 x86_64 GNU/Linux **************************************************** **************************************************** CC Version gcc v Using builtin specs. Target: x86_64linuxgnu Configured with: ../src/configure v withpkgversion='Ubuntu 4.4.14ubuntu9' withbugurl=file:///usr/share/doc/gcc4.4/README.Bugs enablelanguages=c,c++,fortran,objc,objc++ prefix=/usr enableshared enablemultiarch enablelinkerbuildid withsystemzlib libexecdir=/usr/lib withoutincludedgettext enablethreads=posix withgxxincludedir=/usr/include/c++/4.4 programsuffix=4.4 enablenls enableclocale=gnu enablelibstdcxxdebug enableobjcgc disablewerror witharch32=i486 withtune=generic enablechecking=release build=x86_64linuxgnu host=x86_64linuxgnu target=x86_64linuxgnu Thread model: posix gcc version 4.4.1 (Ubuntu 4.4.14ubuntu9) **************************************************** invalid command name 'config_fc noopt noarch' Error building scipy. real 0m2.296s
comment:102 Changed 12 years ago by
I think I have gone overboard with double quotes, attaching a new version shortly.
comment:103 Changed 12 years ago by
Ok. I deleted all outdated patches like you suggested (initially I let them in, because they didn't any harm, and since I didn't know much about every detail of the patches, I first tried out if it works, but of course it's better to throw them out) and it builds without problem. But with the old spkg. the spkginstall.2 you attached still, with the same error, because my machine doesn't know the command config_fc noopt noarch:
invalid command name 'config_fc noopt noarch'
could it be that I have to install something on my system? Or should we solve this with an If clause?
comment:104 followup: 106 Changed 12 years ago by
ok I solved it: You forgot the "" before the config =)
comment:105 Changed 12 years ago by
I think I found the problem, I think the shell was not set up properly.
I mean, these are options passed to setup.py not separate commands. They shouldn't be interpreted as such and aren't in the numpy spkginstall, so there are no reason why they should be in scipy. But the shells for the two spkginstall were different so I think that may be a point of bash/shell semantic. In which case I am curious to find out what I am supposed to do in sh. There is a new spkginstall.2 for you to try.
comment:106 Changed 12 years ago by
Replying to maldun:
ok I solved it: You forgot the "" before the config =)
Ok you posted while I was answering. That may pass but I am not sure that does what it's supposed to do. There shouldn't be a "" in front. I suspect it is now just ignored.
comment:107 Changed 12 years ago by
ok I was to quick... indeed it builded, but not for too long: it took it a c"onfig_fc"... so it didn't ignore, but missinterpreted it.
But the error remains. I don't really get it is there something different with ubuntu's gcc?
comment:108 Changed 12 years ago by
Ah okay: I looked into the scipy docu: This option seems to be for numpy, not for scipy. http://projects.scipy.org/numpy/wiki/DistutilsDoc#specifingconfigfcoptionsforlibrariesinsetuppyscript
Because numpy builded with that command. Does it build on your system?
comment:109 Changed 12 years ago by
OK. I thought it was used in scipy as well. There seems to be 2 files with it, and it shouldn't have an impact. You can remove it then. Strange it doesn't generate errors in Gentoo.
comment:111 Changed 12 years ago by
Help! I couldn't work on this over the weekend, and now I'm not quite sure what I should do to test :) Should I just try installing the new Scipy 0.8, or what? I am very glad that it seems you've figured out what G95 didn't like, though. It would also be nice to see a diff between the spkginstall
scripts.
comment:112 Changed 12 years ago by
Ok short summary: 1) Try the new scipy 2) If this doesn't work try to change the header in gammaincinv.c in the scipy sources from <Python.h> to <python2.6/Python.h> 3) If this results in no change, rewrite the header that it takes the sage header directly means change <Python.h> to "whateverpathitmaybe/python2.6/Python.h" and look if the error message changes. (because there are other files affected, but this should be no prob to patch)
If it's really only the location of the header patching wouldn't be a problem. And Thanx for all the trouble!
I have the diffs made for on my other machine, I can post them tomorrow!
Changed 12 years ago by
Diff of the spkg installs; The lines wich concern config fc are removed due problems in ubuntu
comment:113 followup: 118 Changed 12 years ago by
Well, that seems to have helped...
gcc: scipy/special/c_misc/gammaincinv.c
and it keeps going!
So it was the patches for G95 that were the problem. Which one in particular (which command/line) caused the header confusion, do you think? I'm just curious, and don't know enough about this to say.
By the way, I'm seeing things like
compile options: 'I/Users/student/Desktop/sage4.5.2/local/lib/python2.6/sitepackages/numpy/core/include I/Users/student/Desktop/sage4.5.2/local/include/python2.6 c'
so hopefully that is also a positive sign.
comment:114 Changed 12 years ago by
Success! Now I'll set SAGE_CHECK=yes
, retry numpy, do ./sage ba
, retry scipy, and test the Sage library. It will take a LONG time but hopefully tomorrow I'll have a report for you.
comment:116 Changed 12 years ago by
Ah and don't forget the patch for the doctests in the attachement. Else there would 2 doctests fail due to output changes in numpy.
comment:117 followups: 119 120 Changed 12 years ago by
I'm not worried about that, but I'll do it when I get there. Numpy just finished installing successfully.
Question:
# Program around a bug in SciPY's distutils. unset CFLAGS python setup.py install ${NUMPY_FCONFIG}
I assume this is still needed in the numpy spkginstall
. Just checking.
Question: I know drkirkby is cc:ed on this ticket. Numpy seems to have a very easy to run test suite  see here. However, it requires the Nose package. Maybe this should be a separate ticket, but it would seem reasonable to include Nose, which is about 250 KB, so that we can run the numpy tests. I am curious as to how we run the scipy tests without it!
comment:118 Changed 12 years ago by
Replying to kcrisman:
Well, that seems to have helped...
gcc: scipy/special/c_misc/gammaincinv.cand it keeps going!
So it was the patches for G95 that were the problem. Which one in particular (which command/line) caused the header confusion, do you think? I'm just curious, and don't know enough about this to say.
Well, there are 4 patches that were applied in case g95 was found as the fortran compiler. The faulty one is special.setup.py (or possibly setup.py.special) which replaces the file setup.py in the folder scipy/special.
This setup.py file has changed quite a lot between version 0.7.0 and 0.8.0. The lines at fault:
# C libraries config.add_library('sc_c_misc',sources=[join('c_misc','*.c')]) config.add_library('sc_cephes',sources=[join('cephes','*.c')], include_dirs=[get_python_inc(), get_numpy_include_dirs()], macros=define_macros)
in version 0.7.0 and this is the code found in the "patch" file and
# C libraries config.add_library('sc_c_misc',sources=[join('c_misc','*.c')], include_dirs=[get_python_inc(), get_numpy_include_dirs()], macros=define_macros) config.add_library('sc_cephes',sources=[join('cephes','*.c')], include_dirs=[get_python_inc(), get_numpy_include_dirs()], macros=define_macros)
in version 0.8.0. This is a case were using patches rather copying files wholesale would have prevented the problem. Either the patch wouldn't have worked with the newer version and we would have known straight away or it would have applied with some fuzz without dommaging this particular section.
comment:119 Changed 12 years ago by
Replying to kcrisman:
I'm not worried about that, but I'll do it when I get there. Numpy just finished installing successfully.
Question:
# Program around a bug in SciPY's distutils. unset CFLAGS python setup.py install ${NUMPY_FCONFIG}I assume this is still needed in the numpy
spkginstall
. Just checking.Question: I know drkirkby is cc:ed on this ticket. Numpy seems to have a very easy to run test suite  see here. However, it requires the Nose package. Maybe this should be a separate ticket, but it would seem reasonable to include Nose, which is about 250 KB, so that we can run the numpy tests. I am curious as to how we run the scipy tests without it!
Actually Dave has started a thread about this a few weeks ago on the mailing list, you are welcome to add your opinion and revive the thread.
comment:120 Changed 12 years ago by
Replying to kcrisman:
I'm not worried about that, but I'll do it when I get there. Numpy just finished installing successfully.
Question:
# Program around a bug in SciPY's distutils. unset CFLAGS python setup.py install ${NUMPY_FCONFIG}I assume this is still needed in the numpy
spkginstall
. Just checking.
I thought I had asked maldun to remove this. It's working fine on Gentoo as it is, I am assuming this has been fixed upstream (possibly a while ago).
comment:121 followup: 122 Changed 12 years ago by
I added the
# Program around a bug in SciPY's distutils. unset CFLAGS
into the numpy spkginstall from the old one, just to be sure that we don't run into troubles, and actually it doesn't do any harm... but if you want I can remove it as well.
But didn't we agree that the
setup.py install ${NUMPY_FCONFIG
}
is necessary in numpy? I had to delete it from scipy or else it doesn't build anymore, but in numpy it's correct.
comment:122 Changed 12 years ago by
Replying to maldun:
I added the
# Program around a bug in SciPY's distutils. unset CFLAGSinto the numpy spkginstall from the old one, just to be sure that we don't run into troubles, and actually it doesn't do any harm... but if you want I can remove it as well.
We should remove it. If there is a problem  which I doubt, we can put it back.
But didn't we agree that the
setup.py install ${NUMPY_FCONFIG
} is necessary in numpy? I had to delete it from scipy or else it doesn't build anymore, but in numpy it's correct.
Yes! Leave it here.
comment:123 Changed 12 years ago by
Ok I removed the line.
I also thougth before posting I should check if there are some old rests in there, and I removed some patches that are possible(?) outdated. I think this could affect the cygwin installation, so it would be important to see if it still builds on all machines, or else I have to carefully merge the changed lines, from the old patches with the new versions of the files.
It works for me, so I hope I didn't cause some new problems
comment:124 Changed 12 years ago by
I have a different problem: When I upload the new Version, and try to download it I get the old one. Is there some confusion on the server?
comment:126 followups: 127 129 Changed 12 years ago by
Status:  needs_work → needs_review 

Ok  now this needs review!
I looked at the cygwinlapack_litesetup.py patch. The file patched hasn't changed between numpy1.3.x and 1.4.1, so it can still be applied safely if needed.
In summary the only problems overall are:
 numpy 1.4.1 is broken on linux ppc (on the official list of supported OS  so bad) and alpha (not on the supported list  so OK).
 numpy1.5.0 has an issues with cython that will be solved in 1.5.1  shame because it also fixes linux ppc and alpha.
Anything to add?
comment:127 Changed 12 years ago by
Replying to fbissey:
Anything to add?
as mentioned above: The version of numpy1.5.0 with the patch applied is ok. The only trouble I have are some doctests, which fails because numpy throws a "Division by zero encountered!" Exception, which makes them fail, although the results are correct. But I still don't know how to catch the exceptions.
comment:128 Changed 12 years ago by
I should say that while breaking on linux ppc is not good, I only know of one other person building sage on that platform. So it could take the backseat while problems with the 1.5 series are ironed out.
comment:129 followup: 133 Changed 12 years ago by
 numpy1.5.0 has an issues with cython that will be solved in 1.5.1  shame because it also fixes linux ppc and alpha.
Hmm, dropping an official thing is not so good. I think that one would want to post to the list about this  more likely wait for 1.5.1? Or is the fix for this easily backported?
Also, although I've been going for over 24 hours now on the tests (and several timeouts! so I'll have to run those with SAGE_TIMEOUT_LONG>3600
), I have only seen two errors, both like this:
sage t long "devel/sage/sage/graphs/digraph.py" ********************************************************************** File "/Users/student/Desktop/sage4.5.2/devel/sage/sage/graphs/digraph.py", line 204: sage: DiGraph(A) Exception raised: Traceback (most recent call last): File "/Users/student/Desktop/sage4.5.2/local/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/Users/student/Desktop/sage4.5.2/local/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/Users/student/Desktop/sage4.5.2/local/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_1[17]>", line 1, in <module> DiGraph(A)###line 204: sage: DiGraph(A) File "/Users/student/Desktop/sage4.5.2/local/lib/python/sitepackages/sage/graphs/digraph.py", line 364, in __init__ data = networkx.MultiDiGraph(data) File "/Users/student/Desktop/sage4.5.2/local/lib/python2.6/networkx/classes/digraph.py", line 217, in __init__ convert.from_whatever(data,create_using=self) File "/Users/student/Desktop/sage4.5.2/local/lib/python2.6/networkx/convert.py", line 157, in from_whatever if isinstance(thing,numpy.core.defmatrix.matrix) or \ AttributeError: 'module' object has no attribute 'defmatrix' **********************************************************************
Have you seen this? Clearly it's something to do with numpy, from the end  but it's not clear to me why it would be asking whether a module has defmatrix attribute.
comment:130 followup: 135 Changed 12 years ago by
I know this error. Since you use sage 4.5.2 you will also use networkx1.0.1. See comments 24 and following or ticket #9854.
You have to either use Sage 4.5.3 which holds networkx1.2, upgrade networkx1.0.1 in your current version, or apply the networkx1.0.1.p0.spkg in the attachement, which holds a simple patch.
The reason is that numpy.core.defmatrix moved to numpy.matrixlib.defmatrix.
comment:131 Changed 12 years ago by
@linux ppc I could work over the weekend the solve the failed doctests in numpy1.5.0, because with the patch from the numpy developers the biggest problem i.e. the cython problem is solved. The only thing remains are some "Division by Zero Warnings" as mentioned above.
comment:132 Changed 12 years ago by
And another thing: If we aren't able to merge numpy and scipy to 4.6. in time, why don't we provide the packages as optional or experimental? Because they work on most systems, and it would be a waste to just throw them away.
comment:133 followup: 134 Changed 12 years ago by
Replying to kcrisman:
 numpy1.5.0 has an issues with cython that will be solved in 1.5.1  shame because it also fixes linux ppc and alpha.
Hmm, dropping an official thing is not so good. I think that one would want to post to the list about this  more likely wait for 1.5.1? Or is the fix for this easily backported?
Depending on what list you read, Linux on PPC may or may not be supported. I've tried to get an agreement before on what we do support and what we don't, but never got anywhere.
I'm personally in favor of keeping Sage building on less regular systems, as it often indicates problems. The Solaris port has found endless bugs in Sage  some of which have even according to William been serious.
I have an old IBM RS/6000 7025 F50 (Power PC) which I was setting up AIX on the other day. I contemlated whether I should put Linux on it too for testing Sage. But when I read Sage's README.txt, PowerPC was only supported on OS X, not Linux. So I did not bother. But the machine has space for 18 disk drives, so I could always add it!!
Dave
comment:134 Changed 12 years ago by
I forgot about the networkx change, thanks for the reminder.
 numpy1.5.0 has an issues with cython that will be solved in 1.5.1  shame because it also fixes linux ppc and alpha.
Hmm, dropping an official thing is not so good. I think that one would want to post to the list about this  more likely wait for 1.5.1? Or is the fix for this easily backported?
I'm personally in favor of keeping Sage building on less regular systems, as it often indicates problems. The Solaris port has found endless bugs in Sage  some of which have even according to William been serious.
I have an old IBM RS/6000 7025 F50 (Power PC) which I was setting up AIX on the other day. I contemlated whether I should put Linux on it too for testing Sage. But when I read Sage's README.txt, PowerPC was only supported on OS X, not Linux. So I did not bother. But the machine has space for 18 disk drives, so I could always add it!!
Why not? I've been trying to figure out how to get a thumb drive to boot my PPC machine on Ubuntu (and possibly other distros) but it turns out it's devilishly hard to get that to work, and I haven't had time to do it. I can't find an easy virtualization option either (neither xen nor anything else seem to really be particularly easy, if they even still work with such an old machine...)
And another thing: If we aren't able to merge numpy and scipy to 4.6. in time, why don't we provide the packages as optional or experimental? Because they work on most systems, and it would be a waste to just throw them away.
Certainly one could do this, but even that would require a positive review from someone. You might have to ask on the list about this  I've never heard of having an upgrade to a normal package as only optional.
comment:135 Changed 12 years ago by
You have to either use Sage 4.5.3 which holds networkx1.2, upgrade networkx1.0.1 in your current version, or apply the networkx1.0.1.p0.spkg in the attachement, which holds a simple patch.
Incidentally, I also had to repackage that spkg, because you didn't use ./sage pkg
for it either :) and it had unchecked in changes :) But clearly that is irrelevant for testing, since the upgrade is already in a later Sage.
Okay, with this upgrade those two tests pass. I am currently testing the other four failures, all of which were timeouts, all of which are in files I personally know to have VERY long long
doctests, and all but one of which hopefully will pass with SAGE_TIMEOUT_LONG=10000
(there is one in interfaces/maxima.py
which always fails on OS X 10.4, as far as I know, which is due to tabcompletion testing of Maxima never finishing, totally unrelated to this). I'll let you know if something *doesn't* work.
So I think that my job is done here (pointing out that I was testing the version from before the weekend. fbissey or drkirkby should comment on the spkg itself.
comment:136 followup: 137 Changed 12 years ago by
Well thanks Karl! Your testing was very useful. Since I have contributed to bits and pieces I think I should had myself as an author. But most of the heavy lifting as been done by maldun.
So what's left is the question of whether we go for numpy1.4.1 and leave minor archs unsupported for a little bit. Or we wait for 1.5.1, which works on the minor archs in question and will play well with cython.
As an aside I have already pushed the upgrade in sageongentoo (to avoid tree rote, we already have to keep two old packages that are otherwise removed from Gentoo) and we want to avoid that kind of stuff as much as possible. So the current code is out there and used by a few people.
After that there are question of details. In spkginstall I have set FC to ${SAGE_LOCAL}/bin/sage_fortran, with the idea that it was basically calling "gfortran fpic" or the g95 equivalent. We have been talking about that very subject on sagedevel recently. Is it the best way to go? If one uses sunstudio (and I am planning to give a go) the correct flag would be Kpic but would it be set up properly in sage_fortran? I doubt it.
So what would be the best course of action? Using the variable SAGE_FORTRAN for now and ask it to be set with the proper pic flag and hopefully drop it later on when FC is the mainstay?
comment:137 Changed 12 years ago by
Authors:  Stefan Reiterer → Stefan Reiterer, Francois Bissey 

Replying to fbissey:
Well thanks Karl! Your testing was very useful. Since I have contributed to bits and pieces I think I should had myself as an author. But most of the heavy lifting as been done by maldun.
I think you are right. So I allowed myself to set you as coauthor, since this whole thing wouldn't work without your help =)
So what's left is the question of whether we go for numpy1.4.1 and leave minor archs unsupported for a little bit. Or we wait for 1.5.1, which works on the minor archs in question and will play well with cython.
I would suggest the following: If numpy 1.5.1 doesn't come out before Sage 4.6. let's take 1.4.1 (since ppc linux is not on the current supported list; see readme.tex of sage) I will start this weekend with correcting things of 1.5.0, since the current errors are new warnings, and I'm quite sure numpy 1.5.1 will throw those as well. When numpy 1.5.1 is out before 4.6. I will try to pack it, and then hopefully it is done in no time, since it shared the same probs.
As an aside I have already pushed the upgrade in sageongentoo (to avoid tree rote, we already have to keep two old packages that are otherwise removed from Gentoo) and we want to avoid that kind of stuff as much as possible. So the current code is out there and used by a few people.
After that there are question of details. In spkginstall I have set FC to ${SAGE_LOCAL}/bin/sage_fortran, with the idea that it was basically calling "gfortran fpic" or the g95 equivalent. We have been talking about that very subject on sagedevel recently. Is it the best way to go? If one uses sunstudio (and I am planning to give a go) the correct flag would be Kpic but would it be set up properly in sage_fortran? I doubt it.
So what would be the best course of action? Using the variable SAGE_FORTRAN for now and ask it to be set with the proper pic flag and hopefully drop it later on when FC is the mainstay?
The question is how fast this change with the fortran compiler is done. I think we should wait a little bit now.
My question is: should this given a positive review, or is something still missing?
comment:138 followup: 139 Changed 12 years ago by
I'd like to check this out on sage4.6.alpha1, but its unclear to me exactly what to do. Can someone summarize the current procedure for incorporating this ticket?
comment:139 Changed 12 years ago by
Replying to mhampton:
I'd like to check this out on sage4.6.alpha1, but its unclear to me exactly what to do. Can someone summarize the current procedure for incorporating this ticket?
Install notes and links to the packages are in the discription of this ticket!
Quick and dirty: Install the packagages, do sage ba to compile the whole source, apply the patch in the attachment for the doctests.
greez maldun
comment:140 Changed 12 years ago by
Just a quick note that I used this package in sage4.6alpha1 and it didn't break any tests on linuxx86, I haven't heard of any problems on linuxamd64 from a friend who was also testing this. So it applies cleanly on 4.6alpha1.
comment:141 followup: 142 Changed 12 years ago by
Dave pointed me to this ticket the day before yesterday (and I read it from the beginning). ;)
Despite that, it's unclear to me which diffs are currrent, and the description should IMHO be updated wrt. what's to be done to review this.
./sage ba
is also in my opinion not a solution to get dependent extension modules updated, i.e. if ./sage b
isn't sufficient, dependencies should be added to module_list.py
.
comment:142 followup: 162 Changed 12 years ago by
Description:  modified (diff) 

Replying to leif:
Dave pointed me to this ticket the day before yesterday (and I read it from the beginning). ;)
Despite that, it's unclear to me which diffs are currrent, and the description should IMHO be updated wrt. what's to be done to review this.
./sage ba
is also in my opinion not a solution to get dependent extension modules updated, i.e. if./sage b
isn't sufficient, dependencies should be added tomodule_list.py
.
I cleaned up the discription a little bit, and deleted the obsolete remark about networkx1.2
I don't know if this is a good Idea because this is a whole bunch of files we are talking about, which have to be added. Perhaps you won't save much time either when compiling. I first tried this too, but after about 2 hours searching, I got somehow tired of this... And after merging numpy > 1.3.x into Sage this has not to be done anymore. All versions of 1.5.x builded without problems, and didn't need ba anymore. You only have to do this when big changes are happening, and this is rather occassionally.
package with a linux ppc patch is out: http://code.google.com/p/spkgupload/downloads/detail?name=numpy1.4.1.p0.spkg hope it works this time!
comment:143 followup: 145 Changed 12 years ago by
Ah before I forget it, and you think I'm just lazy: I just looked up again in an non compiled version which files are concerned, and I remembered again what was really the problem Most of them are already in the module_list.py.
However the problem is, that there are actually no changes to that files, so sage b won't do nothing, because it does'nt recognice any changes!
only ba force sage to do that and link the whole c files to the new version of numpy.
You would have to change the behavior of b, not add something to the module_list.py
comment:144 Changed 12 years ago by
Just for the record, gcc
has the following option:
mabi=ieeelongdouble
Change the current ABI to use IEEE extended precision long double. This is a PowerPC 32bit Linux ABI option.
I don't know if this requires a different (or differently built) libc
however.
comment:145 followup: 146 Changed 12 years ago by
Replying to maldun:
Ah before I forget it, and you think I'm just lazy: I just looked up again in an non compiled version which files are concerned, and I remembered again what was really the problem Most of them are already in the module_list.py.
However the problem is, that there are actually no changes to that files, so sage b won't do nothing, because it does'nt recognice any changes!
So we have to add (at least some of) those files to depends
that actually have changed.
only ba force sage to do that and link the whole c files to the new version of numpy.
You would have to change the behavior of b, not add something to the module_list.py
That's IMHO not an option, since missing dependencies will break any Sage upgrade anyway.
P.S.: The attached diffs are still confusing. (One should at least change/update their descriptions.) For review, complete diffs between the old and new spkgs (of course excluding src/
) would be helpful.
comment:146 followups: 147 148 Changed 12 years ago by
Replying to leif:
So we have to add (at least some of) those files to
depends
that actually have changed.
And that's the problem: There are NO files which would have changed! Not within sage/devel/sagewhateverbranch. And that's also the reason for the warnings: numpy wonders why still the old size of flatiter is in the .c files. The only changes are in local/LIB/python/sitepackages/numpy, and b don't care about those. The only way to trick b was to put in a comment to the certain files, delete them again and save it, that sage b recognices the changes.
I when kcrisman gives me feedback if numpy works now on ppc, I will put a complete changelog here on trac. Perhaps over the weekend I find some time for that.
comment:147 followup: 149 Changed 12 years ago by
Reviewers:  KarlDieter Crisman → KarlDieter Crisman, David Kirkby, Leif Leonhardy, Francois Bissey 

I when kcrisman gives me feedback if numpy works now on ppc, I will put a complete changelog here on trac. Perhaps over the weekend I find some time for that.
You mean on OS X, not Linux, right?
Numpy installed fine, and Scipy got past the bad spot fine, so I don't anticipate any problems. Then I'll run tests.
Leif, is this really a problem? (Asked as a query, not a complaint.) Presumably someone will only use these packages via sageupgrade, and then the whole library rebuilds (because sagex.y.z is new), right? Or is that not how sageupgrade works?
I'm adding a few reviewers based on how things have gone over these ~150 comments.
comment:148 Changed 12 years ago by
Replying to maldun:
Replying to leif:
So we have to add (at least some of) those files to
depends
that actually have changed.And that's the problem: There are NO files which would have changed! Not within sage/devel/sagewhateverbranch. And that's also the reason for the warnings: numpy wonders why still the old size of flatiter is in the .c files.
So what? Then just make them also depend on e.g. $SAGE_LOCAL/lib/python/sitepackages/numpy/core/include/numpy/numpyconfig.h
(if that has changed).
comment:149 Changed 12 years ago by
Replying to kcrisman:
Leif, is this really a problem? (Asked as a query, not a complaint.) Presumably someone will only use these packages via sageupgrade, and then the whole library rebuilds (because sagex.y.z is new), right? Or is that not how sageupgrade works?
The Sage library is "new" in every Sage version (even if there were no real changes to it), but does not get rebuilt from scratch (since there's no reason to do so if the dependencies are correct [or at least close to complete].)
comment:150 Changed 12 years ago by
P.S.: $SAGE_LOCAL/lib/python/sitepackages/Cython/Includes/numpy.pxd
could be an option, too.
comment:151 Changed 12 years ago by
I can do that, but this still won't solve the problem. Perhaps you misunderstand me, so I explain it again in more detail.
After installing numpy the only thing that has to be done is to recompile the .pyx files (which don't have changed), so that the old .c files (which haven't changed either) have be builded with the new version of numpy, that actually the change happens. I'm not a pro to sage, but as far I understood that is, that b only recompile changed files. No changes, no compilation. Even if I add now the changed files (the package) sage only would recompile those, and leaves out a whole lot of files which also have to be recompiled. The only way to get now sage to recompile those .pyx which is to change the files after installation for example by putting a comment into them, or force sage to recompile them, that is that what ba does. since there is no option that I know to force recompilation, I don't have an Idea how else I could get it working.
If you have a better way please let me know. I will apply a patch as soon as possible!
@kcrisman: didn't we talk about linux ppc? You said it builds on OS X ppc, or am I wrong?
comment:152 Changed 12 years ago by
No, I'm sorry if you misread my post about that. I would *like* to do so but there are still a number of technical hurdles to me running Linux PPC on my box (I think this should be clear in my sagedevel post). I am testing whether the change you made to support Linux PPC doesn't break anything. Sorry if this is not useful :( and I wish I did have access to Linux PPC more easily; unfortunately, this requires either having access to an already existing Linux PPC machine OR trying to run an image of Linux on a CD even with so little RAM there is no point OR using a portable hard drive, which is a more significant expense.
comment:153 Changed 12 years ago by
Apparently we have to add Carl Friedrich Gauss to the authors... ;)
comment:154 Changed 12 years ago by
As I feared: Making them dependent didn't force compilation either.
comment:155 Changed 12 years ago by
Update: If you are changing the numpy.pxd file (for example insert a blank line and delete it) sage b compile the dependent sources, but this file is in cython not in numpy.
manual changes on the numpy header files didn't do anything.
@Leif I know what you are trying to tell me. I would have expected that making a header dependent would get sage to resolve the dependencies and recompile the .pyx files also.
But even little changes to the header files do nothing even if they are dependent, or I forced dependency.
I didn't meant this as offense or something, I just told you what I had experienced in the early stages of this ticket. If you have a solution for example a diff with an entry that forces dependency, (plot3d for example) I could write the others in no time.
Perhaps I do something wrong. If you know what the trouble is please tell me.
comment:156 followup: 157 Changed 12 years ago by
There are only a few headers that actually have been updated, e.g. _numpyconfig.h
.
A trivial solution is to just touch
one header file after installation in spkginstall
, and make the relevant extension modules (also) depend on that one.
Touching $SAGE_LOCAL/lib/python/sitepackages/Cython/Includes/numpy.pxd
could also help.
comment:157 followup: 159 Changed 12 years ago by
Replying to leif:
Touching
$SAGE_LOCAL/lib/python/sitepackages/Cython/Includes/numpy.pxd
could also help.
Doing so triggers at least recompilation of the 13 extension modules that add the numpy include dir to their include_dirs
.
comment:158 Changed 12 years ago by
Lots happens when I am asleep and lecturing first thing in the morning :)
I cannot comment on the sagerelease thread from work. I am one of the few people running sage on linux ppc  more as hobby to check that it works than real production. The problem for numpy on ppc is fixed in numpy1.5.
cannot comment on the rest right now I have a truckload of exams to prepare.
comment:159 followup: 160 Changed 12 years ago by
Replying to leif:
Replying to leif:
Touching
$SAGE_LOCAL/lib/python/sitepackages/Cython/Includes/numpy.pxd
could also help.Doing so triggers at least recompilation of the 13 extension modules that add the numpy include dir to their
include_dirs
.
... and caused ptestlong
to pass without any errors. So touching just that file in numpy's spkginstall
should fix all dependency issues, without touching module_list.py
at all (which is advantageous, too).
I just wonder that (or if) that file is still compatible with the new numpy.
Regarding the discussion on sagerelease, who prepares a numpy 1.5.0 (or later) spkg? ;)
The 1.4.1 seems obsolete now...
comment:160 Changed 12 years ago by
Replying to leif:
So touching just that file in numpy's
spkginstall
should fix all dependency issues, without touchingmodule_list.py
at all (which is advantageous, too).
Unfortunately this will only work if Cython is not updated/rebuilt (after numpy has been installed), since none of these packages depends on the other, and Cython might still ship one with an old time stamp (which is preserved on installation).
comment:161 Changed 12 years ago by
Regarding the discussion on sagerelease, who prepares a numpy 1.5.0 (or later) spkg? ;)
The 1.4.1 seems obsolete now...
I do. There are only 2 doctests that fails, due to output changes inf > Inf I hope I can complete it over the weekend.
Actually there is another problem with the headers: You would have to make ALL haeaders dependent on the concerning entries in the module_list.py, because in every version of numpy there can be an other headers that changes... and of course that is a matter of taste, but wouldn't get this the module_list.py a little bit messy, with dependencies that actually aren't needed?
just my 2 cents
comment:162 Changed 12 years ago by
Replying to maldun:
Replying to leif:
Dave pointed me to this ticket the day before yesterday (and I read it from the beginning). ;)
Despite that, it's unclear to me which diffs are currrent, and the description should IMHO be updated wrt. what's to be done to review this.
./sage ba
is also in my opinion not a solution to get dependent extension modules updated, i.e. if./sage b
isn't sufficient, dependencies should be added tomodule_list.py
.I cleaned up the discription a little bit, and deleted the obsolete remark about networkx1.2
I don't know if this is a good Idea because this is a whole bunch of files we are talking about, which have to be added. Perhaps you won't save much time either when compiling. I first tried this too, but after about 2 hours searching, I got somehow tired of this... And after merging numpy > 1.3.x into Sage this has not to be done anymore. All versions of 1.5.x builded without problems, and didn't need ba anymore. You only have to do this when big changes are happening, and this is rather occassionally.
package with a linux ppc patch is out: http://code.google.com/p/spkgupload/downloads/detail?name=numpy1.4.1.p0.spkg hope it works this time!
Ok so I tried to build numpy1.4.1 with our patch on ppc. Didn't work. It may be a good start but it fails in the same way it did before:
[39mcompile options: 'Inumpy/core/src/private Inumpy/core/src Inumpy/core Inumpy/core/src/npymath Inumpy/core/src/multiarray Inumpy/core/src/umath Inumpy/core/include I/usr/include/python2.6 c'[0m [39mpowerpcunknownlinuxgnugcc: _configtest.c[0m [39mremoving: _configtest.c _configtest.o[0m Traceback (most recent call last): File "setup.py", line 187, in <module> setup_package() File "setup.py", line 180, in setup_package configuration=configuration ) File "/scratch/portage/portage/devpython/numpy1.4.1/work/numpy1.4.1/numpy/distutils/core.py", line 186, in setup return old_setup(**new_attr) File "/usr/lib/python2.6/distutils/core.py", line 152, in setup dist.run_commands() File "/usr/lib/python2.6/distutils/dist.py", line 975, in run_commands self.run_command(cmd) File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command cmd_obj.run() File "/scratch/portage/portage/devpython/numpy1.4.1/work/numpy1.4.1/numpy/distutils/command/build.py", line 37, in run old_build.run(self) File "/usr/lib/python2.6/distutils/command/build.py", line 134, in run self.run_command(cmd_name) File "/usr/lib/python2.6/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command cmd_obj.run() File "/scratch/portage/portage/devpython/numpy1.4.1/work/numpy1.4.1/numpy/distutils/command/build_src.py", line 152, in run self.build_sources() File "/scratch/portage/portage/devpython/numpy1.4.1/work/numpy1.4.1/numpy/distutils/command/build_src.py", line 169, in build_sources self.build_extension_sources(ext) File "/scratch/portage/portage/devpython/numpy1.4.1/work/numpy1.4.1/numpy/distutils/command/build_src.py", line 328, in build_extension_sources sources = self.generate_sources(sources, ext) File "/scratch/portage/portage/devpython/numpy1.4.1/work/numpy1.4.1/numpy/distutils/command/build_src.py", line 385, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 416, in generate_config_h rep = check_long_double_representation(config_cmd) File "numpy/core/setup_common.py", line 136, in check_long_double_representation type = long_double_representation(pyod(object)) File "numpy/core/setup_common.py", line 244, in long_double_representation raise ValueError("Unrecognized format (%s)" % saw) ValueError: Unrecognized format (['001', '043', '105', '147', '211', '253', '315', '357', '301', '235', '157', '064', '124', '000', '000', '000', '000', '000', '000', '000', '000', '000', '000', '000', '376', '334', '272', '230', '166', '124', '062', '020']) [31;01m*[0m ERROR: devpython/numpy1.4.1 failed:
The expectation from numpy/core/setup_common.py is:
LONG_DOUBLE_REPRESENTATION_SRC = r""" /* "before" is 16 bytes to ensure there's no padding between it and "x". * We're not expecting any "long double" bigger than 16 bytes or with * alignment requirements stricter than 16 bytes. */ typedef %(type)s test_type; struct { char before[16]; test_type x; char after[8]; } foo = { { '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\001', '\043', '\105', '\147', '\211', '\253', '\315', '\357' }, 123456789.0, { '\376', '\334', '\272', '\230', '\166', '\124', '\062', '\020' } }; """
And when you compare the two you can see that it is an endianess problem.
comment:163 Changed 12 years ago by
Description:  modified (diff) 

Summary:  Upgrade numpy to 1.4.1 and scipy to 0.8 → Upgrade numpy to 1.5.0 and scipy to 0.8 
Changed 12 years ago by
Attachment:  numpyspkginstall.diff added 

Changes of the spkginstall from the upgrade to 1.3.x to 1.5.0 (changes that were applied to 1.4.1 are also contained)
Changed 12 years ago by
Attachment:  scipyspkginstall.diff added 

diff of the spkginstall of scipy from 0.7.p5 to 0.8
comment:164 Changed 12 years ago by
Description:  modified (diff) 

comment:165 Changed 12 years ago by
I am incorporating that in my 4.6alpha2 branch on Gentoo right now.
I will unfortunately be unable to test it on linux ppc before Wednesday 6th because of limited access to the hardware. So it may be too short for me to do a comprehensive review for the alpha3 deadline. I already know for sure that numpy1.5.0/scipy0.8.0 will build it's just the testing that will be lacking.
comment:167 Changed 12 years ago by
What do I need to do to install these? I did this: unpacked a sage4.6.alpha2 tarball, downloaded the new numpy and scipy spkg files, applied the file "trac_9808_numpy_doctest_change.patch" to the main Sage library spkg, and built Sage from scratch (with SAGE_CHECK=yes, except when installing the python spkg, which always fails selftests). On taurus (a skynet linux machine, x86_64Linuxnehalemfc), I'm getting doctest failures:
The following tests failed: sage t long devel/sage/sage/plot/plot.py # 12 doctests failed sage t long devel/sage/sage/misc/citation.pyx # 2 doctests failed sage t long devel/sage/sage/plot/misc.py # 4 doctests failed
For example:
sage t long devel/sage/sage/misc/citation.pyx ********************************************************************** File "/home/palmieri/taurus/numpy/sage4.6.alpha2/devel/sagemain/sage/misc/citation.pyx", line 60\ : sage: get_systems('integrate(x^2, x)') Expected: ['ginac', 'Maxima'] Got: ['numpy', 'ginac', 'Maxima'] **********************************************************************
and
sage t long devel/sage/sage/plot/plot.py ********************************************************************** File "/home/palmieri/taurus/numpy/sage4.6.alpha2/devel/sagemain/sage/plot/plot.py", line 207: sage: (g1+g2).show(ticks=pi/6, tick_formatter=pi) # show their sum, nicely formatted Exception raised: Traceback (most recent call last): File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_0[49]>", line 1, in <module> (g1+g2).show(ticks=pi/Integer(6), tick_formatter=pi) # show their sum, nicely formatted###line 207: sage: (g1+g2).show(ticks=pi/6, tick_formatter=pi) # show their sum, nicely formatted File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/sage/plot/plot.py", line 1546, in show self.save(DOCTEST_MODE_FILE, **options) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/sage/plot/plot.py", line 2202, in save figure.savefig(filename,dpi=dpi,bbox_inches='tight',**options) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/figure.py", line 1032, in savefig self.canvas.print_figure(*args, **kwargs) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/backend_bases.py", line 1455, in print_figure **kwargs) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/backends/backend_agg.py", line 358, in print_png FigureCanvasAgg.draw(self) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/backends/backend_agg.py", line 314, in draw self.figure.draw(self.renderer) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/artist.py", line 46, in draw_wrapper draw(artist, renderer, *args, **kwargs) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/figure.py", line 773, in draw for a in self.axes: a.draw(renderer) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/artist.py", line 46, in draw_wrapper draw(artist, renderer, *args, **kwargs) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/axes.py", line 1735, in draw a.draw(renderer) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/artist.py", line 46, in draw_wrapper draw(artist, renderer, *args, **kwargs) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/axis.py", line 742, in draw tick.draw(renderer) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/artist.py", line 46, in draw_wrapper draw(artist, renderer, *args, **kwargs) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/axis.py", line 196, in draw self.label1.draw(renderer) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/text.py", line 518, in draw bbox, info = self._get_layout(renderer) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/text.py", line 280, in _get_layout clean_line, self._fontproperties, ismath=ismath) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/backends/backend_agg.py", line 156, in get_text_width_height_descent self.mathtext_parser.parse(s, self.dpi, prop) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/mathtext.py", line 2797, in parse font_output = fontset_class(prop, backend) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/mathtext.py", line 658, in __init__ self._stix_fallback = StixFonts(*args, **kwargs) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/mathtext.py", line 900, in __init__ fullpath = findfont(name) File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python/sitepackages/matplotlib/font_manager.py", line 1306, in findfont if not os.path.exists(font): File "/home/palmieri/taurus/numpy/sage4.6.alpha2/local/lib/python2.6/genericpath.py", line 18, in exists st = os.stat(path) TypeError: coercing to Unicode: need string or buffer, dict found
comment:168 Changed 12 years ago by
Another report: on the skynet machine fulvia (Solaris on x86), matplotlib fails to find numpy, so it doesn't install:
============================================================================ BUILDING MATPLOTLIB matplotlib: 0.99.3 python: 2.6.4 (r264:75706, Oct 4 2010, 18:22:25) [GCC 4.5.1] platform: sunos5 REQUIRED DEPENDENCIES numpy: no * You must install numpy 1.1 or later to build * matplotlib. Error building matplotlib package. real 0m1.344s user 0m0.062s sys 0m0.093s sage: An error occurred while installing matplotlib0.99.3
The new numpy package has already been installed at this point in the build, so I don't know what's going on here. On the same machine, scipy fails to install:
gcc version 4.5.1 (GCC) **************************************************** ./spkginstall: LDFLAGS=shared: is not an identifier
Perhaps you need something like LDFLAGS="shared"
? I'm not really sure.
comment:169 followup: 170 Changed 12 years ago by
Can you try the new matplotlib spkg on #9221, if you think it is a matplotlib problem?
comment:170 Changed 12 years ago by
Replying to jason:
Can you try the new matplotlib spkg on #9221, if you think it is a matplotlib problem?
For fulvia, it doesn't help: I still get the same error about numpy not being installed. If I run ./sage python
and then import numpy
, I get an error, so although the numpy spkg claims to install correctly, there is some kind of problem:
ImportError: ld.so.1: python: fatal: relocation error: file /home/palmieri/fulvia/numpy/sage4.6.alpha2/local/lib/python2.6/sitepackages/numpy/core/multiarray.so: symbol isfinite: referenced symbol not found
For taurus, using the new matplotlib spkg (and applying the patch from #9221) does seem to fix most of the problems:
 The following tests failed: sage t long devel/sage/sage/misc/citation.pyx # 2 doctests failed 
comment:171 followup: 172 Changed 12 years ago by
Dear me!
And I was going to report numpy1.5/scipy0.8 build on ppc (should be running testall now, I'd do long but on that hardware we would still be there next week).
OK there are several issues to be solved: first "shared" is a good idea. That could be it.
Failing to build matplotlib with an installed numpy :) where did I see something like that... Why, I reported a bug on that on gentoo bugzilla a while ago: http://bugs.gentoo.org/show_bug.cgi?id=320669.
Long story short: that means your lapack (and possibly ATLAS) install is hosed. Probably related to problems producing .so libraries.
I don't get any of the failures you report. A quick package check on Gentoo shows that: matplotlib and rpy depend on numpy and it would be a good idea to rebuild them.
I am a bit worried about the unicode mention in plot.py, that may mean a mismatch between various packages configuration regarding unicode (python/numpy/matplotlib).
comment:172 followups: 173 174 Changed 12 years ago by
Replying to fbissey:
Dear me!
And I was going to report numpy1.5/scipy0.8 build on ppc (should be running testall now, I'd do long but on that hardware we would still be there next week).
OK there are several issues to be solved: first "shared" is a good idea. That could be it.
Oops, it looks like it's already quoted in the spkginstall file. I changed the first line from
#!/bin/sh
to
#!/usr/bin/env bash
I don't know if this was a good idea, but it did get past that error, only to run into the same import error as when trying to install matplotlib.
Failing to build matplotlib with an installed numpy :) where did I see something like that... Why, I reported a bug on that on gentoo bugzilla a while ago: http://bugs.gentoo.org/show_bug.cgi?id=320669.
Long story short: that means your lapack (and possibly ATLAS) install is hosed. Probably related to problems producing .so libraries.
Hmm. We've had problems with shared libraries with ATLAS before on Solaris machines, among others, but it's interesting that this particular problem wouldn't have shown up before this.
I don't get any of the failures you report. A quick package check on Gentoo shows that: matplotlib and rpy depend on numpy and it would be a good idea to rebuild them.
Since I'm building from scratch, all of the dependencies should work themselves out.
I am a bit worried about the unicode mention in plot.py, that may mean a mismatch between various packages configuration regarding unicode (python/numpy/matplotlib).
As Jason suggested, the new matplotlib spkg seems to fix this, at least on taurus. Since that package has been merged as of 4.6.alpha3, it should be taken care of.
comment:173 Changed 12 years ago by
Replying to jhpalmieri:
Replying to fbissey:
Dear me!
And I was going to report numpy1.5/scipy0.8 build on ppc (should be running testall now, I'd do long but on that hardware we would still be there next week).
OK there are several issues to be solved: first "shared" is a good idea. That could be it.
Oops, it looks like it's already quoted in the spkginstall file. I changed the first line from
#!/bin/shto
#!/usr/bin/env bashI don't know if this was a good idea, but it did get past that error, only to run into the same import error as when trying to install matplotlib.
The old version of the spkg was doing that as well. I don't think we should have dropped it. There may be a more elegant solution but in the meantime that should do.
Failing to build matplotlib with an installed numpy :) where did I see something like that... Why, I reported a bug on that on gentoo bugzilla a while ago: http://bugs.gentoo.org/show_bug.cgi?id=320669.
Long story short: that means your lapack (and possibly ATLAS) install is hosed. Probably related to problems producing .so libraries.
Hmm. We've had problems with shared libraries with ATLAS before on Solaris machines, among others, but it's interesting that this particular problem wouldn't have shown up before this.
I cannot comment on that, I don't know the package history of that machine. But searching for numpy in the closed bugs from gentoo's bugzilla show that I haven't been the only victim of that and the answer is always the same: for a reason or another lapack or blas/cblas is foobar.
I don't get any of the failures you report. A quick package check on Gentoo shows that: matplotlib and rpy depend on numpy and it would be a good idea to rebuild them.
Since I'm building from scratch, all of the dependencies should work themselves out.
I am a bit worried about the unicode mention in plot.py, that may mean a mismatch between various packages configuration regarding unicode (python/numpy/matplotlib).
As Jason suggested, the new matplotlib spkg seems to fix this, at least on taurus. Since that package has been merged as of 4.6.alpha3, it should be taken care of.
Ah! I already have matplotlib1.0 here that's probably why I never saw any of that.
comment:174 Changed 12 years ago by
Status:  needs_review → needs_work 

Replying to jhpalmieri:
Replying to fbissey:
Long story short: that means your lapack (and possibly ATLAS) install is hosed. Probably related to problems producing .so libraries.
Hmm. We've had problems with shared libraries with ATLAS before on Solaris machines, among others, but it's interesting that this particular problem wouldn't have shown up before this.
The upstream ATLAS program never builds shared libraries. Mathematica only ships with static libraries for ATLAS. It's not entirely clear to me why we bother building shared libraries for ATLAS, given they have tended to cause problems on several systems.
However, I'm not convinced that's the problem here. The ATLAS package has remained unchanged for a long time.
Dave
comment:175 followup: 176 Changed 12 years ago by
I don't have time to look at this now, but is this being built as C99? If not, that would explain it, since isfinite
was not defined until the C99 standard.
Dave
comment:176 followup: 177 Changed 12 years ago by
Replying to drkirkby:
I don't have time to look at this now, but is this being built as C99? If not, that would explain it, since
isfinite
was not defined until the C99 standard.
Why would fluvia not compile stuff in C99 by default? Compared to the other platforms? As for ATLAS it was a suggestion. I just know that the problem is linked to something broken in lapack and because of linking to libf77blas/libcblas it can come down to a problem with ATLAS. I don't know what exactly is happening with that particular machine.
comment:177 Changed 12 years ago by
Replying to fbissey:
Replying to drkirkby:
I don't have time to look at this now, but is this being built as C99? If not, that would explain it, since
isfinite
was not defined until the C99 standard.Why would fluvia not compile stuff in C99 by default? Compared to the other platforms? As for ATLAS it was a suggestion. I just know that the problem is linked to something broken in lapack and because of linking to libf77blas/libcblas it can come down to a problem with ATLAS. I don't know what exactly is happening with that particular machine.
gcc does not compile C99 by default.
http://gcc.gnu.org/onlinedocs/gcc4.5.1/gcc/CDialectOptions.html#CDialectOptions
says:
std= Determine the language standard. <snip> `gnu89' GNU dialect of ISO C90 (including some C99 features). This is the default for C code.
But isfinite
was not introduced until C99. The Sun headers tend not to define everything under the sun.
I'm not saying that's the issue. But it could be. I've seen a similar issue before.
Dave
comment:178 Changed 12 years ago by
For what it's worth, I redid the build on fulvia with SAGE_CHECK=yes, and ATLAS passed its selftests. The matplotlib package still fails to install, with the import error from numpy.
comment:179 followup: 180 Changed 12 years ago by
Does matplotlib generate any sort of config.log or similar? That might indicate what the problem is.
I know at one point I had issues on 64bit builds, when Numpy thought the static library was 32bit.
http://projects.scipy.org/numpy/ticket/1525 http://trac.sagemath.org/sage_trac/ticket/8086
comment:180 Changed 12 years ago by
By the way, I'm getting the same error on mark2 (Solaris on sparc) as on fulvia (Solaris on x86). Cicero (another linux box) doesn't have this problem with building matplotlib: it's built successfully and is now doctesting.
Replying to drkirkby:
Does matplotlib generate any sort of config.log or similar? That might indicate what the problem is.
I don't see any log file in spkg/build/matplotlib on the machines where this is failing. I think the problem is what I indicated above (comment:170): matplotlib tests whether numpy is installed by running python and trying "import numpy". In this case, that import fails, so matplotlib deduces that numpy is not installed.
Here's the relevant code (from setupext.py):
def check_for_numpy(): try: import numpy except ImportError: print_status("numpy", "no") print_message("You must install numpy 1.1 or later to build matplotlib.") return False
comment:181 Changed 12 years ago by
Here's a recursive grep for isfinite
in /usr/include
on my OpenSolaris machine. I've not tested the package on my OpenSolaris machine, but given the results on Solaris 10, I doubt it would build.
drkirkby@hawk:~$ ggrep R isfinite /usr/include /usr/include/python2.6/pyconfig.h:/* Define to 1 if you have the declaration of `isfinite', and to 0 if you /usr/include/python2.6/pymath.h:#define Py_IS_FINITE(X) isfinite(X) ggrep: warning: /usr/include/gphoto2/gphoto2: recursive directory loop /usr/include/iso/math_c99.h:#undef isfinite /usr/include/iso/math_c99.h:#define isfinite(x) __extension__( \ /usr/include/iso/math_c99.h: { __typeof(x) __x_r = (x); isfinite(__x_r) && \ /usr/include/iso/math_c99.h:#undef isfinite /usr/include/iso/math_c99.h:#define isfinite(x) __builtin_isfinite(x)
Note, there's no reference at all in math.h
to isfinite
. But when the conditions are right, the contents of /usr/include/iso/math_c99.h
will get included, and so the macro isfinite
will be defined.
IMHO, the error message John got:
ImportError: ld.so.1: python: fatal: relocation error: file /home/palmieri/fulvia/numpy/sage4.6.alpha2/local/lib/python2.6/sitepackages/numpy/core/multiarray.so: symbol isfinite: referenced symbol not found
is related to the fact that for whatever reason, the C99 headers are not being included properly.
Unless this is compiled with std=c99
, I doubt this will work.
This contrasts to a Linux box (sage.math) where it looks to me that isfinite
will be defined whenever math.h
is included. That is just an error in my opinion, as isfinite
was not defined until the C99 standard. So Linux is wrong to define it, but some software is assuming it's defined, and so fails when it is not.
IMHO, the bug is in unlikely to be LAPACK or ATLAS.
comment:182 Changed 12 years ago by
The more I look at this, the more I think it's a Numpy bug.
Python is not built with C99 on my machine. The config.log
of Python shows:
ac_cv_have_decl_isfinite=no
which is to be expected since its not built C99. So Numpy is at fault here I think, since that's making a reference to isfinite
despite Python is not built with C99, so it's not defined, as evidenced by Python's config.log
BTW John, do you want an account on my OpenSolaris machine? It can build Sage in under an hour, so is a bit quicker than fulvia.
Dave
comment:183 Changed 12 years ago by
IMHO, we need to see if we can reproduce this with the latest cvs/svn/whatever snapshot of Numpy, and if so, report it as a bug to
http://projects.scipy.org/numpy
If it's been fixed, then perhaps we can add a patch to the stable version, to just correct this bug.
Dave
comment:184 Changed 12 years ago by
I just downloaded the latest code (as listed here), replaced the "src" directory with it, and installed it. Running "./sage python" and then "import numpy" results in the same error about isfinite.
comment:186 Changed 12 years ago by
Report Upstream:  N/A → Reported upstream. Developers acknowledge bug. 

I see this problem on my OpenSolaris machine too, which I expected would be the case.
On my OpenSolaris machine, Python's header file pyconfig.h has in it:
/* Define to 1 if you have the declaration of `isfinite', and to 0 if you don't. */ #define HAVE_DECL_ISFINITE 0
but Numpy's config.h has in it:
#define HAVE_DECL_ISFINITE
Someone by the usename of 'pv' has said on the Numpy bug report http://projects.scipy.org/numpy/ticket/1625
However, the code at numpy/core/setup.py:218 is wrong, since it checks for #ifdef and not #if for Python's HAVE_DECL_ISFINITE.
So it seems this is a bug, and one which should be easy to fix. Note the fix does not need to be Solarisspecific, as the code is basically wrong on any platform  you just happen to get away with it on systems where the header file defines isfinite
even though the code is not built C99.
It's gone midnight here, I'm tired and have a lot to do all this week. But if nobody else patches this, I'll be able to do it in the next day or two.
Dave
comment:187 Changed 12 years ago by
Actually a bigger fix is being proposed on the numpy list. See there  I'm going to bed!
dave
comment:188 followup: 190 Changed 12 years ago by
Dave, thanks a lot for working on this with the numpy people. I prepared a new spkg and so far, the new numpy builds on fulvia, and after "./sage python", "import numpy" works. Matplotlib and scipy have now successfully built. It will be a while before the build is complete, but things look good to me.
I'm putting the spkg up at http://sage.math.washington.edu/home/palmieri/misc/numpy1.5.0.svn20101005.spkg.
This is from a current svn snapshot based on 1.5.0. I don't know if we have a policy about names for such spkg's, so I just used the date. I also don't know if we have a policy about whether to include the repository information, so I've kept it. This makes the spkg much bigger than the old version...
Should we mark this as "needs review" again? It seems to depend on #9221.
comment:189 Changed 12 years ago by
Should we mark this as "needs review" again?
Or should we wait until the patch is positively reviewed on the numpy trac server?
comment:190 followup: 191 Changed 12 years ago by
Replying to jhpalmieri:
Dave, thanks a lot for working on this with the numpy people. I prepared a new spkg and so far, the new numpy builds on fulvia, and after "./sage python", "import numpy" works. Matplotlib and scipy have now successfully built. It will be a while before the build is complete, but things look good to me.
I'm putting the spkg up at http://sage.math.washington.edu/home/palmieri/misc/numpy1.5.0.svn20101005.spkg.
This is from a current svn snapshot based on 1.5.0. I don't know if we have a policy about names for such spkg's, so I just used the date. I also don't know if we have a policy about whether to include the repository information, so I've kept it. This makes the spkg much bigger than the old version...
You don't need to keep the svn repository history, but you do need to put the revision number in the spkg name (numpy1.5.0.svnXXXX).
However, I think it would be better to just include the one patch on top of 1.5.0, rather than pull an unstable release from subversion. See the comments at the top of this trac ticket about using an unstable release (like 1.5.0RC2).
comment:191 Changed 12 years ago by
However, I think it would be better to just include the one patch on top of 1.5.0, rather than pull an unstable release from subversion. See the comments at the top of this trac ticket about using an unstable release (like 1.5.0RC2).
I agree on this point, for what it's worth (since I will just be doing ./sage i
and testing).
comment:192 followup: 193 Changed 12 years ago by
Yes, but the patch was based on the svn code, not vanilla 1.5.0.
comment:193 Changed 12 years ago by
Replying to jhpalmieri:
Yes, but the patch was based on the svn code, not vanilla 1.5.0.
Are you saying that the patch does not apply to 1.5.0 directly?
comment:194 Changed 12 years ago by
The patch affects two files. One part of the patch is this:

numpy/core/setup.py
diff git a/numpy/core/setup.py b/numpy/core/setup.py index ad8d5cb..f71ec10 100644
a b def check_ieee_macros(config): 215 215 _macros = ["isnan", "isinf", "signbit", "isfinite"] 216 216 if sys.version_info[:2] >= (2, 6): 217 217 for f in _macros: 218 already_declared = config.check_decl(fname2def("decl_%s" % f), 218 py_symbol = fname2def("decl_%s" % f) 219 already_declared = config.check_decl(py_symbol, 219 220 headers=["Python.h", "math.h"]) 220 221 if already_declared: 221 pub.append('NPY_%s' % fname2def("decl_%s" % f)) 222 if config.check_macro_true(py_symbol, 223 headers=["Python.h", "math.h"]): 224 pub.append('NPY_%s' % fname2def("decl_%s" % f)) 222 225 else: 223 226 macros.append(f) 224 227 else:
and for example the line "already_declared = config.check_decl(fname2def("decl_%s" % f),
" is not present in the 1.5.0 version.
Maybe you can modify it (and maybe easily) to apply to 1.5.0, but the diff, as written, does not.
comment:195 Changed 12 years ago by
Okay, I have a new version of the spkg, based on vanilla 1.5.0. I haven't tested it because I just lost my internet connection to fulvia. See http://sage.math.washington.edu/home/palmieri/misc/numpy1.5.0.spkg. I'm also attaching the file displaying the differences between this spkg and the old 1.5.0 spkg posted here, except for two things: the new spkg includes modified copies of numpy/core/setup.py and numpy/distutils/command/config.py (in the patches directory  the src directory is unmodified), and I didn't see any reason to include them in the posted patch file. They are of course included in the repository, but to keep the patch file small, I edited them out before posting it.
Oh, the internet connection is back again. I'm rebuilding on fulvia from scratch, which will take a while, but I just rebuilt the numpy package on mark2 successfully: "import numpy" works.
comment:196 Changed 12 years ago by
Ah I'm only a few days away and everyones busy. Sometimes I think this ticket is cursed... How high are the bets that this will work one day before 1.5.1 is released? =D
@jhpalmiere, drkirkby, fbissey: Thanks for all the trouble!
If you want I build the package with the new patches and upload it on google code for testing. (Or do you want to upload it?) I will test it out on Ubuntu tonight for sure.
grettings, maldun
comment:197 Changed 12 years ago by
Description:  modified (diff) 

Ok I oversaw the link in the post... I changed the old link in the discription to your new one. The new package is doing fine on ubuntu. report for doctests will come soon!
I have now workable spkg files for the both packages. Scipy worked well. Numpy need some work, but it's functioning.
I host the files at megaupload, at: Numpy: http://www.megaupload.com/?d=6GCFZD65 Scipy: http://www.megaupload.com/?d=JORT40DK
I think the problem is the same as described in Trac # 7791 (http://trac.sagemath.org/sage_trac/ticket/7791 )
I get the following errors:
What's this? Has someone hardcoded the sizes of the routines?