Opened 12 years ago
Last modified 12 years ago
#9808 closed task
Upgrade numpy to 1.4.1 and scipy to 0.8 — at Version 142
Reported by: | Stefan Reiterer | Owned by: | Stefan Reiterer |
---|---|---|---|
Priority: | major | Milestone: | sage-4.6.1 |
Component: | packages: standard | Keywords: | numpy, scipy |
Cc: | Merged in: | ||
Authors: | Stefan Reiterer, Francois Bissey | Reviewers: | Karl-Dieter Crisman |
Report Upstream: | Fixed upstream, but not in a stable release. | Work issues: | |
Branch: | Commit: | ||
Dependencies: | Stopgaps: |
Description (last modified by )
The packages can be found found under:
http://code.google.com/p/spkg-upload/downloads/detail?name=numpy-1.4.1.spkg http://code.google.com/p/spkg-upload/downloads/detail?name=scipy-0.8.spkg
Files are found with direct links:
http://spkg-upload.googlecode.com/files/numpy-1.4.1.spkg http://spkg-upload.googlecode.com/files/scipy-0.8.spkg
Important notes: after installing numpy, one needs to execute sage -ba, or else one get's runtime warnings.
trac_9808_numpy_doctest_change.patch in the attachment has to be applied, in order to get all doctests running because some of the output has changed.
Change History (148)
comment:1 Changed 12 years ago by
Status: | new → needs_work |
---|
comment:2 Changed 12 years ago by
Summary: | Upgrade numpy to 1.5b and scipy to 0.8 → Upgrade numpy to 1.5.0rc1 and scipy to 0.8 |
---|
numpy 1.5.0rc1: http://www.megaupload.com/?d=KK31RSZV
comment:3 follow-up: 4 Changed 12 years ago by
I don't think that we should upgrade to 1.5.0rc1 -- we should do 1.4.1 for now and wait until 1.5 is released.
comment:4 follow-up: 16 Changed 12 years ago by
Replying to mhansen:
I don't think that we should upgrade to 1.5.0rc1 -- we should do 1.4.1 for now and wait until 1.5 is released.
That has to be double checked but maldun says he needs features in 1.5. It may be that the features he wants are 1.4.x and he doesn't know. Do we know roughly when 1.5 will be released? They are at 1.5rc1 9 days after beta2 so it could very well be upon us in a very short time.
comment:5 Changed 12 years ago by
The warnings go away after rebuilding the source. And the doctests only throw warnings but there don't seem to be errors, and the output is correct
@mhansen, fbissey I think numpy has the same issues since version 1.4. and I'm quite sure that resolving it in 1.5.0rc1 would, solve the problem with 1.4.1. too. (and perhaps also with 1.5. later) So I suggest the following procedure: I solve the issues with the latest. Dobule check if 1.4.1 also works, and we decide then which of the versions we keep, or for example keep 1.4.1 in standard and 1.5.0rc1 as experimental
comment:6 follow-up: 7 Changed 12 years ago by
If 1.5.0rc1 is merged, #7166 can be closed. I don't know about 1.4.1, but in any case that is not a critical bug.
Some time back I made a Solaris-specific change to Numpy, as I wanted to get it reviewed with the least hassle - that generally means making it Solaris specific, as reviewers are easier to please if it only effects a rarer platform.
But I think the change should be implemented on OS X too. Currently there's a really nasty hack on OS X to build Numpy, that involves a script called fake_gcc
, copying that to $SAGE/LOCAL/bin/gcc
, then using the fake gcc to build Numpy. Finally this fake gcc file gets deleted.
The far neater option is the way I did it on Solaris. I suggest the changes I made for Solaris are implemented whenever SAGE64
is set to "yes", irrespective of whatever platform it may be. Then all this fake gcc rubbish can be removed.
If you want, I can create a patch.
comment:7 Changed 12 years ago by
Replying to drkirkby:
If you want, I can create a patch.
would be nice! But first I have to sort some things out, I hope it will get ready soon...
comment:8 follow-up: 9 Changed 12 years ago by
Looking at all that stuff in the spkg and comparing to Gentoo. Not very pretty. Do we really still need to use sage_fortran? On most platform we now use a native fortran compiler rather than a sage shipped one - I don't know if we still do that for OS X. In the patch to gnu.py there is:
+ # Note that sse flags and so on lead to gfortran code that segfaults, so disable arch flags + return opt +
An extract of the Gentoo set up:
export NUMPY_FCONFIG="config_fc --noopt --noarch"
Actually here is the full set up that you might find interesting:
# See progress in http://projects.scipy.org/scipy/numpy/ticket/573 # with the subtle difference that we don't want to break Darwin where # -shared is not a valid linker argument if [[ ${CHOST} != *-darwin* ]] ; then append-ldflags -shared fi # only one fortran to link with: # linking with cblas and lapack library will force # autodetecting and linking to all available fortran compilers use lapack || return [[ -z ${FC} ]] && FC=$(tc-getFC) # when fortran flags are set, pic is removed. FFLAGS="${FFLAGS} -fPIC" export NUMPY_FCONFIG="config_fc --noopt --noarch"
The other patches we have are relatively minor, a fix to the f2py man page, a patch for freebsd - that may be usefull, a patch for interix(!). I cannot comment on the cygwin patches but they look very small. We are are a bit more anal with site.cfg, I don't think it is useful in the context of sage - we just have more complex possible combinations of blas/lapack.
The NUMPY_FCONFIG is passed to distutils as an argument of
python setup.py install
i.e. in the end what we do boils down to "python setup.py install ${NUMPY_FCONFIG}".
Can you point me to your solaris fix Dave? I'd like to see if I can tidy all that up in something that requires less patching and is more based on FLAGS settings.
comment:9 Changed 12 years ago by
Replying to fbissey:
Looking at all that stuff in the spkg and comparing to Gentoo. Not very pretty.
What a surprise.
Do we really still need to use sage_fortran? On most platform we now use a native fortran compiler rather than a sage shipped one - I don't know if we still do that for OS X.
I've never understood the need for this SAGE_FORTRAN
. I've tried arguing for it to be removed and use FC
instead, but I had no luck.
A Fortran compiler is still shipped on OS X, but I don't see why the variable FC
can't be made to point to that, rather than have the variable SAGE_FORTRAN
.
Can you point me to your solaris fix Dave? I'd like to see if I can tidy all that up in something that requires less patching and is more based on FLAGS settings.
On Solaris, and some OS X versions, if you want a 64-bit build, you must add the compiler flag -m64
with gcc. Usually, putting that in CFLAGS
is enough. However, for Numpy that does not work.
Someone came up with a fix for this which was implemented only on OS X, that involved creating a wrapper script called gcc_fake
which basically calls gcc with the -m64
option. You can see the script yourself in the top level directory.
Since they had done this only on OS X, it did not work on Solaris. So I came up with a solution for Solaris, but I avoided the wrapper script. Instead I set the variable to CC=gcc -m64
I'm attaching a patch, which basically uses the Solaris on any system, including OS X. I think this is the sensible way to do it, not have a wrapper script.
I've not tested the attached patch - not even on Solaris!! But I think you can see what I am trying to do. I was going to try to explain it in words, but a bit of code, even if untested, should be more sensible.
Changed 12 years ago by
Attachment: | 9808-remove-gcc_fake.patch added |
---|
An untested patch, which makes Numpy build the same was on OS X as it does on Solaris or other platforms where SAGE64=yes. It removes the stupid wrapper script.
comment:10 follow-up: 11 Changed 12 years ago by
Ok - so we still use g95 on some targets. So we need to keep some patches just for these - bother.
comment:11 follow-up: 12 Changed 12 years ago by
Replying to fbissey:
Ok - so we still use g95 on some targets. So we need to keep some patches just for these - bother.
I don't believe g95 is used anywhere. There are g95 binaries in the Fortran package in Sage, but William said they can be removed. There is a gfortran binary. So as far as I'm aware, all g95 stuff can be removed, but SAGE_FORTRAN
can't be removed.
Dave
comment:12 follow-up: 14 Changed 12 years ago by
Replying to drkirkby:
Replying to fbissey:
Ok - so we still use g95 on some targets. So we need to keep some patches just for these - bother.
I don't believe g95 is used anywhere. There are g95 binaries in the Fortran package in Sage, but William said they can be removed. There is a gfortran binary. So as far as I'm aware, all g95 stuff can be removed, but
SAGE_FORTRAN
can't be removed.
That's good! That means we probably can give the shove to the gnu.py and init.py patches. I wouldn't be sorry to see the back of these.
There is a comment in SPKG.txt:
Special Update/Build Instructions: * The file $SAGE_ROOT/devel/sage/sage/ext/numpy.pxi comes from this file and must be updated if/when the file src/numpy/doc/cython/numpy.pxi is updated.
I cannot find the file in question in that location. There is however a numpy.pxi under src/numpy/random/mtrand but I am not sure that's the file in question. Furthermore I don't appear to have a numpy.pxi in $SAGE_ROOT/devel/sage/sage/ext/numpy.pxi . Does anyone know if these instructions are obsolete?
comment:14 follow-up: 15 Changed 12 years ago by
Replying to fbissey:
Replying to drkirkby:
Replying to fbissey:
Ok - so we still use g95 on some targets. So we need to keep some patches just for these - bother.
I don't believe g95 is used anywhere. There are g95 binaries in the Fortran package in Sage, but William said they can be removed. There is a gfortran binary. So as far as I'm aware, all g95 stuff can be removed, but
SAGE_FORTRAN
can't be removed.That's good! That means we probably can give the shove to the gnu.py and init.py patches. I wouldn't be sorry to see the back of these.
There is a comment in SPKG.txt:
Special Update/Build Instructions: * The file $SAGE_ROOT/devel/sage/sage/ext/numpy.pxi comes from this file and must be updated if/when the file src/numpy/doc/cython/numpy.pxi is updated.I cannot find the file in question in that location. There is however a numpy.pxi under src/numpy/random/mtrand but I am not sure that's the file in question. Furthermore I don't appear to have a numpy.pxi in $SAGE_ROOT/devel/sage/sage/ext/numpy.pxi . Does anyone know if these instructions are obsolete?
I think I was the one that added those instructions, and I'm pretty sure they're obsolete instructions now. I believe we took care of merging the differences between the two numpy.pxi/pxd files a while ago.
comment:15 Changed 12 years ago by
Replying to jason:
I think I was the one that added those instructions, and I'm pretty sure they're obsolete instructions now. I believe we took care of merging the differences between the two numpy.pxi/pxd files a while ago.
Thank you for the confirmation. I have done some cleaning to numpy's spkg-install and it seems to work as intended on my machine. I guess we'll update to 1.5 and give it a spin.
comment:16 follow-ups: 17 18 Changed 12 years ago by
Replying to fbissey:
Replying to mhansen:
I don't think that we should upgrade to 1.5.0rc1 -- we should do 1.4.1 for now and wait until 1.5 is released.
That has to be double checked but maldun says he needs features in 1.5. It may be that the features he wants are 1.4.x and he doesn't know. Do we know roughly when 1.5 will be released? They are at 1.5rc1 9 days after beta2 so it could very well be upon us in a very short time.
In general though, we should not be upgrading to pre-release versions just because one person needs a feature that's only available in a pre-release. Everyone should not have to suffer the extra risks a pre-release gives unless there are compelling arguments for it.
I realise in this case 1.5 has since been released, so it's immaterial now.
Dave
comment:17 Changed 12 years ago by
Replying to drkirkby:
In general though, we should not be upgrading to pre-release versions just because one person needs a feature that's only available in a pre-release. Everyone should not have to suffer the extra risks a pre-release gives unless there are compelling arguments for it.
+1
comment:18 Changed 12 years ago by
Replying to drkirkby:
In general though, we should not be upgrading to pre-release versions just because one person needs a feature that's only available in a pre-release. Everyone should not have to suffer the extra risks a pre-release gives unless there are compelling arguments for it.
I realise in this case 1.5 has since been released, so it's immaterial now.
Yes, I didn't think it would be worth working on that particular version if it wasn't for the real proximity of the release (curses pari svn upgrade). I would probably have put pressure for 1.4.1 otherwise (very keen to see numpy upgraded).
comment:19 Changed 12 years ago by
Replying to jason:
FYI: Numpy 1.5 has been officially released now.
So I was right =) Til I'm done with 1.5.0rc1 1.5 is out...
So changed so far:
The following doctest had to be rewritten:
File "/home/maldun/sage/sage-4.5.2/devel/sage/doc/en/faq/faq-usage.rst", line 341: sage: stats.ttest_ind(list([1,2,3,4,5]),list([2,3,4,5,.6])) Expected: doctest...DeprecationWarning... (0.076752955645333687, 0.94070490247380478) Got: (0.076752955645333687, 0.94070490247380478)
That was easy =)
The next prob was:
File "/home/maldun/sage/sage-4.5.2/devel/sage/sage/graphs/graph.py", line 615: sage: Graph(A) Exception raised: Traceback (most recent call last): File "/home/maldun/sage/sage-4.5.2/local/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/home/maldun/sage/sage-4.5.2/local/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/home/maldun/sage/sage-4.5.2/local/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_1[22]>", line 1, in <module> Graph(A)###line 615: sage: Graph(A) File "/home/maldun/sage/sage-4.5.2/local/lib/python/site-packages/sage/graphs/graph.py", line 846, in __init__ data = networkx.MultiGraph(data) File "/home/maldun/sage/sage-4.5.2/local/lib/python2.6/networkx/classes/graph.py", line 220, in __init__ convert.from_whatever(data,create_using=self) File "/home/maldun/sage/sage-4.5.2/local/lib/python2.6/networkx/convert.py", line 157, in from_whatever if isinstance(thing,numpy.core.defmatrix.matrix) or \ AttributeError: 'module' object has no attribute 'defmatrix'
that was also easy. defmatrix just changed from numpy.core to numpy.matrix to numpy.matrixlib Only changed that line in /local/lib/python2.6/networkx/classes/graph.py
Here comes now a trickier one:
sage -t -valgrind "devel/sage/sage/rings/polynomial/polynomial_element.pyx" Total time for all tests: 716.4 seconds maldun@zauberbuch:~/sage/sage-4.5.2$ sage -t -valgrind "devel/sage/sage/rings/polynomial/real_roots.pyx > " ERROR: File ./devel/sage/sage/rings/polynomial/real_roots.pyx is missing ---------------------------------------------------------------------- The following tests failed: ./devel/sage/sage/rings/polynomial/real_roots.pyx # File not found Total time for all tests: 0.0 seconds maldun@zauberbuch:~/sage/sage-4.5.2$ sage -t -valgrind "devel/sage/sage/rings/polynomial/real_roots.pyx" sage -t -valgrind "devel/sage/sage/rings/polynomial/real_roots.pyx" ********************************************************************** File "/home/maldun/sage/sage-4.5.2/devel/sage/sage/rings/polynomial/real_roots.pyx", line 1819, in __main__.example_76 Failed example: oc.find_roots()###line 3064:_sage_ >>> oc.find_roots() Expected nothing Got: doctest:1: RuntimeWarning: tp_compare didn't return -1 or -2 for exception ********************************************************************** File "/home/maldun/sage/sage-4.5.2/devel/sage/sage/rings/polynomial/real_roots.pyx", line 1840, in __main__.example_77 Failed example: oc.find_roots()###line 3085:_sage_ >>> oc.find_roots() Expected nothing Got: doctest:1: RuntimeWarning: tp_compare didn't return -1 or -2 for exception ********************************************************************** File "/home/maldun/sage/sage-4.5.2/devel/sage/sage/rings/polynomial/real_roots.pyx", line 1934, in __main__.example_80 Failed example: oc.find_roots()###line 3157:_sage_ >>> oc.find_roots() Expected nothing Got: doctest:1: RuntimeWarning: tp_compare didn't return -1 or -2 for exception ********************************************************************** File "/home/maldun/sage/sage-4.5.2/devel/sage/sage/rings/polynomial/real_roots.pyx", line 2320, in __main__.example_98 Failed example: real_roots(x**Integer(5) * (x**Integer(2) - Integer(9999))**Integer(2) - Integer(1))###line 3870:_sage_ >>> real_roots(x^5 * (x^2 - 9999)^2 - 1) Expected: [((-29274496381311/9007199254740992, 419601125186091/2251799813685248), 1), ((2126658450145849453951061654415153249597/21267647932558653966460912964485513216, 4253316902721330018853696359533061621799/42535295865117307932921825928971026432), 1), ((1063329226287740282451317352558954186101/10633823966279326983230456482242756608, 531664614358685696701445201630854654353/5316911983139663491615228241121378304), 1)] Got: doctest:1: RuntimeWarning: tp_compare didn't return -1 or -2 for exception [((-29274496381311/9007199254740992, 419601125186091/2251799813685248), 1), ((2126658450145849453951061654415153249597/21267647932558653966460912964485513216, 4253316902721330018853696359533061621799/42535295865117307932921825928971026432), 1), ((1063329226287740282451317352558954186101/10633823966279326983230456482242756608, 531664614358685696701445201630854654353/5316911983139663491615228241121378304), 1)] ********************************************************************** 4 items had failures: 1 of 10 in __main__.example_76 1 of 9 in __main__.example_77 1 of 10 in __main__.example_80 1 of 44 in __main__.example_98 ***Test Failed*** 4 failures. [227.6 s] ---------------------------------------------------------------------- The following tests failed: sage -t -valgrind "devel/sage/sage/rings/polynomial/real_roots.pyx" Total time for all tests: 227.6 seconds
This is not the only one, but the root of evil seems to be in real_roots (how ironic...) more precisly: I tracked it down, with help of valgrind, to the class ocean. Has anyone an Idea??
comment:20 Changed 12 years ago by
Summary: | Upgrade numpy to 1.5.0rc1 and scipy to 0.8 → Upgrade numpy to 1.5 and scipy to 0.8 |
---|
comment:21 Changed 12 years ago by
Update: Found the problem. see: http://groups.google.com/group/cython-users/browse_thread/thread/624c696293b7fe44
It seems that all versions 1.5.x hold this bug.... Tried downgrading to 1.4.1. and all corrections I have done so far worked.
I will now apply the patch from drkirkby, pack it again, test it overnight and hopefully we are done with 1.4.1, and hopefully they get it right in the next time. perhaps I should send the numpy/scipy guys the doctests I've done so far
scipy 0.8. don't seem to make any problems so far
comment:22 Changed 12 years ago by
Summary: | Upgrade numpy to 1.5 and scipy to 0.8 → Upgrade numpy to 1.4.1 and scipy to 0.8 |
---|
comment:23 follow-up: 24 Changed 12 years ago by
an doctest I oversaw:
Type ``numpy.typecodes`` for a list of the possible typecodes:: sage: import numpy sage: sorted(numpy.typecodes.items()) [('All', '?bhilqpBHILQPfdgFDGSUVOMm'), ('AllFloat', 'fdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Datetime', 'Mm'), ('Float', 'fdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')]
The output is now:
[('All', '?bhilqpBHILQPfdgFDGSUVOMm'), ('AllFloat', 'fdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Datetime', 'Mm'), ('Float', 'fdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')]
But since it's an extension, and no downgrade it is also no prob
comment:24 follow-up: 25 Changed 12 years ago by
Replying to maldun:
an doctest I oversaw:
Type ``numpy.typecodes`` for a list of the possible typecodes:: sage: import numpy sage: sorted(numpy.typecodes.items()) [('All', '?bhilqpBHILQPfdgFDGSUVOMm'), ('AllFloat', 'fdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Datetime', 'Mm'), ('Float', 'fdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')]The output is now:
[('All', '?bhilqpBHILQPfdgFDGSUVOMm'), ('AllFloat', 'fdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Datetime', 'Mm'), ('Float', 'fdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')]But since it's an extension, and no downgrade it is also no prob
the doctest should be fixed as well. I have attached a new version of spkg-install for numpy if you cared to give it a spin. It completely drop the non-cygwin patches. It may need a little bit of work as I haven't looked yet at how you included Dave's fix.
comment:25 Changed 12 years ago by
Status: | needs_work → needs_review |
---|
Replying to fbissey:
the doctest should be fixed as well. I have attached a new version of spkg-install for numpy if you cared to give it a spin. It completely drop the non-cygwin patches. It may need a little bit of work as I haven't looked yet at how you included Dave's fix.
Your install file worked well for me. I have merged it, such that the patches are getting applied. All doctests work now. You can download the packages now at: http://code.google.com/p/spkg-upload/downloads/detail?name=numpy-1.4.1.spkg and http://code.google.com/p/spkg-upload/downloads/detail?name=scipy-0.8.spkg
but you also have to install a modified version of networkx. I have opened a ticket for this, and this also includes a package: http://trac.sagemath.org/sage_trac/ticket/9854
next I will add a patch which includes the changed doctests
comment:26 Changed 12 years ago by
One important note: after updating numpy one has to rebuild the source with sage -ba, or else you get runtime warnings.
The reason is the size change in some of the numpy functions, and then the .pyx files have to be rebuild
Changed 12 years ago by
Attachment: | networkx-1.0.1.p0.spkg added |
---|
modified networkx package (test version)
Changed 12 years ago by
Attachment: | convert.py.diff added |
---|
changes to networkx, which have to be applied
comment:27 follow-up: 28 Changed 12 years ago by
Thanks for the patches. I will probably introduce these in sage-on-gentoo in short order. About networkx, you realize that sage-4.5.3 will use networkx-1.2? see ticket #9567 So have you tried to see if networkx-1.2 needs patching (my guess is no, otherwise I would know by now from gentoo).
comment:28 follow-up: 29 Changed 12 years ago by
Replying to fbissey:
Thanks for the patches. I will probably introduce these in sage-on-gentoo in short order. About networkx, you realize that sage-4.5.3 will use networkx-1.2? see ticket #9567 So have you tried to see if networkx-1.2 needs patching (my guess is no, otherwise I would know by now from gentoo).
I've seen that networkx 1.2. is out but it doesn't seem to work with my sage-4.2 with numpy 1.4.1 nor with my sage-4.1 with the old numpy so I created this spkg in order that it can be used with old versions of sage too
But it seems the line where the problems come from doesn't exist in the new version of networkx anymore. Could you give the numpy packages a spin with the new versions of numpy/scipy? It is very time consuming for me to rebuild every time the source. Thanx in advance!
comment:29 Changed 12 years ago by
ith the new versions of numpy/scipy? It is very time consuming for me to rebuild every time the source. Thanx in advance!
sorry I meant new versions of networkx...
comment:30 Changed 12 years ago by
Understood. We already ship networkx-1.2 with sage-4.5.2 on sage-on-gentoo [because networkx-1.0.1 has been removed from Gentoo] so I can test your patches along with networkx. It may take 1 or 2 days for me to fit it in my schedule.
comment:31 Changed 12 years ago by
ok. Since I had some time this afternoon I build a version of sage-1.4.3.alpha1 (which also has networkx-1.2.) on my machine, applied the packages, the patch, rebuild the source with sage -ba and everything worked fine!
./sage -tp 4 -long devel/sage-numpy .... sage -t -long devel/sage-numpy/doc/en/a_tour_of_sage/index.rst [23.1 s] sage -t -long devel/sage-numpy/doc/en/thematic_tutorials/group_theory.rst [247.1 s] ---------------------------------------------------------------------- All tests passed! Timings have been updated. Total time for all tests: 7320.5 seconds
So networkx-1.2 works!
For Info: I use Ubuntu 10.04 on my machine. So I think it would still be good if someone else would test it
comment:32 Changed 12 years ago by
Milestone: | sage-4.6 → sage-4.5.3 |
---|
comment:33 follow-up: 34 Changed 12 years ago by
It all works in sage-on-gentoo.
However I think there are a few points that should be taken into consideration for both numpy and scipy. Just before posting my spkg-install I edited it to remove a change that I now think is probably necessary. I had set FC to SAGE_FORTRAN... Why? Because numpy tries to autodetect a fortran compiler and will take a gnu compiler first. Unless you set FC. Which means if Dave tries to compile sage with sunstudio on a machine that has also gfortran he is in for a world of hurt. So I think we should either set FC to SAGE_FORTRAN in numpy and possibly scipy. The other option is to set it globally but that may cause problems on OSX.
Thoughts? Francois
comment:34 Changed 12 years ago by
Replying to fbissey:
It all works in sage-on-gentoo.
However I think there are a few points that should be taken into consideration for both numpy and scipy. Just before posting my spkg-install I edited it to remove a change that I now think is probably necessary. I had set FC to SAGE_FORTRAN... Why? Because numpy tries to autodetect a fortran compiler and will take a gnu compiler first. Unless you set FC. Which means if Dave tries to compile sage with sunstudio on a machine that has also gfortran he is in for a world of hurt. So I think we should either set FC to SAGE_FORTRAN in numpy and possibly scipy. The other option is to set it globally but that may cause problems on OSX.
Thoughts? Francois
A very pragmatic thougt: If it doesn"t hurt to set FC to SAGE_FORTRAN, why not? But I think it"s better not to do it globally. Because then we could cause more problems then we solve. Can you give a modified spkg-install? I can try it out over night then.
But I think it would also be good to get some feedback for Solaris and OSX for the current versions. Then we could decide to keep the current, or to take the other.
comment:35 follow-up: 36 Changed 12 years ago by
Good news: It seems that the problem from numpy-1.5.0 has been resolved http://projects.scipy.org/numpy/ticket/1605 I don't think it's a big deal. Shall I give it a try, or should we stick to 1.4.1 for now?
comment:36 Changed 12 years ago by
Replying to maldun:
Good news: It seems that the problem from numpy-1.5.0 has been resolved http://projects.scipy.org/numpy/ticket/1605 I don't think it's a big deal. Shall I give it a try, or should we stick to 1.4.1 for now?
...or wait til 1.5.1 is out?
comment:37 Changed 12 years ago by
May be wait until numpy-1.5.1. it is not a big deal right now I guess. May be we should discuss it on sage-devel. At the moment we don't really have much testing apart from the fact I unleashed the upgrade to numpy-1.4.1/scipy-0.8 on sage-on-gentoo users. I am not sure we can believably merge this in 4.5.3 unless it takes more than one week before it is released. If we target 4.5.3 I say we stay with what we have now, if we target 4.6 which should be a little further down the track we go for 1.5.x.
comment:38 Changed 12 years ago by
I updated my spkg_install as you requested. The other thing about using sage_fortran, that I had forgotten to change back when I removed it, is that it includes "-fPIC". I didn't check the fortran spkg but hopefully the work done by Dave to set the correct pic flag is picked up in sage_fortran.
If we don't adopt this, FFLAG="-fPIC" (or whatever mechanism Dave came up with) should be added in my previous version.
comment:39 Changed 12 years ago by
Replying to fbissey:
May be wait until numpy-1.5.1. it is not a big deal right now I guess. May be we should discuss it on sage-devel. At the moment we don't really have much testing apart from the fact I unleashed the upgrade to numpy-1.4.1/scipy-0.8 on sage-on-gentoo users. I am not sure we can believably merge this in 4.5.3 unless it takes more than one week before it is released. If we target 4.5.3 I say we stay with what we have now, if we target 4.6 which should be a little further down the track we go for 1.5.x.
Ok I think we should do the following: Let's stick to 1.4.1 since it is necessary for scipy 0.8. and quite well tested yet, in contrary to 1.5.x and it seems to work. Especially since the update from scipy 0.7 to 0.8 holds a lot of changes, and we don't know if they build some more bugs into numpy 1.5.1... If it turns out that updating from 1.4.1 to 1.5.1 is no problem, then well... packing a new package would'nt be the prob I think.
@new spkg: Ok I will give it a try tonight!
comment:40 follow-up: 41 Changed 12 years ago by
There's a lot of packages etc. on this ticket. Can someone provide a concise list of what would be needed to apply on a given platform? (For instance, in a very cursory glance, the networkx package being here mystifies.)
comment:41 Changed 12 years ago by
Replying to kcrisman:
There's a lot of packages etc. on this ticket. Can someone provide a concise list of what would be needed to apply on a given platform? (For instance, in a very cursory glance, the networkx package being here mystifies.)
Needed are: http://code.google.com/p/spkg-upload/downloads/detail?name=numpy-1.4.1.spkg http://code.google.com/p/spkg-upload/downloads/detail?name=scipy-0.8.spkg
after installing numpy, one needs so execute sage -ba, or else one get's runtime warnings.
You also need networkx-1.2. (the other networkx is just a small hack for testing, because 1.2. didn't build correctly on my machine, but this is obsolete now, since networkx-1.2 is merged into sage 4.3.alpha1)
comment:42 Changed 12 years ago by
Description: | modified (diff) |
---|
comment:43 Changed 12 years ago by
Description: | modified (diff) |
---|
comment:44 Changed 12 years ago by
I gave the links and (current) build instructions into the discription, so that people can find the latest versions more quickly
comment:45 Changed 12 years ago by
Description: | modified (diff) |
---|
comment:46 Changed 12 years ago by
Description: | modified (diff) |
---|
Included direct links to files in description.
comment:47 Changed 12 years ago by
Milestone: | sage-4.5.3 → sage-4.6 |
---|
comment:48 Changed 12 years ago by
Description: | modified (diff) |
---|
comment:49 Changed 12 years ago by
This seems to work fine on Mac OS X 10.6.4 Intel. I will try to test it tomorrow on a PowerPC machine. Note: I haven't looked at the spkg installs or anything, this is just a data point with regard to whether it works, not a review.
comment:50 follow-up: 51 Changed 12 years ago by
This fails very early on in Cygwin with
building library "npymath" sources creating build/src.cygwin-1.7.5-i686-2.6/src conv_template:> build/src.cygwin-1.7.5-i686-2.6/src/npy_math.c error: src/npy_math.c.src: No such file or directory Error building numpy.
comment:51 Changed 12 years ago by
Replying to mhansen:
This fails very early on in Cygwin with
building library "npymath" sources creating build/src.cygwin-1.7.5-i686-2.6/src conv_template:> build/src.cygwin-1.7.5-i686-2.6/src/npy_math.c error: src/npy_math.c.src: No such file or directory Error building numpy.
Ok I think I have the picture: some files from /src/numpy/core/src moved to subfolders this have to be changed in the cygwin-core-setup.py
I'm installing already cygwin on my laptop and look if I get it up and running, but if anyone could help out I would be thankful!
comment:52 follow-up: 53 Changed 12 years ago by
I've tested it out, and everything works if you just remove the cygwin-core-setup.py patch.
comment:53 follow-up: 54 Changed 12 years ago by
Replying to mhansen:
I've tested it out, and everything works if you just remove the cygwin-core-setup.py patch.
I was just about to upload a patch... But great this is even better than my solution! (and simpler) The reason why it fails is just that cygwin-core-setup.py is outdated.
one question: (because I'm quite the noob...) If I want to upload a modified version of numpy-1.4.1 do I have to overwrite the old version, or shall I rename it with numpy-1.4.1.p0.spkg?
comment:54 Changed 12 years ago by
Replying to maldun:
one question: (because I'm quite the noob...) If I want to upload a modified version of numpy-1.4.1 do I have to overwrite the old version, or shall I rename it with numpy-1.4.1.p0.spkg?
Overwrite the old one.
comment:55 Changed 12 years ago by
Thanx! Numpy is now updated with
- changes from fbissey to the installer (it worked for me)
- removed the cygwin-core-setup.py patch
comment:57 follow-up: 58 Changed 12 years ago by
I'm getting a corrupted package message with the packages right now while trying it on a different system. Did something get changed with respect to the file with this latest update?
comment:58 follow-up: 59 Changed 12 years ago by
Replying to kcrisman:
I'm getting a corrupted package message with the packages right now while trying it on a different system. Did something get changed with respect to the file with this latest update?
Yes we did some changes to the spkg-install. But I downloaded and installed the package right now again. So I don't know wehre the problem is lying?
comment:59 follow-up: 60 Changed 12 years ago by
Replying to maldun:
Replying to kcrisman:
I'm getting a corrupted package message with the packages right now while trying it on a different system. Did something get changed with respect to the file with this latest update?
Yes we did some changes to the spkg-install. But I downloaded and installed the package right now again. So I don't know wehre the problem is lying?
Did you use sage -pkg
to create it or just do some kind of compression? I get a message about tar: this does not look like a file archive
and tar: Archive contains obsolescent base-64 headers
. I just installed the optional biopython package on this machine to test things, so the machine shouldn't be the problem (and it built Sage 4.5.2 just fine).
comment:60 follow-up: 61 Changed 12 years ago by
Replying to kcrisman:
Replying to maldun:
Replying to kcrisman:
I'm getting a corrupted package message with the packages right now while trying it on a different system. Did something get changed with respect to the file with this latest update?
Yes we did some changes to the spkg-install. But I downloaded and installed the package right now again. So I don't know wehre the problem is lying?
Did you use
sage -pkg
to create it or just do some kind of compression? I get a message abouttar: this does not look like a file archive
andtar: Archive contains obsolescent base-64 headers
. I just installed the optional biopython package on this machine to test things, so the machine shouldn't be the problem (and it built Sage 4.5.2 just fine).
Ok I repacked it now with -pkg and uploaded it. Does it work now? And sorry for the newby question: what is the difference between -pkg and just compress it? Is sage using a different version of tar? Because in fact a spkg is noting else then a tar.gz with different ending.
comment:61 Changed 12 years ago by
Ok I repacked it now with -pkg and uploaded it. Does it work now? And sorry for the newby question: what is the difference between -pkg and just compress it? Is sage using a different version of tar? Because in fact a spkg is noting else then a tar.gz with different ending.
No, it's a little more than that - it has an SPKG.txt, it has Mercurial repositories, etc. True that the file type is the same. But sage -pkg tests several things, and a successful review would check all that. See here for more info.
comment:62 Changed 12 years ago by
Reviewers: | → Karl-Dieter Crisman |
---|
Hmm, when I open it manually it seems fine. Though when I run sage -pkg
on that new folder, I get
HG REPO: Unchecked in changes
So you'll need to fix that. Putting myself as a reviewer now.
Calling sage -f
on this new spkg I made *does* seem to work, for whatever reason. This doesn't make sense to me - why is tar
complaining about yours? Hopefully someone who knows about file systems will test this soon and explain why this causes a problem on Mac OS X 10.4 Tiger.
comment:63 follow-up: 65 Changed 12 years ago by
Okay, that worked. Scipy seems to be installing fine directly from the website, I'm not sure why this happened.
comment:64 Changed 12 years ago by
Thanx for the hints! I updated the mercurial entries now to the latest version. Uploaded changed packages. Everything should be fine now
comment:65 Changed 12 years ago by
Replying to kcrisman:
Okay, that worked. Scipy seems to be installing fine directly from the website, I'm not sure why this happened.
I have an Idea: different versions of .tar may cause problems (so far as I know) If you look at the file sizes only it seems that sage does a different kind of compression.
comment:66 follow-up: 67 Changed 12 years ago by
I get the following error with the Scipy for some reason.
creating build/temp.macosx-10.4-ppc-2.6/scipy/special creating build/temp.macosx-10.4-ppc-2.6/scipy/special/c_misc compile options: '-I/Users/student/Desktop/sage-4.5.2/local/lib/python2.6/site-packages/numpy/core/include -c' gcc: scipy/special/c_misc/fsolve.c scipy/special/c_misc/fsolve.c: In function 'false_position': scipy/special/c_misc/fsolve.c:58: warning: 'f3' may be used uninitialized in this function scipy/special/c_misc/fsolve.c:58: warning: 'x3' may be used uninitialized in this function gcc: scipy/special/c_misc/gammaincinv.c scipy/special/c_misc/gammaincinv.c:1:20: error: Python.h: No such file or directory In file included from /Users/student/Desktop/sage-4.5.2/local/lib/python2.6/site-packages/numpy/core/include/numpy/npy_math.h:5, from scipy/special/c_misc/gammaincinv.c:2: /Users/student/Desktop/sage-4.5.2/local/lib/python2.6/site-packages/numpy/core/include/numpy/npy_common.h:79:2: error: #error Must use Python with unicode enabled. scipy/special/c_misc/gammaincinv.c:1:20: error: Python.h: No such file or directory In file included from /Users/student/Desktop/sage-4.5.2/local/lib/python2.6/site-packages/numpy/core/include/numpy/npy_math.h:5, from scipy/special/c_misc/gammaincinv.c:2: /Users/student/Desktop/sage-4.5.2/local/lib/python2.6/site-packages/numpy/core/include/numpy/npy_common.h:79:2: error: #error Must use Python with unicode enabled. error: Command "gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/student/Desktop/sage-4.5.2/local/lib/python2.6/site-packages/numpy/core/include -c scipy/special/c_misc/gammaincinv.c -o build/temp.macosx-10.4-ppc-2.6/scipy/special/c_misc/gammaincinv.o" failed with exit status 1 Error building scipy.
Did you use some special encoding for some of your stuff?
comment:67 Changed 12 years ago by
Replying to kcrisman:
I get the following error with the Scipy for some reason.
creating build/temp.macosx-10.4-ppc-2.6/scipy/special creating build/temp.macosx-10.4-ppc-2.6/scipy/special/c_misc compile options: '-I/Users/student/Desktop/sage-4.5.2/local/lib/python2.6/site-packages/numpy/core/include -c' gcc: scipy/special/c_misc/fsolve.c scipy/special/c_misc/fsolve.c: In function 'false_position': scipy/special/c_misc/fsolve.c:58: warning: 'f3' may be used uninitialized in this function scipy/special/c_misc/fsolve.c:58: warning: 'x3' may be used uninitialized in this function gcc: scipy/special/c_misc/gammaincinv.c scipy/special/c_misc/gammaincinv.c:1:20: error: Python.h: No such file or directory In file included from /Users/student/Desktop/sage-4.5.2/local/lib/python2.6/site-packages/numpy core/include/numpy/npy_math.h:5, from scipy/special/c_misc/gammaincinv.c:2: /Users/student/Desktop/sage-4.5.2/local/lib/python2.6/site-packages/numpy/core/include/numpy/npy_common.h:79:2: error: #error Must use Python with unicode enabled. scipy/special/c_misc/gammaincinv.c:1:20: error: Python.h: No such file or directory In file included from /Users/student/Desktop/sage-4.5.2/local/lib/python2.6/site-packages/numpy/core/include/numpy/npy_math.h:5, from scipy/special/c_misc/gammaincinv.c:2: /Users/student/Desktop/sage-4.5.2/local/lib/python2.6/site-packages/numpy/core/include/numpy/npy_common.h:79:2: error: #error Must use Python with unicode enabled. error: Command "gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/student/Desktop/sage-4.5.2/local/lib/python2.6/site-packages/numpy/core/include -c scipy/special/c_misc/gammaincinv.c -o build/temp.macosx-10.4-ppc-2.6/scipy/special/c_misc/gammaincinv.o" failed with exit status 1 Error building scipy.
Did you use some special encoding for some of your stuff?
Nope. Perhaps something else did go wrong. I repacked it now on a different machine with a newer version of ubuntu and uploaded it. Hope this works. The first install worked for me.
comment:68 follow-up: 69 Changed 12 years ago by
Sadly, same error. This is probably on my end, but this is a supported architecture so it's important to know what went wrong. I don't think that the ./sage -ba
would have anything to do with it.
So I suspect that the "missing: Python.h
is the problem, as I've seen a few other things about this online (including ones like this one, where /include/Python.h definitely exists, here it's within the $SAGE_ROOT
directory). I wonder why it's not finding it this time?
comment:69 follow-up: 70 Changed 12 years ago by
Replying to kcrisman:
Sadly, same error. This is probably on my end, but this is a supported architecture so it's important to know what went wrong. I don't think that the
./sage -ba
would have anything to do with it.So I suspect that the "missing:
Python.h
is the problem, as I've seen a few other things about this online (including ones like this one, where /include/Python.h definitely exists, here it's within the$SAGE_ROOT
directory). I wonder why it's not finding it this time?
Question: Have you tried to give the direct path to the compiler? And does in OS X gcc points to /include/Python.h because in sage it's /include/python2.6/Python.h ?
comment:70 Changed 12 years ago by
Replying to maldun:
Replying to kcrisman:
Sadly, same error. This is probably on my end, but this is a supported architecture so it's important to know what went wrong. I don't think that the
./sage -ba
would have anything to do with it.So I suspect that the "missing:
Python.h
is the problem, as I've seen a few other things about this online (including ones like this one, where /include/Python.h definitely exists, here it's within the$SAGE_ROOT
directory). I wonder why it's not finding it this time?Question: Have you tried to give the direct path to the compiler?
No - how would I do that?
And does in OS X gcc points to /include/Python.h because in sage it's /include/python2.6/Python.h ?
I find that very unlikely, since everything else works fine and in general Sage builds fine on OS X 10.4-10.6. But sometimes things get mixed up, I'm sure. There aren't any weird env variables that would do that here, though, I don't think.
comment:71 follow-up: 72 Changed 12 years ago by
A question: do we impose python to be built with or without unicode support? If not, is the support enabled by default depending on the platform? To me it looks like numpy ships headers that are encoded with unicode and that your sage's python chock on them, that's the first and foremost error. distutils seem to be unable to work things properly after that.
Two things could be tried: 1) have python built with unicode support on OSX. 2) "vet" numpy to convert the headers in /Users/student/Desktop/sage-4.5.2/local/lib/python2.6/site-packages/numpy/core/include/numpy/ to plain text. Actually you probably can do that easily and manually right now on the installed headers.
Give it a spin and see if it works. If it does we'll have to do something about that piece of unicode one way or another.
comment:72 Changed 12 years ago by
Replying to fbissey:
A question: do we impose python to be built with or without unicode support? If not, is the support enabled by default depending on the platform? To me it looks like numpy ships headers that are encoded with unicode and that your sage's python chock on them, that's the first and foremost error. distutils seem to be unable to work things properly after that.
Two things could be tried: 1) have python built with unicode support on OSX. 2) "vet" numpy to convert the headers in /Users/student/Desktop/sage-4.5.2/local/lib/python2.6/site-packages/numpy/core/include/numpy/ to plain text. Actually you probably can do that easily and manually right now on the installed headers.
Umm... except that I have no idea how I might do that. What exact changes would I need to make to Python.h (and/or something else)? It seems unwise to make yet another python spkg change to deal with this; did numpy only recently (between whatever Sage has now and 1.4.1) start making its headers unicode? I am not exactly a C expert.
That said, 1) seems to be more canonical, though I guess if it makes it work, 2) could be an option. This is the sort of thing Leif and/or drkirkby usually have an informed opinion on...
Give it a spin and see if it works. If it does we'll have to do something about that piece of unicode one way or another.
comment:73 Changed 12 years ago by
2) would be good to find if this is indeed the problem. Doing it is another story. Do you have iconv on OSX? I would think so as it is needed by R amongst other.
so go into the right folder and try:
iconv -f UTF-8 -t ISO-8859-1 *.h
comment:75 follow-ups: 76 77 Changed 12 years ago by
In the spkg-install, there is the line
unset CFLAGS LDFLAGS CXXFLAGS SHAREDFLAGS
could this be somehow connected to our problem, because if we unset the flags, gcc doesn't point at Python.h in OS X?
comment:76 Changed 12 years ago by
Replying to maldun:
In the spkg-install, there is the line
unset CFLAGS LDFLAGS CXXFLAGS SHAREDFLAGScould this be somehow connected to our problem, because if we unset the flags, gcc doesn't point at Python.h in OS X?
Ok I reformulate it: Can we use the CFLAGS and LDFLAGS to give to tell gcc to link Python.h directly?
comment:77 Changed 12 years ago by
Replying to maldun:
In the spkg-install, there is the line
unset CFLAGS LDFLAGS CXXFLAGS SHAREDFLAGScould this be somehow connected to our problem, because if we unset the flags, gcc doesn't point at Python.h in OS X?
But this is not a change in the spkg-install script. Sage has built fine on this computer in the very recent past. The new spkg also works fine on other (newer) Macs.
From the numpy website
Reusing npymath In 1.3.0, we started putting portable C math routines in npymath library, so that people can use those to write portable extensions. Unfortunately, it was not possible to easily link against this library: in 1.4.0, support has been added to numpy.distutils so that 3rd party can reuse this library. See coremath documentation for more information.
This seems quite possibly relevant (given the error message). After all, if Python.h
isn't imported, then maybe the first error seen is that PY_UNICODE_DEF
or whatever it's called wouldn't be set... maybe?
I'm just trying to throw out an idea here, I am woefully uniformed when it comes to headers and things like that. But I feel like it has to be a change in Scipy or Numpy, since the previous Sage built fine on this machine.
comment:78 follow-up: 79 Changed 12 years ago by
I have another wild theory: Haven't you said, that the package build one time? And then not. Is this correct?
The only thing I changed was updating the mercurial files, and then commiting. Therefore I had to write the commit messages, and I do this in kate. kate's standard encoding is utf-8. ( My computer's language is german, so everything is utf-8) This (or just packing it, because the encoding on my machine is not unicode) did change something to the encoding. What happens if you untar the package use the iconv from up there, and repack it on your machine?
comment:79 Changed 12 years ago by
Replying to maldun:
I have another wild theory: Haven't you said, that the package build one time? And then not. Is this correct?
The only thing I changed was updating the mercurial files, and then commiting. Therefore I had to write the commit messages, and I do this in kate. kate's standard encoding is utf-8. ( My computer's language is german, so everything is utf-8) This (or just packing it, because the encoding on my machine is not unicode) did change something to the encoding. What happens if you untar the package use the iconv from up there, and repack it on your machine?
I don't think so but it can be tested I guess. I think Karl is on something, I am having a look at scipy's spkg-install to see if there is something that should be done there.
I have an endianess problem with numpy-1.4.1 on linux ppc, funny it works on ppc OSX: http://projects.scipy.org/numpy/ticket/1403 and http://bugs.gentoo.org/show_bug.cgi?id=306237 on the other hand numpy-1.5.0 built on my test machine. I am trying to find the changeset that made it possible to see if it is worth backporting.
comment:80 Changed 12 years ago by
Replying to kcrisman:
Same error.
It may sound unrelated but what fortran compiler are you using in your sage install? g95 or gfortran?
In any case I think some of the patches in scipy would need rebasing. But the fact of the matter is that they are g95 dependent and we dropped g95 - I already did so in numpy so we should do it as well in scipy.
comment:81 Changed 12 years ago by
I don't know exactly, but recall that Sage still includes Fortran compilers for Mac (see here). This is an spkg (sage-fortran...). From the spkg readme:
= gFortran = == Description == G95 is a stable, production Fortran 95 compiler available for multiple CPU architectures and operating systems. == Upstream contacts == URL: http://ftp.g95.org http://www.g95.org
So I don't know how to answer your question - apparently it's both gfortran and g95 :) The spkg-install has more details on this, but it apparently depends on the version (there are different .bz2 files for different Macs); my old PPC Mac would have used g95, I think. Is there some technical reason we should drop this?
Responding to the other comment, sadly, although I thought Scipy installed correctly at first, my machine is REALLY slow, so eventually it turned out that it didn't. I succeeded on a newer Intel Mac, but that is unrelated.
comment:82 follow-up: 83 Changed 12 years ago by
Status: | needs_review → needs_work |
---|
See comment 10 by Dave. g95 can be removed. Unless they forgot about OSX ppc. For my question, what does
$SAGE_LOCAL/bin/sage_fortran --version
says? I am guessing we have gfortran binaries for all platforms. Dropping g95 means that we can remove some (ugly) patches for numpy/scipy there are probably other valid reasons to do so but generally speaking that's less work.
I removed all patches that were applied for g95 in numpy, so if you have g95 the problems may come from there.
I share your pain about building on ppc (although I dropped OS X sometimes ago, this is now linux only and gentoo so everything or almost install from sources. openoffice-3.1 clocks in at 27 hours to build). Did you mean that numpy failed or restating that scipy failed in your latest comment?
If numpy fails for you as well on OSX ppc we may have a strong motivation to move to 1.5.0 again.
I am changing this to need_works until we work out what's wrong in your case.
comment:83 Changed 12 years ago by
Replying to fbissey:
If numpy fails for you as well on OSX ppc we may have a strong motivation to move to 1.5.0 again.
If you want I can prepare a package for 1.5.0 with the patch from the numpy trac, and run the doctests, in the evening.
comment:84 Changed 12 years ago by
Numpy was fine in all cases; Scipy was what didn't build. drkirkby's comment "There are g95 binaries in the Fortran package in Sage, but William said they can be removed." may not be true.
The command that supposedly checks which version I have returns with an error:
local/bin/sage_fortran: line 3: sage_fortran.bin: command not found
This also happens on my (successful) build of 4.6.prealpha4, so maybe the command is wrong?
Incidentally, does this ticket do anything with respect to #7831 and #8010? Just curious.
comment:85 Changed 12 years ago by
Okay, I finally figured out how to check this - I had to run the binary directly, the scripts didn't work.
G95 (GCC 4.0.3 (g95 0.91!) Jun 4 2007)
etcetera. On my Macintel, I get
GNU Fortran (GCC) 4.2.3 Copyright (C) 2007 Free Software Foundation, Inc.
So yes, it is using G95 on Tiger.
comment:86 Changed 12 years ago by
It would be great if this is the problem and it can somehow be solved!
Another thing: I builded numpy-1.5.0 with the new patch now. There only failed some doctests because numpy is throwing some new warnings. Want to give it a try? Perhaps the problem is then obsolete with scipy 0.8. also? I personally would prefer to stick to 1.4.1, but well it is like it is...
comment:87 follow-up: 88 Changed 12 years ago by
If you post it, I can try it - though it might be after the weekend, depending on my physical access to that particular box.
fbissey - can you explain (in layman's terms) how having the wrong fortran compiler might affect things? It does confuse me :)
comment:88 follow-up: 91 Changed 12 years ago by
Replying to kcrisman:
If you post it, I can try it - though it might be after the weekend, depending on my physical access to that particular box.
fbissey - can you explain (in layman's terms) how having the wrong fortran compiler might affect things? It does confuse me :)
Up and ready!
Yes this would be interesting! I don't really get it either because it looks more like a C problem to me?
comment:89 Changed 12 years ago by
comment:90 Changed 12 years ago by
It's more of a hunch than anything special I'll admit to that. The numpy/scipy connection seems to be fragile so I am not discounting it. In actual fact the fortran compiler is used only to compile one file in the whole of numpy (lapack_lite.so). I am more worried that it mixes up the toolchain it prepares for scipy.
One thing I'd like to see however is a complete build log for scipy not just the failing bit. There may be clues earlier in the process.
comment:91 follow-up: 92 Changed 12 years ago by
Replying to maldun:
Replying to kcrisman:
If you post it, I can try it - though it might be after the weekend, depending on my physical access to that particular box.
fbissey - can you explain (in layman's terms) how having the wrong fortran compiler might affect things? It does confuse me :)
Up and ready!
Sorry, fails with exactly the same error (on a different box with similar specs, same G95 in its Sage). So for now probably stick with the better-tested 1.4.1 for numpy. Hopefully we can figure out what's going on here.
The logs for the numpy (1.5.0) and scipy are in this directory, from the second computer. Happy hunting! By the way, Python.h is clearly working fine elsewhere in these logs, and sage_fortran compiles all kinds of neat stuff up to that point.
comment:92 follow-up: 93 Changed 12 years ago by
Replying to kcrisman:
Replying to maldun:
Replying to kcrisman:
If you post it, I can try it - though it might be after the weekend, depending on my physical access to that particular box.
fbissey - can you explain (in layman's terms) how having the wrong fortran compiler might affect things? It does confuse me :)
Up and ready!
Sorry, fails with exactly the same error (on a different box with similar specs, same G95 in its Sage). So for now probably stick with the better-tested 1.4.1 for numpy. Hopefully we can figure out what's going on here.
The logs for the numpy (1.5.0) and scipy are in this directory, from the second computer. Happy hunting! By the way, Python.h is clearly working fine elsewhere in these logs, and sage_fortran compiles all kinds of neat stuff up to that point.
Thanks for the logs. Sorry to be a bother but would you have old build logs for scipy-0.7 as well? Since I don't have a ppc OSX setup that would be very useful to have a successful log even from an older version of scipy.
comment:93 follow-up: 94 Changed 12 years ago by
Sorry, fails with exactly the same error (on a different box with similar specs, same G95 in its Sage). So for now probably stick with the better-tested 1.4.1 for numpy. Hopefully we can figure out what's going on here.
The logs for the numpy (1.5.0) and scipy are in this directory, from the second computer. Happy hunting! By the way, Python.h is clearly working fine elsewhere in these logs, and sage_fortran compiles all kinds of neat stuff up to that point.
Thanks for the logs. Sorry to be a bother but would you have old build logs for scipy-0.7 as well? Since I don't have a ppc OSX setup that would be very useful to have a successful log even from an older version of scipy.
Look in the same place, just posted it. So nice to be operating at 1.25 GHz instead of 700 MHz...
Also, interestingly, I now have a "mixed" installation on this computer:
sage: import numpy sage: numpy.version.version '1.5.0' sage: import scipy sage: scipy.version.version '0.7.0'
At least some of the relevant tests seem to pass with this, though of course I get the RunTimeErrors? mentioned above and did Inf
and inf
change places? I know little about numpy and scipy, though. Anyway, that's better than on the other box, whose issues with bad tarballs seem to not exist on this box.
comment:94 follow-up: 95 Changed 12 years ago by
Replying to kcrisman:
Look in the same place, just posted it. So nice to be operating at 1.25 GHz instead of 700 MHz...
Also, interestingly, I now have a "mixed" installation on this computer:
sage: import numpy sage: numpy.version.version '1.5.0' sage: import scipy sage: scipy.version.version '0.7.0'At least some of the relevant tests seem to pass with this, though of course I get the RunTimeErrors? mentioned above and did
Inf
andinf
change places? I know little about numpy and scipy, though. Anyway, that's better than on the other box, whose issues with bad tarballs seem to not exist on this box.
Found the logs, I may have a theory and the logs may help to (in)validate it. Not sure about Inf/inf either. scipy should work fine with numpy-1.5 - once rebuilt, and sage needs rebuilding too. That may get rid of some of these runtime errors. scipy-0.8 needs numpy-1.4 minimum on the other hand.
comment:95 Changed 12 years ago by
Also, interestingly, I now have a "mixed" installation on this computer:
sage: import numpy sage: numpy.version.version '1.5.0' sage: import scipy sage: scipy.version.version '0.7.0'At least some of the relevant tests seem to pass with this, though of course I get the RunTimeErrors? mentioned above and did
Inf
andinf
change places? I know little about numpy and scipy, though. Anyway, that's better than on the other box, whose issues with bad tarballs seem to not exist on this box.Found the logs, I may have a theory and the logs may help to (in)validate it. Not sure about Inf/inf either. scipy should work fine with numpy-1.5 - once rebuilt, and sage needs rebuilding too. That may get rid of some of these runtime errors. scipy-0.8 needs numpy-1.4 minimum on the other hand.
Yup, tests in relevant modules are passing, with only the RuntimeWarning
s (not errors, sorry for that) in general marring things (as noted, one or two other very minor things). Anyway, none of this not compiling nonsense, so something must have changed in Scipy 0.8 from 0.7 - maybe something that used that change mentioned above about reusing npymath
. There were indeed a couple changes to the gammaincinv
function in Scipy, if you look at their changesets, but as a non-expert how those could have resulted in this Python.h/unicode error is totally beyond me.
comment:96 Changed 12 years ago by
Interestingly enough on Gentoo we have -I/usr/include/python2.6 added to the compilation flags everywhere. I wonder if just finding a way of adding -I${SAGE_LOCAL}/include/python2.6 in sage would help at all. The mystery then would be "how did it work before?" Furthermore why is it not needed for the earlier C compilations.... or on other platforms? May be we are using system python headers without knowing it?
I will work on that over the week end. Hopefully by Monday I will have something you can test.
comment:97 Changed 12 years ago by
It turns out to be more subtle than I thought and I don't know yet why.
It is not easy to add -I${SAGE_LOCAL}/include/python2.6 to compile options, in any case many other compilation lines actually have it already. After careful checking, I found that this particular option is not used in any scipy-0.7.x log I had on sage or Gentoo for these particular objects. But I see it in scipy-0.8.0 in Gentoo.
I am messing up with some of my system at the moment so I cannot generate a successful build log of scipy-0.8.0 from sage at the moment. But it would be interesting to know if this particular compilation option has been added in the successful builds.
comment:98 Changed 12 years ago by
OK - so there is a fair difference on that file between 0.7.2 and 0.8.0.
headers in gammaincinv.c in 0.7.2
#include <stdio.h> #include <math.h> #include "../cephes.h" #undef fabs #include "misc.h"
in 0.8.0
#include <Python.h> #include <numpy/npy_math.h> #include <stdio.h> #include <math.h> #include "../cephes.h" #undef fabs #include "misc.h"
So it will be very interesting to get a successful build log of scipy-0.8.0 in sage for inspection. A quick and dirty fix would be to change <Python.h> to <python2.6/Python.h> and I am not completely sure it would work either.
There are a few more Python.h header in that folder so just fixing that one may not work:
grep -r "Python\.h" * amos_wrappers.h:#include "Python.h" cdf_wrappers.h:#include "Python.h" cephes/sici.c:#include <Python.h> cephes/mconf.h:#include <Python.h> _cephesmodule.c:#include "Python.h" c_misc/gammaincinv.c:#include <Python.h> lambertw.c:#include "Python.h" orthogonal_eval.c:#include "Python.h" specfun_wrappers.h:#include "Python.h" toms_wrappers.h:#include "Python.h" ufunc_extras.h:#include "Python.h"
considering the SConscript the files in the cephes subfolder will probably be trouble too.
comment:99 Changed 12 years ago by
But it would nice to know anyway. If it doesn't work with gammaincinv.c, at least the error message would change. Providing a patch would then be easy for me.
Ironically I already had asked if this had could been the problem... The only thing what me disturbs is the this entry in the error log of kcrisman:
...y/npy_common.h:79:2: error: #error Must use Python with unicode enabled.
This sounds more like that unicode isn't enabled, and perhaps this is related to native python installation on the system, and the gcc uses this header instead that from sage. I think to be absolutely sure, we should try also to set the path really to Python.h in the sage install in the header, if setting to <python2.6/Python.h> doesn't work.
@fbissey Another question: would it help you, if you see my install log from ubuntu?
comment:100 Changed 12 years ago by
No need for extra logs. I found the culprit and why it affects only Karl's ppc machine. I point the finger to g95! or rather the fact that the patches that are installed if g95 is found should have been rebased.
Basically the file special/setup.py is replaced by a version from scipy-0.7.0 which doesn't need the python header. Could you please use spkg-install.2 attached in scipy-0.8.0 it doesn't use the patches for g95. It should work without special patch in my opinion anyway.
comment:101 Changed 12 years ago by
So I've packed the scipy with your spkg-install, but this time it doesn't work for me. I get this error:
Host system uname -a: Linux math181 2.6.31-22-generic #63-Ubuntu SMP Thu Aug 19 00:23:50 UTC 2010 x86_64 GNU/Linux **************************************************** **************************************************** CC Version gcc -v Using built-in specs. Target: x86_64-linux-gnu Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.4.1-4ubuntu9' --with-bugurl=file:///usr/share/doc/gcc-4.4/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --enable-shared --enable-multiarch --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.4 --program-suffix=-4.4 --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc --disable-werror --with-arch-32=i486 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu Thread model: posix gcc version 4.4.1 (Ubuntu 4.4.1-4ubuntu9) **************************************************** invalid command name 'config_fc --noopt --noarch' Error building scipy. real 0m2.296s
comment:102 Changed 12 years ago by
I think I have gone overboard with double quotes, attaching a new version shortly.
comment:103 Changed 12 years ago by
Ok. I deleted all outdated patches like you suggested (initially I let them in, because they didn't any harm, and since I didn't know much about every detail of the patches, I first tried out if it works, but of course it's better to throw them out) and it builds without problem. But with the old spkg. the spkg-install.2 you attached still, with the same error, because my machine doesn't know the command config_fc --noopt --noarch:
invalid command name 'config_fc --noopt --noarch'
could it be that I have to install something on my system? Or should we solve this with an If clause?
comment:104 follow-up: 106 Changed 12 years ago by
ok I solved it: You forgot the "-" before the config =)
comment:105 Changed 12 years ago by
I think I found the problem, I think the shell was not set up properly.
I mean, these are options passed to setup.py not separate commands. They shouldn't be interpreted as such and aren't in the numpy spkg-install, so there are no reason why they should be in scipy. But the shells for the two spkg-install were different so I think that may be a point of bash/shell semantic. In which case I am curious to find out what I am supposed to do in sh. There is a new spkg-install.2 for you to try.
comment:106 Changed 12 years ago by
Replying to maldun:
ok I solved it: You forgot the "-" before the config =)
Ok you posted while I was answering. That may pass but I am not sure that does what it's supposed to do. There shouldn't be a "-" in front. I suspect it is now just ignored.
comment:107 Changed 12 years ago by
ok I was to quick... indeed it builded, but not for too long: it took it a -c"onfig_fc"... so it didn't ignore, but missinterpreted it.
But the error remains. I don't really get it is there something different with ubuntu's gcc?
comment:108 Changed 12 years ago by
Ah okay: I looked into the scipy docu: This option seems to be for numpy, not for scipy. http://projects.scipy.org/numpy/wiki/DistutilsDoc#specifing-config-fc-options-for-libraries-in-setup-py-script
Because numpy builded with that command. Does it build on your system?
comment:109 Changed 12 years ago by
OK. I thought it was used in scipy as well. There seems to be 2 files with it, and it shouldn't have an impact. You can remove it then. Strange it doesn't generate errors in Gentoo.
comment:111 Changed 12 years ago by
Help! I couldn't work on this over the weekend, and now I'm not quite sure what I should do to test :) Should I just try installing the new Scipy 0.8, or what? I am very glad that it seems you've figured out what G95 didn't like, though. It would also be nice to see a diff between the spkg-install
scripts.
comment:112 Changed 12 years ago by
Ok short summary: 1) Try the new scipy 2) If this doesn't work try to change the header in gammaincinv.c in the scipy sources from <Python.h> to <python2.6/Python.h> 3) If this results in no change, rewrite the header that it takes the sage header directly means change <Python.h> to "whateverpathitmaybe/python2.6/Python.h" and look if the error message changes. (because there are other files affected, but this should be no prob to patch)
If it's really only the location of the header patching wouldn't be a problem. And Thanx for all the trouble!
I have the diffs made for on my other machine, I can post them tomorrow!
Changed 12 years ago by
Diff of the spkg installs; The lines wich concern config fc are removed due problems in ubuntu
comment:113 follow-up: 118 Changed 12 years ago by
Well, that seems to have helped...
gcc: scipy/special/c_misc/gammaincinv.c
and it keeps going!
So it was the patches for G95 that were the problem. Which one in particular (which command/line) caused the header confusion, do you think? I'm just curious, and don't know enough about this to say.
By the way, I'm seeing things like
compile options: '-I/Users/student/Desktop/sage-4.5.2/local/lib/python2.6/site-packages/numpy/core/include -I/Users/student/Desktop/sage-4.5.2/local/include/python2.6 -c'
so hopefully that is also a positive sign.
comment:114 Changed 12 years ago by
Success! Now I'll set SAGE_CHECK=yes
, retry numpy, do ./sage -ba
, retry scipy, and test the Sage library. It will take a LONG time but hopefully tomorrow I'll have a report for you.
comment:116 Changed 12 years ago by
Ah and don't forget the patch for the doctests in the attachement. Else there would 2 doctests fail due to output changes in numpy.
comment:117 follow-ups: 119 120 Changed 12 years ago by
I'm not worried about that, but I'll do it when I get there. Numpy just finished installing successfully.
Question:
# Program around a bug in SciPY's distutils. unset CFLAGS python setup.py install ${NUMPY_FCONFIG}
I assume this is still needed in the numpy spkg-install
. Just checking.
Question: I know drkirkby is cc:ed on this ticket. Numpy seems to have a very easy to run test suite - see here. However, it requires the Nose package. Maybe this should be a separate ticket, but it would seem reasonable to include Nose, which is about 250 KB, so that we can run the numpy tests. I am curious as to how we run the scipy tests without it!
comment:118 Changed 12 years ago by
Replying to kcrisman:
Well, that seems to have helped...
gcc: scipy/special/c_misc/gammaincinv.cand it keeps going!
So it was the patches for G95 that were the problem. Which one in particular (which command/line) caused the header confusion, do you think? I'm just curious, and don't know enough about this to say.
Well, there are 4 patches that were applied in case g95 was found as the fortran compiler. The faulty one is special.setup.py (or possibly setup.py.special) which replaces the file setup.py in the folder scipy/special.
This setup.py file has changed quite a lot between version 0.7.0 and 0.8.0. The lines at fault:
# C libraries config.add_library('sc_c_misc',sources=[join('c_misc','*.c')]) config.add_library('sc_cephes',sources=[join('cephes','*.c')], include_dirs=[get_python_inc(), get_numpy_include_dirs()], macros=define_macros)
in version 0.7.0 and this is the code found in the "patch" file and
# C libraries config.add_library('sc_c_misc',sources=[join('c_misc','*.c')], include_dirs=[get_python_inc(), get_numpy_include_dirs()], macros=define_macros) config.add_library('sc_cephes',sources=[join('cephes','*.c')], include_dirs=[get_python_inc(), get_numpy_include_dirs()], macros=define_macros)
in version 0.8.0. This is a case were using patches rather copying files wholesale would have prevented the problem. Either the patch wouldn't have worked with the newer version and we would have known straight away or it would have applied with some fuzz without dommaging this particular section.
comment:119 Changed 12 years ago by
Replying to kcrisman:
I'm not worried about that, but I'll do it when I get there. Numpy just finished installing successfully.
Question:
# Program around a bug in SciPY's distutils. unset CFLAGS python setup.py install ${NUMPY_FCONFIG}I assume this is still needed in the numpy
spkg-install
. Just checking.Question: I know drkirkby is cc:ed on this ticket. Numpy seems to have a very easy to run test suite - see here. However, it requires the Nose package. Maybe this should be a separate ticket, but it would seem reasonable to include Nose, which is about 250 KB, so that we can run the numpy tests. I am curious as to how we run the scipy tests without it!
Actually Dave has started a thread about this a few weeks ago on the mailing list, you are welcome to add your opinion and revive the thread.
comment:120 Changed 12 years ago by
Replying to kcrisman:
I'm not worried about that, but I'll do it when I get there. Numpy just finished installing successfully.
Question:
# Program around a bug in SciPY's distutils. unset CFLAGS python setup.py install ${NUMPY_FCONFIG}I assume this is still needed in the numpy
spkg-install
. Just checking.
I thought I had asked maldun to remove this. It's working fine on Gentoo as it is, I am assuming this has been fixed upstream (possibly a while ago).
comment:121 follow-up: 122 Changed 12 years ago by
I added the
# Program around a bug in SciPY's distutils. unset CFLAGS
into the numpy spkg-install from the old one, just to be sure that we don't run into troubles, and actually it doesn't do any harm... but if you want I can remove it as well.
But didn't we agree that the
setup.py install ${NUMPY_FCONFIG
}
is necessary in numpy? I had to delete it from scipy or else it doesn't build anymore, but in numpy it's correct.
comment:122 Changed 12 years ago by
Replying to maldun:
I added the
# Program around a bug in SciPY's distutils. unset CFLAGSinto the numpy spkg-install from the old one, just to be sure that we don't run into troubles, and actually it doesn't do any harm... but if you want I can remove it as well.
We should remove it. If there is a problem - which I doubt, we can put it back.
But didn't we agree that the
setup.py install ${NUMPY_FCONFIG
} is necessary in numpy? I had to delete it from scipy or else it doesn't build anymore, but in numpy it's correct.
Yes! Leave it here.
comment:123 Changed 12 years ago by
Ok I removed the line.
I also thougth before posting I should check if there are some old rests in there, and I removed some patches that are possible(?) outdated. I think this could affect the cygwin installation, so it would be important to see if it still builds on all machines, or else I have to carefully merge the changed lines, from the old patches with the new versions of the files.
It works for me, so I hope I didn't cause some new problems
comment:124 Changed 12 years ago by
I have a different problem: When I upload the new Version, and try to download it I get the old one. Is there some confusion on the server?
comment:126 follow-ups: 127 129 Changed 12 years ago by
Status: | needs_work → needs_review |
---|
Ok - now this needs review!
I looked at the cygwin-lapack_lite-setup.py patch. The file patched hasn't changed between numpy-1.3.x and 1.4.1, so it can still be applied safely if needed.
In summary the only problems overall are:
- numpy 1.4.1 is broken on linux ppc (on the official list of supported OS - so bad) and alpha (not on the supported list - so OK).
- numpy-1.5.0 has an issues with cython that will be solved in 1.5.1 - shame because it also fixes linux ppc and alpha.
Anything to add?
comment:127 Changed 12 years ago by
Replying to fbissey:
Anything to add?
as mentioned above: The version of numpy-1.5.0 with the patch applied is ok. The only trouble I have are some doctests, which fails because numpy throws a "Division by zero encountered!" Exception, which makes them fail, although the results are correct. But I still don't know how to catch the exceptions.
comment:128 Changed 12 years ago by
I should say that while breaking on linux ppc is not good, I only know of one other person building sage on that platform. So it could take the backseat while problems with the 1.5 series are ironed out.
comment:129 follow-up: 133 Changed 12 years ago by
- numpy-1.5.0 has an issues with cython that will be solved in 1.5.1 - shame because it also fixes linux ppc and alpha.
Hmm, dropping an official thing is not so good. I think that one would want to post to the list about this - more likely wait for 1.5.1? Or is the fix for this easily backported?
Also, although I've been going for over 24 hours now on the tests (and several timeouts! so I'll have to run those with SAGE_TIMEOUT_LONG>3600
), I have only seen two errors, both like this:
sage -t -long "devel/sage/sage/graphs/digraph.py" ********************************************************************** File "/Users/student/Desktop/sage-4.5.2/devel/sage/sage/graphs/digraph.py", line 204: sage: DiGraph(A) Exception raised: Traceback (most recent call last): File "/Users/student/Desktop/sage-4.5.2/local/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/Users/student/Desktop/sage-4.5.2/local/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/Users/student/Desktop/sage-4.5.2/local/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_1[17]>", line 1, in <module> DiGraph(A)###line 204: sage: DiGraph(A) File "/Users/student/Desktop/sage-4.5.2/local/lib/python/site-packages/sage/graphs/digraph.py", line 364, in __init__ data = networkx.MultiDiGraph(data) File "/Users/student/Desktop/sage-4.5.2/local/lib/python2.6/networkx/classes/digraph.py", line 217, in __init__ convert.from_whatever(data,create_using=self) File "/Users/student/Desktop/sage-4.5.2/local/lib/python2.6/networkx/convert.py", line 157, in from_whatever if isinstance(thing,numpy.core.defmatrix.matrix) or \ AttributeError: 'module' object has no attribute 'defmatrix' **********************************************************************
Have you seen this? Clearly it's something to do with numpy, from the end - but it's not clear to me why it would be asking whether a module has defmatrix attribute.
comment:130 follow-up: 135 Changed 12 years ago by
I know this error. Since you use sage 4.5.2 you will also use networkx-1.0.1. See comments 24 and following or ticket #9854.
You have to either use Sage 4.5.3 which holds networkx-1.2, upgrade networkx-1.0.1 in your current version, or apply the networkx-1.0.1.p0.spkg in the attachement, which holds a simple patch.
The reason is that numpy.core.defmatrix moved to numpy.matrixlib.defmatrix.
comment:131 Changed 12 years ago by
@linux ppc I could work over the weekend the solve the failed doctests in numpy-1.5.0, because with the patch from the numpy developers the biggest problem i.e. the cython problem is solved. The only thing remains are some "Division by Zero Warnings" as mentioned above.
comment:132 Changed 12 years ago by
And another thing: If we aren't able to merge numpy and scipy to 4.6. in time, why don't we provide the packages as optional or experimental? Because they work on most systems, and it would be a waste to just throw them away.
comment:133 follow-up: 134 Changed 12 years ago by
Replying to kcrisman:
- numpy-1.5.0 has an issues with cython that will be solved in 1.5.1 - shame because it also fixes linux ppc and alpha.
Hmm, dropping an official thing is not so good. I think that one would want to post to the list about this - more likely wait for 1.5.1? Or is the fix for this easily backported?
Depending on what list you read, Linux on PPC may or may not be supported. I've tried to get an agreement before on what we do support and what we don't, but never got anywhere.
I'm personally in favor of keeping Sage building on less regular systems, as it often indicates problems. The Solaris port has found endless bugs in Sage - some of which have even according to William been serious.
I have an old IBM RS/6000 7025 F50 (Power PC) which I was setting up AIX on the other day. I contemlated whether I should put Linux on it too for testing Sage. But when I read Sage's README.txt, PowerPC was only supported on OS X, not Linux. So I did not bother. But the machine has space for 18 disk drives, so I could always add it!!
Dave
comment:134 Changed 12 years ago by
I forgot about the networkx change, thanks for the reminder.
- numpy-1.5.0 has an issues with cython that will be solved in 1.5.1 - shame because it also fixes linux ppc and alpha.
Hmm, dropping an official thing is not so good. I think that one would want to post to the list about this - more likely wait for 1.5.1? Or is the fix for this easily backported?
I'm personally in favor of keeping Sage building on less regular systems, as it often indicates problems. The Solaris port has found endless bugs in Sage - some of which have even according to William been serious.
I have an old IBM RS/6000 7025 F50 (Power PC) which I was setting up AIX on the other day. I contemlated whether I should put Linux on it too for testing Sage. But when I read Sage's README.txt, PowerPC was only supported on OS X, not Linux. So I did not bother. But the machine has space for 18 disk drives, so I could always add it!!
Why not? I've been trying to figure out how to get a thumb drive to boot my PPC machine on Ubuntu (and possibly other distros) but it turns out it's devilishly hard to get that to work, and I haven't had time to do it. I can't find an easy virtualization option either (neither xen nor anything else seem to really be particularly easy, if they even still work with such an old machine...)
And another thing: If we aren't able to merge numpy and scipy to 4.6. in time, why don't we provide the packages as optional or experimental? Because they work on most systems, and it would be a waste to just throw them away.
Certainly one could do this, but even that would require a positive review from someone. You might have to ask on the list about this - I've never heard of having an upgrade to a normal package as only optional.
comment:135 Changed 12 years ago by
You have to either use Sage 4.5.3 which holds networkx-1.2, upgrade networkx-1.0.1 in your current version, or apply the networkx-1.0.1.p0.spkg in the attachement, which holds a simple patch.
Incidentally, I also had to repackage that spkg, because you didn't use ./sage -pkg
for it either :) and it had unchecked in changes :) But clearly that is irrelevant for testing, since the upgrade is already in a later Sage.
Okay, with this upgrade those two tests pass. I am currently testing the other four failures, all of which were timeouts, all of which are in files I personally know to have VERY long -long
doctests, and all but one of which hopefully will pass with SAGE_TIMEOUT_LONG=10000
(there is one in interfaces/maxima.py
which always fails on OS X 10.4, as far as I know, which is due to tab-completion testing of Maxima never finishing, totally unrelated to this). I'll let you know if something *doesn't* work.
So I think that my job is done here (pointing out that I was testing the version from before the weekend. fbissey or drkirkby should comment on the spkg itself.
comment:136 follow-up: 137 Changed 12 years ago by
Well thanks Karl! Your testing was very useful. Since I have contributed to bits and pieces I think I should had myself as an author. But most of the heavy lifting as been done by maldun.
So what's left is the question of whether we go for numpy-1.4.1 and leave minor archs unsupported for a little bit. Or we wait for 1.5.1, which works on the minor archs in question and will play well with cython.
As an aside I have already pushed the upgrade in sage-on-gentoo (to avoid tree rote, we already have to keep two old packages that are otherwise removed from Gentoo) and we want to avoid that kind of stuff as much as possible. So the current code is out there and used by a few people.
After that there are question of details. In spkg-install I have set FC to ${SAGE_LOCAL}/bin/sage_fortran, with the idea that it was basically calling "gfortran -fpic" or the g95 equivalent. We have been talking about that very subject on sage-devel recently. Is it the best way to go? If one uses sunstudio (and I am planning to give a go) the correct flag would be -Kpic but would it be set up properly in sage_fortran? I doubt it.
So what would be the best course of action? Using the variable SAGE_FORTRAN for now and ask it to be set with the proper pic flag and hopefully drop it later on when FC is the mainstay?
comment:137 Changed 12 years ago by
Authors: | Stefan Reiterer → Stefan Reiterer, Francois Bissey |
---|
Replying to fbissey:
Well thanks Karl! Your testing was very useful. Since I have contributed to bits and pieces I think I should had myself as an author. But most of the heavy lifting as been done by maldun.
I think you are right. So I allowed myself to set you as co-author, since this whole thing wouldn't work without your help =)
So what's left is the question of whether we go for numpy-1.4.1 and leave minor archs unsupported for a little bit. Or we wait for 1.5.1, which works on the minor archs in question and will play well with cython.
I would suggest the following: If numpy 1.5.1 doesn't come out before Sage 4.6. let's take 1.4.1 (since ppc linux is not on the current supported list; see readme.tex of sage) I will start this weekend with correcting things of 1.5.0, since the current errors are new warnings, and I'm quite sure numpy 1.5.1 will throw those as well. When numpy 1.5.1 is out before 4.6. I will try to pack it, and then hopefully it is done in no time, since it shared the same probs.
As an aside I have already pushed the upgrade in sage-on-gentoo (to avoid tree rote, we already have to keep two old packages that are otherwise removed from Gentoo) and we want to avoid that kind of stuff as much as possible. So the current code is out there and used by a few people.
After that there are question of details. In spkg-install I have set FC to ${SAGE_LOCAL}/bin/sage_fortran, with the idea that it was basically calling "gfortran -fpic" or the g95 equivalent. We have been talking about that very subject on sage-devel recently. Is it the best way to go? If one uses sunstudio (and I am planning to give a go) the correct flag would be -Kpic but would it be set up properly in sage_fortran? I doubt it.
So what would be the best course of action? Using the variable SAGE_FORTRAN for now and ask it to be set with the proper pic flag and hopefully drop it later on when FC is the mainstay?
The question is how fast this change with the fortran compiler is done. I think we should wait a little bit now.
My question is: should this given a positive review, or is something still missing?
comment:138 follow-up: 139 Changed 12 years ago by
I'd like to check this out on sage-4.6.alpha1, but its unclear to me exactly what to do. Can someone summarize the current procedure for incorporating this ticket?
comment:139 Changed 12 years ago by
Replying to mhampton:
I'd like to check this out on sage-4.6.alpha1, but its unclear to me exactly what to do. Can someone summarize the current procedure for incorporating this ticket?
Install notes and links to the packages are in the discription of this ticket!
Quick and dirty: Install the packagages, do sage -ba to compile the whole source, apply the patch in the attachment for the doctests.
greez maldun
comment:140 Changed 12 years ago by
Just a quick note that I used this package in sage-4.6alpha1 and it didn't break any tests on linux-x86, I haven't heard of any problems on linux-amd64 from a friend who was also testing this. So it applies cleanly on 4.6alpha1.
comment:141 follow-up: 142 Changed 12 years ago by
Dave pointed me to this ticket the day before yesterday (and I read it from the beginning). ;-)
Despite that, it's unclear to me which diffs are currrent, and the description should IMHO be updated wrt. what's to be done to review this.
./sage -ba
is also in my opinion not a solution to get dependent extension modules updated, i.e. if ./sage -b
isn't sufficient, dependencies should be added to module_list.py
.
comment:142 Changed 12 years ago by
Description: | modified (diff) |
---|
Replying to leif:
Dave pointed me to this ticket the day before yesterday (and I read it from the beginning). ;-)
Despite that, it's unclear to me which diffs are currrent, and the description should IMHO be updated wrt. what's to be done to review this.
./sage -ba
is also in my opinion not a solution to get dependent extension modules updated, i.e. if./sage -b
isn't sufficient, dependencies should be added tomodule_list.py
.
I cleaned up the discription a little bit, and deleted the obsolete remark about networkx-1.2
I don't know if this is a good Idea because this is a whole bunch of files we are talking about, which have to be added. Perhaps you won't save much time either when compiling. I first tried this too, but after about 2 hours searching, I got somehow tired of this... And after merging numpy > 1.3.x into Sage this has not to be done anymore. All versions of 1.5.x builded without problems, and didn't need -ba anymore. You only have to do this when big changes are happening, and this is rather occassionally.
package with a linux ppc patch is out: http://code.google.com/p/spkg-upload/downloads/detail?name=numpy-1.4.1.p0.spkg hope it works this time!
I have now workable spkg files for the both packages. Scipy worked well. Numpy need some work, but it's functioning.
I host the files at megaupload, at: Numpy: http://www.megaupload.com/?d=6GCFZD65 Scipy: http://www.megaupload.com/?d=JORT40DK
I think the problem is the same as described in Trac # 7791 (http://trac.sagemath.org/sage_trac/ticket/7791 )
I get the following errors:
What's this? Has someone hardcoded the sizes of the routines?