Opened 11 years ago

Last modified 11 years ago

#9808 closed task

Upgrade numpy to 1.5.0 and scipy to 0.8 — at Version 197

Reported by: maldun Owned by: maldun
Priority: major Milestone: sage-4.6.1
Component: packages: standard Keywords: numpy, scipy
Cc: Merged in:
Authors: Stefan Reiterer, Francois Bissey Reviewers: Karl-Dieter Crisman, David Kirkby, Leif Leonhardy, Francois Bissey
Report Upstream: Fixed upstream, but not in a stable release. Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by maldun)

The packages can be found found under:

http://sage.math.washington.edu/home/palmieri/misc/numpy-1.5.0.spkg http://code.google.com/p/spkg-upload/downloads/detail?name=scipy-0.8.spkg

(NumPy? 1.4.1 can also be found under http://code.google.com/p/spkg-upload/downloads/detail?name=numpy-1.4.1.spkg when needed)

Files are found with direct links:

http://spkg-upload.googlecode.com/files/scipy-0.8.spkg

Important notes: after installing numpy, one needs to execute sage -ba, or else one get's runtime warnings.

trac_9808_numpy_doctest_change.patch in the attachment has to be applied, in order to get all doctests running because some of the output has changed.

For reviewers: changes.txt holds a summary of all changes with reference to the diffs, and links to other tickets

Change History (205)

comment:1 Changed 11 years ago by maldun

  • Status changed from new to needs_work

I have now workable spkg files for the both packages. Scipy worked well. Numpy need some work, but it's functioning.

I host the files at megaupload, at: Numpy: http://www.megaupload.com/?d=6GCFZD65 Scipy: http://www.megaupload.com/?d=JORT40DK

I think the problem is the same as described in Trac # 7791 (http://trac.sagemath.org/sage_trac/ticket/7791 )

I get the following errors:

/home/maldun/sage/sage-4.5.2/local/lib/python2.6/site-packages/sage/plot/all.py:22: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
  from complex_plot import complex_plot
/home/maldun/sage/sage-4.5.2/local/lib/python2.6/site-packages/sage/plot/all.py:22: RuntimeWarning: numpy.flatiter size changed, may indicate binary incompatibility
  from complex_plot import complex_plot
/home/maldun/sage/sage-4.5.2/local/lib/python2.6/site-packages/sage/plot/plot3d/implicit_plot3d.py:5: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
  from implicit_surface import ImplicitSurface
/home/maldun/sage/sage-4.5.2/local/lib/python2.6/site-packages/sage/plot/plot3d/implicit_plot3d.py:5: RuntimeWarning: numpy.flatiter size changed, may indicate binary incompatibility
  from implicit_surface import ImplicitSurface
/home/maldun/sage/sage-4.5.2/local/lib/python2.6/site-packages/sage/calculus/all.py:17: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
  from riemann import Riemann_Map
/home/maldun/sage/sage-4.5.2/local/lib/python2.6/site-packages/sage/calculus/all.py:17: RuntimeWarning: numpy.flatiter size changed, may indicate binary incompatibility
  from riemann import Riemann_Map
/home/maldun/sage/sage-4.5.2/local/lib/python2.6/site-packages/sage/calculus/all.py:19: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
  from interpolators import polygon_spline, complex_cubic_spline
/home/maldun/sage/sage-4.5.2/local/lib/python2.6/site-packages/sage/calculus/all.py:19: RuntimeWarning: numpy.flatiter size changed, may indicate binary incompatibility
  from interpolators import polygon_spline, complex_cubic_spline
/home/maldun/sage/sage-4.5.2/local/lib/python2.6/site-packages/sage/stats/hmm/all.py:8: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
  from hmm  import DiscreteHiddenMarkovModel
/home/maldun/sage/sage-4.5.2/local/lib/python2.6/site-packages/sage/stats/hmm/all.py:8: RuntimeWarning: numpy.flatiter size changed, may indicate binary incompatibility
  from hmm  import DiscreteHiddenMarkovModel

What's this? Has someone hardcoded the sizes of the routines?

comment:2 Changed 11 years ago by maldun

  • Summary changed from Upgrade numpy to 1.5b and scipy to 0.8 to Upgrade numpy to 1.5.0rc1 and scipy to 0.8

comment:3 follow-up: Changed 11 years ago by mhansen

I don't think that we should upgrade to 1.5.0rc1 -- we should do 1.4.1 for now and wait until 1.5 is released.

comment:4 in reply to: ↑ 3 ; follow-up: Changed 11 years ago by fbissey

Replying to mhansen:

I don't think that we should upgrade to 1.5.0rc1 -- we should do 1.4.1 for now and wait until 1.5 is released.

That has to be double checked but maldun says he needs features in 1.5. It may be that the features he wants are 1.4.x and he doesn't know. Do we know roughly when 1.5 will be released? They are at 1.5rc1 9 days after beta2 so it could very well be upon us in a very short time.

comment:5 Changed 11 years ago by maldun

The warnings go away after rebuilding the source. And the doctests only throw warnings but there don't seem to be errors, and the output is correct

@mhansen, fbissey I think numpy has the same issues since version 1.4. and I'm quite sure that resolving it in 1.5.0rc1 would, solve the problem with 1.4.1. too. (and perhaps also with 1.5. later) So I suggest the following procedure: I solve the issues with the latest. Dobule check if 1.4.1 also works, and we decide then which of the versions we keep, or for example keep 1.4.1 in standard and 1.5.0rc1 as experimental

comment:6 follow-up: Changed 11 years ago by drkirkby

If 1.5.0rc1 is merged, #7166 can be closed. I don't know about 1.4.1, but in any case that is not a critical bug.

Some time back I made a Solaris-specific change to Numpy, as I wanted to get it reviewed with the least hassle - that generally means making it Solaris specific, as reviewers are easier to please if it only effects a rarer platform.

But I think the change should be implemented on OS X too. Currently there's a really nasty hack on OS X to build Numpy, that involves a script called fake_gcc, copying that to $SAGE/LOCAL/bin/gcc, then using the fake gcc to build Numpy. Finally this fake gcc file gets deleted.

The far neater option is the way I did it on Solaris. I suggest the changes I made for Solaris are implemented whenever SAGE64 is set to "yes", irrespective of whatever platform it may be. Then all this fake gcc rubbish can be removed.

If you want, I can create a patch.

comment:7 in reply to: ↑ 6 Changed 11 years ago by maldun

Replying to drkirkby:

If you want, I can create a patch.

would be nice! But first I have to sort some things out, I hope it will get ready soon...

comment:8 follow-up: Changed 11 years ago by fbissey

Looking at all that stuff in the spkg and comparing to Gentoo. Not very pretty. Do we really still need to use sage_fortran? On most platform we now use a native fortran compiler rather than a sage shipped one - I don't know if we still do that for OS X. In the patch to gnu.py there is:

+        # Note that sse flags and so on lead to gfortran code that segfaults, so disable arch flags
+        return opt
+

An extract of the Gentoo set up:

	export NUMPY_FCONFIG="config_fc --noopt --noarch"

Actually here is the full set up that you might find interesting:

	# See progress in http://projects.scipy.org/scipy/numpy/ticket/573
	# with the subtle difference that we don't want to break Darwin where
	# -shared is not a valid linker argument
	if [[ ${CHOST} != *-darwin* ]] ; then
		append-ldflags -shared
	fi

	# only one fortran to link with:
	# linking with cblas and lapack library will force
	# autodetecting and linking to all available fortran compilers
	use lapack || return
	[[ -z ${FC} ]] && FC=$(tc-getFC)
	# when fortran flags are set, pic is removed.
	FFLAGS="${FFLAGS} -fPIC"
	export NUMPY_FCONFIG="config_fc --noopt --noarch"

The other patches we have are relatively minor, a fix to the f2py man page, a patch for freebsd - that may be usefull, a patch for interix(!). I cannot comment on the cygwin patches but they look very small. We are are a bit more anal with site.cfg, I don't think it is useful in the context of sage - we just have more complex possible combinations of blas/lapack.

The NUMPY_FCONFIG is passed to distutils as an argument of

python setup.py install

i.e. in the end what we do boils down to "python setup.py install ${NUMPY_FCONFIG}".

Can you point me to your solaris fix Dave? I'd like to see if I can tidy all that up in something that requires less patching and is more based on FLAGS settings.

comment:9 in reply to: ↑ 8 Changed 11 years ago by drkirkby

Replying to fbissey:

Looking at all that stuff in the spkg and comparing to Gentoo. Not very pretty.

What a surprise.

Do we really still need to use sage_fortran? On most platform we now use a native fortran compiler rather than a sage shipped one - I don't know if we still do that for OS X.

I've never understood the need for this SAGE_FORTRAN. I've tried arguing for it to be removed and use FC instead, but I had no luck.

A Fortran compiler is still shipped on OS X, but I don't see why the variable FC can't be made to point to that, rather than have the variable SAGE_FORTRAN.

Can you point me to your solaris fix Dave? I'd like to see if I can tidy all that up in something that requires less patching and is more based on FLAGS settings.

On Solaris, and some OS X versions, if you want a 64-bit build, you must add the compiler flag -m64 with gcc. Usually, putting that in CFLAGS is enough. However, for Numpy that does not work.

Someone came up with a fix for this which was implemented only on OS X, that involved creating a wrapper script called gcc_fake which basically calls gcc with the -m64 option. You can see the script yourself in the top level directory.

Since they had done this only on OS X, it did not work on Solaris. So I came up with a solution for Solaris, but I avoided the wrapper script. Instead I set the variable to CC=gcc -m64

I'm attaching a patch, which basically uses the Solaris on any system, including OS X. I think this is the sensible way to do it, not have a wrapper script.

I've not tested the attached patch - not even on Solaris!! But I think you can see what I am trying to do. I was going to try to explain it in words, but a bit of code, even if untested, should be more sensible.

Changed 11 years ago by drkirkby

An untested patch, which makes Numpy build the same was on OS X as it does on Solaris or other platforms where SAGE64=yes. It removes the stupid wrapper script.

comment:10 follow-up: Changed 11 years ago by fbissey

Ok - so we still use g95 on some targets. So we need to keep some patches just for these - bother.

comment:11 in reply to: ↑ 10 ; follow-up: Changed 11 years ago by drkirkby

Replying to fbissey:

Ok - so we still use g95 on some targets. So we need to keep some patches just for these - bother.

I don't believe g95 is used anywhere. There are g95 binaries in the Fortran package in Sage, but William said they can be removed. There is a gfortran binary. So as far as I'm aware, all g95 stuff can be removed, but SAGE_FORTRAN can't be removed.

Dave

comment:12 in reply to: ↑ 11 ; follow-up: Changed 11 years ago by fbissey

Replying to drkirkby:

Replying to fbissey:

Ok - so we still use g95 on some targets. So we need to keep some patches just for these - bother.

I don't believe g95 is used anywhere. There are g95 binaries in the Fortran package in Sage, but William said they can be removed. There is a gfortran binary. So as far as I'm aware, all g95 stuff can be removed, but SAGE_FORTRAN can't be removed.

That's good! That means we probably can give the shove to the gnu.py and init.py patches. I wouldn't be sorry to see the back of these.

There is a comment in SPKG.txt:

Special Update/Build Instructions:
  * The file $SAGE_ROOT/devel/sage/sage/ext/numpy.pxi comes from this file and must be updated if/when 
    the file src/numpy/doc/cython/numpy.pxi is updated.

I cannot find the file in question in that location. There is however a numpy.pxi under src/numpy/random/mtrand but I am not sure that's the file in question. Furthermore I don't appear to have a numpy.pxi in $SAGE_ROOT/devel/sage/sage/ext/numpy.pxi . Does anyone know if these instructions are obsolete?

comment:13 follow-up: Changed 11 years ago by jason

FYI: Numpy 1.5 has been officially released now.

comment:14 in reply to: ↑ 12 ; follow-up: Changed 11 years ago by jason

Replying to fbissey:

Replying to drkirkby:

Replying to fbissey:

Ok - so we still use g95 on some targets. So we need to keep some patches just for these - bother.

I don't believe g95 is used anywhere. There are g95 binaries in the Fortran package in Sage, but William said they can be removed. There is a gfortran binary. So as far as I'm aware, all g95 stuff can be removed, but SAGE_FORTRAN can't be removed.

That's good! That means we probably can give the shove to the gnu.py and init.py patches. I wouldn't be sorry to see the back of these.

There is a comment in SPKG.txt:

Special Update/Build Instructions:
  * The file $SAGE_ROOT/devel/sage/sage/ext/numpy.pxi comes from this file and must be updated if/when 
    the file src/numpy/doc/cython/numpy.pxi is updated.

I cannot find the file in question in that location. There is however a numpy.pxi under src/numpy/random/mtrand but I am not sure that's the file in question. Furthermore I don't appear to have a numpy.pxi in $SAGE_ROOT/devel/sage/sage/ext/numpy.pxi . Does anyone know if these instructions are obsolete?

I think I was the one that added those instructions, and I'm pretty sure they're obsolete instructions now. I believe we took care of merging the differences between the two numpy.pxi/pxd files a while ago.

comment:15 in reply to: ↑ 14 Changed 11 years ago by fbissey

Replying to jason:

I think I was the one that added those instructions, and I'm pretty sure they're obsolete instructions now. I believe we took care of merging the differences between the two numpy.pxi/pxd files a while ago.

Thank you for the confirmation. I have done some cleaning to numpy's spkg-install and it seems to work as intended on my machine. I guess we'll update to 1.5 and give it a spin.

comment:16 in reply to: ↑ 4 ; follow-ups: Changed 11 years ago by drkirkby

Replying to fbissey:

Replying to mhansen:

I don't think that we should upgrade to 1.5.0rc1 -- we should do 1.4.1 for now and wait until 1.5 is released.

That has to be double checked but maldun says he needs features in 1.5. It may be that the features he wants are 1.4.x and he doesn't know. Do we know roughly when 1.5 will be released? They are at 1.5rc1 9 days after beta2 so it could very well be upon us in a very short time.

In general though, we should not be upgrading to pre-release versions just because one person needs a feature that's only available in a pre-release. Everyone should not have to suffer the extra risks a pre-release gives unless there are compelling arguments for it.

I realise in this case 1.5 has since been released, so it's immaterial now.

Dave

comment:17 in reply to: ↑ 16 Changed 11 years ago by jason

Replying to drkirkby:

In general though, we should not be upgrading to pre-release versions just because one person needs a feature that's only available in a pre-release. Everyone should not have to suffer the extra risks a pre-release gives unless there are compelling arguments for it.

+1

comment:18 in reply to: ↑ 16 Changed 11 years ago by fbissey

Replying to drkirkby:

In general though, we should not be upgrading to pre-release versions just because one person needs a feature that's only available in a pre-release. Everyone should not have to suffer the extra risks a pre-release gives unless there are compelling arguments for it.

I realise in this case 1.5 has since been released, so it's immaterial now.

Yes, I didn't think it would be worth working on that particular version if it wasn't for the real proximity of the release (curses pari svn upgrade). I would probably have put pressure for 1.4.1 otherwise (very keen to see numpy upgraded).

comment:19 in reply to: ↑ 13 Changed 11 years ago by maldun

Replying to jason:

FYI: Numpy 1.5 has been officially released now.

So I was right =) Til I'm done with 1.5.0rc1 1.5 is out...

So changed so far:

The following doctest had to be rewritten:

File "/home/maldun/sage/sage-4.5.2/devel/sage/doc/en/faq/faq-usage.rst", line 341:
    sage: stats.ttest_ind(list([1,2,3,4,5]),list([2,3,4,5,.6]))
Expected:
    doctest...DeprecationWarning...
    (0.076752955645333687, 0.94070490247380478)
Got:
    (0.076752955645333687, 0.94070490247380478)

That was easy =)

The next prob was:

File "/home/maldun/sage/sage-4.5.2/devel/sage/sage/graphs/graph.py", line 615:
    sage: Graph(A)
Exception raised:
    Traceback (most recent call last):
      File "/home/maldun/sage/sage-4.5.2/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/home/maldun/sage/sage-4.5.2/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/home/maldun/sage/sage-4.5.2/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_1[22]>", line 1, in <module>
        Graph(A)###line 615:
    sage: Graph(A)
      File "/home/maldun/sage/sage-4.5.2/local/lib/python/site-packages/sage/graphs/graph.py", line 846, in __init__
        data = networkx.MultiGraph(data)
      File "/home/maldun/sage/sage-4.5.2/local/lib/python2.6/networkx/classes/graph.py", line 220, in __init__
        convert.from_whatever(data,create_using=self)
      File "/home/maldun/sage/sage-4.5.2/local/lib/python2.6/networkx/convert.py", line 157, in from_whatever
        if isinstance(thing,numpy.core.defmatrix.matrix) or \
    AttributeError: 'module' object has no attribute 'defmatrix'

that was also easy. defmatrix just changed from numpy.core to numpy.matrix to numpy.matrixlib Only changed that line in /local/lib/python2.6/networkx/classes/graph.py

Here comes now a trickier one:

       sage -t -valgrind "devel/sage/sage/rings/polynomial/polynomial_element.pyx"
Total time for all tests: 716.4 seconds
maldun@zauberbuch:~/sage/sage-4.5.2$ sage -t  -valgrind "devel/sage/sage/rings/polynomial/real_roots.pyx
> "
ERROR: File ./devel/sage/sage/rings/polynomial/real_roots.pyx
 is missing
 
----------------------------------------------------------------------
The following tests failed:


        ./devel/sage/sage/rings/polynomial/real_roots.pyx
 # File not found
Total time for all tests: 0.0 seconds
maldun@zauberbuch:~/sage/sage-4.5.2$ sage -t  -valgrind "devel/sage/sage/rings/polynomial/real_roots.pyx"
sage -t -valgrind "devel/sage/sage/rings/polynomial/real_roots.pyx"
**********************************************************************
File "/home/maldun/sage/sage-4.5.2/devel/sage/sage/rings/polynomial/real_roots.pyx", line 1819, in __main__.example_76
Failed example:
    oc.find_roots()###line 3064:_sage_    >>> oc.find_roots()
Expected nothing
Got:
    doctest:1: RuntimeWarning: tp_compare didn't return -1 or -2 for exception
**********************************************************************
File "/home/maldun/sage/sage-4.5.2/devel/sage/sage/rings/polynomial/real_roots.pyx", line 1840, in __main__.example_77
Failed example:
    oc.find_roots()###line 3085:_sage_    >>> oc.find_roots()
Expected nothing
Got:
    doctest:1: RuntimeWarning: tp_compare didn't return -1 or -2 for exception
**********************************************************************
File "/home/maldun/sage/sage-4.5.2/devel/sage/sage/rings/polynomial/real_roots.pyx", line 1934, in __main__.example_80
Failed example:
    oc.find_roots()###line 3157:_sage_    >>> oc.find_roots()
Expected nothing
Got:
    doctest:1: RuntimeWarning: tp_compare didn't return -1 or -2 for exception
**********************************************************************
File "/home/maldun/sage/sage-4.5.2/devel/sage/sage/rings/polynomial/real_roots.pyx", line 2320, in __main__.example_98
Failed example:
    real_roots(x**Integer(5) * (x**Integer(2) - Integer(9999))**Integer(2) - Integer(1))###line 3870:_sage_    >>> real_roots(x^5 * (x^2 - 9999)^2 - 1)
Expected:
    [((-29274496381311/9007199254740992, 419601125186091/2251799813685248), 1), ((2126658450145849453951061654415153249597/21267647932558653966460912964485513216, 4253316902721330018853696359533061621799/42535295865117307932921825928971026432), 1), ((1063329226287740282451317352558954186101/10633823966279326983230456482242756608, 531664614358685696701445201630854654353/5316911983139663491615228241121378304), 1)]
Got:
    doctest:1: RuntimeWarning: tp_compare didn't return -1 or -2 for exception
    [((-29274496381311/9007199254740992, 419601125186091/2251799813685248), 1), ((2126658450145849453951061654415153249597/21267647932558653966460912964485513216, 4253316902721330018853696359533061621799/42535295865117307932921825928971026432), 1), ((1063329226287740282451317352558954186101/10633823966279326983230456482242756608, 531664614358685696701445201630854654353/5316911983139663491615228241121378304), 1)]
**********************************************************************
4 items had failures:
   1 of  10 in __main__.example_76
   1 of   9 in __main__.example_77
   1 of  10 in __main__.example_80
   1 of  44 in __main__.example_98
***Test Failed*** 4 failures.
         [227.6 s]
 
----------------------------------------------------------------------
The following tests failed:


        sage -t -valgrind "devel/sage/sage/rings/polynomial/real_roots.pyx"
Total time for all tests: 227.6 seconds

This is not the only one, but the root of evil seems to be in real_roots (how ironic...) more precisly: I tracked it down, with help of valgrind, to the class ocean. Has anyone an Idea??

comment:20 Changed 11 years ago by maldun

  • Summary changed from Upgrade numpy to 1.5.0rc1 and scipy to 0.8 to Upgrade numpy to 1.5 and scipy to 0.8

comment:21 Changed 11 years ago by maldun

Update: Found the problem. see: http://groups.google.com/group/cython-users/browse_thread/thread/624c696293b7fe44

It seems that all versions 1.5.x hold this bug.... Tried downgrading to 1.4.1. and all corrections I have done so far worked.

I will now apply the patch from drkirkby, pack it again, test it overnight and hopefully we are done with 1.4.1, and hopefully they get it right in the next time. perhaps I should send the numpy/scipy guys the doctests I've done so far

scipy 0.8. don't seem to make any problems so far

comment:22 Changed 11 years ago by maldun

  • Summary changed from Upgrade numpy to 1.5 and scipy to 0.8 to Upgrade numpy to 1.4.1 and scipy to 0.8

comment:23 follow-up: Changed 11 years ago by maldun

an doctest I oversaw:

Type ``numpy.typecodes`` for a list of the possible
        typecodes::

            sage: import numpy
            sage: sorted(numpy.typecodes.items())
            [('All', '?bhilqpBHILQPfdgFDGSUVOMm'), ('AllFloat', 'fdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Datetime', 'Mm'), ('Float', 'fdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')]

The output is now:

[('All', '?bhilqpBHILQPfdgFDGSUVOMm'), ('AllFloat', 'fdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Datetime', 'Mm'), ('Float', 'fdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')]

But since it's an extension, and no downgrade it is also no prob

comment:24 in reply to: ↑ 23 ; follow-up: Changed 11 years ago by fbissey

Replying to maldun:

an doctest I oversaw:

Type ``numpy.typecodes`` for a list of the possible
        typecodes::

            sage: import numpy
            sage: sorted(numpy.typecodes.items())
            [('All', '?bhilqpBHILQPfdgFDGSUVOMm'), ('AllFloat', 'fdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Datetime', 'Mm'), ('Float', 'fdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')]

The output is now:

[('All', '?bhilqpBHILQPfdgFDGSUVOMm'), ('AllFloat', 'fdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Datetime', 'Mm'), ('Float', 'fdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')]

But since it's an extension, and no downgrade it is also no prob

the doctest should be fixed as well. I have attached a new version of spkg-install for numpy if you cared to give it a spin. It completely drop the non-cygwin patches. It may need a little bit of work as I haven't looked yet at how you included Dave's fix.

comment:25 in reply to: ↑ 24 Changed 11 years ago by maldun

  • Status changed from needs_work to needs_review

Replying to fbissey:

the doctest should be fixed as well. I have attached a new version of spkg-install for numpy if you cared to give it a spin. It completely drop the non-cygwin patches. It may need a little bit of work as I haven't looked yet at how you included Dave's fix.

Your install file worked well for me. I have merged it, such that the patches are getting applied. All doctests work now. You can download the packages now at: http://code.google.com/p/spkg-upload/downloads/detail?name=numpy-1.4.1.spkg and http://code.google.com/p/spkg-upload/downloads/detail?name=scipy-0.8.spkg

but you also have to install a modified version of networkx. I have opened a ticket for this, and this also includes a package: http://trac.sagemath.org/sage_trac/ticket/9854

next I will add a patch which includes the changed doctests

comment:26 Changed 11 years ago by maldun

One important note: after updating numpy one has to rebuild the source with sage -ba, or else you get runtime warnings.

The reason is the size change in some of the numpy functions, and then the .pyx files have to be rebuild

Changed 11 years ago by maldun

modified networkx package (test version)

Changed 11 years ago by maldun

changes to networkx, which have to be applied

comment:27 follow-up: Changed 11 years ago by fbissey

Thanks for the patches. I will probably introduce these in sage-on-gentoo in short order. About networkx, you realize that sage-4.5.3 will use networkx-1.2? see ticket #9567 So have you tried to see if networkx-1.2 needs patching (my guess is no, otherwise I would know by now from gentoo).

comment:28 in reply to: ↑ 27 ; follow-up: Changed 11 years ago by maldun

Replying to fbissey:

Thanks for the patches. I will probably introduce these in sage-on-gentoo in short order. About networkx, you realize that sage-4.5.3 will use networkx-1.2? see ticket #9567 So have you tried to see if networkx-1.2 needs patching (my guess is no, otherwise I would know by now from gentoo).

I've seen that networkx 1.2. is out but it doesn't seem to work with my sage-4.2 with numpy 1.4.1 nor with my sage-4.1 with the old numpy so I created this spkg in order that it can be used with old versions of sage too

But it seems the line where the problems come from doesn't exist in the new version of networkx anymore. Could you give the numpy packages a spin with the new versions of numpy/scipy? It is very time consuming for me to rebuild every time the source. Thanx in advance!

comment:29 in reply to: ↑ 28 Changed 11 years ago by maldun

ith the new versions of numpy/scipy? It is very time consuming for me to rebuild every time the source. Thanx in advance!

sorry I meant new versions of networkx...

comment:30 Changed 11 years ago by fbissey

Understood. We already ship networkx-1.2 with sage-4.5.2 on sage-on-gentoo [because networkx-1.0.1 has been removed from Gentoo] so I can test your patches along with networkx. It may take 1 or 2 days for me to fit it in my schedule.

comment:31 Changed 11 years ago by maldun

ok. Since I had some time this afternoon I build a version of sage-1.4.3.alpha1 (which also has networkx-1.2.) on my machine, applied the packages, the patch, rebuild the source with sage -ba and everything worked fine!

./sage -tp 4 -long devel/sage-numpy
....
sage -t -long devel/sage-numpy/doc/en/a_tour_of_sage/index.rst
         [23.1 s]
sage -t -long devel/sage-numpy/doc/en/thematic_tutorials/group_theory.rst
         [247.1 s]
 
----------------------------------------------------------------------
All tests passed!
Timings have been updated.
Total time for all tests: 7320.5 seconds

So networkx-1.2 works!

For Info: I use Ubuntu 10.04 on my machine. So I think it would still be good if someone else would test it

comment:32 Changed 11 years ago by maldun

  • Milestone changed from sage-4.6 to sage-4.5.3

comment:33 follow-up: Changed 11 years ago by fbissey

It all works in sage-on-gentoo.

However I think there are a few points that should be taken into consideration for both numpy and scipy. Just before posting my spkg-install I edited it to remove a change that I now think is probably necessary. I had set FC to SAGE_FORTRAN... Why? Because numpy tries to autodetect a fortran compiler and will take a gnu compiler first. Unless you set FC. Which means if Dave tries to compile sage with sunstudio on a machine that has also gfortran he is in for a world of hurt. So I think we should either set FC to SAGE_FORTRAN in numpy and possibly scipy. The other option is to set it globally but that may cause problems on OSX.

Thoughts? Francois

comment:34 in reply to: ↑ 33 Changed 11 years ago by maldun

Replying to fbissey:

It all works in sage-on-gentoo.

However I think there are a few points that should be taken into consideration for both numpy and scipy. Just before posting my spkg-install I edited it to remove a change that I now think is probably necessary. I had set FC to SAGE_FORTRAN... Why? Because numpy tries to autodetect a fortran compiler and will take a gnu compiler first. Unless you set FC. Which means if Dave tries to compile sage with sunstudio on a machine that has also gfortran he is in for a world of hurt. So I think we should either set FC to SAGE_FORTRAN in numpy and possibly scipy. The other option is to set it globally but that may cause problems on OSX.

Thoughts? Francois

A very pragmatic thougt: If it doesn"t hurt to set FC to SAGE_FORTRAN, why not? But I think it"s better not to do it globally. Because then we could cause more problems then we solve. Can you give a modified spkg-install? I can try it out over night then.

But I think it would also be good to get some feedback for Solaris and OSX for the current versions. Then we could decide to keep the current, or to take the other.

comment:35 follow-up: Changed 11 years ago by maldun

Good news: It seems that the problem from numpy-1.5.0 has been resolved http://projects.scipy.org/numpy/ticket/1605 I don't think it's a big deal. Shall I give it a try, or should we stick to 1.4.1 for now?

comment:36 in reply to: ↑ 35 Changed 11 years ago by maldun

Replying to maldun:

Good news: It seems that the problem from numpy-1.5.0 has been resolved http://projects.scipy.org/numpy/ticket/1605 I don't think it's a big deal. Shall I give it a try, or should we stick to 1.4.1 for now?

...or wait til 1.5.1 is out?

comment:37 Changed 11 years ago by fbissey

May be wait until numpy-1.5.1. it is not a big deal right now I guess. May be we should discuss it on sage-devel. At the moment we don't really have much testing apart from the fact I unleashed the upgrade to numpy-1.4.1/scipy-0.8 on sage-on-gentoo users. I am not sure we can believably merge this in 4.5.3 unless it takes more than one week before it is released. If we target 4.5.3 I say we stay with what we have now, if we target 4.6 which should be a little further down the track we go for 1.5.x.

Changed 11 years ago by fbissey

newer spkg_install setting FC

comment:38 Changed 11 years ago by fbissey

I updated my spkg_install as you requested. The other thing about using sage_fortran, that I had forgotten to change back when I removed it, is that it includes "-fPIC". I didn't check the fortran spkg but hopefully the work done by Dave to set the correct pic flag is picked up in sage_fortran.

If we don't adopt this, FFLAG="-fPIC" (or whatever mechanism Dave came up with) should be added in my previous version.

comment:39 Changed 11 years ago by maldun

Replying to fbissey:

May be wait until numpy-1.5.1. it is not a big deal right now I guess. May be we should discuss it on sage-devel. At the moment we don't really have much testing apart from the fact I unleashed the upgrade to numpy-1.4.1/scipy-0.8 on sage-on-gentoo users. I am not sure we can believably merge this in 4.5.3 unless it takes more than one week before it is released. If we target 4.5.3 I say we stay with what we have now, if we target 4.6 which should be a little further down the track we go for 1.5.x.

Ok I think we should do the following: Let's stick to 1.4.1 since it is necessary for scipy 0.8. and quite well tested yet, in contrary to 1.5.x and it seems to work. Especially since the update from scipy 0.7 to 0.8 holds a lot of changes, and we don't know if they build some more bugs into numpy 1.5.1... If it turns out that updating from 1.4.1 to 1.5.1 is no problem, then well... packing a new package would'nt be the prob I think.

@new spkg: Ok I will give it a try tonight!

comment:40 follow-up: Changed 11 years ago by kcrisman

There's a lot of packages etc. on this ticket. Can someone provide a concise list of what would be needed to apply on a given platform? (For instance, in a very cursory glance, the networkx package being here mystifies.)

comment:41 in reply to: ↑ 40 Changed 11 years ago by maldun

Replying to kcrisman:

There's a lot of packages etc. on this ticket. Can someone provide a concise list of what would be needed to apply on a given platform? (For instance, in a very cursory glance, the networkx package being here mystifies.)

Needed are: http://code.google.com/p/spkg-upload/downloads/detail?name=numpy-1.4.1.spkg http://code.google.com/p/spkg-upload/downloads/detail?name=scipy-0.8.spkg

after installing numpy, one needs so execute sage -ba, or else one get's runtime warnings.

You also need networkx-1.2. (the other networkx is just a small hack for testing, because 1.2. didn't build correctly on my machine, but this is obsolete now, since networkx-1.2 is merged into sage 4.3.alpha1)

comment:42 Changed 11 years ago by maldun

  • Description modified (diff)

comment:43 Changed 11 years ago by maldun

  • Description modified (diff)

comment:44 Changed 11 years ago by maldun

I gave the links and (current) build instructions into the discription, so that people can find the latest versions more quickly

comment:45 Changed 11 years ago by maldun

  • Description modified (diff)

comment:46 Changed 11 years ago by kcrisman

  • Description modified (diff)

Included direct links to files in description.

comment:47 Changed 11 years ago by maldun

  • Milestone changed from sage-4.5.3 to sage-4.6

comment:48 Changed 11 years ago by maldun

  • Description modified (diff)

comment:49 Changed 11 years ago by kcrisman

This seems to work fine on Mac OS X 10.6.4 Intel. I will try to test it tomorrow on a PowerPC machine. Note: I haven't looked at the spkg installs or anything, this is just a data point with regard to whether it works, not a review.

comment:50 follow-up: Changed 11 years ago by mhansen

This fails very early on in Cygwin with

building library "npymath" sources
creating build/src.cygwin-1.7.5-i686-2.6/src
conv_template:> build/src.cygwin-1.7.5-i686-2.6/src/npy_math.c
error: src/npy_math.c.src: No such file or directory
Error building numpy.

comment:51 in reply to: ↑ 50 Changed 11 years ago by maldun

Replying to mhansen:

This fails very early on in Cygwin with

building library "npymath" sources
creating build/src.cygwin-1.7.5-i686-2.6/src
conv_template:> build/src.cygwin-1.7.5-i686-2.6/src/npy_math.c
error: src/npy_math.c.src: No such file or directory
Error building numpy.

Ok I think I have the picture: some files from /src/numpy/core/src moved to subfolders this have to be changed in the cygwin-core-setup.py

I'm installing already cygwin on my laptop and look if I get it up and running, but if anyone could help out I would be thankful!

comment:52 follow-up: Changed 11 years ago by mhansen

I've tested it out, and everything works if you just remove the cygwin-core-setup.py patch.

comment:53 in reply to: ↑ 52 ; follow-up: Changed 11 years ago by maldun

Replying to mhansen:

I've tested it out, and everything works if you just remove the cygwin-core-setup.py patch.

I was just about to upload a patch... But great this is even better than my solution! (and simpler) The reason why it fails is just that cygwin-core-setup.py is outdated.

one question: (because I'm quite the noob...) If I want to upload a modified version of numpy-1.4.1 do I have to overwrite the old version, or shall I rename it with numpy-1.4.1.p0.spkg?

comment:54 in reply to: ↑ 53 Changed 11 years ago by drkirkby

Replying to maldun:

one question: (because I'm quite the noob...) If I want to upload a modified version of numpy-1.4.1 do I have to overwrite the old version, or shall I rename it with numpy-1.4.1.p0.spkg?

Overwrite the old one.

comment:55 Changed 11 years ago by maldun

Thanx! Numpy is now updated with

  • changes from fbissey to the installer (it worked for me)
  • removed the cygwin-core-setup.py patch

comment:56 Changed 11 years ago by mhansen

The SciPy? spkg works fine in Cygwin.

comment:57 follow-up: Changed 11 years ago by kcrisman

I'm getting a corrupted package message with the packages right now while trying it on a different system. Did something get changed with respect to the file with this latest update?

comment:58 in reply to: ↑ 57 ; follow-up: Changed 11 years ago by maldun

Replying to kcrisman:

I'm getting a corrupted package message with the packages right now while trying it on a different system. Did something get changed with respect to the file with this latest update?

Yes we did some changes to the spkg-install. But I downloaded and installed the package right now again. So I don't know wehre the problem is lying?

comment:59 in reply to: ↑ 58 ; follow-up: Changed 11 years ago by kcrisman

Replying to maldun:

Replying to kcrisman:

I'm getting a corrupted package message with the packages right now while trying it on a different system. Did something get changed with respect to the file with this latest update?

Yes we did some changes to the spkg-install. But I downloaded and installed the package right now again. So I don't know wehre the problem is lying?

Did you use sage -pkg to create it or just do some kind of compression? I get a message about tar: this does not look like a file archive and tar: Archive contains obsolescent base-64 headers. I just installed the optional biopython package on this machine to test things, so the machine shouldn't be the problem (and it built Sage 4.5.2 just fine).

comment:60 in reply to: ↑ 59 ; follow-up: Changed 11 years ago by maldun

Replying to kcrisman:

Replying to maldun:

Replying to kcrisman:

I'm getting a corrupted package message with the packages right now while trying it on a different system. Did something get changed with respect to the file with this latest update?

Yes we did some changes to the spkg-install. But I downloaded and installed the package right now again. So I don't know wehre the problem is lying?

Did you use sage -pkg to create it or just do some kind of compression? I get a message about tar: this does not look like a file archive and tar: Archive contains obsolescent base-64 headers. I just installed the optional biopython package on this machine to test things, so the machine shouldn't be the problem (and it built Sage 4.5.2 just fine).

Ok I repacked it now with -pkg and uploaded it. Does it work now? And sorry for the newby question: what is the difference between -pkg and just compress it? Is sage using a different version of tar? Because in fact a spkg is noting else then a tar.gz with different ending.

comment:61 in reply to: ↑ 60 Changed 11 years ago by kcrisman

Ok I repacked it now with -pkg and uploaded it. Does it work now? And sorry for the newby question: what is the difference between -pkg and just compress it? Is sage using a different version of tar? Because in fact a spkg is noting else then a tar.gz with different ending.

No, it's a little more than that - it has an SPKG.txt, it has Mercurial repositories, etc. True that the file type is the same. But sage -pkg tests several things, and a successful review would check all that. See here for more info.

comment:62 Changed 11 years ago by kcrisman

  • Reviewers set to Karl-Dieter Crisman

Hmm, when I open it manually it seems fine. Though when I run sage -pkg on that new folder, I get

HG REPO: Unchecked in changes

So you'll need to fix that. Putting myself as a reviewer now.

Calling sage -f on this new spkg I made *does* seem to work, for whatever reason. This doesn't make sense to me - why is tar complaining about yours? Hopefully someone who knows about file systems will test this soon and explain why this causes a problem on Mac OS X 10.4 Tiger.

comment:63 follow-up: Changed 11 years ago by kcrisman

Okay, that worked. Scipy seems to be installing fine directly from the website, I'm not sure why this happened.

comment:64 Changed 11 years ago by maldun

Thanx for the hints! I updated the mercurial entries now to the latest version. Uploaded changed packages. Everything should be fine now

comment:65 in reply to: ↑ 63 Changed 11 years ago by maldun

Replying to kcrisman:

Okay, that worked. Scipy seems to be installing fine directly from the website, I'm not sure why this happened.

I have an Idea: different versions of .tar may cause problems (so far as I know) If you look at the file sizes only it seems that sage does a different kind of compression.

comment:66 follow-up: Changed 11 years ago by kcrisman

I get the following error with the Scipy for some reason.

creating build/temp.macosx-10.4-ppc-2.6/scipy/special
creating build/temp.macosx-10.4-ppc-2.6/scipy/special/c_misc
compile options: '-I/Users/student/Desktop/sage-4.5.2/local/lib/python2.6/site-packages/numpy/core/include -c'
gcc: scipy/special/c_misc/fsolve.c
scipy/special/c_misc/fsolve.c: In function 'false_position':
scipy/special/c_misc/fsolve.c:58: warning: 'f3' may be used uninitialized in this function
scipy/special/c_misc/fsolve.c:58: warning: 'x3' may be used uninitialized in this function
gcc: scipy/special/c_misc/gammaincinv.c
scipy/special/c_misc/gammaincinv.c:1:20: error: Python.h: No such file or directory
In file included from /Users/student/Desktop/sage-4.5.2/local/lib/python2.6/site-packages/numpy/core/include/numpy/npy_math.h:5,
                 from scipy/special/c_misc/gammaincinv.c:2:
/Users/student/Desktop/sage-4.5.2/local/lib/python2.6/site-packages/numpy/core/include/numpy/npy_common.h:79:2: error: #error Must use Python with unicode enabled.
scipy/special/c_misc/gammaincinv.c:1:20: error: Python.h: No such file or directory
In file included from /Users/student/Desktop/sage-4.5.2/local/lib/python2.6/site-packages/numpy/core/include/numpy/npy_math.h:5,
                 from scipy/special/c_misc/gammaincinv.c:2:
/Users/student/Desktop/sage-4.5.2/local/lib/python2.6/site-packages/numpy/core/include/numpy/npy_common.h:79:2: error: #error Must use Python with unicode enabled.
error: Command "gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/student/Desktop/sage-4.5.2/local/lib/python2.6/site-packages/numpy/core/include -c scipy/special/c_misc/gammaincinv.c -o build/temp.macosx-10.4-ppc-2.6/scipy/special/c_misc/gammaincinv.o" failed with exit status 1
Error building scipy.

Did you use some special encoding for some of your stuff?

comment:67 in reply to: ↑ 66 Changed 11 years ago by maldun

Replying to kcrisman:

I get the following error with the Scipy for some reason.

creating build/temp.macosx-10.4-ppc-2.6/scipy/special
creating build/temp.macosx-10.4-ppc-2.6/scipy/special/c_misc
compile options: '-I/Users/student/Desktop/sage-4.5.2/local/lib/python2.6/site-packages/numpy/core/include -c'
gcc: scipy/special/c_misc/fsolve.c
scipy/special/c_misc/fsolve.c: In function 'false_position':
scipy/special/c_misc/fsolve.c:58: warning: 'f3' may be used uninitialized in this function
scipy/special/c_misc/fsolve.c:58: warning: 'x3' may be used uninitialized in this function
gcc: scipy/special/c_misc/gammaincinv.c
scipy/special/c_misc/gammaincinv.c:1:20: error: Python.h: No such file or directory
In file included from /Users/student/Desktop/sage-4.5.2/local/lib/python2.6/site-packages/numpy core/include/numpy/npy_math.h:5,
                 from scipy/special/c_misc/gammaincinv.c:2:
/Users/student/Desktop/sage-4.5.2/local/lib/python2.6/site-packages/numpy/core/include/numpy/npy_common.h:79:2: error: #error Must use Python with unicode enabled.
scipy/special/c_misc/gammaincinv.c:1:20: error: Python.h: No such file or directory
In file included from /Users/student/Desktop/sage-4.5.2/local/lib/python2.6/site-packages/numpy/core/include/numpy/npy_math.h:5,
                 from scipy/special/c_misc/gammaincinv.c:2:
/Users/student/Desktop/sage-4.5.2/local/lib/python2.6/site-packages/numpy/core/include/numpy/npy_common.h:79:2: error: #error Must use Python with unicode enabled.
error: Command "gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/student/Desktop/sage-4.5.2/local/lib/python2.6/site-packages/numpy/core/include -c scipy/special/c_misc/gammaincinv.c -o build/temp.macosx-10.4-ppc-2.6/scipy/special/c_misc/gammaincinv.o" failed with exit status 1
Error building scipy.

Did you use some special encoding for some of your stuff?

Nope. Perhaps something else did go wrong. I repacked it now on a different machine with a newer version of ubuntu and uploaded it. Hope this works. The first install worked for me.

comment:68 follow-up: Changed 11 years ago by kcrisman

Sadly, same error. This is probably on my end, but this is a supported architecture so it's important to know what went wrong. I don't think that the ./sage -ba would have anything to do with it.

So I suspect that the "missing: Python.h is the problem, as I've seen a few other things about this online (including ones like this one, where /include/Python.h definitely exists, here it's within the $SAGE_ROOT directory). I wonder why it's not finding it this time?

comment:69 in reply to: ↑ 68 ; follow-up: Changed 11 years ago by maldun

Replying to kcrisman:

Sadly, same error. This is probably on my end, but this is a supported architecture so it's important to know what went wrong. I don't think that the ./sage -ba would have anything to do with it.

So I suspect that the "missing: Python.h is the problem, as I've seen a few other things about this online (including ones like this one, where /include/Python.h definitely exists, here it's within the $SAGE_ROOT directory). I wonder why it's not finding it this time?

Question: Have you tried to give the direct path to the compiler? And does in OS X gcc points to /include/Python.h because in sage it's /include/python2.6/Python.h ?

comment:70 in reply to: ↑ 69 Changed 11 years ago by kcrisman

Replying to maldun:

Replying to kcrisman:

Sadly, same error. This is probably on my end, but this is a supported architecture so it's important to know what went wrong. I don't think that the ./sage -ba would have anything to do with it.

So I suspect that the "missing: Python.h is the problem, as I've seen a few other things about this online (including ones like this one, where /include/Python.h definitely exists, here it's within the $SAGE_ROOT directory). I wonder why it's not finding it this time?

Question: Have you tried to give the direct path to the compiler?

No - how would I do that?

And does in OS X gcc points to /include/Python.h because in sage it's /include/python2.6/Python.h ?

I find that very unlikely, since everything else works fine and in general Sage builds fine on OS X 10.4-10.6. But sometimes things get mixed up, I'm sure. There aren't any weird env variables that would do that here, though, I don't think.

comment:71 follow-up: Changed 11 years ago by fbissey

A question: do we impose python to be built with or without unicode support? If not, is the support enabled by default depending on the platform? To me it looks like numpy ships headers that are encoded with unicode and that your sage's python chock on them, that's the first and foremost error. distutils seem to be unable to work things properly after that.

Two things could be tried: 1) have python built with unicode support on OSX. 2) "vet" numpy to convert the headers in /Users/student/Desktop/sage-4.5.2/local/lib/python2.6/site-packages/numpy/core/include/numpy/ to plain text. Actually you probably can do that easily and manually right now on the installed headers.

Give it a spin and see if it works. If it does we'll have to do something about that piece of unicode one way or another.

comment:72 in reply to: ↑ 71 Changed 11 years ago by kcrisman

Replying to fbissey:

A question: do we impose python to be built with or without unicode support? If not, is the support enabled by default depending on the platform? To me it looks like numpy ships headers that are encoded with unicode and that your sage's python chock on them, that's the first and foremost error. distutils seem to be unable to work things properly after that.

Two things could be tried: 1) have python built with unicode support on OSX. 2) "vet" numpy to convert the headers in /Users/student/Desktop/sage-4.5.2/local/lib/python2.6/site-packages/numpy/core/include/numpy/ to plain text. Actually you probably can do that easily and manually right now on the installed headers.

Umm... except that I have no idea how I might do that. What exact changes would I need to make to Python.h (and/or something else)? It seems unwise to make yet another python spkg change to deal with this; did numpy only recently (between whatever Sage has now and 1.4.1) start making its headers unicode? I am not exactly a C expert.

That said, 1) seems to be more canonical, though I guess if it makes it work, 2) could be an option. This is the sort of thing Leif and/or drkirkby usually have an informed opinion on...

Give it a spin and see if it works. If it does we'll have to do something about that piece of unicode one way or another.

comment:73 Changed 11 years ago by fbissey

2) would be good to find if this is indeed the problem. Doing it is another story. Do you have iconv on OSX? I would think so as it is needed by R amongst other.

so go into the right folder and try:

iconv -f UTF-8 -t ISO-8859-1 *.h

comment:74 follow-up: Changed 11 years ago by kcrisman

Same error.

comment:75 follow-ups: Changed 11 years ago by maldun

In the spkg-install, there is the line

unset CFLAGS LDFLAGS CXXFLAGS SHAREDFLAGS

could this be somehow connected to our problem, because if we unset the flags, gcc doesn't point at Python.h in OS X?

comment:76 in reply to: ↑ 75 Changed 11 years ago by maldun

Replying to maldun:

In the spkg-install, there is the line

unset CFLAGS LDFLAGS CXXFLAGS SHAREDFLAGS

could this be somehow connected to our problem, because if we unset the flags, gcc doesn't point at Python.h in OS X?

Ok I reformulate it: Can we use the CFLAGS and LDFLAGS to give to tell gcc to link Python.h directly?

comment:77 in reply to: ↑ 75 Changed 11 years ago by kcrisman

Replying to maldun:

In the spkg-install, there is the line

unset CFLAGS LDFLAGS CXXFLAGS SHAREDFLAGS

could this be somehow connected to our problem, because if we unset the flags, gcc doesn't point at Python.h in OS X?

But this is not a change in the spkg-install script. Sage has built fine on this computer in the very recent past. The new spkg also works fine on other (newer) Macs.

From the numpy website

Reusing npymath

In 1.3.0, we started putting portable C math routines in npymath library, so that people can use those to write portable extensions. Unfortunately, it was not possible to easily link against this library: in 1.4.0, support has been added to numpy.distutils so that 3rd party can reuse this library. See coremath documentation for more information.

This seems quite possibly relevant (given the error message). After all, if Python.h isn't imported, then maybe the first error seen is that PY_UNICODE_DEF or whatever it's called wouldn't be set... maybe?

I'm just trying to throw out an idea here, I am woefully uniformed when it comes to headers and things like that. But I feel like it has to be a change in Scipy or Numpy, since the previous Sage built fine on this machine.

comment:78 follow-up: Changed 11 years ago by maldun

I have another wild theory: Haven't you said, that the package build one time? And then not. Is this correct?

The only thing I changed was updating the mercurial files, and then commiting. Therefore I had to write the commit messages, and I do this in kate. kate's standard encoding is utf-8. ( My computer's language is german, so everything is utf-8) This (or just packing it, because the encoding on my machine is not unicode) did change something to the encoding. What happens if you untar the package use the iconv from up there, and repack it on your machine?

comment:79 in reply to: ↑ 78 Changed 11 years ago by fbissey

Replying to maldun:

I have another wild theory: Haven't you said, that the package build one time? And then not. Is this correct?

The only thing I changed was updating the mercurial files, and then commiting. Therefore I had to write the commit messages, and I do this in kate. kate's standard encoding is utf-8. ( My computer's language is german, so everything is utf-8) This (or just packing it, because the encoding on my machine is not unicode) did change something to the encoding. What happens if you untar the package use the iconv from up there, and repack it on your machine?

I don't think so but it can be tested I guess. I think Karl is on something, I am having a look at scipy's spkg-install to see if there is something that should be done there.

I have an endianess problem with numpy-1.4.1 on linux ppc, funny it works on ppc OSX: http://projects.scipy.org/numpy/ticket/1403 and http://bugs.gentoo.org/show_bug.cgi?id=306237 on the other hand numpy-1.5.0 built on my test machine. I am trying to find the changeset that made it possible to see if it is worth backporting.

comment:80 in reply to: ↑ 74 Changed 11 years ago by fbissey

Replying to kcrisman:

Same error.

It may sound unrelated but what fortran compiler are you using in your sage install? g95 or gfortran?

In any case I think some of the patches in scipy would need rebasing. But the fact of the matter is that they are g95 dependent and we dropped g95 - I already did so in numpy so we should do it as well in scipy.

comment:81 Changed 11 years ago by kcrisman

I don't know exactly, but recall that Sage still includes Fortran compilers for Mac (see here). This is an spkg (sage-fortran...). From the spkg readme:

= gFortran =

== Description ==

G95 is a stable, production Fortran 95 compiler available for multiple
CPU architectures and operating systems.

== Upstream contacts ==

URL: http://ftp.g95.org
     http://www.g95.org

So I don't know how to answer your question - apparently it's both gfortran and g95 :) The spkg-install has more details on this, but it apparently depends on the version (there are different .bz2 files for different Macs); my old PPC Mac would have used g95, I think. Is there some technical reason we should drop this?

Responding to the other comment, sadly, although I thought Scipy installed correctly at first, my machine is REALLY slow, so eventually it turned out that it didn't. I succeeded on a newer Intel Mac, but that is unrelated.

comment:82 follow-up: Changed 11 years ago by fbissey

  • Status changed from needs_review to needs_work

See comment 10 by Dave. g95 can be removed. Unless they forgot about OSX ppc. For my question, what does

$SAGE_LOCAL/bin/sage_fortran --version

says? I am guessing we have gfortran binaries for all platforms. Dropping g95 means that we can remove some (ugly) patches for numpy/scipy there are probably other valid reasons to do so but generally speaking that's less work.

I removed all patches that were applied for g95 in numpy, so if you have g95 the problems may come from there.

I share your pain about building on ppc (although I dropped OS X sometimes ago, this is now linux only and gentoo so everything or almost install from sources. openoffice-3.1 clocks in at 27 hours to build). Did you mean that numpy failed or restating that scipy failed in your latest comment?

If numpy fails for you as well on OSX ppc we may have a strong motivation to move to 1.5.0 again.

I am changing this to need_works until we work out what's wrong in your case.

comment:83 in reply to: ↑ 82 Changed 11 years ago by maldun

Replying to fbissey:

If numpy fails for you as well on OSX ppc we may have a strong motivation to move to 1.5.0 again.

If you want I can prepare a package for 1.5.0 with the patch from the numpy trac, and run the doctests, in the evening.

comment:84 Changed 11 years ago by kcrisman

Numpy was fine in all cases; Scipy was what didn't build. drkirkby's comment "There are g95 binaries in the Fortran package in Sage, but William said they can be removed." may not be true.

The command that supposedly checks which version I have returns with an error:

local/bin/sage_fortran: line 3: sage_fortran.bin: command not found

This also happens on my (successful) build of 4.6.prealpha4, so maybe the command is wrong?

Incidentally, does this ticket do anything with respect to #7831 and #8010? Just curious.

comment:85 Changed 11 years ago by kcrisman

Okay, I finally figured out how to check this - I had to run the binary directly, the scripts didn't work.

G95 (GCC 4.0.3 (g95 0.91!) Jun 4 2007)

etcetera. On my Macintel, I get

GNU Fortran (GCC) 4.2.3
Copyright (C) 2007 Free Software Foundation, Inc.

So yes, it is using G95 on Tiger.

comment:86 Changed 11 years ago by maldun

It would be great if this is the problem and it can somehow be solved!

Another thing: I builded numpy-1.5.0 with the new patch now. There only failed some doctests because numpy is throwing some new warnings. Want to give it a try? Perhaps the problem is then obsolete with scipy 0.8. also? I personally would prefer to stick to 1.4.1, but well it is like it is...

comment:87 follow-up: Changed 11 years ago by kcrisman

If you post it, I can try it - though it might be after the weekend, depending on my physical access to that particular box.

fbissey - can you explain (in layman's terms) how having the wrong fortran compiler might affect things? It does confuse me :)

comment:88 in reply to: ↑ 87 ; follow-up: Changed 11 years ago by maldun

Replying to kcrisman:

If you post it, I can try it - though it might be after the weekend, depending on my physical access to that particular box.

fbissey - can you explain (in layman's terms) how having the wrong fortran compiler might affect things? It does confuse me :)

Up and ready!

Yes this would be interesting! I don't really get it either because it looks more like a C problem to me?

comment:90 Changed 11 years ago by fbissey

It's more of a hunch than anything special I'll admit to that. The numpy/scipy connection seems to be fragile so I am not discounting it. In actual fact the fortran compiler is used only to compile one file in the whole of numpy (lapack_lite.so). I am more worried that it mixes up the toolchain it prepares for scipy.

One thing I'd like to see however is a complete build log for scipy not just the failing bit. There may be clues earlier in the process.

comment:91 in reply to: ↑ 88 ; follow-up: Changed 11 years ago by kcrisman

Replying to maldun:

Replying to kcrisman:

If you post it, I can try it - though it might be after the weekend, depending on my physical access to that particular box.

fbissey - can you explain (in layman's terms) how having the wrong fortran compiler might affect things? It does confuse me :)

Up and ready!

Sorry, fails with exactly the same error (on a different box with similar specs, same G95 in its Sage). So for now probably stick with the better-tested 1.4.1 for numpy. Hopefully we can figure out what's going on here.

The logs for the numpy (1.5.0) and scipy are in this directory, from the second computer. Happy hunting! By the way, Python.h is clearly working fine elsewhere in these logs, and sage_fortran compiles all kinds of neat stuff up to that point.

comment:92 in reply to: ↑ 91 ; follow-up: Changed 11 years ago by fbissey

Replying to kcrisman:

Replying to maldun:

Replying to kcrisman:

If you post it, I can try it - though it might be after the weekend, depending on my physical access to that particular box.

fbissey - can you explain (in layman's terms) how having the wrong fortran compiler might affect things? It does confuse me :)

Up and ready!

Sorry, fails with exactly the same error (on a different box with similar specs, same G95 in its Sage). So for now probably stick with the better-tested 1.4.1 for numpy. Hopefully we can figure out what's going on here.

The logs for the numpy (1.5.0) and scipy are in this directory, from the second computer. Happy hunting! By the way, Python.h is clearly working fine elsewhere in these logs, and sage_fortran compiles all kinds of neat stuff up to that point.

Thanks for the logs. Sorry to be a bother but would you have old build logs for scipy-0.7 as well? Since I don't have a ppc OSX setup that would be very useful to have a successful log even from an older version of scipy.

comment:93 in reply to: ↑ 92 ; follow-up: Changed 11 years ago by kcrisman

Sorry, fails with exactly the same error (on a different box with similar specs, same G95 in its Sage). So for now probably stick with the better-tested 1.4.1 for numpy. Hopefully we can figure out what's going on here.

The logs for the numpy (1.5.0) and scipy are in this directory, from the second computer. Happy hunting! By the way, Python.h is clearly working fine elsewhere in these logs, and sage_fortran compiles all kinds of neat stuff up to that point.

Thanks for the logs. Sorry to be a bother but would you have old build logs for scipy-0.7 as well? Since I don't have a ppc OSX setup that would be very useful to have a successful log even from an older version of scipy.

Look in the same place, just posted it. So nice to be operating at 1.25 GHz instead of 700 MHz...

Also, interestingly, I now have a "mixed" installation on this computer:

sage: import numpy
sage: numpy.version.version
'1.5.0'
sage: import scipy
sage: scipy.version.version
'0.7.0'

At least some of the relevant tests seem to pass with this, though of course I get the RunTimeErrors? mentioned above and did Inf and inf change places? I know little about numpy and scipy, though. Anyway, that's better than on the other box, whose issues with bad tarballs seem to not exist on this box.

comment:94 in reply to: ↑ 93 ; follow-up: Changed 11 years ago by fbissey

Replying to kcrisman:

Look in the same place, just posted it. So nice to be operating at 1.25 GHz instead of 700 MHz...

Also, interestingly, I now have a "mixed" installation on this computer:

sage: import numpy
sage: numpy.version.version
'1.5.0'
sage: import scipy
sage: scipy.version.version
'0.7.0'

At least some of the relevant tests seem to pass with this, though of course I get the RunTimeErrors? mentioned above and did Inf and inf change places? I know little about numpy and scipy, though. Anyway, that's better than on the other box, whose issues with bad tarballs seem to not exist on this box.

Found the logs, I may have a theory and the logs may help to (in)validate it. Not sure about Inf/inf either. scipy should work fine with numpy-1.5 - once rebuilt, and sage needs rebuilding too. That may get rid of some of these runtime errors. scipy-0.8 needs numpy-1.4 minimum on the other hand.

comment:95 in reply to: ↑ 94 Changed 11 years ago by kcrisman

Also, interestingly, I now have a "mixed" installation on this computer:

sage: import numpy
sage: numpy.version.version
'1.5.0'
sage: import scipy
sage: scipy.version.version
'0.7.0'

At least some of the relevant tests seem to pass with this, though of course I get the RunTimeErrors? mentioned above and did Inf and inf change places? I know little about numpy and scipy, though. Anyway, that's better than on the other box, whose issues with bad tarballs seem to not exist on this box.

Found the logs, I may have a theory and the logs may help to (in)validate it. Not sure about Inf/inf either. scipy should work fine with numpy-1.5 - once rebuilt, and sage needs rebuilding too. That may get rid of some of these runtime errors. scipy-0.8 needs numpy-1.4 minimum on the other hand.

Yup, tests in relevant modules are passing, with only the RuntimeWarnings (not errors, sorry for that) in general marring things (as noted, one or two other very minor things). Anyway, none of this not compiling nonsense, so something must have changed in Scipy 0.8 from 0.7 - maybe something that used that change mentioned above about reusing npymath. There were indeed a couple changes to the gammaincinv function in Scipy, if you look at their changesets, but as a non-expert how those could have resulted in this Python.h/unicode error is totally beyond me.

comment:96 Changed 11 years ago by fbissey

Interestingly enough on Gentoo we have -I/usr/include/python2.6 added to the compilation flags everywhere. I wonder if just finding a way of adding -I${SAGE_LOCAL}/include/python2.6 in sage would help at all. The mystery then would be "how did it work before?" Furthermore why is it not needed for the earlier C compilations.... or on other platforms? May be we are using system python headers without knowing it?

I will work on that over the week end. Hopefully by Monday I will have something you can test.

comment:97 Changed 11 years ago by fbissey

It turns out to be more subtle than I thought and I don't know yet why.

It is not easy to add -I${SAGE_LOCAL}/include/python2.6 to compile options, in any case many other compilation lines actually have it already. After careful checking, I found that this particular option is not used in any scipy-0.7.x log I had on sage or Gentoo for these particular objects. But I see it in scipy-0.8.0 in Gentoo.

I am messing up with some of my system at the moment so I cannot generate a successful build log of scipy-0.8.0 from sage at the moment. But it would be interesting to know if this particular compilation option has been added in the successful builds.

comment:98 Changed 11 years ago by fbissey

OK - so there is a fair difference on that file between 0.7.2 and 0.8.0.

headers in gammaincinv.c in 0.7.2

#include <stdio.h>
#include <math.h>
#include "../cephes.h"
#undef fabs
#include "misc.h"

in 0.8.0

#include <Python.h>
#include <numpy/npy_math.h>

#include <stdio.h>
#include <math.h>

#include "../cephes.h"
#undef fabs
#include "misc.h"

So it will be very interesting to get a successful build log of scipy-0.8.0 in sage for inspection. A quick and dirty fix would be to change <Python.h> to <python2.6/Python.h> and I am not completely sure it would work either.

There are a few more Python.h header in that folder so just fixing that one may not work:

grep -r "Python\.h" *
amos_wrappers.h:#include "Python.h"
cdf_wrappers.h:#include "Python.h"
cephes/sici.c:#include <Python.h>
cephes/mconf.h:#include <Python.h>
_cephesmodule.c:#include "Python.h"
c_misc/gammaincinv.c:#include <Python.h>
lambertw.c:#include "Python.h"
orthogonal_eval.c:#include "Python.h"
specfun_wrappers.h:#include "Python.h"
toms_wrappers.h:#include "Python.h"
ufunc_extras.h:#include "Python.h"

considering the SConscript the files in the cephes subfolder will probably be trouble too.

comment:99 Changed 11 years ago by maldun

But it would nice to know anyway. If it doesn't work with gammaincinv.c, at least the error message would change. Providing a patch would then be easy for me.

Ironically I already had asked if this had could been the problem... The only thing what me disturbs is the this entry in the error log of kcrisman:

...y/npy_common.h:79:2: error: #error Must use Python with unicode enabled.

This sounds more like that unicode isn't enabled, and perhaps this is related to native python installation on the system, and the gcc uses this header instead that from sage. I think to be absolutely sure, we should try also to set the path really to Python.h in the sage install in the header, if setting to <python2.6/Python.h> doesn't work.

@fbissey Another question: would it help you, if you see my install log from ubuntu?

comment:100 Changed 11 years ago by fbissey

No need for extra logs. I found the culprit and why it affects only Karl's ppc machine. I point the finger to g95! or rather the fact that the patches that are installed if g95 is found should have been rebased.

Basically the file special/setup.py is replaced by a version from scipy-0.7.0 which doesn't need the python header. Could you please use spkg-install.2 attached in scipy-0.8.0 it doesn't use the patches for g95. It should work without special patch in my opinion anyway.

comment:101 Changed 11 years ago by maldun

So I've packed the scipy with your spkg-install, but this time it doesn't work for me. I get this error:

Host system
uname -a:
Linux math181 2.6.31-22-generic #63-Ubuntu SMP Thu Aug 19 00:23:50 UTC 2010 x86_64 GNU/Linux
****************************************************
****************************************************
CC Version
gcc -v
Using built-in specs.
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.4.1-4ubuntu9' --with-bugurl=file:///usr/share/doc/gcc-4.4/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --enable-shared --enable-multiarch --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.4 --program-suffix=-4.4 --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc --disable-werror --with-arch-32=i486 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.4.1 (Ubuntu 4.4.1-4ubuntu9)
****************************************************
invalid command name 'config_fc --noopt --noarch'
Error building scipy.

real    0m2.296s

comment:102 Changed 11 years ago by fbissey

I think I have gone overboard with double quotes, attaching a new version shortly.

comment:103 Changed 11 years ago by maldun

Ok. I deleted all outdated patches like you suggested (initially I let them in, because they didn't any harm, and since I didn't know much about every detail of the patches, I first tried out if it works, but of course it's better to throw them out) and it builds without problem. But with the old spkg. the spkg-install.2 you attached still, with the same error, because my machine doesn't know the command config_fc --noopt --noarch:

invalid command name 'config_fc --noopt --noarch'

could it be that I have to install something on my system? Or should we solve this with an If clause?

comment:104 follow-up: Changed 11 years ago by maldun

ok I solved it: You forgot the "-" before the config =)

Changed 11 years ago by fbissey

new replacement for scipy

comment:105 Changed 11 years ago by fbissey

I think I found the problem, I think the shell was not set up properly.

I mean, these are options passed to setup.py not separate commands. They shouldn't be interpreted as such and aren't in the numpy spkg-install, so there are no reason why they should be in scipy. But the shells for the two spkg-install were different so I think that may be a point of bash/shell semantic. In which case I am curious to find out what I am supposed to do in sh. There is a new spkg-install.2 for you to try.

comment:106 in reply to: ↑ 104 Changed 11 years ago by fbissey

Replying to maldun:

ok I solved it: You forgot the "-" before the config =)

Ok you posted while I was answering. That may pass but I am not sure that does what it's supposed to do. There shouldn't be a "-" in front. I suspect it is now just ignored.

comment:107 Changed 11 years ago by maldun

ok I was to quick... indeed it builded, but not for too long: it took it a -c"onfig_fc"... so it didn't ignore, but missinterpreted it.

But the error remains. I don't really get it is there something different with ubuntu's gcc?

comment:108 Changed 11 years ago by maldun

Ah okay: I looked into the scipy docu: This option seems to be for numpy, not for scipy. http://projects.scipy.org/numpy/wiki/DistutilsDoc#specifing-config-fc-options-for-libraries-in-setup-py-script

Because numpy builded with that command. Does it build on your system?

comment:109 Changed 11 years ago by fbissey

OK. I thought it was used in scipy as well. There seems to be 2 files with it, and it shouldn't have an impact. You can remove it then. Strange it doesn't generate errors in Gentoo.

comment:110 Changed 11 years ago by maldun

Ok updatet scipy package, with deleted patches

comment:111 Changed 11 years ago by kcrisman

Help! I couldn't work on this over the weekend, and now I'm not quite sure what I should do to test :) Should I just try installing the new Scipy 0.8, or what? I am very glad that it seems you've figured out what G95 didn't like, though. It would also be nice to see a diff between the spkg-install scripts.

comment:112 Changed 11 years ago by maldun

Ok short summary: 1) Try the new scipy 2) If this doesn't work try to change the header in gammaincinv.c in the scipy sources from <Python.h> to <python2.6/Python.h> 3) If this results in no change, rewrite the header that it takes the sage header directly means change <Python.h> to "whateverpathitmaybe/python2.6/Python.h" and look if the error message changes. (because there are other files affected, but this should be no prob to patch)

If it's really only the location of the header patching wouldn't be a problem. And Thanx for all the trouble!

I have the diffs made for on my other machine, I can post them tomorrow!

Changed 11 years ago by maldun

Diff of the spkg installs; The lines wich concern config fc are removed due problems in ubuntu

comment:113 follow-up: Changed 11 years ago by kcrisman

Well, that seems to have helped...

gcc: scipy/special/c_misc/gammaincinv.c

and it keeps going!

So it was the patches for G95 that were the problem. Which one in particular (which command/line) caused the header confusion, do you think? I'm just curious, and don't know enough about this to say.

By the way, I'm seeing things like

compile options: '-I/Users/student/Desktop/sage-4.5.2/local/lib/python2.6/site-packages/numpy/core/include -I/Users/student/Desktop/sage-4.5.2/local/include/python2.6 -c'

so hopefully that is also a positive sign.

comment:114 Changed 11 years ago by kcrisman

Success! Now I'll set SAGE_CHECK=yes, retry numpy, do ./sage -ba, retry scipy, and test the Sage library. It will take a LONG time but hopefully tomorrow I'll have a report for you.

comment:115 Changed 11 years ago by maldun

That are great news! Hopefully everything works!

comment:116 Changed 11 years ago by maldun

Ah and don't forget the patch for the doctests in the attachement. Else there would 2 doctests fail due to output changes in numpy.

comment:117 follow-ups: Changed 11 years ago by kcrisman

I'm not worried about that, but I'll do it when I get there. Numpy just finished installing successfully.

Question:

# Program around a bug in SciPY's distutils.
unset CFLAGS

python setup.py install ${NUMPY_FCONFIG}

I assume this is still needed in the numpy spkg-install. Just checking.

Question: I know drkirkby is cc:ed on this ticket. Numpy seems to have a very easy to run test suite - see here. However, it requires the Nose package. Maybe this should be a separate ticket, but it would seem reasonable to include Nose, which is about 250 KB, so that we can run the numpy tests. I am curious as to how we run the scipy tests without it!

comment:118 in reply to: ↑ 113 Changed 11 years ago by fbissey

Replying to kcrisman:

Well, that seems to have helped...

gcc: scipy/special/c_misc/gammaincinv.c

and it keeps going!

So it was the patches for G95 that were the problem. Which one in particular (which command/line) caused the header confusion, do you think? I'm just curious, and don't know enough about this to say.

Well, there are 4 patches that were applied in case g95 was found as the fortran compiler. The faulty one is special.setup.py (or possibly setup.py.special) which replaces the file setup.py in the folder scipy/special.

This setup.py file has changed quite a lot between version 0.7.0 and 0.8.0. The lines at fault:

    # C libraries
    config.add_library('sc_c_misc',sources=[join('c_misc','*.c')])
    config.add_library('sc_cephes',sources=[join('cephes','*.c')],
                       include_dirs=[get_python_inc(), get_numpy_include_dirs()],
                       macros=define_macros)

in version 0.7.0 and this is the code found in the "patch" file and

    # C libraries
    config.add_library('sc_c_misc',sources=[join('c_misc','*.c')],
                       include_dirs=[get_python_inc(), get_numpy_include_dirs()],
                       macros=define_macros)
    config.add_library('sc_cephes',sources=[join('cephes','*.c')],
                       include_dirs=[get_python_inc(), get_numpy_include_dirs()],
                       macros=define_macros)

in version 0.8.0. This is a case were using patches rather copying files wholesale would have prevented the problem. Either the patch wouldn't have worked with the newer version and we would have known straight away or it would have applied with some fuzz without dommaging this particular section.

comment:119 in reply to: ↑ 117 Changed 11 years ago by fbissey

Replying to kcrisman:

I'm not worried about that, but I'll do it when I get there. Numpy just finished installing successfully.

Question:

# Program around a bug in SciPY's distutils.
unset CFLAGS

python setup.py install ${NUMPY_FCONFIG}

I assume this is still needed in the numpy spkg-install. Just checking.

Question: I know drkirkby is cc:ed on this ticket. Numpy seems to have a very easy to run test suite - see here. However, it requires the Nose package. Maybe this should be a separate ticket, but it would seem reasonable to include Nose, which is about 250 KB, so that we can run the numpy tests. I am curious as to how we run the scipy tests without it!

Actually Dave has started a thread about this a few weeks ago on the mailing list, you are welcome to add your opinion and revive the thread.

comment:120 in reply to: ↑ 117 Changed 11 years ago by fbissey

Replying to kcrisman:

I'm not worried about that, but I'll do it when I get there. Numpy just finished installing successfully.

Question:

# Program around a bug in SciPY's distutils.
unset CFLAGS

python setup.py install ${NUMPY_FCONFIG}

I assume this is still needed in the numpy spkg-install. Just checking.

I thought I had asked maldun to remove this. It's working fine on Gentoo as it is, I am assuming this has been fixed upstream (possibly a while ago).

comment:121 follow-up: Changed 11 years ago by maldun

I added the

# Program around a bug in SciPY's distutils.
unset CFLAGS

into the numpy spkg-install from the old one, just to be sure that we don't run into troubles, and actually it doesn't do any harm... but if you want I can remove it as well.

But didn't we agree that the setup.py install ${NUMPY_FCONFIG} is necessary in numpy? I had to delete it from scipy or else it doesn't build anymore, but in numpy it's correct.

comment:122 in reply to: ↑ 121 Changed 11 years ago by fbissey

Replying to maldun:

I added the

# Program around a bug in SciPY's distutils.
unset CFLAGS

into the numpy spkg-install from the old one, just to be sure that we don't run into troubles, and actually it doesn't do any harm... but if you want I can remove it as well.

We should remove it. If there is a problem - which I doubt, we can put it back.

But didn't we agree that the setup.py install ${NUMPY_FCONFIG} is necessary in numpy? I had to delete it from scipy or else it doesn't build anymore, but in numpy it's correct.

Yes! Leave it here.

comment:123 Changed 11 years ago by maldun

Ok I removed the line.

I also thougth before posting I should check if there are some old rests in there, and I removed some patches that are possible(?) outdated. I think this could affect the cygwin installation, so it would be important to see if it still builds on all machines, or else I have to carefully merge the changed lines, from the old patches with the new versions of the files.

It works for me, so I hope I didn't cause some new problems

comment:124 Changed 11 years ago by maldun

I have a different problem: When I upload the new Version, and try to download it I get the old one. Is there some confusion on the server?

comment:125 Changed 11 years ago by maldun

Ok worked. Don't ask why...

comment:126 follow-ups: Changed 11 years ago by fbissey

  • Status changed from needs_work to needs_review

Ok - now this needs review!

I looked at the cygwin-lapack_lite-setup.py patch. The file patched hasn't changed between numpy-1.3.x and 1.4.1, so it can still be applied safely if needed.

In summary the only problems overall are:

  • numpy 1.4.1 is broken on linux ppc (on the official list of supported OS - so bad) and alpha (not on the supported list - so OK).
  • numpy-1.5.0 has an issues with cython that will be solved in 1.5.1 - shame because it also fixes linux ppc and alpha.

Anything to add?

comment:127 in reply to: ↑ 126 Changed 11 years ago by maldun

Replying to fbissey:

Anything to add?

as mentioned above: The version of numpy-1.5.0 with the patch applied is ok. The only trouble I have are some doctests, which fails because numpy throws a "Division by zero encountered!" Exception, which makes them fail, although the results are correct. But I still don't know how to catch the exceptions.

comment:128 Changed 11 years ago by fbissey

I should say that while breaking on linux ppc is not good, I only know of one other person building sage on that platform. So it could take the backseat while problems with the 1.5 series are ironed out.

comment:129 in reply to: ↑ 126 ; follow-up: Changed 11 years ago by kcrisman

  • numpy-1.5.0 has an issues with cython that will be solved in 1.5.1 - shame because it also fixes linux ppc and alpha.

Hmm, dropping an official thing is not so good. I think that one would want to post to the list about this - more likely wait for 1.5.1? Or is the fix for this easily backported?

Also, although I've been going for over 24 hours now on the tests (and several timeouts! so I'll have to run those with SAGE_TIMEOUT_LONG>3600), I have only seen two errors, both like this:

sage -t -long "devel/sage/sage/graphs/digraph.py"           
**********************************************************************
File "/Users/student/Desktop/sage-4.5.2/devel/sage/sage/graphs/digraph.py", line 204:
    sage: DiGraph(A)
Exception raised:
    Traceback (most recent call last):
      File "/Users/student/Desktop/sage-4.5.2/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/Users/student/Desktop/sage-4.5.2/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/Users/student/Desktop/sage-4.5.2/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_1[17]>", line 1, in <module>
        DiGraph(A)###line 204:
    sage: DiGraph(A)
      File "/Users/student/Desktop/sage-4.5.2/local/lib/python/site-packages/sage/graphs/digraph.py", line 364, in __init__
        data = networkx.MultiDiGraph(data)
      File "/Users/student/Desktop/sage-4.5.2/local/lib/python2.6/networkx/classes/digraph.py", line 217, in __init__
        convert.from_whatever(data,create_using=self)
      File "/Users/student/Desktop/sage-4.5.2/local/lib/python2.6/networkx/convert.py", line 157, in from_whatever
        if isinstance(thing,numpy.core.defmatrix.matrix) or \
    AttributeError: 'module' object has no attribute 'defmatrix'
**********************************************************************

Have you seen this? Clearly it's something to do with numpy, from the end - but it's not clear to me why it would be asking whether a module has defmatrix attribute.

comment:130 follow-up: Changed 11 years ago by maldun

I know this error. Since you use sage 4.5.2 you will also use networkx-1.0.1. See comments 24 and following or ticket #9854.

You have to either use Sage 4.5.3 which holds networkx-1.2, upgrade networkx-1.0.1 in your current version, or apply the networkx-1.0.1.p0.spkg in the attachement, which holds a simple patch.

The reason is that numpy.core.defmatrix moved to numpy.matrixlib.defmatrix.

comment:131 Changed 11 years ago by maldun

@linux ppc I could work over the weekend the solve the failed doctests in numpy-1.5.0, because with the patch from the numpy developers the biggest problem i.e. the cython problem is solved. The only thing remains are some "Division by Zero Warnings" as mentioned above.

comment:132 Changed 11 years ago by maldun

And another thing: If we aren't able to merge numpy and scipy to 4.6. in time, why don't we provide the packages as optional or experimental? Because they work on most systems, and it would be a waste to just throw them away.

comment:133 in reply to: ↑ 129 ; follow-up: Changed 11 years ago by drkirkby

Replying to kcrisman:

  • numpy-1.5.0 has an issues with cython that will be solved in 1.5.1 - shame because it also fixes linux ppc and alpha.

Hmm, dropping an official thing is not so good. I think that one would want to post to the list about this - more likely wait for 1.5.1? Or is the fix for this easily backported?

Depending on what list you read, Linux on PPC may or may not be supported. I've tried to get an agreement before on what we do support and what we don't, but never got anywhere.

I'm personally in favor of keeping Sage building on less regular systems, as it often indicates problems. The Solaris port has found endless bugs in Sage - some of which have even according to William been serious.

I have an old IBM RS/6000 7025 F50 (Power PC) which I was setting up AIX on the other day. I contemlated whether I should put Linux on it too for testing Sage. But when I read Sage's README.txt, PowerPC was only supported on OS X, not Linux. So I did not bother. But the machine has space for 18 disk drives, so I could always add it!!

Dave

comment:134 in reply to: ↑ 133 Changed 11 years ago by kcrisman

I forgot about the networkx change, thanks for the reminder.

  • numpy-1.5.0 has an issues with cython that will be solved in 1.5.1 - shame because it also fixes linux ppc and alpha.

Hmm, dropping an official thing is not so good. I think that one would want to post to the list about this - more likely wait for 1.5.1? Or is the fix for this easily backported?

I'm personally in favor of keeping Sage building on less regular systems, as it often indicates problems. The Solaris port has found endless bugs in Sage - some of which have even according to William been serious.

I have an old IBM RS/6000 7025 F50 (Power PC) which I was setting up AIX on the other day. I contemlated whether I should put Linux on it too for testing Sage. But when I read Sage's README.txt, PowerPC was only supported on OS X, not Linux. So I did not bother. But the machine has space for 18 disk drives, so I could always add it!!

Why not? I've been trying to figure out how to get a thumb drive to boot my PPC machine on Ubuntu (and possibly other distros) but it turns out it's devilishly hard to get that to work, and I haven't had time to do it. I can't find an easy virtualization option either (neither xen nor anything else seem to really be particularly easy, if they even still work with such an old machine...)

And another thing: If we aren't able to merge numpy and scipy to 4.6. in time, why don't we provide the packages as optional or experimental? Because they work on most systems, and it would be a waste to just throw them away.

Certainly one could do this, but even that would require a positive review from someone. You might have to ask on the list about this - I've never heard of having an upgrade to a normal package as only optional.

comment:135 in reply to: ↑ 130 Changed 11 years ago by kcrisman

You have to either use Sage 4.5.3 which holds networkx-1.2, upgrade networkx-1.0.1 in your current version, or apply the networkx-1.0.1.p0.spkg in the attachement, which holds a simple patch.

Incidentally, I also had to repackage that spkg, because you didn't use ./sage -pkg for it either :) and it had unchecked in changes :) But clearly that is irrelevant for testing, since the upgrade is already in a later Sage.

Okay, with this upgrade those two tests pass. I am currently testing the other four failures, all of which were timeouts, all of which are in files I personally know to have VERY long -long doctests, and all but one of which hopefully will pass with SAGE_TIMEOUT_LONG=10000 (there is one in interfaces/maxima.py which always fails on OS X 10.4, as far as I know, which is due to tab-completion testing of Maxima never finishing, totally unrelated to this). I'll let you know if something *doesn't* work.

So I think that my job is done here (pointing out that I was testing the version from before the weekend. fbissey or drkirkby should comment on the spkg itself.

comment:136 follow-up: Changed 11 years ago by fbissey

Well thanks Karl! Your testing was very useful. Since I have contributed to bits and pieces I think I should had myself as an author. But most of the heavy lifting as been done by maldun.

So what's left is the question of whether we go for numpy-1.4.1 and leave minor archs unsupported for a little bit. Or we wait for 1.5.1, which works on the minor archs in question and will play well with cython.

As an aside I have already pushed the upgrade in sage-on-gentoo (to avoid tree rote, we already have to keep two old packages that are otherwise removed from Gentoo) and we want to avoid that kind of stuff as much as possible. So the current code is out there and used by a few people.

After that there are question of details. In spkg-install I have set FC to ${SAGE_LOCAL}/bin/sage_fortran, with the idea that it was basically calling "gfortran -fpic" or the g95 equivalent. We have been talking about that very subject on sage-devel recently. Is it the best way to go? If one uses sunstudio (and I am planning to give a go) the correct flag would be -Kpic but would it be set up properly in sage_fortran? I doubt it.

So what would be the best course of action? Using the variable SAGE_FORTRAN for now and ask it to be set with the proper pic flag and hopefully drop it later on when FC is the mainstay?

comment:137 in reply to: ↑ 136 Changed 11 years ago by maldun

  • Authors changed from Stefan Reiterer to Stefan Reiterer, Francois Bissey

Replying to fbissey:

Well thanks Karl! Your testing was very useful. Since I have contributed to bits and pieces I think I should had myself as an author. But most of the heavy lifting as been done by maldun.

I think you are right. So I allowed myself to set you as co-author, since this whole thing wouldn't work without your help =)

So what's left is the question of whether we go for numpy-1.4.1 and leave minor archs unsupported for a little bit. Or we wait for 1.5.1, which works on the minor archs in question and will play well with cython.

I would suggest the following: If numpy 1.5.1 doesn't come out before Sage 4.6. let's take 1.4.1 (since ppc linux is not on the current supported list; see readme.tex of sage) I will start this weekend with correcting things of 1.5.0, since the current errors are new warnings, and I'm quite sure numpy 1.5.1 will throw those as well. When numpy 1.5.1 is out before 4.6. I will try to pack it, and then hopefully it is done in no time, since it shared the same probs.

As an aside I have already pushed the upgrade in sage-on-gentoo (to avoid tree rote, we already have to keep two old packages that are otherwise removed from Gentoo) and we want to avoid that kind of stuff as much as possible. So the current code is out there and used by a few people.

After that there are question of details. In spkg-install I have set FC to ${SAGE_LOCAL}/bin/sage_fortran, with the idea that it was basically calling "gfortran -fpic" or the g95 equivalent. We have been talking about that very subject on sage-devel recently. Is it the best way to go? If one uses sunstudio (and I am planning to give a go) the correct flag would be -Kpic but would it be set up properly in sage_fortran? I doubt it.

So what would be the best course of action? Using the variable SAGE_FORTRAN for now and ask it to be set with the proper pic flag and hopefully drop it later on when FC is the mainstay?

The question is how fast this change with the fortran compiler is done. I think we should wait a little bit now.

My question is: should this given a positive review, or is something still missing?

comment:138 follow-up: Changed 11 years ago by mhampton

I'd like to check this out on sage-4.6.alpha1, but its unclear to me exactly what to do. Can someone summarize the current procedure for incorporating this ticket?

comment:139 in reply to: ↑ 138 Changed 11 years ago by maldun

Replying to mhampton:

I'd like to check this out on sage-4.6.alpha1, but its unclear to me exactly what to do. Can someone summarize the current procedure for incorporating this ticket?

Install notes and links to the packages are in the discription of this ticket!

Quick and dirty: Install the packagages, do sage -ba to compile the whole source, apply the patch in the attachment for the doctests.

greez maldun

comment:140 Changed 11 years ago by fbissey

Just a quick note that I used this package in sage-4.6alpha1 and it didn't break any tests on linux-x86, I haven't heard of any problems on linux-amd64 from a friend who was also testing this. So it applies cleanly on 4.6alpha1.

comment:141 follow-up: Changed 11 years ago by leif

Dave pointed me to this ticket the day before yesterday (and I read it from the beginning). ;-)

Despite that, it's unclear to me which diffs are currrent, and the description should IMHO be updated wrt. what's to be done to review this.

./sage -ba is also in my opinion not a solution to get dependent extension modules updated, i.e. if ./sage -b isn't sufficient, dependencies should be added to module_list.py.

comment:142 in reply to: ↑ 141 ; follow-up: Changed 11 years ago by maldun

  • Description modified (diff)

Replying to leif:

Dave pointed me to this ticket the day before yesterday (and I read it from the beginning). ;-)

Despite that, it's unclear to me which diffs are currrent, and the description should IMHO be updated wrt. what's to be done to review this.

./sage -ba is also in my opinion not a solution to get dependent extension modules updated, i.e. if ./sage -b isn't sufficient, dependencies should be added to module_list.py.

I cleaned up the discription a little bit, and deleted the obsolete remark about networkx-1.2

I don't know if this is a good Idea because this is a whole bunch of files we are talking about, which have to be added. Perhaps you won't save much time either when compiling. I first tried this too, but after about 2 hours searching, I got somehow tired of this... And after merging numpy > 1.3.x into Sage this has not to be done anymore. All versions of 1.5.x builded without problems, and didn't need -ba anymore. You only have to do this when big changes are happening, and this is rather occassionally.

package with a linux ppc patch is out: http://code.google.com/p/spkg-upload/downloads/detail?name=numpy-1.4.1.p0.spkg hope it works this time!

comment:143 follow-up: Changed 11 years ago by maldun

Ah before I forget it, and you think I'm just lazy: I just looked up again in an non compiled version which files are concerned, and I remembered again what was really the problem Most of them are already in the module_list.py.

However the problem is, that there are actually no changes to that files, so sage -b won't do nothing, because it does'nt recognice any changes!

only -ba force sage to do that and link the whole c files to the new version of numpy.

You would have to change the behavior of -b, not add something to the module_list.py

comment:144 Changed 11 years ago by leif

Just for the record, gcc has the following option:

-mabi=ieeelongdouble

Change the current ABI to use IEEE extended precision long double. This is a PowerPC 32-bit Linux ABI option.

I don't know if this requires a different (or differently built) libc however.

comment:145 in reply to: ↑ 143 ; follow-up: Changed 11 years ago by leif

Replying to maldun:

Ah before I forget it, and you think I'm just lazy: I just looked up again in an non compiled version which files are concerned, and I remembered again what was really the problem Most of them are already in the module_list.py.

However the problem is, that there are actually no changes to that files, so sage -b won't do nothing, because it does'nt recognice any changes!

So we have to add (at least some of) those files to depends that actually have changed.

only -ba force sage to do that and link the whole c files to the new version of numpy.

You would have to change the behavior of -b, not add something to the module_list.py

That's IMHO not an option, since missing dependencies will break any Sage upgrade anyway.

P.S.: The attached diffs are still confusing. (One should at least change/update their descriptions.) For review, complete diffs between the old and new spkgs (of course excluding src/) would be helpful.

comment:146 in reply to: ↑ 145 ; follow-ups: Changed 11 years ago by maldun

Replying to leif:

So we have to add (at least some of) those files to depends that actually have changed.

And that's the problem: There are NO files which would have changed! Not within sage/devel/sage-whateverbranch. And that's also the reason for the warnings: numpy wonders why still the old size of flatiter is in the .c files. The only changes are in local/LIB/python/site-packages/numpy, and -b don't care about those. The only way to trick -b was to put in a comment to the certain files, delete them again and save it, that sage -b recognices the changes.

I when kcrisman gives me feedback if numpy works now on ppc, I will put a complete changelog here on trac. Perhaps over the weekend I find some time for that.

comment:147 in reply to: ↑ 146 ; follow-up: Changed 11 years ago by kcrisman

  • Reviewers changed from Karl-Dieter Crisman to Karl-Dieter Crisman, David Kirkby, Leif Leonhardy, Francois Bissey

I when kcrisman gives me feedback if numpy works now on ppc, I will put a complete changelog here on trac. Perhaps over the weekend I find some time for that.

You mean on OS X, not Linux, right?

Numpy installed fine, and Scipy got past the bad spot fine, so I don't anticipate any problems. Then I'll run tests.

Leif, is this really a problem? (Asked as a query, not a complaint.) Presumably someone will only use these packages via sage-upgrade, and then the whole library rebuilds (because sage-x.y.z is new), right? Or is that not how sage-upgrade works?

I'm adding a few reviewers based on how things have gone over these ~150 comments.

comment:148 in reply to: ↑ 146 Changed 11 years ago by leif

Replying to maldun:

Replying to leif:

So we have to add (at least some of) those files to depends that actually have changed.

And that's the problem: There are NO files which would have changed! Not within sage/devel/sage-whateverbranch. And that's also the reason for the warnings: numpy wonders why still the old size of flatiter is in the .c files.

So what? Then just make them also depend on e.g. $SAGE_LOCAL/lib/python/site-packages/numpy/core/include/numpy/numpyconfig.h (if that has changed).

comment:149 in reply to: ↑ 147 Changed 11 years ago by leif

Replying to kcrisman:

Leif, is this really a problem? (Asked as a query, not a complaint.) Presumably someone will only use these packages via sage-upgrade, and then the whole library rebuilds (because sage-x.y.z is new), right? Or is that not how sage-upgrade works?

The Sage library is "new" in every Sage version (even if there were no real changes to it), but does not get rebuilt from scratch (since there's no reason to do so if the dependencies are correct [or at least close to complete].)

comment:150 Changed 11 years ago by leif

P.S.: $SAGE_LOCAL/lib/python/site-packages/Cython/Includes/numpy.pxd could be an option, too.

comment:151 Changed 11 years ago by maldun

I can do that, but this still won't solve the problem. Perhaps you misunderstand me, so I explain it again in more detail.

After installing numpy the only thing that has to be done is to recompile the .pyx files (which don't have changed), so that the old .c files (which haven't changed either) have be builded with the new version of numpy, that actually the change happens. I'm not a pro to sage, but as far I understood that is, that -b only recompile changed files. No changes, no compilation. Even if I add now the changed files (the package) sage only would recompile those, and leaves out a whole lot of files which also have to be recompiled. The only way to get now sage to recompile those .pyx which is to change the files after installation for example by putting a comment into them, or force sage to recompile them, that is that what -ba does. since there is no option that I know to force recompilation, I don't have an Idea how else I could get it working.

If you have a better way please let me know. I will apply a patch as soon as possible!

@kcrisman: didn't we talk about linux ppc? You said it builds on OS X ppc, or am I wrong?

comment:152 Changed 11 years ago by kcrisman

No, I'm sorry if you misread my post about that. I would *like* to do so but there are still a number of technical hurdles to me running Linux PPC on my box (I think this should be clear in my sage-devel post). I am testing whether the change you made to support Linux PPC doesn't break anything. Sorry if this is not useful :( and I wish I did have access to Linux PPC more easily; unfortunately, this requires either having access to an already existing Linux PPC machine OR trying to run an image of Linux on a CD even with so little RAM there is no point OR using a portable hard drive, which is a more significant expense.

comment:153 Changed 11 years ago by leif

Apparently we have to add Carl Friedrich Gauss to the authors... ;-)

comment:154 Changed 11 years ago by maldun

As I feared: Making them dependent didn't force compilation either.

comment:155 Changed 11 years ago by maldun

Update: If you are changing the numpy.pxd file (for example insert a blank line and delete it) sage -b compile the dependent sources, but this file is in cython not in numpy.

manual changes on the numpy header files didn't do anything.

@Leif I know what you are trying to tell me. I would have expected that making a header dependent would get sage to resolve the dependencies and recompile the .pyx files also.

But even little changes to the header files do nothing even if they are dependent, or I forced dependency.

I didn't meant this as offense or something, I just told you what I had experienced in the early stages of this ticket. If you have a solution for example a diff with an entry that forces dependency, (plot3d for example) I could write the others in no time.

Perhaps I do something wrong. If you know what the trouble is please tell me.

comment:156 follow-up: Changed 11 years ago by leif

There are only a few headers that actually have been updated, e.g. _numpyconfig.h.

A trivial solution is to just touch one header file after installation in spkg-install, and make the relevant extension modules (also) depend on that one.

Touching $SAGE_LOCAL/lib/python/site-packages/Cython/Includes/numpy.pxd could also help.

comment:157 in reply to: ↑ 156 ; follow-up: Changed 11 years ago by leif

Replying to leif:

Touching $SAGE_LOCAL/lib/python/site-packages/Cython/Includes/numpy.pxd could also help.

Doing so triggers at least recompilation of the 13 extension modules that add the numpy include dir to their include_dirs.

comment:158 Changed 11 years ago by fbissey

Lots happens when I am asleep and lecturing first thing in the morning :)

I cannot comment on the sage-release thread from work. I am one of the few people running sage on linux ppc - more as hobby to check that it works than real production. The problem for numpy on ppc is fixed in numpy-1.5.

cannot comment on the rest right now I have a truckload of exams to prepare.

comment:159 in reply to: ↑ 157 ; follow-up: Changed 11 years ago by leif

Replying to leif:

Replying to leif:

Touching $SAGE_LOCAL/lib/python/site-packages/Cython/Includes/numpy.pxd could also help.

Doing so triggers at least recompilation of the 13 extension modules that add the numpy include dir to their include_dirs.

... and caused ptestlong to pass without any errors. So touching just that file in numpy's spkg-install should fix all dependency issues, without touching module_list.py at all (which is advantageous, too).

I just wonder that (or if) that file is still compatible with the new numpy.


Regarding the discussion on sage-release, who prepares a numpy 1.5.0 (or later) spkg? ;-)

The 1.4.1 seems obsolete now...

comment:160 in reply to: ↑ 159 Changed 11 years ago by leif

Replying to leif:

So touching just that file in numpy's spkg-install should fix all dependency issues, without touching module_list.py at all (which is advantageous, too).

Unfortunately this will only work if Cython is not updated/rebuilt (after numpy has been installed), since none of these packages depends on the other, and Cython might still ship one with an old time stamp (which is preserved on installation).

comment:161 Changed 11 years ago by maldun

Regarding the discussion on sage-release, who prepares a numpy 1.5.0 (or later) spkg? ;-)

The 1.4.1 seems obsolete now...

I do. There are only 2 doctests that fails, due to output changes inf -> Inf I hope I can complete it over the weekend.

Actually there is another problem with the headers: You would have to make ALL haeaders dependent on the concerning entries in the module_list.py, because in every version of numpy there can be an other headers that changes... and of course that is a matter of taste, but wouldn't get this the module_list.py a little bit messy, with dependencies that actually aren't needed?

just my 2 cents

comment:162 in reply to: ↑ 142 Changed 11 years ago by fbissey

Replying to maldun:

Replying to leif:

Dave pointed me to this ticket the day before yesterday (and I read it from the beginning). ;-)

Despite that, it's unclear to me which diffs are currrent, and the description should IMHO be updated wrt. what's to be done to review this.

./sage -ba is also in my opinion not a solution to get dependent extension modules updated, i.e. if ./sage -b isn't sufficient, dependencies should be added to module_list.py.

I cleaned up the discription a little bit, and deleted the obsolete remark about networkx-1.2

I don't know if this is a good Idea because this is a whole bunch of files we are talking about, which have to be added. Perhaps you won't save much time either when compiling. I first tried this too, but after about 2 hours searching, I got somehow tired of this... And after merging numpy > 1.3.x into Sage this has not to be done anymore. All versions of 1.5.x builded without problems, and didn't need -ba anymore. You only have to do this when big changes are happening, and this is rather occassionally.

package with a linux ppc patch is out: http://code.google.com/p/spkg-upload/downloads/detail?name=numpy-1.4.1.p0.spkg hope it works this time!

Ok so I tried to build numpy-1.4.1 with our patch on ppc. Didn't work. It may be a good start but it fails in the same way it did before:

[39mcompile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/usr/include/python2.6 -c'[0m
[39mpowerpc-unknown-linux-gnu-gcc: _configtest.c[0m
[39mremoving: _configtest.c _configtest.o[0m
Traceback (most recent call last):
  File "setup.py", line 187, in <module>
    setup_package()
  File "setup.py", line 180, in setup_package
    configuration=configuration )
  File "/scratch/portage/portage/dev-python/numpy-1.4.1/work/numpy-1.4.1/numpy/distutils/core.py", line 186, in setup
    return old_setup(**new_attr)
  File "/usr/lib/python2.6/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib/python2.6/distutils/dist.py", line 975, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command
    cmd_obj.run()
  File "/scratch/portage/portage/dev-python/numpy-1.4.1/work/numpy-1.4.1/numpy/distutils/command/build.py", line 37, in run
    old_build.run(self)
  File "/usr/lib/python2.6/distutils/command/build.py", line 134, in run
    self.run_command(cmd_name)
  File "/usr/lib/python2.6/distutils/cmd.py", line 333, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command
    cmd_obj.run()
  File "/scratch/portage/portage/dev-python/numpy-1.4.1/work/numpy-1.4.1/numpy/distutils/command/build_src.py", line 152, in run
    self.build_sources()
  File "/scratch/portage/portage/dev-python/numpy-1.4.1/work/numpy-1.4.1/numpy/distutils/command/build_src.py", line 169, in build_sources
    self.build_extension_sources(ext)
  File "/scratch/portage/portage/dev-python/numpy-1.4.1/work/numpy-1.4.1/numpy/distutils/command/build_src.py", line 328, in build_extension_sources
    sources = self.generate_sources(sources, ext)
  File "/scratch/portage/portage/dev-python/numpy-1.4.1/work/numpy-1.4.1/numpy/distutils/command/build_src.py", line 385, in generate_sources
    source = func(extension, build_dir)
  File "numpy/core/setup.py", line 416, in generate_config_h
    rep = check_long_double_representation(config_cmd)
  File "numpy/core/setup_common.py", line 136, in check_long_double_representation
    type = long_double_representation(pyod(object))
  File "numpy/core/setup_common.py", line 244, in long_double_representation
    raise ValueError("Unrecognized format (%s)" % saw)
ValueError: Unrecognized format (['001', '043', '105', '147', '211', '253', '315', '357', '301', '235', '157', '064', '124', '000', '000', '000', '000', '000', '000', '000', '000', '000', '000', '000', '376', '334', '272', '230', '166', '124', '062', '020'])
 [31;01m*[0m ERROR: dev-python/numpy-1.4.1 failed:

The expectation from numpy/core/setup_common.py is:

LONG_DOUBLE_REPRESENTATION_SRC = r"""
/* "before" is 16 bytes to ensure there's no padding between it and "x".
 *    We're not expecting any "long double" bigger than 16 bytes or with
 *       alignment requirements stricter than 16 bytes.  */
typedef %(type)s test_type;

struct {
        char         before[16];
        test_type    x;
        char         after[8];
} foo = {
        { '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0',
          '\001', '\043', '\105', '\147', '\211', '\253', '\315', '\357' },
        -123456789.0,
        { '\376', '\334', '\272', '\230', '\166', '\124', '\062', '\020' }
};
"""

And when you compare the two you can see that it is an endianess problem.

comment:163 Changed 11 years ago by maldun

  • Description modified (diff)
  • Summary changed from Upgrade numpy to 1.4.1 and scipy to 0.8 to Upgrade numpy to 1.5.0 and scipy to 0.8

Changed 11 years ago by maldun

Changes of the spkg-install from the upgrade to 1.3.x to 1.5.0 (changes that were applied to 1.4.1 are also contained)

Changed 11 years ago by maldun

diff of the spkg-install of scipy from 0.7.p5 to 0.8

comment:164 Changed 11 years ago by maldun

  • Description modified (diff)

comment:165 Changed 11 years ago by fbissey

I am incorporating that in my 4.6alpha2 branch on Gentoo right now.

I will unfortunately be unable to test it on linux ppc before Wednesday 6th because of limited access to the hardware. So it may be too short for me to do a comprehensive review for the alpha3 deadline. I already know for sure that numpy-1.5.0/scipy-0.8.0 will build it's just the testing that will be lacking.

comment:166 Changed 11 years ago by fbissey

Build on linux x86 and tests pass here on 4.6alpha2.

comment:167 Changed 11 years ago by jhpalmieri

What do I need to do to install these? I did this: unpacked a sage-4.6.alpha2 tarball, downloaded the new numpy and scipy spkg files, applied the file "trac_9808_numpy_doctest_change.patch" to the main Sage library spkg, and built Sage from scratch (with SAGE_CHECK=yes, except when installing the python spkg, which always fails self-tests). On taurus (a skynet linux machine, x86_64-Linux-nehalem-fc), I'm getting doctest failures:

The following tests failed:

        sage -t  -long devel/sage/sage/plot/plot.py # 12 doctests failed
        sage -t  -long devel/sage/sage/misc/citation.pyx # 2 doctests failed
        sage -t  -long devel/sage/sage/plot/misc.py # 4 doctests failed

For example:

sage -t  -long devel/sage/sage/misc/citation.pyx
**********************************************************************
File "/home/palmieri/taurus/numpy/sage-4.6.alpha2/devel/sage-main/sage/misc/citation.pyx", line 60\
:
    sage: get_systems('integrate(x^2, x)')
Expected:
    ['ginac', 'Maxima']
Got:
    ['numpy', 'ginac', 'Maxima']
**********************************************************************

and

sage -t  -long devel/sage/sage/plot/plot.py
**********************************************************************
File "/home/palmieri/taurus/numpy/sage-4.6.alpha2/devel/sage-main/sage/plot/plot.py", line 207:
    sage: (g1+g2).show(ticks=pi/6, tick_formatter=pi)  # show their sum, nicely formatted
Exception raised:
    Traceback (most recent call last):
      File "/home/palmieri/taurus/numpy/sage-4.6.alpha2/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/home/palmieri/taurus/numpy/sage-4.6.alpha2/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/home/palmieri/taurus/numpy/sage-4.6.alpha2/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_0[49]>", line 1, in <module>
        (g1+g2).show(ticks=pi/Integer(6), tick_formatter=pi)  # show their sum, nicely formatted###line 207:
    sage: (g1+g2).show(ticks=pi/6, tick_formatter=pi)  # show their sum, nicely formatted
      File "/home/palmieri/taurus/numpy/sage-4.6.alpha2/local/lib/python/site-packages/sage/plot/plot.py", line 1546, in show
        self.save(DOCTEST_MODE_FILE, **options)
      File "/home/palmieri/taurus/numpy/sage-4.6.alpha2/local/lib/python/site-packages/sage/plot/plot.py", line 2202, in save
        figure.savefig(filename,dpi=dpi,bbox_inches='tight',**options)
      File "/home/palmieri/taurus/numpy/sage-4.6.alpha2/local/lib/python/site-packages/matplotlib/figure.py", line 1032, in savefig
        self.canvas.print_figure(*args, **kwargs)
      File "/home/palmieri/taurus/numpy/sage-4.6.alpha2/local/lib/python/site-packages/matplotlib/backend_bases.py", line 1455, in print_figure
        **kwargs)
      File "/home/palmieri/taurus/numpy/sage-4.6.alpha2/local/lib/python/site-packages/matplotlib/backends/backend_agg.py", line 358, in print_png
        FigureCanvasAgg.draw(self)
      File "/home/palmieri/taurus/numpy/sage-4.6.alpha2/local/lib/python/site-packages/matplotlib/backends/backend_agg.py", line 314, in draw
        self.figure.draw(self.renderer)
      File "/home/palmieri/taurus/numpy/sage-4.6.alpha2/local/lib/python/site-packages/matplotlib/artist.py", line 46, in draw_wrapper
        draw(artist, renderer, *args, **kwargs)
      File "/home/palmieri/taurus/numpy/sage-4.6.alpha2/local/lib/python/site-packages/matplotlib/figure.py", line 773, in draw
        for a in self.axes: a.draw(renderer)
      File "/home/palmieri/taurus/numpy/sage-4.6.alpha2/local/lib/python/site-packages/matplotlib/artist.py", line 46, in draw_wrapper
        draw(artist, renderer, *args, **kwargs)
      File "/home/palmieri/taurus/numpy/sage-4.6.alpha2/local/lib/python/site-packages/matplotlib/axes.py", line 1735, in draw
        a.draw(renderer)
      File "/home/palmieri/taurus/numpy/sage-4.6.alpha2/local/lib/python/site-packages/matplotlib/artist.py", line 46, in draw_wrapper
        draw(artist, renderer, *args, **kwargs)
      File "/home/palmieri/taurus/numpy/sage-4.6.alpha2/local/lib/python/site-packages/matplotlib/axis.py", line 742, in draw
        tick.draw(renderer)
      File "/home/palmieri/taurus/numpy/sage-4.6.alpha2/local/lib/python/site-packages/matplotlib/artist.py", line 46, in draw_wrapper
        draw(artist, renderer, *args, **kwargs)
      File "/home/palmieri/taurus/numpy/sage-4.6.alpha2/local/lib/python/site-packages/matplotlib/axis.py", line 196, in draw
        self.label1.draw(renderer)
      File "/home/palmieri/taurus/numpy/sage-4.6.alpha2/local/lib/python/site-packages/matplotlib/text.py", line 518, in draw
        bbox, info = self._get_layout(renderer)
      File "/home/palmieri/taurus/numpy/sage-4.6.alpha2/local/lib/python/site-packages/matplotlib/text.py", line 280, in _get_layout
        clean_line, self._fontproperties, ismath=ismath)
      File "/home/palmieri/taurus/numpy/sage-4.6.alpha2/local/lib/python/site-packages/matplotlib/backends/backend_agg.py", line 156, in get_text_width_height_descent
        self.mathtext_parser.parse(s, self.dpi, prop)
      File "/home/palmieri/taurus/numpy/sage-4.6.alpha2/local/lib/python/site-packages/matplotlib/mathtext.py", line 2797, in parse
        font_output = fontset_class(prop, backend)
      File "/home/palmieri/taurus/numpy/sage-4.6.alpha2/local/lib/python/site-packages/matplotlib/mathtext.py", line 658, in __init__
        self._stix_fallback = StixFonts(*args, **kwargs)
      File "/home/palmieri/taurus/numpy/sage-4.6.alpha2/local/lib/python/site-packages/matplotlib/mathtext.py", line 900, in __init__
        fullpath = findfont(name)
      File "/home/palmieri/taurus/numpy/sage-4.6.alpha2/local/lib/python/site-packages/matplotlib/font_manager.py", line 1306, in findfont
        if not os.path.exists(font):
      File "/home/palmieri/taurus/numpy/sage-4.6.alpha2/local/lib/python2.6/genericpath.py", line 18, in exists
        st = os.stat(path)
    TypeError: coercing to Unicode: need string or buffer, dict found

comment:168 Changed 11 years ago by jhpalmieri

Another report: on the skynet machine fulvia (Solaris on x86), matplotlib fails to find numpy, so it doesn't install:

============================================================================
BUILDING MATPLOTLIB
            matplotlib: 0.99.3
                python: 2.6.4 (r264:75706, Oct  4 2010, 18:22:25)  [GCC
                        4.5.1]
              platform: sunos5

REQUIRED DEPENDENCIES
                 numpy: no
                        * You must install numpy 1.1 or later to build
                        * matplotlib.
Error building matplotlib package.

real    0m1.344s
user    0m0.062s
sys     0m0.093s
sage: An error occurred while installing matplotlib-0.99.3

The new numpy package has already been installed at this point in the build, so I don't know what's going on here. On the same machine, scipy fails to install:

gcc version 4.5.1 (GCC)
****************************************************
./spkg-install: LDFLAGS=-shared: is not an identifier

Perhaps you need something like LDFLAGS="-shared"? I'm not really sure.

comment:169 follow-up: Changed 11 years ago by jason

Can you try the new matplotlib spkg on #9221, if you think it is a matplotlib problem?

comment:170 in reply to: ↑ 169 Changed 11 years ago by jhpalmieri

Replying to jason:

Can you try the new matplotlib spkg on #9221, if you think it is a matplotlib problem?

For fulvia, it doesn't help: I still get the same error about numpy not being installed. If I run ./sage -python and then import numpy, I get an error, so although the numpy spkg claims to install correctly, there is some kind of problem:

ImportError: ld.so.1: python: fatal: relocation error: file /home/palmieri/fulvia/numpy/sage-4.6.alpha2/local/lib/python2.6/site-packages/numpy/core/multiarray.so: symbol isfinite: referenced symbol not found

For taurus, using the new matplotlib spkg (and applying the patch from #9221) does seem to fix most of the problems:

----------------------------------------------------------------------

The following tests failed:

        sage -t  -long devel/sage/sage/misc/citation.pyx # 2 doctests failed
----------------------------------------------------------------------

comment:171 follow-up: Changed 11 years ago by fbissey

Dear me!

And I was going to report numpy-1.5/scipy-0.8 build on ppc (should be running testall now, I'd do -long but on that hardware we would still be there next week).

OK there are several issues to be solved: first "-shared" is a good idea. That could be it.

Failing to build matplotlib with an installed numpy :) where did I see something like that... Why, I reported a bug on that on gentoo bugzilla a while ago: http://bugs.gentoo.org/show_bug.cgi?id=320669.

Long story short: that means your lapack (and possibly ATLAS) install is hosed. Probably related to problems producing .so libraries.

I don't get any of the failures you report. A quick package check on Gentoo shows that: matplotlib and rpy depend on numpy and it would be a good idea to rebuild them.

I am a bit worried about the unicode mention in plot.py, that may mean a mismatch between various packages configuration regarding unicode (python/numpy/matplotlib).

comment:172 in reply to: ↑ 171 ; follow-ups: Changed 11 years ago by jhpalmieri

Replying to fbissey:

Dear me!

And I was going to report numpy-1.5/scipy-0.8 build on ppc (should be running testall now, I'd do -long but on that hardware we would still be there next week).

OK there are several issues to be solved: first "-shared" is a good idea. That could be it.

Oops, it looks like it's already quoted in the spkg-install file. I changed the first line from

#!/bin/sh

to

#!/usr/bin/env bash 

I don't know if this was a good idea, but it did get past that error, only to run into the same import error as when trying to install matplotlib.

Failing to build matplotlib with an installed numpy :) where did I see something like that... Why, I reported a bug on that on gentoo bugzilla a while ago: http://bugs.gentoo.org/show_bug.cgi?id=320669.

Long story short: that means your lapack (and possibly ATLAS) install is hosed. Probably related to problems producing .so libraries.

Hmm. We've had problems with shared libraries with ATLAS before on Solaris machines, among others, but it's interesting that this particular problem wouldn't have shown up before this.

I don't get any of the failures you report. A quick package check on Gentoo shows that: matplotlib and rpy depend on numpy and it would be a good idea to rebuild them.

Since I'm building from scratch, all of the dependencies should work themselves out.

I am a bit worried about the unicode mention in plot.py, that may mean a mismatch between various packages configuration regarding unicode (python/numpy/matplotlib).

As Jason suggested, the new matplotlib spkg seems to fix this, at least on taurus. Since that package has been merged as of 4.6.alpha3, it should be taken care of.

comment:173 in reply to: ↑ 172 Changed 11 years ago by fbissey

Replying to jhpalmieri:

Replying to fbissey:

Dear me!

And I was going to report numpy-1.5/scipy-0.8 build on ppc (should be running testall now, I'd do -long but on that hardware we would still be there next week).

OK there are several issues to be solved: first "-shared" is a good idea. That could be it.

Oops, it looks like it's already quoted in the spkg-install file. I changed the first line from

#!/bin/sh

to

#!/usr/bin/env bash 

I don't know if this was a good idea, but it did get past that error, only to run into the same import error as when trying to install matplotlib.

The old version of the spkg was doing that as well. I don't think we should have dropped it. There may be a more elegant solution but in the meantime that should do.

Failing to build matplotlib with an installed numpy :) where did I see something like that... Why, I reported a bug on that on gentoo bugzilla a while ago: http://bugs.gentoo.org/show_bug.cgi?id=320669.

Long story short: that means your lapack (and possibly ATLAS) install is hosed. Probably related to problems producing .so libraries.

Hmm. We've had problems with shared libraries with ATLAS before on Solaris machines, among others, but it's interesting that this particular problem wouldn't have shown up before this.

I cannot comment on that, I don't know the package history of that machine. But searching for numpy in the closed bugs from gentoo's bugzilla show that I haven't been the only victim of that and the answer is always the same: for a reason or another lapack or blas/cblas is foobar.

I don't get any of the failures you report. A quick package check on Gentoo shows that: matplotlib and rpy depend on numpy and it would be a good idea to rebuild them.

Since I'm building from scratch, all of the dependencies should work themselves out.

I am a bit worried about the unicode mention in plot.py, that may mean a mismatch between various packages configuration regarding unicode (python/numpy/matplotlib).

As Jason suggested, the new matplotlib spkg seems to fix this, at least on taurus. Since that package has been merged as of 4.6.alpha3, it should be taken care of.

Ah! I already have matplotlib-1.0 here that's probably why I never saw any of that.

comment:174 in reply to: ↑ 172 Changed 11 years ago by drkirkby

  • Status changed from needs_review to needs_work

Replying to jhpalmieri:

Replying to fbissey:

Long story short: that means your lapack (and possibly ATLAS) install is hosed. Probably related to problems producing .so libraries.

Hmm. We've had problems with shared libraries with ATLAS before on Solaris machines, among others, but it's interesting that this particular problem wouldn't have shown up before this.

The upstream ATLAS program never builds shared libraries. Mathematica only ships with static libraries for ATLAS. It's not entirely clear to me why we bother building shared libraries for ATLAS, given they have tended to cause problems on several systems.

However, I'm not convinced that's the problem here. The ATLAS package has remained unchanged for a long time.

Dave

comment:175 follow-up: Changed 11 years ago by drkirkby

I don't have time to look at this now, but is this being built as C99? If not, that would explain it, since isfinite was not defined until the C99 standard.

Dave

comment:176 in reply to: ↑ 175 ; follow-up: Changed 11 years ago by fbissey

Replying to drkirkby:

I don't have time to look at this now, but is this being built as C99? If not, that would explain it, since isfinite was not defined until the C99 standard.

Why would fluvia not compile stuff in C99 by default? Compared to the other platforms? As for ATLAS it was a suggestion. I just know that the problem is linked to something broken in lapack and because of linking to libf77blas/libcblas it can come down to a problem with ATLAS. I don't know what exactly is happening with that particular machine.

comment:177 in reply to: ↑ 176 Changed 11 years ago by drkirkby

Replying to fbissey:

Replying to drkirkby:

I don't have time to look at this now, but is this being built as C99? If not, that would explain it, since isfinite was not defined until the C99 standard.

Why would fluvia not compile stuff in C99 by default? Compared to the other platforms? As for ATLAS it was a suggestion. I just know that the problem is linked to something broken in lapack and because of linking to libf77blas/libcblas it can come down to a problem with ATLAS. I don't know what exactly is happening with that particular machine.

gcc does not compile C99 by default.

http://gcc.gnu.org/onlinedocs/gcc-4.5.1/gcc/C-Dialect-Options.html#C-Dialect-Options

says:

std=
    Determine the language standard.

<snip>

`gnu89'
    GNU dialect of ISO C90 (including some C99 features). This is the default for C code. 

But isfinite was not introduced until C99. The Sun headers tend not to define everything under the sun.

I'm not saying that's the issue. But it could be. I've seen a similar issue before.

Dave

comment:178 Changed 11 years ago by jhpalmieri

For what it's worth, I redid the build on fulvia with SAGE_CHECK=yes, and ATLAS passed its self-tests. The matplotlib package still fails to install, with the import error from numpy.

comment:179 follow-up: Changed 11 years ago by drkirkby

Does matplotlib generate any sort of config.log or similar? That might indicate what the problem is.

I know at one point I had issues on 64-bit builds, when Numpy thought the static library was 32-bit.

http://projects.scipy.org/numpy/ticket/1525 http://trac.sagemath.org/sage_trac/ticket/8086

comment:180 in reply to: ↑ 179 Changed 11 years ago by jhpalmieri

By the way, I'm getting the same error on mark2 (Solaris on sparc) as on fulvia (Solaris on x86). Cicero (another linux box) doesn't have this problem with building matplotlib: it's built successfully and is now doctesting.

Replying to drkirkby:

Does matplotlib generate any sort of config.log or similar? That might indicate what the problem is.

I don't see any log file in spkg/build/matplotlib on the machines where this is failing. I think the problem is what I indicated above (comment:170): matplotlib tests whether numpy is installed by running python and trying "import numpy". In this case, that import fails, so matplotlib deduces that numpy is not installed.

Here's the relevant code (from setupext.py):

def check_for_numpy():
    try:
        import numpy
    except ImportError:
	print_status("numpy", "no")
        print_message("You must install numpy 1.1 or later to build matplotlib.")
        return False

comment:181 Changed 11 years ago by drkirkby

Here's a recursive grep for isfinite in /usr/include on my OpenSolaris machine. I've not tested the package on my OpenSolaris machine, but given the results on Solaris 10, I doubt it would build.

drkirkby@hawk:~$ ggrep -R isfinite /usr/include
/usr/include/python2.6/pyconfig.h:/* Define to 1 if you have the declaration of `isfinite', and to 0 if you
/usr/include/python2.6/pymath.h:#define Py_IS_FINITE(X) isfinite(X)
ggrep: warning: /usr/include/gphoto2/gphoto2: recursive directory loop

/usr/include/iso/math_c99.h:#undef	isfinite
/usr/include/iso/math_c99.h:#define	isfinite(x)	__extension__( \
/usr/include/iso/math_c99.h:			{ __typeof(x) __x_r = (x); isfinite(__x_r) && \
/usr/include/iso/math_c99.h:#undef	isfinite
/usr/include/iso/math_c99.h:#define	isfinite(x)	__builtin_isfinite(x)

Note, there's no reference at all in math.h to isfinite. But when the conditions are right, the contents of /usr/include/iso/math_c99.h will get included, and so the macro isfinite will be defined.

IMHO, the error message John got:

ImportError: ld.so.1: python: fatal: relocation error: file /home/palmieri/fulvia/numpy/sage-4.6.alpha2/local/lib/python2.6/site-packages/numpy/core/multiarray.so: symbol isfinite: referenced symbol not found

is related to the fact that for whatever reason, the C99 headers are not being included properly.

Unless this is compiled with -std=c99, I doubt this will work.

This contrasts to a Linux box (sage.math) where it looks to me that isfinite will be defined whenever math.h is included. That is just an error in my opinion, as isfinite was not defined until the C99 standard. So Linux is wrong to define it, but some software is assuming it's defined, and so fails when it is not.

IMHO, the bug is in unlikely to be LAPACK or ATLAS.

comment:182 Changed 11 years ago by drkirkby

The more I look at this, the more I think it's a Numpy bug.

Python is not built with C99 on my machine. The config.log of Python shows:

ac_cv_have_decl_isfinite=no

which is to be expected since its not built C99. So Numpy is at fault here I think, since that's making a reference to isfinite despite Python is not built with C99, so it's not defined, as evidenced by Python's config.log

BTW John, do you want an account on my OpenSolaris machine? It can build Sage in under an hour, so is a bit quicker than fulvia.

Dave

comment:183 Changed 11 years ago by drkirkby

IMHO, we need to see if we can reproduce this with the latest cvs/svn/whatever snapshot of Numpy, and if so, report it as a bug to

http://projects.scipy.org/numpy

If it's been fixed, then perhaps we can add a patch to the stable version, to just correct this bug.

Dave

comment:184 Changed 11 years ago by jhpalmieri

I just downloaded the latest code (as listed here), replaced the "src" directory with it, and installed it. Running "./sage -python" and then "import numpy" results in the same error about isfinite.

comment:185 Changed 11 years ago by drkirkby

I reported this as a Numpy bug:

http://projects.scipy.org/numpy/ticket/1625

comment:186 Changed 11 years ago by drkirkby

  • Report Upstream changed from N/A to Reported upstream. Developers acknowledge bug.

I see this problem on my OpenSolaris machine too, which I expected would be the case.

On my OpenSolaris machine, Python's header file pyconfig.h has in it:

/* Define to 1 if you have the declaration of `isfinite', and to 0 if you
   don't. */
#define HAVE_DECL_ISFINITE 0

but Numpy's config.h has in it:

#define HAVE_DECL_ISFINITE

Someone by the usename of 'pv' has said on the Numpy bug report http://projects.scipy.org/numpy/ticket/1625

However, the code at numpy/core/setup.py:218 is wrong, since it checks for #ifdef and not #if for Python's HAVE_DECL_ISFINITE. 

So it seems this is a bug, and one which should be easy to fix. Note the fix does not need to be Solaris-specific, as the code is basically wrong on any platform - you just happen to get away with it on systems where the header file defines isfinite even though the code is not built C99.

It's gone midnight here, I'm tired and have a lot to do all this week. But if nobody else patches this, I'll be able to do it in the next day or two.

Dave

comment:187 Changed 11 years ago by drkirkby

Actually a bigger fix is being proposed on the numpy list. See there - I'm going to bed!

dave

comment:188 follow-up: Changed 11 years ago by jhpalmieri

Dave, thanks a lot for working on this with the numpy people. I prepared a new spkg and so far, the new numpy builds on fulvia, and after "./sage -python", "import numpy" works. Matplotlib and scipy have now successfully built. It will be a while before the build is complete, but things look good to me.

I'm putting the spkg up at http://sage.math.washington.edu/home/palmieri/misc/numpy-1.5.0.svn-20101005.spkg.

This is from a current svn snapshot based on 1.5.0. I don't know if we have a policy about names for such spkg's, so I just used the date. I also don't know if we have a policy about whether to include the repository information, so I've kept it. This makes the spkg much bigger than the old version...

Should we mark this as "needs review" again? It seems to depend on #9221.

comment:189 Changed 11 years ago by jhpalmieri

Should we mark this as "needs review" again?

Or should we wait until the patch is positively reviewed on the numpy trac server?

comment:190 in reply to: ↑ 188 ; follow-up: Changed 11 years ago by jason

Replying to jhpalmieri:

Dave, thanks a lot for working on this with the numpy people. I prepared a new spkg and so far, the new numpy builds on fulvia, and after "./sage -python", "import numpy" works. Matplotlib and scipy have now successfully built. It will be a while before the build is complete, but things look good to me.

I'm putting the spkg up at http://sage.math.washington.edu/home/palmieri/misc/numpy-1.5.0.svn-20101005.spkg.

This is from a current svn snapshot based on 1.5.0. I don't know if we have a policy about names for such spkg's, so I just used the date. I also don't know if we have a policy about whether to include the repository information, so I've kept it. This makes the spkg much bigger than the old version...

You don't need to keep the svn repository history, but you do need to put the revision number in the spkg name (numpy-1.5.0.svnXXXX).

However, I think it would be better to just include the one patch on top of 1.5.0, rather than pull an unstable release from subversion. See the comments at the top of this trac ticket about using an unstable release (like 1.5.0RC2).

comment:191 in reply to: ↑ 190 Changed 11 years ago by kcrisman

However, I think it would be better to just include the one patch on top of 1.5.0, rather than pull an unstable release from subversion. See the comments at the top of this trac ticket about using an unstable release (like 1.5.0RC2).

I agree on this point, for what it's worth (since I will just be doing ./sage -i and testing).

comment:192 follow-up: Changed 11 years ago by jhpalmieri

Yes, but the patch was based on the svn code, not vanilla 1.5.0.

comment:193 in reply to: ↑ 192 Changed 11 years ago by jason

Replying to jhpalmieri:

Yes, but the patch was based on the svn code, not vanilla 1.5.0.

Are you saying that the patch does not apply to 1.5.0 directly?

comment:194 Changed 11 years ago by jhpalmieri

The patch affects two files. One part of the patch is this:

  • numpy/core/setup.py

    diff --git a/numpy/core/setup.py b/numpy/core/setup.py
    index ad8d5cb..f71ec10 100644
    a b def check_ieee_macros(config): 
    215215    _macros = ["isnan", "isinf", "signbit", "isfinite"]
    216216    if sys.version_info[:2] >= (2, 6):
    217217        for f in _macros:
    218             already_declared = config.check_decl(fname2def("decl_%s" % f),
     218            py_symbol = fname2def("decl_%s" % f)
     219            already_declared = config.check_decl(py_symbol,
    219220                    headers=["Python.h", "math.h"])
    220221            if already_declared:
    221                 pub.append('NPY_%s' % fname2def("decl_%s" % f))
     222                if config.check_macro_true(py_symbol,
     223                        headers=["Python.h", "math.h"]):
     224                    pub.append('NPY_%s' % fname2def("decl_%s" % f))
    222225            else:
    223226                macros.append(f)
    224227    else:

and for example the line "already_declared = config.check_decl(fname2def("decl_%s" % f)," is not present in the 1.5.0 version.

Maybe you can modify it (and maybe easily) to apply to 1.5.0, but the diff, as written, does not.

comment:195 Changed 11 years ago by jhpalmieri

Okay, I have a new version of the spkg, based on vanilla 1.5.0. I haven't tested it because I just lost my internet connection to fulvia. See http://sage.math.washington.edu/home/palmieri/misc/numpy-1.5.0.spkg. I'm also attaching the file displaying the differences between this spkg and the old 1.5.0 spkg posted here, except for two things: the new spkg includes modified copies of numpy/core/setup.py and numpy/distutils/command/config.py (in the patches directory -- the src directory is unmodified), and I didn't see any reason to include them in the posted patch file. They are of course included in the repository, but to keep the patch file small, I edited them out before posting it.

Oh, the internet connection is back again. I'm rebuilding on fulvia from scratch, which will take a while, but I just rebuilt the numpy package on mark2 successfully: "import numpy" works.

comment:196 Changed 11 years ago by maldun

Ah I'm only a few days away and everyones busy. Sometimes I think this ticket is cursed... How high are the bets that this will work one day before 1.5.1 is released? =D

@jhpalmiere, drkirkby, fbissey: Thanks for all the trouble!

If you want I build the package with the new patches and upload it on google code for testing. (Or do you want to upload it?) I will test it out on Ubuntu tonight for sure.

grettings, maldun

comment:197 Changed 11 years ago by maldun

  • Description modified (diff)

Ok I oversaw the link in the post... I changed the old link in the discription to your new one. The new package is doing fine on ubuntu. report for doctests will come soon!

Note: See TracTickets for help on using tickets.