Remove the hardcoding of python version in setup.py and SConstruct to build sage_clib and sage itself

[fbissey]

Currently to build sage (the spkg) four files are involved in major way:

1. SConstuct to build sage-clib
2. setup.py
3. module_list.py
4. sage/misc/cython.py

The last three to build the sage python extensions. setup.py, and SConstruct both have the version of python used hardcoded in them, module_list.py doesn't refer to the python version anywhere. cython.py mention the python version explicitly in doctest but smartly find the version used for building all by itself.

In this ticket I bring this smartness to SConstruct and setup.py. The advantage of doing this is that sage then can be build with either python-2.6 or 2.7 (and 3.x in the future) without patching. Of course it may still require patches to fix doctest and syntax in some parts - but you would be able to build sage in the first place without having to patch the build system. It has been found to be an impediment to testing in #9958.

Depend: #11373

Apply:

• trac_11376-SConstruct.patch
• trac_11376-setuppy.patch
• trac_11376-msic_cythonpy-p2.patch
• trac_11376-fix_cythonpy_doctest.patch

[fbissey]

Remove the python version hardcoding in SConstruct, remove debian cruft as well.

[fbissey]

I also remove the Debian cruft from SConstruct and setup.py at the same time.

[mariah]

patch trac_11376-build-sage_clib.patch failed to apply to sage-4.7.rc4:

sage: hg_sage.apply("/home/mariah/trac_11376-build-sage_clib.patch")
cd "/home/mariah/sage/sage-4.7.rc4-x86_64-Linux-core2-fc/devel/sage" && hg status
cd "/home/mariah/sage/sage-4.7.rc4-x86_64-Linux-core2-fc/devel/sage" && hg status
cd "/home/mariah/sage/sage-4.7.rc4-x86_64-Linux-core2-fc/devel/sage" && hg import   "/home/mariah/trac_11376-build-sage_clib.patch"
applying /home/mariah/trac_11376-build-sage_clib.patch
patching file c_lib/SConstruct
Hunk #3 FAILED at 133
1 out of 3 hunks FAILED -- saving rejects to file c_lib/SConstruct.rej
abort: patch failed to apply
sage:

[fbissey]

Thanks for looking at this Mariah. It doesn't apply because it was based on 4.7.1.alpha1. It was based on 4.7.1.alpha1 because of #11167 I cannot produce a patch that will work equally on 4.7.rc and 4.7.1.alpha. That's unfortunate but such is life. In fact I may have to rebase trac_11376-build_setuppy.patch because of #11373 in 4.7.1.alpha2.

[fbissey]

a version of this patch that works with 4.7.rc4

[fbissey]

Just in case you still want to test it against 4.7.rc4 use this patch trac_11376-build-sage_clib-4.7.patch instead.

[mariah]

Replying to fbissey:

Thanks for looking at this Mariah. It doesn't apply because it was based on 4.7.1.alpha1.

Apologies for the noise. I did not realize that 4.7.1.alpha1 was out. I have not seen any announcement of 4.7.1-alpha1 on sage-release. Is there someplace I should be looking for such announcements?

[fbissey]

Replying to mariah:

Replying to fbissey:

Thanks for looking at this Mariah. It doesn't apply because it was based on 4.7.1.alpha1.

Apologies for the noise. I did not realize that 4.7.1.alpha1 was out. I have not seen any announcement of 4.7.1-alpha1 on sage-release. Is there someplace I should be looking for such announcements?

No apologies required. It isn't officially out but Jeroen has "prototypes" alpha0, alpha1 and alpha2 already on  http://sage.math.washington.edu/home/release/. It is a bit a pain to know that if you do something against 4.7.rc4 it will have to be put back to "needs work" once 4.7 is finally out and Jeroen officially announce the next alpha.

[fbissey]

I will add that it is perfectly acceptable to me if you want to leave this ticket for now and come back to it once a 4.7.1.alpha is released.

[fbissey]

Now that 4.7.1.alpha0 is released this ticket should be in a reviewable state.

[fbissey]

Make setup.py python version smart, remove debian cruft. (made dependent on #11373)

[fbissey]

Decided to avoid problem with #11373 so make it depend on it.

[jhpalmieri]

The python documentation for sys.version says "Do not extract version information out of it, rather, use version_info and the functions provided by the platform module." It looks to me as though you could use

platform.python_version()  # gives '2.6.4'
platform.python_version().rstrip('.', 1)[0]  # gives '2.6'
ver = platform.python_version_tuple()  # gives ('2', '6', '4')
ver[0] + '.' + ver[1]  # gives '2.6'

or

ver = sys.version_info
str(ver[0]) + '.' + str(ver[1])   # gives '2.6'

I'll take a look at the rest of the patches...

[fbissey]

Replying to jhpalmieri:

The python documentation for sys.version says "Do not extract version information out of it, rather, use version_info and the functions provided by the platform module." It looks to me as though you could use

platform.python_version()  # gives '2.6.4'
platform.python_version().rstrip('.', 1)[0]  # gives '2.6'
ver = platform.python_version_tuple()  # gives ('2', '6', '4')
ver[0] + '.' + ver[1]  # gives '2.6'

or

ver = sys.version_info
str(ver[0]) + '.' + str(ver[1])   # gives '2.6'

I'll take a look at the rest of the patches...

OK I will rewrite that then. It was purely a case of stealing code from somewhere else. Looks cool in misc/cython.py that will be useful here. I should bring cython.py in line too.

[jhpalmieri]

By the way, I should have provided a link: see  http://docs.python.org/library/sys.html#sys.version for information about sys.version, and pointers to the other options.

[fbissey]

New version of patches all using the platform module. I also included sage/misc/cython.py to switch it from sys to platform. The cython patch in #9958 will be updated consequently.

[jhpalmieri]

Can there ever be version numbers like '2.10'? Choosing the first three characters of the version string seems riskier than removing everything starting from the second dot.

[fbissey]

Replying to jhpalmieri:

Can there ever be version numbers like '2.10'? Choosing the first three characters of the version string seems riskier than removing everything starting from the second dot.

2.10 in particular no. But your general point is valid. I will address this. Thanks for pointing this out.

[fbissey]

It also looks like I made a big typo in the placement of a % in one of the patch as well. I'll get to it ASAP (about 4hours from now).

[fbissey]

Patch updated to use .rstrip('.', 1)[0] as it was the quickest thing to do. Fixed the syntax errors in the setup.py patch as well.

[fbissey]

updated 20110603 #2

[fbissey]

updated 20110603 take 2

[fbissey]

updated 20110603 take 2

[fbissey]

OK so I should have tested what John said before adopting it, even at half past midnight. He meant rsplit not rstrip.

[mariah]

On skynet/eno (x86_64-Linux-core2-fc), I applied the three patches to sage-4.7.1.alpha2, did 'sage -b', then 'make testlong'. I got the following failure:

eno% ./sage -t  -long -force_lib "devel/sage/sage/misc/cython.py"
sage -t -long -force_lib "devel/sage/sage/misc/cython.py"   
**********************************************************************
File "/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/devel/sage/sage/misc/cython.py", line 146:
    sage: pyx_preparse("")
Expected:
    ('\ninclude "interrupt.pxi"  # ctrl-c interrupt block support\ninclude "stdsage.pxi"  # ctrl-c interrupt block support\n\ninclude "cdefs.pxi"\n',
    ['mpfr',
    'gmp',
    'gmpxx',
    'stdc++',
    'pari',
    'm',
    'curvesntl',
    'g0nntl',
    'jcntl',
    'rankntl',
    'gsl',
    '...blas',
    ...,
    'ntl',
    'csage'],
    ['.../local/include/csage/',
    '.../local/include/',
    '.../local/include/python2.6/',
    '.../devel/sage/sage/ext/',
    '.../devel/sage/',
    '.../devel/sage/sage/gsl/'],
    'c',
    [])
Got:
    ('\ninclude "interrupt.pxi"  # ctrl-c interrupt block support\ninclude "stdsage.pxi"  # ctrl-c interrupt block support\n\ninclude "cdefs.pxi"\n', ['mpfr', 'gmp', 'gmpxx', 'stdc++', 'pari', 'm', 'curvesntl', 'g0nntl', 'jcntl', 'rankntl', 'gsl', 'gslcblas', 'atlas', 'ntl', 'csage'], ['/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/local//include/csage/', '/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/local//include/', '/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/local//include/python2.6/', '/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/local//lib/python2.6/site-packages/numpy/core/include', '/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/devel/sage/sage/ext/', '/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/devel/sage/', '/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/devel/sage/sage/gsl/'], 'c', [])
**********************************************************************
File "/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/devel/sage/sage/misc/cython.py", line 188:
    sage: inc
Expected:
    ['bar',
    '.../local/include/csage/',
    '.../local/include/',
    '.../local/include/python2.6/',
    '.../devel/sage/sage/ext/',
    '.../devel/sage/',
    '.../devel/sage/sage/gsl/']
Got:
    ['bar', '/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/local//include/csage/', '/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/local//include/', '/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/local//include/python2.6/', '/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/local//lib/python2.6/site-packages/numpy/core/include', '/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/devel/sage/sage/ext/', '/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/devel/sage/', '/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/devel/sage/sage/gsl/']
**********************************************************************
1 items had failures:
   2 of  10 in __main__.example_2

[fbissey]

Hi Mariah,

I fixed this test as part of #9958 in  http://trac.sagemath.org/sage_trac/attachment/ticket/9958/trac_9958-fix-misc_cythonpy.patch I thought it was appearing because of python-2.7.

If it shows up for you with python-2.6 it is strange that it didn't before the patch.

In any case if it shows up for you the above patch will have to be split and moved to this ticket. Just a few minutes.

[fbissey]

fix the cython.py doctest for the appearance on numpy

[fbissey]

OK new patch added.

[strogdon]

What about the 

/localinclude/  and  /locallib/

in the returned results? Could this also cause a failure? I'm seeing this also in failure of the cython.py test when the patches of #9958 are applied. And the numpy fix is already in those patches.

[mariah]

On skynet/eno (x86_64-Linux-core2-fc), I applied all four patches to sage-4.7.1.alpha1, did 'sage -b', but I still get the error

sage -t -long -force_lib "devel/sage/sage/misc/cython.py"   
**********************************************************************
File "/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/devel/sage/sage/misc/cython.py", line 146:
    sage: pyx_preparse("")
Expected:
    ('\ninclude "interrupt.pxi"  # ctrl-c interrupt block support\ninclude "stdsage.pxi"  # ctrl-c interrupt block support\n\ninclude "cdefs.pxi"\n',
    ['mpfr',
    'gmp',
    'gmpxx',
    'stdc++',
    'pari',
    'm',
    'curvesntl',
    'g0nntl',
    'jcntl',
    'rankntl',
    'gsl',
    '...blas',
    ...,
    'ntl',
    'csage'],
    ['.../local/include/csage/',
    '.../local/include/',
    '.../local/include/python2.6/',
    '.../local/lib/python2.6/site-packages/numpy/core/include',
    '.../devel/sage/sage/ext/',
    '.../devel/sage/',
    '.../devel/sage/sage/gsl/'],
    'c',
    [])
Got:
    ('\ninclude "interrupt.pxi"  # ctrl-c interrupt block support\ninclude "stdsage.pxi"  # ctrl-c interrupt block support\n\ninclude "cdefs.pxi"\n', ['mpfr', 'gmp', 'gmpxx', 'stdc++', 'pari', 'm', 'curvesntl', 'g0nntl', 'jcntl', 'rankntl', 'gsl', 'gslcblas', 'atlas', 'ntl', 'csage'], ['/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/local//include/csage/', '/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/local//include/', '/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/local//include/python2.6/', '/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/local//lib/python2.6/site-packages/numpy/core/include', '/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/devel/sage/sage/ext/', '/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/devel/sage/', '/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/devel/sage/sage/gsl/'], 'c', [])
**********************************************************************
File "/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/devel/sage/sage/misc/cython.py", line 189:
    sage: inc
Expected:
    ['bar',
    '.../local/include/csage/',
    '.../local/include/',
    '.../local/include/python2.6/',
    '.../local/lib/python2.6/site-packages/numpy/core/include',
    '.../devel/sage/sage/ext/',
    '.../devel/sage/',
    '.../devel/sage/sage/gsl/']
Got:
    ['bar', '/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/local//include/csage/', '/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/local//include/', '/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/local//include/python2.6/', '/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/local//lib/python2.6/site-packages/numpy/core/include', '/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/devel/sage/sage/ext/', '/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/devel/sage/', '/home/mariah/sage/sage-4.7.1.alpha2-x86_64-Linux-core2-fc-review-11376/devel/sage/sage/gsl/']
**********************************************************************
1 items had failures:
   2 of  10 in __main__.example_2
***Test Failed*** 2 failures.
For whitespace errors, see the file /home/mariah/.sage//tmp/.doctest_cython.py
         [4.7 s]
 
----------------------------------------------------------------------
The following tests failed:

[fbissey]

Patch to misc/cython.py with the right number of slashes

[fbissey]

Thanks for pointing it out Steve I had missed it completely. We should have a uniform definition of SAGE_LOCAL, sometimes it has a final "/" and sometimes it doesn't. So I updated the misc_cythonpy patch to -p2. Hope we got everything now.

[mariah]

Applied all patches to sage-4.7.1.alpha2, did 'sage -b', and 'make testlong'. All tests passed. Positive review! (finally!)

[fbissey]

A little word of thanks, in the right bug this time (note to self don't do this again at 1:30am):

Thank you Mariah for the review. Thank you John for the suggestion to make it better. Thank you Steve for pointing the obvious.

Thank you all for making this reach this stage. Hopefully this will be in 4.7.2.alpha0.