Opened 3 years ago

Closed 3 years ago

#24605 closed defect (fixed)

Upgrading from older Python versions doesn't work

Reported by: jdemeyer Owned by:
Priority: critical Milestone: sage-8.2
Component: packages: optional Keywords:
Cc: embray Merged in:
Authors: Jeroen Demeyer Reviewers: Erik Bray
Report Upstream: N/A Work issues:
Branch: 379adfd (Commits) Commit: 379adfd8c10222a59e9ad598c924cd1060efeb53
Dependencies: Stopgaps:

Description (last modified by jdemeyer)

This has been reported several time on sage-devel but now I finally managed to reproduce it.

The builds ends with

gcc -pthread -shared -L. -L/home/jdemeyer/sage-git/local/lib -Wl,-rpath,/home/jdemeyer/sage-git/local/lib -L. -L/home/jdemeyer/sage-git/local/lib -Wl,-rpath,/home/jdeme
yer/sage-git/local/lib build/temp.linux-ppc64le-2.7/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/_ctypes.o build/temp.linux-ppc
64le-2.7/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/callbacks.o build/temp.linux-ppc64le-2.7/home/jdemeyer/sage-git/local/var
/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/callproc.o build/temp.linux-ppc64le-2.7/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Modu
les/_ctypes/stgdict.o build/temp.linux-ppc64le-2.7/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/cfield.o build/temp.linux-ppc64
le-2.7/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/libffi/src/prep_cif.o build/temp.linux-ppc64le-2.7/home/jdemeyer/sage-git/l
ocal/var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/libffi/src/closures.o build/temp.linux-ppc64le-2.7/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2
-2.7.14.p2/src/Modules/_ctypes/libffi/src/powerpc/ffi.o build/temp.linux-ppc64le-2.7/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctyp
es/libffi/src/powerpc/ffi_sysv.o build/temp.linux-ppc64le-2.7/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/libffi/src/powerpc/f
fi_linux64.o build/temp.linux-ppc64le-2.7/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/libffi/src/powerpc/sysv.o build/temp.lin
ux-ppc64le-2.7/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/libffi/src/powerpc/ppc_closure.o build/temp.linux-ppc64le-2.7/home/
jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/libffi/src/powerpc/linux64.o build/temp.linux-ppc64le-2.7/home/jdemeyer/sage-git/local/
var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/libffi/src/powerpc/linux64_closure.o -L/home/jdemeyer/sage-git/local/lib -L/usr/lib/powerpc64le-linux-gnu -L/us
r/local/lib -L. -lpython2.7 -o build/lib.linux-ppc64le-2.7/
*** WARNING: renaming "_ctypes" since importing it failed: build/lib.linux-ppc64le-2.7/ undefined symbol: _PySlice_Unpack

Python build finished, but the necessary bits to build these modules were not found:
_bsddb             _ssl               _tkinter        
bsddb185           dbm                dl              
gdbm               imageop            sunaudiodev     
To find the necessary bits, look in in detect_modules() for the module's name.

Failed to build these modules:
_ctypes            _elementtree       _multiprocessing
array              mmap                               

running build_scripts
creating build/scripts-2.7
copying and adjusting /home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Tools/scripts/pydoc -> build/scripts-2.7
copying and adjusting /home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Tools/scripts/idle -> build/scripts-2.7
copying and adjusting /home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Tools/scripts/2to3 -> build/scripts-2.7
copying and adjusting /home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Lib/ -> build/scripts-2.7
changing mode of build/scripts-2.7/pydoc from 664 to 775
changing mode of build/scripts-2.7/idle from 664 to 775
changing mode of build/scripts-2.7/2to3 from 664 to 775
changing mode of build/scripts-2.7/ from 664 to 775
make[3]: Leaving directory '/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src'
Testing importing of various modules...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Lib/ctypes/", line 7, in <module>
    from _ctypes import Union, Structure, Array
ImportError: No module named _ctypes
ctypes module failed to import

The problem is that the Python build system imports ctypes at build-time but this will use the wrong version of libpython. The build system tried to work around that with

# Make sure -L. is placed before -L$SAGE_LOCAL/lib so that python and extension
# modules are linked with the right libpython; we pass this in at make time
# only, since we don't want -L. to be saved as one of the default LDFLAGS
# used for building third-party extension modules

However, this affects building only, not running. And even using LD_LIBRARY_PATH won't work since an rpath takes priority over that.

Change History (17)

comment:1 Changed 3 years ago by jdemeyer

  • Description modified (diff)

comment:2 Changed 3 years ago by jdemeyer

  • Description modified (diff)

comment:3 Changed 3 years ago by jdemeyer

  • Branch set to u/jdemeyer/upgrading_from_older_python_versions_doesn_t_work

comment:4 Changed 3 years ago by jdemeyer

  • Commit set to 4a64f3e3033d18b554a37627cbf6cecae0d82009
  • Status changed from new to needs_review

New commits:

4a64f3eRemove old Python libraries *before* build

comment:5 Changed 3 years ago by embray

I don't like the idea of removing the existing libs before building. Let me think about this a bit.

comment:6 Changed 3 years ago by jdemeyer

OK, I'll let you think a bit. But if you cannot come up with a better solution, please accept this one which works.

comment:7 Changed 3 years ago by embray

It is annoying that, in this sense, Python's build process is basically incompatible with setting a custom rpath. I wonder if this has been discussed anywhere else.

comment:8 Changed 3 years ago by jdemeyer

The unusual thing about Python is that it actually runs itself as part of the build process...

comment:9 Changed 3 years ago by jdemeyer

Just wanted to point out that we should fix this before the next Python upgrade. But as there isn't even a plan for 2.7.15 at the moment, that doesn't look urgent.

comment:10 Changed 3 years ago by jdemeyer

Python 2.7.15 is planned for the end of this month.

comment:11 Changed 3 years ago by embray

  • Reviewers set to Erik Bray
  • Status changed from needs_review to positive_review

I'm okay with it as a temporary solution. I think a better solution is needed in the long term, but that could come more or less automatically with #22510 (which would simplify completely uninstalling a package before upgrading it). I'm working on updating that work again so that we can start using it soon, hopefully.

comment:12 Changed 3 years ago by chapoton

  • Authors set to Jeroen Demeyer

comment:13 Changed 3 years ago by jhpalmieri

I feel like I'm missing something: the ticket description deals with Python 2 but the branch only deals with Python 3. What's going on?

comment:14 Changed 3 years ago by jdemeyer

There is a symlink from the Python 2 build scripts to the Python 3 build scripts.

comment:15 Changed 3 years ago by git

  • Commit changed from 4a64f3e3033d18b554a37627cbf6cecae0d82009 to 379adfd8c10222a59e9ad598c924cd1060efeb53
  • Status changed from positive_review to needs_review

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. This was a forced push. New commits:

379adfdRemove old Python libraries *before* build

comment:16 Changed 3 years ago by jdemeyer

  • Status changed from needs_review to positive_review

Rebased to 8.2.rc3

comment:17 Changed 3 years ago by vbraun

  • Branch changed from u/jdemeyer/upgrading_from_older_python_versions_doesn_t_work to 379adfd8c10222a59e9ad598c924cd1060efeb53
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.