Opened 3 years ago
Closed 3 years ago
#24605 closed defect (fixed)
Upgrading from older Python versions doesn't work
Reported by: | jdemeyer | Owned by: | |
---|---|---|---|
Priority: | critical | Milestone: | sage-8.2 |
Component: | packages: optional | Keywords: | |
Cc: | embray | Merged in: | |
Authors: | Jeroen Demeyer | Reviewers: | Erik Bray |
Report Upstream: | N/A | Work issues: | |
Branch: | 379adfd (Commits) | Commit: | 379adfd8c10222a59e9ad598c924cd1060efeb53 |
Dependencies: | Stopgaps: |
Description (last modified by )
This has been reported several time on sage-devel but now I finally managed to reproduce it.
The builds ends with
gcc -pthread -shared -L. -L/home/jdemeyer/sage-git/local/lib -Wl,-rpath,/home/jdemeyer/sage-git/local/lib -L. -L/home/jdemeyer/sage-git/local/lib -Wl,-rpath,/home/jdeme yer/sage-git/local/lib build/temp.linux-ppc64le-2.7/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/_ctypes.o build/temp.linux-ppc 64le-2.7/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/callbacks.o build/temp.linux-ppc64le-2.7/home/jdemeyer/sage-git/local/var /tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/callproc.o build/temp.linux-ppc64le-2.7/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Modu les/_ctypes/stgdict.o build/temp.linux-ppc64le-2.7/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/cfield.o build/temp.linux-ppc64 le-2.7/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/libffi/src/prep_cif.o build/temp.linux-ppc64le-2.7/home/jdemeyer/sage-git/l ocal/var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/libffi/src/closures.o build/temp.linux-ppc64le-2.7/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2 -2.7.14.p2/src/Modules/_ctypes/libffi/src/powerpc/ffi.o build/temp.linux-ppc64le-2.7/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctyp es/libffi/src/powerpc/ffi_sysv.o build/temp.linux-ppc64le-2.7/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/libffi/src/powerpc/f fi_linux64.o build/temp.linux-ppc64le-2.7/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/libffi/src/powerpc/sysv.o build/temp.lin ux-ppc64le-2.7/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/libffi/src/powerpc/ppc_closure.o build/temp.linux-ppc64le-2.7/home/ jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/libffi/src/powerpc/linux64.o build/temp.linux-ppc64le-2.7/home/jdemeyer/sage-git/local/ var/tmp/sage/build/python2-2.7.14.p2/src/Modules/_ctypes/libffi/src/powerpc/linux64_closure.o -L/home/jdemeyer/sage-git/local/lib -L/usr/lib/powerpc64le-linux-gnu -L/us r/local/lib -L. -lpython2.7 -o build/lib.linux-ppc64le-2.7/_ctypes.so *** WARNING: renaming "_ctypes" since importing it failed: build/lib.linux-ppc64le-2.7/_ctypes.so: undefined symbol: _PySlice_Unpack Python build finished, but the necessary bits to build these modules were not found: _bsddb _ssl _tkinter bsddb185 dbm dl gdbm imageop sunaudiodev To find the necessary bits, look in setup.py in detect_modules() for the module's name. Failed to build these modules: _ctypes _elementtree _multiprocessing array mmap running build_scripts creating build/scripts-2.7 copying and adjusting /home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Tools/scripts/pydoc -> build/scripts-2.7 copying and adjusting /home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Tools/scripts/idle -> build/scripts-2.7 copying and adjusting /home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Tools/scripts/2to3 -> build/scripts-2.7 copying and adjusting /home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Lib/smtpd.py -> build/scripts-2.7 changing mode of build/scripts-2.7/pydoc from 664 to 775 changing mode of build/scripts-2.7/idle from 664 to 775 changing mode of build/scripts-2.7/2to3 from 664 to 775 changing mode of build/scripts-2.7/smtpd.py from 664 to 775 make[3]: Leaving directory '/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src' Testing importing of various modules... Traceback (most recent call last): File "<string>", line 1, in <module> File "/home/jdemeyer/sage-git/local/var/tmp/sage/build/python2-2.7.14.p2/src/Lib/ctypes/__init__.py", line 7, in <module> from _ctypes import Union, Structure, Array ImportError: No module named _ctypes ctypes module failed to import
The problem is that the Python build system imports ctypes
at build-time but this will use the wrong version of libpython. The build system tried to work around that with
# Make sure -L. is placed before -L$SAGE_LOCAL/lib so that python and extension # modules are linked with the right libpython; we pass this in at make time # only, since we don't want -L. to be saved as one of the default LDFLAGS # used for building third-party extension modules $MAKE LDFLAGS="-L. $LDFLAGS"
However, this affects building only, not running. And even using LD_LIBRARY_PATH
won't work since an rpath takes priority over that.
Change History (17)
comment:1 Changed 3 years ago by
- Description modified (diff)
comment:2 Changed 3 years ago by
- Description modified (diff)
comment:3 Changed 3 years ago by
- Branch set to u/jdemeyer/upgrading_from_older_python_versions_doesn_t_work
comment:4 Changed 3 years ago by
- Commit set to 4a64f3e3033d18b554a37627cbf6cecae0d82009
- Status changed from new to needs_review
comment:5 Changed 3 years ago by
I don't like the idea of removing the existing libs before building. Let me think about this a bit.
comment:6 Changed 3 years ago by
OK, I'll let you think a bit. But if you cannot come up with a better solution, please accept this one which works.
comment:7 Changed 3 years ago by
It is annoying that, in this sense, Python's build process is basically incompatible with setting a custom rpath. I wonder if this has been discussed anywhere else.
comment:8 Changed 3 years ago by
The unusual thing about Python is that it actually runs itself as part of the build process...
comment:9 Changed 3 years ago by
Just wanted to point out that we should fix this before the next Python upgrade. But as there isn't even a plan for 2.7.15 at the moment, that doesn't look urgent.
comment:10 Changed 3 years ago by
Python 2.7.15 is planned for the end of this month.
comment:11 Changed 3 years ago by
- Reviewers set to Erik Bray
- Status changed from needs_review to positive_review
I'm okay with it as a temporary solution. I think a better solution is needed in the long term, but that could come more or less automatically with #22510 (which would simplify completely uninstalling a package before upgrading it). I'm working on updating that work again so that we can start using it soon, hopefully.
comment:12 Changed 3 years ago by
comment:13 Changed 3 years ago by
I feel like I'm missing something: the ticket description deals with Python 2 but the branch only deals with Python 3. What's going on?
comment:14 Changed 3 years ago by
There is a symlink from the Python 2 build scripts to the Python 3 build scripts.
comment:15 Changed 3 years ago by
- Commit changed from 4a64f3e3033d18b554a37627cbf6cecae0d82009 to 379adfd8c10222a59e9ad598c924cd1060efeb53
- Status changed from positive_review to needs_review
Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. This was a forced push. New commits:
379adfd | Remove old Python libraries *before* build
|
comment:16 Changed 3 years ago by
- Status changed from needs_review to positive_review
Rebased to 8.2.rc3
comment:17 Changed 3 years ago by
- Branch changed from u/jdemeyer/upgrading_from_older_python_versions_doesn_t_work to 379adfd8c10222a59e9ad598c924cd1060efeb53
- Resolution set to fixed
- Status changed from positive_review to closed
New commits:
Remove old Python libraries *before* build