#9210 closed defect (fixed)
pkg-config's prefix statements in SAGE_LOCAL/lib/pkgconfig not changed upon Sage move
Reported by: | jason | Owned by: | drkirkby |
---|---|---|---|
Priority: | blocker | Milestone: | sage-4.6 |
Component: | build | Keywords: | |
Cc: | drkirkby, kcrisman | Merged in: | sage-4.6.rc0 |
Authors: | Jason Grout | Reviewers: | David Kirkby, Karl-Dieter Crisman |
Report Upstream: | N/A | Work issues: | |
Branch: | Commit: | ||
Dependencies: | Stopgaps: |
Description
The attached patch fixes the above issue which causes matplotlib to pick up the system freetype2 after moving Sage (even after #9208). One thing led to another and I ended up restructuring and rewriting lots of the code in the file too. I hope it's cleaner and has less portability issues (I changed a lot of things to use os.path.join, for example).
Attachments (4)
Change History (64)
comment:1 Changed 6 years ago by jason
- Status changed from new to needs_review
comment:2 Changed 6 years ago by jason
#5776 is related (it's a general ticket about the stuff that old paths that still hang around in various places).
comment:3 Changed 6 years ago by jason
- Status changed from needs_review to needs_work
In the R .pc file, there are still some other directories that are not changed (besides just the prefix directory).
comment:4 Changed 6 years ago by jason
- Status changed from needs_work to needs_review
Updated script to initially set a SAGE_ROOT variable inside the file, and then just update that variable. This should take care of all paths in the file that need to be changed.
comment:5 Changed 6 years ago by drkirkby
I've started to build the binary on my old Blade 1000. It's not the fastest machine in the world, (not even the fastest SPARC I own, but its convenient at the minute). It will take a few hours to build this, unpack it, then check for hard-coded paths. But I'm working on it. If I can stay awake, I might have something by 0200 GMT on Saturday.
Dave
comment:6 Changed 6 years ago by drkirkby
It was a lot quicker than I expected.
Something has gone wrong here though. The installation worked fine before it was moved. First I create the binary.
drkirkby@redstart:~/32/sage-4.4.3$ ./sage -bdist 4.4.3-ax Sage works! Copying files over to tmp directory local/lib/python2.6/python2.6: failed to get acl entries local/lib/libgfortran.so: failed to get acl entries cp: cannot access *.sage Copying Sage library over Making empty spkg's
I think we need to copy libgfortran (see #8049) though the failure to do so should not cause me a problem, as the system has it. But when I tried to move the distribution, I get:
-bash-3.00$ ./sage ---------------------------------------------------------------------- | Sage Version 4.4.3, Release Date: 2010-06-04 | | Type notebook() for the GUI, and license() for information. | ---------------------------------------------------------------------- The Sage install tree may have moved (from /export/home/drkirkby/32/sage-4.4.3 to /export/home/sageserv/sage-4.4.3-after-rewrite-sage-location-patch-Solaris10_03_2005-sun4u-SunOS) Changing various hardcoded paths (please wait at most a few minutes)... Do not interrupt this. Done resetting paths ------------------------------------------------------------ Unhandled SIGSEGV: A segmentation fault occured in Sage. This probably occured because a *compiled* component of Sage has a bug in it (typically accessing invalid memory) or is not properly wrapped with _sig_on, _sig_off. You might want to run Sage under gdb with 'sage -gdb' to debug this. Sage will now terminate (sorry). ------------------------------------------------------------
I don't have gdb on this machine. I'll investigate, but for now, there may be a problem here.
Dave
comment:7 Changed 6 years ago by drkirkby
- Owner changed from GeorgSWeber to drkirkby
On closer inspection, gdb was on the machine, so I run sage with the option -gdb.
The SIGSEGV seems to be caused by Singular failing to find a file 'tesths.cc' though that file does exist.
-bash-3.00$ export PATH=$PATH:/usr/local/bin -bash-3.00$ ./sage -gdb ---------------------------------------------------------------------- | Sage Version 4.4.3, Release Date: 2010-06-04 | | Type notebook() for the GUI, and license() for information. | ---------------------------------------------------------------------- /export/home/sageserv/sage-4.4.3-after-rewrite-sage-location-patch-Solaris10_03_2005-sun4u-SunOS/local/bin/sage-ipython GNU gdb (GDB) 7.0.1 Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "sparc-sun-solaris2.10". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /export/home/sageserv/sage-4.4.3-after-rewrite-sage-location-patch-Solaris10_03_2005-sun4u-SunOS/local/bin/python...done. [Thread debugging using libthread_db enabled] [New Thread 1 (LWP 1)] Python 2.6.4 (r264:75706, Jun 6 2010, 00:37:24) [GCC 4.4.3] on sunos5 Type "help", "copyright", "credits" or "license" for more information. Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 1 (LWP 1)] siInit (name=0x1e169b4 <error reading variable>) at tesths.cc:65 65 tesths.cc: No such file or directory. in tesths.cc Current language: auto The current source language is "auto; currently c++". (gdb) quit A debugging session is active. Inferior 1 [process 6182 ] will be killed. Quit anyway? (y or n) y -bash-3.00$ find . -name tesths.cc ./spkg/standard/singular-3.1.0.4.p6/src/Singular/tesths.cc -bash-3.00$ cat ./spkg/standard/singular-3.1.0.4.p6/src/Singular/tesths.cc /**************************************** * Computer Algebra System SINGULAR * ****************************************/ /* $Id: tesths.cc,v 1.115 2008/09/10 09:33:58 Singular Exp $ */
Do you think this can be related? It was fine before the distribution was moved.
Dave
comment:8 Changed 6 years ago by jason
It doesn't seem like it should be related (i.e., my guess is that you'd have this problem without the patch as well). Can you zip up the SAGE_LOCAL/lib/pkgconfig/ directory and the SAGE_LOCAL/lib/sage-current-location.txt file so I can just double-check that the paths are right?
comment:9 Changed 6 years ago by drkirkby
Sure. I've attached two .tar.gz files. It should be obvious which is which from the name. Note I created a second user account to test the moved version on, so you might see different user names on the two tarballs.
Changed 6 years ago by drkirkby
After creating a binary distribution and moving it to somewhere else.
comment:10 follow-up: ↓ 11 Changed 6 years ago by jason
Enough stuff looked fishy in your output that I have to ask: Are you using the most recent version of the patch? From the pkg-config files, it doesn't look like it.
What commands are you running? The following two sequences of commands should work to update the .pc files:
- Build sage from source in directory A
- Run Sage (this may be optional---it depends on if the build creates a correct local/lib/sage-current-location.txt file)
- Move the sage build directory to directory B
- Run Sage in the new directory (this updates the sage-current-location.txt file, and also updates the .pc files, etc.)
Then the file paths should be updated.
Alternatively, this should work:
- Build sage from source in directory A
- Make a bdist (this automatically runs Sage, so it takes care of step 2 above)
- Extract the bdist
- Run Sage in the new directory (this updates the sage-current-location.txt file, and also updates the .pc files, etc.)
In either situation, at the top of each local/lib/pkgconfig/*.pc file, there should be a SAGE_ROOT=new directory after step 3, and the local/lib/sage-current-location.txt file should also contain the new directory name. Neither of those are true for your attached outputs, so I think either you are using the old version of the patch or didn't follow one of the steps. Or my code has a bug :).
comment:11 in reply to: ↑ 10 Changed 6 years ago by jason
Replying to jason:
In either situation, at the top of each local/lib/pkgconfig/*.pc file, there should be a SAGE_ROOT=new directory after step 3
I meant after step 4.
comment:12 Changed 6 years ago by drkirkby
I'll have to look at this later today - it is 3:19 am here! It's possible I don't have the latest patch, or I've somehow mess up installing it. Could you attach the full file (not just the patch) of whatever you think I might have wrong. I'll then copy that in directly.
Dave
comment:13 follow-up: ↓ 15 Changed 6 years ago by drkirkby
I thought I might have messed up, as I was tired, but as far as I can see, I do get a problem.
The method I used was:
- Build sage from source in directory /export/home/drkirkby/32/sage-4.4.3
- Make a binary distribution using your updated sage-bdist, which I've confirmed works on other systems and mine.
- Log in as a different user 'sageserv' on my system.
- Extract the binary .tar.gz file, which was extracted to
/export/home/sageserv/sage-4.4.3-after-rewrite-sage-location-patch-Solaris10_03_2005-sun4u-SunOS
- Run Sage in the new directory, where upon I get the seg fault. But in the other directory, as a different user name, it works fine.
As far as I can see, I have the latest patches. These are the sizes of the patched files.
-rwxr-xr-x 1 drkirkby other 3680 Jun 9 23:54 sage-bdist -rwxr-xr-x 1 drkirkby other 10216 Jun 10 20:00 sage-env -rwxr-xr-x 1 drkirkby other 7133 Jun 12 15:40 sage-location
The 'sage-location' script starts
#!/usr/bin/env python import os, sys OLD_SAGE_ROOT = None SAGE_ROOT = os.path.realpath(os.environ['SAGE_ROOT']) location_file = os.path.join(SAGE_ROOT, 'local', 'lib', 'sage-curent-location.txt') flags_file = os.path.join(SAGE_ROOT, 'local', 'lib', 'sage-flags.txt') # The flags we care about recording in the local/lib/sage-flags.txt file # In SAGE_FAT_BINARY mode we only require that ['sse', 'sse2', '3d', # 'mmx', 'cmov'] be available, and in particular, don't require pni # or ssse3. try: SAGE_FAT_BINARY = os.environ['SAGE_FAT_BINARY'] except: SAGE_FAT_BINARY = ""
Anything look wrong?
Dave
Changed 6 years ago by jason
new sage-location file (after patch) - do not apply; only for reviewing convenience
comment:14 Changed 6 years ago by jason
I've attached the full (new) sage-location script, for reviewing convenience.
comment:15 in reply to: ↑ 13 Changed 6 years ago by jason
Replying to drkirkby:
The 'sage-location' script starts
#!/usr/bin/env python import os, sys OLD_SAGE_ROOT = None SAGE_ROOT = os.path.realpath(os.environ['SAGE_ROOT']) location_file = os.path.join(SAGE_ROOT, 'local', 'lib', 'sage-curent-location.txt') flags_file = os.path.join(SAGE_ROOT, 'local', 'lib', 'sage-flags.txt') # The flags we care about recording in the local/lib/sage-flags.txt file # In SAGE_FAT_BINARY mode we only require that ['sse', 'sse2', '3d', # 'mmx', 'cmov'] be available, and in particular, don't require pni # or ssse3. try: SAGE_FAT_BINARY = os.environ['SAGE_FAT_BINARY'] except: SAGE_FAT_BINARY = ""Anything look wrong?
Yep. Notice that the sage-location file is 'sage-curent-location' (misspelled "current"). That's the old patch. I've attached the full new file for convenience, or you can pull the patch attached to this ticket currently.
Jason
comment:16 Changed 6 years ago by drkirkby
I'm just building another binary distribution.
I should be able to update this within the next hour or two.
comment:17 Changed 6 years ago by drkirkby
- Status changed from needs_review to needs_work
Hi Jason,
I made another binary distribution, and moved it again. (I mistakenly used the number 4.4.4 in the name, rather than 4.4.3, but it makes no difference. I could call it zebra if I wanted! But this is based on the 4.4.3 in Sage, not on the 4.4.4.alpha0 source code).
Sage reports it has been moved, and an attempt to factorise a Mersenne prime behaves as expected. So there is no segmentation fault as there was before.
-bash-3.00$ cd sage-4.4.4-revised-sage-location-without-misspelling-sun4u-SunOS -bash-3.00$ ./sage ---------------------------------------------------------------------- | Sage Version 4.4.3, Release Date: 2010-06-04 | | Type notebook() for the GUI, and license() for information. | ---------------------------------------------------------------------- The Sage install tree may have moved (from /export/home/drkirkby/32/sage-4.4.3 to /export/home/sageserv/sage-4.4.4-revised-sage-location-without-misspelling-sun4u-SunOS) Changing various hardcoded paths (please wait at most a few minutes)... Do not interrupt this. Done resetting paths sage: factor(2^607 -1) 531137992816767098689588206552468627329593117727031923199444138200403559860852242739162502265229285668889329486246501015346579337652707239409519978766587351943831270835393219031728127 sage:
But when I look at the .pc files, the prefix is not updated. This does not look right to me.
-bash-3.00$ pwd /export/home/sageserv/sage-4.4.4-revised-sage-location-without-misspelling-sun4u-SunOS/local/lib/pkgconfig -bash-3.00$ cat freetype2.pc prefix=/export/home/drkirkby/32/sage-4.4.3/local exec_prefix=${prefix} libdir=${exec_prefix}/lib includedir=${prefix}/include Name: FreeType 2 Description: A free, high-quality, and portable font engine. Version: 9.16.3 Requires: Libs: -L${libdir} -lfreetype -lz Cflags: -I${includedir}/freetype2 -I${includedir}SAGE_ROOT=/export/home/sageserv/sage-4.4.4-revised-sage-location-without-misspelling-sun4u-SunOS -bash-3.00$
The file sage-location is the one attached to this ticket
-bash-3.00$ digest -a md5 ./local/bin/sage-location d73f790e23a60751f6b1b58941515744
comment:18 Changed 6 years ago by jason
You're right, that .pc file does not look right. I'll try to take a look at this early next week.
comment:19 Changed 6 years ago by jason
I think we need to be more careful about how we review this patch. I know what files to delete to re-initialize things, but just to make sure, here are the new review instructions. I'm following these myself, and I'll post them in a bit.
comment:20 Changed 6 years ago by jason
- Status changed from needs_work to needs_review
Okay, here are I think better instructions for reviewing this patch:
- Download and extract the Sage 4.4.3 source tarball
- Download http://sage.math.washington.edu/home/jason/sage_scripts-4.4.3.spkg and put it in the spkg/standard directory, replacing the existing file of the same name. This is just an spkg with the patch on this ticket applied, so that the .pc files are initialized correctly
- Build the source (i.e., type "make")
- Run Sage (this is very important, as hardcoded paths will depend on the build directory, and this step will initialize things to know what the build directory is)
- Then either move the sage directory, or create a bdist. Now running Sage after moving directories or extracting and running sage from the bdist should correctly update the .pc files. The freetype2.pc file, for example, should be something like:
SAGE_ROOT=/home/grout/sage-4.4.3-pkgconfig prefix=${SAGE_ROOT}/local exec_prefix=${prefix} libdir=${exec_prefix}/lib includedir=${prefix}/include Name: FreeType 2 Description: A free, high-quality, and portable font engine. Version: 9.16.3 Requires: Libs: -L${libdir} -lfreetype -lz Cflags: -I${includedir}/freetype2 -I${includedir}
where the SAGE_ROOT variable should update to the current sage root directory.
comment:21 Changed 6 years ago by jason
FYI, in April, some discussion happened on the pkg-config mailing list about making some automatic variables to support relocation:
http://lists.freedesktop.org/archives/pkg-config/2010-April/000520.html
comment:22 Changed 6 years ago by jason
(nothing came out of that discussion yet, but it might be nice to keep an eye on it in the future...)
comment:23 Changed 6 years ago by drkirkby
Since 4.4.4 has now been released, should I be able to apply trac-9210-rewrite-sage-location.patch and get this to work? I've built 4.4.4 but have not moved it yet.
Dave
comment:24 follow-up: ↓ 57 Changed 5 years ago by mpatel
- Cc kcrisman added
Not updating the .pc files in SAGE_LOCAL/lib/pkgconfig may also cause problems during an upgrade after moving Sage. Please see sage-release for Karl-Dieter's report about a 4.5.2.rc1-4.5.3.alpha1 upgrade on PPC OS X 10.4
Also, for some packages that compile Python extension modules during the upgrade, it seems to help to update old paths in SAGE_LOCAL/lib/python/config/Makefile.
I checked the .pc file and Makefile changes on sage.math with the equivalent of
$ tar xf /home/release/sage-4.5.2/sage-4.5.2.tar $ mv sage-4.5.2 sage-FOOO $ cd sage-FOOO $ make build $ cd .. $ cp -a sage-FOOO/ sage-yyyy $ cd sage-yyyy $ sed -i 's/FOOO/yyyy/g' local/lib/pkgconfig/* $ sed -i 's/FOOO/yyyy/g' local/lib/python/config/Makefile $ echo "y" | ./sage -upgrade http://sage.math.washington.edu/home/release/sage-4.5.3.alpha1/sage-4.5.3.alpha1.tar | tee -a upgrade.log $ grep FOOO upgrade.log $
If I don't update the .pc files and/or the Makefile, I get many matches for FOOO.
comment:25 Changed 5 years ago by jason
To review, this should be sufficient:
- Download and build Sage (important; it must be a fresh build from source).
- Apply the patch at the ticket to the scripts repository (in SAGE_ROOT/local/bin):
http://trac.sagemath.org/sage_trac/raw-attachment/ticket/9210/trac-9210-rewrite-sage-location.patch
- Delete the file SAGE_ROOT/local/lib/sage-current-location.txt, if it exists
- Now run Sage, then exit (this updates the sage-current-location and pkgconfig files appropriately).
- Move the sage directory and run Sage again. The SAGE_ROOT/local/lib/sage-current-location.txt should be updated, and the pkgconfig files in SAGE_ROOT/local/lib/pkgconfig/ should also be updated to the new path (you can check the freetype2.pc file, for example, to see the new sage directory is listed there in the SAGE_ROOT variable).
comment:26 follow-up: ↓ 30 Changed 5 years ago by drkirkby
- Status changed from needs_review to needs_work
This is simply not working for me. The system is a Sun Ultra 27 running OpenSolaris on an Intel Xeon processor. This is what I did.
1) Build Sage fresh in the directory
/export/home/drkirkby/9/sage-4.5.3.alpha2
Sage had not been run. There was no file SAGE_ROOT/local/lib/sage-current-location.txt
2) Created a tar file with all this in:
$ tar cvf test.tar sage-4.5.3.alpha2
3) Changed to the directory /tmp.
$ cd /tmp
4) Extracted the tar file in /tmp
$ tar xf $HOME/9/test.tar
5) Run Sage from /tmp.
That created
$SAGE_LOCAL/lib/sage-current-location.txt
which had /tmp/sage-4.5.3.alpha2
as the contents - I note there is no end of line.
6) Changed to the directory where all the .pc files are
$ cd /tmp/sage-4.5.3.alpha2/local/lib/pkgconfig
7) Now run grep, and look for my user name "drkirkby" which was in the path before, but now I'm in /tmp, it should not be there.
drkirkby@hawk:/tmp/sage-4.5.3.alpha2/local/lib/pkgconfig$ grep drkirkby * bdw-gc.pc:prefix=/export/home/drkirkby/9/sage-4.5.3.alpha2/local freetype2.pc:prefix=/export/home/drkirkby/9/sage-4.5.3.alpha2/local gnutls-extra.pc:prefix=/export/home/drkirkby/9/sage-4.5.3.alpha2/local gnutls-extra.pc:Libs.private: -L${exec_prefix}/lib -lgnutls-extra -L/export/home/drkirkby/9/sage-4.5.3.alpha2/local/lib -lopencdk -L/export/home/drkirkby/9/sage-4.5.3.alpha2/local/lib -lgcrypt -lsocket -L/export/home/drkirkby/9/sage-4.5.3.alpha2/local/lib -lgpg-error -L/export/home/drkirkby/9/sage-4.5.3.alpha2/local/lib -lz -R/export/home/drkirkby/9/sage-4.5.3.alpha2/local/lib -L${exec_prefix}/lib -lgnutls -L/export/home/drkirkby/9/sage-4.5.3.alpha2/local/lib -lgcrypt -lgpg-error -lnsl -lsocket gnutls.pc:prefix=/export/home/drkirkby/9/sage-4.5.3.alpha2/local gnutls.pc:Libs.private: -L${exec_prefix}/lib -lgnutls -L/export/home/drkirkby/9/sage-4.5.3.alpha2/local/lib -lgcrypt -lgpg-error -lnsl -lsocket gsl.pc:prefix=/export/home/drkirkby/9/sage-4.5.3.alpha2/local gsl.pc:exec_prefix=/export/home/drkirkby/9/sage-4.5.3.alpha2/local gsl.pc:libdir=/export/home/drkirkby/9/sage-4.5.3.alpha2/local/lib gsl.pc:includedir=/export/home/drkirkby/9/sage-4.5.3.alpha2/local/include gsl.pc:Libs: -L/export/home/drkirkby/9/sage-4.5.3.alpha2/local/lib -lgsl -lgslcblas -lm gsl.pc:Cflags: -I/export/home/drkirkby/9/sage-4.5.3.alpha2/local/includeSAGE_ROOT=/tmp/sage-4.5.3.alpha2 libpng.pc:prefix=/export/home/drkirkby/9/sage-4.5.3.alpha2/local libpng12.pc:prefix=/export/home/drkirkby/9/sage-4.5.3.alpha2/local libR.pc:rhome=/export/home/drkirkby/9/sage-4.5.3.alpha2/local/lib/R libR.pc:rincludedir=/export/home/drkirkby/9/sage-4.5.3.alpha2/local/lib/R/include opencdk.pc:prefix=/export/home/drkirkby/9/sage-4.5.3.alpha2/local opencdk.pc:Libs.private: -L${exec_prefix}/lib -lopencdk -L/export/home/drkirkby/9/sage-4.5.3.alpha2/local/lib -lgcrypt -lgpg-error -lz pynac.pc:prefix=/export/home/drkirkby/9/sage-4.5.3.alpha2/local sqlite3.pc:prefix=/export/home/drkirkby/9/sage-4.5.3.alpha2/local zlib.pc:prefix=/export/home/drkirkby/9/sage-4.5.3.alpha2/local zlib.pc:includedir=/export/home/drkirkby/9/sage-4.5.3.alpha2/local/include
As you can see, there are tons of references to the old location.
Now lets see how many of those files have "tmp" in them.
drkirkby@hawk:/tmp/sage-4.5.3.alpha2/local/lib/pkgconfig$ grep drkirkby * | grep tmp gsl.pc:Cflags: -I/export/home/drkirkby/9/sage-4.5.3.alpha2/local/includeSAGE_ROOT=/tmp/sage-4.5.3.alpha2 drkirkby@hawk:/tmp/sage-4.5.3.alpha2/local/lib/pkgconfig$
So for me at least, those .pc files are not being updated.
Would it be easier to do this with a simple sed script? I don't know precisely how to do this, but I could ask on comp.unix.shell I reckon this could be done in a very short unix shell script - it does not need loads of Python IMHO.
Dave
comment:27 Changed 5 years ago by drkirkby
I forgot to add - I did apply the patch!!
comment:28 follow-up: ↓ 29 Changed 5 years ago by kcrisman
Data point - this does seem to work for me.
Crismans-Computer:~ crisman$ cd Desktop/sage-4.6/local/lib/pkgconfig/ Crismans-Computer:~/Desktop/sage-4.6/local/lib/pkgconfig crisman$ grep 4.5.3 *
Though it wasn't a *totally* fresh build... I did delete everything, though. Why does this have to be totally fresh? On this computer (one of the original report computers) it takes a long time to build, that's why I ask.
comment:29 in reply to: ↑ 28 Changed 5 years ago by jason
Replying to kcrisman:
Data point - this does seem to work for me.
Crismans-Computer:~ crisman$ cd Desktop/sage-4.6/local/lib/pkgconfig/ Crismans-Computer:~/Desktop/sage-4.6/local/lib/pkgconfig crisman$ grep 4.5.3 *Though it wasn't a *totally* fresh build... I did delete everything, though. Why does this have to be totally fresh? On this computer (one of the original report computers) it takes a long time to build, that's why I ask.
The totally fresh part is to make sure that the initialization the first time Sage is started up is done correctly. In reality, as long as the current directory is the same as the build directory, and the sage-current-location.txt is deleted, it will probably also work.
comment:30 in reply to: ↑ 26 ; follow-up: ↓ 31 Changed 5 years ago by jason
- Status changed from needs_work to needs_review
Replying to drkirkby:
This is simply not working for me. The system is a Sun Ultra 27 running OpenSolaris on an Intel Xeon processor. This is what I did.
You didn't quite follow the instructions.
1) Build Sage fresh in the directory
/export/home/drkirkby/9/sage-4.5.3.alpha2
Sage had not been run. There was no file SAGE_ROOT/local/lib/sage-current-location.txt
Here you need to apply the patch and run Sage (step 4). Sage must be run, with the patch applied, from the original build directory, to properly initialize the pkconfig files and the sage-current-location.txt file.
After applying the patch, making sure there is no sage-current-location.txt file, then running Sage once, only *then* should you move the directory.
Would it be easier to do this with a simple sed script? I don't know precisely how to do this, but I could ask on comp.unix.shell I reckon this could be done in a very short unix shell script - it does not need loads of Python IMHO.
More than just the pkgconfig files are being updated, so it's not just that simple. However, a sufficiently talented (or un-busy) shell programmer could do the job, probably.
comment:31 in reply to: ↑ 30 ; follow-up: ↓ 32 Changed 5 years ago by drkirkby
Replying to jason:
Replying to drkirkby:
This is simply not working for me. The system is a Sun Ultra 27 running OpenSolaris on an Intel Xeon processor. This is what I did.
You didn't quite follow the instructions.
I think it was a case of not exactly explaining what I did. I believe I did run Sage in the original location. When I extracted the tar file for a second time, it showed the file
$SAGE_LOCAL/lib/sage-current-location.txt
But prior to running Sage that file did not exist. Then when I moved Sage, the contents of
$SAGE_LOCAL/lib/sage-current-location.txt
changed, but the contents of the .pc files did not substantially change.
I will revisit this - but I'm still not convinced it is working myself. It might be a platform specific problem.
Dave
comment:32 in reply to: ↑ 31 Changed 5 years ago by jason
Replying to drkirkby:
Replying to jason:
Replying to drkirkby:
This is simply not working for me. The system is a Sun Ultra 27 running OpenSolaris on an Intel Xeon processor. This is what I did.
You didn't quite follow the instructions.
I think it was a case of not exactly explaining what I did. I believe I did run Sage in the original location. When I extracted the tar file for a second time, it showed the file
$SAGE_LOCAL/lib/sage-current-location.txt
But prior to running Sage that file did not exist. Then when I moved Sage, the contents of
$SAGE_LOCAL/lib/sage-current-location.txt
changed, but the contents of the .pc files did not substantially change.
I will revisit this - but I'm still not convinced it is working myself. It might be a platform specific problem.
The fact that in your before tarball, the pkgconfig files did not start with SAGE_ROOT leads me to believe that the sage-current-location.txt file was created *before* the patch was applied, and was never deleted and sage run before moving the directory. With the new patch applied, when that file is created, the pkgconfig files should be modified to include a SAGE_ROOT line at the very top.
I tried to use the python OS-agnostic functions everywhere, so on the surface, I doubt it's an OS-specific problem.
So, if you have time, please try again, and make sure that after the patch is applied, the sage-current-location.txt file is deleted which for convenience, are repeated below:
To review, this should be sufficient:
#. Download and build Sage (important; it must be a fresh build from source).
#. Apply the patch at the ticket to the scripts repository (in SAGE_ROOT/local/bin):
http://trac.sagemath.org/sage_trac/raw-attachment/ticket/9210/trac-9210-rewrite-sage-location.patch
#. Delete the file SAGE_ROOT/local/lib/sage-current-location.txt, if it exists
#. Now run Sage, then exit (this updates the sage-current-location and pkgconfig files appropriately). Check to make sure that sage-current-location.txt is now created, and that each pkgconfig file starts with a line "SAGE_ROOT=" (followed by the build directory)
#. Move the sage directory and run Sage again. The SAGE_ROOT/local/lib/sage-current-location.txt should be updated, and the pkgconfig files in SAGE_ROOT/local/lib/pkgconfig/ should also be updated to the new path (you can check the freetype2.pc file, for example, to see the new sage directory is listed there in the SAGE_ROOT variable).
Thanks!
comment:33 follow-up: ↓ 34 Changed 5 years ago by jason
Apparently I don't remember the right wiki syntax. Here are the 5 steps again:
- Download and build Sage (important; it must be a fresh build from source).
- Apply the patch at the ticket to the scripts repository (in SAGE_ROOT/local/bin):http://trac.sagemath.org/sage_trac/raw-attachment/ticket/9210/trac-9210-rewrite-sage-location.patch
- Delete the file SAGE_ROOT/local/lib/sage-current-location.txt, if it exists
- Now run Sage, then exit (this updates the sage-current-location and pkgconfig files appropriately). Check to make sure that sage-current-location.txt is now created, and that each pkgconfig file starts with a line "SAGE_ROOT=" (followed by the build directory)
- Move the sage directory and run Sage again. The SAGE_ROOT/local/lib/sage-current-location.txt should be updated, and the pkgconfig files in SAGE_ROOT/local/lib/pkgconfig/ should also be updated to the new path (you can check the freetype2.pc file, for example, to see the new sage directory is listed there in the SAGE_ROOT variable).
comment:34 in reply to: ↑ 33 Changed 5 years ago by kcrisman
- Apply the patch at the ticket to the scripts repository (in SAGE_ROOT/local/bin):http://trac.sagemath.org/sage_trac/raw-attachment/ticket/9210/trac-9210-rewrite-sage-location.patch
The link is wrong but the text is right; http://trac.sagemath.org/sage_trac/raw-attachment/ticket/9210/trac-9210-rewrite-sage-location.patch
comment:35 follow-up: ↓ 36 Changed 5 years ago by kcrisman
- Reviewers set to David Kirkby, Karl-Dieter Crisman
After doing this on a NON fresh build, just to fix something dumb, I get
new-host-2:pkgconfig karl-dietercrisman$ grep 4.6 * bdw-gc.pc:SAGE_ROOT=/Users/.../sage-4.6.alpha4 freetype2.pc:SAGE_ROOT=/Users/.../sage-4.6.alpha4 gnutls-extra.pc:SAGE_ROOT=/Users/.../sage-4.6.alpha4 gnutls.pc:SAGE_ROOT=/Users/.../sage-4.6.alpha4 gsl.pc:SAGE_ROOT=/Users/.../sage-4.6.alpha4 libR.pc:SAGE_ROOT=/Users/.../sage-4.6.alpha4 libR.pc:rhome=/Users/.../sage-4.6.alpha1/local/lib/R libR.pc:rincludedir=/Users/.../sage-4.6.alpha1/local/lib/R/include libpng.pc:SAGE_ROOT=/Users/.../sage-4.6.alpha4 libpng12.pc:SAGE_ROOT=/Users/.../sage-4.6.alpha4 opencdk.pc:SAGE_ROOT=/Users/.../sage-4.6.alpha4 pynac.pc:SAGE_ROOT=/Users/.../sage-4.6.alpha4 pynac.pc:prefix=/Users/.../sage-4.6.alpha2/local sqlite3.pc:SAGE_ROOT=/Users/.../sage-4.6.alpha4 zlib.pc:SAGE_ROOT=/Users/.../sage-4.6.alpha4 new-host-2:pkgconfig karl-dietercrisman$ grep 4.5 * bdw-gc.pc:prefix=/Users/.../sage-4.5.1/local freetype2.pc:prefix=/Users/.../sage-4.5.1/local gnutls-extra.pc:prefix=/Users/.../sage-4.5.1/local gnutls-extra.pc:Libs.private: -L${exec_prefix}/lib -lgnutls-extra -L/Users/.../sage-4.5.1/local/lib -lopencdk -L/Users/.../sage-4.5.1/local/lib -lgcrypt -L/Users/.../sage-4.5.1/local/lib -lgpg-error -L/Users/.../sage-4.5.1/local/lib -lz -R/Users/.../sage-4.5.1/local/lib -L${exec_prefix}/lib -lgnutls -L/Users/.../sage-4.5.1/local/lib -lgcrypt -lgpg-error gnutls.pc:prefix=/Users/.../sage-4.5.1/local gnutls.pc:Libs.private: -L${exec_prefix}/lib -lgnutls -L/Users/.../sage-4.5.1/local/lib -lgcrypt -lgpg-error gsl.pc:prefix=/Users/.../sage-4.5.2/local gsl.pc:exec_prefix=/Users/.../sage-4.5.2/local gsl.pc:libdir=/Users/.../sage-4.5.2/local/lib gsl.pc:includedir=/Users/.../sage-4.5.2/local/include gsl.pc:Libs: -L/Users/.../sage-4.5.2/local/lib -lgsl -lgslcblas -lm gsl.pc:Cflags: -I/Users/.../sage-4.5.2/local/include libpng.pc:prefix=/Users/.../sage-4.5.1/local libpng12.pc:prefix=/Users/.../sage-4.5.1/local opencdk.pc:prefix=/Users/.../sage-4.5.1/local opencdk.pc:Libs.private: -L${exec_prefix}/lib -lopencdk -L/Users/.../sage-4.5.1/local/lib -lgcrypt -lgpg-error -lz sqlite3.pc:prefix=/Users/.../sage-4.5.1/local zlib.pc:prefix=/Users/.../sage-4.5.1/local zlib.pc:includedir=/Users/.../sage-4.5.1/local/include
So although it relists the SAGE_ROOT variable correctly, all those prefixes and includes and whatever else still are bad news. I don't know how to use sed like Mitesh suggested on sage-release, but I think I can change the freetype one by hand.
comment:36 in reply to: ↑ 35 Changed 5 years ago by jason
Replying to kcrisman:
After doing this on a NON fresh build, just to fix something dumb, I get
The result is expected. The SAGE_ROOT directory is no longer the same as the build directory (as evidenced by the plethora of other build directories in the pkgconfig files). This patch only works correctly if things are initialized in the original build directory.
comment:37 follow-up: ↓ 38 Changed 5 years ago by kcrisman
I had a long followup, but your reply explained things. So why can't we also update those things? The new sage-location script update_pkgconfig_files doesn't seem to touch anything but that SAGE_ROOT guy, but it looks like there isn't any reason it couldn't, in theory, do those prefix statements too and just make them ${SAGE_ROOT}/local or whatever is appropriate.
(I'm going to try that change 'by hand' on freetype, by the way.)
comment:38 in reply to: ↑ 37 ; follow-up: ↓ 39 Changed 5 years ago by jason
Replying to kcrisman:
I had a long followup, but your reply explained things. So why can't we also update those things? The new sage-location script update_pkgconfig_files doesn't seem to touch anything but that SAGE_ROOT guy, but it looks like there isn't any reason it couldn't, in theory, do those prefix statements too and just make them ${SAGE_ROOT}/local or whatever is appropriate.
(I'm going to try that change 'by hand' on freetype, by the way.)
I suppose the general problem is that we only know what the previous SAGE_ROOT was (given in the sage-location.txt file). We don't know what any other path in those pkgconfig files are, including the SAGE_ROOT from the time before the last move.
I suppose we could be more invasive and always change the prefix path as well. But there are other paths too. What paths do you change? In your example, you had at least 3-4 paths from old build directories, and lots of them are in things other than prefix statements. Which ones are old SAGE_ROOTs, and which ones are still valid locations that shouldn't be changed? For the ones that are old SAGE_ROOTs, which are the SAGE_ROOT parts and which are the parts that shouldn't be changed?
The strategy now is to replace all SAGE_ROOT paths with the SAGE_ROOT variable on the first startup, and then only change that in subsequent runs if needed. That seems *much* safer.
Right now, the strategy breaks down for pkgconfig files created after the initial build.
Another ticket should be created to update pkgconfig files that don't have SAGE_ROOT already defined, which takes care of spkgs installed later.
comment:39 in reply to: ↑ 38 ; follow-up: ↓ 40 Changed 5 years ago by kcrisman
(I'm going to try that change 'by hand' on freetype, by the way.)
Which worked!
On another machine, doing that wasn't enough to get out of this issue... there was still a reference to the wrong thing in libR.dylib, which is a binary file, and which I have no idea how to change. But that's not this ticket. As far as I have time to test this, it did what it said, so positive review from me. I don't have time to do the complete build from scratch, but it definitely changed what it claimed to in the right way from a non-scratch situation - and it WAS changed.
Another ticket should be created to update pkgconfig files that don't have SAGE_ROOT already defined, which takes care of spkgs installed later.
This seems reasonable, and hopefully with time won't be much of an issue. Sorry I can't give final positive review.
comment:40 in reply to: ↑ 39 ; follow-up: ↓ 41 Changed 5 years ago by jason
Replying to kcrisman:
Another ticket should be created to update pkgconfig files that don't have SAGE_ROOT already defined, which takes care of spkgs installed later.
This seems reasonable, and hopefully with time won't be much of an issue. Sorry I can't give final positive review.
So I think that means we need at least one more reviewer. Dave, do you have time to look at this again?
comment:41 in reply to: ↑ 40 Changed 5 years ago by drkirkby
Replying to jason:
So I think that means we need at least one more reviewer. Dave, do you have time to look at this again?
I won't have much time to play with this until Wednesday, but I'm just doing a fresh build. This time I'm using the 'script' command so it will have an exact record of what I did - even my typos are recorded. If there's a problem I'll make that file available, if there's not, I'll give it a positive review.
BTW, someone above said they did not know how to use 'sed'. There are plenty of references on the web, but this file I always find useful, as it normally has something there which is sufficiently similar to what I want to achieve, that I can modify it to do what I want.
http://sed.sourceforge.net/sed1line.txt
dave
comment:42 Changed 5 years ago by drkirkby
OK, Sage is built. Let's try my luck for the Nth time!
comment:43 Changed 5 years ago by drkirkby
I think that has worked. I am just going to run the doctests to make sure.
Dave
comment:44 follow-up: ↓ 45 Changed 5 years ago by drkirkby
Em, this is odd. I used 'mv' to move the Sage directory from:
/export/home/drkirkby/9210/sage-4.6.alpha3/local
to:
/tmp/9210/sage-4.6.alpha3/local
I got some warnings about unable to save ACLs. Now when I run the doctests, several are failing. One at least is due to the fact 'Maxima' can't run. When I look at the script $SAGE_ROOT/local/bin/maxima, I see that still have the old location:
drkirkby@hawk:/tmp/9210/sage-4.6.alpha3$ pwd /tmp/9210/sage-4.6.alpha3 drkirkby@hawk:/tmp/9210/sage-4.6.alpha3$ grep drkirkby local/bin/maxima prefix=`unixize "/export/home/drkirkby/9210/sage-4.6.alpha3/local"` top_srcdir=`unixize "/export/home/drkirkby"`
So perhaps the .pc files are not the only things that need changing. I realise that's outside of the scope of this ticket, and it may be due to the warnings about unable to save ALCs, so I build Sage again and use 'tar' to move it.
Dave
comment:45 in reply to: ↑ 44 Changed 5 years ago by jason
Replying to drkirkby:
So perhaps the .pc files are not the only things that need changing. I realise that's outside of the scope of this ticket, and it may be due to the warnings about unable to save ALCs, so I build Sage again and use 'tar' to move it.
Most definitely there are other issues besides the pkgconfig files. I'm not confident in moving my installation at all anymore. This ticket just takes care of one aspect of the problem that prevented compiling things in a new location.
comment:46 follow-up: ↓ 47 Changed 5 years ago by drkirkby
- Status changed from needs_review to positive_review
I've taken another look at this. It looks like I was mistaken when I thought the .pc file were not being updated properly, since they do now need to be.
Prior to moving the distribution, there were two doctest failures - #10041 and 10042.
sage -t -long "devel/sage/sage/rings/polynomial/polynomial_element.pyx" sage -t -long "devel/sage/sage/tensor/differential_forms.py"
After moving the location, there are 5 doctest failures.
The following tests failed: sage -t -long -force_lib devel/sage/doc/en/tutorial/tour_advanced.rst # 2 doctests failed sage -t -long -force_lib devel/sage/doc/fr/tutorial/tour_advanced.rst # 2 doctests failed sage -t -long -force_lib devel/sage/sage/tensor/differential_forms.py # 1 doctests failed sage -t -long -force_lib devel/sage/sage/rings/polynomial/polynomial_element.pyx # 1 doctests failed sage -t -long -force_lib devel/sage/sage/rings/polynomial/groebner_fan.py # 76 doctests failed sage -t -long -force_lib devel/sage/sage/rings/polynomial/multi_polynomial_ideal.py # 1 doctests failed ---------------------------------------------------------------------- Total time for all tests: 1583.2 seconds
So clearly this is not a complete fix for moving Sage, but it's a step in the right direction.
So I'm giving this positive review.
I'm sorry I have dragged this ticket out so long. I was of the belief it was not working correctly, so that is why I declined to give it positive review before, but now I'm convinced it is ok.
Like Jason, I'm not happy to move Sage. It makes me wonder how good the binaries are that we distribute. Does anyone doctest the binaries after they are created before they are made public?
Dave
comment:47 in reply to: ↑ 46 ; follow-up: ↓ 48 Changed 5 years ago by mpatel
Replying to drkirkby:
The following tests failed: sage -t -long -force_lib devel/sage/doc/en/tutorial/tour_advanced.rst # 2 doctests failed sage -t -long -force_lib devel/sage/doc/fr/tutorial/tour_advanced.rst # 2 doctests failed sage -t -long -force_lib devel/sage/sage/tensor/differential_forms.py # 1 doctests failed sage -t -long -force_lib devel/sage/sage/rings/polynomial/polynomial_element.pyx # 1 doctests failed sage -t -long -force_lib devel/sage/sage/rings/polynomial/groebner_fan.py # 76 doctests failed sage -t -long -force_lib devel/sage/sage/rings/polynomial/multi_polynomial_ideal.py # 1 doctests failed ---------------------------------------------------------------------- Total time for all tests: 1583.2 seconds
If you still have it, could you attach or give a link to the test log?
comment:48 in reply to: ↑ 47 ; follow-up: ↓ 49 Changed 5 years ago by drkirkby
Replying to mpatel:
If you still have it, could you attach or give a link to the test log?
Sure, it is here. The tests were run on 'hawk', my OpenSolaris? machine.
comment:49 in reply to: ↑ 48 Changed 5 years ago by mpatel
Replying to drkirkby:
Replying to mpatel:
If you still have it, could you attach or give a link to the test log?
Sure, it is here. The tests were run on 'hawk', my OpenSolaris? machine.
Thanks. At least, the new errors all have one "cause":
RuntimeError: ld.so.1: gfan: fatal: relocation error: file /export/home/test/sage-4.6.alpha3/local/bin/gfan: symbol _ZNSt15_List_node_base7_M_hookEPS_: referenced symbol not found
Of course, we can make this a separate ticket, but why would this fatal: relocation error happen?
comment:50 follow-up: ↓ 51 Changed 5 years ago by leif
I wonder if this should be a blocker...
comment:51 in reply to: ↑ 50 Changed 5 years ago by jason
Replying to leif:
I wonder if this should be a blocker...
No. At most critical. Sage certainly has been released for a long time without this fix.
comment:52 Changed 5 years ago by drkirkby
I found the reason gfan has unresolved variables when moved - it is to do with gcc and the linker run time search path.
In a Sage shell, before the move:
(sage subshell) hawk:sage-4.6.alpha3 drkirkby$ ldd local/bin/gfan libgmp.so.3 => /export/home/drkirkby/sage-4.6.alpha3/local/lib//libgmp.so.3 libstdc++.so.6 => /usr/local/gcc-4.5.0/lib/libstdc++.so.6 libm.so.2 => /lib/libm.so.2 libgcc_s.so.1 => /usr/local/gcc-4.5.0/lib/libgcc_s.so.1 libc.so.1 => /lib/libc.so.1 SAGE_ROOT=/export/home/drkirkby/sage-4.6.alpha3
After creating another account, and moving Sage, I must have not set LD_LIBRARY_PATH identically, so ldd shows:
(sage subshell) hawk:bin test$ ldd gfan libgmp.so.3 => /export/home/test/sage-4.6.alpha3/local/lib//libgmp.so.3 libstdc++.so.6 => /usr/lib/libstdc++.so.6 libm.so.2 => /lib/libm.so.2 libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 libc.so.1 => /lib/libc.so.1 SAGE_ROOT=/export/home/test/sage-4.6.alpha3
i.e. instead of gfan finding the C++ libary it was linked against (e..g. /usr/local/gcc-4.5.0/lib/libstdc++.so.6) if finds an older library in /usr/lib/libstdc++.so.6. That is from gcc 3.4.3 - i.e. very old.
This is something we have to be very weary of on Solaris. We should make available recent run-time libraries, otherwise the user will probably have trouble. I know there have been reports of this on sage-support, and other sage-developers have hit it on t2.math. If we compile Sage with a recent gcc, and the person using a binary only had old gcc libraries, and/or new libraries are not in their linker search path, Sage will not run well.
There was one other thing I found odd. Before moving the code, though after Sage had been run, I tried this in a Sage shell.
(sage subshell) hawk:sage-4.6.alpha3 drkirkby$ local/bin/gfan 1+2 PARSER ERROR: Expected: "field" Found: "1" Assertion failed: 0, file parser.cpp, line 509 Abort (core dumped) SAGE_ROOT=/export/home/drkirkby/sage-4.6.alpha3
I've got no idea how gfan is supposed to be used, but trying 1+2 is enough to make it dump core, which is a bug in my opinion.
I tried it on sage.math too. Whilst it does not dump core, it is not very elegant:
(sage subshell) sage:bin kirkby$ gfan 1+2 PARSER ERROR: Expected: "field" Found: "1" gfan: parser.cpp:509: Field CharacterBasedParser::parseField(): Assertion `0' failed. Aborted SAGE_ROOT=/usr/local/sage
And on bsd.math (OS X)
(sage subshell) bsd:bin kirkby$ gfan 1+1 PARSER ERROR: Expected: "field" Found: "1" Assertion failed: (0), function parseField, file parser.cpp, line 509. Abort trap SAGE_ROOT=/Users/kirkby/sage-4.6.alpha0
So as soon as the parser sees something it does not reconise, it either aborts or dumps core. Not very elegant.
Dave
comment:53 Changed 5 years ago by jason
Dave,
Just to clarify, are you saying something more needs to be done on this ticket, or are you commenting on another issue (with valuable information that should go on another ticket to fix the gfan issue)?
comment:54 Changed 5 years ago by drkirkby
No, I'm happy with this ticket.
The gfan core dump should go on another ticket.
I'm not sure how best to handle the situation with libraries that are older than the version Sage was built with.
Neither issue is related to the ticket as such.
Dave
comment:55 Changed 5 years ago by mpatel
- Priority changed from major to blocker
comment:56 Changed 5 years ago by mpatel
- Merged in set to sage-4.6.rc0
- Resolution set to fixed
- Status changed from positive_review to closed
comment:57 in reply to: ↑ 24 Changed 5 years ago by leif
Replying to mpatel:
Not updating the .pc files in SAGE_LOCAL/lib/pkgconfig may also cause problems during an upgrade after moving Sage. Please see sage-release for Karl-Dieter's report about a 4.5.2.rc1-4.5.3.alpha1 upgrade on PPC OS X 10.4
Also, for some packages that compile Python extension modules during the upgrade, it seems to help to update old paths in SAGE_LOCAL/lib/python/config/Makefile. [...]
Cf. #9896. Python's Makefile and pyconfig.h define lots of variables that contain hard-coded paths. These all go into distutils.sysconfig.get_config_vars() (a dictionary), but only some are used, and not all obsolete paths are actually harmful.
My patch to devel/sage/setup.py at #9896 e.g. just fixes invalid / obsolete library search directories coded into distutils' dynamic linker command(!) (on Darwin).
An alternative is to patch distutils itself.
comment:58 Changed 5 years ago by leif
- Summary changed from pkg-config prefix statements in SAGE_LOCAL/lib/pkg-config not changed upon Sage move to pkg-config's prefix statements in SAGE_LOCAL/lib/pkgconfig not changed upon Sage move
comment:59 follow-up: ↓ 60 Changed 5 years ago by jhpalmieri
See #10162 for a follow-up.
Note that if you've already moved your build directory, you will have to move it again to trigger the code in this patch that updates the pkg-config file paths.