Opened 7 years ago
Closed 7 years ago
#15901 closed defect (fixed)
gitrelated doctest failures after moving sage installation
Reported by:  dkrenn  Owned by:  

Priority:  major  Milestone:  sage6.3 
Component:  misc  Keywords:  doctest, git, relocate 
Cc:  ppurka  Merged in:  
Authors:  John Palmieri  Reviewers:  Peter Bruin 
Report Upstream:  N/A  Work issues:  
Branch:  782e3f6 (Commits)  Commit:  782e3f6cc0a3852dbb83e26d1f1e41563821a8e5 
Dependencies:  Stopgaps: 
Description
I've compiled sage in /local/data/krenn/sagedev/sage6.1.1
and then moved it to another directory. I am getting the following when testing sage:
File "src/sage/dev/test/config.py", line 43, in sage.dev.test.config.DoctestConfig Failed example: DoctestConfig() Expected: Config(''' [trac] username = doctest ticket_cache = ... [UI] log_level = 1 [git] ssh_key_set = True repository_anonymous = remote_repository_undefined repository = remote_repository_undefined src = ... dot_git = ... [sagedev] ''') Got: warning: templates not found /local/data/krenn/sagedev/sage6.1.1/local/share/gitcore/templates Config(''' [trac] username = doctest ticket_cache = /home/krenn/.sage/temp/brown/8826/dir_0ABZH9/ticket_cache [UI] log_level = 1 [git] ssh_key_set = True repository_anonymous = remote_repository_undefined repository = remote_repository_undefined src = /home/blub/.sage/temp/brown/8826/dir_0ABZH9/repo dot_git = /home/blub/.sage/temp/brown/8826/dir_0ABZH9/repo/.git user.name = doctest user.email = doc@test.test user_email_set = True [sagedev] ''')
There are a couple of other files with similar warnings.
Change History (32)
comment:1 followup: ↓ 2 Changed 7 years ago by
comment:2 in reply to: ↑ 1 Changed 7 years ago by
Replying to dkrenn:
FYI: A manual fix is to use
rm local/var/lib/sage/installed/git1.8.4.4 maketo rebuild git.
Or just
./sage f git
comment:3 Changed 7 years ago by
 Cc ppurka added
comment:4 Changed 7 years ago by
 Branch set to u/jhpalmieri/gitrelocate
 Commit set to 62524f6e67df98602d37d6be82315357e8ea6b0b
 Status changed from new to needs_review
Here's an attempt at a patch. Works for me.
New commits:
62524f6  Get rid of warnings about missing git templates after relocating Sage

comment:5 followup: ↓ 6 Changed 7 years ago by
How do you compile sage after this?
sage dev checkout ticket 15901 sage btp long src/sage/dev
does not work.
Even the following doesn't work
cp {src,local}/bin/sageenv sage btp long src/sage/dev
comment:6 in reply to: ↑ 5 Changed 7 years ago by
Replying to ppurka:
How do you compile sage after this?
You have to touch the src/sage/dev/[test/]
Python files (or remove their .pyc
files).
(I think.)
comment:7 Changed 7 years ago by
Ahem, or maybe something else... 8)
comment:8 Changed 7 years ago by
Hmm. I guess I should have said "Worked for me yesterday on one machine." I can't get it to work today on a different machine (and haven't tried on the first one yet).
comment:9 Changed 7 years ago by
 Commit changed from 62524f6e67df98602d37d6be82315357e8ea6b0b to 782e3f6cc0a3852dbb83e26d1f1e41563821a8e5
Branch pushed to git repo; I updated commit sha1. New commits:
782e3f6  Get rid of warnings about missing git templates after relocating Sage

comment:10 Changed 7 years ago by
Okay, here's another attempt. Please try this one. (I hope that just sage tp src/sage/dev
works. There are no changes to Python files, so sage b
should be unnecessary.)
comment:11 Changed 7 years ago by
 Milestone changed from sage6.2 to sage6.3
comment:12 Changed 7 years ago by
This must explain the failures in the first patchbot run of 6.2 (http://patchbot.sagemath.org/ticket/0/
, 20140506 20:45:32 +0100); I made a fresh Sage install for experimenting with the patchbot, but then decided to move it. Patchbot is now running in the original location, but I'll try this patch on another copy.
comment:13 Changed 7 years ago by
The patchbot instance is now getting doctest failures in sage/plot/plot.py
:
RuntimeError: Could not open facefile /home/pbruin/src/sage/local/lib/python2.7/sitepackages/matplotlib1.3.1py2.7linuxx86_64.egg/matplotlib/mpldata/fonts/ttf/Vera.ttf; Cannot_Open_Resource
The directory it is looking in is where my other Sage copy was before moving that copy to test this ticket! The path seems to have been obtained from a cache living in my home directory: ~/.cache/matplotlib/fontList.cache
. Does this mean that sagelocation
potentially has to worry about hardcoded paths existing in $DOT_SAGE
?
comment:14 followup: ↓ 15 Changed 7 years ago by
Curiously, sometime after the above failures in sage/plot
, my patchbot copy of Sage seems to have updated the paths in ~/.cache/matplotlib
. Anyway, I can confirm that the failures in sage/dev
are fixed by this patch.
comment:15 in reply to: ↑ 14 ; followups: ↓ 16 ↓ 17 Changed 7 years ago by
 Reviewers set to Peter Bruin
 Status changed from needs_review to positive_review
Replying to pbruin:
I can confirm that the failures in
sage/dev
are fixed by this patch.
... which should be enough for a positive review. If there really is a problem with the matplotlib
cache, it should be dealt with on a different ticket.
comment:16 in reply to: ↑ 15 Changed 7 years ago by
Replying to pbruin:
Replying to pbruin:
I can confirm that the failures in
sage/dev
are fixed by this patch.... which should be enough for a positive review. If there really is a problem with the
matplotlib
cache, it should be dealt with on a different ticket.
Then we should probably also change the ticket's title accordingly.
comment:17 in reply to: ↑ 15 ; followups: ↓ 18 ↓ 20 Changed 7 years ago by
Replying to pbruin:
If there really is a problem with the
matplotlib
cache, it should be dealt with on a different ticket.
This really deserves its own ticket (and btw. is pretty unrelated to relocating Sage; just imagine having different versions of Sage running at the same time).
IMHO Sage's matplotlib must not put its cache into $HOME
(if at all, it should be in $DOT_SAGE/.cache/...
), and it should somehow be versioned (with matplotlib's and/or Sage's version in the folder name).
comment:18 in reply to: ↑ 17 Changed 7 years ago by
Replying to leif:
IMHO Sage's matplotlib must not put its cache into
$HOME
(if at all, it should be in$DOT_SAGE/.cache/...
), and it should somehow be versioned (with matplotlib's and/or Sage's version in the folder name).
... or probably $SAGE_LOCAL/var/{lib,tmp}/matplotlib/...
comment:19 Changed 7 years ago by
 Summary changed from doctest failures after moving sage installation to gitrelated doctest failures after moving sage installation
Does it need to be writeable? Anyway, it looks like we can modify this by setting the environment variable XDG_CACHE_HOME
: see the functions _get_cachedir
and _get_xdg_cache_dir
in sage/local/lib/python/sitepackages/matplotlib1.3.1py2.7macosx10.8x86_64.egg/matplotlib/__init__.py
. I also think this belongs on another ticket.
comment:20 in reply to: ↑ 17 Changed 7 years ago by
Replying to leif:
IMHO Sage's matplotlib must not put its cache into
$HOME
(if at all, it should be in$DOT_SAGE/.cache/...
), and it should somehow be versioned (with matplotlib's and/or Sage's version in the folder name).
In fact that is where it used to be: I have an old directory ~/.sage/matplotlib1.2.1/
lying around, containing fontList.cache
and tex.cache/
, just like ~/.cache/matplotlib/
looks now.
comment:21 followups: ↓ 22 ↓ 24 Changed 7 years ago by
By default, Sage's matplotlib should write to .sage/matplotlib1.3.1
. Do you have any environment variables set, like MPLCONFIGDIR
? On my OS X machine, I just checked: MPLCONFIGDIR
(which Sage sets to .sage/matplotlib1.3.1
if it's unset) still gets fontList.cache
and tex.cache/
added to it when you call plot
from Sage. My $HOME/.cache
directory only contains fontconfig/
.
comment:22 in reply to: ↑ 21 ; followup: ↓ 23 Changed 7 years ago by
Replying to jhpalmieri:
By default, Sage's matplotlib should write to
.sage/matplotlib1.3.1
. Do you have any environment variables set, likeMPLCONFIGDIR
? On my OS X machine, I just checked:MPLCONFIGDIR
(which Sage sets to.sage/matplotlib1.3.1
if it's unset) still getsfontList.cache
andtex.cache/
added to it when you callplot
from Sage. My$HOME/.cache
directory only containsfontconfig/
.
Well, setting up MPL's config dir based on its version in sageenv
in turn can indeed break upon relocation... :)
(as it calls Python before sagelocation
is called, and the Python interpreter may try to load libraries that have moved, as I already mentioned elsewhere).
comment:23 in reply to: ↑ 22 Changed 7 years ago by
Replying to leif:
Well, setting up MPL's config dir based on its version in
sageenv
in turn can indeed break upon relocation... :)(as it calls Python before
sagelocation
is called, and the Python interpreter may try to load libraries that have moved, as I already mentioned elsewhere).
See for example http://trac.sagemath.org/ticket/16044#comment:7 ff.
(Note that IIRC sagelocation
currently wouldn't help there anyway, as it doesn't mess with install_name
s.)
comment:24 in reply to: ↑ 21 ; followup: ↓ 25 Changed 7 years ago by
Replying to jhpalmieri:
By default, Sage's matplotlib should write to
.sage/matplotlib1.3.1
. Do you have any environment variables set, likeMPLCONFIGDIR
? On my OS X machine, I just checked:MPLCONFIGDIR
(which Sage sets to.sage/matplotlib1.3.1
if it's unset) still getsfontList.cache
andtex.cache/
added to it when you callplot
from Sage. My$HOME/.cache
directory only containsfontconfig/
.
On GNU/Linux x86_64, I don't seem to have any environment variables with names containing "MPL", neither in the normal shell nor in sage sh
. This is despite `logs/pkgs/matplotlib1.3.1.log saying
When Sage runs, MPLCONFIGDIR will be set to '$DOT_SAGE/matplotlib1.3.1'.
comment:25 in reply to: ↑ 24 ; followup: ↓ 26 Changed 7 years ago by
Replying to pbruin:
On GNU/Linux x86_64, I don't seem to have any environment variables with names containing "MPL", neither in the normal shell nor in
sage sh
.
True. B)
$ ./sage sh c 'env  grep MPL'; uname sm Linux x86_64
comment:26 in reply to: ↑ 25 ; followup: ↓ 27 Changed 7 years ago by
Replying to leif:
Replying to pbruin:
On GNU/Linux x86_64, I don't seem to have any environment variables with names containing "MPL", neither in the normal shell nor in
sage sh
.True. B)
It turns out that this is because the following command in sageenv
fails (I removed the 2>/dev/null
at the end:
"$SAGE_ROOT/local/bin/python" c 'import pkg_resources; pkg_resources.get_distribution("matplotlib").version' Traceback (most recent call last): File "<string>", line 1, in <module> zipimport.ZipImportError: can't decompress data; zlib not available
comment:27 in reply to: ↑ 26 Changed 7 years ago by
Replying to pbruin:
Replying to leif:
Replying to pbruin:
On GNU/Linux x86_64, I don't seem to have any environment variables with names containing "MPL", neither in the normal shell nor in
sage sh
.True. B)
It turns out that this is because the following command in
sageenv
fails (I removed the2>/dev/null
at the end:"$SAGE_ROOT/local/bin/python" c 'import pkg_resources; pkg_resources.get_distribution("matplotlib").version' Traceback (most recent call last): File "<string>", line 1, in <module> zipimport.ZipImportError: can't decompress data; zlib not available
I'm getting
.../local/bin/python: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory
So apparently LD_LIBRARY_PATH
isn't set up properly to that time.
comment:28 Changed 7 years ago by
The problem in comment:26 was on 6.2.rc0; with 6.2 on a different system I get
Traceback (most recent call last): ... ImportError: cannot import name MAXREPEAT
I guess the MPLCONFIGDIR
relevant part of sageenv
should be moved down to a point where all required setting up has been done.
comment:29 Changed 7 years ago by
Oh, just noticed:
# Use a matplotlib config directory specific to Sage and specific to # the version number of matplotlib, by setting the environment # variable MPLCONFIGDIR. Note that we can't find the version number by # importing matplotlib, because that could create matplotlib's standard # config directory. So we use pkg_resources. "$SAGE_ROOT/local/bin/python" c 'import pkg_resources; pkg_resources.get_distribution("matplotlib").version' 2>/dev/null if [ $? eq 0 ]; then MPLVERSION=`"$SAGE_ROOT/local/bin/python" c 'import pkg_resources; print pkg_resources.get_distribution("matplotlib").version'` MPLCONFIGDIR="$DOT_SAGE/matplotlib$MPLVERSION" export MPLCONFIGDIR # The directory is created when Sage starts (see sage.misc.misc). fi # Add some directories to $LD_LIBRARY_PATH: # * lib/openmpi is needed for openmpi. # * lib/R/lib is needed for R in case the Sage install is moved. # * lib32 and lib64 are needed for GCC, see #12405. for d in lib/openmpi lib/R/lib lib32 lib64 lib; do libdir="$SAGE_LOCAL/$d" # Add only existing directories if [ d "$libdir" ]; then [ z "$LD_LIBRARY_PATH" ]  LD_LIBRARY_PATH=":${LD_LIBRARY_PATH}" LD_LIBRARY_PATH="${libdir}$LD_LIBRARY_PATH" fi done export LD_LIBRARY_PATH
>8O
comment:30 Changed 7 years ago by
(I meant to say it's done right after we try to call python
; I've slightly edited the snippet.)
comment:31 Changed 7 years ago by
I created #16305 for this.
comment:32 Changed 7 years ago by
 Branch changed from u/jhpalmieri/gitrelocate to 782e3f6cc0a3852dbb83e26d1f1e41563821a8e5
 Resolution set to fixed
 Status changed from positive_review to closed
FYI: A manual fix is to use
to rebuild git.