Opened 8 years ago

Closed 8 years ago

#15901 closed defect (fixed)

git-related doctest failures after moving sage installation

Reported by: dkrenn Owned by:
Priority: major Milestone: sage-6.3
Component: misc Keywords: doctest, git, relocate
Cc: ppurka Merged in:
Authors: John Palmieri Reviewers: Peter Bruin
Report Upstream: N/A Work issues:
Branch: 782e3f6 (Commits, GitHub, GitLab) Commit: 782e3f6cc0a3852dbb83e26d1f1e41563821a8e5
Dependencies: Stopgaps:

Status badges

Description

I've compiled sage in /local/data/krenn/sage-dev/sage-6.1.1 and then moved it to another directory. I am getting the following when testing sage:

File "src/sage/dev/test/config.py", line 43, in sage.dev.test.config.DoctestConfig
Failed example:
    DoctestConfig()
Expected:
    Config('''
    [trac]
    username = doctest
    ticket_cache = ...
    [UI]
    log_level = 1
    [git]
    ssh_key_set = True
    repository_anonymous = remote_repository_undefined
    repository = remote_repository_undefined
    src = ...
    dot_git = ...
    [sagedev]
    ''')
Got:
    warning: templates not found /local/data/krenn/sage-dev/sage-6.1.1/local/share/git-core/templates
    Config('''
    [trac]
    username = doctest
    ticket_cache = /home/krenn/.sage/temp/brown/8826/dir_0ABZH9/ticket_cache
    [UI]
    log_level = 1
    [git]
    ssh_key_set = True
    repository_anonymous = remote_repository_undefined
    repository = remote_repository_undefined
    src = /home/blub/.sage/temp/brown/8826/dir_0ABZH9/repo
    dot_git = /home/blub/.sage/temp/brown/8826/dir_0ABZH9/repo/.git
    user.name = doctest
    user.email = doc@test.test
    user_email_set = True
    [sagedev]
    ''')

There are a couple of other files with similar warnings.

Change History (32)

comment:1 follow-up: Changed 8 years ago by dkrenn

FYI: A manual fix is to use

rm local/var/lib/sage/installed/git-1.8.4.4
make

to rebuild git.

comment:2 in reply to: ↑ 1 Changed 8 years ago by leif

Replying to dkrenn:

FYI: A manual fix is to use

rm local/var/lib/sage/installed/git-1.8.4.4
make

to rebuild git.

Or just

./sage -f git

comment:3 Changed 8 years ago by ppurka

  • Cc ppurka added

comment:4 Changed 8 years ago by jhpalmieri

  • Authors set to John Palmieri
  • Branch set to u/jhpalmieri/git-relocate
  • Commit set to 62524f6e67df98602d37d6be82315357e8ea6b0b
  • Status changed from new to needs_review

Here's an attempt at a patch. Works for me.


New commits:

62524f6Get rid of warnings about missing git templates after relocating Sage

comment:5 follow-up: Changed 8 years ago by ppurka

How do you compile sage after this?

sage --dev checkout --ticket 15901
sage -btp --long src/sage/dev

does not work.

Even the following doesn't work

cp {src,local}/bin/sage-env
sage -btp --long src/sage/dev

comment:6 in reply to: ↑ 5 Changed 8 years ago by leif

Replying to ppurka:

How do you compile sage after this?

You have to touch the src/sage/dev/[test/] Python files (or remove their .pyc files).

(I think.)

comment:7 Changed 8 years ago by leif

Ahem, or maybe something else... 8-)

comment:8 Changed 8 years ago by jhpalmieri

Hmm. I guess I should have said "Worked for me yesterday on one machine." I can't get it to work today on a different machine (and haven't tried on the first one yet).

comment:9 Changed 8 years ago by git

  • Commit changed from 62524f6e67df98602d37d6be82315357e8ea6b0b to 782e3f6cc0a3852dbb83e26d1f1e41563821a8e5

Branch pushed to git repo; I updated commit sha1. New commits:

782e3f6Get rid of warnings about missing git templates after relocating Sage

comment:10 Changed 8 years ago by jhpalmieri

Okay, here's another attempt. Please try this one. (I hope that just sage -tp src/sage/dev works. There are no changes to Python files, so sage -b should be unnecessary.)

comment:11 Changed 8 years ago by vbraun_spam

  • Milestone changed from sage-6.2 to sage-6.3

comment:12 Changed 8 years ago by pbruin

This must explain the failures in the first patchbot run of 6.2 (http://patchbot.sagemath.org/ticket/0/, 2014-05-06 20:45:32 +0100); I made a fresh Sage install for experimenting with the patchbot, but then decided to move it. Patchbot is now running in the original location, but I'll try this patch on another copy.

comment:13 Changed 8 years ago by pbruin

The patchbot instance is now getting doctest failures in sage/plot/plot.py:

RuntimeError: Could not open facefile /home/pbruin/src/sage/local/lib/python2.7/site-packages/matplotlib-1.3.1-py2.7-linux-x86_64.egg/matplotlib/mpl-data/fonts/ttf/Vera.ttf; Cannot_Open_Resource

The directory it is looking in is where my other Sage copy was before moving that copy to test this ticket! The path seems to have been obtained from a cache living in my home directory: ~/.cache/matplotlib/fontList.cache. Does this mean that sage-location potentially has to worry about hardcoded paths existing in $DOT_SAGE?

comment:14 follow-up: Changed 8 years ago by pbruin

Curiously, sometime after the above failures in sage/plot, my patchbot copy of Sage seems to have updated the paths in ~/.cache/matplotlib. Anyway, I can confirm that the failures in sage/dev are fixed by this patch.

comment:15 in reply to: ↑ 14 ; follow-ups: Changed 8 years ago by pbruin

  • Reviewers set to Peter Bruin
  • Status changed from needs_review to positive_review

Replying to pbruin:

I can confirm that the failures in sage/dev are fixed by this patch.

... which should be enough for a positive review. If there really is a problem with the matplotlib cache, it should be dealt with on a different ticket.

comment:16 in reply to: ↑ 15 Changed 8 years ago by leif

Replying to pbruin:

Replying to pbruin:

I can confirm that the failures in sage/dev are fixed by this patch.

... which should be enough for a positive review. If there really is a problem with the matplotlib cache, it should be dealt with on a different ticket.

Then we should probably also change the ticket's title accordingly.

comment:17 in reply to: ↑ 15 ; follow-ups: Changed 8 years ago by leif

Replying to pbruin:

If there really is a problem with the matplotlib cache, it should be dealt with on a different ticket.

This really deserves its own ticket (and btw. is pretty unrelated to relocating Sage; just imagine having different versions of Sage running at the same time).

IMHO Sage's matplotlib must not put its cache into $HOME (if at all, it should be in $DOT_SAGE/.cache/...), and it should somehow be versioned (with matplotlib's and/or Sage's version in the folder name).

comment:18 in reply to: ↑ 17 Changed 8 years ago by leif

Replying to leif:

IMHO Sage's matplotlib must not put its cache into $HOME (if at all, it should be in $DOT_SAGE/.cache/...), and it should somehow be versioned (with matplotlib's and/or Sage's version in the folder name).

... or probably $SAGE_LOCAL/var/{lib,tmp}/matplotlib/...

Last edited 8 years ago by leif (previous) (diff)

comment:19 Changed 8 years ago by jhpalmieri

  • Summary changed from doctest failures after moving sage installation to git-related doctest failures after moving sage installation

Does it need to be writeable? Anyway, it looks like we can modify this by setting the environment variable XDG_CACHE_HOME: see the functions _get_cachedir and _get_xdg_cache_dir in sage/local/lib/python/site-packages/matplotlib-1.3.1-py2.7-macosx-10.8-x86_64.egg/matplotlib/__init__.py. I also think this belongs on another ticket.

comment:20 in reply to: ↑ 17 Changed 8 years ago by pbruin

Replying to leif:

IMHO Sage's matplotlib must not put its cache into $HOME (if at all, it should be in $DOT_SAGE/.cache/...), and it should somehow be versioned (with matplotlib's and/or Sage's version in the folder name).

In fact that is where it used to be: I have an old directory ~/.sage/matplotlib-1.2.1/ lying around, containing fontList.cache and tex.cache/, just like ~/.cache/matplotlib/ looks now.

comment:21 follow-ups: Changed 8 years ago by jhpalmieri

By default, Sage's matplotlib should write to .sage/matplotlib-1.3.1. Do you have any environment variables set, like MPLCONFIGDIR? On my OS X machine, I just checked: MPLCONFIGDIR (which Sage sets to .sage/matplotlib-1.3.1 if it's unset) still gets fontList.cache and tex.cache/ added to it when you call plot from Sage. My $HOME/.cache directory only contains fontconfig/.

Last edited 8 years ago by jhpalmieri (previous) (diff)

comment:22 in reply to: ↑ 21 ; follow-up: Changed 8 years ago by leif

Replying to jhpalmieri:

By default, Sage's matplotlib should write to .sage/matplotlib-1.3.1. Do you have any environment variables set, like MPLCONFIGDIR? On my OS X machine, I just checked: MPLCONFIGDIR (which Sage sets to .sage/matplotlib-1.3.1 if it's unset) still gets fontList.cache and tex.cache/ added to it when you call plot from Sage. My $HOME/.cache directory only contains fontconfig/.

Well, setting up MPL's config dir based on its version in sage-env in turn can indeed break upon relocation... :-)

(as it calls Python before sage-location is called, and the Python interpreter may try to load libraries that have moved, as I already mentioned elsewhere).

comment:23 in reply to: ↑ 22 Changed 8 years ago by leif

Replying to leif:

Well, setting up MPL's config dir based on its version in sage-env in turn can indeed break upon relocation... :-)

(as it calls Python before sage-location is called, and the Python interpreter may try to load libraries that have moved, as I already mentioned elsewhere).

See for example http://trac.sagemath.org/ticket/16044#comment:7 ff.

(Note that IIRC sage-location currently wouldn't help there anyway, as it doesn't mess with install_names.)

comment:24 in reply to: ↑ 21 ; follow-up: Changed 8 years ago by pbruin

Replying to jhpalmieri:

By default, Sage's matplotlib should write to .sage/matplotlib-1.3.1. Do you have any environment variables set, like MPLCONFIGDIR? On my OS X machine, I just checked: MPLCONFIGDIR (which Sage sets to .sage/matplotlib-1.3.1 if it's unset) still gets fontList.cache and tex.cache/ added to it when you call plot from Sage. My $HOME/.cache directory only contains fontconfig/.

On GNU/Linux x86_64, I don't seem to have any environment variables with names containing "MPL", neither in the normal shell nor in sage -sh. This is despite `logs/pkgs/matplotlib-1.3.1.log saying

When Sage runs, MPLCONFIGDIR will be set to '$DOT_SAGE/matplotlib-1.3.1'.

comment:25 in reply to: ↑ 24 ; follow-up: Changed 8 years ago by leif

Replying to pbruin:

On GNU/Linux x86_64, I don't seem to have any environment variables with names containing "MPL", neither in the normal shell nor in sage -sh.

True. B)

$ ./sage --sh -c 'env | grep MPL'; uname -sm
Linux x86_64

comment:26 in reply to: ↑ 25 ; follow-up: Changed 8 years ago by pbruin

Replying to leif:

Replying to pbruin:

On GNU/Linux x86_64, I don't seem to have any environment variables with names containing "MPL", neither in the normal shell nor in sage -sh.

True. B)

It turns out that this is because the following command in sage-env fails (I removed the 2>/dev/null at the end:

"$SAGE_ROOT/local/bin/python" -c 'import pkg_resources; pkg_resources.get_distribution("matplotlib").version'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
zipimport.ZipImportError: can't decompress data; zlib not available

comment:27 in reply to: ↑ 26 Changed 8 years ago by leif

Replying to pbruin:

Replying to leif:

Replying to pbruin:

On GNU/Linux x86_64, I don't seem to have any environment variables with names containing "MPL", neither in the normal shell nor in sage -sh.

True. B)

It turns out that this is because the following command in sage-env fails (I removed the 2>/dev/null at the end:

"$SAGE_ROOT/local/bin/python" -c 'import pkg_resources; pkg_resources.get_distribution("matplotlib").version'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
zipimport.ZipImportError: can't decompress data; zlib not available

I'm getting

.../local/bin/python: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory

So apparently LD_LIBRARY_PATH isn't set up properly to that time.

comment:28 Changed 8 years ago by pbruin

The problem in comment:26 was on 6.2.rc0; with 6.2 on a different system I get

Traceback (most recent call last):
...
ImportError: cannot import name MAXREPEAT

I guess the MPLCONFIGDIR-relevant part of sage-env should be moved down to a point where all required setting up has been done.

comment:29 Changed 8 years ago by leif

Oh, just noticed:

# Use a matplotlib config directory specific to Sage and specific to
# the version number of matplotlib, by setting the environment
# variable MPLCONFIGDIR. Note that we can't find the version number by
# importing matplotlib, because that could create matplotlib's standard
# config directory. So we use pkg_resources.
"$SAGE_ROOT/local/bin/python" -c 'import pkg_resources; pkg_resources.get_distribution("matplotlib").version' 2>/dev/null
if [ $? -eq 0 ]; then
    MPLVERSION=`"$SAGE_ROOT/local/bin/python" -c 'import pkg_resources; print pkg_resources.get_distribution("matplotlib").version'`
    MPLCONFIGDIR="$DOT_SAGE/matplotlib-$MPLVERSION"
    export MPLCONFIGDIR
    # The directory is created when Sage starts (see sage.misc.misc).
fi

# Add some directories to $LD_LIBRARY_PATH:
# * lib/openmpi is needed for openmpi.
# * lib/R/lib is needed for R in case the Sage install is moved.
# * lib32 and lib64 are needed for GCC, see #12405.
for d in lib/openmpi lib/R/lib lib32 lib64 lib; do
    libdir="$SAGE_LOCAL/$d"
    # Add only existing directories
    if [ -d "$libdir" ]; then
        [ -z "$LD_LIBRARY_PATH" ] || LD_LIBRARY_PATH=":${LD_LIBRARY_PATH}"
        LD_LIBRARY_PATH="${libdir}$LD_LIBRARY_PATH"
    fi
done
export LD_LIBRARY_PATH

>8-O

Last edited 8 years ago by leif (previous) (diff)

comment:30 Changed 8 years ago by leif

(I meant to say it's done right after we try to call python; I've slightly edited the snippet.)

comment:31 Changed 8 years ago by pbruin

I created #16305 for this.

comment:32 Changed 8 years ago by vbraun

  • Branch changed from u/jhpalmieri/git-relocate to 782e3f6cc0a3852dbb83e26d1f1e41563821a8e5
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.