#6235 closed defect (fixed)
set MPLCONFIGDIR environment variable when Sage starts up
Reported by: | was | Owned by: | cwitty |
---|---|---|---|
Priority: | blocker | Milestone: | sage-4.6 |
Component: | misc | Keywords: | |
Cc: | leif, mpatel, justin | Merged in: | sage-4.6.rc0 |
Authors: | John Palmieri | Reviewers: | Leif Leonhardy |
Report Upstream: | N/A | Work issues: | |
Branch: | Commit: | ||
Dependencies: | Stopgaps: |
Description
Attachments (1)
Change History (38)
comment:1 Changed 11 years ago by
comment:2 Changed 10 years ago by
- Report Upstream set to N/A
comment:3 Changed 10 years ago by
Problems are cropping up at #9122 dealing with configuration files and font caches, etc. Any comment on setting MPLCONFIGDIR to be something inside of the Sage hierarchy, maybe SAGE_LOCAL/etc/matplotlib? For example, the font caches should point to the matplotlib directory in the current Sage being run, but this would not happen usually with multiple Sage instances laying around (or a system matplotlib install) and a global MPLCONFIGDIR (even if that global directory was in the .sage directory).
comment:4 Changed 10 years ago by
- Cc leif added
comment:5 follow-up: ↓ 6 Changed 10 years ago by
comment:6 in reply to: ↑ 5 ; follow-up: ↓ 7 Changed 10 years ago by
Replying to jason:
The problems in #9221 (I mistakenly said #9122 above) were fixed with a patch to the matplotlib spkg that has been applied upstream as well. The issue on this ticket still stands, but it is not urgent for #9221 anymore.
I just noticed that some (I don't know yet which, i.e. the one from alpha2 or alpha3) matplotlib 1.0.0 spkg broke all previous Sage installations, due to ~/.matplotlib/
...
(And especially made testing #9896 totally useless. :( )
comment:7 in reply to: ↑ 6 ; follow-up: ↓ 8 Changed 10 years ago by
Replying to leif:
I just noticed that some (I don't know yet which, i.e. the one from alpha2 or alpha3) matplotlib 1.0.0 spkg broke all previous Sage installations, due to
~/.matplotlib/
...
Can you elaborate? What do you mean "due to ~/.matplotlib/
"?
comment:8 in reply to: ↑ 7 ; follow-up: ↓ 9 Changed 10 years ago by
Replying to jason:
Replying to leif:
I just noticed that some (I don't know yet which, i.e. the one from alpha2 or alpha3) matplotlib 1.0.0 spkg broke all previous Sage installations, due to
~/.matplotlib/
...Can you elaborate? What do you mean "due to
~/.matplotlib/
"?
Without deleting ~/.matplotlib/
, I get the doctest errors in sage/plot/
mentioned here.
Also, I did get compilation errors related to freetype like you reported.
comment:9 in reply to: ↑ 8 Changed 10 years ago by
Replying to leif:
Also, I did get compilation errors related to freetype like you reported.
(Even rebuilding Sage 4.5.3 from scratch failed.)
comment:10 follow-up: ↓ 11 Changed 10 years ago by
Can we just do
MPLCONFIGDIR="$DOT_SAGE/matplotlib" export MPLCONFIGDIR
in sage-env? I suppose we could add yet another environment variable, like SAGE_MPLCONFIGDIR, and do
if [ "$SAGE_MPLCONFIGDIR" = "" ]; then MPLCONFIGDIR=$DOT_SAGE/matplotlib else MPLCONFIGDIR=$SAGE_MPLCONFIGDIR fi export MPLCONFIGDIR
but I don't think that's necessary. We already have too many environment variables. I suppose we could test whether MPLCONFIGDIR is set, and if so, print a warning (once) that Sage is not using the user's setting of this variable. I'm not sure where to test this, though.
If we export our setting for MPLCONFIGDIR, then we also need to document it, probably in the installation guide, saying that Sage uses its own matplotlib config directory, not the default one or whatever the user may have set.
comment:11 in reply to: ↑ 10 ; follow-up: ↓ 12 Changed 10 years ago by
Replying to jhpalmieri:
Can we just do
MPLCONFIGDIR="$DOT_SAGE/matplotlib" export MPLCONFIGDIR
in sage-env?
As mentioned above, the configuration should be inside a specific Sage installation hierarchy, not just yet another user-specific directory, which is (usually) the same for all Sage installations.
I don't understand why the matplotlib developers broke compatibility with older versions; I think it's likely to have different MPL installations included in other software packages, but I might be wrong. IMHO bad design anyway; also the exceptions raised are odd.
comment:12 in reply to: ↑ 11 ; follow-up: ↓ 13 Changed 10 years ago by
Replying to leif:
Replying to jhpalmieri:
Can we just do
MPLCONFIGDIR="$DOT_SAGE/matplotlib" export MPLCONFIGDIRin sage-env?
As mentioned above, the configuration should be inside a specific Sage installation hierarchy, not just yet another user-specific directory, which is (usually) the same for all Sage installations.
At the same time, this directory should not be inside a specific Sage installation (i.e., below SAGE_ROOT) since that means system-wide installs can't have individual customizations, and it also breaks system-wide font cache generation (i.e., matplotlib assumes that a user can update the font cache file, I believe). So where are we now? Separate matplotlib config directories for each version of Sage inside of the .sage directory?
I don't understand why the matplotlib developers broke compatibility with older versions; I think it's likely to have different MPL installations included in other software packages, but I might be wrong. IMHO bad design anyway; also the exceptions raised are odd.
comment:13 in reply to: ↑ 12 ; follow-up: ↓ 14 Changed 10 years ago by
Replying to jason:
At the same time, this directory should not be inside a specific Sage installation (i.e., below SAGE_ROOT) since that means system-wide installs can't have individual customizations, and it also breaks system-wide font cache generation (i.e., matplotlib assumes that a user can update the font cache file, I believe). So where are we now? Separate matplotlib config directories for each version of Sage inside of the .sage directory?
I was thinking of that, too. Not that easy, though. (E.g. using the Sage version as an "index" isn't reliable either.)
Is the font cache the only problem? If so, we could just delete it upon every Sage [script] start-up... (quite ugly, of course)
But older versions of MPL should simply recognize config files from newer versions and e.g. (partially) ignore them. Or at least print an appropriate error message. (Not very pythonic, I know. *SCNR*)
comment:14 in reply to: ↑ 13 Changed 10 years ago by
Replying to leif:
Replying to jason:
At the same time, this directory should not be inside a specific Sage installation (i.e., below SAGE_ROOT) since that means system-wide installs can't have individual customizations, and it also breaks system-wide font cache generation (i.e., matplotlib assumes that a user can update the font cache file, I believe). So where are we now? Separate matplotlib config directories for each version of Sage inside of the .sage directory?
I was thinking of that, too. Not that easy, though. (E.g. using the Sage version as an "index" isn't reliable either.)
Is the font cache the only problem? If so, we could just delete it upon every Sage [script] start-up... (quite ugly, of course)
I believe (off the top of my head) that the error " TypeError?: coercing to Unicode: need string or buffer, dict found" comes from the newer matplotlib including some stix fonts, and so it updates the font cache file to include those files. However, older versions of matplotlib did not deal gracefully with font cache files that referred to nonexistant directories. So if you install the new Sage, then matplotlib updates the font cache file to include the new fonts in the new matplotlib, then you move the new Sage install, the old Sage install will probably die when trying to open the nonexistant new font. Of course, matplotlib should just silently regenerate the cache file, and that is what the bugfix in the 1.0.0 spkg is.
comment:15 follow-up: ↓ 18 Changed 10 years ago by
How about separate matplotlib config directories for each version of matplotlib? We could read the version from the file SAGE_ROOT/local/lib/python/site-packages/matplotlib/__init__.py
-- search for "__version__ = ..."
. If the file matplotlib/__init__.py
does not exist, then matplotlib hasn't been installed yet, so we don't care what we set MPLCONFIGDIR to, but if it exists, we can set MPLCONFIGDIR to something like "$DOT_SAGE/matplotlib-$VER".
(We could instead look at the name of the file SAGE_ROOT/local/lib/python/site-packages/matplotlib-VER-py2.6.egg-info, although if we upgrade, there could be several of these files present, and this doesn't seem as safe to me.)
comment:16 Changed 10 years ago by
Of course, by the reasoning in my previous post, simply moving an old sage version directory should have caused the same problem. So I guess my explanation doesn't seem right anymore.
comment:17 Changed 10 years ago by
In case my idea works, when you install the matplotlib spkg, does it need to know the value of MPLCONFIGDIR, or is it safe to set that only after matplotlib has been installed?
comment:18 in reply to: ↑ 15 ; follow-up: ↓ 19 Changed 10 years ago by
Replying to jhpalmieri:
How about separate matplotlib config directories for each version of matplotlib? [...] we can set
MPLCONFIGDIR
to something like$DOT_SAGE/matplotlib-$VER
.
Sounds like a good idea to me. (You should suggest similar upstream; they could read MPLCONFIGDIR
but write incompatible things to a versioned subdirectory of that.)
In case my idea works, when you install the matplotlib spkg, does it need to know the value of
MPLCONFIGDIR
, or is it safe to set that only after matplotlib has been installed?
I'm not sure if MPL writes anything to that during installation; it's perhaps sufficient to set MPLCONFIGDIR before using MPL (i.e., after installation) to fix the TypeError
issue with parallel installations of older versions.
comment:19 in reply to: ↑ 18 Changed 10 years ago by
Replying to leif:
I'm not sure if MPL writes anything to that during installation; it's perhaps sufficient to set MPLCONFIGDIR before using MPL (i.e., after installation) to fix the
TypeError
issue with parallel installations of older versions.
At least our current 1.0.0 doesn't write to / create $HOME/.matplotlib/
during installation.
(I simply renamed the directory and did ./sage -f matplotlib-1.0.0
.)
Doing
sage: import matplotlib
recreates the directory ($HOME/.matplotlib/
).
comment:20 follow-up: ↓ 21 Changed 10 years ago by
:-) Try:
$ export MPLCONFIGDIR=/some/non-existent/dir/ && ./sage -c "import matplotlib"
(The trailing slash doesn't matter. Also, using $HOME/non-existent/
doesn't make a difference.)
comment:21 in reply to: ↑ 20 Changed 10 years ago by
Replying to leif:
:-) Try:
$ export MPLCONFIGDIR=/some/non-existent/dir/ && ./sage -c "import matplotlib"
(The trailing slash doesn't matter. Also, using
$HOME/non-existent/
doesn't make a difference.)
Cool.
comment:22 Changed 10 years ago by
- Status changed from new to needs_review
Here's a patch. The "sed" business could be done more efficiently by someone who actually knows how to use sed. You can consider this a first draft if you want, but I'm marking it as ready for review.
comment:23 Changed 10 years ago by
- Cc mpatel added
comment:24 Changed 10 years ago by
Replying to jhpalmieri:
Here's a patch. The "sed" business could be done more efficiently by someone who actually knows how to use sed. You can consider this a first draft if you want, but I'm marking it as ready for review.
... MPLVER=`sed -n "/^__version__[ ]*=[ ]*'[^']*'/s/[^']*'\([^']*\)'.*$/\1/p" "$SAGE_LOCAL"/lib/python/site-packages/matplotlib/__init__.py` # Or just (if we check the result): # MPLVER=`sed -n "/^__version__[ ]*=/s/[^']*'\([^']*\)'.*$/\1/p" "$SAGE_LOCAL"/lib/python/site-packages/matplotlib/__init__.py` # Hopefully they never switch to double quotes... ... $MKDIR -p "$MPLCONFIGDIR" # better quote the dir
Perhaps also check that "$MPLVER"
is non-empty.
comment:25 Changed 10 years ago by
More funny:
eval `sed -n "/^__version__[ ]*=/s/ //gp" "$SAGE_LOCAL"/lib/python/site-packages/matplotlib/__init__.py` MPLVER=$__version__
comment:26 Changed 10 years ago by
Here's a new patch using leif's less funny version. It also does not set MPLCONFIGDIR if the file matplotlib/__init__.py
is not found, partly because I don't want to have to create $DOT_SAGE/matplotlib early in the installation process. One small drawback to the current approach is that perhaps during an upgrade from a version of Sage using matplotlib-0.99 to a version using matplotlib-1.0.0, the directory $DOT_SAGE/matplotlib-0.99 will be created first but will remain empty and will never be used. But I seem to have various subdirectories in $DOT_SAGE which I never pay attention to, so having one more doesn't seem like a big deal.
comment:27 follow-up: ↓ 28 Changed 10 years ago by
s/MPLCONFIGIDIR/MPLCONFIGDIR/ (minor; in the comment)
In principle, you can now omit the outer test (if [ -f ... ]; then
) and simply redirect stderr to /dev/null
inside the backquotes.
According to Dave, we no longer use variables for simple (POSIX) commands like mkdir
and cp
etc., but I don't mind.
comment:28 in reply to: ↑ 27 Changed 10 years ago by
Replying to leif:
s/MPLCONFIGIDIR/MPLCONFIGDIR/ (minor; in the comment)
Fixed.
In principle, you can now omit the outer test (
if [ -f ... ]; then
) and simply redirect stderr to/dev/null
inside the backquotes.
Okay, but it also works this way, and seems readable to me this way, so I'm leaving it as is.
According to Dave, we no longer use variables for simple (POSIX) commands like
mkdir
andcp
etc., but I don't mind.
You're right, the scripts in local/bin use "mkdir", not "$MKDIR", so I've changed that, too.
comment:29 Changed 10 years ago by
Ok, "dry" positive review. (Not yet tested, but should work and fix the issue).
(Using also the second minor version number is certainly an overkill though.)
comment:30 Changed 10 years ago by
- Priority changed from minor to critical
- Reviewers set to Leif Leonhardy
- Status changed from needs_review to positive_review
Tested with Sage 4.6.alpha3.
Also works after deletion of $HOME/.sage/
(i.e., dirs get properly recreated s.t. MPL doesn't raise an error).
Replying to jason:
The problems in #9221 (I mistakenly said #9122 above) were fixed with a patch to the matplotlib spkg that has been applied upstream as well. The issue on this ticket still stands, but it is not urgent for #9221 anymore.
Since this fixes MPL 1.0.0 (#9221) breaking other, older Sage installations, I'm increasing the priority.
comment:31 Changed 10 years ago by
- Priority changed from critical to blocker
comment:32 Changed 10 years ago by
- Merged in set to sage-4.6.rc0
- Resolution set to fixed
- Status changed from positive_review to closed
comment:33 Changed 10 years ago by
See #10154 for a follow-up: documenting Sage's use of MPLCONFIGDIR.
comment:34 Changed 10 years ago by
Justin Walker reports a doctest failure in doc/en/constructions/plotting.rst
:
sage -t -long -force_lib devel/sage/doc/en/constructions/plotting.rst ********************************************************************** File "/Users/Sage/sage-4.6.alpha0/devel/sage-main/doc/en/constructions/plotting.rst", line 42: sage: f.plot() Exception raised: Traceback (most recent call last): File "/Users/Sage/sage-4.6.alpha0/local/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/Users/Sage/sage-4.6.alpha0/local/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/Users/Sage/sage-4.6.alpha0/local/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_0[7]>", line 1, in <module> f.plot()###line 42: sage: f.plot() File "/Users/Sage/sage-4.6.alpha0/local/lib/python/site-packages/sage/misc/displayhook.py", line 174, in displayhook print_obj(sys.stdout, obj) File "/Users/Sage/sage-4.6.alpha0/local/lib/python/site-packages/sage/misc/displayhook.py", line 142, in print_obj print >>out_stream, `obj` File "sage_object.pyx", line 101, in sage.structure.sage_object.SageObject.__repr__ (sage/structure/sage_object.c:1341) File "/Users/Sage/sage-4.6.alpha0/local/lib/python/site-packages/sage/plot/plot.py", line 1080, in _repr_ self.show() File "/Users/Sage/sage-4.6.alpha0/local/lib/python/site-packages/sage/plot/misc.py", line 84, in wrapper return func(*args, **kwds) File "/Users/Sage/sage-4.6.alpha0/local/lib/python/site-packages/sage/plot/plot.py", line 1727, in show self.save(DOCTEST_MODE_FILE, **options) File "/Users/Sage/sage-4.6.alpha0/local/lib/python/site-packages/sage/plot/plot.py", line 2388, in save figure=self.matplotlib(*args, **kwds) File "/Users/Sage/sage-4.6.alpha0/local/lib/python/site-packages/sage/plot/plot.py", line 1927, in matplotlib from matplotlib.figure import Figure, figaspect File "/Users/Sage/sage-4.6.alpha0/local/lib/python/site-packages/matplotlib/figure.py", line 18, in <module> from axes import Axes, SubplotBase, subplot_class_factory File "/Users/Sage/sage-4.6.alpha0/local/lib/python/site-packages/matplotlib/axes.py", line 18, in <module> import matplotlib.contour as mcontour File "/Users/Sage/sage-4.6.alpha0/local/lib/python/site-packages/matplotlib/contour.py", line 21, in <module> import matplotlib.texmanager as texmanager File "/Users/Sage/sage-4.6.alpha0/local/lib/python/site-packages/matplotlib/texmanager.py", line 72, in <module> class TexManager: File "/Users/Sage/sage-4.6.alpha0/local/lib/python/site-packages/matplotlib/texmanager.py", line 92, in TexManager os.mkdir(texcache) OSError: [Errno 17] File exists: '/Users/justin/.sage//matplotlib-1.0.0/tex.cache'
comment:35 follow-up: ↓ 36 Changed 10 years ago by
- Cc justin added
I wonder if Justin's error occured because matplotlib tried to create tex.cache
in two or more "simultaneous" test processes.
comment:36 in reply to: ↑ 35 Changed 10 years ago by
Replying to mpatel:
I wonder if Justin's error occured because matplotlib tried to create
tex.cache
in two or more "simultaneous" test processes.
That's just what I posted to sage-release. The relevant lines in matplotlib/texmanager.py are
if not os.path.exists(texcache): os.mkdir(texcache)
so that seems a likely explanation.
comment:37 Changed 10 years ago by
See #10159 for a followup, dealing with the race condition.