Sage: Ticket #11678: sage-4.7 gives error on relocating the root directory
https://trac.sagemath.org/ticket/11678
<p>
I installed sage-4.7 in a directory as <strong>~/tmp/Downloadz/sage/sage-4.7</strong> and I could run sage by doing <strong>cd ~/tmp/Downloadz/sage/sage-4.7; ./sage -n</strong>
</p>
<p>
A day ago, I decided to move the directory into a more permanent location (from ~/tmp :)) and moved the whole folder by <strong>mv ~/tmp/Downloadz/sage/sage-4.7 ~/Installations/.</strong>. Now, on trying to run sage, I get the following errors:
</p>
<p>
First, for the notebook:
</p>
<pre class="wiki">~/Installations/sage-4.7> ./sage -n
----------------------------------------------------------------------
| Sage Version 4.7, Release Date: 2011-05-23 |
| Type notebook() for the GUI, and license() for information. |
----------------------------------------------------------------------
Please wait while the Sage Notebook server starts...
Traceback (most recent call last):
File "/home/ppurka/Installations/sage-4.7/local/bin/sage-notebook", line 9, in <module>
from sage.server.notebook.all import notebook
File "/home/ppurka/Installations/sage-4.7/local/lib/python2.6/site-packages/sage/server/notebook/all.py", line 22, in <module>
from sagenb.notebook.all import *
ImportError: No module named sagenb.notebook.all
</pre><p>
For sage itself:
</p>
<pre class="wiki">...allations/sage-4.7 [130] > ./sage
----------------------------------------------------------------------
| Sage Version 4.7, Release Date: 2011-05-23 |
| Type notebook() for the GUI, and license() for information. |
----------------------------------------------------------------------
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
/home/ppurka/Installations/sage-4.7/local/bin/<string> in <module>()
/home/ppurka/Installations/sage-4.7/local/lib/python2.6/site-packages/sage/misc/preparser_ipython.py in <module>()
6 ###########################################################################
7
----> 8 import sage.misc.interpreter
9
10 import preparser
/home/ppurka/Installations/sage-4.7/local/lib/python2.6/site-packages/sage/misc/interpreter.py in <module>()
100
101 import os
--> 102 import log
103 import re
104
/home/ppurka/Installations/sage-4.7/local/lib/python2.6/site-packages/sage/misc/log.py in <module>()
63
64 import interpreter
---> 65 import latex
66 import misc
67
/home/ppurka/Installations/sage-4.7/local/lib/python2.6/site-packages/sage/misc/latex.py in <module>()
39 import subprocess
40
---> 41 from misc import tmp_dir, graphics_filename
42 import sage_eval
43 from sage.misc.misc import SAGE_DOC
/home/ppurka/Installations/sage-4.7/local/lib/python2.6/site-packages/sage/misc/misc.py in <module>()
2104 return "0"*(size-len(str(s))) + str(s)
2105
-> 2106 import sage.server.support
2107
2108 def embedded():
/home/ppurka/Installations/sage-4.7/local/lib/python2.6/site-packages/sage/server/support.py in <module>()
17 import sage.misc.pager
18
---> 19 import sage.misc.sagedoc as sagedoc
20 import sage.misc.sageinspect as sageinspect
21
/home/ppurka/Installations/sage-4.7/local/lib/python2.6/site-packages/sage/misc/sagedoc.py in <module>()
26 from sage.misc.viewer import browser
27 from sage.misc.misc import SAGE_DOC, tmp_dir
---> 28 from sagenb.misc.sphinxify import sphinxify
29 import sage.version
30
ImportError: No module named sagenb.misc.sphinxify
WARNING: Failure executing code: 'import sage.misc.preparser_ipython; sage.misc.preparser_ipython.magma_colon_equals=True'
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
/home/ppurka/Installations/sage-4.7/local/lib/python2.6/site-packages/IPython/ipmaker.pyc in force_import(modname)
64 reload(sys.modules[modname])
65 else:
---> 66 __import__(modname)
67
68
/home/ppurka/Installations/sage-4.7/local/bin/ipy_profile_sage.py in <module>()
1 import os
2 if 'SAGE_CLEAN' not in os.environ:
----> 3 import sage.misc.misc
4 from sage.misc.interpreter import preparser, _ip
5 preparser(True)
/home/ppurka/Installations/sage-4.7/local/lib/python2.6/site-packages/sage/misc/misc.py in <module>()
2104 return "0"*(size-len(str(s))) + str(s)
2105
-> 2106 import sage.server.support
2107
2108 def embedded():
/home/ppurka/Installations/sage-4.7/local/lib/python2.6/site-packages/sage/server/support.py in <module>()
17 import sage.misc.pager
18
---> 19 import sage.misc.sagedoc as sagedoc
20 import sage.misc.sageinspect as sageinspect
21
/home/ppurka/Installations/sage-4.7/local/lib/python2.6/site-packages/sage/misc/sagedoc.py in <module>()
26 from sage.misc.viewer import browser
27 from sage.misc.misc import SAGE_DOC, tmp_dir
---> 28 from sagenb.misc.sphinxify import sphinxify
29 import sage.version
30
ImportError: No module named sagenb.misc.sphinxify
Error importing ipy_profile_sage - perhaps you should run %upgrade?
WARNING: Loading of ipy_profile_sage failed.
<ERROR: name 'sage_prompt' is not defined>
</pre><p>
grep-ing the sources leads to tons of files which have the earlier path hardcoded into the source files. Just a sample:
</p>
<pre class="wiki">~/Installations/sage-4.7> grep -r Downloadz *
<showing only some output here>
Binary file devel/sage/sage/__init__.pyc matches
Binary file devel/sage/sage/misc/package.pyc matches
Binary file devel/sage/sage/misc/lazy_import_cache.pyc matches
Binary file devel/sage/sage/misc/__init__.pyc matches
Binary file local/lib/python2.6/site-packages/speaklater-1.2-py2.6.egg/speaklater.pyc matches
</pre><p>
All the errors were fixed as soon as I made a symbolic link as follows:
</p>
<pre class="wiki">ln -s ~/Installations/sage-4.7 ~/tmp/Downloadz/sage/.
</pre><p>
Since the file <strong>~/Installations/sage-4.7/README.txt</strong> claims that sage should be relocatable, I believe this is a relocation bug in sage.
</p>
<p>
If it matters, my system is a Gentoo Linux amd64 system. I do have a sage-4.7 installation in / installed via the sage-on-gentoo overlay. This installation (the one in ~/Installations) was compiled "manually" by downloading the sage tarball from the website. It was originally compiled inside the ~/tmp/Downloadz/sage directory.
</p>
en-usSagehttps://trac.sagemath.org/chrome/site/logo_sagemath_trac.png
https://trac.sagemath.org/ticket/11678
Trac 1.1.6leifThu, 11 Aug 2011 08:43:19 GMTcc set
https://trac.sagemath.org/ticket/11678#comment:1
https://trac.sagemath.org/ticket/11678#comment:1
<ul>
<li><strong>cc</strong>
<em>leif</em> added
</li>
</ul>
TicketleifThu, 11 Aug 2011 09:16:11 GMT
https://trac.sagemath.org/ticket/11678#comment:2
https://trac.sagemath.org/ticket/11678#comment:2
<p>
Did you notice a message
</p>
<pre class="wiki">The Sage install tree may have moved
(from ... to ...)
Changing various hardcoded paths
(please wait at most a few minutes)...
Do not interrupt this.
</pre><p>
when you first started <code>./sage</code> in the new directory (<code>~/Installations/sage-4.7/</code>)?
</p>
<p>
<br />
</p>
<p>
I'm not sure if (or rather doubt that) Sage will fully work in the presence of another system-wide Sage installation, i.e., if another (different) <code>sage</code> is in your <code>PATH</code>.
</p>
<p>
The latter <em>might</em> be the problem.
</p>
TicketleifThu, 11 Aug 2011 09:31:10 GMT
https://trac.sagemath.org/ticket/11678#comment:3
https://trac.sagemath.org/ticket/11678#comment:3
<p>
P.S.: If it's just the <code>.pyc</code> files, you may cure your installation by simply removing them; they'll get recompiled the next time Python executes their respective <code>.py</code> files.
</p>
<p>
(I of course don't consider this a solution to the general problem, but it also would be helpful to know whether there are further files which still contain the old location <em>and</em> hinder Sage from starting up without the symbolic link from its original location. AFAIK there'll always remain some "harmless" <code>RPATH</code>s in shared libraries pointing to the old location.)
</p>
TicketppurkaThu, 11 Aug 2011 13:06:57 GMT
https://trac.sagemath.org/ticket/11678#comment:4
https://trac.sagemath.org/ticket/11678#comment:4
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/11678#comment:2" title="Comment 2">leif</a>:
</p>
<blockquote class="citation">
<p>
Did you notice a message
</p>
<pre class="wiki">The Sage install tree may have moved
(from ... to ...)
Changing various hardcoded paths
(please wait at most a few minutes)...
Do not interrupt this.
</pre><p>
when you first started <code>./sage</code> in the new directory (<code>~/Installations/sage-4.7/</code>)?
</p>
<p>
<br />
</p>
<p>
I'm not sure if (or rather doubt that) Sage will fully work in the presence of another system-wide Sage installation, i.e., if another (different) <code>sage</code> is in your <code>PATH</code>.
</p>
<p>
The latter <em>might</em> be the problem.
</p>
</blockquote>
<p>
Sorry, I don't clearly remember what was the output at the very first run of sage.
</p>
<p>
Deleting the .pyc files doesn't work. I get the same error as I got earlier. If I have the symlink, then it does work fine, irrespective of the system wide sage installation. I know that it uses the sage (at least the notebook) from the directory ~/Installations, since I am running the rkirov-flask notebook in that directory. But you are right, there might be other things which are getting mixed up. I will simply try recompiling the local sage installation again.
</p>
TicketjhpalmieriThu, 11 Aug 2011 16:14:04 GMT
https://trac.sagemath.org/ticket/11678#comment:5
https://trac.sagemath.org/ticket/11678#comment:5
<p>
In the file <code>SAGE_ROOT/local/lib/python/site-packages/easy-install.pth</code>, there should be a line containing "sagenb". Is the path there relative (<code>../../../../devel/sagenb</code>) or absolute? If it's absolute, what happens if you replace it with the relative path I gave?
</p>
TicketleifThu, 11 Aug 2011 16:20:44 GMT
https://trac.sagemath.org/ticket/11678#comment:6
https://trac.sagemath.org/ticket/11678#comment:6
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/11678#comment:5" title="Comment 5">jhpalmieri</a>:
</p>
<blockquote class="citation">
<p>
In the file <code>SAGE_ROOT/local/lib/python/site-packages/easy-install.pth</code>, there should be a line containing "sagenb". Is the path there relative (<code>../../../../devel/sagenb</code>) or absolute? If it's absolute, what happens if you replace it with the relative path I gave?
</p>
</blockquote>
<p>
I know I once fixed this (especially if <code>SAGE_PATH</code> was set), but I don't recall if that sagenb spkg ever finally got merged.
</p>
TicketjhpalmieriThu, 11 Aug 2011 16:26:51 GMT
https://trac.sagemath.org/ticket/11678#comment:7
https://trac.sagemath.org/ticket/11678#comment:7
<p>
Leif: I think the issue is the flask notebook. With the "classic" notebook, I don't seem to have this problem, and the spkg-install file tries to make the path relative, but when I tried the <a class="ext-link" href="http://code.google.com/r/rkirov-flask/"><span class="icon"></span>installation instructions</a> for the flask notebook, I got an absolute path for the notebook (referring to "rkirov-flask" instead of "sagenb"), and it renders my Sage installation non-relocateable.
</p>
TicketleifThu, 11 Aug 2011 16:28:05 GMT
https://trac.sagemath.org/ticket/11678#comment:8
https://trac.sagemath.org/ticket/11678#comment:8
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/11678#comment:4" title="Comment 4">ppurka</a>:
</p>
<blockquote class="citation">
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/11678#comment:2" title="Comment 2">leif</a>:
</p>
<blockquote class="citation">
<p>
Did you notice a message
</p>
</blockquote>
</blockquote>
<pre class="wiki">The Sage install tree may have moved
(from ... to ...)
Changing various hardcoded paths
(please wait at most a few minutes)...
Do not interrupt this.
</pre><blockquote class="citation">
<blockquote class="citation">
<p>
when you first started <code>./sage</code> in the new directory (<code>~/Installations/sage-4.7/</code>)?
</p>
<p>
<br />
</p>
<p>
I'm not sure if (or rather doubt that) Sage will fully work in the presence of another system-wide Sage installation, i.e., if another (different) <code>sage</code> is in your <code>PATH</code>.
</p>
<p>
The latter <em>might</em> be the problem.
</p>
</blockquote>
<p>
Sorry, I don't clearly remember what was the output at the very first run of sage.
</p>
</blockquote>
<p>
Well, you can retry by simply renaming the <code>$SAGE_ROOT</code> directory. ;-)
</p>
<blockquote class="citation">
<p>
If I have the symlink, then it does work fine, irrespective of the system wide sage installation. I know that it uses the sage (at least the notebook) from the directory ~/Installations, since I am running the rkirov-flask notebook in that directory. But you are right, there might be other things which are getting mixed up. I will simply try recompiling the local sage installation again.
</p>
</blockquote>
<p>
You could just rename your system-wide <code>sage</code> command, in <code>/usr/bin/</code> or <code>/usr/local/bin/</code> I guess. (<code>command -v sage</code> should show which one is used when you're outside the separate Sage installation.)
</p>
TicketleifThu, 11 Aug 2011 16:34:32 GMT
https://trac.sagemath.org/ticket/11678#comment:9
https://trac.sagemath.org/ticket/11678#comment:9
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/11678#comment:7" title="Comment 7">jhpalmieri</a>:
</p>
<blockquote class="citation">
<p>
Leif: I think the issue is the flask notebook. With the "classic" notebook, I don't seem to have this problem, and the spkg-install file tries to make the path relative, but when I tried the <a class="ext-link" href="http://code.google.com/r/rkirov-flask/"><span class="icon"></span>installation instructions</a> for the flask notebook, I got an absolute path for the notebook (referring to "rkirov-flask" instead of "sagenb"), and it renders my Sage installation non-relocateable.
</p>
</blockquote>
<p>
Ah, so it's an upstream problem... :D
</p>
TicketppurkaThu, 11 Aug 2011 16:37:09 GMT
https://trac.sagemath.org/ticket/11678#comment:10
https://trac.sagemath.org/ticket/11678#comment:10
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/11678#comment:5" title="Comment 5">jhpalmieri</a>:
</p>
<blockquote class="citation">
<p>
In the file <code>SAGE_ROOT/local/lib/python/site-packages/easy-install.pth</code>, there should be a line containing "sagenb". Is the path there relative (<code>../../../../devel/sagenb</code>) or absolute? If it's absolute, what happens if you replace it with the relative path I gave?
</p>
</blockquote>
<p>
Excellent! This fixes the relocation problem. Indeed rkirov-flask was written as a full path instead of the relative one. So, this is not a bug in sage relocation per se.
</p>
TicketleifThu, 11 Aug 2011 16:41:05 GMT
https://trac.sagemath.org/ticket/11678#comment:11
https://trac.sagemath.org/ticket/11678#comment:11
<p>
Can someone report this upstream, to Rado?
</p>
TicketppurkaThu, 11 Aug 2011 16:42:11 GMT
https://trac.sagemath.org/ticket/11678#comment:12
https://trac.sagemath.org/ticket/11678#comment:12
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/11678#comment:11" title="Comment 11">leif</a>:
</p>
<blockquote class="citation">
<p>
Can someone report this upstream, to Rado?
</p>
</blockquote>
<p>
I will see him tomorrow, so I will tell him :)
</p>
TicketrkirovThu, 11 Aug 2011 17:44:03 GMT
https://trac.sagemath.org/ticket/11678#comment:13
https://trac.sagemath.org/ticket/11678#comment:13
<p>
I don't understand python's setuptools, but I haven't hardcoded the path anywhere. My guess is that doing "python setup.py develop" does that. Once flask becomes a proper spkg this should go away (hopefully).
</p>
TicketleifThu, 11 Aug 2011 18:31:53 GMTkeywords set
https://trac.sagemath.org/ticket/11678#comment:14
https://trac.sagemath.org/ticket/11678#comment:14
<ul>
<li><strong>keywords</strong>
<em>flask</em> <em>notebook</em> <em>easy-install.pth</em> <em>easy_install</em> added
</li>
</ul>
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/11678#comment:13" title="Comment 13">rkirov</a>:
</p>
<blockquote class="citation">
<p>
I don't understand python's setuptools, but I haven't hardcoded the path anywhere. My guess is that doing "python setup.py develop" does that. Once flask becomes a proper spkg this should go away (hopefully).
</p>
</blockquote>
<p>
In SageNB, we use
</p>
<div class="wiki-code"><div class="code"><pre>python setup.py develop --egg-path ../../../../devel/sagenb
</pre></div></div><p>
but that apparently doesn't always work, so I hacked SageNB's <code>spkg-dist</code> which creates its <code>spkg-install</code>:
</p>
<div class="wiki-code"><div class="code"><pre>...
<span class="nb">cd</span> <span class="s2">"$SAGE_ROOT/local/lib/python/site-packages"</span>
...
<span class="k">if</span> ! grep sagenb easy-install.pth >/dev/null; <span class="k">then</span>
<span class="c"># Ugly work-around, we haven't found the real cause yet (see #10176):
</span> <span class="nb">echo</span> <span class="s2">"No sagenb path found in 'easy-install.pth'"</span><span class="s1">'!'</span>
<span class="nb">echo</span> <span class="s2">"Adding relative sagenb path to 'easy-install.pth'..."</span>
sed -e <span class="s1">'$ i \../../../../devel/sagenb'</span> easy-install.pth > easy-install.pth.<span class="nv">$$</span>
<span class="k">if</span> <span class="o">[</span> <span class="nv">$?</span> -ne 0 <span class="o">]</span>; <span class="k">then
</span><span class="nb">echo</span> >&2 <span class="s2">"Error adding relative sagenb path to 'easy-install.pth'."</span>
<span class="nb">exit </span>1
<span class="k">fi
else
</span><span class="nb">echo</span> <span class="s2">"Making sagenb path in 'easy-install.pth' relative..."</span>
sed <span class="s1">'s/^.*sagenb.*$/..\/..\/..\/..\/devel\/sagenb/'</span> easy-install.pth > easy-install.pth.<span class="nv">$$</span>
<span class="k">if</span> <span class="o">[</span> <span class="nv">$?</span> -ne 0 <span class="o">]</span>; <span class="k">then
</span><span class="nb">echo</span> >&2 <span class="s2">"Error patching 'easy-install.pth' to have relative path to SageNB."</span>
<span class="nb">exit </span>1
<span class="k">fi
fi</span>
...
<span class="c"># The following fails only on wrong file permissions etc.:
</span>mv -f easy-install.pth.<span class="nv">$$</span> easy-install.pth
<span class="k">if</span> <span class="o">[</span> <span class="nv">$?</span> -ne 0 <span class="o">]</span>; <span class="k">then
</span><span class="nb">echo</span> >&2 <span class="s2">"Error overwriting original 'easy-install.pth'."</span>
<span class="nb">exit </span>1
<span class="k">fi</span>
...
</pre></div></div><p>
(IIRC the trouble with <em>a completely missing</em> <code>sagenb</code> path was caused by having <code>SAGE_PATH</code> set to "<code>.</code>", but I'm not sure.)
</p>
TicketppurkaThu, 11 Aug 2011 18:37:09 GMT
https://trac.sagemath.org/ticket/11678#comment:15
https://trac.sagemath.org/ticket/11678#comment:15
<p>
There is actually absolutely no difference between rkirov-flask/spkg-dist and sagenb-main/spkg-dist. So, this problem will probably "magically go away" once rkirov-flask is actually packaged.
</p>
TicketleifThu, 11 Aug 2011 19:42:27 GMT
https://trac.sagemath.org/ticket/11678#comment:16
https://trac.sagemath.org/ticket/11678#comment:16
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/11678#comment:15" title="Comment 15">ppurka</a>:
</p>
<blockquote class="citation">
<p>
There is actually absolutely no difference between rkirov-flask/spkg-dist and sagenb-main/spkg-dist. So, this problem will probably "magically go away" once rkirov-flask is actually packaged.
</p>
</blockquote>
<p>
Yeah, but Rado could add a script (or just a note on that) to the project page.
</p>
<hr />
<p>
For the record:
</p>
<p>
The <code>--egg-path ...</code> was actually introduced at <a class="closed ticket" href="https://trac.sagemath.org/ticket/10097" title="defect: Sage won't start after moving or renaming the 4.6.alpha3 root directory (closed: fixed)">#10097</a>, and only affects the <code>sagenb.egg-link</code> file, therefore Mitesh added replacing the absolute path in <code>easy-install.pth</code> by a relative one.
</p>
<p>
<a class="closed ticket" href="https://trac.sagemath.org/ticket/10176" title="defect: ImportError: No module named sagenb on openSUSE (closed: invalid)">#10176</a> and <a class="closed ticket" href="https://trac.sagemath.org/ticket/10192" title="defect: SageNB broken when SAGE_PATH contains flavours of '.' during installation (closed: fixed)">#10192</a> (the latter patching <code>sage-spkg</code>, and still needing review) dealt with a completely missing <code>sagenb</code> entry in <code>easy-install.pth</code>, due to <code>SAGE_PATH="."</code> (and removed <code>ln -snf</code>, see below).
</p>
<p>
Note that the <code>-n</code> option of <code>ln -snf</code> isn't defined by POSIX, and -- worse -- shows different behaviour on GNU/Linux and Solaris, so we must not use it. I'd also change the <code>rkirov-flask</code> installation instructions, though currently explicitly for Linux, accordingly.
</p>
TicketdimpaseSat, 20 Aug 2011 15:59:29 GMTcc changed
https://trac.sagemath.org/ticket/11678#comment:17
https://trac.sagemath.org/ticket/11678#comment:17
<ul>
<li><strong>cc</strong>
<em>rkirov</em> added
</li>
</ul>
TicketppurkaThu, 01 Dec 2011 12:55:40 GMT
https://trac.sagemath.org/ticket/11678#comment:18
https://trac.sagemath.org/ticket/11678#comment:18
<p>
This is still a problem with sage-4.7.2 + jasongrout-flask.
</p>
TicketppurkaTue, 11 Dec 2012 16:42:37 GMTstatus, milestone changed; reviewer set
https://trac.sagemath.org/ticket/11678#comment:19
https://trac.sagemath.org/ticket/11678#comment:19
<ul>
<li><strong>status</strong>
changed from <em>new</em> to <em>needs_review</em>
</li>
<li><strong>reviewer</strong>
set to <em>Punarbasu Purkayastha</em>
</li>
<li><strong>milestone</strong>
changed from <em>sage-5.6</em> to <em>sage-duplicate/invalid/wontfix</em>
</li>
</ul>
TicketppurkaTue, 11 Dec 2012 16:42:47 GMTstatus changed
https://trac.sagemath.org/ticket/11678#comment:20
https://trac.sagemath.org/ticket/11678#comment:20
<ul>
<li><strong>status</strong>
changed from <em>needs_review</em> to <em>positive_review</em>
</li>
</ul>
TicketppurkaTue, 11 Dec 2012 16:44:12 GMT
https://trac.sagemath.org/ticket/11678#comment:21
https://trac.sagemath.org/ticket/11678#comment:21
<p>
sagenb has long ago been merged. And this ticket is pointless now.
</p>
TicketjdemeyerFri, 21 Dec 2012 22:47:41 GMTstatus changed; resolution set
https://trac.sagemath.org/ticket/11678#comment:22
https://trac.sagemath.org/ticket/11678#comment:22
<ul>
<li><strong>status</strong>
changed from <em>positive_review</em> to <em>closed</em>
</li>
<li><strong>resolution</strong>
set to <em>worksforme</em>
</li>
</ul>
Ticket