sage-4.7 gives error on relocating the root directory

[ppurka]

I installed sage-4.7 in a directory as ~/tmp/Downloadz/sage/sage-4.7 and I could run sage by doing cd ~/tmp/Downloadz/sage/sage-4.7; ./sage -n

A day ago, I decided to move the directory into a more permanent location (from ~/tmp :)) and moved the whole folder by mv ~/tmp/Downloadz/sage/sage-4.7 ~/Installations/.. Now, on trying to run sage, I get the following errors:

First, for the notebook:

~/Installations/sage-4.7> ./sage -n
----------------------------------------------------------------------
| Sage Version 4.7, Release Date: 2011-05-23                         |
| Type notebook() for the GUI, and license() for information.        |
----------------------------------------------------------------------

Please wait while the Sage Notebook server starts...
Traceback (most recent call last):
  File "/home/ppurka/Installations/sage-4.7/local/bin/sage-notebook", line 9, in <module>
    from sage.server.notebook.all import notebook
  File "/home/ppurka/Installations/sage-4.7/local/lib/python2.6/site-packages/sage/server/notebook/all.py", line 22, in <module>
    from sagenb.notebook.all import *
ImportError: No module named sagenb.notebook.all

For sage itself:

...allations/sage-4.7 [130] > ./sage
----------------------------------------------------------------------
| Sage Version 4.7, Release Date: 2011-05-23                         |
| Type notebook() for the GUI, and license() for information.        |
----------------------------------------------------------------------
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)

/home/ppurka/Installations/sage-4.7/local/bin/<string> in <module>()

/home/ppurka/Installations/sage-4.7/local/lib/python2.6/site-packages/sage/misc/preparser_ipython.py in <module>()
      6 ###########################################################################

      7 
----> 8 import sage.misc.interpreter
      9 
     10 import preparser

/home/ppurka/Installations/sage-4.7/local/lib/python2.6/site-packages/sage/misc/interpreter.py in <module>()
    100 
    101 import os
--> 102 import log
    103 import re
    104 

/home/ppurka/Installations/sage-4.7/local/lib/python2.6/site-packages/sage/misc/log.py in <module>()
     63 
     64 import interpreter
---> 65 import latex
     66 import misc
     67 

/home/ppurka/Installations/sage-4.7/local/lib/python2.6/site-packages/sage/misc/latex.py in <module>()
     39 import subprocess
     40 
---> 41 from misc import tmp_dir, graphics_filename
     42 import sage_eval
     43 from sage.misc.misc import SAGE_DOC

/home/ppurka/Installations/sage-4.7/local/lib/python2.6/site-packages/sage/misc/misc.py in <module>()
   2104     return "0"*(size-len(str(s))) + str(s)
   2105 
-> 2106 import sage.server.support
   2107 
   2108 def embedded():

/home/ppurka/Installations/sage-4.7/local/lib/python2.6/site-packages/sage/server/support.py in <module>()
     17 import sage.misc.pager
     18 
---> 19 import sage.misc.sagedoc as sagedoc
     20 import sage.misc.sageinspect as sageinspect
     21 

/home/ppurka/Installations/sage-4.7/local/lib/python2.6/site-packages/sage/misc/sagedoc.py in <module>()
     26 from sage.misc.viewer import browser
     27 from sage.misc.misc import SAGE_DOC, tmp_dir
---> 28 from sagenb.misc.sphinxify import sphinxify
     29 import sage.version
     30 

ImportError: No module named sagenb.misc.sphinxify
WARNING: Failure executing code: 'import sage.misc.preparser_ipython;  sage.misc.preparser_ipython.magma_colon_equals=True'
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)

/home/ppurka/Installations/sage-4.7/local/lib/python2.6/site-packages/IPython/ipmaker.pyc in force_import(modname)
     64         reload(sys.modules[modname])
     65     else:
---> 66         __import__(modname)
     67 
     68 

/home/ppurka/Installations/sage-4.7/local/bin/ipy_profile_sage.py in <module>()
      1 import os
      2 if 'SAGE_CLEAN' not in os.environ:
----> 3     import sage.misc.misc
      4     from sage.misc.interpreter import preparser, _ip
      5     preparser(True)

/home/ppurka/Installations/sage-4.7/local/lib/python2.6/site-packages/sage/misc/misc.py in <module>()
   2104     return "0"*(size-len(str(s))) + str(s)
   2105 
-> 2106 import sage.server.support
   2107 
   2108 def embedded():

/home/ppurka/Installations/sage-4.7/local/lib/python2.6/site-packages/sage/server/support.py in <module>()
     17 import sage.misc.pager
     18 
---> 19 import sage.misc.sagedoc as sagedoc
     20 import sage.misc.sageinspect as sageinspect
     21 

/home/ppurka/Installations/sage-4.7/local/lib/python2.6/site-packages/sage/misc/sagedoc.py in <module>()
     26 from sage.misc.viewer import browser
     27 from sage.misc.misc import SAGE_DOC, tmp_dir
---> 28 from sagenb.misc.sphinxify import sphinxify
     29 import sage.version
     30 

ImportError: No module named sagenb.misc.sphinxify
Error importing ipy_profile_sage - perhaps you should run %upgrade?
WARNING: Loading of ipy_profile_sage failed.

<ERROR: name 'sage_prompt' is not defined>

grep-ing the sources leads to tons of files which have the earlier path hardcoded into the source files. Just a sample:

~/Installations/sage-4.7> grep -r Downloadz *
<showing only some output here>
Binary file devel/sage/sage/__init__.pyc matches
Binary file devel/sage/sage/misc/package.pyc matches
Binary file devel/sage/sage/misc/lazy_import_cache.pyc matches
Binary file devel/sage/sage/misc/__init__.pyc matches
Binary file local/lib/python2.6/site-packages/speaklater-1.2-py2.6.egg/speaklater.pyc matches

All the errors were fixed as soon as I made a symbolic link as follows:

ln -s ~/Installations/sage-4.7 ~/tmp/Downloadz/sage/.

Since the file ~/Installations/sage-4.7/README.txt claims that sage should be relocatable, I believe this is a relocation bug in sage.

If it matters, my system is a Gentoo Linux amd64 system. I do have a sage-4.7 installation in / installed via the sage-on-gentoo overlay. This installation (the one in ~/Installations) was compiled "manually" by downloading the sage tarball from the website. It was originally compiled inside the ~/tmp/Downloadz/sage directory.

[leif]

Did you notice a message

The Sage install tree may have moved
(from ... to ...)
Changing various hardcoded paths
(please wait at most a few minutes)...
Do not interrupt this.

when you first started ./sage in the new directory (~/Installations/sage-4.7/)?

I'm not sure if (or rather doubt that) Sage will fully work in the presence of another system-wide Sage installation, i.e., if another (different) sage is in your PATH.

The latter might be the problem.

[leif]

P.S.: If it's just the .pyc files, you may cure your installation by simply removing them; they'll get recompiled the next time Python executes their respective .py files.

(I of course don't consider this a solution to the general problem, but it also would be helpful to know whether there are further files which still contain the old location and hinder Sage from starting up without the symbolic link from its original location. AFAIK there'll always remain some "harmless" RPATHs in shared libraries pointing to the old location.)

[ppurka]

Replying to leif:

Did you notice a message

The Sage install tree may have moved
(from ... to ...)
Changing various hardcoded paths
(please wait at most a few minutes)...
Do not interrupt this.

when you first started ./sage in the new directory (~/Installations/sage-4.7/)?

I'm not sure if (or rather doubt that) Sage will fully work in the presence of another system-wide Sage installation, i.e., if another (different) sage is in your PATH.

The latter might be the problem.

Sorry, I don't clearly remember what was the output at the very first run of sage.

Deleting the .pyc files doesn't work. I get the same error as I got earlier. If I have the symlink, then it does work fine, irrespective of the system wide sage installation. I know that it uses the sage (at least the notebook) from the directory ~/Installations, since I am running the rkirov-flask notebook in that directory. But you are right, there might be other things which are getting mixed up. I will simply try recompiling the local sage installation again.

[jhpalmieri]

In the file SAGE_ROOT/local/lib/python/site-packages/easy-install.pth, there should be a line containing "sagenb". Is the path there relative (../../../../devel/sagenb) or absolute? If it's absolute, what happens if you replace it with the relative path I gave?

[leif]

Replying to jhpalmieri:

In the file SAGE_ROOT/local/lib/python/site-packages/easy-install.pth, there should be a line containing "sagenb". Is the path there relative (../../../../devel/sagenb) or absolute? If it's absolute, what happens if you replace it with the relative path I gave?

I know I once fixed this (especially if SAGE_PATH was set), but I don't recall if that sagenb spkg ever finally got merged.

[jhpalmieri]

Leif: I think the issue is the flask notebook. With the "classic" notebook, I don't seem to have this problem, and the spkg-install file tries to make the path relative, but when I tried the ​installation instructions for the flask notebook, I got an absolute path for the notebook (referring to "rkirov-flask" instead of "sagenb"), and it renders my Sage installation non-relocateable.

[leif]

Replying to ppurka:

Replying to leif:

Did you notice a message

The Sage install tree may have moved
(from ... to ...)
Changing various hardcoded paths
(please wait at most a few minutes)...
Do not interrupt this.

when you first started ./sage in the new directory (~/Installations/sage-4.7/)?

I'm not sure if (or rather doubt that) Sage will fully work in the presence of another system-wide Sage installation, i.e., if another (different) sage is in your PATH.

The latter might be the problem.

Sorry, I don't clearly remember what was the output at the very first run of sage.

Well, you can retry by simply renaming the $SAGE_ROOT directory. ;-)

If I have the symlink, then it does work fine, irrespective of the system wide sage installation. I know that it uses the sage (at least the notebook) from the directory ~/Installations, since I am running the rkirov-flask notebook in that directory. But you are right, there might be other things which are getting mixed up. I will simply try recompiling the local sage installation again.

You could just rename your system-wide sage command, in /usr/bin/ or /usr/local/bin/ I guess. (command -v sage should show which one is used when you're outside the separate Sage installation.)

[leif]

Replying to jhpalmieri:

Leif: I think the issue is the flask notebook. With the "classic" notebook, I don't seem to have this problem, and the spkg-install file tries to make the path relative, but when I tried the ​installation instructions for the flask notebook, I got an absolute path for the notebook (referring to "rkirov-flask" instead of "sagenb"), and it renders my Sage installation non-relocateable.

Ah, so it's an upstream problem... :D

[ppurka]

Replying to jhpalmieri:

In the file SAGE_ROOT/local/lib/python/site-packages/easy-install.pth, there should be a line containing "sagenb". Is the path there relative (../../../../devel/sagenb) or absolute? If it's absolute, what happens if you replace it with the relative path I gave?

Excellent! This fixes the relocation problem. Indeed rkirov-flask was written as a full path instead of the relative one. So, this is not a bug in sage relocation per se.

[leif]

Can someone report this upstream, to Rado?

[ppurka]

Replying to leif:

Can someone report this upstream, to Rado?

I will see him tomorrow, so I will tell him :)

[rkirov]

I don't understand python's setuptools, but I haven't hardcoded the path anywhere. My guess is that doing "python setup.py develop" does that. Once flask becomes a proper spkg this should go away (hopefully).

[leif]

Replying to rkirov:

I don't understand python's setuptools, but I haven't hardcoded the path anywhere. My guess is that doing "python setup.py develop" does that. Once flask becomes a proper spkg this should go away (hopefully).

In SageNB, we use

python setup.py develop --egg-path ../../../../devel/sagenb

but that apparently doesn't always work, so I hacked SageNB's spkg-dist which creates its spkg-install:

...
cd "$SAGE_ROOT/local/lib/python/site-packages"
...
if ! grep sagenb easy-install.pth >/dev/null; then
    # Ugly work-around, we haven't found the real cause yet (see #10176):
    echo "No sagenb path found in 'easy-install.pth'"'!'
    echo "Adding relative sagenb path to 'easy-install.pth'..."
    sed -e '$ i \../../../../devel/sagenb' easy-install.pth > easy-install.pth.$$
    if [ $? -ne 0 ]; then
        echo >&2 "Error adding relative sagenb path to 'easy-install.pth'."
        exit 1
    fi
else
    echo "Making sagenb path in 'easy-install.pth' relative..."
    sed 's/^.*sagenb.*$/..\/..\/..\/..\/devel\/sagenb/' easy-install.pth > easy-install.pth.$$
    if [ $? -ne 0 ]; then
        echo >&2 "Error patching 'easy-install.pth' to have relative path to SageNB."
        exit 1
    fi
fi
...
# The following fails only on wrong file permissions etc.:
mv -f easy-install.pth.$$ easy-install.pth
if [ $? -ne 0 ]; then
    echo >&2 "Error overwriting original 'easy-install.pth'."
    exit 1
fi
...

(IIRC the trouble with a completely missing sagenb path was caused by having SAGE_PATH set to ".", but I'm not sure.)

[ppurka]

There is actually absolutely no difference between rkirov-flask/spkg-dist and sagenb-main/spkg-dist. So, this problem will probably "magically go away" once rkirov-flask is actually packaged.

[leif]

Replying to ppurka:

There is actually absolutely no difference between rkirov-flask/spkg-dist and sagenb-main/spkg-dist. So, this problem will probably "magically go away" once rkirov-flask is actually packaged.

Yeah, but Rado could add a script (or just a note on that) to the project page.

---

For the record:

The --egg-path ... was actually introduced at #10097, and only affects the sagenb.egg-link file, therefore Mitesh added replacing the absolute path in easy-install.pth by a relative one.

#10176 and #10192 (the latter patching sage-spkg, and still needing review) dealt with a completely missing sagenb entry in easy-install.pth, due to SAGE_PATH="." (and removed ln -snf, see below).

Note that the -n option of ln -snf isn't defined by POSIX, and -- worse -- shows different behaviour on GNU/Linux and Solaris, so we must not use it. I'd also change the rkirov-flask installation instructions, though currently explicitly for Linux, accordingly.

[ppurka]

This is still a problem with sage-4.7.2 + jasongrout-flask.

[ppurka]

sagenb has long ago been merged. And this ticket is pointless now.