Fix doctests and methods that want write access to $SAGE_LOCAL

[mabshoff]

All doctests in Sage should pass when they are run as a user that does not have write access to the Sage install. To do that, set SAGE_TESTDIR to some place writable, for example

export SAGE_TESTDIR=$HOME/tmp

and run the doctests on a Sage install that isn't owned by the user:

$ ./sage -tp 4 -long devel/sage/doc/common devel/sage/doc/de devel/sage/doc/en devel/sage/doc/fr devel/sage/doc/ru devel/sage/sage

Here, we assume that the owner of the Sage install has run Sage at least once (or has applied #11926).

On sage-4.7.2.alpha4, this causes:

The following tests failed:

        sage -t -long devel/sage/sage/modular/hecke/submodule.py # 1 doctests failed
        sage -t -long devel/sage/sage/modular/abvar/abvar.py # 1 doctests failed
        sage -t -long devel/sage/sage/lfunctions/sympow.py # 13 doctests failed
        sage -t -long devel/sage/sage/interfaces/qepcad.py # 3 doctests failed
        sage -t -long devel/sage/sage/schemes/elliptic_curves/ell_rational_field.py # 17 doctests failed

This seems to be because of:

1. Installing an rc file for the experimental package qepcad (in a very odd location $SAGE_ROOT/local/default.qepcadrc)
2. Sympow which needs to create datafiles. This should probably be done in sympow's spkg-install instead. See #11920.

Apply:

1. 5155_scripts.patch to the scripts repository.
2. 5155_qepcad.patch to the Sage library.

[jhpalmieri]

Before applying the patch, I get two doctest failures (with a non-writeable Sage install). Applying the patch "trac_5155.patch" fixes both. So positive review for that.

The scripts patch wasn't necessary to get those tests to pass, and it also doesn't apply cleanly to 4.3.1:

applying /Users/palmieri/Downloads/scripts_5155.patch
patching file sage-doctest
Hunk #1 FAILED at 55
1 out of 1 hunks FAILED -- saving rejects to file sage-doctest.rej
unable to find 'sage-dsage-trial' for patching
1 out of 1 hunks FAILED -- saving rejects to file sage-dsage-trial.rej
patching file sage-maketest
Hunk #1 succeeded at 1 with fuzz 1 (offset -1 lines).
patching file sage-test
Hunk #1 FAILED at 40
1 out of 1 hunks FAILED -- saving rejects to file sage-test.rej
sage-dsage-trial: No such file or directory
abort: patch failed to apply

The changes there look fine in principle, though. So once it's rebased, positive review there, too.

[lftabera]

Some of the doctest are ok nowadays but there are other new fails. See for example #9965

I can confirm that, with sage 4.5.3 I get the following doctest failures:

sage -t -long "devel/sage/doc/common/builder.py" sage -t -long "devel/sage/sage/interfaces/qepcad.py" sage -t -long "devel/sage/sage/schemes/elliptic_curves/ell_rational_field.py" sage -t -long "devel/sage/sage/modular/hecke/submodule.py" sage -t -long "devel/sage/sage/modular/abvar/abvar.py" sage -t -long "devel/sage/sage/lfunctions/sympow.py"

All except quecad are trying to be solved in #9965

[lftabera]

qepcad failures

sage -t -long "devel/sage/sage/interfaces/qepcad.py"        
**********************************************************************
File "/opt/SAGE/sage/devel/sage/sage/interfaces/qepcad.py", line 638:
    sage: _rewrite_qepcadrc()
Exception raised:
    Traceback (most recent call last):
      File "/opt/SAGE/sage/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/opt/SAGE/sage/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/opt/SAGE/sage/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_3[3]>", line 1, in <module>
        _rewrite_qepcadrc()###line 638:
    sage: _rewrite_qepcadrc()
      File "/opt/SAGE/sage/local/lib/python/site-packages/sage/interfaces/qepcad.py", line 660, in _rewrite_qepcadrc
        open(fn, 'w').write(text)
    IOError: [Errno 13] Permission denied: '/opt/SAGE/sage/local//default.qepcadrc'
**********************************************************************
File "/opt/SAGE/sage/devel/sage/sage/interfaces/qepcad.py", line 689:
    sage: Qepcad_expect(memcells=100000, logfile=sys.stdout)
Exception raised:
    Traceback (most recent call last):
      File "/opt/SAGE/sage/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/opt/SAGE/sage/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/opt/SAGE/sage/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_6[3]>", line 1, in <module>
        Qepcad_expect(memcells=Integer(100000), logfile=sys.stdout)###line 689:
    sage: Qepcad_expect(memcells=100000, logfile=sys.stdout)
      File "/opt/SAGE/sage/local/lib/python/site-packages/sage/interfaces/qepcad.py", line 692, in __init__
        _rewrite_qepcadrc()
      File "/opt/SAGE/sage/local/lib/python/site-packages/sage/interfaces/qepcad.py", line 660, in _rewrite_qepcadrc
        open(fn, 'w').write(text)
    IOError: [Errno 13] Permission denied: '/opt/SAGE/sage/local//default.qepcadrc'
**********************************************************************
2 items had failures:
   1 of   6 in __main__.example_3
   1 of   4 in __main__.example_6
***Test Failed*** 2 failures.
For whitespace errors, see the file /home/usuario/.sage//tmp/.doctest_qepcad.py
         [1.9 s]

[lftabera]

Another issue, if you compile sage but the very first access of sage is made by a user without write permissions, you get the following error:

---

| Sage Version 4.5.3, Release Date: 2010-09-04 | | Type notebook() for the GUI, and license() for information. |

---

Traceback (most recent call last):

File "/opt/SAGE/sage-4.5.3/local/bin/sage-location", line 174, in <module>

t, R = install_moved()

File "/opt/SAGE/sage-4.5.3/local/bin/sage-location", line 18, in install_moved

write_flags_file()

File "/opt/SAGE/sage-4.5.3/local/bin/sage-location", line 82, in write_flags_file

open(flags_file,'w').write(get_flags_info())

IOError: [Errno 13] Permission denied: '/opt/SAGE/sage-4.5.3/local/lib/sage-flags.txt'

[jhpalmieri]

With Sage 4.7.2.alpha2, I see problems with qepcad and sympow. I think the qepcad problem should be easy to solve, basically as mhansen did before:

sage/interfaces/qepcad.py

@@ -636 +636 @@
     EXAMPLES:
         sage: from sage.interfaces.qepcad import _rewrite_qepcadrc
         sage: _rewrite_qepcadrc()
-        sage: from sage.misc.misc import SAGE_LOCAL
-        sage: open('%s/default.qepcadrc'%SAGE_LOCAL).readlines()[-1]
+        sage: from sage.misc.misc import DOT_SAGE
+        sage: open('%s/default.qepcadrc'%DOT_SAGE).readlines()[-1]
         'SINGULAR .../local//bin'
     """
     global _rewrote_qepcadrc
     if _rewrote_qepcadrc: return
 
-    SL = sage.misc.misc.SAGE_LOCAL
+    SL = sage.misc.misc.DOT_SAGE
     fn = '%s/default.qepcadrc'%SL
     text = \
 """# THIS FILE IS AUTOMATICALLY GENERATED -- DO NOT EDIT

Sympow will be harder to deal with, because of how the spkg is written: it tries to write files to SAGE_LOCAL/lib/sympow. See ​http://trac.sagemath.org/sage_trac/ticket/9703#comment:9 for a possible fix.

I'm attaching a patch to try to deal with the situation when you run Sage for the very first time as a user without write permissions.

Finally, there may be other issues if you compile Sage but don't run it, and then run doctests as a user without write permissions (the first time doctests get run, they might write some files which don't need to be written later). These issues should be fixed, too.

[jdemeyer]

I'm not sure this patch is a good idea or needed, considering #11926. If the Sage install was moved, it is probably good to bail out with an error if there is no write access.

[jhpalmieri]

A few quick comments:

• you have a typo in line 93, "eyactly"
• on line 583, # optional - qepcad, note "algeraic" [sic], this is not the way to format an optional doctest: it will only run if you do "sage -t -optional-only=qepcad,not,algeraic,sic" or something like that.
• same on line 1220

For the scripts patch:

• why not apply os.path.abspath to SAGE_ROOT?

[jdemeyer]

Replying to jhpalmieri:

A few quick comments:

• you have a typo in line 93, "eyactly"

No idea how that happened...

• on line 583, # optional - qepcad, note "algeraic" [sic], this is not the way to format an optional doctest: it will only run if you do "sage -t -optional-only=qepcad,not,algeraic,sic" or something like that.

Very true, thanks!

For the scripts patch:

• why not apply os.path.abspath to SAGE_ROOT?

Because SAGE_ROOT is already canonicalized by realpath at the top of that file:

SAGE_ROOT     = os.path.realpath(os.environ['SAGE_ROOT'])

[jdemeyer]

Moved the "clean up" part of the qepcad patch to #11933.