Opened 13 years ago

Last modified 9 years ago

#5155 closed defect

Fix doctests that want write access to $SAGE_ROOT — at Version 35

Reported by: mabshoff Owned by: jdemeyer
Priority: critical Milestone: sage-5.7
Component: scripts Keywords:
Cc: lftabera, leif Merged in:
Authors: Mike Hansen, Jeroen Demeyer, John Palmieri Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: #13157, #13397, #13452, #13459, #13457, #13123 Stopgaps:

Status badges

Description (last modified by jdemeyer)

All doctests in Sage should pass when they are run as a user that does not have write access to the Sage install. To do that, run the doctests on a Sage install that isn't owned by the user:

$ ./sage -tp 4 -long devel/sage/doc/common devel/sage/doc/de devel/sage/doc/en devel/sage/doc/fr devel/sage/doc/ru devel/sage/sage

Here, we assume that the owner of the Sage install has run Sage at least once (or has applied #11926).

On sage-4.7.2.alpha4, this causes:

The following tests failed:

        sage -t -long devel/sage/sage/modular/hecke/submodule.py # 1 doctests failed
        sage -t -long devel/sage/sage/modular/abvar/abvar.py # 1 doctests failed
        sage -t -long devel/sage/sage/lfunctions/sympow.py # 13 doctests failed
        sage -t -long devel/sage/sage/interfaces/qepcad.py # 3 doctests failed
        sage -t -long devel/sage/sage/schemes/elliptic_curves/ell_rational_field.py # 17 doctests failed

This seems to be because of:

  1. Installing an rc file for the experimental package qepcad (in a very odd location $SAGE_ROOT/local/default.qepcadrc)
  2. Sympow which needs to create datafiles. This should probably be done in sympow's spkg-install instead. This is dealt with at #11920.

Apply:

  1. 5155_sage_location.patch to the scripts repository.
  2. 5155_root.patch to the SAGE_ROOT repository.
  3. 5155_qepcad.patch and 5155_fortran.patch to the Sage library.

Change History (35)

comment:1 Changed 12 years ago by mhansen

  • Authors set to Mike Hansen
  • Report Upstream set to N/A
  • Status changed from new to needs_review

comment:2 Changed 12 years ago by jhpalmieri

  • Status changed from needs_review to needs_work

Before applying the patch, I get two doctest failures (with a non-writeable Sage install). Applying the patch "trac_5155.patch" fixes both. So positive review for that.

The scripts patch wasn't necessary to get those tests to pass, and it also doesn't apply cleanly to 4.3.1:

applying /Users/palmieri/Downloads/scripts_5155.patch
patching file sage-doctest
Hunk #1 FAILED at 55
1 out of 1 hunks FAILED -- saving rejects to file sage-doctest.rej
unable to find 'sage-dsage-trial' for patching
1 out of 1 hunks FAILED -- saving rejects to file sage-dsage-trial.rej
patching file sage-maketest
Hunk #1 succeeded at 1 with fuzz 1 (offset -1 lines).
patching file sage-test
Hunk #1 FAILED at 40
1 out of 1 hunks FAILED -- saving rejects to file sage-test.rej
sage-dsage-trial: No such file or directory
abort: patch failed to apply

The changes there look fine in principle, though. So once it's rebased, positive review there, too.

comment:3 Changed 12 years ago by lftabera

Some of the doctest are ok nowadays but there are other new fails. See for example #9965

I can confirm that, with sage 4.5.3 I get the following doctest failures:

sage -t -long "devel/sage/doc/common/builder.py" sage -t -long "devel/sage/sage/interfaces/qepcad.py" sage -t -long "devel/sage/sage/schemes/elliptic_curves/ell_rational_field.py" sage -t -long "devel/sage/sage/modular/hecke/submodule.py" sage -t -long "devel/sage/sage/modular/abvar/abvar.py" sage -t -long "devel/sage/sage/lfunctions/sympow.py"

All except quecad are trying to be solved in #9965

comment:4 Changed 12 years ago by lftabera

qepcad failures

sage -t -long "devel/sage/sage/interfaces/qepcad.py"        
**********************************************************************
File "/opt/SAGE/sage/devel/sage/sage/interfaces/qepcad.py", line 638:
    sage: _rewrite_qepcadrc()
Exception raised:
    Traceback (most recent call last):
      File "/opt/SAGE/sage/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/opt/SAGE/sage/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/opt/SAGE/sage/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_3[3]>", line 1, in <module>
        _rewrite_qepcadrc()###line 638:
    sage: _rewrite_qepcadrc()
      File "/opt/SAGE/sage/local/lib/python/site-packages/sage/interfaces/qepcad.py", line 660, in _rewrite_qepcadrc
        open(fn, 'w').write(text)
    IOError: [Errno 13] Permission denied: '/opt/SAGE/sage/local//default.qepcadrc'
**********************************************************************
File "/opt/SAGE/sage/devel/sage/sage/interfaces/qepcad.py", line 689:
    sage: Qepcad_expect(memcells=100000, logfile=sys.stdout)
Exception raised:
    Traceback (most recent call last):
      File "/opt/SAGE/sage/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/opt/SAGE/sage/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/opt/SAGE/sage/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_6[3]>", line 1, in <module>
        Qepcad_expect(memcells=Integer(100000), logfile=sys.stdout)###line 689:
    sage: Qepcad_expect(memcells=100000, logfile=sys.stdout)
      File "/opt/SAGE/sage/local/lib/python/site-packages/sage/interfaces/qepcad.py", line 692, in __init__
        _rewrite_qepcadrc()
      File "/opt/SAGE/sage/local/lib/python/site-packages/sage/interfaces/qepcad.py", line 660, in _rewrite_qepcadrc
        open(fn, 'w').write(text)
    IOError: [Errno 13] Permission denied: '/opt/SAGE/sage/local//default.qepcadrc'
**********************************************************************
2 items had failures:
   1 of   6 in __main__.example_3
   1 of   4 in __main__.example_6
***Test Failed*** 2 failures.
For whitespace errors, see the file /home/usuario/.sage//tmp/.doctest_qepcad.py
         [1.9 s]
Last edited 10 years ago by jdemeyer (previous) (diff)

comment:5 Changed 12 years ago by lftabera

  • Cc lftabera added
  • Summary changed from Sage 3.3.a3: fix doctests that want write access to $SAGE_LOCAL to Fix doctests and methods that want write access to $SAGE_LOCAL

Another issue, if you compile sage but the very first access of sage is made by a user without write permissions, you get the following error:


| Sage Version 4.5.3, Release Date: 2010-09-04 | | Type notebook() for the GUI, and license() for information. |


Traceback (most recent call last):

File "/opt/SAGE/sage-4.5.3/local/bin/sage-location", line 174, in <module>

t, R = install_moved()

File "/opt/SAGE/sage-4.5.3/local/bin/sage-location", line 18, in install_moved

write_flags_file()

File "/opt/SAGE/sage-4.5.3/local/bin/sage-location", line 82, in write_flags_file

open(flags_file,'w').write(get_flags_info())

IOError: [Errno 13] Permission denied: '/opt/SAGE/sage-4.5.3/local/lib/sage-flags.txt'

comment:6 Changed 11 years ago by jhpalmieri

With Sage 4.7.2.alpha2, I see problems with qepcad and sympow. I think the qepcad problem should be easy to solve, basically as mhansen did before:

  • sage/interfaces/qepcad.py

    diff --git a/sage/interfaces/qepcad.py b/sage/interfaces/qepcad.py
    a b def _rewrite_qepcadrc(): 
    636636    EXAMPLES:
    637637        sage: from sage.interfaces.qepcad import _rewrite_qepcadrc
    638638        sage: _rewrite_qepcadrc()
    639         sage: from sage.misc.misc import SAGE_LOCAL
    640         sage: open('%s/default.qepcadrc'%SAGE_LOCAL).readlines()[-1]
     639        sage: from sage.misc.misc import DOT_SAGE
     640        sage: open('%s/default.qepcadrc'%DOT_SAGE).readlines()[-1]
    641641        'SINGULAR .../local//bin'
    642642    """
    643643    global _rewrote_qepcadrc
    644644    if _rewrote_qepcadrc: return
    645645
    646     SL = sage.misc.misc.SAGE_LOCAL
     646    SL = sage.misc.misc.DOT_SAGE
    647647    fn = '%s/default.qepcadrc'%SL
    648648    text = \
    649649"""# THIS FILE IS AUTOMATICALLY GENERATED -- DO NOT EDIT

Sympow will be harder to deal with, because of how the spkg is written: it tries to write files to SAGE_LOCAL/lib/sympow. See http://trac.sagemath.org/sage_trac/ticket/9703#comment:9 for a possible fix.

I'm attaching a patch to try to deal with the situation when you run Sage for the very first time as a user without write permissions.

Finally, there may be other issues if you compile Sage but don't run it, and then run doctests as a user without write permissions (the first time doctests get run, they might write some files which don't need to be written later). These issues should be fixed, too.

comment:7 Changed 11 years ago by jdemeyer

I'm not sure this patch is a good idea or needed, considering #11926. If the Sage install was moved, it is probably good to bail out with an error if there is no write access.

comment:8 Changed 11 years ago by jdemeyer

  • Dependencies set to #11926
  • Description modified (diff)
  • Owner changed from mabshoff to jdemeyer

comment:9 Changed 11 years ago by jdemeyer

  • Description modified (diff)

comment:10 Changed 11 years ago by jdemeyer

  • Description modified (diff)

comment:11 Changed 11 years ago by jdemeyer

  • Description modified (diff)

comment:12 Changed 11 years ago by jdemeyer

  • Description modified (diff)

comment:13 Changed 11 years ago by jdemeyer

  • Component changed from doctest to scripts
  • Description modified (diff)
  • Milestone changed from sage-4.7.2 to sage-4.7.3

comment:14 follow-up: Changed 11 years ago by jhpalmieri

  • Cc leif added

A few quick comments:

  • you have a typo in line 93, "eyactly"
  • on line 583, # optional - qepcad, note "algeraic" [sic], this is not the way to format an optional doctest: it will only run if you do "sage -t -optional-only=qepcad,not,algeraic,sic" or something like that.
  • same on line 1220

For the scripts patch:

  • why not apply os.path.abspath to SAGE_ROOT?

comment:15 in reply to: ↑ 14 Changed 11 years ago by jdemeyer

Replying to jhpalmieri:

A few quick comments:

  • you have a typo in line 93, "eyactly"

No idea how that happened...

  • on line 583, # optional - qepcad, note "algeraic" [sic], this is not the way to format an optional doctest: it will only run if you do "sage -t -optional-only=qepcad,not,algeraic,sic" or something like that.

Very true, thanks!

For the scripts patch:

  • why not apply os.path.abspath to SAGE_ROOT?

Because SAGE_ROOT is already canonicalized by realpath at the top of that file:

SAGE_ROOT     = os.path.realpath(os.environ['SAGE_ROOT'])

comment:16 Changed 11 years ago by jdemeyer

  • Dependencies changed from #11926 to #11926, #11933

comment:17 Changed 11 years ago by jdemeyer

Moved the "clean up" part of the qepcad patch to #11933.

comment:18 Changed 11 years ago by jdemeyer

  • Dependencies changed from #11926, #11933 to #11920, #11926, #11933
  • Description modified (diff)

comment:19 Changed 11 years ago by jhpalmieri

  • Description modified (diff)

For the scripts patch: should there be any error checking when writing to files? If a sysadmin installs Sage, runs it once (to generate the appropriate files) and then moves it but doesn't run it again, it would be nice if other users got helpful error messages. One issue is the code

    check_processor_flags()
    # Note: install_moved() may also run e.g. initialize_pkgconfig_files().
    if install_moved():
        print "The Sage installation tree may have moved"
        print "(from %s to %s)." % (OLD_SAGE_ROOT, SAGE_ROOT)

The problem is that check_processor_flags and install_moved already try to write to files, so if permissions are bad, the message about the installation tree may not get printed. I'm attaching a patch to apply on top of yours (basically wraps everything in a try...except block).

Also, as discussed at #11760, the alternate implementation (commented out) of searching for SAGE_ROOT using regular expressions was slower than the one currently used, so I just deleted the comments altogether.

I'll keep looking at your patch.

comment:20 follow-up: Changed 11 years ago by jdemeyer

John, you are assuming a failure means that there was a problem was permissions. We should probably catch IOError (and also OSError perhaps?) and print the message which came with the exception.

comment:21 in reply to: ↑ 20 Changed 11 years ago by jhpalmieri

Replying to jdemeyer:

John, you are assuming a failure means that there was a problem was permissions. We should probably catch IOError (and also OSError perhaps?) and print the message which came with the exception.

I think that a permissions problem is the most likely issue. I've modified the patch a bit: if it looks like bad permissions, print a friendly error message. If it's an IOError unrelated to permissions, explicitly raise the error. If it's some other kind of error, don't catch it at all, so it should get raised, too. You can test this by changing my patch: change the line

if e.strerror.find('Permission denied') != -1:

by changing != to ==, or on the previous line, change IOError to ValueError or something else irrelevant. Move sage and make the directory unwriteable; then when you run sage, you should see the raw error message.

comment:22 Changed 11 years ago by jdemeyer

John, your patch looks good on first glance.

comment:23 Changed 11 years ago by jdemeyer

  • Description modified (diff)

#11920 (sympow) needs review.

comment:24 Changed 11 years ago by jdemeyer

  • Summary changed from Fix doctests and methods that want write access to $SAGE_LOCAL to Fix doctests that want write access to $SAGE_ROOT

comment:25 Changed 11 years ago by jdemeyer

I plan to come back to this once the dependencies are settled. Otherwise I would just be rebasing pointlessly.

comment:26 Changed 11 years ago by jdemeyer

  • Milestone sage-4.7.3 deleted

Milestone sage-4.7.3 deleted

comment:27 Changed 11 years ago by jdemeyer

  • Description modified (diff)
  • Milestone set to sage-4.8

comment:28 Changed 10 years ago by jdemeyer

  • Authors changed from Mike Hansen to Mike Hansen, Jeroen Demeyer, John Palmieri
  • Dependencies changed from #11920, #11926, #11933 to #11920, #11926, #11933, #13452

comment:29 Changed 10 years ago by jdemeyer

  • Description modified (diff)

comment:30 Changed 10 years ago by jdemeyer

  • Description modified (diff)

comment:31 Changed 10 years ago by jdemeyer

  • Dependencies changed from #11920, #11926, #11933, #13452 to #11920, #11926, #11933, #13397, #13452

comment:32 Changed 10 years ago by jdemeyer

  • Dependencies changed from #11920, #11926, #11933, #13397, #13452 to #11920, #11926, #11933, #13397, #13452, #13459

comment:33 Changed 10 years ago by jdemeyer

  • Dependencies changed from #11920, #11926, #11933, #13397, #13452, #13459 to #11920, #11926, #11933, #13397, #13452, #13459, #13457, #13123

comment:34 Changed 10 years ago by jdemeyer

  • Dependencies changed from #11920, #11926, #11933, #13397, #13452, #13459, #13457, #13123 to #13157, #13397, #13452, #13459, #13457, #13123

comment:35 Changed 10 years ago by jdemeyer

  • Description modified (diff)
Note: See TracTickets for help on using tickets.