Opened 11 years ago

Closed 7 years ago

Last modified 7 years ago

#5155 closed defect (fixed)

Fix doctests that want write access to $SAGE_ROOT

Reported by: mabshoff Owned by: jdemeyer
Priority: critical Milestone: sage-5.7
Component: scripts Keywords:
Cc: lftabera, leif Merged in: sage-5.7.beta0
Authors: Mike Hansen, Jeroen Demeyer, John Palmieri Reviewers: François Bissey
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: #13157, #13397, #13452, #13407, #13459, #13457, #13123, #13887 Stopgaps:

Description (last modified by jdemeyer)

All doctests in Sage should pass when they are run as a user that does not have write access to the Sage install. To do that, run the doctests on a Sage install that isn't owned by the user:

$ ./sage -tp 4 --long devel/sage/doc/common devel/sage/doc/en devel/sage/sage

On sage-5.4.beta1, this causes:

The following tests failed:

        sage -t --long devel/sage/sage/interfaces/qepcad.py # 3 doctests failed
        sage -t --long devel/sage/sage/misc/inline_fortran.py # 3 doctests failed
        sage -t --long devel/sage/sage/tests/cmdline.py # 2 doctests failed

Apply:

  1. 5155_sage_location.patch to the scripts repository.
  2. 5155_root.patch to the SAGE_ROOT repository.
  3. 5155_sagelib.patch to the Sage library.

Attachments (3)

5155_root.patch (2.2 KB) - added by jdemeyer 7 years ago.
5155_sage_location.patch (1.5 KB) - added by jdemeyer 7 years ago.
5155_sagelib.patch (2.9 KB) - added by jdemeyer 7 years ago.

Download all attachments as: .zip

Change History (50)

comment:1 Changed 10 years ago by mhansen

  • Authors set to Mike Hansen
  • Report Upstream set to N/A
  • Status changed from new to needs_review

comment:2 Changed 10 years ago by jhpalmieri

  • Status changed from needs_review to needs_work

Before applying the patch, I get two doctest failures (with a non-writeable Sage install). Applying the patch "trac_5155.patch" fixes both. So positive review for that.

The scripts patch wasn't necessary to get those tests to pass, and it also doesn't apply cleanly to 4.3.1:

applying /Users/palmieri/Downloads/scripts_5155.patch
patching file sage-doctest
Hunk #1 FAILED at 55
1 out of 1 hunks FAILED -- saving rejects to file sage-doctest.rej
unable to find 'sage-dsage-trial' for patching
1 out of 1 hunks FAILED -- saving rejects to file sage-dsage-trial.rej
patching file sage-maketest
Hunk #1 succeeded at 1 with fuzz 1 (offset -1 lines).
patching file sage-test
Hunk #1 FAILED at 40
1 out of 1 hunks FAILED -- saving rejects to file sage-test.rej
sage-dsage-trial: No such file or directory
abort: patch failed to apply

The changes there look fine in principle, though. So once it's rebased, positive review there, too.

comment:3 Changed 9 years ago by lftabera

Some of the doctest are ok nowadays but there are other new fails. See for example #9965

I can confirm that, with sage 4.5.3 I get the following doctest failures:

sage -t -long "devel/sage/doc/common/builder.py" sage -t -long "devel/sage/sage/interfaces/qepcad.py" sage -t -long "devel/sage/sage/schemes/elliptic_curves/ell_rational_field.py" sage -t -long "devel/sage/sage/modular/hecke/submodule.py" sage -t -long "devel/sage/sage/modular/abvar/abvar.py" sage -t -long "devel/sage/sage/lfunctions/sympow.py"

All except quecad are trying to be solved in #9965

comment:4 Changed 9 years ago by lftabera

qepcad failures

sage -t -long "devel/sage/sage/interfaces/qepcad.py"        
**********************************************************************
File "/opt/SAGE/sage/devel/sage/sage/interfaces/qepcad.py", line 638:
    sage: _rewrite_qepcadrc()
Exception raised:
    Traceback (most recent call last):
      File "/opt/SAGE/sage/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/opt/SAGE/sage/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/opt/SAGE/sage/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_3[3]>", line 1, in <module>
        _rewrite_qepcadrc()###line 638:
    sage: _rewrite_qepcadrc()
      File "/opt/SAGE/sage/local/lib/python/site-packages/sage/interfaces/qepcad.py", line 660, in _rewrite_qepcadrc
        open(fn, 'w').write(text)
    IOError: [Errno 13] Permission denied: '/opt/SAGE/sage/local//default.qepcadrc'
**********************************************************************
File "/opt/SAGE/sage/devel/sage/sage/interfaces/qepcad.py", line 689:
    sage: Qepcad_expect(memcells=100000, logfile=sys.stdout)
Exception raised:
    Traceback (most recent call last):
      File "/opt/SAGE/sage/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/opt/SAGE/sage/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/opt/SAGE/sage/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_6[3]>", line 1, in <module>
        Qepcad_expect(memcells=Integer(100000), logfile=sys.stdout)###line 689:
    sage: Qepcad_expect(memcells=100000, logfile=sys.stdout)
      File "/opt/SAGE/sage/local/lib/python/site-packages/sage/interfaces/qepcad.py", line 692, in __init__
        _rewrite_qepcadrc()
      File "/opt/SAGE/sage/local/lib/python/site-packages/sage/interfaces/qepcad.py", line 660, in _rewrite_qepcadrc
        open(fn, 'w').write(text)
    IOError: [Errno 13] Permission denied: '/opt/SAGE/sage/local//default.qepcadrc'
**********************************************************************
2 items had failures:
   1 of   6 in __main__.example_3
   1 of   4 in __main__.example_6
***Test Failed*** 2 failures.
For whitespace errors, see the file /home/usuario/.sage//tmp/.doctest_qepcad.py
         [1.9 s]
Last edited 7 years ago by jdemeyer (previous) (diff)

comment:5 Changed 9 years ago by lftabera

  • Cc lftabera added
  • Summary changed from Sage 3.3.a3: fix doctests that want write access to $SAGE_LOCAL to Fix doctests and methods that want write access to $SAGE_LOCAL

Another issue, if you compile sage but the very first access of sage is made by a user without write permissions, you get the following error:


| Sage Version 4.5.3, Release Date: 2010-09-04 | | Type notebook() for the GUI, and license() for information. |


Traceback (most recent call last):

File "/opt/SAGE/sage-4.5.3/local/bin/sage-location", line 174, in <module>

t, R = install_moved()

File "/opt/SAGE/sage-4.5.3/local/bin/sage-location", line 18, in install_moved

write_flags_file()

File "/opt/SAGE/sage-4.5.3/local/bin/sage-location", line 82, in write_flags_file

open(flags_file,'w').write(get_flags_info())

IOError: [Errno 13] Permission denied: '/opt/SAGE/sage-4.5.3/local/lib/sage-flags.txt'

comment:6 Changed 8 years ago by jhpalmieri

With Sage 4.7.2.alpha2, I see problems with qepcad and sympow. I think the qepcad problem should be easy to solve, basically as mhansen did before:

  • sage/interfaces/qepcad.py

    diff --git a/sage/interfaces/qepcad.py b/sage/interfaces/qepcad.py
    a b def _rewrite_qepcadrc(): 
    636636    EXAMPLES:
    637637        sage: from sage.interfaces.qepcad import _rewrite_qepcadrc
    638638        sage: _rewrite_qepcadrc()
    639         sage: from sage.misc.misc import SAGE_LOCAL
    640         sage: open('%s/default.qepcadrc'%SAGE_LOCAL).readlines()[-1]
     639        sage: from sage.misc.misc import DOT_SAGE
     640        sage: open('%s/default.qepcadrc'%DOT_SAGE).readlines()[-1]
    641641        'SINGULAR .../local//bin'
    642642    """
    643643    global _rewrote_qepcadrc
    644644    if _rewrote_qepcadrc: return
    645645
    646     SL = sage.misc.misc.SAGE_LOCAL
     646    SL = sage.misc.misc.DOT_SAGE
    647647    fn = '%s/default.qepcadrc'%SL
    648648    text = \
    649649"""# THIS FILE IS AUTOMATICALLY GENERATED -- DO NOT EDIT

Sympow will be harder to deal with, because of how the spkg is written: it tries to write files to SAGE_LOCAL/lib/sympow. See http://trac.sagemath.org/sage_trac/ticket/9703#comment:9 for a possible fix.

I'm attaching a patch to try to deal with the situation when you run Sage for the very first time as a user without write permissions.

Finally, there may be other issues if you compile Sage but don't run it, and then run doctests as a user without write permissions (the first time doctests get run, they might write some files which don't need to be written later). These issues should be fixed, too.

comment:7 Changed 8 years ago by jdemeyer

I'm not sure this patch is a good idea or needed, considering #11926. If the Sage install was moved, it is probably good to bail out with an error if there is no write access.

comment:8 Changed 8 years ago by jdemeyer

  • Dependencies set to #11926
  • Description modified (diff)
  • Owner changed from mabshoff to jdemeyer

comment:9 Changed 8 years ago by jdemeyer

  • Description modified (diff)

comment:10 Changed 8 years ago by jdemeyer

  • Description modified (diff)

comment:11 Changed 8 years ago by jdemeyer

  • Description modified (diff)

comment:12 Changed 8 years ago by jdemeyer

  • Description modified (diff)

comment:13 Changed 8 years ago by jdemeyer

  • Component changed from doctest to scripts
  • Description modified (diff)
  • Milestone changed from sage-4.7.2 to sage-4.7.3

comment:14 follow-up: Changed 8 years ago by jhpalmieri

  • Cc leif added

A few quick comments:

  • you have a typo in line 93, "eyactly"
  • on line 583, # optional - qepcad, note "algeraic" [sic], this is not the way to format an optional doctest: it will only run if you do "sage -t -optional-only=qepcad,not,algeraic,sic" or something like that.
  • same on line 1220

For the scripts patch:

  • why not apply os.path.abspath to SAGE_ROOT?

comment:15 in reply to: ↑ 14 Changed 8 years ago by jdemeyer

Replying to jhpalmieri:

A few quick comments:

  • you have a typo in line 93, "eyactly"

No idea how that happened...

  • on line 583, # optional - qepcad, note "algeraic" [sic], this is not the way to format an optional doctest: it will only run if you do "sage -t -optional-only=qepcad,not,algeraic,sic" or something like that.

Very true, thanks!

For the scripts patch:

  • why not apply os.path.abspath to SAGE_ROOT?

Because SAGE_ROOT is already canonicalized by realpath at the top of that file:

SAGE_ROOT     = os.path.realpath(os.environ['SAGE_ROOT'])

comment:16 Changed 8 years ago by jdemeyer

  • Dependencies changed from #11926 to #11926, #11933

comment:17 Changed 8 years ago by jdemeyer

Moved the "clean up" part of the qepcad patch to #11933.

comment:18 Changed 8 years ago by jdemeyer

  • Dependencies changed from #11926, #11933 to #11920, #11926, #11933
  • Description modified (diff)

comment:19 Changed 8 years ago by jhpalmieri

  • Description modified (diff)

For the scripts patch: should there be any error checking when writing to files? If a sysadmin installs Sage, runs it once (to generate the appropriate files) and then moves it but doesn't run it again, it would be nice if other users got helpful error messages. One issue is the code

    check_processor_flags()
    # Note: install_moved() may also run e.g. initialize_pkgconfig_files().
    if install_moved():
        print "The Sage installation tree may have moved"
        print "(from %s to %s)." % (OLD_SAGE_ROOT, SAGE_ROOT)

The problem is that check_processor_flags and install_moved already try to write to files, so if permissions are bad, the message about the installation tree may not get printed. I'm attaching a patch to apply on top of yours (basically wraps everything in a try...except block).

Also, as discussed at #11760, the alternate implementation (commented out) of searching for SAGE_ROOT using regular expressions was slower than the one currently used, so I just deleted the comments altogether.

I'll keep looking at your patch.

comment:20 follow-up: Changed 8 years ago by jdemeyer

John, you are assuming a failure means that there was a problem was permissions. We should probably catch IOError (and also OSError perhaps?) and print the message which came with the exception.

comment:21 in reply to: ↑ 20 Changed 8 years ago by jhpalmieri

Replying to jdemeyer:

John, you are assuming a failure means that there was a problem was permissions. We should probably catch IOError (and also OSError perhaps?) and print the message which came with the exception.

I think that a permissions problem is the most likely issue. I've modified the patch a bit: if it looks like bad permissions, print a friendly error message. If it's an IOError unrelated to permissions, explicitly raise the error. If it's some other kind of error, don't catch it at all, so it should get raised, too. You can test this by changing my patch: change the line

if e.strerror.find('Permission denied') != -1:

by changing != to ==, or on the previous line, change IOError to ValueError or something else irrelevant. Move sage and make the directory unwriteable; then when you run sage, you should see the raw error message.

comment:22 Changed 8 years ago by jdemeyer

John, your patch looks good on first glance.

comment:23 Changed 8 years ago by jdemeyer

  • Description modified (diff)

#11920 (sympow) needs review.

comment:24 Changed 8 years ago by jdemeyer

  • Summary changed from Fix doctests and methods that want write access to $SAGE_LOCAL to Fix doctests that want write access to $SAGE_ROOT

comment:25 Changed 8 years ago by jdemeyer

I plan to come back to this once the dependencies are settled. Otherwise I would just be rebasing pointlessly.

comment:26 Changed 8 years ago by jdemeyer

  • Milestone sage-4.7.3 deleted

Milestone sage-4.7.3 deleted

comment:27 Changed 8 years ago by jdemeyer

  • Description modified (diff)
  • Milestone set to sage-4.8

comment:28 Changed 7 years ago by jdemeyer

  • Authors changed from Mike Hansen to Mike Hansen, Jeroen Demeyer, John Palmieri
  • Dependencies changed from #11920, #11926, #11933 to #11920, #11926, #11933, #13452

comment:29 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:30 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:31 Changed 7 years ago by jdemeyer

  • Dependencies changed from #11920, #11926, #11933, #13452 to #11920, #11926, #11933, #13397, #13452

comment:32 Changed 7 years ago by jdemeyer

  • Dependencies changed from #11920, #11926, #11933, #13397, #13452 to #11920, #11926, #11933, #13397, #13452, #13459

comment:33 Changed 7 years ago by jdemeyer

  • Dependencies changed from #11920, #11926, #11933, #13397, #13452, #13459 to #11920, #11926, #11933, #13397, #13452, #13459, #13457, #13123

comment:34 Changed 7 years ago by jdemeyer

  • Dependencies changed from #11920, #11926, #11933, #13397, #13452, #13459, #13457, #13123 to #13157, #13397, #13452, #13459, #13457, #13123

comment:35 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:36 Changed 7 years ago by jdemeyer

  • Description modified (diff)

comment:37 Changed 7 years ago by jdemeyer

  • Dependencies changed from #13157, #13397, #13452, #13459, #13457, #13123 to #13157, #13397, #13452, #13407, #13459, #13457, #13123

comment:38 Changed 7 years ago by jdemeyer

  • Status changed from needs_work to needs_review

This seems to be fixed with these patches, so I'm putting it to needs_review.

However, this depends on so many other tickets, so it might be good to postpone the review and wait for the other tickets.

Changed 7 years ago by jdemeyer

Changed 7 years ago by jdemeyer

comment:39 Changed 7 years ago by jdemeyer

The Fortran patch needed to be rebased because of #13579. Apart from that, things look fine.

comment:40 Changed 7 years ago by jdemeyer

  • Status changed from needs_review to needs_work

Some doctest failures...

Changed 7 years ago by jdemeyer

comment:41 Changed 7 years ago by jdemeyer

  • Description modified (diff)
  • Status changed from needs_work to needs_review

comment:42 Changed 7 years ago by jdemeyer

  • Dependencies changed from #13157, #13397, #13452, #13407, #13459, #13457, #13123 to #13157, #13397, #13452, #13407, #13459, #13457, #13123, #13887
  • Description modified (diff)

comment:43 Changed 7 years ago by fbissey

SAGE_LOCAL is a strange location for default.qepcadrc I suppose it really needs to be changed in the spkg first but SAGE_LOCAL/etc is a better location for this kind of file.

comment:44 follow-up: Changed 7 years ago by fbissey

  • Reviewers set to François Bissey
  • Status changed from needs_review to positive_review

My only objection is the location of default.qepcadrc but since it is for an experimental spkg and would require another ticket to change it I am giving this a positive review. Moving the default.qepcadrc location should be the object of a follow up ticket.

comment:45 in reply to: ↑ 44 Changed 7 years ago by jdemeyer

  • Milestone changed from sage-5.6 to sage-5.7

Replying to fbissey:

My only objection is the location of default.qepcadrc

Also, you shouldn't blame this ticket for that, it has always been like that.

Thanks for the review!

comment:46 Changed 7 years ago by jdemeyer

  • Merged in set to sage-5.7.beta0
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:47 Changed 7 years ago by jdemeyer

I made a patch for the analogous problem for $HOME at #13985. It would be nice if somebody could review that.

Note: See TracTickets for help on using tickets.