#5155 closed defect (fixed)
Fix doctests that want write access to $SAGE_ROOT
Reported by:  Michael Abshoff  Owned by:  Jeroen Demeyer 

Priority:  critical  Milestone:  sage5.7 
Component:  scripts  Keywords:  
Cc:  Luis Felipe Tabera Alonso, Leif Leonhardy  Merged in:  sage5.7.beta0 
Authors:  Mike Hansen, Jeroen Demeyer, John Palmieri  Reviewers:  François Bissey 
Report Upstream:  N/A  Work issues:  
Branch:  Commit:  
Dependencies:  #13157, #13397, #13452, #13407, #13459, #13457, #13123, #13887  Stopgaps: 
Description (last modified by )
All doctests in Sage should pass when they are run as a user that does not have write access to the Sage install. To do that, run the doctests on a Sage install that isn't owned by the user:
$ ./sage tp 4 long devel/sage/doc/common devel/sage/doc/en devel/sage/sage
On sage5.4.beta1, this causes:
The following tests failed: sage t long devel/sage/sage/interfaces/qepcad.py # 3 doctests failed sage t long devel/sage/sage/misc/inline_fortran.py # 3 doctests failed sage t long devel/sage/sage/tests/cmdline.py # 2 doctests failed
Apply:
 5155_sage_location.patch to the scripts repository.
 5155_root.patch to the
SAGE_ROOT
repository.  5155_sagelib.patch to the Sage library.
Attachments (3)
Change History (50)
comment:1 Changed 13 years ago by
Authors:  → Mike Hansen 

Report Upstream:  → N/A 
Status:  new → needs_review 
comment:2 Changed 13 years ago by
Status:  needs_review → needs_work 

comment:3 Changed 12 years ago by
Some of the doctest are ok nowadays but there are other new fails. See for example #9965
I can confirm that, with sage 4.5.3 I get the following doctest failures:
sage t long "devel/sage/doc/common/builder.py" sage t long "devel/sage/sage/interfaces/qepcad.py" sage t long "devel/sage/sage/schemes/elliptic_curves/ell_rational_field.py" sage t long "devel/sage/sage/modular/hecke/submodule.py" sage t long "devel/sage/sage/modular/abvar/abvar.py" sage t long "devel/sage/sage/lfunctions/sympow.py"
All except quecad are trying to be solved in #9965
comment:4 Changed 12 years ago by
qepcad failures
sage t long "devel/sage/sage/interfaces/qepcad.py" File "/opt/SAGE/sage/devel/sage/sage/interfaces/qepcad.py", line 638:
sage: _rewrite_qepcadrc()
Exception raised:
Traceback (most recent call last):
File "/opt/SAGE/sage/local/bin/ncadoctest.py", line 1231, in run_one_test
self.run_one_example(test, example, filename, compileflags)
File "/opt/SAGE/sage/local/bin/sagedoctest.py", line 38, in run_one_example
OrigDocTestRunner?.run_one_example(self, test, example, filename, compileflags)
File "/opt/SAGE/sage/local/bin/ncadoctest.py", line 1172, in run_one_example
compileflags, 1) in test.globs
File "<doctest main.example_3[3]>", line 1, in <module>
_rewrite_qepcadrc()###line 638:
sage: _rewrite_qepcadrc()
File "/opt/SAGE/sage/local/lib/python/sitepackages/sage/interfaces/qepcad.py", line 660, in _rewrite_qepcadrc
open(fn, 'w').write(text)
IOError: [Errno 13] Permission denied: '/opt/SAGE/sage/localdefault.qepcadrc'
File "/opt/SAGE/sage/devel/sage/sage/interfaces/qepcad.py", line 689:
sage: Qepcad_expect(memcells=100000, logfile=sys.stdout)
Exception raised:
Traceback (most recent call last):
File "/opt/SAGE/sage/local/bin/ncadoctest.py", line 1231, in run_one_test
self.run_one_example(test, example, filename, compileflags)
File "/opt/SAGE/sage/local/bin/sagedoctest.py", line 38, in run_one_example
OrigDocTestRunner?.run_one_example(self, test, example, filename, compileflags)
File "/opt/SAGE/sage/local/bin/ncadoctest.py", line 1172, in run_one_example
compileflags, 1) in test.globs
File "<doctest main.example_6[3]>", line 1, in <module>
Qepcad_expect(memcells=Integer(100000), logfile=sys.stdout)###line 689:
sage: Qepcad_expect(memcells=100000, logfile=sys.stdout)
File "/opt/SAGE/sage/local/lib/python/sitepackages/sage/interfaces/qepcad.py", line 692, in init
_rewrite_qepcadrc()
File "/opt/SAGE/sage/local/lib/python/sitepackages/sage/interfaces/qepcad.py", line 660, in _rewrite_qepcadrc
open(fn, 'w').write(text)
IOError: [Errno 13] Permission denied: '/opt/SAGE/sage/localdefault.qepcadrc'
2 items had failures:
1 of 6 in main.example_3 1 of 4 in main.example_6
*Test Failed* 2 failures. For whitespace errors, see the file /home/usuario/.sagetmp/.doctest_qepcad.py
[1.9 s]
comment:5 Changed 12 years ago by
Cc:  Luis Felipe Tabera Alonso added 

Summary:  Sage 3.3.a3: fix doctests that want write access to $SAGE_LOCAL → Fix doctests and methods that want write access to $SAGE_LOCAL 
Another issue, if you compile sage but the very first access of sage is made by a user without write permissions, you get the following error:
 Sage Version 4.5.3, Release Date: 20100904   Type notebook() for the GUI, and license() for information. 
Traceback (most recent call last):
File "/opt/SAGE/sage4.5.3/local/bin/sagelocation", line 174, in <module>
t, R = install_moved()
File "/opt/SAGE/sage4.5.3/local/bin/sagelocation", line 18, in install_moved
write_flags_file()
File "/opt/SAGE/sage4.5.3/local/bin/sagelocation", line 82, in write_flags_file
open(flags_file,'w').write(get_flags_info())
IOError: [Errno 13] Permission denied: '/opt/SAGE/sage4.5.3/local/lib/sageflags.txt'
comment:6 Changed 11 years ago by
With Sage 4.7.2.alpha2, I see problems with qepcad and sympow. I think the qepcad problem should be easy to solve, basically as mhansen did before:

sage/interfaces/qepcad.py
diff git a/sage/interfaces/qepcad.py b/sage/interfaces/qepcad.py
a b def _rewrite_qepcadrc(): 636 636 EXAMPLES: 637 637 sage: from sage.interfaces.qepcad import _rewrite_qepcadrc 638 638 sage: _rewrite_qepcadrc() 639 sage: from sage.misc.misc import SAGE_LOCAL640 sage: open('%s/default.qepcadrc'% SAGE_LOCAL).readlines()[1]639 sage: from sage.misc.misc import DOT_SAGE 640 sage: open('%s/default.qepcadrc'%DOT_SAGE).readlines()[1] 641 641 'SINGULAR .../local//bin' 642 642 """ 643 643 global _rewrote_qepcadrc 644 644 if _rewrote_qepcadrc: return 645 645 646 SL = sage.misc.misc. SAGE_LOCAL646 SL = sage.misc.misc.DOT_SAGE 647 647 fn = '%s/default.qepcadrc'%SL 648 648 text = \ 649 649 """# THIS FILE IS AUTOMATICALLY GENERATED  DO NOT EDIT
Sympow will be harder to deal with, because of how the spkg is written: it tries to write files to SAGE_LOCAL/lib/sympow. See http://trac.sagemath.org/sage_trac/ticket/9703#comment:9 for a possible fix.
I'm attaching a patch to try to deal with the situation when you run Sage for the very first time as a user without write permissions.
Finally, there may be other issues if you compile Sage but don't run it, and then run doctests as a user without write permissions (the first time doctests get run, they might write some files which don't need to be written later). These issues should be fixed, too.
comment:7 Changed 11 years ago by
I'm not sure this patch is a good idea or needed, considering #11926. If the Sage install was moved, it is probably good to bail out with an error if there is no write access.
comment:8 Changed 11 years ago by
Dependencies:  → #11926 

Description:  modified (diff) 
Owner:  changed from Michael Abshoff to Jeroen Demeyer 
comment:9 Changed 11 years ago by
Description:  modified (diff) 

comment:10 Changed 11 years ago by
Description:  modified (diff) 

comment:11 Changed 11 years ago by
Description:  modified (diff) 

comment:12 Changed 11 years ago by
Description:  modified (diff) 

comment:13 Changed 11 years ago by
Component:  doctest → scripts 

Description:  modified (diff) 
Milestone:  sage4.7.2 → sage4.7.3 
comment:14 followup: 15 Changed 11 years ago by
Cc:  Leif Leonhardy added 

A few quick comments:
 you have a typo in line 93, "eyactly"
 on line 583,
# optional  qepcad, note "algeraic" [sic]
, this is not the way to format an optional doctest: it will only run if you do "sage t optionalonly=qepcad,not,algeraic,sic" or something like that.  same on line 1220
For the scripts patch:
 why not apply os.path.abspath to SAGE_ROOT?
comment:15 Changed 11 years ago by
Replying to jhpalmieri:
A few quick comments:
 you have a typo in line 93, "eyactly"
No idea how that happened...
 on line 583,
# optional  qepcad, note "algeraic" [sic]
, this is not the way to format an optional doctest: it will only run if you do "sage t optionalonly=qepcad,not,algeraic,sic" or something like that.
Very true, thanks!
For the scripts patch:
 why not apply os.path.abspath to SAGE_ROOT?
Because SAGE_ROOT
is already canonicalized by realpath
at the top of that file:
SAGE_ROOT = os.path.realpath(os.environ['SAGE_ROOT'])
comment:16 Changed 11 years ago by
Dependencies:  #11926 → #11926, #11933 

comment:18 Changed 11 years ago by
Dependencies:  #11926, #11933 → #11920, #11926, #11933 

Description:  modified (diff) 
comment:19 Changed 11 years ago by
Description:  modified (diff) 

For the scripts patch: should there be any error checking when writing to files? If a sysadmin installs Sage, runs it once (to generate the appropriate files) and then moves it but doesn't run it again, it would be nice if other users got helpful error messages. One issue is the code
check_processor_flags() # Note: install_moved() may also run e.g. initialize_pkgconfig_files(). if install_moved(): print "The Sage installation tree may have moved" print "(from %s to %s)." % (OLD_SAGE_ROOT, SAGE_ROOT)
The problem is that check_processor_flags
and install_moved
already try to write to files, so if permissions are bad, the message about the installation tree may not get printed. I'm attaching a patch to apply on top of yours (basically wraps everything in a try...except block).
Also, as discussed at #11760, the alternate implementation (commented out) of searching for SAGE_ROOT using regular expressions was slower than the one currently used, so I just deleted the comments altogether.
I'll keep looking at your patch.
comment:20 followup: 21 Changed 11 years ago by
John, you are assuming a failure means that there was a problem was permissions. We should probably catch IOError
(and also OSError
perhaps?) and print the message which came with the exception.
comment:21 Changed 11 years ago by
Replying to jdemeyer:
John, you are assuming a failure means that there was a problem was permissions. We should probably catch
IOError
(and alsoOSError
perhaps?) and print the message which came with the exception.
I think that a permissions problem is the most likely issue. I've modified the patch a bit: if it looks like bad permissions, print a friendly error message. If it's an IOError
unrelated to permissions, explicitly raise the error. If it's some other kind of error, don't catch it at all, so it should get raised, too. You can test this by changing my patch: change the line
if e.strerror.find('Permission denied') != 1:
by changing !=
to ==
, or on the previous line, change IOError
to ValueError
or something else irrelevant. Move sage and make the directory unwriteable; then when you run sage, you should see the raw error message.
comment:24 Changed 11 years ago by
Summary:  Fix doctests and methods that want write access to $SAGE_LOCAL → Fix doctests that want write access to $SAGE_ROOT 

comment:25 Changed 11 years ago by
I plan to come back to this once the dependencies are settled. Otherwise I would just be rebasing pointlessly.
comment:27 Changed 11 years ago by
Description:  modified (diff) 

Milestone:  → sage4.8 
comment:28 Changed 10 years ago by
Authors:  Mike Hansen → Mike Hansen, Jeroen Demeyer, John Palmieri 

Dependencies:  #11920, #11926, #11933 → #11920, #11926, #11933, #13452 
comment:29 Changed 10 years ago by
Description:  modified (diff) 

comment:30 Changed 10 years ago by
Description:  modified (diff) 

comment:31 Changed 10 years ago by
Dependencies:  #11920, #11926, #11933, #13452 → #11920, #11926, #11933, #13397, #13452 

comment:32 Changed 10 years ago by
Dependencies:  #11920, #11926, #11933, #13397, #13452 → #11920, #11926, #11933, #13397, #13452, #13459 

comment:33 Changed 10 years ago by
Dependencies:  #11920, #11926, #11933, #13397, #13452, #13459 → #11920, #11926, #11933, #13397, #13452, #13459, #13457, #13123 

comment:34 Changed 10 years ago by
Dependencies:  #11920, #11926, #11933, #13397, #13452, #13459, #13457, #13123 → #13157, #13397, #13452, #13459, #13457, #13123 

comment:35 Changed 10 years ago by
Description:  modified (diff) 

comment:36 Changed 10 years ago by
Description:  modified (diff) 

comment:37 Changed 10 years ago by
Dependencies:  #13157, #13397, #13452, #13459, #13457, #13123 → #13157, #13397, #13452, #13407, #13459, #13457, #13123 

comment:38 Changed 10 years ago by
Status:  needs_work → needs_review 

This seems to be fixed with these patches, so I'm putting it to needs_review.
However, this depends on so many other tickets, so it might be good to postpone the review and wait for the other tickets.
Changed 10 years ago by
Attachment:  5155_root.patch added 

Changed 10 years ago by
Attachment:  5155_sage_location.patch added 

comment:39 Changed 10 years ago by
The Fortran patch needed to be rebased because of #13579. Apart from that, things look fine.
Changed 10 years ago by
Attachment:  5155_sagelib.patch added 

comment:41 Changed 10 years ago by
Description:  modified (diff) 

Status:  needs_work → needs_review 
comment:42 Changed 10 years ago by
Dependencies:  #13157, #13397, #13452, #13407, #13459, #13457, #13123 → #13157, #13397, #13452, #13407, #13459, #13457, #13123, #13887 

Description:  modified (diff) 
comment:43 Changed 10 years ago by
SAGE_LOCAL is a strange location for default.qepcadrc I suppose it really needs to be changed in the spkg first but SAGE_LOCAL/etc is a better location for this kind of file.
comment:44 followup: 45 Changed 10 years ago by
Reviewers:  → François Bissey 

Status:  needs_review → positive_review 
My only objection is the location of default.qepcadrc but since it is for an experimental spkg and would require another ticket to change it I am giving this a positive review. Moving the default.qepcadrc location should be the object of a follow up ticket.
comment:45 Changed 10 years ago by
Milestone:  sage5.6 → sage5.7 

Replying to fbissey:
My only objection is the location of default.qepcadrc
Also, you shouldn't blame this ticket for that, it has always been like that.
Thanks for the review!
comment:46 Changed 10 years ago by
Merged in:  → sage5.7.beta0 

Resolution:  → fixed 
Status:  positive_review → closed 
comment:47 Changed 10 years ago by
I made a patch for the analogous problem for $HOME
at #13985. It would be nice if somebody could review that.
Before applying the patch, I get two doctest failures (with a nonwriteable Sage install). Applying the patch "trac_5155.patch" fixes both. So positive review for that.
The scripts patch wasn't necessary to get those tests to pass, and it also doesn't apply cleanly to 4.3.1:
The changes there look fine in principle, though. So once it's rebased, positive review there, too.