#5155 closed defect (fixed)
Fix doctests that want write access to $SAGE_ROOT
Reported by:  mabshoff  Owned by:  jdemeyer 

Priority:  critical  Milestone:  sage5.7 
Component:  scripts  Keywords:  
Cc:  lftabera, leif  Merged in:  sage5.7.beta0 
Authors:  Mike Hansen, Jeroen Demeyer, John Palmieri  Reviewers:  François Bissey 
Report Upstream:  N/A  Work issues:  
Branch:  Commit:  
Dependencies:  #13157, #13397, #13452, #13407, #13459, #13457, #13123, #13887  Stopgaps: 
Description (last modified by )
All doctests in Sage should pass when they are run as a user that does not have write access to the Sage install. To do that, run the doctests on a Sage install that isn't owned by the user:
$ ./sage tp 4 long devel/sage/doc/common devel/sage/doc/en devel/sage/sage
On sage5.4.beta1, this causes:
The following tests failed: sage t long devel/sage/sage/interfaces/qepcad.py # 3 doctests failed sage t long devel/sage/sage/misc/inline_fortran.py # 3 doctests failed sage t long devel/sage/sage/tests/cmdline.py # 2 doctests failed
Apply:
 5155_sage_location.patch to the scripts repository.
 5155_root.patch to the
SAGE_ROOT
repository.  5155_sagelib.patch to the Sage library.
Attachments (3)
Change History (50)
comment:1 Changed 11 years ago by
 Report Upstream set to N/A
 Status changed from new to needs_review
comment:2 Changed 11 years ago by
 Status changed from needs_review to needs_work
comment:3 Changed 10 years ago by
Some of the doctest are ok nowadays but there are other new fails. See for example #9965
I can confirm that, with sage 4.5.3 I get the following doctest failures:
sage t long "devel/sage/doc/common/builder.py" sage t long "devel/sage/sage/interfaces/qepcad.py" sage t long "devel/sage/sage/schemes/elliptic_curves/ell_rational_field.py" sage t long "devel/sage/sage/modular/hecke/submodule.py" sage t long "devel/sage/sage/modular/abvar/abvar.py" sage t long "devel/sage/sage/lfunctions/sympow.py"
All except quecad are trying to be solved in #9965
comment:4 Changed 10 years ago by
qepcad failures
sage t long "devel/sage/sage/interfaces/qepcad.py" ********************************************************************** File "/opt/SAGE/sage/devel/sage/sage/interfaces/qepcad.py", line 638: sage: _rewrite_qepcadrc() Exception raised: Traceback (most recent call last): File "/opt/SAGE/sage/local/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/opt/SAGE/sage/local/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/opt/SAGE/sage/local/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_3[3]>", line 1, in <module> _rewrite_qepcadrc()###line 638: sage: _rewrite_qepcadrc() File "/opt/SAGE/sage/local/lib/python/sitepackages/sage/interfaces/qepcad.py", line 660, in _rewrite_qepcadrc open(fn, 'w').write(text) IOError: [Errno 13] Permission denied: '/opt/SAGE/sage/local//default.qepcadrc' ********************************************************************** File "/opt/SAGE/sage/devel/sage/sage/interfaces/qepcad.py", line 689: sage: Qepcad_expect(memcells=100000, logfile=sys.stdout) Exception raised: Traceback (most recent call last): File "/opt/SAGE/sage/local/bin/ncadoctest.py", line 1231, in run_one_test self.run_one_example(test, example, filename, compileflags) File "/opt/SAGE/sage/local/bin/sagedoctest.py", line 38, in run_one_example OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags) File "/opt/SAGE/sage/local/bin/ncadoctest.py", line 1172, in run_one_example compileflags, 1) in test.globs File "<doctest __main__.example_6[3]>", line 1, in <module> Qepcad_expect(memcells=Integer(100000), logfile=sys.stdout)###line 689: sage: Qepcad_expect(memcells=100000, logfile=sys.stdout) File "/opt/SAGE/sage/local/lib/python/sitepackages/sage/interfaces/qepcad.py", line 692, in __init__ _rewrite_qepcadrc() File "/opt/SAGE/sage/local/lib/python/sitepackages/sage/interfaces/qepcad.py", line 660, in _rewrite_qepcadrc open(fn, 'w').write(text) IOError: [Errno 13] Permission denied: '/opt/SAGE/sage/local//default.qepcadrc' ********************************************************************** 2 items had failures: 1 of 6 in __main__.example_3 1 of 4 in __main__.example_6 ***Test Failed*** 2 failures. For whitespace errors, see the file /home/usuario/.sage//tmp/.doctest_qepcad.py [1.9 s]
comment:5 Changed 10 years ago by
 Cc lftabera added
 Summary changed from Sage 3.3.a3: fix doctests that want write access to $SAGE_LOCAL to Fix doctests and methods that want write access to $SAGE_LOCAL
Another issue, if you compile sage but the very first access of sage is made by a user without write permissions, you get the following error:
 Sage Version 4.5.3, Release Date: 20100904   Type notebook() for the GUI, and license() for information. 
Traceback (most recent call last):
File "/opt/SAGE/sage4.5.3/local/bin/sagelocation", line 174, in <module>
t, R = install_moved()
File "/opt/SAGE/sage4.5.3/local/bin/sagelocation", line 18, in install_moved
write_flags_file()
File "/opt/SAGE/sage4.5.3/local/bin/sagelocation", line 82, in write_flags_file
open(flags_file,'w').write(get_flags_info())
IOError: [Errno 13] Permission denied: '/opt/SAGE/sage4.5.3/local/lib/sageflags.txt'
comment:6 Changed 9 years ago by
With Sage 4.7.2.alpha2, I see problems with qepcad and sympow. I think the qepcad problem should be easy to solve, basically as mhansen did before:

sage/interfaces/qepcad.py
diff git a/sage/interfaces/qepcad.py b/sage/interfaces/qepcad.py
a b def _rewrite_qepcadrc(): 636 636 EXAMPLES: 637 637 sage: from sage.interfaces.qepcad import _rewrite_qepcadrc 638 638 sage: _rewrite_qepcadrc() 639 sage: from sage.misc.misc import SAGE_LOCAL640 sage: open('%s/default.qepcadrc'% SAGE_LOCAL).readlines()[1]639 sage: from sage.misc.misc import DOT_SAGE 640 sage: open('%s/default.qepcadrc'%DOT_SAGE).readlines()[1] 641 641 'SINGULAR .../local//bin' 642 642 """ 643 643 global _rewrote_qepcadrc 644 644 if _rewrote_qepcadrc: return 645 645 646 SL = sage.misc.misc. SAGE_LOCAL646 SL = sage.misc.misc.DOT_SAGE 647 647 fn = '%s/default.qepcadrc'%SL 648 648 text = \ 649 649 """# THIS FILE IS AUTOMATICALLY GENERATED  DO NOT EDIT
Sympow will be harder to deal with, because of how the spkg is written: it tries to write files to SAGE_LOCAL/lib/sympow. See http://trac.sagemath.org/sage_trac/ticket/9703#comment:9 for a possible fix.
I'm attaching a patch to try to deal with the situation when you run Sage for the very first time as a user without write permissions.
Finally, there may be other issues if you compile Sage but don't run it, and then run doctests as a user without write permissions (the first time doctests get run, they might write some files which don't need to be written later). These issues should be fixed, too.
comment:7 Changed 9 years ago by
I'm not sure this patch is a good idea or needed, considering #11926. If the Sage install was moved, it is probably good to bail out with an error if there is no write access.
comment:8 Changed 9 years ago by
 Dependencies set to #11926
 Description modified (diff)
 Owner changed from mabshoff to jdemeyer
comment:9 Changed 9 years ago by
 Description modified (diff)
comment:10 Changed 9 years ago by
 Description modified (diff)
comment:11 Changed 9 years ago by
 Description modified (diff)
comment:12 Changed 9 years ago by
 Description modified (diff)
comment:13 Changed 9 years ago by
 Component changed from doctest to scripts
 Description modified (diff)
 Milestone changed from sage4.7.2 to sage4.7.3
comment:14 followup: ↓ 15 Changed 9 years ago by
 Cc leif added
A few quick comments:
 you have a typo in line 93, "eyactly"
 on line 583,
# optional  qepcad, note "algeraic" [sic]
, this is not the way to format an optional doctest: it will only run if you do "sage t optionalonly=qepcad,not,algeraic,sic" or something like that.  same on line 1220
For the scripts patch:
 why not apply os.path.abspath to SAGE_ROOT?
comment:15 in reply to: ↑ 14 Changed 9 years ago by
Replying to jhpalmieri:
A few quick comments:
 you have a typo in line 93, "eyactly"
No idea how that happened...
 on line 583,
# optional  qepcad, note "algeraic" [sic]
, this is not the way to format an optional doctest: it will only run if you do "sage t optionalonly=qepcad,not,algeraic,sic" or something like that.
Very true, thanks!
For the scripts patch:
 why not apply os.path.abspath to SAGE_ROOT?
Because SAGE_ROOT
is already canonicalized by realpath
at the top of that file:
SAGE_ROOT = os.path.realpath(os.environ['SAGE_ROOT'])
comment:16 Changed 9 years ago by
 Dependencies changed from #11926 to #11926, #11933
comment:17 Changed 9 years ago by
Moved the "clean up" part of the qepcad patch to #11933.
comment:18 Changed 9 years ago by
 Dependencies changed from #11926, #11933 to #11920, #11926, #11933
 Description modified (diff)
comment:19 Changed 9 years ago by
 Description modified (diff)
For the scripts patch: should there be any error checking when writing to files? If a sysadmin installs Sage, runs it once (to generate the appropriate files) and then moves it but doesn't run it again, it would be nice if other users got helpful error messages. One issue is the code
check_processor_flags() # Note: install_moved() may also run e.g. initialize_pkgconfig_files(). if install_moved(): print "The Sage installation tree may have moved" print "(from %s to %s)." % (OLD_SAGE_ROOT, SAGE_ROOT)
The problem is that check_processor_flags
and install_moved
already try to write to files, so if permissions are bad, the message about the installation tree may not get printed. I'm attaching a patch to apply on top of yours (basically wraps everything in a try...except block).
Also, as discussed at #11760, the alternate implementation (commented out) of searching for SAGE_ROOT using regular expressions was slower than the one currently used, so I just deleted the comments altogether.
I'll keep looking at your patch.
comment:20 followup: ↓ 21 Changed 9 years ago by
John, you are assuming a failure means that there was a problem was permissions. We should probably catch IOError
(and also OSError
perhaps?) and print the message which came with the exception.
comment:21 in reply to: ↑ 20 Changed 9 years ago by
Replying to jdemeyer:
John, you are assuming a failure means that there was a problem was permissions. We should probably catch
IOError
(and alsoOSError
perhaps?) and print the message which came with the exception.
I think that a permissions problem is the most likely issue. I've modified the patch a bit: if it looks like bad permissions, print a friendly error message. If it's an IOError
unrelated to permissions, explicitly raise the error. If it's some other kind of error, don't catch it at all, so it should get raised, too. You can test this by changing my patch: change the line
if e.strerror.find('Permission denied') != 1:
by changing !=
to ==
, or on the previous line, change IOError
to ValueError
or something else irrelevant. Move sage and make the directory unwriteable; then when you run sage, you should see the raw error message.
comment:22 Changed 9 years ago by
John, your patch looks good on first glance.
comment:24 Changed 9 years ago by
 Summary changed from Fix doctests and methods that want write access to $SAGE_LOCAL to Fix doctests that want write access to $SAGE_ROOT
comment:25 Changed 9 years ago by
I plan to come back to this once the dependencies are settled. Otherwise I would just be rebasing pointlessly.
comment:27 Changed 9 years ago by
 Description modified (diff)
 Milestone set to sage4.8
comment:28 Changed 8 years ago by
 Dependencies changed from #11920, #11926, #11933 to #11920, #11926, #11933, #13452
comment:29 Changed 8 years ago by
 Description modified (diff)
comment:30 Changed 8 years ago by
 Description modified (diff)
comment:31 Changed 8 years ago by
 Dependencies changed from #11920, #11926, #11933, #13452 to #11920, #11926, #11933, #13397, #13452
comment:32 Changed 8 years ago by
 Dependencies changed from #11920, #11926, #11933, #13397, #13452 to #11920, #11926, #11933, #13397, #13452, #13459
comment:33 Changed 8 years ago by
 Dependencies changed from #11920, #11926, #11933, #13397, #13452, #13459 to #11920, #11926, #11933, #13397, #13452, #13459, #13457, #13123
comment:34 Changed 8 years ago by
 Dependencies changed from #11920, #11926, #11933, #13397, #13452, #13459, #13457, #13123 to #13157, #13397, #13452, #13459, #13457, #13123
comment:35 Changed 8 years ago by
 Description modified (diff)
comment:36 Changed 8 years ago by
 Description modified (diff)
comment:37 Changed 8 years ago by
 Dependencies changed from #13157, #13397, #13452, #13459, #13457, #13123 to #13157, #13397, #13452, #13407, #13459, #13457, #13123
comment:38 Changed 8 years ago by
 Status changed from needs_work to needs_review
This seems to be fixed with these patches, so I'm putting it to needs_review.
However, this depends on so many other tickets, so it might be good to postpone the review and wait for the other tickets.
Changed 8 years ago by
Changed 8 years ago by
comment:39 Changed 8 years ago by
The Fortran patch needed to be rebased because of #13579. Apart from that, things look fine.
comment:40 Changed 8 years ago by
 Status changed from needs_review to needs_work
Some doctest failures...
Changed 8 years ago by
comment:41 Changed 8 years ago by
 Description modified (diff)
 Status changed from needs_work to needs_review
comment:42 Changed 8 years ago by
 Dependencies changed from #13157, #13397, #13452, #13407, #13459, #13457, #13123 to #13157, #13397, #13452, #13407, #13459, #13457, #13123, #13887
 Description modified (diff)
comment:43 Changed 8 years ago by
SAGE_LOCAL is a strange location for default.qepcadrc I suppose it really needs to be changed in the spkg first but SAGE_LOCAL/etc is a better location for this kind of file.
comment:44 followup: ↓ 45 Changed 8 years ago by
 Reviewers set to François Bissey
 Status changed from needs_review to positive_review
My only objection is the location of default.qepcadrc but since it is for an experimental spkg and would require another ticket to change it I am giving this a positive review. Moving the default.qepcadrc location should be the object of a follow up ticket.
comment:45 in reply to: ↑ 44 Changed 8 years ago by
 Milestone changed from sage5.6 to sage5.7
Replying to fbissey:
My only objection is the location of default.qepcadrc
Also, you shouldn't blame this ticket for that, it has always been like that.
Thanks for the review!
comment:46 Changed 8 years ago by
 Merged in set to sage5.7.beta0
 Resolution set to fixed
 Status changed from positive_review to closed
comment:47 Changed 8 years ago by
I made a patch for the analogous problem for $HOME
at #13985. It would be nice if somebody could review that.
Before applying the patch, I get two doctest failures (with a nonwriteable Sage install). Applying the patch "trac_5155.patch" fixes both. So positive review for that.
The scripts patch wasn't necessary to get those tests to pass, and it also doesn't apply cleanly to 4.3.1:
The changes there look fine in principle, though. So once it's rebased, positive review there, too.