#21785 closed enhancement (fixed)
Install SAGE_SRC/ext/ in package data of sage package by setup.py, not SAGE_LOCAL/share/sage/ext/ by build/make/Makefile
Reported by:  mkoeppe  Owned by:  

Priority:  major  Milestone:  sage9.1 
Component:  build  Keywords:  
Cc:  jdemeyer, embray, fbissey, slelievre, saraedum, isuruf, dimpase, jhpalmieri  Merged in:  
Authors:  Matthias Koeppe  Reviewers:  François Bissey 
Report Upstream:  N/A  Work issues:  
Branch:  fef5a5f (Commits, GitHub, GitLab)  Commit:  
Dependencies:  #29233  Stopgaps: 
Description (last modified by )
A lot of the stuff (not necessarily all) in $SAGE_SRC/ext
belongs directly in the sage Python packagein particular any files that are needed for the package itself to function properly.
A model for that is sage.gaprc
, a file required for the libgap
module to work, which is installed in sage.libs.gap
as "package data" as of #22626.
In this ticket, we install all of ext
as package data of sage
.
 In a follow up ticket, if we identify any files in
$SAGE_SRC/ext
that aren't explicitly required from the package, we will move them to a more appropriate place.
Change History (63)
comment:1 Changed 5 years ago by
 Cc fbissey added
comment:2 Changed 5 years ago by
comment:3 Changed 3 years ago by
 Cc slelievre added
 Milestone changed from sage7.5 to sage8.9
There was a reference to this ticket in a "sagepackaging" discussion:
comment:4 Changed 3 years ago by
I would add, a lot of the stuff (not necessarily all) in $SAGE_SRC/ext
might better belong directly in the sage
Python packagein particular any files that are needed for the package itself to function properly. For example you can see in #22626 that I put sage.gaprc
, a file required for the libgap module to work, in sage.libs.gap
as "package data".
There are several files in $SAGE_SRC/ext
that there's no reason to ever allow to be out of sync with the Python package, and that could be given the same treatment.
For files in $SAGE_SRC/ext
in general, that aren't explicitly required from the package (are there any such files) it might be better to move out of src/
entirely?
comment:5 Changed 2 years ago by
 Milestone changed from sage8.9 to sage9.1
Ticket retargeted after milestone closed
comment:6 Changed 2 years ago by
 Cc saraedum isuruf added
comment:7 Changed 2 years ago by
setuptools experts: To move this ticket forward, should it go through #22655 (Support package_datalike of nonPython resource files in Python packages)?
comment:8 Changed 2 years ago by
Something like it, yes, though I think maybe it can be simplified.
comment:9 Changed 2 years ago by
Erik, should discussion then take place on #22655 or here?
comment:10 Changed 2 years ago by
 Cc dimpase added
 Description modified (diff)
 Summary changed from Installation of SAGE_SRC/ext/ in SAGE_LOCAL/share/sage/ext/ should be done by setup.py, not build/make/Makefile to Install SAGE_SRC/ext/ in package data of sage package by setup.py, not SAGE_LOCAL/share/sage/ext/ by build/make/Makefile
Description adapted from embray's comment 4.
comment:11 Changed 2 years ago by
 Dependencies set to #29062
comment:12 Changed 2 years ago by
 Branch set to u/mkoeppe/install_sage_src_ext__in_package_data_of_sage_package_by_setup_py__not_sage_local_share_sage_ext__by_build_make_makefile
comment:13 Changed 2 years ago by
 Commit set to 06a0226ff6541bdd75ddd40f5158732b7650e29e
Branch is on top of #29062.
Installation of the package data does not work yet... what am I missing?
New commits:
568afdc  build/pkgs/python3/spkginstall: Install valgrindpython.supp directly in SAGE_LOCAL, not first in SAGE_SRC

ceb35bb  src/bin/sagevalgrind: Find python.supp/python3.supp in SAGE_LOCAL/lib/valgrind or system directories

06a0226  Move src/ext to src/sage/ext_data and install it as 'package_data'

comment:14 Changed 2 years ago by
 Commit changed from 06a0226ff6541bdd75ddd40f5158732b7650e29e to 1edc6541e127a15caef36700394dc644954537d8
Branch pushed to git repo; I updated commit sha1. New commits:
1edc654  src/setup.py [package_data]: List all ext_data subdirectories

comment:15 Changed 2 years ago by
 Status changed from new to needs_review
comment:16 Changed 2 years ago by
@embray Also this ticket would need review.
comment:17 Changed 2 years ago by
 Commit changed from 1edc6541e127a15caef36700394dc644954537d8 to 749928c8234605d5977a2a2376e15879b2e0fc2c
comment:18 Changed 2 years ago by
Rebased on top of current #29062 (and 9.1.beta2)
comment:19 Changed 2 years ago by
 Commit changed from 749928c8234605d5977a2a2376e15879b2e0fc2c to 278d68fdd72a6a7cf3be2d8e06634b528e354646
Branch pushed to git repo; I updated commit sha1. New commits:
278d68f  Merge tag '9.1.beta3' into t/21785/install_sage_src_ext__in_package_data_of_sage_package_by_setup_py__not_sage_local_share_sage_ext__by_build_make_makefile

comment:20 Changed 2 years ago by
Merged current beta, needs review
comment:21 Changed 2 years ago by
 Cc jhpalmieri added
comment:22 Changed 2 years ago by
I haven't looked at this in detail yet, but I'm basically happy with what I see; this is what I've been wanting to see done all along.
comment:23 Changed 2 years ago by
 Commit changed from 278d68fdd72a6a7cf3be2d8e06634b528e354646 to 3995b6a7d71373e0c25731d788c53e1837098a98
Branch pushed to git repo; I updated commit sha1. New commits:
3995b6a  Merge tag '9.1.beta4' into t/21785/install_sage_src_ext__in_package_data_of_sage_package_by_setup_py__not_sage_local_share_sage_ext__by_build_make_makefile

comment:24 Changed 2 years ago by
 Dependencies #29062 deleted
comment:25 Changed 2 years ago by
Looks good but there are a few doctests errors on the patchbots and I think at least some of them are really related to this ticket
sage t long warnlong 313.2 src/sage/ext_data/nbconvert/postprocess.py FileNotFoundError in doctesting framework ********************************************************************** Traceback (most recent call last): File "/tmp/tmpy2iq33kqsagegittemp21785/local/lib/python3.7/sitepackages/sage/doctest/forker.py", line 2504, in __call__ doctests, extras = self._run(runner, options, results) File "/tmp/tmpy2iq33kqsagegittemp21785/local/lib/python3.7/sitepackages/sage/doctest/forker.py", line 2549, in _run doctests, extras = self.source.create_doctests(sage_namespace) File "/tmp/tmpy2iq33kqsagegittemp21785/local/lib/python3.7/sitepackages/sage/doctest/sources.py", line 734, in create_doctests load(filename, namespace) # errors raised here will be caught in DocTestTask File "/tmp/tmpy2iq33kqsagegittemp21785/local/lib/python3.7/sitepackages/sage/repl/load.py", line 252, in load exec(code, globals) File "./postprocess.py", line 19, in <module> FileNotFoundError: [Errno 2] No such file or directory: 'p'
Looking at this file, there is no tests in it, which suggests it may not have been tested before and may be nothing in ext_data
should.
This one is actually the exact same error but on another file
sage t long warnlong 313.2 src/sage/doctest/sources.py ********************************************************************** File "src/sage/doctest/sources.py", line 765, in sage.doctest.sources.FileDocTestSource._test_enough_doctests Failed example: for path, dirs, files in itertools.chain(os.walk('sage'), os.walk('doc')): # long time path = os.path.relpath(path) dirs.sort(); files.sort() for F in files: _, ext = os.path.splitext(F) if ext in ('.py', '.pyx', '.pxd', '.pxi', '.sage', '.spyx', '.rst'): filename = os.path.join(path, F) FDS = FileDocTestSource(filename, DocTestDefaults(long=True,optional=True)) FDS._test_enough_doctests(verbose=False) Exception raised: Traceback (most recent call last): File "/tmp/tmpy2iq33kqsagegittemp21785/local/lib/python3.7/sitepackages/sage/doctest/forker.py", line 681, in _run self.compile_and_execute(example, compiler, test.globs) File "/tmp/tmpy2iq33kqsagegittemp21785/local/lib/python3.7/sitepackages/sage/doctest/forker.py", line 1123, in compile_and_execute exec(compiled, globs) File "<doctest sage.doctest.sources.FileDocTestSource._test_enough_doctests[6]>", line 9, in <module> FDS._test_enough_doctests(verbose=False) File "/tmp/tmpy2iq33kqsagegittemp21785/local/lib/python3.7/sitepackages/sage/doctest/sources.py", line 809, in _test_enough_doctests tests, _ = self.create_doctests({}) File "/tmp/tmpy2iq33kqsagegittemp21785/local/lib/python3.7/sitepackages/sage/doctest/sources.py", line 734, in create_doctests load(filename, namespace) # errors raised here will be caught in DocTestTask File "/tmp/tmpy2iq33kqsagegittemp21785/local/lib/python3.7/sitepackages/sage/repl/load.py", line 252, in load exec(code, globals) File "./postprocess.py", line 19, in <module> FileNotFoundError: [Errno 2] No such file or directory: 'p'
Something different
sage t long warnlong 313.2 src/sage/homology/simplicial_set.py ********************************************************************** File "src/sage/homology/simplicial_set.py", line 1679, in sage.homology.simplicial_set.SimplicialSet_arbitrary.graph Failed example: CP3 = simplicial_sets.ComplexProjectiveSpace(3) Exception raised: Traceback (most recent call last): File "/tmp/tmpy2iq33kqsagegittemp21785/local/lib/python3.7/sitepackages/sage/doctest/forker.py", line 681, in _run self.compile_and_execute(example, compiler, test.globs) File "/tmp/tmpy2iq33kqsagegittemp21785/local/lib/python3.7/sitepackages/sage/doctest/forker.py", line 1123, in compile_and_execute exec(compiled, globs) File "<doctest sage.homology.simplicial_set.SimplicialSet_arbitrary.graph[6]>", line 1, in <module> CP3 = simplicial_sets.ComplexProjectiveSpace(Integer(3)) File "/tmp/tmpy2iq33kqsagegittemp21785/local/lib/python3.7/sitepackages/sage/homology/simplicial_set_examples.py", line 612, in ComplexProjectiveSpace data = simplicial_data_from_kenzo_output(file) File "/tmp/tmpy2iq33kqsagegittemp21785/local/lib/python3.7/sitepackages/sage/homology/simplicial_set_examples.py", line 649, in simplicial_data_from_kenzo_output with open(filename, 'r') as f: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpy2iq33kqsagegittemp21785/local/lib/python3.7/sitepackages/sage/ext_data/kenzo/CP3.txt' ********************************************************************** File "src/sage/homology/simplicial_set.py", line 1680, in sage.homology.simplicial_set.SimplicialSet_arbitrary.graph Failed example: G = CP3.graph() Exception raised: Traceback (most recent call last): File "/tmp/tmpy2iq33kqsagegittemp21785/local/lib/python3.7/sitepackages/sage/doctest/forker.py", line 681, in _run self.compile_and_execute(example, compiler, test.globs) File "/tmp/tmpy2iq33kqsagegittemp21785/local/lib/python3.7/sitepackages/sage/doctest/forker.py", line 1123, in compile_and_execute exec(compiled, globs) File "<doctest sage.homology.simplicial_set.SimplicialSet_arbitrary.graph[7]>", line 1, in <module> G = CP3.graph() NameError: name 'CP3' is not defined ********************************************************************** File "src/sage/homology/simplicial_set.py", line 1681, in sage.homology.simplicial_set.SimplicialSet_arbitrary.graph Failed example: len(G.vertices()) Expected: 1 Got: 4 ********************************************************************** File "src/sage/homology/simplicial_set.py", line 1683, in sage.homology.simplicial_set.SimplicialSet_arbitrary.graph Failed example: len(G.edges()) Expected: 0 Got: 6 **********************************************************************
Was a file lost in the move? And again in the same folder
sage t long warnlong 313.2 src/sage/homology/simplicial_set_examples.py ********************************************************************** File "src/sage/homology/simplicial_set_examples.py", line 535, in sage.homology.simplicial_set_examples.ComplexProjectiveSpace Failed example: CP3 = simplicial_sets.ComplexProjectiveSpace(3) Exception raised: Traceback (most recent call last): File "/tmp/tmpy2iq33kqsagegittemp21785/local/lib/python3.7/sitepackages/sage/doctest/forker.py", line 681, in _run self.compile_and_execute(example, compiler, test.globs) File "/tmp/tmpy2iq33kqsagegittemp21785/local/lib/python3.7/sitepackages/sage/doctest/forker.py", line 1123, in compile_and_execute exec(compiled, globs) File "<doctest sage.homology.simplicial_set_examples.ComplexProjectiveSpace[1]>", line 1, in <module> CP3 = simplicial_sets.ComplexProjectiveSpace(Integer(3)) File "/tmp/tmpy2iq33kqsagegittemp21785/local/lib/python3.7/sitepackages/sage/homology/simplicial_set_examples.py", line 612, in ComplexProjectiveSpace data = simplicial_data_from_kenzo_output(file) File "/tmp/tmpy2iq33kqsagegittemp21785/local/lib/python3.7/sitepackages/sage/homology/simplicial_set_examples.py", line 649, in simplicial_data_from_kenzo_output with open(filename, 'r') as f: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpy2iq33kqsagegittemp21785/local/lib/python3.7/sitepackages/sage/ext_data/kenzo/CP3.txt'
File "src/sage/homology/simplicial_set_examples.py", line 551, in sage.homology.simplicial_set_examples.ComplexProjectiveSpace Failed example: simplicial_sets.ComplexProjectiveSpace(4).f_vector() Exception raised: Traceback (most recent call last): File "/tmp/tmpy2iq33kqsagegittemp21785/local/lib/python3.7/sitepackages/sage/doctest/forker.py", line 681, in _run self.compile_and_execute(example, compiler, test.globs) File "/tmp/tmpy2iq33kqsagegittemp21785/local/lib/python3.7/sitepackages/sage/doctest/forker.py", line 1123, in compile_and_execute exec(compiled, globs) File "<doctest sage.homology.simplicial_set_examples.ComplexProjectiveSpace[10]>", line 1, in <module> simplicial_sets.ComplexProjectiveSpace(Integer(4)).f_vector() File "/tmp/tmpy2iq33kqsagegittemp21785/local/lib/python3.7/sitepackages/sage/homology/simplicial_set_examples.py", line 619, in ComplexProjectiveSpace data = simplicial_data_from_kenzo_output(file) File "/tmp/tmpy2iq33kqsagegittemp21785/local/lib/python3.7/sitepackages/sage/homology/simplicial_set_examples.py", line 649, in simplicial_data_from_kenzo_output with open(filename, 'r') as f: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpy2iq33kqsagegittemp21785/local/lib/python3.7/sitepackages/sage/ext_data/kenzo/CP4.txt'
File "src/sage/homology/simplicial_set_examples.py", line 646, in sage.homology.simplicial_set_examples.simplicial_data_from_kenzo_output Failed example: S4.homology(reduced=False) Exception raised: Traceback (most recent call last): File "/tmp/tmpy2iq33kqsagegittemp21785/local/lib/python3.7/sitepackages/sage/doctest/forker.py", line 681, in _run self.compile_and_execute(example, compiler, test.globs) File "/tmp/tmpy2iq33kqsagegittemp21785/local/lib/python3.7/sitepackages/sage/doctest/forker.py", line 1123, in compile_and_execute exec(compiled, globs) File "<doctest sage.homology.simplicial_set_examples.simplicial_data_from_kenzo_output[4]>", line 1, in <module> S4.homology(reduced=False) NameError: name 'S4' is not defined
comment:26 Changed 2 years ago by
I have a felling the .txt
in the kenso
folder are not actually installed. At least they may not have been on the patchbot I have taken these logs from.
comment:27 Changed 2 years ago by
Yes, it looks like they are not getting installed. I'll investigate
comment:28 Changed 2 years ago by
 Commit changed from 3995b6a7d71373e0c25731d788c53e1837098a98 to ef3c7d33850a0e05b62aa05b7cf50eb9693e46c1
Branch pushed to git repo; I updated commit sha1. New commits:
ef3c7d3  Fix install of ext_data/kenzo

comment:29 Changed 2 years ago by
Ok, the problem with kenzo
was a simple typo.
comment:30 Changed 2 years ago by
 Commit changed from ef3c7d33850a0e05b62aa05b7cf50eb9693e46c1 to 6967bc13f62fd8e18d7c0aa8a1b10c6ba2bca924
Branch pushed to git repo; I updated commit sha1. New commits:
6967bc1  Mark src/sage/ext_data to be skipped in doctesting

comment:31 Changed 2 years ago by
I am trying to find if there was a reason it was named nodotest.py
(specifically the .py
part), this is an annoyance at install time, at least on sageongentoo, for no good reason when examined closely.
comment:32 Changed 2 years ago by
 Status changed from needs_review to needs_work
Sorry for the noise, I want to trigger the patchbot.
comment:33 Changed 2 years ago by
 Status changed from needs_work to needs_review
comment:34 Changed 2 years ago by
Somewhat incidental, and doing it in another ticket would be fine. src/sage/doctest/tests
is currently not installed because there is no __init__.py
 at least on distros. But it contains a nodoctest.py
which is tested for. So it should really be added to package_data
as well.
comment:35 Changed 2 years ago by
 Commit changed from 6967bc13f62fd8e18d7c0aa8a1b10c6ba2bca924 to 2c13f048a566401d0f4bd0479cb9cefe4eabe564
Branch pushed to git repo; I updated commit sha1. New commits:
2c13f04  src/setup.py: Add sage/doctest/tests to package_data

comment:36 Changed 2 years ago by
Next, to make it work, the cleaner of stale files needs to be tamed:
[sagelib9.1.beta4] Cleaning up stale file: /Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/local/lib/python3.7/sitepackages/sage/ext_data/__pycache__/nodoctest.cpython37.pyc [sagelib9.1.beta4] Cleaning up stale file: /Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/local/lib/python3.7/sitepackages/sage/ext_data/nodoctest.py [sagelib9.1.beta4] Cleaning up stale file: /Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/local/lib/python3.7/sitepackages/sage/ext_data/nbconvert/postprocess.py [sagelib9.1.beta4] Cleaning up stale file: /Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/local/lib/python3.7/sitepackages/sage/ext_data/nbconvert/__pycache__/postprocess.cpython37.pyc [sagelib9.1.beta4] Cleaning up stale file: /Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/local/lib/python3.7/sitepackages/sage/doctest/tests/__pycache__/nodoctest.cpython37.pyc [sagelib9.1.beta4] Cleaning up stale file: /Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/local/lib/python3.7/sitepackages/sage/doctest/tests/nodoctest.py [sagelib9.1.beta4]  cleaning build/lib.macosx10.9x86_643.7 [sagelib9.1.beta4] Cleaning up stale file: build/lib.macosx10.9x86_643.7/sage/ext_data/nodoctest.py [sagelib9.1.beta4] Cleaning up stale file: build/lib.macosx10.9x86_643.7/sage/ext_data/nbconvert/postprocess.py [sagelib9.1.beta4] Cleaning up stale file: build/lib.macosx10.9x86_643.7/sage/doctest/tests/nodoctest.py [sagelib9.1.beta4] Finished cleaning, time: 0.25 seconds.
comment:37 Changed 2 years ago by
OK I have somewhat asked on #14575 if there was any reason it was nodoctest.py
 in particular the extension. I think we should have a separate ticket for this but the problem of __pycache__
with nodoctest
would be eliminated. Still leave us with postprocess.py
but at least, that one is genuine.
comment:38 Changed 2 years ago by
I'm working on the cleaner.
comment:39 Changed 2 years ago by
 Commit changed from 2c13f048a566401d0f4bd0479cb9cefe4eabe564 to 86e20791c7a468b7196f844de9741ce5cc97ce20
Branch pushed to git repo; I updated commit sha1. New commits:
86e2079  src/setup.py, src/sage_setup/clean.py: Clean carefully around package_data

comment:40 Changed 2 years ago by
Now it looks like this:
[sagelib9.1.beta4] Cleaning up stale installed files.... [sagelib9.1.beta4]  cleaning /Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/local/lib/python3.7/sitepackages [sagelib9.1.beta4] Cleaning up stale file: /Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/local/lib/python3.7/sitepackages/sage/ext_data/__pycache__/nodoctest.cpython37.pyc [sagelib9.1.beta4] Cleaning up stale file: /Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/local/lib/python3.7/sitepackages/sage/ext_data/nbconvert/__pycache__/postprocess.cpython37.pyc [sagelib9.1.beta4] Cleaning up stale file: /Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/local/lib/python3.7/sitepackages/sage/doctest/tests/__pycache__/nodoctest.cpython37.pyc [sagelib9.1.beta4]  cleaning build/lib.macosx10.9x86_643.7
comment:41 Changed 2 years ago by
New round of patchbot
sage t long warnlong 286.7 src/sage/rings/function_field/function_field.py # Timed out sage t long warnlong 286.7 src/sage/doctest/sources.py # 1 doctest failed sage t long warnlong 286.7 src/sage_setup/clean.py # 1 doctest failed
The time out is strange but was already there in the previous round so there may be something about it (but what?). The testing in doctest/sources.py
is still present even so we don't test postprocess.py
anymore so there is something more to do somewhere in there.
And lastly we have some stale files  not unexpectedly at this stage
sage t long warnlong 286.7 src/sage_setup/clean.py ********************************************************************** File "src/sage_setup/clean.py", line 100, in sage_setup.clean._find_stale_files Failed example: for f in stale_iter: if f.endswith(skip_extensions): continue print('Found stale file: ' + f) Expected nothing Got: Found stale file: sage/doctest/tests/nodoctest.py Found stale file: sage/ext_data/nodoctest.py Found stale file: sage/ext_data/nbconvert/postprocess.py
comment:42 Changed 2 years ago by
 Commit changed from 86e20791c7a468b7196f844de9741ce5cc97ce20 to 50a5d2587c1a003327dd844bbf15ad4e98011666
comment:43 Changed 2 years ago by
comment:44 Changed 2 years ago by
This doctest works without problem on my machine:
sage t long src/sage/rings/function_field/function_field.py [862 tests, 220.61 s]
comment:45 Changed 2 years ago by
There is one strange thing left:
running install running install_lib bytecompiling /Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/local/lib/python3.7/sitepackages/sage/ext_data/nbconvert/postprocess.py to postprocess.cpython37.pyc running install_egg_info Removing /Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/local/lib/python3.7/sitepackages/sage9.1.beta4py3.7.egginfo Writing /Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/local/lib/python3.7/sitepackages/sage9.1.beta4py3.7.egginfo Cleaning up stale installed files....  cleaning build/lib.macosx10.9x86_643.7  cleaning /Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/local/lib/python3.7/sitepackages Cleaning up stale file: /Users/mkoeppe/s/sage/sagerebasing/worktreealgebraic2018spring/local/lib/python3.7/sitepackages/sage/ext_data/nbconvert/__pycache__/postprocess.cpython37.pyc Finished cleaning, time: 0.23 seconds.
comment:46 Changed 2 years ago by
Real cool! Everything in the one ticket, the nodoctest stuff means that I can ditch packaging code I had to add because of QA complaints in Gentoo.
Which bit do you find strange? the stale file in ext_data
? It looks like when you install stuff with distutils all python file are bytecompiled by default, including the ones in package_data
(which I wasn't expecting but this is probably a blanket scan of what's installed).
I think this is ready for positive review.
comment:47 Changed 2 years ago by
Note that the bytecompiled file there is deleted by the cleaner 1 second later. This could be fixed but it's not essential.
comment:48 Changed 2 years ago by
 Reviewers set to François Bissey
 Status changed from needs_review to positive_review
I'll send it to the bots as is.
comment:49 Changed 2 years ago by
Thanks for reviewing!
comment:50 Changed 2 years ago by
 Status changed from positive_review to needs_work
I'm still seeing this on the buildbot:
********************************************************************** File "src/sage/doctest/sources.py", line 765, in sage.doctest.sources.FileDocTestSource._test_enough_doctests Failed example: for path, dirs, files in itertools.chain(os.walk('sage'), os.walk('doc')): # long time path = os.path.relpath(path) dirs.sort(); files.sort() for F in files: _, ext = os.path.splitext(F) if ext in ('.py', '.pyx', '.pxd', '.pxi', '.sage', '.spyx', '.rst'): filename = os.path.join(path, F) FDS = FileDocTestSource(filename, DocTestDefaults(long=True,optional=True)) FDS._test_enough_doctests(verbose=False) Exception raised: Traceback (most recent call last): File "/Users/buildbotsage/slave/sage_git/build/local/lib/python3.7/sitepackages/sage/doctest/forker.py", line 681, in _run self.compile_and_execute(example, compiler, test.globs) File "/Users/buildbotsage/slave/sage_git/build/local/lib/python3.7/sitepackages/sage/doctest/forker.py", line 1123, in compile_and_execute exec(compiled, globs) File "<doctest sage.doctest.sources.FileDocTestSource._test_enough_doctests[6]>", line 9, in <module> FDS._test_enough_doctests(verbose=False) File "/Users/buildbotsage/slave/sage_git/build/local/lib/python3.7/sitepackages/sage/doctest/sources.py", line 809, in _test_enough_doctests tests, _ = self.create_doctests({}) File "/Users/buildbotsage/slave/sage_git/build/local/lib/python3.7/sitepackages/sage/doctest/sources.py", line 734, in create_doctests load(filename, namespace) # errors raised here will be caught in DocTestTask File "/Users/buildbotsage/slave/sage_git/build/local/lib/python3.7/sitepackages/sage/repl/load.py", line 252, in load exec(code, globals) File "./postprocess.py", line 19, in <module> FileNotFoundError: [Errno 2] No such file or directory: 'p' ********************************************************************** 1 item had failures: 1 of 9 in sage.doctest.sources.FileDocTestSource._test_enough_doctests [367 tests, 1 failure, 26.05 s] 1 memlimit not run  sage t long src/sage/doctest/sources.py # 1 doctest failed 
And this one on my machine:
[dochtml] [installat] The HTML pages are in local/share/doc/sage/html/en/installation. [dochtml] Error building the documentation. [dochtml] Traceback (most recent call last): [dochtml] File "/home/release/Sage/local/lib/python2.7/runpy.py", line 174, in _run_module_as_main [dochtml] "__main__", fname, loader, pkg_name) [dochtml] File "/home/release/Sage/local/lib/python2.7/runpy.py", line 72, in _run_code [dochtml] exec code in run_globals [dochtml] File "/home/release/Sage/local/lib/python2.7/sitepackages/sage_setup/docbuild/__main__.py", line 2, in <module> [dochtml] main() [dochtml] File "/home/release/Sage/local/lib/python2.7/sitepackages/sage_setup/docbuild/__init__.py", line 1720, in main [dochtml] builder() [dochtml] File "/home/release/Sage/local/lib/python2.7/sitepackages/sage_setup/docbuild/__init__.py", line 336, in _wrapper [dochtml] build_many(build_other_doc, L) [dochtml] File "/home/release/Sage/local/lib/python2.7/sitepackages/sage_setup/docbuild/__init__.py", line 280, in build_many [dochtml] _build_many(target, args, processes=NUM_THREADS) [dochtml] File "/home/release/Sage/local/lib/python2.7/sitepackages/sage_setup/docbuild/utils.py", line 283, in build_many [dochtml] raise worker_exc.original_exception [dochtml] OSError: /home/release/Sage/src/doc/en/installation/source.rst:228: WARNING: Include file u'/home/release/Sage/src/doc/en/installation/debian.txt' not found or reading it failed
I'm guessing this ticket has issues with incremental builds, can you give this a try?
comment:51 Changed 2 years ago by
I made #29233 for the documentation build issue
comment:52 Changed 2 years ago by
 Dependencies set to #29233
comment:53 Changed 2 years ago by
 Commit changed from 50a5d2587c1a003327dd844bbf15ad4e98011666 to 51e7f51332390708148f107a104f48d92c7aa5d3
Branch pushed to git repo; I updated commit sha1. New commits:
0dc97db  Clean src/doc/en/installation/*.txt in bootstrapclean, not miscclean

51e7f51  Merge branch 't/29233/bootstrap____make_distclean_broken' into t/21785/install_sage_src_ext__in_package_data_of_sage_package_by_setup_py__not_sage_local_share_sage_ext__by_build_make_makefile

comment:54 Changed 2 years ago by
 Status changed from needs_work to needs_review
I don't know how to reproduce this error:
FileNotFoundError: [Errno 2] No such file or directory: 'p'
comment:55 Changed 2 years ago by
Ic ouldn't reproduce it either. I suspect this is an incremental build issue.
comment:56 Changed 2 years ago by
 Status changed from needs_review to needs_work
comment:57 Changed 2 years ago by
 Commit changed from 51e7f51332390708148f107a104f48d92c7aa5d3 to fef5a5f69cecea754eadecc2e4ae94a7903f68d6
Branch pushed to git repo; I updated commit sha1. New commits:
fef5a5f  FileDocTestSource._test_enough_doctest doctest: Do not load .py files that are not a sage module

comment:58 Changed 2 years ago by
 Status changed from needs_work to needs_review
Found it, fixed. Needs review
comment:59 Changed 2 years ago by
 Status changed from needs_review to positive_review
OK let's try it again.
comment:60 Changed 23 months ago by
 Branch changed from u/mkoeppe/install_sage_src_ext__in_package_data_of_sage_package_by_setup_py__not_sage_local_share_sage_ext__by_build_make_makefile to fef5a5f69cecea754eadecc2e4ae94a7903f68d6
 Resolution set to fixed
 Status changed from positive_review to closed
comment:61 Changed 23 months ago by
 Commit fef5a5f69cecea754eadecc2e4ae94a7903f68d6 deleted
Could you check that .nodoctest
files are properly installed in vanilla sage? Because vanilla sage runs doctests on original sage sources (while in sageongentoo it is run on the install because the sources are not available at runtime) it may have been missed.
comment:62 Changed 23 months ago by
This ticket messes up anyone with data in the old src/ext
directory. Would have been nice to have some advance notice on sagedevel, you know?
comment:63 Changed 23 months ago by
Let me know if you need help with resolving a merge conflict.
That would actually help me. I see extra potential to this for my jupyter kernel install woes as well (even if I know Jeroen doesn't see it that way).