#21785 closed enhancement (fixed)
Install SAGE_SRC/ext/ in package data of sage package by setup.py, not SAGE_LOCAL/share/sage/ext/ by build/make/Makefile
Reported by: | mkoeppe | Owned by: | |
---|---|---|---|
Priority: | major | Milestone: | sage-9.1 |
Component: | build | Keywords: | |
Cc: | jdemeyer, embray, fbissey, slelievre, saraedum, isuruf, dimpase, jhpalmieri | Merged in: | |
Authors: | Matthias Koeppe | Reviewers: | François Bissey |
Report Upstream: | N/A | Work issues: | |
Branch: | fef5a5f (Commits) | Commit: | |
Dependencies: | #29233 | Stopgaps: |
Description (last modified by )
A lot of the stuff (not necessarily all) in $SAGE_SRC/ext
belongs directly in the sage Python package--in particular any files that are needed for the package itself to function properly.
A model for that is sage.gaprc
, a file required for the libgap
module to work, which is installed in sage.libs.gap
as "package data" as of #22626.
In this ticket, we install all of ext
as package data of sage
.
- In a follow up ticket, if we identify any files in
$SAGE_SRC/ext
that aren't explicitly required from the package, we will move them to a more appropriate place.
Change History (63)
comment:1 Changed 4 years ago by
- Cc fbissey added
comment:2 Changed 4 years ago by
comment:3 Changed 19 months ago by
- Cc slelievre added
- Milestone changed from sage-7.5 to sage-8.9
There was a reference to this ticket in a "sage-packaging" discussion:
comment:4 Changed 19 months ago by
I would add, a lot of the stuff (not necessarily all) in $SAGE_SRC/ext
might better belong directly in the sage
Python package--in particular any files that are needed for the package itself to function properly. For example you can see in #22626 that I put sage.gaprc
, a file required for the libgap module to work, in sage.libs.gap
as "package data".
There are several files in $SAGE_SRC/ext
that there's no reason to ever allow to be out of sync with the Python package, and that could be given the same treatment.
For files in $SAGE_SRC/ext
in general, that aren't explicitly required from the package (are there any such files) it might be better to move out of src/
entirely?
comment:5 Changed 13 months ago by
- Milestone changed from sage-8.9 to sage-9.1
Ticket retargeted after milestone closed
comment:6 Changed 13 months ago by
- Cc saraedum isuruf added
comment:7 Changed 13 months ago by
setuptools experts: To move this ticket forward, should it go through #22655 (Support package_data-like of non-Python resource files in Python packages)?
comment:8 Changed 13 months ago by
Something like it, yes, though I think maybe it can be simplified.
comment:9 Changed 12 months ago by
Erik, should discussion then take place on #22655 or here?
comment:10 Changed 12 months ago by
- Cc dimpase added
- Description modified (diff)
- Summary changed from Installation of SAGE_SRC/ext/ in SAGE_LOCAL/share/sage/ext/ should be done by setup.py, not build/make/Makefile to Install SAGE_SRC/ext/ in package data of sage package by setup.py, not SAGE_LOCAL/share/sage/ext/ by build/make/Makefile
Description adapted from embray's comment 4.
comment:11 Changed 12 months ago by
- Dependencies set to #29062
comment:12 Changed 12 months ago by
- Branch set to u/mkoeppe/install_sage_src_ext__in_package_data_of_sage_package_by_setup_py__not_sage_local_share_sage_ext__by_build_make_makefile
comment:13 Changed 12 months ago by
- Commit set to 06a0226ff6541bdd75ddd40f5158732b7650e29e
Branch is on top of #29062.
Installation of the package data does not work yet... what am I missing?
New commits:
568afdc | build/pkgs/python3/spkg-install: Install valgrind-python.supp directly in SAGE_LOCAL, not first in SAGE_SRC
|
ceb35bb | src/bin/sage-valgrind: Find python.supp/python3.supp in SAGE_LOCAL/lib/valgrind or system directories
|
06a0226 | Move src/ext to src/sage/ext_data and install it as 'package_data'
|
comment:14 Changed 12 months ago by
- Commit changed from 06a0226ff6541bdd75ddd40f5158732b7650e29e to 1edc6541e127a15caef36700394dc644954537d8
Branch pushed to git repo; I updated commit sha1. New commits:
1edc654 | src/setup.py [package_data]: List all ext_data subdirectories
|
comment:15 Changed 12 months ago by
- Status changed from new to needs_review
comment:16 Changed 12 months ago by
@embray Also this ticket would need review.
comment:17 Changed 12 months ago by
- Commit changed from 1edc6541e127a15caef36700394dc644954537d8 to 749928c8234605d5977a2a2376e15879b2e0fc2c
comment:18 Changed 12 months ago by
Rebased on top of current #29062 (and 9.1.beta2)
comment:19 Changed 12 months ago by
- Commit changed from 749928c8234605d5977a2a2376e15879b2e0fc2c to 278d68fdd72a6a7cf3be2d8e06634b528e354646
Branch pushed to git repo; I updated commit sha1. New commits:
278d68f | Merge tag '9.1.beta3' into t/21785/install_sage_src_ext__in_package_data_of_sage_package_by_setup_py__not_sage_local_share_sage_ext__by_build_make_makefile
|
comment:20 Changed 12 months ago by
Merged current beta, needs review
comment:21 Changed 12 months ago by
- Cc jhpalmieri added
comment:22 Changed 12 months ago by
I haven't looked at this in detail yet, but I'm basically happy with what I see; this is what I've been wanting to see done all along.
comment:23 Changed 11 months ago by
- Commit changed from 278d68fdd72a6a7cf3be2d8e06634b528e354646 to 3995b6a7d71373e0c25731d788c53e1837098a98
Branch pushed to git repo; I updated commit sha1. New commits:
3995b6a | Merge tag '9.1.beta4' into t/21785/install_sage_src_ext__in_package_data_of_sage_package_by_setup_py__not_sage_local_share_sage_ext__by_build_make_makefile
|
comment:24 Changed 11 months ago by
- Dependencies #29062 deleted
comment:25 Changed 11 months ago by
Looks good but there are a few doctests errors on the patchbots and I think at least some of them are really related to this ticket
sage -t --long --warn-long 313.2 src/sage/ext_data/nbconvert/postprocess.py FileNotFoundError in doctesting framework ********************************************************************** Traceback (most recent call last): File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 2504, in __call__ doctests, extras = self._run(runner, options, results) File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 2549, in _run doctests, extras = self.source.create_doctests(sage_namespace) File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/sources.py", line 734, in create_doctests load(filename, namespace) # errors raised here will be caught in DocTestTask File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/repl/load.py", line 252, in load exec(code, globals) File "./postprocess.py", line 19, in <module> FileNotFoundError: [Errno 2] No such file or directory: '-p'
Looking at this file, there is no tests in it, which suggests it may not have been tested before and may be nothing in ext_data
should.
This one is actually the exact same error but on another file
sage -t --long --warn-long 313.2 src/sage/doctest/sources.py ********************************************************************** File "src/sage/doctest/sources.py", line 765, in sage.doctest.sources.FileDocTestSource._test_enough_doctests Failed example: for path, dirs, files in itertools.chain(os.walk('sage'), os.walk('doc')): # long time path = os.path.relpath(path) dirs.sort(); files.sort() for F in files: _, ext = os.path.splitext(F) if ext in ('.py', '.pyx', '.pxd', '.pxi', '.sage', '.spyx', '.rst'): filename = os.path.join(path, F) FDS = FileDocTestSource(filename, DocTestDefaults(long=True,optional=True)) FDS._test_enough_doctests(verbose=False) Exception raised: Traceback (most recent call last): File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run self.compile_and_execute(example, compiler, test.globs) File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute exec(compiled, globs) File "<doctest sage.doctest.sources.FileDocTestSource._test_enough_doctests[6]>", line 9, in <module> FDS._test_enough_doctests(verbose=False) File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/sources.py", line 809, in _test_enough_doctests tests, _ = self.create_doctests({}) File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/sources.py", line 734, in create_doctests load(filename, namespace) # errors raised here will be caught in DocTestTask File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/repl/load.py", line 252, in load exec(code, globals) File "./postprocess.py", line 19, in <module> FileNotFoundError: [Errno 2] No such file or directory: '-p'
Something different
sage -t --long --warn-long 313.2 src/sage/homology/simplicial_set.py ********************************************************************** File "src/sage/homology/simplicial_set.py", line 1679, in sage.homology.simplicial_set.SimplicialSet_arbitrary.graph Failed example: CP3 = simplicial_sets.ComplexProjectiveSpace(3) Exception raised: Traceback (most recent call last): File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run self.compile_and_execute(example, compiler, test.globs) File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute exec(compiled, globs) File "<doctest sage.homology.simplicial_set.SimplicialSet_arbitrary.graph[6]>", line 1, in <module> CP3 = simplicial_sets.ComplexProjectiveSpace(Integer(3)) File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/homology/simplicial_set_examples.py", line 612, in ComplexProjectiveSpace data = simplicial_data_from_kenzo_output(file) File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/homology/simplicial_set_examples.py", line 649, in simplicial_data_from_kenzo_output with open(filename, 'r') as f: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/ext_data/kenzo/CP3.txt' ********************************************************************** File "src/sage/homology/simplicial_set.py", line 1680, in sage.homology.simplicial_set.SimplicialSet_arbitrary.graph Failed example: G = CP3.graph() Exception raised: Traceback (most recent call last): File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run self.compile_and_execute(example, compiler, test.globs) File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute exec(compiled, globs) File "<doctest sage.homology.simplicial_set.SimplicialSet_arbitrary.graph[7]>", line 1, in <module> G = CP3.graph() NameError: name 'CP3' is not defined ********************************************************************** File "src/sage/homology/simplicial_set.py", line 1681, in sage.homology.simplicial_set.SimplicialSet_arbitrary.graph Failed example: len(G.vertices()) Expected: 1 Got: 4 ********************************************************************** File "src/sage/homology/simplicial_set.py", line 1683, in sage.homology.simplicial_set.SimplicialSet_arbitrary.graph Failed example: len(G.edges()) Expected: 0 Got: 6 **********************************************************************
Was a file lost in the move? And again in the same folder
sage -t --long --warn-long 313.2 src/sage/homology/simplicial_set_examples.py ********************************************************************** File "src/sage/homology/simplicial_set_examples.py", line 535, in sage.homology.simplicial_set_examples.ComplexProjectiveSpace Failed example: CP3 = simplicial_sets.ComplexProjectiveSpace(3) Exception raised: Traceback (most recent call last): File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run self.compile_and_execute(example, compiler, test.globs) File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute exec(compiled, globs) File "<doctest sage.homology.simplicial_set_examples.ComplexProjectiveSpace[1]>", line 1, in <module> CP3 = simplicial_sets.ComplexProjectiveSpace(Integer(3)) File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/homology/simplicial_set_examples.py", line 612, in ComplexProjectiveSpace data = simplicial_data_from_kenzo_output(file) File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/homology/simplicial_set_examples.py", line 649, in simplicial_data_from_kenzo_output with open(filename, 'r') as f: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/ext_data/kenzo/CP3.txt'
File "src/sage/homology/simplicial_set_examples.py", line 551, in sage.homology.simplicial_set_examples.ComplexProjectiveSpace Failed example: simplicial_sets.ComplexProjectiveSpace(4).f_vector() Exception raised: Traceback (most recent call last): File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run self.compile_and_execute(example, compiler, test.globs) File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute exec(compiled, globs) File "<doctest sage.homology.simplicial_set_examples.ComplexProjectiveSpace[10]>", line 1, in <module> simplicial_sets.ComplexProjectiveSpace(Integer(4)).f_vector() File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/homology/simplicial_set_examples.py", line 619, in ComplexProjectiveSpace data = simplicial_data_from_kenzo_output(file) File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/homology/simplicial_set_examples.py", line 649, in simplicial_data_from_kenzo_output with open(filename, 'r') as f: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/ext_data/kenzo/CP4.txt'
File "src/sage/homology/simplicial_set_examples.py", line 646, in sage.homology.simplicial_set_examples.simplicial_data_from_kenzo_output Failed example: S4.homology(reduced=False) Exception raised: Traceback (most recent call last): File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run self.compile_and_execute(example, compiler, test.globs) File "/tmp/tmpy2iq33kq-sage-git-temp-21785/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute exec(compiled, globs) File "<doctest sage.homology.simplicial_set_examples.simplicial_data_from_kenzo_output[4]>", line 1, in <module> S4.homology(reduced=False) NameError: name 'S4' is not defined
comment:26 Changed 11 months ago by
I have a felling the .txt
in the kenso
folder are not actually installed. At least they may not have been on the patchbot I have taken these logs from.
comment:27 Changed 11 months ago by
Yes, it looks like they are not getting installed. I'll investigate
comment:28 Changed 11 months ago by
- Commit changed from 3995b6a7d71373e0c25731d788c53e1837098a98 to ef3c7d33850a0e05b62aa05b7cf50eb9693e46c1
Branch pushed to git repo; I updated commit sha1. New commits:
ef3c7d3 | Fix install of ext_data/kenzo
|
comment:29 Changed 11 months ago by
Ok, the problem with kenzo
was a simple typo.
comment:30 Changed 11 months ago by
- Commit changed from ef3c7d33850a0e05b62aa05b7cf50eb9693e46c1 to 6967bc13f62fd8e18d7c0aa8a1b10c6ba2bca924
Branch pushed to git repo; I updated commit sha1. New commits:
6967bc1 | Mark src/sage/ext_data to be skipped in doctesting
|
comment:31 Changed 11 months ago by
I am trying to find if there was a reason it was named nodotest.py
(specifically the .py
part), this is an annoyance at install time, at least on sage-on-gentoo, for no good reason when examined closely.
comment:32 Changed 11 months ago by
- Status changed from needs_review to needs_work
Sorry for the noise, I want to trigger the patchbot.
comment:33 Changed 11 months ago by
- Status changed from needs_work to needs_review
comment:34 Changed 11 months ago by
Somewhat incidental, and doing it in another ticket would be fine. src/sage/doctest/tests
is currently not installed because there is no __init__.py
- at least on distros. But it contains a nodoctest.py
which is tested for. So it should really be added to package_data
as well.
comment:35 Changed 11 months ago by
- Commit changed from 6967bc13f62fd8e18d7c0aa8a1b10c6ba2bca924 to 2c13f048a566401d0f4bd0479cb9cefe4eabe564
Branch pushed to git repo; I updated commit sha1. New commits:
2c13f04 | src/setup.py: Add sage/doctest/tests to package_data
|
comment:36 Changed 11 months ago by
Next, to make it work, the cleaner of stale files needs to be tamed:
[sagelib-9.1.beta4] Cleaning up stale file: /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages/sage/ext_data/__pycache__/nodoctest.cpython-37.pyc [sagelib-9.1.beta4] Cleaning up stale file: /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages/sage/ext_data/nodoctest.py [sagelib-9.1.beta4] Cleaning up stale file: /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages/sage/ext_data/nbconvert/postprocess.py [sagelib-9.1.beta4] Cleaning up stale file: /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages/sage/ext_data/nbconvert/__pycache__/postprocess.cpython-37.pyc [sagelib-9.1.beta4] Cleaning up stale file: /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages/sage/doctest/tests/__pycache__/nodoctest.cpython-37.pyc [sagelib-9.1.beta4] Cleaning up stale file: /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages/sage/doctest/tests/nodoctest.py [sagelib-9.1.beta4] - cleaning build/lib.macosx-10.9-x86_64-3.7 [sagelib-9.1.beta4] Cleaning up stale file: build/lib.macosx-10.9-x86_64-3.7/sage/ext_data/nodoctest.py [sagelib-9.1.beta4] Cleaning up stale file: build/lib.macosx-10.9-x86_64-3.7/sage/ext_data/nbconvert/postprocess.py [sagelib-9.1.beta4] Cleaning up stale file: build/lib.macosx-10.9-x86_64-3.7/sage/doctest/tests/nodoctest.py [sagelib-9.1.beta4] Finished cleaning, time: 0.25 seconds.
comment:37 Changed 11 months ago by
OK I have somewhat asked on #14575 if there was any reason it was nodoctest.py
- in particular the extension. I think we should have a separate ticket for this but the problem of __pycache__
with nodoctest
would be eliminated. Still leave us with postprocess.py
but at least, that one is genuine.
comment:38 Changed 11 months ago by
I'm working on the cleaner.
comment:39 Changed 11 months ago by
- Commit changed from 2c13f048a566401d0f4bd0479cb9cefe4eabe564 to 86e20791c7a468b7196f844de9741ce5cc97ce20
Branch pushed to git repo; I updated commit sha1. New commits:
86e2079 | src/setup.py, src/sage_setup/clean.py: Clean carefully around package_data
|
comment:40 Changed 11 months ago by
Now it looks like this:
[sagelib-9.1.beta4] Cleaning up stale installed files.... [sagelib-9.1.beta4] - cleaning /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages [sagelib-9.1.beta4] Cleaning up stale file: /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages/sage/ext_data/__pycache__/nodoctest.cpython-37.pyc [sagelib-9.1.beta4] Cleaning up stale file: /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages/sage/ext_data/nbconvert/__pycache__/postprocess.cpython-37.pyc [sagelib-9.1.beta4] Cleaning up stale file: /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages/sage/doctest/tests/__pycache__/nodoctest.cpython-37.pyc [sagelib-9.1.beta4] - cleaning build/lib.macosx-10.9-x86_64-3.7
comment:41 Changed 11 months ago by
New round of patchbot
sage -t --long --warn-long 286.7 src/sage/rings/function_field/function_field.py # Timed out sage -t --long --warn-long 286.7 src/sage/doctest/sources.py # 1 doctest failed sage -t --long --warn-long 286.7 src/sage_setup/clean.py # 1 doctest failed
The time out is strange but was already there in the previous round so there may be something about it (but what?). The testing in doctest/sources.py
is still present even so we don't test postprocess.py
anymore so there is something more to do somewhere in there.
And lastly we have some stale files - not unexpectedly at this stage
sage -t --long --warn-long 286.7 src/sage_setup/clean.py ********************************************************************** File "src/sage_setup/clean.py", line 100, in sage_setup.clean._find_stale_files Failed example: for f in stale_iter: if f.endswith(skip_extensions): continue print('Found stale file: ' + f) Expected nothing Got: Found stale file: sage/doctest/tests/nodoctest.py Found stale file: sage/ext_data/nodoctest.py Found stale file: sage/ext_data/nbconvert/postprocess.py
comment:42 Changed 11 months ago by
- Commit changed from 86e20791c7a468b7196f844de9741ce5cc97ce20 to 50a5d2587c1a003327dd844bbf15ad4e98011666
comment:43 Changed 11 months ago by
comment:44 Changed 11 months ago by
This doctest works without problem on my machine:
sage -t --long src/sage/rings/function_field/function_field.py [862 tests, 220.61 s]
comment:45 Changed 11 months ago by
There is one strange thing left:
running install running install_lib byte-compiling /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages/sage/ext_data/nbconvert/postprocess.py to postprocess.cpython-37.pyc running install_egg_info Removing /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages/sage-9.1.beta4-py3.7.egg-info Writing /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages/sage-9.1.beta4-py3.7.egg-info Cleaning up stale installed files.... - cleaning build/lib.macosx-10.9-x86_64-3.7 - cleaning /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages Cleaning up stale file: /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages/sage/ext_data/nbconvert/__pycache__/postprocess.cpython-37.pyc Finished cleaning, time: 0.23 seconds.
comment:46 Changed 11 months ago by
Real cool! Everything in the one ticket, the nodoctest stuff means that I can ditch packaging code I had to add because of QA complaints in Gentoo.
Which bit do you find strange? the stale file in ext_data
? It looks like when you install stuff with distutils all python file are byte-compiled by default, including the ones in package_data
(which I wasn't expecting but this is probably a blanket scan of what's installed).
I think this is ready for positive review.
comment:47 Changed 11 months ago by
Note that the byte-compiled file there is deleted by the cleaner 1 second later. This could be fixed but it's not essential.
comment:48 Changed 11 months ago by
- Reviewers set to François Bissey
- Status changed from needs_review to positive_review
I'll send it to the bots as is.
comment:49 Changed 11 months ago by
Thanks for reviewing!
comment:50 Changed 11 months ago by
- Status changed from positive_review to needs_work
I'm still seeing this on the buildbot:
********************************************************************** File "src/sage/doctest/sources.py", line 765, in sage.doctest.sources.FileDocTestSource._test_enough_doctests Failed example: for path, dirs, files in itertools.chain(os.walk('sage'), os.walk('doc')): # long time path = os.path.relpath(path) dirs.sort(); files.sort() for F in files: _, ext = os.path.splitext(F) if ext in ('.py', '.pyx', '.pxd', '.pxi', '.sage', '.spyx', '.rst'): filename = os.path.join(path, F) FDS = FileDocTestSource(filename, DocTestDefaults(long=True,optional=True)) FDS._test_enough_doctests(verbose=False) Exception raised: Traceback (most recent call last): File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run self.compile_and_execute(example, compiler, test.globs) File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute exec(compiled, globs) File "<doctest sage.doctest.sources.FileDocTestSource._test_enough_doctests[6]>", line 9, in <module> FDS._test_enough_doctests(verbose=False) File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.7/site-packages/sage/doctest/sources.py", line 809, in _test_enough_doctests tests, _ = self.create_doctests({}) File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.7/site-packages/sage/doctest/sources.py", line 734, in create_doctests load(filename, namespace) # errors raised here will be caught in DocTestTask File "/Users/buildbot-sage/slave/sage_git/build/local/lib/python3.7/site-packages/sage/repl/load.py", line 252, in load exec(code, globals) File "./postprocess.py", line 19, in <module> FileNotFoundError: [Errno 2] No such file or directory: '-p' ********************************************************************** 1 item had failures: 1 of 9 in sage.doctest.sources.FileDocTestSource._test_enough_doctests [367 tests, 1 failure, 26.05 s] 1 memlimit not run ---------------------------------------------------------------------- sage -t --long src/sage/doctest/sources.py # 1 doctest failed ----------------------------------------------------------------------
And this one on my machine:
[dochtml] [installat] The HTML pages are in local/share/doc/sage/html/en/installation. [dochtml] Error building the documentation. [dochtml] Traceback (most recent call last): [dochtml] File "/home/release/Sage/local/lib/python2.7/runpy.py", line 174, in _run_module_as_main [dochtml] "__main__", fname, loader, pkg_name) [dochtml] File "/home/release/Sage/local/lib/python2.7/runpy.py", line 72, in _run_code [dochtml] exec code in run_globals [dochtml] File "/home/release/Sage/local/lib/python2.7/site-packages/sage_setup/docbuild/__main__.py", line 2, in <module> [dochtml] main() [dochtml] File "/home/release/Sage/local/lib/python2.7/site-packages/sage_setup/docbuild/__init__.py", line 1720, in main [dochtml] builder() [dochtml] File "/home/release/Sage/local/lib/python2.7/site-packages/sage_setup/docbuild/__init__.py", line 336, in _wrapper [dochtml] build_many(build_other_doc, L) [dochtml] File "/home/release/Sage/local/lib/python2.7/site-packages/sage_setup/docbuild/__init__.py", line 280, in build_many [dochtml] _build_many(target, args, processes=NUM_THREADS) [dochtml] File "/home/release/Sage/local/lib/python2.7/site-packages/sage_setup/docbuild/utils.py", line 283, in build_many [dochtml] raise worker_exc.original_exception [dochtml] OSError: /home/release/Sage/src/doc/en/installation/source.rst:228: WARNING: Include file u'/home/release/Sage/src/doc/en/installation/debian.txt' not found or reading it failed
I'm guessing this ticket has issues with incremental builds, can you give this a try?
comment:51 Changed 11 months ago by
I made #29233 for the documentation build issue
comment:52 Changed 11 months ago by
- Dependencies set to #29233
comment:53 Changed 11 months ago by
- Commit changed from 50a5d2587c1a003327dd844bbf15ad4e98011666 to 51e7f51332390708148f107a104f48d92c7aa5d3
Branch pushed to git repo; I updated commit sha1. New commits:
0dc97db | Clean src/doc/en/installation/*.txt in bootstrap-clean, not misc-clean
|
51e7f51 | Merge branch 't/29233/bootstrap____make_distclean_broken' into t/21785/install_sage_src_ext__in_package_data_of_sage_package_by_setup_py__not_sage_local_share_sage_ext__by_build_make_makefile
|
comment:54 Changed 11 months ago by
- Status changed from needs_work to needs_review
I don't know how to reproduce this error:
FileNotFoundError: [Errno 2] No such file or directory: '-p'
comment:55 Changed 11 months ago by
Ic ouldn't reproduce it either. I suspect this is an incremental build issue.
comment:56 Changed 11 months ago by
- Status changed from needs_review to needs_work
comment:57 Changed 11 months ago by
- Commit changed from 51e7f51332390708148f107a104f48d92c7aa5d3 to fef5a5f69cecea754eadecc2e4ae94a7903f68d6
Branch pushed to git repo; I updated commit sha1. New commits:
fef5a5f | FileDocTestSource._test_enough_doctest doctest: Do not load .py files that are not a sage module
|
comment:58 Changed 11 months ago by
- Status changed from needs_work to needs_review
Found it, fixed. Needs review
comment:59 Changed 11 months ago by
- Status changed from needs_review to positive_review
OK let's try it again.
comment:60 Changed 11 months ago by
- Branch changed from u/mkoeppe/install_sage_src_ext__in_package_data_of_sage_package_by_setup_py__not_sage_local_share_sage_ext__by_build_make_makefile to fef5a5f69cecea754eadecc2e4ae94a7903f68d6
- Resolution set to fixed
- Status changed from positive_review to closed
comment:61 Changed 11 months ago by
- Commit fef5a5f69cecea754eadecc2e4ae94a7903f68d6 deleted
Could you check that .nodoctest
files are properly installed in vanilla sage? Because vanilla sage runs doctests on original sage sources (while in sage-on-gentoo it is run on the install because the sources are not available at runtime) it may have been missed.
comment:62 Changed 11 months ago by
This ticket messes up anyone with data in the old sage/ext
directory. Would have been nice to have some advance notice on sage-devel, you know?
comment:63 Changed 11 months ago by
Let me know if you need help with resolving a merge conflict.
That would actually help me. I see extra potential to this for my jupyter kernel install woes as well (even if I know Jeroen doesn't see it that way).