Opened 19 months ago
Closed 6 months ago
#31306 closed enhancement (fixed)
sage.repl: Replace use of SAGE_EXTCODE by importlib.resources
Reported by: | mkoeppe | Owned by: | |
---|---|---|---|
Priority: | major | Milestone: | sage-9.6 |
Component: | refactoring | Keywords: | |
Cc: | gh-tobiasdiez, chapoton, fbissey, dimpase, soehms, tscrim | Merged in: | |
Authors: | Matthias Koeppe | Reviewers: | Dima Pasechnik |
Report Upstream: | N/A | Work issues: | |
Branch: | d925cd2 (Commits, GitHub, GitLab) | Commit: | d925cd2f322cf8230a6e9f86bef03ce020fc12bf |
Dependencies: | Stopgaps: |
Description (last modified by )
We eliminate direct reading of files from the package directories of sagelib by using importlib.resources
(available since python 3.7)
git grep 'SAGE_EXTCODE' src/sage
git grep '__file__' src/sage
This will help make sagelib zip_safe
(https://setuptools.readthedocs.io/en/latest/userguide/miscellaneous.html#setting-the-zip-safe-flag)
In this ticket, we take care of sage.repl
, moving data needed by its doctests from SAGE_EXTCODE
(= src/sage/ext_data
) into the package directory. (Thus, we avoid a dependency on support for resources in namespace packages brought by Python 3.10(?) or the backport package importlib-resources
, see https://github.com/python/importlib_resources/pull/196)
Follow-up tickets will deal with other parts of the library. In the end, the directory src/sage/ext_data
will be eliminated.
References:
Change History (27)
comment:1 Changed 13 months ago by
- Milestone changed from sage-9.4 to sage-9.5
comment:2 Changed 8 months ago by
- Dependencies changed from #30551 to #30551, #31386
- Description modified (diff)
comment:3 Changed 8 months ago by
- Branch set to u/mkoeppe/replace_use_of_sage_extcode_by_importlib_resources
comment:4 Changed 8 months ago by
- Commit set to b2cb173ba4080b77d80e845313ed0cc325d13e92
- Summary changed from Replace use of SAGE_EXTCODE by importlib.resources to sage.repl: Replace use of SAGE_EXTCODE by importlib.resources
comment:5 Changed 8 months ago by
- Commit changed from b2cb173ba4080b77d80e845313ed0cc325d13e92 to 0f63cce4ed14254a74ebf5627f88495db3896576
Branch pushed to git repo; I updated commit sha1. New commits:
0f63cce | src/sage/repl/display/formatter.py: Replace use of SAGE_EXTCODE in doctest by importlib.resources
|
comment:6 Changed 8 months ago by
- Dependencies changed from #30551, #31386 to #31386
- Status changed from new to needs_review
comment:7 Changed 8 months ago by
- Cc dimpase added
comment:8 Changed 8 months ago by
- Description modified (diff)
comment:9 Changed 8 months ago by
- Commit changed from 0f63cce4ed14254a74ebf5627f88495db3896576 to 09787af8a65445158522c07c8e76b01eacd5fdbe
Branch pushed to git repo; I updated commit sha1. New commits:
09787af | Merge tag '9.5.beta8' into t/31306/replace_use_of_sage_extcode_by_importlib_resources
|
comment:10 Changed 8 months ago by
- Cc soehms tscrim added
comment:11 Changed 8 months ago by
- Description modified (diff)
comment:12 Changed 8 months ago by
- Description modified (diff)
comment:13 Changed 8 months ago by
The testsuite failures are unrelated
comment:14 Changed 8 months ago by
- Reviewers set to Dima Pasechnik
- Status changed from needs_review to positive_review
lgtm
comment:15 Changed 8 months ago by
Thanks!
comment:16 Changed 8 months ago by
- Commit changed from 09787af8a65445158522c07c8e76b01eacd5fdbe to 9873958f9f942637116a8742b8a179117c95b20e
- Status changed from positive_review to needs_review
Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:
9873958 | Merge tag '9.5.beta9' into t/31306/replace_use_of_sage_extcode_by_importlib_resources
|
comment:17 Changed 8 months ago by
- Status changed from needs_review to positive_review
trivial merge
comment:18 Changed 7 months ago by
- Milestone changed from sage-9.5 to sage-9.6
comment:19 Changed 7 months ago by
- Commit changed from 9873958f9f942637116a8742b8a179117c95b20e to d925cd2f322cf8230a6e9f86bef03ce020fc12bf
- Status changed from positive_review to needs_review
Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:
d925cd2 | Merge tag '9.5.rc3' into t/31306/replace_use_of_sage_extcode_by_importlib_resources
|
comment:20 Changed 7 months ago by
- Dependencies #31386 deleted
- Status changed from needs_review to positive_review
comment:21 Changed 6 months ago by
OK, Volker has started merging stuff for 9.6 including this. And I get
$ sage -t --long --random-seed=241985268132368173591124256826068888319 /usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py too many failed tests, not using stored timings Running doctests with ID 2022-01-31-20-37-00-111581fe. Using --optional=pip,sage Features to be detected: 4ti2,benzene,bliss,buckygen,conway_polynomials,csdp,database_cremona_ellcurve,database_cremona_mini_ellcurve,database_jones_numfield,database_knotinfo,dvipng,graphviz,imagemagick,jupymake,kenzo,latte_int,lrslib,mcqd,meataxe,pandoc,pdf2svg,plantri,pynormaliz,python_igraph,rubiks,sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.groups,sage.plot,sage.rings.number_field,sage.rings.padics,sage.rings.real_double,sage.symbolic,sage_numerical_backends_coin,sagemath_doc_html,sphinx,tdlib Doctesting 1 file. sage -t --long --random-seed=241985268132368173591124256826068888319 /usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py ********************************************************************** File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 248, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate Failed example: backend.validate(dm.types.OutputSceneWavefront.example()) Exception raised: Traceback (most recent call last): File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run self.compile_and_execute(example, compiler, test.globs) File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute exec(compiled, globs) File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[15]>", line 1, in <module> backend.validate(dm.types.OutputSceneWavefront.example()) File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_graphics3d.py", line 354, in example scene_obj = OutputBuffer.from_file(filename) File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file buf._chmod_readonly(buf._filename) File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly os.chmod(filename, mode) PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example_wavefront_scene.obj' ********************************************************************** File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 249, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate Failed example: backend.validate(dm.types.OutputSceneCanvas3d.example()) Exception raised: Traceback (most recent call last): File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run self.compile_and_execute(example, compiler, test.globs) File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute exec(compiled, globs) File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[16]>", line 1, in <module> backend.validate(dm.types.OutputSceneCanvas3d.example()) File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_graphics3d.py", line 161, in example return cls(OutputBuffer.from_file(filename)) File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file buf._chmod_readonly(buf._filename) File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly os.chmod(filename, mode) PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.canvas3d' ********************************************************************** File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 250, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate Failed example: backend.validate(dm.types.OutputVideoOgg.example()) Exception raised: Traceback (most recent call last): File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run self.compile_and_execute(example, compiler, test.globs) File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute exec(compiled, globs) File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[17]>", line 1, in <module> backend.validate(dm.types.OutputVideoOgg.example()) File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example return cls(OutputBuffer.from_file(filename), File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file buf._chmod_readonly(buf._filename) File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly os.chmod(filename, mode) PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.ogv' ********************************************************************** File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 251, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate Failed example: backend.validate(dm.types.OutputVideoWebM.example()) Exception raised: Traceback (most recent call last): File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run self.compile_and_execute(example, compiler, test.globs) File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute exec(compiled, globs) File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[18]>", line 1, in <module> backend.validate(dm.types.OutputVideoWebM.example()) File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example return cls(OutputBuffer.from_file(filename), File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file buf._chmod_readonly(buf._filename) File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly os.chmod(filename, mode) PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.webm' ********************************************************************** File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 252, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate Failed example: backend.validate(dm.types.OutputVideoMp4.example()) Exception raised: Traceback (most recent call last): File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run self.compile_and_execute(example, compiler, test.globs) File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute exec(compiled, globs) File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[19]>", line 1, in <module> backend.validate(dm.types.OutputVideoMp4.example()) File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example return cls(OutputBuffer.from_file(filename), File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file buf._chmod_readonly(buf._filename) File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly os.chmod(filename, mode) PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.mp4' ********************************************************************** File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 253, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate Failed example: backend.validate(dm.types.OutputVideoFlash.example()) Exception raised: Traceback (most recent call last): File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run self.compile_and_execute(example, compiler, test.globs) File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute exec(compiled, globs) File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[20]>", line 1, in <module> backend.validate(dm.types.OutputVideoFlash.example()) File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example return cls(OutputBuffer.from_file(filename), File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file buf._chmod_readonly(buf._filename) File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly os.chmod(filename, mode) PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.flv' ********************************************************************** File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 254, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate Failed example: backend.validate(dm.types.OutputVideoMatroska.example()) Exception raised: Traceback (most recent call last): File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run self.compile_and_execute(example, compiler, test.globs) File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute exec(compiled, globs) File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[21]>", line 1, in <module> backend.validate(dm.types.OutputVideoMatroska.example()) File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example return cls(OutputBuffer.from_file(filename), File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file buf._chmod_readonly(buf._filename) File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly os.chmod(filename, mode) PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.mkv' ********************************************************************** File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 255, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate Failed example: backend.validate(dm.types.OutputVideoAvi.example()) Exception raised: Traceback (most recent call last): File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run self.compile_and_execute(example, compiler, test.globs) File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute exec(compiled, globs) File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[22]>", line 1, in <module> backend.validate(dm.types.OutputVideoAvi.example()) File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example return cls(OutputBuffer.from_file(filename), File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file buf._chmod_readonly(buf._filename) File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly os.chmod(filename, mode) PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.avi' ********************************************************************** File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 256, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate Failed example: backend.validate(dm.types.OutputVideoWmv.example()) Exception raised: Traceback (most recent call last): File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run self.compile_and_execute(example, compiler, test.globs) File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute exec(compiled, globs) File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[23]>", line 1, in <module> backend.validate(dm.types.OutputVideoWmv.example()) File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example return cls(OutputBuffer.from_file(filename), File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file buf._chmod_readonly(buf._filename) File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly os.chmod(filename, mode) PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.wmv' ********************************************************************** File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 257, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate Failed example: backend.validate(dm.types.OutputVideoQuicktime.example()) Exception raised: Traceback (most recent call last): File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run self.compile_and_execute(example, compiler, test.globs) File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute exec(compiled, globs) File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[24]>", line 1, in <module> backend.validate(dm.types.OutputVideoQuicktime.example()) File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example return cls(OutputBuffer.from_file(filename), File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file buf._chmod_readonly(buf._filename) File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly os.chmod(filename, mode) PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.mov' **********************************************************************
And it comes down to that block https://github.com/sagemath/sage/blob/develop/src/sage/repl/rich_output/buffer.py#L139 - chmod
is not done if the file is in the SAGE_EXTCODE
folder but executed otherwise. Of course all those files have been moved out of SAGE_EXTCODE
.
comment:22 Changed 6 months ago by
I also have
sage -t --long --random-seed=241985268132368173591124256826068888319 /usr/lib/python3.10/site-packages/sage/repl/rich_output/output_graphics3d.py # 25 doctests failed sage -t --long --random-seed=241985268132368173591124256826068888319 /usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py # 14 doctests failed
because of the same thing. I guess it doesn't fail on user space build because you can safely chmod
anything.
comment:23 Changed 6 months ago by
Follow up at #33256.
comment:24 follow-up: ↓ 25 Changed 6 months ago by
Wow, doctesting of /usr/lib/python3.10/site-packages/sage/
...
comment:25 in reply to: ↑ 24 Changed 6 months ago by
Replying to dimpase:
Wow, doctesting of
/usr/lib/python3.10/site-packages/sage/
...
Is it the python3.10
or the /usr/lib/
that makes you go "wow"? I can do the python3.9
flavor too if you want. I have been doctesting system installed sage and reporting from those tests for the last 10 years so I may be a bit blazé about it.
comment:26 Changed 6 months ago by
it's about /usr/lib
comment:27 Changed 6 months ago by
- Branch changed from u/mkoeppe/replace_use_of_sage_extcode_by_importlib_resources to d925cd2f322cf8230a6e9f86bef03ce020fc12bf
- Resolution set to fixed
- Status changed from positive_review to closed
Last 10 new commits:
src/doc/en/developer/packaging_sage_library.rst: Put hierarchy section one level higher
pkgs/sagemath-standard/tox.ini: Use SAGE_VENV or venv symlink to find wheels
Prettier diagram
Use :mod: as markup for packages/modules
Improve ABC example
src/doc/en/developer/packaging_sage_library.rst: More :mod: and :class: markup
src/doc/en/developer/packaging_sage_library.rst: Link to pypi.org and to documentation of packaging metadata
Merge #32899
Merge #31386
src/sage/repl/rich_output/: Move example files here from src/sage/ext_data, use importlib.resources