Opened 19 months ago

Closed 6 months ago

#31306 closed enhancement (fixed)

sage.repl: Replace use of SAGE_EXTCODE by importlib.resources

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-9.6
Component: refactoring Keywords:
Cc: gh-tobiasdiez, chapoton, fbissey, dimpase, soehms, tscrim Merged in:
Authors: Matthias Koeppe Reviewers: Dima Pasechnik
Report Upstream: N/A Work issues:
Branch: d925cd2 (Commits, GitHub, GitLab) Commit: d925cd2f322cf8230a6e9f86bef03ce020fc12bf
Dependencies: Stopgaps:

Status badges

Description (last modified by mkoeppe)

We eliminate direct reading of files from the package directories of sagelib by using importlib.resources (available since python 3.7)

  • git grep 'SAGE_EXTCODE' src/sage
  • git grep '__file__' src/sage

This will help make sagelib zip_safe (https://setuptools.readthedocs.io/en/latest/userguide/miscellaneous.html#setting-the-zip-safe-flag)

In this ticket, we take care of sage.repl, moving data needed by its doctests from SAGE_EXTCODE (= src/sage/ext_data) into the package directory. (Thus, we avoid a dependency on support for resources in namespace packages brought by Python 3.10(?) or the backport package importlib-resources, see https://github.com/python/importlib_resources/pull/196)

Follow-up tickets will deal with other parts of the library. In the end, the directory src/sage/ext_data will be eliminated.

References:

Change History (27)

comment:1 Changed 13 months ago by mkoeppe

  • Milestone changed from sage-9.4 to sage-9.5

comment:2 Changed 8 months ago by mkoeppe

  • Dependencies changed from #30551 to #30551, #31386
  • Description modified (diff)

comment:3 Changed 8 months ago by mkoeppe

  • Branch set to u/mkoeppe/replace_use_of_sage_extcode_by_importlib_resources

comment:4 Changed 8 months ago by mkoeppe

  • Commit set to b2cb173ba4080b77d80e845313ed0cc325d13e92
  • Summary changed from Replace use of SAGE_EXTCODE by importlib.resources to sage.repl: Replace use of SAGE_EXTCODE by importlib.resources

Last 10 new commits:

77c957fsrc/doc/en/developer/packaging_sage_library.rst: Put hierarchy section one level higher
9375ef8pkgs/sagemath-standard/tox.ini: Use SAGE_VENV or venv symlink to find wheels
c6fc111Prettier diagram
7d6a44cUse :mod: as markup for packages/modules
f83d424Improve ABC example
817a8d0src/doc/en/developer/packaging_sage_library.rst: More :mod: and :class: markup
b555735src/doc/en/developer/packaging_sage_library.rst: Link to pypi.org and to documentation of packaging metadata
81e9c9aMerge #32899
af47b5fMerge #31386
b2cb173src/sage/repl/rich_output/: Move example files here from src/sage/ext_data, use importlib.resources

comment:5 Changed 8 months ago by git

  • Commit changed from b2cb173ba4080b77d80e845313ed0cc325d13e92 to 0f63cce4ed14254a74ebf5627f88495db3896576

Branch pushed to git repo; I updated commit sha1. New commits:

0f63ccesrc/sage/repl/display/formatter.py: Replace use of SAGE_EXTCODE in doctest by importlib.resources

comment:6 Changed 8 months ago by mkoeppe

  • Authors set to Matthias Koeppe
  • Dependencies changed from #30551, #31386 to #31386
  • Status changed from new to needs_review

comment:7 Changed 8 months ago by mkoeppe

  • Cc dimpase added

comment:8 Changed 8 months ago by mkoeppe

  • Description modified (diff)

comment:9 Changed 8 months ago by git

  • Commit changed from 0f63cce4ed14254a74ebf5627f88495db3896576 to 09787af8a65445158522c07c8e76b01eacd5fdbe

Branch pushed to git repo; I updated commit sha1. New commits:

09787afMerge tag '9.5.beta8' into t/31306/replace_use_of_sage_extcode_by_importlib_resources

comment:10 Changed 8 months ago by mkoeppe

  • Cc soehms tscrim added

comment:11 Changed 8 months ago by mkoeppe

  • Description modified (diff)

comment:12 Changed 8 months ago by mkoeppe

  • Description modified (diff)

comment:13 Changed 8 months ago by mkoeppe

The testsuite failures are unrelated

comment:14 Changed 8 months ago by dimpase

  • Reviewers set to Dima Pasechnik
  • Status changed from needs_review to positive_review

lgtm

comment:15 Changed 8 months ago by mkoeppe

Thanks!

comment:16 Changed 8 months ago by git

  • Commit changed from 09787af8a65445158522c07c8e76b01eacd5fdbe to 9873958f9f942637116a8742b8a179117c95b20e
  • Status changed from positive_review to needs_review

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

9873958Merge tag '9.5.beta9' into t/31306/replace_use_of_sage_extcode_by_importlib_resources

comment:17 Changed 8 months ago by mkoeppe

  • Status changed from needs_review to positive_review

trivial merge

comment:18 Changed 7 months ago by mkoeppe

  • Milestone changed from sage-9.5 to sage-9.6

comment:19 Changed 7 months ago by git

  • Commit changed from 9873958f9f942637116a8742b8a179117c95b20e to d925cd2f322cf8230a6e9f86bef03ce020fc12bf
  • Status changed from positive_review to needs_review

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

d925cd2Merge tag '9.5.rc3' into t/31306/replace_use_of_sage_extcode_by_importlib_resources

comment:20 Changed 7 months ago by mkoeppe

  • Dependencies #31386 deleted
  • Status changed from needs_review to positive_review

comment:21 Changed 6 months ago by fbissey

OK, Volker has started merging stuff for 9.6 including this. And I get

$ sage -t --long --random-seed=241985268132368173591124256826068888319 /usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py
too many failed tests, not using stored timings
Running doctests with ID 2022-01-31-20-37-00-111581fe.
Using --optional=pip,sage
Features to be detected: 4ti2,benzene,bliss,buckygen,conway_polynomials,csdp,database_cremona_ellcurve,database_cremona_mini_ellcurve,database_jones_numfield,database_knotinfo,dvipng,graphviz,imagemagick,jupymake,kenzo,latte_int,lrslib,mcqd,meataxe,pandoc,pdf2svg,plantri,pynormaliz,python_igraph,rubiks,sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.groups,sage.plot,sage.rings.number_field,sage.rings.padics,sage.rings.real_double,sage.symbolic,sage_numerical_backends_coin,sagemath_doc_html,sphinx,tdlib
Doctesting 1 file.
sage -t --long --random-seed=241985268132368173591124256826068888319 /usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 248, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate
Failed example:
    backend.validate(dm.types.OutputSceneWavefront.example())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[15]>", line 1, in <module>
        backend.validate(dm.types.OutputSceneWavefront.example())
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_graphics3d.py", line 354, in example
        scene_obj = OutputBuffer.from_file(filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file
        buf._chmod_readonly(buf._filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly
        os.chmod(filename, mode)
    PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example_wavefront_scene.obj'
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 249, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate
Failed example:
    backend.validate(dm.types.OutputSceneCanvas3d.example())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[16]>", line 1, in <module>
        backend.validate(dm.types.OutputSceneCanvas3d.example())
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_graphics3d.py", line 161, in example
        return cls(OutputBuffer.from_file(filename))
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file
        buf._chmod_readonly(buf._filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly
        os.chmod(filename, mode)
    PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.canvas3d'
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 250, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate
Failed example:
    backend.validate(dm.types.OutputVideoOgg.example())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[17]>", line 1, in <module>
        backend.validate(dm.types.OutputVideoOgg.example())
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example
        return cls(OutputBuffer.from_file(filename),
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file
        buf._chmod_readonly(buf._filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly
        os.chmod(filename, mode)
    PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.ogv'
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 251, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate
Failed example:
    backend.validate(dm.types.OutputVideoWebM.example())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[18]>", line 1, in <module>
        backend.validate(dm.types.OutputVideoWebM.example())
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example
        return cls(OutputBuffer.from_file(filename),
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file
        buf._chmod_readonly(buf._filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly
        os.chmod(filename, mode)
    PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.webm'
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 252, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate
Failed example:
    backend.validate(dm.types.OutputVideoMp4.example())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[19]>", line 1, in <module>
        backend.validate(dm.types.OutputVideoMp4.example())
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example
        return cls(OutputBuffer.from_file(filename),
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file
        buf._chmod_readonly(buf._filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly
        os.chmod(filename, mode)
    PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.mp4'
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 253, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate
Failed example:
    backend.validate(dm.types.OutputVideoFlash.example())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[20]>", line 1, in <module>
        backend.validate(dm.types.OutputVideoFlash.example())
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example
        return cls(OutputBuffer.from_file(filename),
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file
        buf._chmod_readonly(buf._filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly
        os.chmod(filename, mode)
    PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.flv'
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 254, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate
Failed example:
    backend.validate(dm.types.OutputVideoMatroska.example())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[21]>", line 1, in <module>
        backend.validate(dm.types.OutputVideoMatroska.example())
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example
        return cls(OutputBuffer.from_file(filename),
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file
        buf._chmod_readonly(buf._filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly
        os.chmod(filename, mode)
    PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.mkv'
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 255, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate
Failed example:
    backend.validate(dm.types.OutputVideoAvi.example())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[22]>", line 1, in <module>
        backend.validate(dm.types.OutputVideoAvi.example())
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example
        return cls(OutputBuffer.from_file(filename),
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file
        buf._chmod_readonly(buf._filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly
        os.chmod(filename, mode)
    PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.avi'
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 256, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate
Failed example:
    backend.validate(dm.types.OutputVideoWmv.example())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[23]>", line 1, in <module>
        backend.validate(dm.types.OutputVideoWmv.example())
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example
        return cls(OutputBuffer.from_file(filename),
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file
        buf._chmod_readonly(buf._filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly
        os.chmod(filename, mode)
    PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.wmv'
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 257, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate
Failed example:
    backend.validate(dm.types.OutputVideoQuicktime.example())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[24]>", line 1, in <module>
        backend.validate(dm.types.OutputVideoQuicktime.example())
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example
        return cls(OutputBuffer.from_file(filename),
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file
        buf._chmod_readonly(buf._filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly
        os.chmod(filename, mode)
    PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.mov'
**********************************************************************

And it comes down to that block https://github.com/sagemath/sage/blob/develop/src/sage/repl/rich_output/buffer.py#L139 - chmod is not done if the file is in the SAGE_EXTCODE folder but executed otherwise. Of course all those files have been moved out of SAGE_EXTCODE.

comment:22 Changed 6 months ago by fbissey

I also have

sage -t --long --random-seed=241985268132368173591124256826068888319 /usr/lib/python3.10/site-packages/sage/repl/rich_output/output_graphics3d.py  # 25 doctests failed
sage -t --long --random-seed=241985268132368173591124256826068888319 /usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py  # 14 doctests failed

because of the same thing. I guess it doesn't fail on user space build because you can safely chmod anything.

comment:23 Changed 6 months ago by fbissey

Follow up at #33256.

comment:24 follow-up: Changed 6 months ago by dimpase

Wow, doctesting of /usr/lib/python3.10/site-packages/sage/...

comment:25 in reply to: ↑ 24 Changed 6 months ago by fbissey

Replying to dimpase:

Wow, doctesting of /usr/lib/python3.10/site-packages/sage/...

Is it the python3.10 or the /usr/lib/ that makes you go "wow"? I can do the python3.9 flavor too if you want. I have been doctesting system installed sage and reporting from those tests for the last 10 years so I may be a bit blazé about it.

comment:26 Changed 6 months ago by dimpase

it's about /usr/lib

comment:27 Changed 6 months ago by vbraun

  • Branch changed from u/mkoeppe/replace_use_of_sage_extcode_by_importlib_resources to d925cd2f322cf8230a6e9f86bef03ce020fc12bf
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.