sage.repl: Replace use of SAGE_EXTCODE by importlib.resources

[mkoeppe]

We eliminate direct reading of files from the package directories of sagelib by using importlib.resources (available since python 3.7)

• git grep 'SAGE_EXTCODE' src/sage
• git grep '__file__' src/sage

This will help make sagelib zip_safe (​https://setuptools.readthedocs.io/en/latest/userguide/miscellaneous.html#setting-the-zip-safe-flag)

In this ticket, we take care of sage.repl, moving data needed by its doctests from SAGE_EXTCODE (= src/sage/ext_data) into the package directory. (Thus, we avoid a dependency on support for resources in namespace packages brought by Python 3.10(?) or the backport package importlib-resources, see ​https://github.com/python/importlib_resources/pull/196)

Follow-up tickets will deal with other parts of the library. In the end, the directory src/sage/ext_data will be eliminated.

References:

• ​https://docs.python.org/3/library/importlib.html#module-importlib.resources
• ​https://importlib-resources.readthedocs.io/en/latest/migration.html
• ​https://importlib-resources.readthedocs.io/en/latest/using.html

[mkoeppe]

Last 10 new commits:

​77c957f	src/doc/en/developer/packaging_sage_library.rst: Put hierarchy section one level higher
​9375ef8	pkgs/sagemath-standard/tox.ini: Use SAGE_VENV or venv symlink to find wheels
​c6fc111	Prettier diagram
​7d6a44c	Use :mod: as markup for packages/modules
​f83d424	Improve ABC example
​817a8d0	src/doc/en/developer/packaging_sage_library.rst: More :mod: and :class: markup
​b555735	src/doc/en/developer/packaging_sage_library.rst: Link to pypi.org and to documentation of packaging metadata
​81e9c9a	Merge #32899
​af47b5f	Merge #31386
​b2cb173	src/sage/repl/rich_output/: Move example files here from src/sage/ext_data, use importlib.resources

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​0f63cce

src/sage/repl/display/formatter.py: Replace use of SAGE_EXTCODE in doctest by importlib.resources

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​09787af

Merge tag '9.5.beta8' into t/31306/replace_use_of_sage_extcode_by_importlib_resources

[mkoeppe]

The testsuite failures are unrelated

[dimpase]

lgtm

[mkoeppe]

Thanks!

[git]

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

​9873958

Merge tag '9.5.beta9' into t/31306/replace_use_of_sage_extcode_by_importlib_resources

[mkoeppe]

trivial merge

[git]

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

​d925cd2

Merge tag '9.5.rc3' into t/31306/replace_use_of_sage_extcode_by_importlib_resources

[fbissey]

OK, Volker has started merging stuff for 9.6 including this. And I get

$ sage -t --long --random-seed=241985268132368173591124256826068888319 /usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py
too many failed tests, not using stored timings
Running doctests with ID 2022-01-31-20-37-00-111581fe.
Using --optional=pip,sage
Features to be detected: 4ti2,benzene,bliss,buckygen,conway_polynomials,csdp,database_cremona_ellcurve,database_cremona_mini_ellcurve,database_jones_numfield,database_knotinfo,dvipng,graphviz,imagemagick,jupymake,kenzo,latte_int,lrslib,mcqd,meataxe,pandoc,pdf2svg,plantri,pynormaliz,python_igraph,rubiks,sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.groups,sage.plot,sage.rings.number_field,sage.rings.padics,sage.rings.real_double,sage.symbolic,sage_numerical_backends_coin,sagemath_doc_html,sphinx,tdlib
Doctesting 1 file.
sage -t --long --random-seed=241985268132368173591124256826068888319 /usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 248, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate
Failed example:
    backend.validate(dm.types.OutputSceneWavefront.example())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[15]>", line 1, in <module>
        backend.validate(dm.types.OutputSceneWavefront.example())
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_graphics3d.py", line 354, in example
        scene_obj = OutputBuffer.from_file(filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file
        buf._chmod_readonly(buf._filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly
        os.chmod(filename, mode)
    PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example_wavefront_scene.obj'
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 249, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate
Failed example:
    backend.validate(dm.types.OutputSceneCanvas3d.example())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[16]>", line 1, in <module>
        backend.validate(dm.types.OutputSceneCanvas3d.example())
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_graphics3d.py", line 161, in example
        return cls(OutputBuffer.from_file(filename))
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file
        buf._chmod_readonly(buf._filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly
        os.chmod(filename, mode)
    PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.canvas3d'
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 250, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate
Failed example:
    backend.validate(dm.types.OutputVideoOgg.example())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[17]>", line 1, in <module>
        backend.validate(dm.types.OutputVideoOgg.example())
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example
        return cls(OutputBuffer.from_file(filename),
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file
        buf._chmod_readonly(buf._filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly
        os.chmod(filename, mode)
    PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.ogv'
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 251, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate
Failed example:
    backend.validate(dm.types.OutputVideoWebM.example())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[18]>", line 1, in <module>
        backend.validate(dm.types.OutputVideoWebM.example())
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example
        return cls(OutputBuffer.from_file(filename),
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file
        buf._chmod_readonly(buf._filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly
        os.chmod(filename, mode)
    PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.webm'
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 252, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate
Failed example:
    backend.validate(dm.types.OutputVideoMp4.example())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[19]>", line 1, in <module>
        backend.validate(dm.types.OutputVideoMp4.example())
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example
        return cls(OutputBuffer.from_file(filename),
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file
        buf._chmod_readonly(buf._filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly
        os.chmod(filename, mode)
    PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.mp4'
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 253, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate
Failed example:
    backend.validate(dm.types.OutputVideoFlash.example())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[20]>", line 1, in <module>
        backend.validate(dm.types.OutputVideoFlash.example())
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example
        return cls(OutputBuffer.from_file(filename),
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file
        buf._chmod_readonly(buf._filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly
        os.chmod(filename, mode)
    PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.flv'
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 254, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate
Failed example:
    backend.validate(dm.types.OutputVideoMatroska.example())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[21]>", line 1, in <module>
        backend.validate(dm.types.OutputVideoMatroska.example())
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example
        return cls(OutputBuffer.from_file(filename),
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file
        buf._chmod_readonly(buf._filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly
        os.chmod(filename, mode)
    PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.mkv'
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 255, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate
Failed example:
    backend.validate(dm.types.OutputVideoAvi.example())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[22]>", line 1, in <module>
        backend.validate(dm.types.OutputVideoAvi.example())
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example
        return cls(OutputBuffer.from_file(filename),
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file
        buf._chmod_readonly(buf._filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly
        os.chmod(filename, mode)
    PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.avi'
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 256, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate
Failed example:
    backend.validate(dm.types.OutputVideoWmv.example())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[23]>", line 1, in <module>
        backend.validate(dm.types.OutputVideoWmv.example())
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example
        return cls(OutputBuffer.from_file(filename),
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file
        buf._chmod_readonly(buf._filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly
        os.chmod(filename, mode)
    PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.wmv'
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/backend_doctest.py", line 257, in sage.repl.rich_output.backend_doctest.BackendDoctest.validate
Failed example:
    backend.validate(dm.types.OutputVideoQuicktime.example())
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.rich_output.backend_doctest.BackendDoctest.validate[24]>", line 1, in <module>
        backend.validate(dm.types.OutputVideoQuicktime.example())
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py", line 75, in example
        return cls(OutputBuffer.from_file(filename),
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 114, in from_file
        buf._chmod_readonly(buf._filename)
      File "/usr/lib/python3.10/site-packages/sage/repl/rich_output/buffer.py", line 146, in _chmod_readonly
        os.chmod(filename, mode)
    PermissionError: [Errno 1] Operation not permitted: '/usr/lib/python3.10/site-packages/sage/repl/rich_output/example.mov'
**********************************************************************

And it comes down to that block ​https://github.com/sagemath/sage/blob/develop/src/sage/repl/rich_output/buffer.py#L139 - chmod is not done if the file is in the SAGE_EXTCODE folder but executed otherwise. Of course all those files have been moved out of SAGE_EXTCODE.

[fbissey]

I also have

sage -t --long --random-seed=241985268132368173591124256826068888319 /usr/lib/python3.10/site-packages/sage/repl/rich_output/output_graphics3d.py  # 25 doctests failed
sage -t --long --random-seed=241985268132368173591124256826068888319 /usr/lib/python3.10/site-packages/sage/repl/rich_output/output_video.py  # 14 doctests failed

because of the same thing. I guess it doesn't fail on user space build because you can safely chmod anything.

[fbissey]

Follow up at #33256.

[dimpase]

Wow, doctesting of /usr/lib/python3.10/site-packages/sage/...

[fbissey]

Replying to dimpase:

Wow, doctesting of /usr/lib/python3.10/site-packages/sage/...

Is it the python3.10 or the /usr/lib/ that makes you go "wow"? I can do the python3.9 flavor too if you want. I have been doctesting system installed sage and reporting from those tests for the last 10 years so I may be a bit blazé about it.

[dimpase]

it's about /usr/lib