Opened 2 years ago

Closed 22 months ago

#30912 closed enhancement (fixed)

sagelib: Update metadata for PyPI deployment as package sagemath-standard

Reported by: Matthias Köppe Owned by:
Priority: critical Milestone: sage-9.3
Component: build Keywords: sd111
Cc: Dima Pasechnik, John Palmieri, Marc Masdeu, Vincent Delecroix, Martin Albrecht, François Bissey, Erik Bray, ​dunfield, Samuel Lelièvre, Michael Orlitzky, Frédéric Chapoton, Volker Braun, Sébastien Labbé, William Stein, Markus Wageringel, Julian Rüth Merged in:
Authors: Matthias Koeppe Reviewers: John Palmieri
Report Upstream: N/A Work issues:
Branch: 7ad4c0e (Commits, GitHub, GitLab) Commit: 7ad4c0e6da58f8851cfbced0e9c1a66c9479f918
Dependencies: Stopgaps:

Status badges

Description (last modified by Matthias Köppe)

We update the following items in build/pkgs/sagelib/src/:

  • Add README.rst
  • Add LICENSE.txt (for sagelib and documentation only - see discussion in #21571)
  • setup.py metadata (author, license, compatibility, ....)

We switch from distutils to setuptools so we can put the new metadata in the setup.cfg format.

sage-devel post regarding naming:

Naming was discussed during https://wiki.sagemath.org/days111 and a decision made to use the sagemath-... format and in particular sagemath-standard.

Change History (49)

comment:1 Changed 2 years ago by Matthias Köppe

Currently, setup.py thinks that the name of the distribution is sage.

I was planning to change it to sagelib. But we could also try a systematic approach regarding branding and call it sagemath-standard (in anticipation of the modularization project's smaller distributions that could be called sagemath-core, sagemath-ntl, etc., see #29705).

comment:2 Changed 2 years ago by Matthias Köppe

Branch: u/mkoeppe/sagelib__update_metadata_for_pypi_deployment

comment:3 Changed 2 years ago by Matthias Köppe

Authors: Matthias Koeppe
Commit: 2c40179e64bc01894e77de02a4b3399cea36c6f4
Status: newneeds_review

New commits:

2c40179build/pkgs/sagelib/src: Update metadata

comment:4 Changed 2 years ago by git

Commit: 2c40179e64bc01894e77de02a4b3399cea36c6f42a91862c279caa62eee6b7d6d57fac1d99dd4998

Branch pushed to git repo; I updated commit sha1. New commits:

2a91862build/pkgs/sagelib/src/README.rst: Add some text

comment:5 Changed 2 years ago by Dima Pasechnik

should we also claim https://pypi.org/project/sagemath/ ?

comment:6 Changed 2 years ago by Dima Pasechnik

Cc: Marc Masdeu added

comment:7 Changed 2 years ago by Marc Masdeu

The dummy package on pypi.org/project/sagemath was created so that other projects could be made to depend on it, and thus allow a user to install a custom package without having to recompile Sage. This has worked well to this day, you can install the darmonpoints package with a one-liner and alongside it checks that the installed version of Sage is the correct one.

At the time (2016 maybe?) we thought that this was a good idea, since it seemed like it was a long time until modularization would be a reality.

The dummy package only has one function (check_version) which compares the version passed as parameter with the sage version (via from sage.all import version).

I don't mind giving away this package if modularity is happening, since then there would a proper way to check for each of the required packages (there should be a way to check for the "whole sage" being installed, since it may be hard to decide on all the dependencies).

How should we go about it? I would rather have this kind of functionality (and I am assuming that the other two package maintainers that use the dummy package would be with me) available during the transition, specially if it takes more than a week or two...

comment:8 Changed 2 years ago by Matthias Köppe

I certainly don't need the sagemath name itself any time soon - I would use suffixed names such as sagemath-standard.

comment:9 Changed 2 years ago by Matthias Köppe

Cc: Vincent Delecroix added

comment:10 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:11 Changed 2 years ago by Matthias Köppe

Cc: Martin Albrecht added

comment:12 Changed 2 years ago by Matthias Köppe

Cc: François Bissey added

comment:13 Changed 2 years ago by Tobias Diez

What's the reason that this new metadata is only in build/.../sagelib and not (also?) under src? Other than that, the changes look good to me.

Moreover, FYI, PEP 621 for Storing project metadata in pyproject.toml has been provisionally accepted (and will be fully accepted latest on 2021-03-01). Once setuptools supports this, I would propose to follow it and put the metadata in pyproject.toml.

comment:14 Changed 2 years ago by git

Commit: 2a91862c279caa62eee6b7d6d57fac1d99dd4998ea182d7302b3d362bbd38b548cda65c86e3b6f66

Branch pushed to git repo; I updated commit sha1. New commits:

6432727Merge tag '9.3.beta2' into t/30912/sagelib__update_metadata_for_pypi_deployment
4a693f2Move build/pkgs/sagelib/src/setup.cfg to SAGE_ROOT/src, replace by symlink
ea182d7Copy changes from build/pkgs/sagelib/src to src

comment:15 in reply to:  13 Changed 2 years ago by Matthias Köppe

Replying to gh-tobiasdiez:

What's the reason that this new metadata is only in build/.../sagelib and not (also?) under src?

I guess I had wanted to avoid merge conflicts with #30371, but here you go.

comment:16 in reply to:  13 Changed 2 years ago by Matthias Köppe

Replying to gh-tobiasdiez:

Moreover, FYI, PEP 621 for Storing project metadata in pyproject.toml has been provisionally accepted (and will be fully accepted latest on 2021-03-01). Once setuptools supports this, I would propose to follow it and put the metadata in pyproject.toml.

Yes, it's good to see the progress in this direction.

comment:17 Changed 2 years ago by git

Commit: ea182d7302b3d362bbd38b548cda65c86e3b6f66a1a10b9b524e63574bbf1de2c4ce6f149116f147

Branch pushed to git repo; I updated commit sha1. New commits:

a1a10b9src/VERSION.txt: New

comment:18 Changed 2 years ago by Matthias Köppe

Cc: Erik Bray added

comment:19 Changed 2 years ago by Matthias Köppe

Keywords: sd111 added

comment:20 Changed 2 years ago by Matthias Köppe

Summary: sagelib: Update metadata for PyPI deploymentsagelib: Update metadata for PyPI deployment as package sagemath-standard

comment:21 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:22 Changed 2 years ago by Matthias Köppe

Cc: ​dunfield added

comment:23 in reply to:  7 Changed 2 years ago by Samuel Lelièvre

Cc: Samuel Lelièvre added

Replying to mmasdeu:

The dummy package on pypi.org/project/sagemath was created so that other projects could be made to depend on it, and thus allow a user to install a custom package without having to recompile Sage. This has worked well to this day, you can install the darmonpoints package with a one-liner and alongside it checks that the installed version of Sage is the correct one.

At the time (2016 maybe?) we thought that this was a good idea, since it seemed like it was a long time until modularization would be a reality.

The dummy package only has one function (check_version) which compares the version passed as parameter with the sage version (via from sage.all import version).

I don't mind giving away this package if modularity is happening, since then there would a proper way to check for each of the required packages (there should be a way to check for the "whole sage" being installed, since it may be hard to decide on all the dependencies).

How should we go about it? I would rather have this kind of functionality (and I am assuming that the other two package maintainers that use the dummy package would be with me) available during the transition, specially if it takes more than a week or two...

Maybe there could be a package with a more descriptive name for that use case?

Maybe one of the following names, or something similar?

  • sagemath_installed
  • sagemath_version
  • sagemath_checkinstalled
  • sagemath_checkversion
  • sagemath_probe

Some users assume from the name that it actually installs Sage, and are surprised after installing it that things don't work as expected; this comes up once in a while on the help and support channels such as Ask Sage, sage-support, Stack Overflow...

Not an emergency, but something to consider.

comment:24 Changed 2 years ago by Matthias Köppe

Status: needs_reviewneeds_work
Work issues: add license file

comment:25 Changed 2 years ago by git

Commit: a1a10b9b524e63574bbf1de2c4ce6f149116f147deb9eb3a33ee6474f755a9bee6f487d839c9d247

Branch pushed to git repo; I updated commit sha1. New commits:

5697335src/setup.cfg: Add license_file=LICENSE.txt
deb9eb3Merge tag '9.3.beta3' into t/30912/sagelib__update_metadata_for_pypi_deployment

comment:26 Changed 2 years ago by Matthias Köppe

Cc: Michael Orlitzky added
Description: modified (diff)
Status: needs_workneeds_review
Work issues: add license file

comment:27 Changed 2 years ago by Matthias Köppe

Cc: Frédéric Chapoton Volker Braun added

comment:28 Changed 2 years ago by git

Commit: deb9eb3a33ee6474f755a9bee6f487d839c9d2477ad4c0e6da58f8851cfbced0e9c1a66c9479f918

Branch pushed to git repo; I updated commit sha1. New commits:

7ad4c0eMerge tag '9.3.beta4' into t/30912/sagelib__update_metadata_for_pypi_deployment

comment:29 Changed 2 years ago by Matthias Köppe

Still hoping for someone with recent Python packaging experience to review this ticket.

comment:30 Changed 2 years ago by Dima Pasechnik

Cc: Sébastien Labbé added

comment:31 Changed 2 years ago by Matthias Köppe

Cc: William Stein added
Description: modified (diff)

comment:32 Changed 2 years ago by Matthias Köppe

Priority: majorcritical

comment:33 Changed 2 years ago by Matthias Köppe

Cc: Markus Wageringel Julian Rüth added

comment:34 Changed 23 months ago by Matthias Köppe

Needs review.

comment:35 Changed 22 months ago by Matthias Köppe

Can we please get this into 9.3?

comment:36 Changed 22 months ago by John Palmieri

I'm seeing a doctest failure:

sage -t --warn-long 95.5 --random-seed=0 src/sage/all.py
**********************************************************************
File "src/sage/all.py", line 333, in sage.all._write_started_file
Failed example:
    os.path.isfile(started_file)  # optional - build
Expected:
    True
Got:
    False
**********************************************************************

Is it spurious or could it be caused by this ticket?

comment:37 Changed 22 months ago by Matthias Köppe

If the docbuild fails with an error, then make does not get to the target that creates the started_file. You can fix this with make build-start

comment:38 Changed 22 months ago by John Palmieri

That's too bad. I thought, but maybe I'm wrong about this, that starting Sage used to create the started_file, and I made sure to run Sage before rerunning this doctest. You're right, in any case: make build-start fixed the doctest.

comment:39 Changed 22 months ago by Matthias Köppe

I don't think starting sage creates the started file. It's only happening as part of the build in build/bin/sage-starts

comment:40 Changed 22 months ago by Matthias Köppe

since #13988

comment:41 Changed 22 months ago by John Palmieri

Reviewers: John Palmieri
Status: needs_reviewpositive_review

I am far from an expert in setuptools, but this looks okay to me.

comment:42 Changed 22 months ago by Matthias Köppe

Thanks!

comment:43 Changed 22 months ago by François Bissey

This is fun. Volker is in the process of merging the ticket, which means I get to test it in sage-on-gentoo

>>> Compiling source in /dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src ...
 * python3_7: running distutils-r1_run_phase python_compile
python3.7 setup.py build -j 8
distributions = ['', 'sage-bliss', 'sage-meataxe']
Discovering Python/Cython source code....
Discovered Python/Cython sources, time: 0.20 seconds.
************************************************************************
Traceback (most recent call last):
  File "setup.py", line 153, in <module>
    ext_modules = cython_modules)
  File "/usr/lib/python3.7/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.7/distutils/core.py", line 121, in setup
    dist.parse_config_files()
  File "/usr/lib/python3.7/site-packages/setuptools/dist.py", line 668, in parse_config_files
    ignore_option_errors=ignore_option_errors)
  File "/usr/lib/python3.7/site-packages/setuptools/config.py", line 157, in parse_configuration
    meta.parse()
  File "/usr/lib/python3.7/site-packages/setuptools/config.py", line 463, in parse
    section_parser_method(section_options)
  File "/usr/lib/python3.7/site-packages/setuptools/config.py", line 436, in parse_section
    self[name] = value
  File "/usr/lib/python3.7/site-packages/setuptools/config.py", line 220, in __setitem__
    value = parser(value)
  File "/usr/lib/python3.7/site-packages/setuptools/config.py", line 548, in _parse_version
    raise DistutilsOptionError(tmpl.format(**locals()))
distutils.errors.DistutilsOptionError: Version loaded from file: VERSION.txt does not comply with PEP 440: SageMath version 9.3.beta7, Release Date: 2021-02-07
************************************************************************
Error building the Sage library

You can read the PEP in question here https://www.python.org/dev/peps/pep-0440/. I am currently using the Gentoo stable setuptools which is 50.3.0. sage is shipping 51.1.1 so I am guessing someone had a change of heart about strict enforcement between those two. I'll check by updating my own to the latest in Gentoo's tree (53.0.0).

But if the PEP get enforced the days of naming release .betaN and .rcN are numbered.

comment:44 Changed 22 months ago by François Bissey

Reading more closely a, b and rc are allowed.

comment:45 Changed 22 months ago by François Bissey

Still broken for me with setuptools-53.0.0.

comment:46 Changed 22 months ago by Matthias Köppe

This does not affect the Sage distribution (it uses the other copy of setup.py) and is fixed already in #31357.

comment:47 Changed 22 months ago by François Bissey

I see. Switching to that copy is possible but that messes up how I currently do the documentation. I am not quite sure how much effort yet but it probably should be done.

comment:48 in reply to:  47 Changed 22 months ago by François Bissey

Replying to fbissey:

I see. Switching to that copy is possible but that messes up how I currently do the documentation. I am not quite sure how much effort yet but it probably should be done.

Lesson of the day, patch doesn't like symbolic links which means that I am better off waiting for #31357.

comment:49 Changed 22 months ago by Volker Braun

Branch: u/mkoeppe/sagelib__update_metadata_for_pypi_deployment7ad4c0e6da58f8851cfbced0e9c1a66c9479f918
Resolution: fixed
Status: positive_reviewclosed
Note: See TracTickets for help on using tickets.