SPKG type: Make "normal/script/pip" orthogonal to "base/standard/optional/experimental"

[mkoeppe]

This ticket, inspired by a proposal by embray in #24586, removes the misuse of "package type" for the undocumented "script package" category (introduced in #19427). Likewise, it removes the same misuse of "package type" for the undocumented (as noted in #21033) "pip package" category (introduced in #19187).

With this ticket, type has to be one of base / standard / optional / experimental.

All type=pip packages are changed to optional.

All type=script packages are changed to optional; with the exception of sage_conf, which is a dependency of sagelib and is changed to standard.

Orthogonal to type is the new notion of a package "source", which is defined as follows:

• normal packages have a checksum.ini file
• pip packages have a requirements.txt file instead (which is used for pip install). (This allows us to change the pyopenssl package from script to pip.)
• script packages have neither of the two

The ticket also makes script packages more similar to normal packages: They can now optionally have a package-version.txt file; and their installation status is recorded.

Finally, this ticket adds documentation to the developer manual.

[mkoeppe]

New commits:

​c5dcfd6	build/pkgs: Change all 'type=pip' packages to 'type=optional' and add requirements.txt
​9a9b3c3	build/pkgs: Change all 'type=script' packages to 'type=optional'
​90054f6	m4/sage_spkg_collect.m4: Make SPKG_SOURCE={normal|pip|script} orthogonal to SPKG_TYPE={base,standard,optional,experimental}

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f7a661c

sage.misc.package.list_packages: New argument pkg_sources, orthogonal to pkg_types

[jhpalmieri]

This needs documentation.

[mkoeppe]

Indeed, as noted already in #21033.

[jhpalmieri]

Just putting the list in the description somewhere in the docs would help. What is a "script" package, anyway?

[mkoeppe]

Yes, I'll work on it

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​09cc8ba	build/pkgs: Change all 'type=pip' packages to 'type=optional' and add requirements.txt
​8a8100a	build/pkgs: Change all 'type=script' packages to 'type=optional'
​ca47dc4	m4/sage_spkg_collect.m4: Make SPKG_SOURCE={normal|pip|script} orthogonal to SPKG_TYPE={base,standard,optional,experimental}
​ff4db4b	sage.misc.package.list_packages: New argument pkg_sources, orthogonal to pkg_types
​041c76d	Rename spkg-build, spkg-install etc. templates as spkg-build.in, spkg-install.in etc.
​4524b04	Update developer manual
​2724083	Undo rename of build/pkgs/*/spkg-install.py
​ec540dd	Fix up sage-spkg
​f819b51	Merge branch 't/29096/rename_spkg_build__spkg_install_etc__templates_as_spkg_build_in__spkg_install_in_etc_' into t/29287/spkg_type__make__regular_script_pip__orthogonal_to__standard_optional_experimental_

[mkoeppe]

Rebased on 9.1.beta7, merged #29096 to avoid conflicts in documentation

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​678f4cd

src/doc/en/developer/packaging.rst: Update doc

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​28d5c92	build/make/Makefile.in: Install pip packages using 'pip install -r requirements.txt'
​b80d88f	Update doc

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​4793466

build/pkgs/sage_conf: Change type to standard

[mkoeppe]

Added documentation describing normal, pip, and script packages.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​72e7719	m4/sage_spkg_collect.m4: Remove handling of type=script
​dcaaa85	m4/sage_spkg_collect.m4: Remove special-casing of unversioned (script/pip) packages
​52902eb	m4/sage_spkg_collect.m4: For unversioned packages, set version to 'none' instead of name of spkg
​ff871b7	build/make/Makefile.in: Version and record script packages like normal packages
​aa8cdc5	Update doc regarding versioning/recording of script packages
​a6cea8e	build/make/Makefile.in: Mark the targets SPKG and SPKG-clean as phony

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d6005a3

src/bin/sage-list-packages: Remove use of type=pip

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d4b4d81

sage.misc.package.list_packages: For unversioned packages, use 'none' as version

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​ec2805e	m4/sage_spkg_collect.m4: Fix up collection of dependencies
​cb3c913	build/pkgs/texlive/spkg-install: Fix up path

[vdelecroix]

Why do you had all these .in? Such a change should be agreed on sage-devel.

[vdelecroix]

Unless I misunderstood something, in the section Install scripts of script packages, ``spkg-install`` should be ``spkg-install.in`` (twice).

[mkoeppe]

Changing to .in happened on #29096.

No, script packages don't to templating.

[dimpase]

Replying to vdelecroix:

Why do you had all these .in? Such a change should be agreed on sage-devel.

we had an argument about it, and that change should have been done when the actual conversion of most of sage-install scripts into templates happened, a couple of years back. So this change merely corrects an old fault of the naming scheme.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​ad9e524	m4/sage_spkg_collect.m4: Remove handling of type=script
​6c977be	m4/sage_spkg_collect.m4: Remove special-casing of unversioned (script/pip) packages
​d4e8cd9	m4/sage_spkg_collect.m4: For unversioned packages, set version to 'none' instead of name of spkg
​8370225	build/make/Makefile.in: Version and record script packages like normal packages
​495d499	Update doc regarding versioning/recording of script packages
​bc4ed34	build/make/Makefile.in: Mark the targets SPKG and SPKG-clean as phony
​d1fd7a7	src/bin/sage-list-packages: Remove use of type=pip
​75175db	sage.misc.package.list_packages: For unversioned packages, use 'none' as version
​d3d78f4	m4/sage_spkg_collect.m4: Fix up collection of dependencies
​dbb9860	build/pkgs/texlive/spkg-install: Fix up path

[mkoeppe]

Rebased on 9.1.beta8 -- so the diff is easier to read now. Needs review

[jhpalmieri]

I get a doctest failure:

sage -t --long --warn-long 65.3 src/sage/env.py
**********************************************************************
File "src/sage/env.py", line 14, in sage.env
Failed example:
    res = subprocess.call([sys.executable, "-c", cmd], env=env)  # long time
Expected:
    None
Got:
    /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/29090/29287/sage-9.1.beta8
**********************************************************************
1 item had failures:
   1 of   5 in sage.env
    [45 tests, 1 failure, 1.41 s]
----------------------------------------------------------------------
sage -t --long --warn-long 65.3 src/sage/env.py  # 1 doctest failed
----------------------------------------------------------------------
Total time for all tests: 1.4 seconds
    cpu time: 0.1 seconds
    cumulative wall time: 1.4 seconds

[jhpalmieri]

I don't know m4 syntax. Is the indentation remaining after you delete if test "$SPKG_NAME" != "$SPKG_VERSION"; then meaningful?

[mkoeppe]

No, it is not meaningful, I was just keeping the diff small.

[jhpalmieri]

What do you think about this proposed change:

src/doc/en/developer/packaging.rst

@@ -87 +87 @@
    - is obtained directly from https://pypi.org/;
 
    - the version to be installed is determined using the required file
-     ``requirements.txt`` (see
-     https://pip.pypa.io/en/stable/reference/pip_install/#requirements-file-format);
+     ``requirements.txt`` -- in its simplest form, this file just
+     contains the name of the package (more details at
+     https://pip.pypa.io/en/stable/user_guide/#requirements-files);
 
    - Sage installs the package using the ``pip`` package manager;
 

[mkoeppe]

Excellent, please push it to the ticket

[jhpalmieri]

Not necessarily for this ticket, but should pyopenssl be a pip package?

---

New commits:

​8b1d5ca

trac 29287: rewording documentation

[mkoeppe]

Good idea. I'll make this change.

[mkoeppe]

Replying to jhpalmieri:

I get a doctest failure:

sage -t --long --warn-long 65.3 src/sage/env.py
**********************************************************************
File "src/sage/env.py", line 14, in sage.env
Failed example:
    res = subprocess.call([sys.executable, "-c", cmd], env=env)  # long time
Expected:
    None
Got:
    /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/29090/29287/sage-9.1.beta8

Ha, this doctest is funny. I'll fix it.

---

New commits:

​057a66b

Make pyopenssl a pip package

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​8f6323e

src/sage/env.py: Fix up doctest on starting sage without SAGE_* variables

[jhpalmieri]

Can you explain the doctest failure? For some reason, there was no output before, but now there is output?

[mkoeppe]

The doctest was actually "broken" by #29038 in the situation when the sage_conf package that it added is installed. See ticket description of #29038 -- in particular, it provides SAGE_ROOT in the situation that Python imports sage.all outside of a sage-env. That doctest was testing that SAGE_ROOT is not provided in that situation -- when it should simply have tested that the sage.all module works.

The present ticket makes sure that sage_conf is always installed (fixes #29365, for example), and so you now saw the doctest failure.

[jhpalmieri]

Why do we need to give a longer path in the spkg-install file for texlive?

[mkoeppe]

Replying to jhpalmieri:

Why do we need to give a longer path in the spkg-install file for texlive?

This is an incidental bugfix for something that was broken by some earlier ticket. Seems nobody has tried sage -i texlive in a while.

[jhpalmieri]

A few small changes. If you're happy with these, positive review from me.

---

New commits:

​0919086

trac 29287: doc fixes

[mkoeppe]

Great, thanks very much.