Cleaning fileserver.sagemath.org for optional packages

[Vincent Delecroix]

The pages ​http://files.sagemath.org/spkg/optional/list and ​http://files.sagemath.org/spkg/optional/ contain outdated information. The following are now new style packages

• beautifulsoup-3.2.1 (pip)
• biopython-1.61 (pip)
• brian-1.4.1.p0 (pip)
• coxeter3-1.1 (in #19178)
• database_jones_numfield-v4 (in #19174)
• database_kohel-20060803 (in #21086)
• guppy-0.1.8 (pip)
• lie-2.2.2.p5 (? moved in experimental in #19781)
• mpi4py-1.2.2 (pip)
• ore_algebra-0.2 (#19175)
• pybtex-20120618 (pip)
• pyx-0.10 (#21076)
• TOPCOM-0.17.4 (now lower case topcom)
• trac-0.11.5.p0 (pip)

See also #19220.

[John Palmieri]

I think the pip packages can be removed from the server: users can download and install them using sage -i ..., which should use pip.

[Dinakar Muthiah]

I think so. One difference will be that the package names will not appear anymore with $ sage -optional. However, that script that list packages would be easily changed. But since pip type is transitional I am not sure it is what we want.

[John Palmieri]

See #19213.

[Leif Leonhardy]

Replying to vdelecroix (ticket description):

See also #21130

???

---

You (or we) shouldn't search for optional packages in spkg/optional/ unless there's no new-style version of a package.

All packages below spkg/ are for "backwards-compatibility", i.e., older versions of Sage.

At some point, all old-style packages should be converted, such that we can (in principle) disable the installation of legacy spkgs into newer versions of Sage.

---

As you may know, I've always been against the "unified repo" as is (tying any spkg to specific Sage versions and vice versa, and offering only a single version of each at the same time), which is completely against the original philosophy (that anybody could offer his/her spkg, most being self-contained, some needing Sage library support of course).

Because of this, it's difficult to generally disallow the installation of "old-style" spkgs, unless or until we have a new format and mechanism for truly "external" packages. (I suggested to split spkgs into two parts, the Sage package metadata, .spam, and an upstream tarball, years ago. We could still do the same for at least "external" packages, namely create/use a tarball with just what is now in build/pkgs/<pkgname>/, although a single plain text file à la Arch Linux or Gentoo/Funtoo would be better.)

A related problem is that our build / package system lacks any notion of version requirements, it always [en]forces to use exactly a single specific version of a package, no matter whether that's necessary or not, and true requirements are hardly documented.

[Leif Leonhardy]

Replying to leif:

Replying to vdelecroix (ticket description):

See also #21130

???

Ah ok, fixed now.

[Dinakar Muthiah]

Replying to leif:

Replying to vdelecroix (ticket description):

I did no try to look at spkg/optional. I just did sage -optional that shows wrong information (e.g. lie being optional but sage -i lie install a different experimental package). I want to be able to install all optional packages using sage -optional and this is badly broken.

I also consider the current packaging system as very bad. But this ticket is about cleaning the websites mentioned in the description. And I think cleaning is a good step forward.

[Dinakar Muthiah]

Another consequence of this

$ sage -i brian
... #snip 
$ sage -optional|grep brian -C1
boost................................... 1_61_0 (not_installed)
brian................................... 1.4.1.p0 (not_installed)
buckygen................................ 1.0 (not_installed)

So +1 from removing pip packages as well...

[Leif Leonhardy]

Replying to dmuthiah:

Replying to leif:

Replying to vdelecroix (ticket description):

I did no try to look at spkg/optional. I just did sage -optional that shows wrong information (e.g. lie being optional but sage -i lie install a different experimental package).

That's because the "new-style" package is now categorized "experimental".

I want to be able to install all optional packages using sage -optional and this is badly broken.

Yes, it is. Not long ago, it kept showing only legacy spkgs, now it gives an inconsistent mixture instead of what I mentioned above (namely legacy spkgs iff there's no "new-style" version at all).

I also consider the current packaging system as very bad. But this ticket is about cleaning the websites mentioned in the description. And I think cleaning is a good step forward.

It's not the website that's broken, it's sage --optional, sage --experimental etc. We should fix that, and not just try to hide the problem by more or less randomly deleting or moving some stuff.

[Vincent Delecroix]

Replying to leif:

Replying to dmuthiah:

Replying to leif: I also consider the current packaging system as very bad. But this ticket is about cleaning the websites mentioned in the description. And I think cleaning is a good step forward.

It's not the website that's broken, it's sage --optional, sage --experimental etc. We should fix that, and not just try to hide the problem by more or less randomly deleting or moving some stuff.

What do you suggest to fix them?

[Leif Leonhardy]

Replying to vdelecroix:

Replying to leif:

Replying to dmuthiah:

Replying to leif: I also consider the current packaging system as very bad. But this ticket is about cleaning the websites mentioned in the description. And I think cleaning is a good step forward.

It's not the website that's broken, it's sage --optional, sage --experimental etc. We should fix that, and not just try to hide the problem by more or less randomly deleting or moving some stuff.

What do you suggest to fix them?

What I said, fix the function showing optional/experimental packages.

Start with build/pkgs/* (as we do in configure), and if we at all still want to list legacy spkgs as well, only show those from spkg/* on files.sagemath.org that are not in the local repo (build/pkgs/).

[Vincent Delecroix]

Replying to leif:

Replying to vdelecroix:

Replying to leif:

Replying to dmuthiah:

Replying to leif: I also consider the current packaging system as very bad. But this ticket is about cleaning the websites mentioned in the description. And I think cleaning is a good step forward.

It's not the website that's broken, it's sage --optional, sage --experimental etc. We should fix that, and not just try to hide the problem by more or less randomly deleting or moving some stuff.

What do you suggest to fix them?

What I said, fix the function showing optional/experimental packages.

Start with build/pkgs/* (as we do in configure), and if we at all still want to list legacy spkgs as well, only show those from spkg/* on files.sagemath.org that are not in the local repo (build/pkgs/).

What about TOPCOM versus topcom?

[Leif Leonhardy]

TOPCOM is old-style, all new-style packages have all-lowercase.

[Leif Leonhardy]

(There are probably other packages that got renamed, also by adding or deleting some dashes or underscores, so we perhaps need a short alias list, but in general comparing tolower(name) should suffice.)

[Leif Leonhardy]

P.P.S.: The "name" of a new-style package is its folder name in build/pkgs/ (these contain only [a-zA-Z0-9_]), while that of the (upstream) tarball, encrypted in checksums.ini -- by the way a wonderful choice, might differ.

Old-style package names are in constrast everything up to the first dash of the filename of the spkg/tarball, more likely to match what is now in checksums.ini.

[Leif Leonhardy]

And old-style packages usually have a Sage patchlevel in addition, .pN, right before the .spkg, so that should get removed before trying to match names as well.

[John Palmieri]

It's not just sage --optional etc. that may need fixing, it's also sage -i ..., which looks at the website for matches. As long as there are valid old-style packages on the website, we should search for them, so we should also clean out the crap on the website so that we don't get incorrect matches. So cleaning the website is a good idea.

By the way, although the name of each new-style spkg is lowercase, its tarball need not be. Relying on cases to determine whether a package is old or new seems like a dangerous idea.

[Vincent Delecroix]

I am cleaning the sage-list-packages script in #19213. Once this is done, the website could be cleaned as well.

[Leif Leonhardy]

Replying to jhpalmieri:

It's not just sage --optional etc. that may need fixing, it's also sage -i ..., which looks at the website for matches.

Yes, see below. We shouldn't have two implementations of the same thing (in addition not acting in the same manner).

As long as there are valid old-style packages on the website, we should search for them, so we should also clean out the crap on the website so that we don't get incorrect matches. So cleaning the website is a good idea.

While the last version of Sage without "new-style" packages was released quite a while ago, removing anything from spkg/ breaks all the old versions. I'm not aware of any announcement regarding that (although it's not unlikely I missed it), but I'm not aware of any "static" statement anywhere either.

If we do no longer support older versions, then wipe out everything below spkg/.

If there are still valuable optional/experimental packages which (after years?) haven't been converted, that's a bug, and there should be tickets for each.

By the way, although the name of each new-style spkg is lowercase, its tarball need not be. Relying on cases to determine whether a package is old or new seems like a dangerous idea.

I didn't say that. I meant if you encounter an old-style package (from the file server), strip the version, including the Sage patch level, and of course the extension, convert it to lowercase, and see if that matches any of the folder names in build/pkgs/, or everything after the equals sign up to the first dash from the first line (or the line containing tarball=) of each build/pkgs/*/checksum.ini, again converted to lowercase. [Not my design.] It's presumably sufficient to also simply remove any special characters including underscores; I doubt there will be any name clash of actually different packages.

What happened to the work of Nathann and others (about a year ago IIRC)? We have been dealing with this a couple of times in the past...

Another thing we have to solve is of course any discrepancy between sage-list-packages or whatever its name is, and (the behavior of) sage-spkg. Again, the easiest solution is to disable downloading old-style packages (automatically). That's orthogonal to disallowing installing such (sage -i /path/to/spkg).

[Vincent Delecroix]

Replying to leif:

Replying to jhpalmieri: What happened to the work of Nathann and others (about a year ago IIRC)? We have been dealing with this a couple of times in the past...

Still there and useful sage.misc.package. I am cleaning, updating this file in #19213.

[Leif Leonhardy]

Replying to vdelecroix:

Replying to leif:

Replying to jhpalmieri: What happened to the work of Nathann and others (about a year ago IIRC)? We have been dealing with this a couple of times in the past...

Still there and useful sage.misc.package. I am cleaning, updating this file in #19213.

Since we meanwhile (rather "by accident") require a system python (>=2.6) anyway, if the python files there are stand-alone (i.e., do not need the Sage library), you could use them from sage-spkg as well.

[Vincent Delecroix]

Replying to leif:

Replying to vdelecroix:

Replying to leif:

Replying to jhpalmieri: What happened to the work of Nathann and others (about a year ago IIRC)? We have been dealing with this a couple of times in the past...

Still there and useful sage.misc.package. I am cleaning, updating this file in #19213.

Since we meanwhile (rather "by accident") require a system python (>=2.6) anyway, if the python files there are stand-alone (i.e., do not need the Sage library), you could use them from sage-spkg as well.

It is more or less standalone: only needs various variables from sage.env.

[Leif Leonhardy]

But since we won't upload any new old-style packages (I guess), we could use a static list of everything what's currently there (or worth to be kept / installed) anyway. Never ever looking at spkg/{optional,experimental,huge}/list again.

[Vincent Delecroix]

This is exactly what I proposed in #19213: two files build/old_style_packages_optional and build/old_style_packages_experimental. And I would actually be happier if we move all old style optional to experimental.

[Leif Leonhardy]

Replying to vdelecroix:

This is exactly what I proposed in #19213: two files build/old_style_packages_optional and build/old_style_packages_experimental. And I would actually be happier if we move all old style optional to experimental.

Why not directly move them into build/pkgs/ (with their "simplified" names), putting legacy into type, tarball=... (and checksums) into checksums.ini, what we consider the version into package-version.txt, and if one downloads them, into upstream/ as usual. (You could even already extract spkg-{install,check} and SPKG.txt.)

Then conversion will probably happen sooner...

[Vincent Delecroix]

Replying to leif:

Replying to vdelecroix:

This is exactly what I proposed in #19213: two files build/old_style_packages_optional and build/old_style_packages_experimental. And I would actually be happier if we move all old style optional to experimental.

Why not directly move them into build/pkgs/ (with their "simplified" names), putting legacy into type, tarball=... (and checksums) into checksums.ini, what we consider the version into package-version.txt, and if one downloads them, into upstream/ as usual. (You could even already extract spkg-{install,check} and SPKG.txt.)

Then conversion will probably happen sooner...

I like very much the idea. But out of scope of #19213... I am opening a ticket for this which will be a dependency of #19213.

[Jeroen Demeyer]

Why is this not a duplicate of #19220?

[Leif Leonhardy]

Replying to jdemeyer:

Why is this not a duplicate of #19220?

I guess because #19220 deals with throwing some away, while we here (or on #19213) try to keep the valuable ones, but presumably not by leaving them a) there b) as they are, which may also involve changes to the build system (perhaps on follow-ups though)...

[Leif Leonhardy]

Or maybe not... I actually missed the ticket references in the description, but for a couple it's not obvious what the tickets actually achieve, i.e., whether a closed ticket there really means "we can safely delete that package" (for newer versions of Sage at least), or whether when all tickets referenced in the description are merged, "stop looking for packages in spkg/" in sage-spkg as well as the scripts / code listing packages.

[Leif Leonhardy]

P.S.: There's no point in moving packages when we simply stop looking there, which doesn't require completely converting all of them to (fully) new-style packages, and that's the subject here, not on #19220 I'd say.

[Jeroen Demeyer]

The list of packages here is a subset of #19220, so this is just a duplicate of #19220.

[Erik Bray]

Determined to be invalid/duplicate/wontfix (closing as "wontfix" as a catch-all resolution).