Opened 7 years ago

Closed 6 years ago

#21130 closed task (wontfix)

Cleaning fileserver.sagemath.org for optional packages

Reported by: Vincent Delecroix Owned by:
Priority: major Milestone: sage-duplicate/invalid/wontfix
Component: packages: optional Keywords:
Cc: Jeroen Demeyer, Harald Schilly, John Palmieri Merged in:
Authors: Reviewers: Jeroen Demeyer
Report Upstream: N/A Work issues:
Branch: #19213 Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by Dinakar Muthiah)

The pages http://files.sagemath.org/spkg/optional/list and http://files.sagemath.org/spkg/optional/ contain outdated information. The following are now new style packages

  • beautifulsoup-3.2.1 (pip)
  • biopython-1.61 (pip)
  • brian-1.4.1.p0 (pip)
  • coxeter3-1.1 (in #19178)
  • database_jones_numfield-v4 (in #19174)
  • database_kohel-20060803 (in #21086)
  • guppy-0.1.8 (pip)
  • lie-2.2.2.p5 (? moved in experimental in #19781)
  • mpi4py-1.2.2 (pip)
  • ore_algebra-0.2 (#19175)
  • pybtex-20120618 (pip)
  • pyx-0.10 (#21076)
  • TOPCOM-0.17.4 (now lower case topcom)
  • trac-0.11.5.p0 (pip)

See also #19220.

Change History (34)

comment:1 Changed 7 years ago by Vincent Delecroix

Cc: John Palmieri added
Description: modified (diff)

comment:2 Changed 7 years ago by John Palmieri

Description: modified (diff)

comment:3 Changed 7 years ago by John Palmieri

I think the pip packages can be removed from the server: users can download and install them using sage -i ..., which should use pip.

comment:4 Changed 7 years ago by Dinakar Muthiah

I think so. One difference will be that the package names will not appear anymore with $ sage -optional. However, that script that list packages would be easily changed. But since pip type is transitional I am not sure it is what we want.

comment:5 Changed 7 years ago by John Palmieri

See #19213.

comment:6 in reply to:  description ; Changed 7 years ago by Leif Leonhardy

Replying to vdelecroix (ticket description):

See also #21130

???


You (or we) shouldn't search for optional packages in spkg/optional/ unless there's no new-style version of a package.

All packages below spkg/ are for "backwards-compatibility", i.e., older versions of Sage.

At some point, all old-style packages should be converted, such that we can (in principle) disable the installation of legacy spkgs into newer versions of Sage.


As you may know, I've always been against the "unified repo" as is (tying any spkg to specific Sage versions and vice versa, and offering only a single version of each at the same time), which is completely against the original philosophy (that anybody could offer his/her spkg, most being self-contained, some needing Sage library support of course).

Because of this, it's difficult to generally disallow the installation of "old-style" spkgs, unless or until we have a new format and mechanism for truly "external" packages. (I suggested to split spkgs into two parts, the Sage package metadata, .spam, and an upstream tarball, years ago. We could still do the same for at least "external" packages, namely create/use a tarball with just what is now in build/pkgs/<pkgname>/, although a single plain text file à la Arch Linux or Gentoo/Funtoo would be better.)

A related problem is that our build / package system lacks any notion of version requirements, it always [en]forces to use exactly a single specific version of a package, no matter whether that's necessary or not, and true requirements are hardly documented.

comment:7 in reply to:  6 Changed 7 years ago by Leif Leonhardy

Replying to leif:

Replying to vdelecroix (ticket description):

See also #21130

???

Ah ok, fixed now.

comment:8 in reply to:  6 ; Changed 7 years ago by Dinakar Muthiah

Replying to leif:

Replying to vdelecroix (ticket description):

I did no try to look at spkg/optional. I just did sage -optional that shows wrong information (e.g. lie being optional but sage -i lie install a different experimental package). I want to be able to install all optional packages using sage -optional and this is badly broken.

I also consider the current packaging system as very bad. But this ticket is about cleaning the websites mentioned in the description. And I think cleaning is a good step forward.

comment:9 Changed 7 years ago by Dinakar Muthiah

Description: modified (diff)

Another consequence of this

$ sage -i brian
... #snip 
$ sage -optional|grep brian -C1
boost................................... 1_61_0 (not_installed)
brian................................... 1.4.1.p0 (not_installed)
buckygen................................ 1.0 (not_installed)

So +1 from removing pip packages as well...

comment:10 in reply to:  8 ; Changed 7 years ago by Leif Leonhardy

Replying to dmuthiah:

Replying to leif:

Replying to vdelecroix (ticket description):

I did no try to look at spkg/optional. I just did sage -optional that shows wrong information (e.g. lie being optional but sage -i lie install a different experimental package).

That's because the "new-style" package is now categorized "experimental".


I want to be able to install all optional packages using sage -optional and this is badly broken.

Yes, it is. Not long ago, it kept showing only legacy spkgs, now it gives an inconsistent mixture instead of what I mentioned above (namely legacy spkgs iff there's no "new-style" version at all).

I also consider the current packaging system as very bad. But this ticket is about cleaning the websites mentioned in the description. And I think cleaning is a good step forward.

It's not the website that's broken, it's sage --optional, sage --experimental etc. We should fix that, and not just try to hide the problem by more or less randomly deleting or moving some stuff.

comment:11 in reply to:  10 ; Changed 7 years ago by Vincent Delecroix

Replying to leif:

Replying to dmuthiah:

Replying to leif: I also consider the current packaging system as very bad. But this ticket is about cleaning the websites mentioned in the description. And I think cleaning is a good step forward.

It's not the website that's broken, it's sage --optional, sage --experimental etc. We should fix that, and not just try to hide the problem by more or less randomly deleting or moving some stuff.

What do you suggest to fix them?

comment:12 in reply to:  11 ; Changed 7 years ago by Leif Leonhardy

Replying to vdelecroix:

Replying to leif:

Replying to dmuthiah:

Replying to leif: I also consider the current packaging system as very bad. But this ticket is about cleaning the websites mentioned in the description. And I think cleaning is a good step forward.

It's not the website that's broken, it's sage --optional, sage --experimental etc. We should fix that, and not just try to hide the problem by more or less randomly deleting or moving some stuff.

What do you suggest to fix them?

What I said, fix the function showing optional/experimental packages.

Start with build/pkgs/* (as we do in configure), and if we at all still want to list legacy spkgs as well, only show those from spkg/* on files.sagemath.org that are not in the local repo (build/pkgs/).

comment:13 in reply to:  12 Changed 7 years ago by Vincent Delecroix

Replying to leif:

Replying to vdelecroix:

Replying to leif:

Replying to dmuthiah:

Replying to leif: I also consider the current packaging system as very bad. But this ticket is about cleaning the websites mentioned in the description. And I think cleaning is a good step forward.

It's not the website that's broken, it's sage --optional, sage --experimental etc. We should fix that, and not just try to hide the problem by more or less randomly deleting or moving some stuff.

What do you suggest to fix them?

What I said, fix the function showing optional/experimental packages.

Start with build/pkgs/* (as we do in configure), and if we at all still want to list legacy spkgs as well, only show those from spkg/* on files.sagemath.org that are not in the local repo (build/pkgs/).

What about TOPCOM versus topcom?

comment:14 Changed 7 years ago by Leif Leonhardy

TOPCOM is old-style, all new-style packages have all-lowercase.

comment:15 Changed 7 years ago by Leif Leonhardy

(There are probably other packages that got renamed, also by adding or deleting some dashes or underscores, so we perhaps need a short alias list, but in general comparing tolower(name) should suffice.)

comment:16 Changed 7 years ago by Leif Leonhardy

P.P.S.: The "name" of a new-style package is its folder name in build/pkgs/ (these contain only [a-zA-Z0-9_]), while that of the (upstream) tarball, encrypted in checksums.ini -- by the way a wonderful choice, might differ.

Old-style package names are in constrast everything up to the first dash of the filename of the spkg/tarball, more likely to match what is now in checksums.ini.

Last edited 7 years ago by Leif Leonhardy (previous) (diff)

comment:17 Changed 7 years ago by Leif Leonhardy

And old-style packages usually have a Sage patchlevel in addition, .pN, right before the .spkg, so that should get removed before trying to match names as well.

comment:18 Changed 7 years ago by John Palmieri

It's not just sage --optional etc. that may need fixing, it's also sage -i ..., which looks at the website for matches. As long as there are valid old-style packages on the website, we should search for them, so we should also clean out the crap on the website so that we don't get incorrect matches. So cleaning the website is a good idea.

By the way, although the name of each new-style spkg is lowercase, its tarball need not be. Relying on cases to determine whether a package is old or new seems like a dangerous idea.

comment:19 Changed 7 years ago by Vincent Delecroix

Branch: #19213

I am cleaning the sage-list-packages script in #19213. Once this is done, the website could be cleaned as well.

comment:20 in reply to:  18 ; Changed 7 years ago by Leif Leonhardy

Replying to jhpalmieri:

It's not just sage --optional etc. that may need fixing, it's also sage -i ..., which looks at the website for matches.

Yes, see below. We shouldn't have two implementations of the same thing (in addition not acting in the same manner).


As long as there are valid old-style packages on the website, we should search for them, so we should also clean out the crap on the website so that we don't get incorrect matches. So cleaning the website is a good idea.

While the last version of Sage without "new-style" packages was released quite a while ago, removing anything from spkg/ breaks all the old versions. I'm not aware of any announcement regarding that (although it's not unlikely I missed it), but I'm not aware of any "static" statement anywhere either.

If we do no longer support older versions, then wipe out everything below spkg/.

If there are still valuable optional/experimental packages which (after years?) haven't been converted, that's a bug, and there should be tickets for each.


By the way, although the name of each new-style spkg is lowercase, its tarball need not be. Relying on cases to determine whether a package is old or new seems like a dangerous idea.

I didn't say that. I meant if you encounter an old-style package (from the file server), strip the version, including the Sage patch level, and of course the extension, convert it to lowercase, and see if that matches any of the folder names in build/pkgs/, or everything after the equals sign up to the first dash from the first line (or the line containing tarball=) of each build/pkgs/*/checksum.ini, again converted to lowercase. [Not my design.] It's presumably sufficient to also simply remove any special characters including underscores; I doubt there will be any name clash of actually different packages.

What happened to the work of Nathann and others (about a year ago IIRC)? We have been dealing with this a couple of times in the past...

Another thing we have to solve is of course any discrepancy between sage-list-packages or whatever its name is, and (the behavior of) sage-spkg. Again, the easiest solution is to disable downloading old-style packages (automatically). That's orthogonal to disallowing installing such (sage -i /path/to/spkg).

comment:21 in reply to:  20 ; Changed 7 years ago by Vincent Delecroix

Replying to leif:

Replying to jhpalmieri: What happened to the work of Nathann and others (about a year ago IIRC)? We have been dealing with this a couple of times in the past...

Still there and useful sage.misc.package. I am cleaning, updating this file in #19213.

comment:22 in reply to:  21 ; Changed 7 years ago by Leif Leonhardy

Replying to vdelecroix:

Replying to leif:

Replying to jhpalmieri: What happened to the work of Nathann and others (about a year ago IIRC)? We have been dealing with this a couple of times in the past...

Still there and useful sage.misc.package. I am cleaning, updating this file in #19213.

Since we meanwhile (rather "by accident") require a system python (>=2.6) anyway, if the python files there are stand-alone (i.e., do not need the Sage library), you could use them from sage-spkg as well.

comment:23 in reply to:  22 Changed 7 years ago by Vincent Delecroix

Replying to leif:

Replying to vdelecroix:

Replying to leif:

Replying to jhpalmieri: What happened to the work of Nathann and others (about a year ago IIRC)? We have been dealing with this a couple of times in the past...

Still there and useful sage.misc.package. I am cleaning, updating this file in #19213.

Since we meanwhile (rather "by accident") require a system python (>=2.6) anyway, if the python files there are stand-alone (i.e., do not need the Sage library), you could use them from sage-spkg as well.

It is more or less standalone: only needs various variables from sage.env.

comment:24 Changed 7 years ago by Leif Leonhardy

But since we won't upload any new old-style packages (I guess), we could use a static list of everything what's currently there (or worth to be kept / installed) anyway. Never ever looking at spkg/{optional,experimental,huge}/list again.

comment:25 Changed 7 years ago by Vincent Delecroix

This is exactly what I proposed in #19213: two files build/old_style_packages_optional and build/old_style_packages_experimental. And I would actually be happier if we move all old style optional to experimental.

Last edited 7 years ago by Vincent Delecroix (previous) (diff)

comment:26 in reply to:  25 ; Changed 7 years ago by Leif Leonhardy

Replying to vdelecroix:

This is exactly what I proposed in #19213: two files build/old_style_packages_optional and build/old_style_packages_experimental. And I would actually be happier if we move all old style optional to experimental.

Why not directly move them into build/pkgs/ (with their "simplified" names), putting legacy into type, tarball=... (and checksums) into checksums.ini, what we consider the version into package-version.txt, and if one downloads them, into upstream/ as usual. (You could even already extract spkg-{install,check} and SPKG.txt.)

Then conversion will probably happen sooner...

comment:27 in reply to:  26 Changed 7 years ago by Vincent Delecroix

Replying to leif:

Replying to vdelecroix:

This is exactly what I proposed in #19213: two files build/old_style_packages_optional and build/old_style_packages_experimental. And I would actually be happier if we move all old style optional to experimental.

Why not directly move them into build/pkgs/ (with their "simplified" names), putting legacy into type, tarball=... (and checksums) into checksums.ini, what we consider the version into package-version.txt, and if one downloads them, into upstream/ as usual. (You could even already extract spkg-{install,check} and SPKG.txt.)

Then conversion will probably happen sooner...

I like very much the idea. But out of scope of #19213... I am opening a ticket for this which will be a dependency of #19213.

comment:28 Changed 7 years ago by Jeroen Demeyer

Why is this not a duplicate of #19220?

comment:29 in reply to:  28 Changed 7 years ago by Leif Leonhardy

Replying to jdemeyer:

Why is this not a duplicate of #19220?

I guess because #19220 deals with throwing some away, while we here (or on #19213) try to keep the valuable ones, but presumably not by leaving them a) there b) as they are, which may also involve changes to the build system (perhaps on follow-ups though)...

comment:30 Changed 7 years ago by Leif Leonhardy

Or maybe not... I actually missed the ticket references in the description, but for a couple it's not obvious what the tickets actually achieve, i.e., whether a closed ticket there really means "we can safely delete that package" (for newer versions of Sage at least), or whether when all tickets referenced in the description are merged, "stop looking for packages in spkg/" in sage-spkg as well as the scripts / code listing packages.

comment:31 Changed 7 years ago by Leif Leonhardy

P.S.: There's no point in moving packages when we simply stop looking there, which doesn't require completely converting all of them to (fully) new-style packages, and that's the subject here, not on #19220 I'd say.

comment:32 Changed 7 years ago by Jeroen Demeyer

Milestone: sage-7.3sage-duplicate/invalid/wontfix
Reviewers: Jeroen Demeyer
Status: newneeds_review

The list of packages here is a subset of #19220, so this is just a duplicate of #19220.

comment:33 Changed 7 years ago by Jeroen Demeyer

Status: needs_reviewpositive_review

comment:34 Changed 6 years ago by Erik Bray

Resolution: wontfix
Status: positive_reviewclosed

Determined to be invalid/duplicate/wontfix (closing as "wontfix" as a catch-all resolution).

Note: See TracTickets for help on using tickets.