Opened 7 years ago
Closed 6 years ago
#21130 closed task (wontfix)
Cleaning fileserver.sagemath.org for optional packages
Reported by: | Vincent Delecroix | Owned by: | |
---|---|---|---|
Priority: | major | Milestone: | sage-duplicate/invalid/wontfix |
Component: | packages: optional | Keywords: | |
Cc: | Jeroen Demeyer, Harald Schilly, John Palmieri | Merged in: | |
Authors: | Reviewers: | Jeroen Demeyer | |
Report Upstream: | N/A | Work issues: | |
Branch: | #19213 | Commit: | |
Dependencies: | Stopgaps: |
Description (last modified by )
The pages http://files.sagemath.org/spkg/optional/list and http://files.sagemath.org/spkg/optional/ contain outdated information. The following are now new style packages
- beautifulsoup-3.2.1 (pip)
- biopython-1.61 (pip)
- brian-1.4.1.p0 (pip)
- coxeter3-1.1 (in #19178)
- database_jones_numfield-v4 (in #19174)
- database_kohel-20060803 (in #21086)
- guppy-0.1.8 (pip)
- lie-2.2.2.p5 (? moved in experimental in #19781)
- mpi4py-1.2.2 (pip)
- ore_algebra-0.2 (#19175)
- pybtex-20120618 (pip)
- pyx-0.10 (#21076)
- TOPCOM-0.17.4 (now lower case topcom)
- trac-0.11.5.p0 (pip)
See also #19220.
Change History (34)
comment:1 Changed 7 years ago by
Cc: | John Palmieri added |
---|---|
Description: | modified (diff) |
comment:2 Changed 7 years ago by
Description: | modified (diff) |
---|
comment:3 Changed 7 years ago by
comment:4 Changed 7 years ago by
I think so. One difference will be that the package names will not appear anymore with $ sage -optional
. However, that script that list packages would be easily changed. But since pip type is transitional I am not sure it is what we want.
comment:6 follow-ups: 7 8 Changed 7 years ago by
Replying to vdelecroix (ticket description):
See also #21130
???
You (or we) shouldn't search for optional packages in spkg/optional/
unless there's no new-style version of a package.
All packages below spkg/
are for "backwards-compatibility", i.e., older versions of Sage.
At some point, all old-style packages should be converted, such that we can (in principle) disable the installation of legacy spkgs into newer versions of Sage.
As you may know, I've always been against the "unified repo" as is (tying any spkg to specific Sage versions and vice versa, and offering only a single version of each at the same time), which is completely against the original philosophy (that anybody could offer his/her spkg, most being self-contained, some needing Sage library support of course).
Because of this, it's difficult to generally disallow the installation of "old-style" spkgs, unless or until we have a new format and mechanism for truly "external" packages. (I suggested to split spkgs into two parts, the Sage package metadata, .spam
, and an upstream tarball, years ago. We could still do the same for at least "external" packages, namely create/use a tarball with just what is now in build/pkgs/<pkgname>/
, although a single plain text file à la Arch Linux or Gentoo/Funtoo would be better.)
A related problem is that our build / package system lacks any notion of version requirements, it always [en]forces to use exactly a single specific version of a package, no matter whether that's necessary or not, and true requirements are hardly documented.
comment:7 Changed 7 years ago by
Replying to leif:
Replying to vdelecroix (ticket description):
See also #21130
???
Ah ok, fixed now.
comment:8 follow-up: 10 Changed 7 years ago by
Replying to leif:
Replying to vdelecroix (ticket description):
I did no try to look at spkg/optional
. I just did sage -optional
that shows wrong information (e.g. lie being optional but sage -i lie
install a different experimental package). I want to be able to install all optional packages using sage -optional
and this is badly broken.
I also consider the current packaging system as very bad. But this ticket is about cleaning the websites mentioned in the description. And I think cleaning is a good step forward.
comment:9 Changed 7 years ago by
Description: | modified (diff) |
---|
Another consequence of this
$ sage -i brian ... #snip $ sage -optional|grep brian -C1 boost................................... 1_61_0 (not_installed) brian................................... 1.4.1.p0 (not_installed) buckygen................................ 1.0 (not_installed)
So +1 from removing pip packages as well...
comment:10 follow-up: 11 Changed 7 years ago by
Replying to dmuthiah:
Replying to leif:
Replying to vdelecroix (ticket description):
I did no try to look at
spkg/optional
. I just didsage -optional
that shows wrong information (e.g. lie being optional butsage -i lie
install a different experimental package).
That's because the "new-style" package is now categorized "experimental".
I want to be able to install all optional packages using
sage -optional
and this is badly broken.
Yes, it is. Not long ago, it kept showing only legacy spkgs, now it gives an inconsistent mixture instead of what I mentioned above (namely legacy spkgs iff there's no "new-style" version at all).
I also consider the current packaging system as very bad. But this ticket is about cleaning the websites mentioned in the description. And I think cleaning is a good step forward.
It's not the website that's broken, it's sage --optional
, sage --experimental
etc. We should fix that, and not just try to hide the problem by more or less randomly deleting or moving some stuff.
comment:11 follow-up: 12 Changed 7 years ago by
Replying to leif:
Replying to dmuthiah:
Replying to leif: I also consider the current packaging system as very bad. But this ticket is about cleaning the websites mentioned in the description. And I think cleaning is a good step forward.
It's not the website that's broken, it's
sage --optional
,sage --experimental
etc. We should fix that, and not just try to hide the problem by more or less randomly deleting or moving some stuff.
What do you suggest to fix them?
comment:12 follow-up: 13 Changed 7 years ago by
Replying to vdelecroix:
Replying to leif:
Replying to dmuthiah:
Replying to leif: I also consider the current packaging system as very bad. But this ticket is about cleaning the websites mentioned in the description. And I think cleaning is a good step forward.
It's not the website that's broken, it's
sage --optional
,sage --experimental
etc. We should fix that, and not just try to hide the problem by more or less randomly deleting or moving some stuff.What do you suggest to fix them?
What I said, fix the function showing optional/experimental packages.
Start with build/pkgs/*
(as we do in configure
), and if we at all still want to list legacy spkgs as well, only show those from spkg/*
on files.sagemath.org that are not in the local repo (build/pkgs/
).
comment:13 Changed 7 years ago by
Replying to leif:
Replying to vdelecroix:
Replying to leif:
Replying to dmuthiah:
Replying to leif: I also consider the current packaging system as very bad. But this ticket is about cleaning the websites mentioned in the description. And I think cleaning is a good step forward.
It's not the website that's broken, it's
sage --optional
,sage --experimental
etc. We should fix that, and not just try to hide the problem by more or less randomly deleting or moving some stuff.What do you suggest to fix them?
What I said, fix the function showing optional/experimental packages.
Start with
build/pkgs/*
(as we do inconfigure
), and if we at all still want to list legacy spkgs as well, only show those fromspkg/*
on files.sagemath.org that are not in the local repo (build/pkgs/
).
What about TOPCOM
versus topcom
?
comment:15 Changed 7 years ago by
(There are probably other packages that got renamed, also by adding or deleting some dashes or underscores, so we perhaps need a short alias list, but in general comparing tolower(name) should suffice.)
comment:16 Changed 7 years ago by
P.P.S.: The "name" of a new-style package is its folder name in build/pkgs/
(these contain only [a-zA-Z0-9_]
), while that of the (upstream) tarball, encrypted in checksums.ini
-- by the way a wonderful choice, might differ.
Old-style package names are in constrast everything up to the first dash of the filename of the spkg/tarball, more likely to match what is now in checksums.ini
.
comment:17 Changed 7 years ago by
And old-style packages usually have a Sage patchlevel in addition, .pN
, right before the .spkg
, so that should get removed before trying to match names as well.
comment:18 follow-up: 20 Changed 7 years ago by
It's not just sage --optional
etc. that may need fixing, it's also sage -i ...
, which looks at the website for matches. As long as there are valid old-style packages on the website, we should search for them, so we should also clean out the crap on the website so that we don't get incorrect matches. So cleaning the website is a good idea.
By the way, although the name of each new-style spkg is lowercase, its tarball need not be. Relying on cases to determine whether a package is old or new seems like a dangerous idea.
comment:19 Changed 7 years ago by
Branch: | → #19213 |
---|
I am cleaning the sage-list-packages script in #19213. Once this is done, the website could be cleaned as well.
comment:20 follow-up: 21 Changed 7 years ago by
Replying to jhpalmieri:
It's not just
sage --optional
etc. that may need fixing, it's alsosage -i ...
, which looks at the website for matches.
Yes, see below. We shouldn't have two implementations of the same thing (in addition not acting in the same manner).
As long as there are valid old-style packages on the website, we should search for them, so we should also clean out the crap on the website so that we don't get incorrect matches. So cleaning the website is a good idea.
While the last version of Sage without "new-style" packages was released quite a while ago, removing anything from spkg/
breaks all the old versions. I'm not aware of any announcement regarding that (although it's not unlikely I missed it), but I'm not aware of any "static" statement anywhere either.
If we do no longer support older versions, then wipe out everything below spkg/
.
If there are still valuable optional/experimental packages which (after years?) haven't been converted, that's a bug, and there should be tickets for each.
By the way, although the name of each new-style spkg is lowercase, its tarball need not be. Relying on cases to determine whether a package is old or new seems like a dangerous idea.
I didn't say that. I meant if you encounter an old-style package (from the file server), strip the version, including the Sage patch level, and of course the extension, convert it to lowercase, and see if that matches any of the folder names in build/pkgs/
, or everything after the equals sign up to the first dash from the first line (or the line containing tarball=
) of each build/pkgs/*/checksum.ini
, again converted to lowercase. [Not my design.] It's presumably sufficient to also simply remove any special characters including underscores; I doubt there will be any name clash of actually different packages.
What happened to the work of Nathann and others (about a year ago IIRC)? We have been dealing with this a couple of times in the past...
Another thing we have to solve is of course any discrepancy between sage-list-packages
or whatever its name is, and (the behavior of) sage-spkg
. Again, the easiest solution is to disable downloading old-style packages (automatically). That's orthogonal to disallowing installing such (sage -i /path/to/spkg
).
comment:21 follow-up: 22 Changed 7 years ago by
Replying to leif:
Replying to jhpalmieri: What happened to the work of Nathann and others (about a year ago IIRC)? We have been dealing with this a couple of times in the past...
Still there and useful sage.misc.package
. I am cleaning, updating this file in #19213.
comment:22 follow-up: 23 Changed 7 years ago by
Replying to vdelecroix:
Replying to leif:
Replying to jhpalmieri: What happened to the work of Nathann and others (about a year ago IIRC)? We have been dealing with this a couple of times in the past...
Still there and useful
sage.misc.package
. I am cleaning, updating this file in #19213.
Since we meanwhile (rather "by accident") require a system python (>=2.6) anyway, if the python files there are stand-alone (i.e., do not need the Sage library), you could use them from sage-spkg
as well.
comment:23 Changed 7 years ago by
Replying to leif:
Replying to vdelecroix:
Replying to leif:
Replying to jhpalmieri: What happened to the work of Nathann and others (about a year ago IIRC)? We have been dealing with this a couple of times in the past...
Still there and useful
sage.misc.package
. I am cleaning, updating this file in #19213.Since we meanwhile (rather "by accident") require a system python (>=2.6) anyway, if the python files there are stand-alone (i.e., do not need the Sage library), you could use them from
sage-spkg
as well.
It is more or less standalone: only needs various variables from sage.env
.
comment:24 Changed 7 years ago by
But since we won't upload any new old-style packages (I guess), we could use a static list of everything what's currently there (or worth to be kept / installed) anyway. Never ever looking at spkg/{optional,experimental,huge}/list
again.
comment:25 follow-up: 26 Changed 7 years ago by
This is exactly what I proposed in #19213: two files build/old_style_packages_optional
and build/old_style_packages_experimental
. And I would actually be happier if we move all old style optional to experimental.
comment:26 follow-up: 27 Changed 7 years ago by
Replying to vdelecroix:
This is exactly what I proposed in #19213: two files
build/old_style_packages_optional
andbuild/old_style_packages_experimental
. And I would actually be happier if we move all old style optional to experimental.
Why not directly move them into build/pkgs/
(with their "simplified" names), putting legacy
into type
, tarball=...
(and checksums) into checksums.ini
, what we consider the version into package-version.txt
, and if one downloads them, into upstream/
as usual. (You could even already extract spkg-{install,check}
and SPKG.txt
.)
Then conversion will probably happen sooner...
comment:27 Changed 7 years ago by
Replying to leif:
Replying to vdelecroix:
This is exactly what I proposed in #19213: two files
build/old_style_packages_optional
andbuild/old_style_packages_experimental
. And I would actually be happier if we move all old style optional to experimental.Why not directly move them into
build/pkgs/
(with their "simplified" names), puttinglegacy
intotype
,tarball=...
(and checksums) intochecksums.ini
, what we consider the version intopackage-version.txt
, and if one downloads them, intoupstream/
as usual. (You could even already extractspkg-{install,check}
andSPKG.txt
.)Then conversion will probably happen sooner...
I like very much the idea. But out of scope of #19213... I am opening a ticket for this which will be a dependency of #19213.
comment:29 Changed 7 years ago by
Replying to jdemeyer:
Why is this not a duplicate of #19220?
I guess because #19220 deals with throwing some away, while we here (or on #19213) try to keep the valuable ones, but presumably not by leaving them a) there b) as they are, which may also involve changes to the build system (perhaps on follow-ups though)...
comment:30 Changed 7 years ago by
Or maybe not... I actually missed the ticket references in the description, but for a couple it's not obvious what the tickets actually achieve, i.e., whether a closed ticket there really means "we can safely delete that package" (for newer versions of Sage at least), or whether when all tickets referenced in the description are merged, "stop looking for packages in spkg/
" in sage-spkg
as well as the scripts / code listing packages.
comment:31 Changed 7 years ago by
P.S.: There's no point in moving packages when we simply stop looking there, which doesn't require completely converting all of them to (fully) new-style packages, and that's the subject here, not on #19220 I'd say.
comment:32 Changed 7 years ago by
Milestone: | sage-7.3 → sage-duplicate/invalid/wontfix |
---|---|
Reviewers: | → Jeroen Demeyer |
Status: | new → needs_review |
comment:33 Changed 7 years ago by
Status: | needs_review → positive_review |
---|
comment:34 Changed 6 years ago by
Resolution: | → wontfix |
---|---|
Status: | positive_review → closed |
Determined to be invalid/duplicate/wontfix (closing as "wontfix" as a catch-all resolution).
I think the pip packages can be removed from the server: users can download and install them using
sage -i ...
, which should use pip.