Opened 6 years ago
Closed 5 years ago
#18408 closed enhancement (fixed)
List of packages in the documentation of sage.misc.package
Reported by:  ncohen  Owned by:  

Priority:  major  Milestone:  sage7.1 
Component:  documentation  Keywords:  
Cc:  vdelecroix, leif  Merged in:  
Authors:  Nathann Cohen  Reviewers:  Javier Honrubia González 
Report Upstream:  N/A  Work issues:  
Branch:  15b9583 (Commits, GitHub, GitLab)  Commit:  15b95833e3928fb055ca8d34f2cea2ca646dc236 
Dependencies:  Stopgaps: 
Description
It seems that we do not have an uptodate list of Sage's packages in our documentation.
This branch solves the problem, by adding the list of packages in our reference manual (in sage.misc.package
, where all functions dealing with packages are defined). This list is *generated* from python lists, so that they are:
 Easy to read (in a nice sorted table)
 Checked by a doctest, so that we get a warning if they are not up to date anymore
Nathann
Change History (61)
comment:1 Changed 6 years ago by
 Branch set to public/18408
 Status changed from new to needs_review
comment:2 Changed 6 years ago by
 Commit set to f43eb0119fb60ea1367247bcf383506782e4e81a
comment:3 Changed 6 years ago by
 Commit changed from f43eb0119fb60ea1367247bcf383506782e4e81a to 4a968a88a8eb55bad8dd0f88ccaa1a856eb9c092
Branch pushed to git repo; I updated commit sha1. New commits:
4a968a8  trac #18408: A leftover from a past test

comment:4 Changed 6 years ago by
As mentioned, those package lists refer to Sage versions < 6.0.
comment:5 Changed 6 years ago by
 Status changed from needs_review to needs_info
What's up with this?
singular31
(there are many more optional packages like this)
comment:6 followup: ↓ 10 Changed 6 years ago by
 Status changed from needs_info to needs_work
Some packages from build/pkgs
don't appear in your list (arb
is the first, but there are more).
comment:7 Changed 6 years ago by
And extcode
, seriously? That's no longer a package...
comment:8 Changed 6 years ago by
gcc
appears in every list :)
comment:9 Changed 6 years ago by
 Commit changed from 4a968a88a8eb55bad8dd0f88ccaa1a856eb9c092 to 869b195106b1fcd06332a38d12ac2a5bcd2d86ec
Branch pushed to git repo; I updated commit sha1. New commits:
869b195  trac #18408: Change the regexp

comment:10 in reply to: ↑ 6 ; followup: ↓ 12 Changed 6 years ago by
Some packages from
build/pkgs
don't appear in your list (arb
is the first, but there are more). And extcode, seriously? That's no longer a package... gcc appears in every list :)
Jeroen, please do not ask me to rewrite everything at once. I am no magician. As you can see from the source code, the list of packages is generated from the output of sage standard
, sage optional
and sage experimental
. And yes, all three contain gcc. And extcode
. The point of this branch is that we see the list somewhere, and apparently it also helps us notice other bugs. Please do not force me to rewrite everything here.
Nathann
comment:11 Changed 6 years ago by
 Status changed from needs_work to needs_review
comment:12 in reply to: ↑ 10 Changed 6 years ago by
 Status changed from needs_review to needs_work
Replying to ncohen:
Jeroen, please do not ask me to rewrite everything at once. I am no magician. As you can see from the source code, the list of packages is generated from the output of
sage standard
,sage optional
andsage experimental
. And yes, all three contain gcc. Andextcode
. The point of this branch is that we see the list somewhere, and apparently it also helps us notice other bugs. Please do not force me to rewrite everything here.
Well, the logical thing to do would be to first fix sage standard
, sage optional
and sage experimental
and then add the lists to the documentation.
I really do not see the point of adding wrong and outdated lists to the documentation.
And in any case, this is wrong: singular31
comment:13 Changed 6 years ago by
Why is singular wrong? It displays singular, and only singular?..
comment:14 followups: ↓ 16 ↓ 17 Changed 6 years ago by
FWIW, I'm currently working on categorizing the packages (obsolete, removed, renamed, replaced, newly added...)
If we fix sagelistpackages
w.r.t. this, Nathann's work still makes sense (without modifying his code, just regenerating the "hardcoded" lists, I think).
comment:15 Changed 6 years ago by
Sorry, I didn't realize that you changed the singular
thing. Never mind...
comment:16 in reply to: ↑ 14 ; followups: ↓ 18 ↓ 19 Changed 6 years ago by
Replying to leif:
If we fix
sagelistpackages
w.r.t. this, Nathann's work still makes sense (without modifying his code, just regenerating the "hardcoded" lists, I think).
Of course Nathann's work makes sense, but obviously this ticket should depend on fixing the lists.
comment:17 in reply to: ↑ 14 Changed 6 years ago by
If we fix
sagelistpackages
w.r.t. this, Nathann's work still makes sense (without modifying his code, just regenerating the "hardcoded" lists, I think).
Indeed. Plus my goal is also to add a list more exhaustive than the one our doc currently contains:
http://www.sagemath.org/doc/installation/introduction.html
I am rewriting the doc of that manual currently, and moving a lot of things around. See #18405
Nathann
comment:18 in reply to: ↑ 16 ; followup: ↓ 21 Changed 6 years ago by
Of course Nathann's work makes sense, but obviously this ticket should depend on fixing the lists.
Why should it *depend* on it? The list currenty generated by sage standard
is wrong at the moment (not caused by this patch) and what this patch does will still work when that will be fixed.
In the meantime we will have a "more correct" list of packages, I will be able to remove the list of Sage packages from the 'installation' document (and instead link to that list). There is no gain in blocking this ticket until the other problem is fixed.
Nathann
comment:19 in reply to: ↑ 16 Changed 6 years ago by
Replying to jdemeyer:
obviously this ticket should depend on fixing the lists.
Well, it's just as borked as sagelistpackages
and our package repositories, so I wouldn't even mind merging it as is... ;)
comment:20 followups: ↓ 22 ↓ 23 Changed 6 years ago by
Remove this:
pname = re.sub("(\.)p[09]+$","",pname) # strip .p0
(version numbers are always separated by a 
, the .p0
is added in addition to a version number)
comment:21 in reply to: ↑ 18 Changed 6 years ago by
Replying to ncohen:
In the meantime we will have a "more correct" list of packages
The lists in this ticket are far from correct.
There is no gain in blocking this ticket until the other problem is fixed.
Yes, there is gain: Sage will have less wrong documentation.
I think wrong documentation is worse than no documentation.
comment:22 in reply to: ↑ 20 ; followup: ↓ 25 Changed 6 years ago by
Replying to jdemeyer:
Remove this:
pname = re.sub("(\.)p[09]+$","",pname) # strip .p0(version numbers are always separated by a

, the.p0
is added in addition to a version number)
Nope, we do have packages with a patchlevel but no version.
comment:23 in reply to: ↑ 20 Changed 6 years ago by
(version numbers are always separated by a

, the.p0
is added in addition to a version number)
That's what I also believed at first
~$ sage experimental  grep  "p" pyniftip0 ............................. not installed
comment:24 Changed 6 years ago by
That's p0
, not .p0
comment:25 in reply to: ↑ 22 ; followup: ↓ 26 Changed 6 years ago by
comment:26 in reply to: ↑ 25 Changed 6 years ago by
Replying to leif:
Such as
database_stein_watkins_mini.p0.spkg
for example.
 The name of that package actually is
database_stein_watkins_mini.p0
(of course, that's a wrong name).  That package no longer exists anyway (the wrong name was fixed).
comment:27 followup: ↓ 29 Changed 6 years ago by
Let's see it this way Jeroen: if you have reasons to believe that the list from sage standard
(which I believe is the same as what you see there [1]) is wrong then why don't you report it so that we update it (and my code will match it)?
My question at [2] was about whether we have some (any) reliable source containing our packages, for each category. It seems that we have none, so here is the deal: if you see something wrong in the list (i.e. the online one, or sage standard
) then let us fix that. Otherwise you are telling me that "the list is wrong" without telling me how exactly.
On the other hand, you cannot block this patch until the "problematic way in which the packages are handled in Sage" is fixed. All I want to do here is clear our doc, and make it more reliable *than it is*.
Nathann
[1] http://www.sagemath.org/packages/standard/ [2] https://groups.google.com/forum/#!topic/sagedevel/aEmUmFOwJYQ
comment:28 Changed 6 years ago by
So, here it is.
As of Sage 6.7.beta4, we have:
Obsolete packages (since Sage 6.0), *.spkg
:
 extcode
 sage (also used to be versioned)
 sage_root
 sage_scripts
Renamed packages:
 libm4ri > m4ri
 libm4rie > m4rie
 database_stein_watkins_mini.p0 > database_stein_watkins_mini (optional)
 TOPCOM > topcom (optional; "newstyle" is lowercase)
 gd > libgd ? (probably replaced, see below)
Removed (standard) packages:
 genus2reduction (now part of the Sage library IIRC)
 mercurial (in principle replaced by git, but no "dropin" replacement)
 sqlalchemy
Replaced (standard) packages:
 pil > Pillow
 gd, gdmodule > libgd (I think)
Newly added (standard) packages:
 backports_ssl_match_hostname (WTF?)
 certifi
 combinatorial_designs
 configure (see below)
 csage (see below)
 dateutil
 git (used to be optional, so there's still an old one, too)
 jsonschema
 libgd (presumably the replacement of gd, gdmodule; see above)
 markupsafe
 mathjax
 mistune
 pari_galdata
 pari_seadata_small
 pip
 pkgconf
 pkgconfig
 prereq (see below)
 pyparsing
 pyzmq
 six
 tornado
 zeromq
Then we have some weirdness with the following "packages":
 in
build/pkgs/
, but not (listed as) installed: configure
 (listed as) installed, but not in
build/pkgs/
, and without version: csage
 sage ("conflicts" with the previous, versioned spkg in Sage 5.x)
 prereq
And then we have new "newstyle" packages (just in the git repo) which have
no legacy (*.spkg
) counterpart, hence currently not listed by sagelistpackages
:
 arb (optional)
 ...
... besides that so far only some optional packages have been converted to "newstyle"; the versions in the git repo might meanwhile be newer than their legacy counterparts. (Similar for packages that used to be optional but are standard now, cf. "newly added packages" above.)
comment:29 in reply to: ↑ 27 ; followup: ↓ 30 Changed 6 years ago by
Replying to ncohen:
All I want to do here is clear our doc, and make it more reliable *than it is*.
Again, please tell me why adding wrong documentation is making it more reliable.
comment:30 in reply to: ↑ 29 Changed 6 years ago by
Again, please tell me why adding wrong documentation is making it more reliable.
Well, because that new page will match the most reliable information we have (i.e. sage standard
) in the future, and because by having this new page I will be able to remove the other pages whose information is even older, and redirect them toward that page.
Nathann
comment:31 followup: ↓ 33 Changed 6 years ago by
gdmodule
has genuinly been removed. And gd
was renamed to libgd
.
comment:32 followup: ↓ 45 Changed 6 years ago by
You seem really bothered by the information I am adding, and not bothered at all by the wrong information which stays available for as long as we do nothing.
comment:33 in reply to: ↑ 31 ; followup: ↓ 34 Changed 6 years ago by
gdmodule
has genuinly be removed. Andgd
was renamed tolibgd
.
I just generated the list from our build/install file:
http://www.steinertriples.fr/ncohen/tmp/std
I was about to write to sagedevel to ask <whoever can> to put it online, but in the meantime I noticed that the official list has been removed: http://www.sagemath.org/packages/standard/list
So something is apparently happening.
Nathann
comment:34 in reply to: ↑ 33 Changed 6 years ago by
Replying to ncohen:
I was about to write to sagedevel to ask <whoever can> to put it online, but in the meantime I noticed that the official list has been removed: http://www.sagemath.org/packages/standard/list
The site is currently moving, while packages/
is obsolete anyway, since it is incompatible with the mirrors. Take spkg/standard/list
from some mirror instead.
comment:35 followup: ↓ 37 Changed 6 years ago by
Well I do not want to take that list, I want to replace it with the list I just generated from the latest beta :P
comment:36 Changed 6 years ago by
Even the reference manual was removed O_o
comment:37 in reply to: ↑ 35 Changed 6 years ago by
Replying to ncohen:
Well I do not want to take that list, I want to replace it with the list I just generated from the latest beta
:P
That doesn't make much sense, since
 the
list
files are generated from the directory contents,  the site is moving, as I said.
We'll have to wait with such at least until the infrastructure is stable again (hopefully in a few days).
comment:38 Changed 6 years ago by
 Dependencies changed from #18407 to #18407, #18456
comment:39 Changed 6 years ago by
 Commit changed from 869b195106b1fcd06332a38d12ac2a5bcd2d86ec to 868fa71f00ed26bf97158e184a93fe8517dd211e
Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
3efc9fb  trac #18431: packagespecific 'dependencies' files

308ceff  trac #18431: Autogenerate make rules for standard packages

77f293f  trac #18431: autogenerated Make target "sagestandardpackages"

e114874  trac #18431: typo

766798e  trac #18431: tab>spaces

47e438d  Merge tag '6.7' into t/18441/base_packages_except_configure_should_be_standard

4de8e37  Various changes to build system

f83b0c4  Correct variable name

39cc007  trac #18456: ReFix standard_packages(), optional_packages(), and experimental_packages()

868fa71  trac #18408: Merged with #18456

comment:40 Changed 6 years ago by
 Status changed from needs_work to needs_review
comment:41 Changed 6 years ago by
 Commit changed from 868fa71f00ed26bf97158e184a93fe8517dd211e to ac93f992452bebd569183f8f032d5b71c086dbf4
comment:42 Changed 6 years ago by
 Status changed from needs_review to needs_work
(small problems with git history)
comment:43 Changed 6 years ago by
 Commit changed from ac93f992452bebd569183f8f032d5b71c086dbf4 to a8b9103987226c9a5768c4f571840da3fd7aef19
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
a3402c9  trac #18456: ReFix standard_packages(), optional_packages(), and experimental_packages()

687e426  trac #18456: Column names

d671f1b  trac #18456: don't use explicitly '/' in file paths

b093007  trac #18456: doc and doctest

a8b9103  trac #18408: List of packages in the documentation of sage.misc.package

comment:44 Changed 6 years ago by
 Status changed from needs_work to needs_review
comment:45 in reply to: ↑ 32 ; followup: ↓ 46 Changed 6 years ago by
 Milestone changed from sage6.7 to sage6.8
 Status changed from needs_review to needs_work
 Work issues set to Remove obsolete packages
Replying to ncohen:
You seem really bothered by the information I am adding, and not bothered at all by the wrong information which stays available for as long as we do nothing.
I am bothered by both. The fact that one thing is wrong is not an excuse to add a second wrong thing.
This patch should not be merged as long as stuff like extcode
is listed.
comment:46 in reply to: ↑ 45 Changed 6 years ago by
This patch should not be merged as long as stuff like
extcode
is listed.
What is wrong with extcode?
comment:47 Changed 6 years ago by
 Status changed from needs_work to needs_review
 Work issues Remove obsolete packages deleted
It seems that updating the online packages is harder than I thought. And I also feel slightly guilty for "forcing" #18581 against your will. So I will stop complaining about this ticket.
comment:48 Changed 6 years ago by
Thanks. And I hope that they will end up updating the lists at some point :/
comment:49 Changed 6 years ago by
 Commit changed from a8b9103987226c9a5768c4f571840da3fd7aef19 to a9a45bf4314373cda000c34cf8296b457acb5fde
comment:50 Changed 6 years ago by
The lists have been updated. And so, this patch too!
Nathann
comment:51 followup: ↓ 53 Changed 6 years ago by
These are no longer standard:
gd...................................... 2.0.35.p7 (not_installed) gdmodule................................ 0.56.p8 (not_installed) genus2reduction......................... 0.3.p8 (not_installed) mercurial............................... 2.2.2.p0 (not_installed) pil..................................... 1.1.6.p4 (not_installed) python.................................. 2.7.5.p1 (not_installed) sage.................................... 5.13 (not_installed) sage_root............................... 5.13 (not_installed) sage_scripts............................ 5.13 (not_installed) sqlalchemy.............................. 0.5.8 (not_installed)
comment:52 Changed 6 years ago by
Concerning Python, I do find it very confusing that ./sage i python
installs the old python2.7.5.p1.spkg
(the newstyle package is called python2
)
comment:53 in reply to: ↑ 51 ; followup: ↓ 54 Changed 6 years ago by
These are no longer standard:
This conversation again? Didn't you say in 47 that you would give it up?
These packages appear there: http://mirror.switch.ch/mirror/sagemath/spkg/standard/
If you do not agree with the list, please write to Harald. I already did it once, you were in Cc.
comment:54 in reply to: ↑ 53 ; followup: ↓ 55 Changed 6 years ago by
comment:55 in reply to: ↑ 54 Changed 6 years ago by
I'm not complaining, just informing...
I see...
Nathann
comment:56 Changed 5 years ago by
Pom Pom Pom ...
comment:57 Changed 5 years ago by
 Dependencies changed from #18407, #18456 to #19158
comment:58 Changed 5 years ago by
 Dependencies #19158 deleted
 Milestone changed from sage6.8 to sage6.9
comment:59 Changed 5 years ago by
 Commit changed from a9a45bf4314373cda000c34cf8296b457acb5fde to 15b95833e3928fb055ca8d34f2cea2ca646dc236
Branch pushed to git repo; I updated commit sha1. New commits:
15b9583  Merge branch 'public/18408' of git://trac.sagemath.org/sage into public/18408

comment:60 Changed 5 years ago by
 Milestone changed from sage6.9 to sage7.1
 Reviewers set to Javier Honrubia González
 Status changed from needs_review to positive_review
comment:61 Changed 5 years ago by
 Branch changed from public/18408 to 15b95833e3928fb055ca8d34f2cea2ca646dc236
 Resolution set to fixed
 Status changed from positive_review to closed
Branch pushed to git repo; I updated commit sha1. New commits:
trac #18407: Fix standard_packages(), optional_packages(), and experimental_packages()
Fix exception message of _package_lists_from_sage_output() (#18407)
trac #18407: better parsing of package name
trac #18408: List of packages in the documentation of sage.misc.package