Opened 6 years ago

Closed 5 years ago

# List of packages in the documentation of sage.misc.package

Reported by: Owned by: ncohen major sage-7.1 documentation vdelecroix, leif Nathann Cohen Javier Honrubia González N/A 15b9583 (Commits) 15b95833e3928fb055ca8d34f2cea2ca646dc236

### Description

It seems that we do not have an up-to-date list of Sage's packages in our documentation.

This branch solves the problem, by adding the list of packages in our reference manual (in sage.misc.package, where all functions dealing with packages are defined). This list is *generated* from python lists, so that they are:

• Easy to read (in a nice sorted table)
• Checked by a doctest, so that we get a warning if they are not up to date anymore

Nathann

### comment:1 Changed 6 years ago by ncohen

• Branch set to public/18408
• Status changed from new to needs_review

### comment:2 Changed 6 years ago by git

• Commit set to f43eb0119fb60ea1367247bcf383506782e4e81a

Branch pushed to git repo; I updated commit sha1. New commits:

 ​c400ee6 trac #18407: Fix standard_packages(), optional_packages(), and experimental_packages() ​10933bc Fix exception message of _package_lists_from_sage_output() (#18407) ​cc02304 trac #18407: better parsing of package name ​f43eb01 trac #18408: List of packages in the documentation of sage.misc.package

### comment:3 Changed 6 years ago by git

• Commit changed from f43eb0119fb60ea1367247bcf383506782e4e81a to 4a968a88a8eb55bad8dd0f88ccaa1a856eb9c092

Branch pushed to git repo; I updated commit sha1. New commits:

 ​4a968a8 trac #18408: A leftover from a past test

### comment:4 Changed 6 years ago by leif

As mentioned, those package lists refer to Sage versions < 6.0.

### comment:5 Changed 6 years ago by jdemeyer

• Status changed from needs_review to needs_info

What's up with this?

singular-3-1


(there are many more optional packages like this)

Last edited 6 years ago by jdemeyer (previous) (diff)

### comment:6 follow-up: ↓ 10 Changed 6 years ago by jdemeyer

• Status changed from needs_info to needs_work

Some packages from build/pkgs don't appear in your list (arb is the first, but there are more).

### comment:7 Changed 6 years ago by jdemeyer

And extcode, seriously? That's no longer a package...

### comment:8 Changed 6 years ago by jdemeyer

gcc appears in every list :-)

### comment:9 Changed 6 years ago by git

• Commit changed from 4a968a88a8eb55bad8dd0f88ccaa1a856eb9c092 to 869b195106b1fcd06332a38d12ac2a5bcd2d86ec

Branch pushed to git repo; I updated commit sha1. New commits:

 ​869b195 trac #18408: Change the regexp

### comment:10 in reply to: ↑ 6 ; follow-up: ↓ 12 Changed 6 years ago by ncohen

Some packages from build/pkgs don't appear in your list (arb is the first, but there are more). And extcode, seriously? That's no longer a package... gcc appears in every list :-)

Jeroen, please do not ask me to rewrite everything at once. I am no magician. As you can see from the source code, the list of packages is generated from the output of sage -standard, sage -optional and sage -experimental. And yes, all three contain gcc. And extcode. The point of this branch is that we see the list somewhere, and apparently it also helps us notice other bugs. Please do not force me to rewrite everything here.

Nathann

### comment:11 Changed 6 years ago by ncohen

• Status changed from needs_work to needs_review

### comment:12 in reply to: ↑ 10 Changed 6 years ago by jdemeyer

• Status changed from needs_review to needs_work

Jeroen, please do not ask me to rewrite everything at once. I am no magician. As you can see from the source code, the list of packages is generated from the output of sage -standard, sage -optional and sage -experimental. And yes, all three contain gcc. And extcode. The point of this branch is that we see the list somewhere, and apparently it also helps us notice other bugs. Please do not force me to rewrite everything here.

Well, the logical thing to do would be to first fix sage --standard, sage --optional and sage --experimental and then add the lists to the documentation.

I really do not see the point of adding wrong and outdated lists to the documentation.

And in any case, this is wrong: singular-3-1

### comment:13 Changed 6 years ago by ncohen

Why is singular wrong? It displays singular, and only singular?..

### comment:14 follow-ups: ↓ 16 ↓ 17 Changed 6 years ago by leif

FWIW, I'm currently working on categorizing the packages (obsolete, removed, renamed, replaced, newly added...)

If we fix sage-list-packages w.r.t. this, Nathann's work still makes sense (without modifying his code, just regenerating the "hardcoded" lists, I think).

### comment:15 Changed 6 years ago by jdemeyer

Sorry, I didn't realize that you changed the singular thing. Never mind...

### comment:16 in reply to: ↑ 14 ; follow-ups: ↓ 18 ↓ 19 Changed 6 years ago by jdemeyer

If we fix sage-list-packages w.r.t. this, Nathann's work still makes sense (without modifying his code, just regenerating the "hardcoded" lists, I think).

Of course Nathann's work makes sense, but obviously this ticket should depend on fixing the lists.

### comment:17 in reply to: ↑ 14 Changed 6 years ago by ncohen

If we fix sage-list-packages w.r.t. this, Nathann's work still makes sense (without modifying his code, just regenerating the "hardcoded" lists, I think).

Indeed. Plus my goal is also to add a list more exhaustive than the one our doc currently contains:

I am rewriting the doc of that manual currently, and moving a lot of things around. See #18405

Nathann

### comment:18 in reply to: ↑ 16 ; follow-up: ↓ 21 Changed 6 years ago by ncohen

Of course Nathann's work makes sense, but obviously this ticket should depend on fixing the lists.

Why should it *depend* on it? The list currenty generated by sage -standard is wrong at the moment (not caused by this patch) and what this patch does will still work when that will be fixed.

In the meantime we will have a "more correct" list of packages, I will be able to remove the list of Sage packages from the 'installation' document (and instead link to that list). There is no gain in blocking this ticket until the other problem is fixed.

Nathann

### comment:19 in reply to: ↑ 16 Changed 6 years ago by leif

obviously this ticket should depend on fixing the lists.

Well, it's just as borked as sage-list-packages and our package repositories, so I wouldn't even mind merging it as is... ;-)

Remove this:

pname = re.sub("(\.|-)p[0-9]+$","",pname) # strip .p0  (version numbers are always separated by a -, the .p0 is added in addition to a version number) ### comment:21 in reply to: ↑ 18 Changed 6 years ago by jdemeyer Replying to ncohen: In the meantime we will have a "more correct" list of packages The lists in this ticket are far from correct. There is no gain in blocking this ticket until the other problem is fixed. Yes, there is gain: Sage will have less wrong documentation. I think wrong documentation is worse than no documentation. ### comment:22 in reply to: ↑ 20 ; follow-up: ↓ 25 Changed 6 years ago by leif Replying to jdemeyer: Remove this: pname = re.sub("(\.|-)p[0-9]+$","",pname) # strip .p0


(version numbers are always separated by a -, the .p0 is added in addition to a version number)

Nope, we do have packages with a patchlevel but no version.

### comment:23 in reply to: ↑ 20 Changed 6 years ago by ncohen

(version numbers are always separated by a -, the .p0 is added in addition to a version number)

That's what I also believed at first

~\$ sage -experimental | grep -- "-p"
pynifti-p0 ............................. not installed

Last edited 6 years ago by ncohen (previous) (diff)

### comment:24 Changed 6 years ago by jdemeyer

That's -p0, not .p0

### comment:25 in reply to: ↑ 22 ; follow-up: ↓ 26 Changed 6 years ago by leif

(version numbers are always separated by a -, the .p0 is added in addition to a version number)

Nope, we do have packages with a patchlevel but no version.

Such as database_stein_watkins_mini.p0.spkg for example.

### comment:26 in reply to: ↑ 25 Changed 6 years ago by jdemeyer

Such as database_stein_watkins_mini.p0.spkg for example.

1. The name of that package actually is database_stein_watkins_mini.p0 (of course, that's a wrong name).
2. That package no longer exists anyway (the wrong name was fixed).

### comment:27 follow-up: ↓ 29 Changed 6 years ago by ncohen

Let's see it this way Jeroen: if you have reasons to believe that the list from sage -standard (which I believe is the same as what you see there [1]) is wrong then why don't you report it so that we update it (and my code will match it)?

My question at [2] was about whether we have some (any) reliable source containing our packages, for each category. It seems that we have none, so here is the deal: if you see something wrong in the list (i.e. the online one, or sage -standard) then let us fix that. Otherwise you are telling me that "the list is wrong" without telling me how exactly.

On the other hand, you cannot block this patch until the "problematic way in which the packages are handled in Sage" is fixed. All I want to do here is clear our doc, and make it more reliable *than it is*.

Nathann

Last edited 6 years ago by ncohen (previous) (diff)

### comment:28 Changed 6 years ago by leif

So, here it is.

As of Sage 6.7.beta4, we have:

Obsolete packages (since Sage 6.0), *.spkg:

• extcode
• sage (also used to be versioned)
• sage_root
• sage_scripts

Renamed packages:

• libm4ri -> m4ri
• libm4rie -> m4rie
• database_stein_watkins_mini.p0 -> database_stein_watkins_mini (optional)
• TOPCOM -> topcom (optional; "new-style" is lowercase)
• gd -> libgd ? (probably replaced, see below)

Removed (standard) packages:

• genus2reduction (now part of the Sage library IIRC)
• mercurial (in principle replaced by git, but no "drop-in" replacement)
• sqlalchemy

Replaced (standard) packages:

• pil -> Pillow
• gd, gdmodule -> libgd (I think)

• backports_ssl_match_hostname (WTF?)
• certifi
• combinatorial_designs
• configure (see below)
• csage (see below)
• dateutil
• git (used to be optional, so there's still an old one, too)
• jsonschema
• libgd (presumably the replacement of gd, gdmodule; see above)
• markupsafe
• mathjax
• mistune
• pari_galdata
• pip
• pkgconf
• pkgconfig
• prereq (see below)
• pyparsing
• pyzmq
• six
• zeromq

Then we have some weirdness with the following "packages":

• in build/pkgs/, but not (listed as) installed:
• configure
• (listed as) installed, but not in build/pkgs/, and without version:
• csage
• sage ("conflicts" with the previous, versioned spkg in Sage 5.x)
• prereq

And then we have new "new-style" packages (just in the git repo) which have no legacy (*.spkg) counterpart, hence currently not listed by sage-list-packages:

• arb (optional)
• ...

... besides that so far only some optional packages have been converted to "new-style"; the versions in the git repo might meanwhile be newer than their legacy counterparts. (Similar for packages that used to be optional but are standard now, cf. "newly added packages" above.)

### comment:29 in reply to: ↑ 27 ; follow-up: ↓ 30 Changed 6 years ago by jdemeyer

All I want to do here is clear our doc, and make it more reliable *than it is*.

Again, please tell me why adding wrong documentation is making it more reliable.

### comment:30 in reply to: ↑ 29 Changed 6 years ago by ncohen

Again, please tell me why adding wrong documentation is making it more reliable.

Well, because that new page will match the most reliable information we have (i.e. sage -standard) in the future, and because by having this new page I will be able to remove the other pages whose information is even older, and redirect them toward that page.

Nathann

### comment:31 follow-up: ↓ 33 Changed 6 years ago by jdemeyer

gdmodule has genuinly been removed. And gd was renamed to libgd.

Last edited 6 years ago by jdemeyer (previous) (diff)

### comment:32 follow-up: ↓ 45 Changed 6 years ago by ncohen

You seem really bothered by the information I am adding, and not bothered at all by the wrong information which stays available for as long as we do nothing.

### comment:33 in reply to: ↑ 31 ; follow-up: ↓ 34 Changed 6 years ago by ncohen

gdmodule has genuinly be removed. And gd was renamed to libgd.

I just generated the list from our build/install file:

I was about to write to sage-devel to ask <whoever can> to put it online, but in the meantime I noticed that the official list has been removed: http://www.sagemath.org/packages/standard/list

So something is apparently happening.

Nathann

### comment:34 in reply to: ↑ 33 Changed 6 years ago by leif

I was about to write to sage-devel to ask <whoever can> to put it online, but in the meantime I noticed that the official list has been removed: http://www.sagemath.org/packages/standard/list

The site is currently moving, while packages/ is obsolete anyway, since it is incompatible with the mirrors. Take spkg/standard/list from some mirror instead.

### comment:35 follow-up: ↓ 37 Changed 6 years ago by ncohen

Well I do not want to take that list, I want to replace it with the list I just generated from the latest beta :-P

### comment:36 Changed 6 years ago by ncohen

Even the reference manual was removed O_o

### comment:37 in reply to: ↑ 35 Changed 6 years ago by leif

Well I do not want to take that list, I want to replace it with the list I just generated from the latest beta :-P

That doesn't make much sense, since

• the list files are generated from the directory contents,
• the site is moving, as I said.

We'll have to wait with such at least until the infrastructure is stable again (hopefully in a few days).

### comment:38 Changed 6 years ago by ncohen

• Dependencies changed from #18407 to #18407, #18456

### comment:39 Changed 6 years ago by git

• Commit changed from 869b195106b1fcd06332a38d12ac2a5bcd2d86ec to 868fa71f00ed26bf97158e184a93fe8517dd211e

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

 ​3efc9fb trac #18431: package-specific 'dependencies' files ​308ceff trac #18431: Auto-generate make rules for standard packages ​77f293f trac #18431: auto-generated Make target "sage-standard-packages" ​e114874 trac #18431: typo ​766798e trac #18431: tab->spaces ​47e438d Merge tag '6.7' into t/18441/base_packages_except_configure_should_be_standard ​4de8e37 Various changes to build system ​f83b0c4 Correct variable name ​39cc007 trac #18456: Re-Fix standard_packages(), optional_packages(), and experimental_packages() ​868fa71 trac #18408: Merged with #18456

### comment:40 Changed 6 years ago by ncohen

• Status changed from needs_work to needs_review

### comment:41 Changed 6 years ago by git

• Commit changed from 868fa71f00ed26bf97158e184a93fe8517dd211e to ac93f992452bebd569183f8f032d5b71c086dbf4

Branch pushed to git repo; I updated commit sha1. New commits:

 ​fe40c54 Edge connectivity working ​9a0e868 Improved code for conversion ​b0ca92c Added sig_on, sig_off, and corrected edge_connectivity ​7d12ea9 trac #18408: Merged with 6.8.beta2 ​ac93f99 trac #18408: python2/python3 were missing

### comment:42 Changed 6 years ago by ncohen

• Status changed from needs_review to needs_work

(small problems with git history)

### comment:43 Changed 6 years ago by git

• Commit changed from ac93f992452bebd569183f8f032d5b71c086dbf4 to a8b9103987226c9a5768c4f571840da3fd7aef19

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

 ​a3402c9 trac #18456: Re-Fix standard_packages(), optional_packages(), and experimental_packages() ​687e426 trac #18456: Column names ​d671f1b trac #18456: don't use explicitly '/' in file paths ​b093007 trac #18456: doc and doctest ​a8b9103 trac #18408: List of packages in the documentation of sage.misc.package

### comment:44 Changed 6 years ago by ncohen

• Status changed from needs_work to needs_review

### comment:45 in reply to: ↑ 32 ; follow-up: ↓ 46 Changed 6 years ago by jdemeyer

• Milestone changed from sage-6.7 to sage-6.8
• Status changed from needs_review to needs_work
• Work issues set to Remove obsolete packages

You seem really bothered by the information I am adding, and not bothered at all by the wrong information which stays available for as long as we do nothing.

I am bothered by both. The fact that one thing is wrong is not an excuse to add a second wrong thing.

This patch should not be merged as long as stuff like extcode is listed.

### comment:46 in reply to: ↑ 45 Changed 6 years ago by ncohen

This patch should not be merged as long as stuff like extcode is listed.

What is wrong with extcode?

### comment:47 Changed 6 years ago by jdemeyer

• Status changed from needs_work to needs_review
• Work issues Remove obsolete packages deleted

It seems that updating the online packages is harder than I thought. And I also feel slightly guilty for "forcing" #18581 against your will. So I will stop complaining about this ticket.

### comment:48 Changed 6 years ago by ncohen

Thanks. And I hope that they will end up updating the lists at some point :-/

### comment:49 Changed 6 years ago by git

• Commit changed from a8b9103987226c9a5768c4f571840da3fd7aef19 to a9a45bf4314373cda000c34cf8296b457acb5fde

Branch pushed to git repo; I updated commit sha1. New commits:

 ​10a0222 trac #18408: Merged with 6.8.beta7 ​a9a45bf trac #18408: Updated lists

### comment:50 Changed 6 years ago by ncohen

The lists have been updated. And so, this patch too!

Nathann

### comment:51 follow-up: ↓ 53 Changed 6 years ago by jdemeyer

These are no longer standard:

gd...................................... 2.0.35.p7 (not_installed)
gdmodule................................ 0.56.p8 (not_installed)
genus2reduction......................... 0.3.p8 (not_installed)
mercurial............................... 2.2.2.p0 (not_installed)
pil..................................... 1.1.6.p4 (not_installed)
python.................................. 2.7.5.p1 (not_installed)
sage.................................... 5.13 (not_installed)
sage_root............................... 5.13 (not_installed)
sage_scripts............................ 5.13 (not_installed)
sqlalchemy.............................. 0.5.8 (not_installed)


### comment:52 Changed 6 years ago by jdemeyer

Concerning Python, I do find it very confusing that ./sage -i python installs the old python-2.7.5.p1.spkg (the new-style package is called python2)

### comment:53 in reply to: ↑ 51 ; follow-up: ↓ 54 Changed 6 years ago by ncohen

These are no longer standard:

This conversation again? Didn't you say in 47 that you would give it up?

These packages appear there: http://mirror.switch.ch/mirror/sagemath/spkg/standard/

If you do not agree with the list, please write to Harald. I already did it once, you were in Cc.

### comment:54 in reply to: ↑ 53 ; follow-up: ↓ 55 Changed 6 years ago by jdemeyer

This conversation again? Didn't you say in 47 that you would give it up?

I'm not complaining, just informing...

### comment:55 in reply to: ↑ 54 Changed 6 years ago by ncohen

I'm not complaining, just informing...

I see...

Nathann

Pom Pom Pom ...

### comment:57 Changed 5 years ago by jdemeyer

• Dependencies changed from #18407, #18456 to #19158

### comment:58 Changed 5 years ago by jdemeyer

• Dependencies #19158 deleted
• Milestone changed from sage-6.8 to sage-6.9

### comment:59 Changed 5 years ago by git

• Commit changed from a9a45bf4314373cda000c34cf8296b457acb5fde to 15b95833e3928fb055ca8d34f2cea2ca646dc236

Branch pushed to git repo; I updated commit sha1. New commits:

 ​15b9583 Merge branch 'public/18408' of git://trac.sagemath.org/sage into public/18408

### comment:60 Changed 5 years ago by jhonrubia6

• Milestone changed from sage-6.9 to sage-7.1
• Reviewers set to Javier Honrubia González
• Status changed from needs_review to positive_review

### comment:61 Changed 5 years ago by vbraun

• Branch changed from public/18408 to 15b95833e3928fb055ca8d34f2cea2ca646dc236
• Resolution set to fixed
• Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.