Opened 6 years ago

Closed 5 years ago

#18408 closed enhancement (fixed)

List of packages in the documentation of sage.misc.package

Reported by: ncohen Owned by:
Priority: major Milestone: sage-7.1
Component: documentation Keywords:
Cc: vdelecroix, leif Merged in:
Authors: Nathann Cohen Reviewers: Javier Honrubia González
Report Upstream: N/A Work issues:
Branch: 15b9583 (Commits) Commit: 15b95833e3928fb055ca8d34f2cea2ca646dc236
Dependencies: Stopgaps:

Description

It seems that we do not have an up-to-date list of Sage's packages in our documentation.

This branch solves the problem, by adding the list of packages in our reference manual (in sage.misc.package, where all functions dealing with packages are defined). This list is *generated* from python lists, so that they are:

  • Easy to read (in a nice sorted table)
  • Checked by a doctest, so that we get a warning if they are not up to date anymore

Nathann

Change History (61)

comment:1 Changed 6 years ago by ncohen

  • Branch set to public/18408
  • Status changed from new to needs_review

comment:2 Changed 6 years ago by git

  • Commit set to f43eb0119fb60ea1367247bcf383506782e4e81a

Branch pushed to git repo; I updated commit sha1. New commits:

c400ee6trac #18407: Fix standard_packages(), optional_packages(), and experimental_packages()
10933bcFix exception message of _package_lists_from_sage_output() (#18407)
cc02304trac #18407: better parsing of package name
f43eb01trac #18408: List of packages in the documentation of sage.misc.package

comment:3 Changed 6 years ago by git

  • Commit changed from f43eb0119fb60ea1367247bcf383506782e4e81a to 4a968a88a8eb55bad8dd0f88ccaa1a856eb9c092

Branch pushed to git repo; I updated commit sha1. New commits:

4a968a8trac #18408: A leftover from a past test

comment:4 Changed 6 years ago by leif

As mentioned, those package lists refer to Sage versions < 6.0.

comment:5 Changed 6 years ago by jdemeyer

  • Status changed from needs_review to needs_info

What's up with this?

singular-3-1

(there are many more optional packages like this)

Last edited 6 years ago by jdemeyer (previous) (diff)

comment:6 follow-up: Changed 6 years ago by jdemeyer

  • Status changed from needs_info to needs_work

Some packages from build/pkgs don't appear in your list (arb is the first, but there are more).

comment:7 Changed 6 years ago by jdemeyer

And extcode, seriously? That's no longer a package...

comment:8 Changed 6 years ago by jdemeyer

gcc appears in every list :-)

comment:9 Changed 6 years ago by git

  • Commit changed from 4a968a88a8eb55bad8dd0f88ccaa1a856eb9c092 to 869b195106b1fcd06332a38d12ac2a5bcd2d86ec

Branch pushed to git repo; I updated commit sha1. New commits:

869b195trac #18408: Change the regexp

comment:10 in reply to: ↑ 6 ; follow-up: Changed 6 years ago by ncohen

Some packages from build/pkgs don't appear in your list (arb is the first, but there are more). And extcode, seriously? That's no longer a package... gcc appears in every list :-)

Jeroen, please do not ask me to rewrite everything at once. I am no magician. As you can see from the source code, the list of packages is generated from the output of sage -standard, sage -optional and sage -experimental. And yes, all three contain gcc. And extcode. The point of this branch is that we see the list somewhere, and apparently it also helps us notice other bugs. Please do not force me to rewrite everything here.

Nathann

comment:11 Changed 6 years ago by ncohen

  • Status changed from needs_work to needs_review

comment:12 in reply to: ↑ 10 Changed 6 years ago by jdemeyer

  • Status changed from needs_review to needs_work

Replying to ncohen:

Jeroen, please do not ask me to rewrite everything at once. I am no magician. As you can see from the source code, the list of packages is generated from the output of sage -standard, sage -optional and sage -experimental. And yes, all three contain gcc. And extcode. The point of this branch is that we see the list somewhere, and apparently it also helps us notice other bugs. Please do not force me to rewrite everything here.

Well, the logical thing to do would be to first fix sage --standard, sage --optional and sage --experimental and then add the lists to the documentation.

I really do not see the point of adding wrong and outdated lists to the documentation.

And in any case, this is wrong: singular-3-1

comment:13 Changed 6 years ago by ncohen

Why is singular wrong? It displays singular, and only singular?..

comment:14 follow-ups: Changed 6 years ago by leif

FWIW, I'm currently working on categorizing the packages (obsolete, removed, renamed, replaced, newly added...)

If we fix sage-list-packages w.r.t. this, Nathann's work still makes sense (without modifying his code, just regenerating the "hardcoded" lists, I think).

comment:15 Changed 6 years ago by jdemeyer

Sorry, I didn't realize that you changed the singular thing. Never mind...

comment:16 in reply to: ↑ 14 ; follow-ups: Changed 6 years ago by jdemeyer

Replying to leif:

If we fix sage-list-packages w.r.t. this, Nathann's work still makes sense (without modifying his code, just regenerating the "hardcoded" lists, I think).

Of course Nathann's work makes sense, but obviously this ticket should depend on fixing the lists.

comment:17 in reply to: ↑ 14 Changed 6 years ago by ncohen

If we fix sage-list-packages w.r.t. this, Nathann's work still makes sense (without modifying his code, just regenerating the "hardcoded" lists, I think).

Indeed. Plus my goal is also to add a list more exhaustive than the one our doc currently contains:

http://www.sagemath.org/doc/installation/introduction.html

I am rewriting the doc of that manual currently, and moving a lot of things around. See #18405

Nathann

comment:18 in reply to: ↑ 16 ; follow-up: Changed 6 years ago by ncohen

Of course Nathann's work makes sense, but obviously this ticket should depend on fixing the lists.

Why should it *depend* on it? The list currenty generated by sage -standard is wrong at the moment (not caused by this patch) and what this patch does will still work when that will be fixed.

In the meantime we will have a "more correct" list of packages, I will be able to remove the list of Sage packages from the 'installation' document (and instead link to that list). There is no gain in blocking this ticket until the other problem is fixed.

Nathann

comment:19 in reply to: ↑ 16 Changed 6 years ago by leif

Replying to jdemeyer:

obviously this ticket should depend on fixing the lists.

Well, it's just as borked as sage-list-packages and our package repositories, so I wouldn't even mind merging it as is... ;-)

comment:20 follow-ups: Changed 6 years ago by jdemeyer

Remove this:

pname = re.sub("(\.|-)p[0-9]+$","",pname) # strip .p0

(version numbers are always separated by a -, the .p0 is added in addition to a version number)

comment:21 in reply to: ↑ 18 Changed 6 years ago by jdemeyer

Replying to ncohen:

In the meantime we will have a "more correct" list of packages

The lists in this ticket are far from correct.

There is no gain in blocking this ticket until the other problem is fixed.

Yes, there is gain: Sage will have less wrong documentation.

I think wrong documentation is worse than no documentation.

comment:22 in reply to: ↑ 20 ; follow-up: Changed 6 years ago by leif

Replying to jdemeyer:

Remove this:

pname = re.sub("(\.|-)p[0-9]+$","",pname) # strip .p0

(version numbers are always separated by a -, the .p0 is added in addition to a version number)

Nope, we do have packages with a patchlevel but no version.

comment:23 in reply to: ↑ 20 Changed 6 years ago by ncohen

(version numbers are always separated by a -, the .p0 is added in addition to a version number)

That's what I also believed at first

~$ sage -experimental | grep -- "-p"
pynifti-p0 ............................. not installed
Last edited 6 years ago by ncohen (previous) (diff)

comment:24 Changed 6 years ago by jdemeyer

That's -p0, not .p0

comment:25 in reply to: ↑ 22 ; follow-up: Changed 6 years ago by leif

Replying to leif:

Replying to jdemeyer:

(version numbers are always separated by a -, the .p0 is added in addition to a version number)

Nope, we do have packages with a patchlevel but no version.

Such as database_stein_watkins_mini.p0.spkg for example.

comment:26 in reply to: ↑ 25 Changed 6 years ago by jdemeyer

Replying to leif:

Such as database_stein_watkins_mini.p0.spkg for example.

  1. The name of that package actually is database_stein_watkins_mini.p0 (of course, that's a wrong name).
  2. That package no longer exists anyway (the wrong name was fixed).

comment:27 follow-up: Changed 6 years ago by ncohen

Let's see it this way Jeroen: if you have reasons to believe that the list from sage -standard (which I believe is the same as what you see there [1]) is wrong then why don't you report it so that we update it (and my code will match it)?

My question at [2] was about whether we have some (any) reliable source containing our packages, for each category. It seems that we have none, so here is the deal: if you see something wrong in the list (i.e. the online one, or sage -standard) then let us fix that. Otherwise you are telling me that "the list is wrong" without telling me how exactly.

On the other hand, you cannot block this patch until the "problematic way in which the packages are handled in Sage" is fixed. All I want to do here is clear our doc, and make it more reliable *than it is*.

Nathann

[1] http://www.sagemath.org/packages/standard/ [2] https://groups.google.com/forum/#!topic/sage-devel/aEmUmFOwJYQ

Last edited 6 years ago by ncohen (previous) (diff)

comment:28 Changed 6 years ago by leif

So, here it is.


As of Sage 6.7.beta4, we have:

Obsolete packages (since Sage 6.0), *.spkg:

  • extcode
  • sage (also used to be versioned)
  • sage_root
  • sage_scripts

Renamed packages:

  • libm4ri -> m4ri
  • libm4rie -> m4rie
  • database_stein_watkins_mini.p0 -> database_stein_watkins_mini (optional)
  • TOPCOM -> topcom (optional; "new-style" is lowercase)
  • gd -> libgd ? (probably replaced, see below)

Removed (standard) packages:

  • genus2reduction (now part of the Sage library IIRC)
  • mercurial (in principle replaced by git, but no "drop-in" replacement)
  • sqlalchemy

Replaced (standard) packages:

  • pil -> Pillow
  • gd, gdmodule -> libgd (I think)

Newly added (standard) packages:

  • backports_ssl_match_hostname (WTF?)
  • certifi
  • combinatorial_designs
  • configure (see below)
  • csage (see below)
  • dateutil
  • git (used to be optional, so there's still an old one, too)
  • jsonschema
  • libgd (presumably the replacement of gd, gdmodule; see above)
  • markupsafe
  • mathjax
  • mistune
  • pari_galdata
  • pari_seadata_small
  • pip
  • pkgconf
  • pkgconfig
  • prereq (see below)
  • pyparsing
  • pyzmq
  • six
  • tornado
  • zeromq

Then we have some weirdness with the following "packages":

  • in build/pkgs/, but not (listed as) installed:
    • configure
  • (listed as) installed, but not in build/pkgs/, and without version:
    • csage
    • sage ("conflicts" with the previous, versioned spkg in Sage 5.x)
    • prereq

And then we have new "new-style" packages (just in the git repo) which have no legacy (*.spkg) counterpart, hence currently not listed by sage-list-packages:

  • arb (optional)
  • ...

... besides that so far only some optional packages have been converted to "new-style"; the versions in the git repo might meanwhile be newer than their legacy counterparts. (Similar for packages that used to be optional but are standard now, cf. "newly added packages" above.)

comment:29 in reply to: ↑ 27 ; follow-up: Changed 6 years ago by jdemeyer

Replying to ncohen:

All I want to do here is clear our doc, and make it more reliable *than it is*.

Again, please tell me why adding wrong documentation is making it more reliable.

comment:30 in reply to: ↑ 29 Changed 6 years ago by ncohen

Again, please tell me why adding wrong documentation is making it more reliable.

Well, because that new page will match the most reliable information we have (i.e. sage -standard) in the future, and because by having this new page I will be able to remove the other pages whose information is even older, and redirect them toward that page.

Nathann

comment:31 follow-up: Changed 6 years ago by jdemeyer

gdmodule has genuinly been removed. And gd was renamed to libgd.

Last edited 6 years ago by jdemeyer (previous) (diff)

comment:32 follow-up: Changed 6 years ago by ncohen

You seem really bothered by the information I am adding, and not bothered at all by the wrong information which stays available for as long as we do nothing.

comment:33 in reply to: ↑ 31 ; follow-up: Changed 6 years ago by ncohen

gdmodule has genuinly be removed. And gd was renamed to libgd.

I just generated the list from our build/install file:

http://www.steinertriples.fr/ncohen/tmp/std

I was about to write to sage-devel to ask <whoever can> to put it online, but in the meantime I noticed that the official list has been removed: http://www.sagemath.org/packages/standard/list

So something is apparently happening.

Nathann

comment:34 in reply to: ↑ 33 Changed 6 years ago by leif

Replying to ncohen:

I was about to write to sage-devel to ask <whoever can> to put it online, but in the meantime I noticed that the official list has been removed: http://www.sagemath.org/packages/standard/list

The site is currently moving, while packages/ is obsolete anyway, since it is incompatible with the mirrors. Take spkg/standard/list from some mirror instead.

comment:35 follow-up: Changed 6 years ago by ncohen

Well I do not want to take that list, I want to replace it with the list I just generated from the latest beta :-P

comment:36 Changed 6 years ago by ncohen

Even the reference manual was removed O_o

comment:37 in reply to: ↑ 35 Changed 6 years ago by leif

Replying to ncohen:

Well I do not want to take that list, I want to replace it with the list I just generated from the latest beta :-P

That doesn't make much sense, since

  • the list files are generated from the directory contents,
  • the site is moving, as I said.

We'll have to wait with such at least until the infrastructure is stable again (hopefully in a few days).

comment:38 Changed 6 years ago by ncohen

  • Dependencies changed from #18407 to #18407, #18456

comment:39 Changed 6 years ago by git

  • Commit changed from 869b195106b1fcd06332a38d12ac2a5bcd2d86ec to 868fa71f00ed26bf97158e184a93fe8517dd211e

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

3efc9fbtrac #18431: package-specific 'dependencies' files
308cefftrac #18431: Auto-generate make rules for standard packages
77f293ftrac #18431: auto-generated Make target "sage-standard-packages"
e114874trac #18431: typo
766798etrac #18431: tab->spaces
47e438dMerge tag '6.7' into t/18441/base_packages_except_configure_should_be_standard
4de8e37Various changes to build system
f83b0c4Correct variable name
39cc007trac #18456: Re-Fix standard_packages(), optional_packages(), and experimental_packages()
868fa71trac #18408: Merged with #18456

comment:40 Changed 6 years ago by ncohen

  • Status changed from needs_work to needs_review

comment:41 Changed 6 years ago by git

  • Commit changed from 868fa71f00ed26bf97158e184a93fe8517dd211e to ac93f992452bebd569183f8f032d5b71c086dbf4

Branch pushed to git repo; I updated commit sha1. New commits:

fe40c54Edge connectivity working
9a0e868Improved code for conversion
b0ca92cAdded sig_on, sig_off, and corrected edge_connectivity
7d12ea9trac #18408: Merged with 6.8.beta2
ac93f99trac #18408: python2/python3 were missing

comment:42 Changed 6 years ago by ncohen

  • Status changed from needs_review to needs_work

(small problems with git history)

comment:43 Changed 6 years ago by git

  • Commit changed from ac93f992452bebd569183f8f032d5b71c086dbf4 to a8b9103987226c9a5768c4f571840da3fd7aef19

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

a3402c9trac #18456: Re-Fix standard_packages(), optional_packages(), and experimental_packages()
687e426trac #18456: Column names
d671f1btrac #18456: don't use explicitly '/' in file paths
b093007trac #18456: doc and doctest
a8b9103trac #18408: List of packages in the documentation of sage.misc.package

comment:44 Changed 6 years ago by ncohen

  • Status changed from needs_work to needs_review

comment:45 in reply to: ↑ 32 ; follow-up: Changed 6 years ago by jdemeyer

  • Milestone changed from sage-6.7 to sage-6.8
  • Status changed from needs_review to needs_work
  • Work issues set to Remove obsolete packages

Replying to ncohen:

You seem really bothered by the information I am adding, and not bothered at all by the wrong information which stays available for as long as we do nothing.

I am bothered by both. The fact that one thing is wrong is not an excuse to add a second wrong thing.

This patch should not be merged as long as stuff like extcode is listed.

comment:46 in reply to: ↑ 45 Changed 6 years ago by ncohen

This patch should not be merged as long as stuff like extcode is listed.

What is wrong with extcode?

comment:47 Changed 6 years ago by jdemeyer

  • Status changed from needs_work to needs_review
  • Work issues Remove obsolete packages deleted

It seems that updating the online packages is harder than I thought. And I also feel slightly guilty for "forcing" #18581 against your will. So I will stop complaining about this ticket.

comment:48 Changed 6 years ago by ncohen

Thanks. And I hope that they will end up updating the lists at some point :-/

comment:49 Changed 6 years ago by git

  • Commit changed from a8b9103987226c9a5768c4f571840da3fd7aef19 to a9a45bf4314373cda000c34cf8296b457acb5fde

Branch pushed to git repo; I updated commit sha1. New commits:

10a0222trac #18408: Merged with 6.8.beta7
a9a45bftrac #18408: Updated lists

comment:50 Changed 6 years ago by ncohen

The lists have been updated. And so, this patch too!

Nathann

comment:51 follow-up: Changed 6 years ago by jdemeyer

These are no longer standard:

gd...................................... 2.0.35.p7 (not_installed)
gdmodule................................ 0.56.p8 (not_installed)
genus2reduction......................... 0.3.p8 (not_installed)
mercurial............................... 2.2.2.p0 (not_installed)
pil..................................... 1.1.6.p4 (not_installed)
python.................................. 2.7.5.p1 (not_installed)
sage.................................... 5.13 (not_installed)
sage_root............................... 5.13 (not_installed)
sage_scripts............................ 5.13 (not_installed)
sqlalchemy.............................. 0.5.8 (not_installed)

comment:52 Changed 6 years ago by jdemeyer

Concerning Python, I do find it very confusing that ./sage -i python installs the old python-2.7.5.p1.spkg (the new-style package is called python2)

comment:53 in reply to: ↑ 51 ; follow-up: Changed 6 years ago by ncohen

These are no longer standard:

This conversation again? Didn't you say in 47 that you would give it up?

These packages appear there: http://mirror.switch.ch/mirror/sagemath/spkg/standard/

If you do not agree with the list, please write to Harald. I already did it once, you were in Cc.

comment:54 in reply to: ↑ 53 ; follow-up: Changed 6 years ago by jdemeyer

Replying to ncohen:

This conversation again? Didn't you say in 47 that you would give it up?

I'm not complaining, just informing...

comment:55 in reply to: ↑ 54 Changed 6 years ago by ncohen

I'm not complaining, just informing...

I see...

Nathann

comment:56 Changed 5 years ago by ncohen

Pom Pom Pom ...

comment:57 Changed 5 years ago by jdemeyer

  • Dependencies changed from #18407, #18456 to #19158

comment:58 Changed 5 years ago by jdemeyer

  • Dependencies #19158 deleted
  • Milestone changed from sage-6.8 to sage-6.9

comment:59 Changed 5 years ago by git

  • Commit changed from a9a45bf4314373cda000c34cf8296b457acb5fde to 15b95833e3928fb055ca8d34f2cea2ca646dc236

Branch pushed to git repo; I updated commit sha1. New commits:

15b9583Merge branch 'public/18408' of git://trac.sagemath.org/sage into public/18408

comment:60 Changed 5 years ago by jhonrubia6

  • Milestone changed from sage-6.9 to sage-7.1
  • Reviewers set to Javier Honrubia González
  • Status changed from needs_review to positive_review

comment:61 Changed 5 years ago by vbraun

  • Branch changed from public/18408 to 15b95833e3928fb055ca8d34f2cea2ca646dc236
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.