Opened 6 years ago

Closed 5 years ago

Last modified 4 years ago

#19213 closed enhancement (fixed)

Cleaning sage-package-list

Reported by: jhpalmieri Owned by:
Priority: minor Milestone: sage-7.4
Component: scripts Keywords:
Cc: vdelecroix, jdemeyer, vbraun, embray Merged in:
Authors: Vincent Delecroix Reviewers: Matthias Koeppe, Jeroen Demeyer
Report Upstream: N/A Work issues:
Branch: f054998 (Commits, GitHub, GitLab) Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by vdelecroix)

This ticket enhances the module sage.misc.package and the script sage-list-packages:

  • add independent functions in sage.misc.package to list the (new style) packages
  • add pip functionalities to sage.misc.package
  • do not list anymore old style packages (see also #21132)
  • relies on sage.misc.packages in sage-list-packages

pip type packages were created in #19187. This ticket is such that these packages will be listed with sage -optional. Sadly, it will still not be possible for these packages to be used in optional doctests. The reason for that is that we do not have proper version control for them.

Change History (64)

comment:1 Changed 6 years ago by jhpalmieri

  • Status changed from new to needs_review
  • Type changed from PLEASE CHANGE to enhancement

comment:2 Changed 6 years ago by jhpalmieri

  • Branch set to u/jhpalmieri/pip_packages

comment:3 follow-up: Changed 6 years ago by vdelecroix

  • Commit set to 00224aafc17efcb3a4d5504bdbcf036fae5bca57

To my mind, it would nice to also list non installed pip packages as well as the last version available. The pip version can be obtained from Python with the json API

import urllib2
import json

def pip_version(package_name):
    url = "https://pypi.python.org/pypi/{}/json".format(package_name)

    try:
        data = json.load(urllib2.urlopen(urllib2.Request(url)))
    except urllib2.HTTPError:
        return None

    try:
        return data["info"]["version"]
    except KeyError:
        return None

The last version available might differ from what is installed. And it is worth mentioning that an update is available.


New commits:

93ec09bChange sage -p to always install a package
c6ddd0eBetter help
f3d7ef3Remove -f option to sage-spkg when running sage -i
752e269Use PKG-clean target to implement sage -f PKG
1812e4fMerge tag '6.9.beta6' into t/19119/ticket/19119
adef308Add rules for installing packages with pip
fdba3eeAdd mercurial and sqlalchemy (ex-standard packages)
00224aatrac 19213: include packages from build/pkgs/piprules in 'sage --optional'

comment:4 in reply to: ↑ 3 Changed 6 years ago by jhpalmieri

Replying to vdelecroix:

To my mind, it would nice to also list non installed pip packages as well as the last version available. The pip version can be obtained from Python with the json API

One disadvantage to this is that it requires internet access, which sage --optional --local does not do. But this would be a good addition if --local were not given. If we do list the non-installed packages, then it's going to get unwieldy if there are hundreds of entries in build/pkgs/piprules. What if that file gets big?

Do we want to list these as optional packages or in another category? sage --pip_packages?

comment:5 follow-up: Changed 6 years ago by jdemeyer

  • Component changed from packages: optional to scripts
  • Status changed from needs_review to needs_work

I do not want these pip packages to be treated like optional packages. If anything, they are closer to "experimental" than to "optional". We don't have control over them, so they are more likely to break when the upstream package gets upgraded.

comment:6 in reply to: ↑ 5 ; follow-up: Changed 6 years ago by jhpalmieri

Replying to jdemeyer:

I do not want these pip packages to be treated like optional packages. If anything, they are closer to "experimental" than to "optional". We don't have control over them, so they are more likely to break when the upstream package gets upgraded.

Do you mean

  • you want them listed in sage --experimental, not sage --optional, or
  • you don't want them detected for the purpose of doctests,

or both?

comment:7 in reply to: ↑ 6 ; follow-up: Changed 6 years ago by jdemeyer

Replying to jhpalmieri:

Do you mean

  • you want them listed in sage --experimental, not sage --optional, or
  • you don't want them detected for the purpose of doctests,

or both?

The default for --optional flags in doctests really should be optional packages only, so I don't want pip packages to be tested by doctests by default. Therefore, I also don't want those packages to appear in sage --optional. So I really prefer to see them as a new category: not standard, not optional, not experimental.

comment:8 Changed 6 years ago by jhpalmieri

Wait, if an (old-style) optional package is listed in piprules, you still don't want it listed in sage --optional? Should it be removed altogether from the listing, or listed as not_installed? If it's old-style, by the way, having it listed in sage --optional is somewhat independent of having it used in doctests, since the doctest framework only looks at local packages. So in my current branch, if we delete the changes to doctest/control.py, then these pip packages won't be used automatically in doctests.

comment:9 in reply to: ↑ 7 Changed 6 years ago by vdelecroix

Replying to jdemeyer:

Replying to jhpalmieri:

Do you mean

  • you want them listed in sage --experimental, not sage --optional, or
  • you don't want them detected for the purpose of doctests,

or both?

The default for --optional flags in doctests really should be optional packages only, so I don't want pip packages to be tested by doctests by default. Therefore, I also don't want those packages to appear in sage --optional. So I really prefer to see them as a new category: not standard, not optional, not experimental.

I see several solutions to deal with "pip packages":

  1. having a new category of packages (let say python-optional)
  2. be stricter about versioning, i.e. in piprules specify the version of the packages we support
    brian=1.3.1
    trac=2.3
    
    Very simple to maintain!

I agree that it would be hard to prevent a pip package to just break because versions are not backward compatible. So solution 2 looks appropriate to me. Otherwise, we can have a more precise tagging system for python-optional packages

sage: my_test()  # python-optional -- 3.5 <= beautifulsoup <= 4.1.2

It is more flexible than the piprules strict versioning solution but the cost of maintenance is much higher.

Vincent

comment:10 follow-up: Changed 6 years ago by vdelecroix

I have a working version of sage-list-packages doing

$ ./sage-list-packages python
[package] .............................. [latest version] ([installed version])

beautifulsoup........................... 3.2.1 (3.2.1)
biopython............................... 1.65 (1.65)
brian................................... 1.4.1 (1.4.1)
guppy................................... 0.1.10 (0.1.10)
mercurial............................... 3.5.1 (3.5.1)
mpi4py.................................. 1.3.1 (1.3.1)
nibabel................................. 2.0.1 (2.0.1)
pybtex.................................. 0.18 (0.18)
pyflakes................................ 0.9.2 (0.9.2)
pyopenssl............................... 0.15.1 (0.15.1)
sqlalchemy.............................. 1.0.8 (1.0.8)
trac.................................... 1.0.9 (1.0.9)

The only problem is that it is relatively long to get the information from the pip json API for all the packages. Perhaps, one can be smarter with urllib2.

John, am I free to change the branch in order to make my proposition publicly available?

Vincent

comment:11 Changed 6 years ago by vdelecroix

Furthermore, could we separate the issues of:

  • listing python packages
  • including optional doctests for them

Vincent

comment:12 in reply to: ↑ 10 Changed 6 years ago by jhpalmieri

Replying to vdelecroix:

John, am I free to change the branch in order to make my proposition publicly available?

Please do so.

comment:13 Changed 6 years ago by jhpalmieri

I have a few questions and comments:

  • if #19220 is merged (archiving superseded packages, including the piprules packages), how are users supposed to find out about biopython, brian, etc.? We need to include an option to sage which will call sage-list-packages python. Maybe we should also have an option which lists all packages: first standard, then optional, then experimental, then python/pip. Or maybe group all of the non-standard packages together, sorted by name. In any case, we shouldn't expect users to even be aware that there are several different types of non-standard packages; they should instead just be able to get one list to see if some particular package is supported.

Maybe a list like

[package (type)] .......... [latest version] ([installed version])

...
biopython (pip) ........... latest version (installed version)
...
chomp (experimental) ...... latest version (installed version)
...
ore_algebra (optional) .... latest version (installed version)
...

I'm not really sure, though.

  • Regarding the versions, I am not convinced that we can accurately specify which version of each package we support. Who is going to maintain this? Unless there is some automatic way of determining the versions supported by Sage, the versions will quickly get out of date. If brian gets updated to 1.4.2 but we list brian=1.4.1, what happens if a user does sage -i brian? What happens if a user does sage --pip install brian?

comment:14 Changed 6 years ago by vdelecroix

  • Branch changed from u/jhpalmieri/pip_packages to public/pip_packages
  • Commit 00224aafc17efcb3a4d5504bdbcf036fae5bca57 deleted

comment:15 Changed 6 years ago by vdelecroix

  • Branch changed from public/pip_packages to public/19213
  • Commit set to 19e3b86ac89a4441bf101130674b4daec9e14910

New commits:

93ec09bChange sage -p to always install a package
c6ddd0eBetter help
f3d7ef3Remove -f option to sage-spkg when running sage -i
752e269Use PKG-clean target to implement sage -f PKG
1812e4fMerge tag '6.9.beta6' into t/19119/ticket/19119
adef308Add rules for installing packages with pip
fdba3eeAdd mercurial and sqlalchemy (ex-standard packages)
00224aatrac 19213: include packages from build/pkgs/piprules in 'sage --optional'
19e3b86trac 19213: separation of python packages

comment:16 Changed 5 years ago by mkoeppe

  • Cc vdelecroix jdemeyer vbraun embray added
  • Milestone changed from sage-6.9 to sage-7.3

What is the status on this?

comment:17 Changed 5 years ago by dmuthiah

  • Dependencies #19187 deleted
  • Description modified (diff)

comment:18 Changed 5 years ago by dmuthiah

  • Description modified (diff)
  • Summary changed from For packages listed in build/pkgs/piprules, allow 'sage --optional' to list them to Allow 'sage --optional' to list pip packages

comment:19 Changed 5 years ago by vdelecroix

  • Authors changed from John Palmieri to Vincent Delecroix
  • Branch changed from public/19213 to u/vdelecroix/19213
  • Commit changed from 19e3b86ac89a4441bf101130674b4daec9e14910 to 9af0741b99d457dea3dcf19b8a7acf179f8a139c
  • Description modified (diff)
  • Status changed from needs_work to needs_review
  • Summary changed from Allow 'sage --optional' to list pip packages to Cleaning the sage-list-packages script

New commits:

cf6398019213: sage_setup.packages
ab4012319213: add standalone files to list old style packages
9af074119213: clean sage-list-packages script

comment:20 Changed 5 years ago by git

  • Commit changed from 9af0741b99d457dea3dcf19b8a7acf179f8a139c to 79d2e0011a5d851a03bb4b93f14f51d9e4887571

Branch pushed to git repo; I updated commit sha1. New commits:

79d2e0019213: clean imports

comment:21 Changed 5 years ago by git

  • Commit changed from 79d2e0011a5d851a03bb4b93f14f51d9e4887571 to 9b058ef05334f517637683f907c3a32d03dbae95

Branch pushed to git repo; I updated commit sha1. New commits:

9b058ef19213: fix doctest

comment:22 follow-up: Changed 5 years ago by dimpase

ACHTUNG! According to an email by user dmuthiah, changes by this user on this ticket were done by an unautorised person...

comment:23 in reply to: ↑ 22 Changed 5 years ago by vdelecroix

Replying to dimpase:

ACHTUNG! According to an email by user dmuthiah, changes by this user on this ticket were done by an unautorised person...

that was me... (I sent an e-mail). Sorry.

comment:24 Changed 5 years ago by vdelecroix

  • Status changed from needs_review to needs_work

need to superseed sage/misc/packages which contains *builtin* list of packages!!

comment:25 Changed 5 years ago by vdelecroix

  • Description modified (diff)

comment:26 Changed 5 years ago by vdelecroix

  • Description modified (diff)
  • Summary changed from Cleaning the sage-list-packages script to Cleaning sage-package-list

comment:27 Changed 5 years ago by git

  • Commit changed from 9b058ef05334f517637683f907c3a32d03dbae95 to faff48bd5628e808790ac83c8c827dd046224afd

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

7d72e0419213: clean sage.misc.package
64cd48219213: clean sage-list-packages script
faff48b19213: use list_packages in sage.doctest.control

comment:28 Changed 5 years ago by vdelecroix

  • Status changed from needs_work to needs_review

comment:29 Changed 5 years ago by mkoeppe

This is probably a FAQ, but I'm getting this error:

$ sage -optional
[package]...............................[latest version] ([version])

Traceback (most recent call last):
  File "/Users/mkoeppe/cvs/sage/src/bin/sage-list-packages", line 67, in <module>
    L = list_packages('optional', 'pip', local=args['local']).values()
  File "/Users/mkoeppe/cvs/sage/local/lib/python2.7/site-packages/sage/misc/package.py", line 196, in list_packages
    pkg['remote_version'] = pip_remote_version(p)
  File "/Users/mkoeppe/cvs/sage/local/lib/python2.7/site-packages/sage/misc/package.py", line 87, in pip_remote_version
    f = urlopen(url)
  File "/Users/mkoeppe/cvs/sage/local/lib/python/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/Users/mkoeppe/cvs/sage/local/lib/python/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/Users/mkoeppe/cvs/sage/local/lib/python/urllib2.py", line 449, in _open
    '_open', req)
  File "/Users/mkoeppe/cvs/sage/local/lib/python/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/Users/mkoeppe/cvs/sage/local/lib/python/urllib2.py", line 1240, in https_open
    context=self._context)
  File "/Users/mkoeppe/cvs/sage/local/lib/python/urllib2.py", line 1197, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>

(I don't have problems with sage -pip otherwise.)

comment:30 follow-up: Changed 5 years ago by vbraun

You are on OSX; Apple's openssl library is outdated and lacks tls 1.2 support.

comment:31 Changed 5 years ago by vdelecroix

And PyPI does not seem to accept http connections... I can catch URLError and just return version='?' in that case. What do you think?

comment:32 Changed 5 years ago by vdelecroix

  • Description modified (diff)

comment:33 Changed 5 years ago by git

  • Commit changed from faff48bd5628e808790ac83c8c827dd046224afd to 41aaa3a0d8c68c65036e161a1eb709f29e16da69

Branch pushed to git repo; I updated commit sha1. New commits:

41aaa3a19213: work around MacOSX (absent) TLS support

comment:34 follow-up: Changed 5 years ago by vdelecroix

Please test again. Normally you should see the same output as with sage-list-packages optional --local.

comment:35 Changed 5 years ago by git

  • Commit changed from 41aaa3a0d8c68c65036e161a1eb709f29e16da69 to 8750c2f728da978d7f782ebde0ff9927326742cf

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

8750c2f19213: work around MacOSX (absent) TLS support

comment:36 in reply to: ↑ 30 Changed 5 years ago by mkoeppe

Replying to vbraun:

You are on OSX; Apple's openssl library is outdated and lacks tls 1.2 support.

Thanks. Is there a documented workaround for this for Sage users/developers?

comment:37 in reply to: ↑ 34 ; follow-up: Changed 5 years ago by mkoeppe

Replying to vdelecroix:

Please test again. Normally you should see the same output as with sage-list-packages optional --local.

Seems to work now.

comment:38 in reply to: ↑ 37 ; follow-up: Changed 5 years ago by vdelecroix

Replying to mkoeppe:

Replying to vdelecroix:

Please test again. Normally you should see the same output as with sage-list-packages optional --local.

Seems to work now.

Yes but normally you can not see remote pip version (look at the lines for the packages beautifulsoup, biopython, sqlalchemy, etc)

comment:39 Changed 5 years ago by mkoeppe

  • Reviewers set to Matthias Koeppe, NEEDS MORE REVIEWERS

I only took a cursory look at the code changes. Perhaps someone familiar with this code should take a closer look.

comment:40 in reply to: ↑ 38 Changed 5 years ago by mkoeppe

Replying to vdelecroix:

Yes but normally you can not see remote pip version (look at the lines for the packages beautifulsoup, biopython, sqlalchemy, etc)

Yes, I see question marks for these packages.

comment:41 Changed 5 years ago by mkoeppe

Perhaps there should be a warning/notice when the pip version query failed.

comment:42 follow-up: Changed 5 years ago by git

  • Commit changed from 8750c2f728da978d7f782ebde0ff9927326742cf to de597b27c45d0f629698832a4677b92b48472c3a

Branch pushed to git repo; I updated commit sha1. New commits:

de597b219213: a warning with tests

comment:43 follow-up: Changed 5 years ago by jdemeyer

It is possible to somehow compute the package lists lazily? I am just a bit worried that those _STANDARD_PACKAGES lists will need to be computed every time that Sage starts.

comment:44 in reply to: ↑ 43 Changed 5 years ago by vdelecroix

Replying to jdemeyer:

It is possible to somehow compute the package lists lazily? I am just a bit worried that those _STANDARD_PACKAGES lists will need to be computed every time that Sage starts.

The module would better not be imported on startup. I will change the imports to become lazy.

comment:45 Changed 5 years ago by vdelecroix

Actually, it seems nearly impossible to be sure that sage.misc.package is not imported on startup using sage.misc.lazy_import.is_during_startup. The main reason is that is_package_installed is currently used in sage_setup.

comment:46 Changed 5 years ago by vdelecroix

As this is rather disjoint from the preoccupations of this ticket, I would prefer to postpone this problem. One solution is to wait for #20382 that get rid of is_installed_package. Then it would be trivial to forbid the import on startup.

comment:47 in reply to: ↑ 42 Changed 5 years ago by mkoeppe

  • Milestone changed from sage-7.3 to sage-7.4
  • Reviewers changed from Matthias Koeppe, NEEDS MORE REVIEWERS to Matthias Koeppe, Jeroen Demeyer

Replying to git:

Branch pushed to git repo; I updated commit sha1. New commits:

de597b219213: a warning with tests

OK, that's much better with these warnings.

From my side, this would be a 'positive_review'. Jeroen?

comment:48 follow-up: Changed 5 years ago by jdemeyer

I would like to see what the patchbot thinks of the startup time (however, it seems down now). If this ticket does not give a statistically significant increase to the startup time, it is ok for me. See 43

comment:49 in reply to: ↑ 48 ; follow-up: Changed 5 years ago by vdelecroix

Replying to jdemeyer:

I would like to see what the patchbot thinks of the startup time (however, it seems down now). If this ticket does not give a statistically significant increase to the startup time, it is ok for me. See 43

Since the lists are built with local=True this should even be faster than before. The old way to build these lists was via the script sage-package-list...

comment:50 in reply to: ↑ 49 ; follow-up: Changed 5 years ago by jdemeyer

Replying to vdelecroix:

The old way to build these lists was via the script sage-package-list...

Unless I'm missing something, in the old version, the list of packages was not computed every time that Sage starts.

comment:51 in reply to: ↑ 50 Changed 5 years ago by vdelecroix

Replying to jdemeyer:

Replying to vdelecroix:

The old way to build these lists was via the script sage-package-list...

Unless I'm missing something, in the old version, the list of packages was not computed every time that Sage starts.

It was the very same. All of _STANDARD_PACKAGES, _OPTIONAL_PACKAGES, _EXPERIMENTAL_PACKAGES where computed at module loading time (since they are used in the doc of the module). And the module is loaded on startup.

comment:52 Changed 5 years ago by jhpalmieri

  • Status changed from needs_review to needs_work

I see a significant difference in startup time: on one OS X machine it takes about 800ms longer on average with this branch. (I ran sage --startuptime 10 times with and without the branch, discarding the first run in each case. Average without: 1760ms. With: 2560ms.) If I run sage --startuptime sage.misc.package, without the branch it takes about 0.8ms to import that module, while with the branch: 800ms.

comment:53 Changed 5 years ago by git

  • Commit changed from de597b27c45d0f629698832a4677b92b48472c3a to c6578f47420f1ca835e20d596221c1835b831dcf

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

ad9249019213: clean sage.misc.package
8840aa319213: clean sage-list-packages script
383171a19213: use list_packages in sage.doctest.control
3b9716619213: work around MacOSX (absent) TLS support
c91197c19213: a warning with tests
c6578f4Trac 19213: remove X_PACKAGES from the doc + deprecations

comment:54 follow-up: Changed 5 years ago by vdelecroix

  • Status changed from needs_work to needs_review

Indeed... I was wrong the variables X_PACKAGES where hardcoded (and unmaintained).

I rebased and added some deprecations to all of standard_packages, optional_packages, experimental_packages and package_versions (since all the information is now available from list_packages).

comment:55 Changed 5 years ago by vdelecroix

  • Description modified (diff)

comment:56 in reply to: ↑ 54 Changed 5 years ago by jdemeyer

Replying to vdelecroix:

Indeed... I was wrong the variables X_PACKAGES where hardcoded

Yes, that's what I meant.

comment:57 Changed 5 years ago by jdemeyer

I don't see why standard_packages() and friends should be deprecated. They offer a user-friendly interface for list_packages().

comment:58 Changed 5 years ago by jdemeyer

Anyway, I don't really care much about this.

comment:59 Changed 5 years ago by git

  • Commit changed from c6578f47420f1ca835e20d596221c1835b831dcf to f0549980f6951f0fe6638afcdb309cdf50aa1378

Branch pushed to git repo; I updated commit sha1. New commits:

f054998Trac 19213: remove deprecations

comment:60 Changed 5 years ago by vdelecroix

I don't care either. But a pair of lists is not a lot more friendly than a dictionary.

comment:61 Changed 5 years ago by vdelecroix

  • Description modified (diff)

comment:62 Changed 5 years ago by jdemeyer

  • Status changed from needs_review to positive_review

comment:63 Changed 5 years ago by vbraun

  • Branch changed from u/vdelecroix/19213 to f0549980f6951f0fe6638afcdb309cdf50aa1378
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:64 Changed 4 years ago by mderickx

  • Commit f0549980f6951f0fe6638afcdb309cdf50aa1378 deleted

It seems that the deprecation warnings were removed in the latest commit f054998 , but the documentation still says they are deprecated. I modified the documentation accordingly in #23849 .

Note: See TracTickets for help on using tickets.