Cleaning sage-package-list

[jhpalmieri]

This ticket enhances the module sage.misc.package and the script sage-list-packages:

• add independent functions in sage.misc.package to list the (new style) packages
• add pip functionalities to sage.misc.package
• do not list anymore old style packages (see also #21132)
• relies on sage.misc.packages in sage-list-packages

pip type packages were created in #19187. This ticket is such that these packages will be listed with sage -optional. Sadly, it will still not be possible for these packages to be used in optional doctests. The reason for that is that we do not have proper version control for them.

[vdelecroix]

To my mind, it would nice to also list non installed pip packages as well as the last version available. The pip version can be obtained from Python with the json API

import urllib2
import json

def pip_version(package_name):
    url = "https://pypi.python.org/pypi/{}/json".format(package_name)

    try:
        data = json.load(urllib2.urlopen(urllib2.Request(url)))
    except urllib2.HTTPError:
        return None

    try:
        return data["info"]["version"]
    except KeyError:
        return None

The last version available might differ from what is installed. And it is worth mentioning that an update is available.

---

New commits:

​93ec09b	Change sage -p to always install a package
​c6ddd0e	Better help
​f3d7ef3	Remove -f option to sage-spkg when running sage -i
​752e269	Use PKG-clean target to implement sage -f PKG
​1812e4f	Merge tag '6.9.beta6' into t/19119/ticket/19119
​adef308	Add rules for installing packages with pip
​fdba3ee	Add mercurial and sqlalchemy (ex-standard packages)
​00224aa	trac 19213: include packages from build/pkgs/piprules in 'sage --optional'

[jhpalmieri]

Replying to vdelecroix:

To my mind, it would nice to also list non installed pip packages as well as the last version available. The pip version can be obtained from Python with the json API

One disadvantage to this is that it requires internet access, which sage --optional --local does not do. But this would be a good addition if --local were not given. If we do list the non-installed packages, then it's going to get unwieldy if there are hundreds of entries in build/pkgs/piprules. What if that file gets big?

Do we want to list these as optional packages or in another category? sage --pip_packages?

[jdemeyer]

I do not want these pip packages to be treated like optional packages. If anything, they are closer to "experimental" than to "optional". We don't have control over them, so they are more likely to break when the upstream package gets upgraded.

[jhpalmieri]

Replying to jdemeyer:

I do not want these pip packages to be treated like optional packages. If anything, they are closer to "experimental" than to "optional". We don't have control over them, so they are more likely to break when the upstream package gets upgraded.

Do you mean

• you want them listed in sage --experimental, not sage --optional, or
• you don't want them detected for the purpose of doctests,

or both?

[jdemeyer]

Replying to jhpalmieri:

Do you mean

• you want them listed in sage --experimental, not sage --optional, or
• you don't want them detected for the purpose of doctests,

or both?

The default for --optional flags in doctests really should be optional packages only, so I don't want pip packages to be tested by doctests by default. Therefore, I also don't want those packages to appear in sage --optional. So I really prefer to see them as a new category: not standard, not optional, not experimental.

[jhpalmieri]

Wait, if an (old-style) optional package is listed in piprules, you still don't want it listed in sage --optional? Should it be removed altogether from the listing, or listed as not_installed? If it's old-style, by the way, having it listed in sage --optional is somewhat independent of having it used in doctests, since the doctest framework only looks at local packages. So in my current branch, if we delete the changes to doctest/control.py, then these pip packages won't be used automatically in doctests.

[vdelecroix]

Replying to jdemeyer:

Replying to jhpalmieri:

Do you mean

• you want them listed in sage --experimental, not sage --optional, or
• you don't want them detected for the purpose of doctests,

or both?

The default for --optional flags in doctests really should be optional packages only, so I don't want pip packages to be tested by doctests by default. Therefore, I also don't want those packages to appear in sage --optional. So I really prefer to see them as a new category: not standard, not optional, not experimental.

I see several solutions to deal with "pip packages":

1. having a new category of packages (let say python-optional)
2. be stricter about versioning, i.e. in piprules specify the version of the packages we support

   brian=1.3.1
   trac=2.3
   

   Very simple to maintain!

I agree that it would be hard to prevent a pip package to just break because versions are not backward compatible. So solution 2 looks appropriate to me. Otherwise, we can have a more precise tagging system for python-optional packages

sage: my_test()  # python-optional -- 3.5 <= beautifulsoup <= 4.1.2

It is more flexible than the piprules strict versioning solution but the cost of maintenance is much higher.

Vincent

[vdelecroix]

I have a working version of sage-list-packages doing

$ ./sage-list-packages python
[package] .............................. [latest version] ([installed version])

beautifulsoup........................... 3.2.1 (3.2.1)
biopython............................... 1.65 (1.65)
brian................................... 1.4.1 (1.4.1)
guppy................................... 0.1.10 (0.1.10)
mercurial............................... 3.5.1 (3.5.1)
mpi4py.................................. 1.3.1 (1.3.1)
nibabel................................. 2.0.1 (2.0.1)
pybtex.................................. 0.18 (0.18)
pyflakes................................ 0.9.2 (0.9.2)
pyopenssl............................... 0.15.1 (0.15.1)
sqlalchemy.............................. 1.0.8 (1.0.8)
trac.................................... 1.0.9 (1.0.9)

The only problem is that it is relatively long to get the information from the pip json API for all the packages. Perhaps, one can be smarter with urllib2.

John, am I free to change the branch in order to make my proposition publicly available?

Vincent

[vdelecroix]

Furthermore, could we separate the issues of:

• listing python packages
• including optional doctests for them

Vincent

[jhpalmieri]

Replying to vdelecroix:

John, am I free to change the branch in order to make my proposition publicly available?

Please do so.

[jhpalmieri]

I have a few questions and comments:

• if #19220 is merged (archiving superseded packages, including the piprules packages), how are users supposed to find out about biopython, brian, etc.? We need to include an option to sage which will call sage-list-packages python. Maybe we should also have an option which lists all packages: first standard, then optional, then experimental, then python/pip. Or maybe group all of the non-standard packages together, sorted by name. In any case, we shouldn't expect users to even be aware that there are several different types of non-standard packages; they should instead just be able to get one list to see if some particular package is supported.

Maybe a list like

[package (type)] .......... [latest version] ([installed version])

...
biopython (pip) ........... latest version (installed version)
...
chomp (experimental) ...... latest version (installed version)
...
ore_algebra (optional) .... latest version (installed version)
...

I'm not really sure, though.

• Regarding the versions, I am not convinced that we can accurately specify which version of each package we support. Who is going to maintain this? Unless there is some automatic way of determining the versions supported by Sage, the versions will quickly get out of date. If brian gets updated to 1.4.2 but we list brian=1.4.1, what happens if a user does sage -i brian? What happens if a user does sage --pip install brian?

[vdelecroix]

New commits:

​93ec09b	Change sage -p to always install a package
​c6ddd0e	Better help
​f3d7ef3	Remove -f option to sage-spkg when running sage -i
​752e269	Use PKG-clean target to implement sage -f PKG
​1812e4f	Merge tag '6.9.beta6' into t/19119/ticket/19119
​adef308	Add rules for installing packages with pip
​fdba3ee	Add mercurial and sqlalchemy (ex-standard packages)
​00224aa	trac 19213: include packages from build/pkgs/piprules in 'sage --optional'
​19e3b86	trac 19213: separation of python packages

[mkoeppe]

What is the status on this?

[vdelecroix]

New commits:

​cf63980	19213: sage_setup.packages
​ab40123	19213: add standalone files to list old style packages
​9af0741	19213: clean sage-list-packages script

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​79d2e00

19213: clean imports

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​9b058ef

19213: fix doctest

[dimpase]

ACHTUNG! According to an email by user dmuthiah, changes by this user on this ticket were done by an unautorised person...

[vdelecroix]

Replying to dimpase:

ACHTUNG! According to an email by user dmuthiah, changes by this user on this ticket were done by an unautorised person...

that was me... (I sent an e-mail). Sorry.

[vdelecroix]

need to superseed sage/misc/packages which contains *builtin* list of packages!!

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​7d72e04	19213: clean sage.misc.package
​64cd482	19213: clean sage-list-packages script
​faff48b	19213: use list_packages in sage.doctest.control

[mkoeppe]

This is probably a FAQ, but I'm getting this error:

$ sage -optional
[package]...............................[latest version] ([version])

Traceback (most recent call last):
  File "/Users/mkoeppe/cvs/sage/src/bin/sage-list-packages", line 67, in <module>
    L = list_packages('optional', 'pip', local=args['local']).values()
  File "/Users/mkoeppe/cvs/sage/local/lib/python2.7/site-packages/sage/misc/package.py", line 196, in list_packages
    pkg['remote_version'] = pip_remote_version(p)
  File "/Users/mkoeppe/cvs/sage/local/lib/python2.7/site-packages/sage/misc/package.py", line 87, in pip_remote_version
    f = urlopen(url)
  File "/Users/mkoeppe/cvs/sage/local/lib/python/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/Users/mkoeppe/cvs/sage/local/lib/python/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/Users/mkoeppe/cvs/sage/local/lib/python/urllib2.py", line 449, in _open
    '_open', req)
  File "/Users/mkoeppe/cvs/sage/local/lib/python/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/Users/mkoeppe/cvs/sage/local/lib/python/urllib2.py", line 1240, in https_open
    context=self._context)
  File "/Users/mkoeppe/cvs/sage/local/lib/python/urllib2.py", line 1197, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>

(I don't have problems with sage -pip otherwise.)

[vbraun]

You are on OSX; Apple's openssl library is outdated and lacks tls 1.2 support.

[vdelecroix]

And PyPI does not seem to accept http connections... I can catch URLError and just return version='?' in that case. What do you think?

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​41aaa3a

19213: work around MacOSX (absent) TLS support

[vdelecroix]

Please test again. Normally you should see the same output as with sage-list-packages optional --local.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​8750c2f

19213: work around MacOSX (absent) TLS support

[mkoeppe]

Replying to vbraun:

You are on OSX; Apple's openssl library is outdated and lacks tls 1.2 support.

Thanks. Is there a documented workaround for this for Sage users/developers?

[mkoeppe]

Replying to vdelecroix:

Please test again. Normally you should see the same output as with sage-list-packages optional --local.

Seems to work now.

[vdelecroix]

Replying to mkoeppe:

Replying to vdelecroix:

Please test again. Normally you should see the same output as with sage-list-packages optional --local.

Seems to work now.

Yes but normally you can not see remote pip version (look at the lines for the packages beautifulsoup, biopython, sqlalchemy, etc)

[mkoeppe]

I only took a cursory look at the code changes. Perhaps someone familiar with this code should take a closer look.

[mkoeppe]

Replying to vdelecroix:

Yes but normally you can not see remote pip version (look at the lines for the packages beautifulsoup, biopython, sqlalchemy, etc)

Yes, I see question marks for these packages.

[mkoeppe]

Perhaps there should be a warning/notice when the pip version query failed.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​de597b2

19213: a warning with tests

[jdemeyer]

It is possible to somehow compute the package lists lazily? I am just a bit worried that those _STANDARD_PACKAGES lists will need to be computed every time that Sage starts.

[vdelecroix]

Replying to jdemeyer:

It is possible to somehow compute the package lists lazily? I am just a bit worried that those _STANDARD_PACKAGES lists will need to be computed every time that Sage starts.

The module would better not be imported on startup. I will change the imports to become lazy.

[vdelecroix]

Actually, it seems nearly impossible to be sure that sage.misc.package is not imported on startup using sage.misc.lazy_import.is_during_startup. The main reason is that is_package_installed is currently used in sage_setup.

[vdelecroix]

As this is rather disjoint from the preoccupations of this ticket, I would prefer to postpone this problem. One solution is to wait for #20382 that get rid of is_installed_package. Then it would be trivial to forbid the import on startup.

[mkoeppe]

Replying to git:

Branch pushed to git repo; I updated commit sha1. New commits:

​de597b2

19213: a warning with tests

OK, that's much better with these warnings.

From my side, this would be a 'positive_review'. Jeroen?

[jdemeyer]

I would like to see what the patchbot thinks of the startup time (however, it seems down now). If this ticket does not give a statistically significant increase to the startup time, it is ok for me. See 43

[vdelecroix]

Replying to jdemeyer:

I would like to see what the patchbot thinks of the startup time (however, it seems down now). If this ticket does not give a statistically significant increase to the startup time, it is ok for me. See 43

Since the lists are built with local=True this should even be faster than before. The old way to build these lists was via the script sage-package-list...

[jdemeyer]

Replying to vdelecroix:

The old way to build these lists was via the script sage-package-list...

Unless I'm missing something, in the old version, the list of packages was not computed every time that Sage starts.

[vdelecroix]

Replying to jdemeyer:

Replying to vdelecroix:

The old way to build these lists was via the script sage-package-list...

Unless I'm missing something, in the old version, the list of packages was not computed every time that Sage starts.

It was the very same. All of _STANDARD_PACKAGES, _OPTIONAL_PACKAGES, _EXPERIMENTAL_PACKAGES where computed at module loading time (since they are used in the doc of the module). And the module is loaded on startup.

[jhpalmieri]

I see a significant difference in startup time: on one OS X machine it takes about 800ms longer on average with this branch. (I ran sage --startuptime 10 times with and without the branch, discarding the first run in each case. Average without: 1760ms. With: 2560ms.) If I run sage --startuptime sage.misc.package, without the branch it takes about 0.8ms to import that module, while with the branch: 800ms.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​ad92490	19213: clean sage.misc.package
​8840aa3	19213: clean sage-list-packages script
​383171a	19213: use list_packages in sage.doctest.control
​3b97166	19213: work around MacOSX (absent) TLS support
​c91197c	19213: a warning with tests
​c6578f4	Trac 19213: remove X_PACKAGES from the doc + deprecations

[vdelecroix]

Indeed... I was wrong the variables X_PACKAGES where hardcoded (and unmaintained).

I rebased and added some deprecations to all of standard_packages, optional_packages, experimental_packages and package_versions (since all the information is now available from list_packages).

[jdemeyer]

Replying to vdelecroix:

Indeed... I was wrong the variables X_PACKAGES where hardcoded

Yes, that's what I meant.

[jdemeyer]

I don't see why standard_packages() and friends should be deprecated. They offer a user-friendly interface for list_packages().

[jdemeyer]

Anyway, I don't really care much about this.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f054998

Trac 19213: remove deprecations

[vdelecroix]

I don't care either. But a pair of lists is not a lot more friendly than a dictionary.

[mderickx]

It seems that the deprecation warnings were removed in the latest commit f054998 , but the documentation still says they are deprecated. I modified the documentation accordingly in #23849 .