#19213 closed enhancement (fixed)
Cleaning sagepackagelist
Reported by:  jhpalmieri  Owned by:  

Priority:  minor  Milestone:  sage7.4 
Component:  scripts  Keywords:  
Cc:  vdelecroix, jdemeyer, vbraun, embray  Merged in:  
Authors:  Vincent Delecroix  Reviewers:  Matthias Koeppe, Jeroen Demeyer 
Report Upstream:  N/A  Work issues:  
Branch:  f054998 (Commits, GitHub, GitLab)  Commit:  
Dependencies:  Stopgaps: 
Description (last modified by )
This ticket enhances the module sage.misc.package
and the script sagelistpackages
:
 add independent functions in
sage.misc.package
to list the (new style) packages  add pip functionalities to
sage.misc.package
 do not list anymore old style packages (see also #21132)
 relies on
sage.misc.packages
insagelistpackages
pip type packages were created in #19187. This ticket is such that these packages will be listed with sage optional
. Sadly, it will still not be possible for these packages to be used in optional doctests. The reason for that is that we do not have proper version control for them.
Change History (64)
comment:1 Changed 7 years ago by
 Status changed from new to needs_review
 Type changed from PLEASE CHANGE to enhancement
comment:2 Changed 7 years ago by
 Branch set to u/jhpalmieri/pip_packages
comment:3 followup: ↓ 4 Changed 7 years ago by
 Commit set to 00224aafc17efcb3a4d5504bdbcf036fae5bca57
comment:4 in reply to: ↑ 3 Changed 7 years ago by
Replying to vdelecroix:
To my mind, it would nice to also list non installed pip packages as well as the last version available. The pip version can be obtained from Python with the json API
One disadvantage to this is that it requires internet access, which sage optional local
does not do. But this would be a good addition if local
were not given. If we do list the noninstalled packages, then it's going to get unwieldy if there are hundreds of entries in build/pkgs/piprules
. What if that file gets big?
Do we want to list these as optional packages or in another category? sage pip_packages
?
comment:5 followup: ↓ 6 Changed 7 years ago by
 Component changed from packages: optional to scripts
 Status changed from needs_review to needs_work
I do not want these pip packages to be treated like optional packages. If anything, they are closer to "experimental" than to "optional". We don't have control over them, so they are more likely to break when the upstream package gets upgraded.
comment:6 in reply to: ↑ 5 ; followup: ↓ 7 Changed 7 years ago by
Replying to jdemeyer:
I do not want these pip packages to be treated like optional packages. If anything, they are closer to "experimental" than to "optional". We don't have control over them, so they are more likely to break when the upstream package gets upgraded.
Do you mean
 you want them listed in
sage experimental
, notsage optional
, or  you don't want them detected for the purpose of doctests,
or both?
comment:7 in reply to: ↑ 6 ; followup: ↓ 9 Changed 7 years ago by
Replying to jhpalmieri:
Do you mean
 you want them listed in
sage experimental
, notsage optional
, or you don't want them detected for the purpose of doctests,
or both?
The default for optional
flags in doctests really should be optional packages only, so I don't want pip packages to be tested by doctests by default. Therefore, I also don't want those packages to appear in sage optional
. So I really prefer to see them as a new category: not standard, not optional, not experimental.
comment:8 Changed 7 years ago by
Wait, if an (oldstyle) optional package is listed in piprules
, you still don't want it listed in sage optional
? Should it be removed altogether from the listing, or listed as not_installed
? If it's oldstyle, by the way, having it listed in sage optional
is somewhat independent of having it used in doctests, since the doctest framework only looks at local packages. So in my current branch, if we delete the changes to doctest/control.py
, then these pip packages won't be used automatically in doctests.
comment:9 in reply to: ↑ 7 Changed 7 years ago by
Replying to jdemeyer:
Replying to jhpalmieri:
Do you mean
 you want them listed in
sage experimental
, notsage optional
, or you don't want them detected for the purpose of doctests,
or both?
The default for
optional
flags in doctests really should be optional packages only, so I don't want pip packages to be tested by doctests by default. Therefore, I also don't want those packages to appear insage optional
. So I really prefer to see them as a new category: not standard, not optional, not experimental.
I see several solutions to deal with "pip packages":
 having a new category of packages (let say pythonoptional)
 be stricter about versioning, i.e. in piprules specify the version of the packages we support
brian=1.3.1 trac=2.3
Very simple to maintain!
I agree that it would be hard to prevent a pip package to just break because versions are not backward compatible. So solution 2 looks appropriate to me. Otherwise, we can have a more precise tagging system for pythonoptional packages
sage: my_test() # pythonoptional  3.5 <= beautifulsoup <= 4.1.2
It is more flexible than the piprules strict versioning solution but the cost of maintenance is much higher.
Vincent
comment:10 followup: ↓ 12 Changed 7 years ago by
I have a working version of sagelistpackages
doing
$ ./sagelistpackages python [package] .............................. [latest version] ([installed version]) beautifulsoup........................... 3.2.1 (3.2.1) biopython............................... 1.65 (1.65) brian................................... 1.4.1 (1.4.1) guppy................................... 0.1.10 (0.1.10) mercurial............................... 3.5.1 (3.5.1) mpi4py.................................. 1.3.1 (1.3.1) nibabel................................. 2.0.1 (2.0.1) pybtex.................................. 0.18 (0.18) pyflakes................................ 0.9.2 (0.9.2) pyopenssl............................... 0.15.1 (0.15.1) sqlalchemy.............................. 1.0.8 (1.0.8) trac.................................... 1.0.9 (1.0.9)
The only problem is that it is relatively long to get the information from the pip json API for all the packages. Perhaps, one can be smarter with urllib2
.
John, am I free to change the branch in order to make my proposition publicly available?
Vincent
comment:11 Changed 7 years ago by
Furthermore, could we separate the issues of:
 listing python packages
 including optional doctests for them
Vincent
comment:12 in reply to: ↑ 10 Changed 7 years ago by
Replying to vdelecroix:
John, am I free to change the branch in order to make my proposition publicly available?
Please do so.
comment:13 Changed 7 years ago by
I have a few questions and comments:
 if #19220 is merged (archiving superseded packages, including the
piprules
packages), how are users supposed to find out aboutbiopython
,brian
, etc.? We need to include an option tosage
which will callsagelistpackages python
. Maybe we should also have an option which lists all packages: first standard, then optional, then experimental, then python/pip. Or maybe group all of the nonstandard packages together, sorted by name. In any case, we shouldn't expect users to even be aware that there are several different types of nonstandard packages; they should instead just be able to get one list to see if some particular package is supported.
Maybe a list like
[package (type)] .......... [latest version] ([installed version]) ... biopython (pip) ........... latest version (installed version) ... chomp (experimental) ...... latest version (installed version) ... ore_algebra (optional) .... latest version (installed version) ...I'm not really sure, though.
 Regarding the versions, I am not convinced that we can accurately specify which version of each package we support. Who is going to maintain this? Unless there is some automatic way of determining the versions supported by Sage, the versions will quickly get out of date. If
brian
gets updated to 1.4.2 but we listbrian=1.4.1
, what happens if a user doessage i brian
? What happens if a user doessage pip install brian
?
comment:14 Changed 7 years ago by
 Branch changed from u/jhpalmieri/pip_packages to public/pip_packages
 Commit 00224aafc17efcb3a4d5504bdbcf036fae5bca57 deleted
comment:15 Changed 7 years ago by
 Branch changed from public/pip_packages to public/19213
 Commit set to 19e3b86ac89a4441bf101130674b4daec9e14910
New commits:
93ec09b  Change sage p to always install a package

c6ddd0e  Better help

f3d7ef3  Remove f option to sagespkg when running sage i

752e269  Use PKGclean target to implement sage f PKG

1812e4f  Merge tag '6.9.beta6' into t/19119/ticket/19119

adef308  Add rules for installing packages with pip

fdba3ee  Add mercurial and sqlalchemy (exstandard packages)

00224aa  trac 19213: include packages from build/pkgs/piprules in 'sage optional'

19e3b86  trac 19213: separation of python packages

comment:16 Changed 6 years ago by
 Cc vdelecroix jdemeyer vbraun embray added
 Milestone changed from sage6.9 to sage7.3
What is the status on this?
comment:17 Changed 6 years ago by
 Dependencies #19187 deleted
 Description modified (diff)
comment:18 Changed 6 years ago by
 Description modified (diff)
 Summary changed from For packages listed in build/pkgs/piprules, allow 'sage optional' to list them to Allow 'sage optional' to list pip packages
comment:19 Changed 6 years ago by
 Branch changed from public/19213 to u/vdelecroix/19213
 Commit changed from 19e3b86ac89a4441bf101130674b4daec9e14910 to 9af0741b99d457dea3dcf19b8a7acf179f8a139c
 Description modified (diff)
 Status changed from needs_work to needs_review
 Summary changed from Allow 'sage optional' to list pip packages to Cleaning the sagelistpackages script
comment:20 Changed 6 years ago by
 Commit changed from 9af0741b99d457dea3dcf19b8a7acf179f8a139c to 79d2e0011a5d851a03bb4b93f14f51d9e4887571
Branch pushed to git repo; I updated commit sha1. New commits:
79d2e00  19213: clean imports

comment:21 Changed 6 years ago by
 Commit changed from 79d2e0011a5d851a03bb4b93f14f51d9e4887571 to 9b058ef05334f517637683f907c3a32d03dbae95
Branch pushed to git repo; I updated commit sha1. New commits:
9b058ef  19213: fix doctest

comment:22 followup: ↓ 23 Changed 6 years ago by
ACHTUNG! According to an email by user dmuthiah, changes by this user on this ticket were done by an unautorised person...
comment:23 in reply to: ↑ 22 Changed 6 years ago by
Replying to dimpase:
ACHTUNG! According to an email by user dmuthiah, changes by this user on this ticket were done by an unautorised person...
that was me... (I sent an email). Sorry.
comment:24 Changed 6 years ago by
 Status changed from needs_review to needs_work
need to superseed sage/misc/packages
which contains *builtin* list of packages!!
comment:25 Changed 6 years ago by
 Description modified (diff)
comment:26 Changed 6 years ago by
 Description modified (diff)
 Summary changed from Cleaning the sagelistpackages script to Cleaning sagepackagelist
comment:27 Changed 6 years ago by
 Commit changed from 9b058ef05334f517637683f907c3a32d03dbae95 to faff48bd5628e808790ac83c8c827dd046224afd
comment:28 Changed 6 years ago by
 Status changed from needs_work to needs_review
comment:29 Changed 6 years ago by
This is probably a FAQ, but I'm getting this error:
$ sage optional [package]...............................[latest version] ([version]) Traceback (most recent call last): File "/Users/mkoeppe/cvs/sage/src/bin/sagelistpackages", line 67, in <module> L = list_packages('optional', 'pip', local=args['local']).values() File "/Users/mkoeppe/cvs/sage/local/lib/python2.7/sitepackages/sage/misc/package.py", line 196, in list_packages pkg['remote_version'] = pip_remote_version(p) File "/Users/mkoeppe/cvs/sage/local/lib/python2.7/sitepackages/sage/misc/package.py", line 87, in pip_remote_version f = urlopen(url) File "/Users/mkoeppe/cvs/sage/local/lib/python/urllib2.py", line 154, in urlopen return opener.open(url, data, timeout) File "/Users/mkoeppe/cvs/sage/local/lib/python/urllib2.py", line 431, in open response = self._open(req, data) File "/Users/mkoeppe/cvs/sage/local/lib/python/urllib2.py", line 449, in _open '_open', req) File "/Users/mkoeppe/cvs/sage/local/lib/python/urllib2.py", line 409, in _call_chain result = func(*args) File "/Users/mkoeppe/cvs/sage/local/lib/python/urllib2.py", line 1240, in https_open context=self._context) File "/Users/mkoeppe/cvs/sage/local/lib/python/urllib2.py", line 1197, in do_open raise URLError(err) urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>
(I don't have problems with sage pip
otherwise.)
comment:30 followup: ↓ 36 Changed 6 years ago by
You are on OSX; Apple's openssl library is outdated and lacks tls 1.2 support.
comment:31 Changed 6 years ago by
And PyPI does not seem to accept http connections... I can catch URLError and just return version='?'
in that case. What do you think?
comment:32 Changed 6 years ago by
 Description modified (diff)
comment:33 Changed 6 years ago by
 Commit changed from faff48bd5628e808790ac83c8c827dd046224afd to 41aaa3a0d8c68c65036e161a1eb709f29e16da69
Branch pushed to git repo; I updated commit sha1. New commits:
41aaa3a  19213: work around MacOSX (absent) TLS support

comment:34 followup: ↓ 37 Changed 6 years ago by
Please test again. Normally you should see the same output as with sagelistpackages optional local
.
comment:35 Changed 6 years ago by
 Commit changed from 41aaa3a0d8c68c65036e161a1eb709f29e16da69 to 8750c2f728da978d7f782ebde0ff9927326742cf
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
8750c2f  19213: work around MacOSX (absent) TLS support

comment:36 in reply to: ↑ 30 Changed 6 years ago by
Replying to vbraun:
You are on OSX; Apple's openssl library is outdated and lacks tls 1.2 support.
Thanks. Is there a documented workaround for this for Sage users/developers?
comment:37 in reply to: ↑ 34 ; followup: ↓ 38 Changed 6 years ago by
Replying to vdelecroix:
Please test again. Normally you should see the same output as with
sagelistpackages optional local
.
Seems to work now.
comment:38 in reply to: ↑ 37 ; followup: ↓ 40 Changed 6 years ago by
Replying to mkoeppe:
Replying to vdelecroix:
Please test again. Normally you should see the same output as with
sagelistpackages optional local
.Seems to work now.
Yes but normally you can not see remote pip version (look at the lines for the packages beautifulsoup
, biopython
, sqlalchemy
, etc)
comment:39 Changed 6 years ago by
 Reviewers set to Matthias Koeppe, NEEDS MORE REVIEWERS
I only took a cursory look at the code changes. Perhaps someone familiar with this code should take a closer look.
comment:40 in reply to: ↑ 38 Changed 6 years ago by
Replying to vdelecroix:
Yes but normally you can not see remote pip version (look at the lines for the packages
beautifulsoup
,biopython
,sqlalchemy
, etc)
Yes, I see question marks for these packages.
comment:41 Changed 6 years ago by
Perhaps there should be a warning/notice when the pip version query failed.
comment:42 followup: ↓ 47 Changed 6 years ago by
 Commit changed from 8750c2f728da978d7f782ebde0ff9927326742cf to de597b27c45d0f629698832a4677b92b48472c3a
Branch pushed to git repo; I updated commit sha1. New commits:
de597b2  19213: a warning with tests

comment:43 followup: ↓ 44 Changed 6 years ago by
It is possible to somehow compute the package lists lazily? I am just a bit worried that those _STANDARD_PACKAGES
lists will need to be computed every time that Sage starts.
comment:44 in reply to: ↑ 43 Changed 6 years ago by
Replying to jdemeyer:
It is possible to somehow compute the package lists lazily? I am just a bit worried that those
_STANDARD_PACKAGES
lists will need to be computed every time that Sage starts.
The module would better not be imported on startup. I will change the imports to become lazy.
comment:45 Changed 6 years ago by
Actually, it seems nearly impossible to be sure that sage.misc.package
is not imported on startup using sage.misc.lazy_import.is_during_startup
. The main reason is that is_package_installed
is currently used in sage_setup
.
comment:46 Changed 6 years ago by
As this is rather disjoint from the preoccupations of this ticket, I would prefer to postpone this problem. One solution is to wait for #20382 that get rid of is_installed_package
. Then it would be trivial to forbid the import on startup.
comment:47 in reply to: ↑ 42 Changed 6 years ago by
 Milestone changed from sage7.3 to sage7.4
 Reviewers changed from Matthias Koeppe, NEEDS MORE REVIEWERS to Matthias Koeppe, Jeroen Demeyer
comment:48 followup: ↓ 49 Changed 6 years ago by
I would like to see what the patchbot thinks of the startup time (however, it seems down now). If this ticket does not give a statistically significant increase to the startup time, it is ok for me. See 43
comment:49 in reply to: ↑ 48 ; followup: ↓ 50 Changed 6 years ago by
Replying to jdemeyer:
I would like to see what the patchbot thinks of the startup time (however, it seems down now). If this ticket does not give a statistically significant increase to the startup time, it is ok for me. See 43
Since the lists are built with local=True
this should even be faster than before. The old way to build these lists was via the script sagepackagelist
...
comment:50 in reply to: ↑ 49 ; followup: ↓ 51 Changed 6 years ago by
Replying to vdelecroix:
The old way to build these lists was via the script
sagepackagelist
...
Unless I'm missing something, in the old version, the list of packages was not computed every time that Sage starts.
comment:51 in reply to: ↑ 50 Changed 6 years ago by
Replying to jdemeyer:
Replying to vdelecroix:
The old way to build these lists was via the script
sagepackagelist
...Unless I'm missing something, in the old version, the list of packages was not computed every time that Sage starts.
It was the very same. All of _STANDARD_PACKAGES
, _OPTIONAL_PACKAGES
, _EXPERIMENTAL_PACKAGES
where computed at module loading time (since they are used in the doc of the module). And the module is loaded on startup.
comment:52 Changed 6 years ago by
 Status changed from needs_review to needs_work
I see a significant difference in startup time: on one OS X machine it takes about 800ms longer on average with this branch. (I ran sage startuptime
10 times with and without the branch, discarding the first run in each case. Average without: 1760ms. With: 2560ms.) If I run sage startuptime sage.misc.package
, without the branch it takes about 0.8ms to import that module, while with the branch: 800ms.
comment:53 Changed 6 years ago by
 Commit changed from de597b27c45d0f629698832a4677b92b48472c3a to c6578f47420f1ca835e20d596221c1835b831dcf
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
ad92490  19213: clean sage.misc.package

8840aa3  19213: clean sagelistpackages script

383171a  19213: use list_packages in sage.doctest.control

3b97166  19213: work around MacOSX (absent) TLS support

c91197c  19213: a warning with tests

c6578f4  Trac 19213: remove X_PACKAGES from the doc + deprecations

comment:54 followup: ↓ 56 Changed 6 years ago by
 Status changed from needs_work to needs_review
Indeed... I was wrong the variables X_PACKAGES
where hardcoded (and unmaintained).
I rebased and added some deprecations to all of standard_packages
, optional_packages
, experimental_packages
and package_versions
(since all the information is now available from list_packages
).
comment:55 Changed 6 years ago by
 Description modified (diff)
comment:56 in reply to: ↑ 54 Changed 6 years ago by
Replying to vdelecroix:
Indeed... I was wrong the variables
X_PACKAGES
where hardcoded
Yes, that's what I meant.
comment:57 Changed 6 years ago by
I don't see why standard_packages()
and friends should be deprecated. They offer a userfriendly interface for list_packages()
.
comment:58 Changed 6 years ago by
Anyway, I don't really care much about this.
comment:59 Changed 6 years ago by
 Commit changed from c6578f47420f1ca835e20d596221c1835b831dcf to f0549980f6951f0fe6638afcdb309cdf50aa1378
Branch pushed to git repo; I updated commit sha1. New commits:
f054998  Trac 19213: remove deprecations

comment:60 Changed 6 years ago by
I don't care either. But a pair of lists is not a lot more friendly than a dictionary.
comment:61 Changed 6 years ago by
 Description modified (diff)
comment:62 Changed 6 years ago by
 Status changed from needs_review to positive_review
comment:63 Changed 6 years ago by
 Branch changed from u/vdelecroix/19213 to f0549980f6951f0fe6638afcdb309cdf50aa1378
 Resolution set to fixed
 Status changed from positive_review to closed
comment:64 Changed 5 years ago by
 Commit f0549980f6951f0fe6638afcdb309cdf50aa1378 deleted
It seems that the deprecation warnings were removed in the latest commit f054998 , but the documentation still says they are deprecated. I modified the documentation accordingly in #23849 .
To my mind, it would nice to also list non installed pip packages as well as the last version available. The pip version can be obtained from Python with the json API
The last version available might differ from what is installed. And it is worth mentioning that an update is available.
New commits:
Change sage p to always install a package
Better help
Remove f option to sagespkg when running sage i
Use PKGclean target to implement sage f PKG
Merge tag '6.9.beta6' into t/19119/ticket/19119
Add rules for installing packages with pip
Add mercurial and sqlalchemy (exstandard packages)
trac 19213: include packages from build/pkgs/piprules in 'sage optional'