#19213 closed enhancement (fixed)
Cleaning sage-package-list
Reported by: | jhpalmieri | Owned by: | |
---|---|---|---|
Priority: | minor | Milestone: | sage-7.4 |
Component: | scripts | Keywords: | |
Cc: | vdelecroix, jdemeyer, vbraun, embray | Merged in: | |
Authors: | Vincent Delecroix | Reviewers: | Matthias Koeppe, Jeroen Demeyer |
Report Upstream: | N/A | Work issues: | |
Branch: | f054998 (Commits, GitHub, GitLab) | Commit: | |
Dependencies: | Stopgaps: |
Description (last modified by )
This ticket enhances the module sage.misc.package
and the script sage-list-packages
:
- add independent functions in
sage.misc.package
to list the (new style) packages - add pip functionalities to
sage.misc.package
- do not list anymore old style packages (see also #21132)
- relies on
sage.misc.packages
insage-list-packages
pip type packages were created in #19187. This ticket is such that these packages will be listed with sage -optional
. Sadly, it will still not be possible for these packages to be used in optional doctests. The reason for that is that we do not have proper version control for them.
Change History (64)
comment:1 Changed 7 years ago by
- Status changed from new to needs_review
- Type changed from PLEASE CHANGE to enhancement
comment:2 Changed 7 years ago by
- Branch set to u/jhpalmieri/pip_packages
comment:3 follow-up: ↓ 4 Changed 7 years ago by
- Commit set to 00224aafc17efcb3a4d5504bdbcf036fae5bca57
comment:4 in reply to: ↑ 3 Changed 7 years ago by
Replying to vdelecroix:
To my mind, it would nice to also list non installed pip packages as well as the last version available. The pip version can be obtained from Python with the json API
One disadvantage to this is that it requires internet access, which sage --optional --local
does not do. But this would be a good addition if --local
were not given. If we do list the non-installed packages, then it's going to get unwieldy if there are hundreds of entries in build/pkgs/piprules
. What if that file gets big?
Do we want to list these as optional packages or in another category? sage --pip_packages
?
comment:5 follow-up: ↓ 6 Changed 7 years ago by
- Component changed from packages: optional to scripts
- Status changed from needs_review to needs_work
I do not want these pip packages to be treated like optional packages. If anything, they are closer to "experimental" than to "optional". We don't have control over them, so they are more likely to break when the upstream package gets upgraded.
comment:6 in reply to: ↑ 5 ; follow-up: ↓ 7 Changed 7 years ago by
Replying to jdemeyer:
I do not want these pip packages to be treated like optional packages. If anything, they are closer to "experimental" than to "optional". We don't have control over them, so they are more likely to break when the upstream package gets upgraded.
Do you mean
- you want them listed in
sage --experimental
, notsage --optional
, or - you don't want them detected for the purpose of doctests,
or both?
comment:7 in reply to: ↑ 6 ; follow-up: ↓ 9 Changed 7 years ago by
Replying to jhpalmieri:
Do you mean
- you want them listed in
sage --experimental
, notsage --optional
, or- you don't want them detected for the purpose of doctests,
or both?
The default for --optional
flags in doctests really should be optional packages only, so I don't want pip packages to be tested by doctests by default. Therefore, I also don't want those packages to appear in sage --optional
. So I really prefer to see them as a new category: not standard, not optional, not experimental.
comment:8 Changed 7 years ago by
Wait, if an (old-style) optional package is listed in piprules
, you still don't want it listed in sage --optional
? Should it be removed altogether from the listing, or listed as not_installed
? If it's old-style, by the way, having it listed in sage --optional
is somewhat independent of having it used in doctests, since the doctest framework only looks at local packages. So in my current branch, if we delete the changes to doctest/control.py
, then these pip packages won't be used automatically in doctests.
comment:9 in reply to: ↑ 7 Changed 7 years ago by
Replying to jdemeyer:
Replying to jhpalmieri:
Do you mean
- you want them listed in
sage --experimental
, notsage --optional
, or- you don't want them detected for the purpose of doctests,
or both?
The default for
--optional
flags in doctests really should be optional packages only, so I don't want pip packages to be tested by doctests by default. Therefore, I also don't want those packages to appear insage --optional
. So I really prefer to see them as a new category: not standard, not optional, not experimental.
I see several solutions to deal with "pip packages":
- having a new category of packages (let say python-optional)
- be stricter about versioning, i.e. in piprules specify the version of the packages we support
brian=1.3.1 trac=2.3
Very simple to maintain!
I agree that it would be hard to prevent a pip package to just break because versions are not backward compatible. So solution 2 looks appropriate to me. Otherwise, we can have a more precise tagging system for python-optional packages
sage: my_test() # python-optional -- 3.5 <= beautifulsoup <= 4.1.2
It is more flexible than the piprules strict versioning solution but the cost of maintenance is much higher.
Vincent
comment:10 follow-up: ↓ 12 Changed 7 years ago by
I have a working version of sage-list-packages
doing
$ ./sage-list-packages python [package] .............................. [latest version] ([installed version]) beautifulsoup........................... 3.2.1 (3.2.1) biopython............................... 1.65 (1.65) brian................................... 1.4.1 (1.4.1) guppy................................... 0.1.10 (0.1.10) mercurial............................... 3.5.1 (3.5.1) mpi4py.................................. 1.3.1 (1.3.1) nibabel................................. 2.0.1 (2.0.1) pybtex.................................. 0.18 (0.18) pyflakes................................ 0.9.2 (0.9.2) pyopenssl............................... 0.15.1 (0.15.1) sqlalchemy.............................. 1.0.8 (1.0.8) trac.................................... 1.0.9 (1.0.9)
The only problem is that it is relatively long to get the information from the pip json API for all the packages. Perhaps, one can be smarter with urllib2
.
John, am I free to change the branch in order to make my proposition publicly available?
Vincent
comment:11 Changed 7 years ago by
Furthermore, could we separate the issues of:
- listing python packages
- including optional doctests for them
Vincent
comment:12 in reply to: ↑ 10 Changed 7 years ago by
Replying to vdelecroix:
John, am I free to change the branch in order to make my proposition publicly available?
Please do so.
comment:13 Changed 7 years ago by
I have a few questions and comments:
- if #19220 is merged (archiving superseded packages, including the
piprules
packages), how are users supposed to find out aboutbiopython
,brian
, etc.? We need to include an option tosage
which will callsage-list-packages python
. Maybe we should also have an option which lists all packages: first standard, then optional, then experimental, then python/pip. Or maybe group all of the non-standard packages together, sorted by name. In any case, we shouldn't expect users to even be aware that there are several different types of non-standard packages; they should instead just be able to get one list to see if some particular package is supported.
Maybe a list like
[package (type)] .......... [latest version] ([installed version]) ... biopython (pip) ........... latest version (installed version) ... chomp (experimental) ...... latest version (installed version) ... ore_algebra (optional) .... latest version (installed version) ...I'm not really sure, though.
- Regarding the versions, I am not convinced that we can accurately specify which version of each package we support. Who is going to maintain this? Unless there is some automatic way of determining the versions supported by Sage, the versions will quickly get out of date. If
brian
gets updated to 1.4.2 but we listbrian=1.4.1
, what happens if a user doessage -i brian
? What happens if a user doessage --pip install brian
?
comment:14 Changed 7 years ago by
- Branch changed from u/jhpalmieri/pip_packages to public/pip_packages
- Commit 00224aafc17efcb3a4d5504bdbcf036fae5bca57 deleted
comment:15 Changed 7 years ago by
- Branch changed from public/pip_packages to public/19213
- Commit set to 19e3b86ac89a4441bf101130674b4daec9e14910
New commits:
93ec09b | Change sage -p to always install a package
|
c6ddd0e | Better help
|
f3d7ef3 | Remove -f option to sage-spkg when running sage -i
|
752e269 | Use PKG-clean target to implement sage -f PKG
|
1812e4f | Merge tag '6.9.beta6' into t/19119/ticket/19119
|
adef308 | Add rules for installing packages with pip
|
fdba3ee | Add mercurial and sqlalchemy (ex-standard packages)
|
00224aa | trac 19213: include packages from build/pkgs/piprules in 'sage --optional'
|
19e3b86 | trac 19213: separation of python packages
|
comment:16 Changed 6 years ago by
- Cc vdelecroix jdemeyer vbraun embray added
- Milestone changed from sage-6.9 to sage-7.3
What is the status on this?
comment:17 Changed 6 years ago by
- Dependencies #19187 deleted
- Description modified (diff)
comment:18 Changed 6 years ago by
- Description modified (diff)
- Summary changed from For packages listed in build/pkgs/piprules, allow 'sage --optional' to list them to Allow 'sage --optional' to list pip packages
comment:19 Changed 6 years ago by
- Branch changed from public/19213 to u/vdelecroix/19213
- Commit changed from 19e3b86ac89a4441bf101130674b4daec9e14910 to 9af0741b99d457dea3dcf19b8a7acf179f8a139c
- Description modified (diff)
- Status changed from needs_work to needs_review
- Summary changed from Allow 'sage --optional' to list pip packages to Cleaning the sage-list-packages script
comment:20 Changed 6 years ago by
- Commit changed from 9af0741b99d457dea3dcf19b8a7acf179f8a139c to 79d2e0011a5d851a03bb4b93f14f51d9e4887571
Branch pushed to git repo; I updated commit sha1. New commits:
79d2e00 | 19213: clean imports
|
comment:21 Changed 6 years ago by
- Commit changed from 79d2e0011a5d851a03bb4b93f14f51d9e4887571 to 9b058ef05334f517637683f907c3a32d03dbae95
Branch pushed to git repo; I updated commit sha1. New commits:
9b058ef | 19213: fix doctest
|
comment:22 follow-up: ↓ 23 Changed 6 years ago by
ACHTUNG! According to an email by user dmuthiah, changes by this user on this ticket were done by an unautorised person...
comment:23 in reply to: ↑ 22 Changed 6 years ago by
Replying to dimpase:
ACHTUNG! According to an email by user dmuthiah, changes by this user on this ticket were done by an unautorised person...
that was me... (I sent an e-mail). Sorry.
comment:24 Changed 6 years ago by
- Status changed from needs_review to needs_work
need to superseed sage/misc/packages
which contains *builtin* list of packages!!
comment:25 Changed 6 years ago by
- Description modified (diff)
comment:26 Changed 6 years ago by
- Description modified (diff)
- Summary changed from Cleaning the sage-list-packages script to Cleaning sage-package-list
comment:27 Changed 6 years ago by
- Commit changed from 9b058ef05334f517637683f907c3a32d03dbae95 to faff48bd5628e808790ac83c8c827dd046224afd
comment:28 Changed 6 years ago by
- Status changed from needs_work to needs_review
comment:29 Changed 6 years ago by
This is probably a FAQ, but I'm getting this error:
$ sage -optional [package]...............................[latest version] ([version]) Traceback (most recent call last): File "/Users/mkoeppe/cvs/sage/src/bin/sage-list-packages", line 67, in <module> L = list_packages('optional', 'pip', local=args['local']).values() File "/Users/mkoeppe/cvs/sage/local/lib/python2.7/site-packages/sage/misc/package.py", line 196, in list_packages pkg['remote_version'] = pip_remote_version(p) File "/Users/mkoeppe/cvs/sage/local/lib/python2.7/site-packages/sage/misc/package.py", line 87, in pip_remote_version f = urlopen(url) File "/Users/mkoeppe/cvs/sage/local/lib/python/urllib2.py", line 154, in urlopen return opener.open(url, data, timeout) File "/Users/mkoeppe/cvs/sage/local/lib/python/urllib2.py", line 431, in open response = self._open(req, data) File "/Users/mkoeppe/cvs/sage/local/lib/python/urllib2.py", line 449, in _open '_open', req) File "/Users/mkoeppe/cvs/sage/local/lib/python/urllib2.py", line 409, in _call_chain result = func(*args) File "/Users/mkoeppe/cvs/sage/local/lib/python/urllib2.py", line 1240, in https_open context=self._context) File "/Users/mkoeppe/cvs/sage/local/lib/python/urllib2.py", line 1197, in do_open raise URLError(err) urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>
(I don't have problems with sage -pip
otherwise.)
comment:30 follow-up: ↓ 36 Changed 6 years ago by
You are on OSX; Apple's openssl library is outdated and lacks tls 1.2 support.
comment:31 Changed 6 years ago by
And PyPI does not seem to accept http connections... I can catch URLError and just return version='?'
in that case. What do you think?
comment:32 Changed 6 years ago by
- Description modified (diff)
comment:33 Changed 6 years ago by
- Commit changed from faff48bd5628e808790ac83c8c827dd046224afd to 41aaa3a0d8c68c65036e161a1eb709f29e16da69
Branch pushed to git repo; I updated commit sha1. New commits:
41aaa3a | 19213: work around MacOSX (absent) TLS support
|
comment:34 follow-up: ↓ 37 Changed 6 years ago by
Please test again. Normally you should see the same output as with sage-list-packages optional --local
.
comment:35 Changed 6 years ago by
- Commit changed from 41aaa3a0d8c68c65036e161a1eb709f29e16da69 to 8750c2f728da978d7f782ebde0ff9927326742cf
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
8750c2f | 19213: work around MacOSX (absent) TLS support
|
comment:36 in reply to: ↑ 30 Changed 6 years ago by
Replying to vbraun:
You are on OSX; Apple's openssl library is outdated and lacks tls 1.2 support.
Thanks. Is there a documented workaround for this for Sage users/developers?
comment:37 in reply to: ↑ 34 ; follow-up: ↓ 38 Changed 6 years ago by
Replying to vdelecroix:
Please test again. Normally you should see the same output as with
sage-list-packages optional --local
.
Seems to work now.
comment:38 in reply to: ↑ 37 ; follow-up: ↓ 40 Changed 6 years ago by
Replying to mkoeppe:
Replying to vdelecroix:
Please test again. Normally you should see the same output as with
sage-list-packages optional --local
.Seems to work now.
Yes but normally you can not see remote pip version (look at the lines for the packages beautifulsoup
, biopython
, sqlalchemy
, etc)
comment:39 Changed 6 years ago by
- Reviewers set to Matthias Koeppe, NEEDS MORE REVIEWERS
I only took a cursory look at the code changes. Perhaps someone familiar with this code should take a closer look.
comment:40 in reply to: ↑ 38 Changed 6 years ago by
Replying to vdelecroix:
Yes but normally you can not see remote pip version (look at the lines for the packages
beautifulsoup
,biopython
,sqlalchemy
, etc)
Yes, I see question marks for these packages.
comment:41 Changed 6 years ago by
Perhaps there should be a warning/notice when the pip version query failed.
comment:42 follow-up: ↓ 47 Changed 6 years ago by
- Commit changed from 8750c2f728da978d7f782ebde0ff9927326742cf to de597b27c45d0f629698832a4677b92b48472c3a
Branch pushed to git repo; I updated commit sha1. New commits:
de597b2 | 19213: a warning with tests
|
comment:43 follow-up: ↓ 44 Changed 6 years ago by
It is possible to somehow compute the package lists lazily? I am just a bit worried that those _STANDARD_PACKAGES
lists will need to be computed every time that Sage starts.
comment:44 in reply to: ↑ 43 Changed 6 years ago by
Replying to jdemeyer:
It is possible to somehow compute the package lists lazily? I am just a bit worried that those
_STANDARD_PACKAGES
lists will need to be computed every time that Sage starts.
The module would better not be imported on startup. I will change the imports to become lazy.
comment:45 Changed 6 years ago by
Actually, it seems nearly impossible to be sure that sage.misc.package
is not imported on startup using sage.misc.lazy_import.is_during_startup
. The main reason is that is_package_installed
is currently used in sage_setup
.
comment:46 Changed 6 years ago by
As this is rather disjoint from the preoccupations of this ticket, I would prefer to postpone this problem. One solution is to wait for #20382 that get rid of is_installed_package
. Then it would be trivial to forbid the import on startup.
comment:47 in reply to: ↑ 42 Changed 6 years ago by
- Milestone changed from sage-7.3 to sage-7.4
- Reviewers changed from Matthias Koeppe, NEEDS MORE REVIEWERS to Matthias Koeppe, Jeroen Demeyer
comment:48 follow-up: ↓ 49 Changed 6 years ago by
I would like to see what the patchbot thinks of the startup time (however, it seems down now). If this ticket does not give a statistically significant increase to the startup time, it is ok for me. See 43
comment:49 in reply to: ↑ 48 ; follow-up: ↓ 50 Changed 6 years ago by
Replying to jdemeyer:
I would like to see what the patchbot thinks of the startup time (however, it seems down now). If this ticket does not give a statistically significant increase to the startup time, it is ok for me. See 43
Since the lists are built with local=True
this should even be faster than before. The old way to build these lists was via the script sage-package-list
...
comment:50 in reply to: ↑ 49 ; follow-up: ↓ 51 Changed 6 years ago by
Replying to vdelecroix:
The old way to build these lists was via the script
sage-package-list
...
Unless I'm missing something, in the old version, the list of packages was not computed every time that Sage starts.
comment:51 in reply to: ↑ 50 Changed 6 years ago by
Replying to jdemeyer:
Replying to vdelecroix:
The old way to build these lists was via the script
sage-package-list
...Unless I'm missing something, in the old version, the list of packages was not computed every time that Sage starts.
It was the very same. All of _STANDARD_PACKAGES
, _OPTIONAL_PACKAGES
, _EXPERIMENTAL_PACKAGES
where computed at module loading time (since they are used in the doc of the module). And the module is loaded on startup.
comment:52 Changed 6 years ago by
- Status changed from needs_review to needs_work
I see a significant difference in startup time: on one OS X machine it takes about 800ms longer on average with this branch. (I ran sage --startuptime
10 times with and without the branch, discarding the first run in each case. Average without: 1760ms. With: 2560ms.) If I run sage --startuptime sage.misc.package
, without the branch it takes about 0.8ms to import that module, while with the branch: 800ms.
comment:53 Changed 6 years ago by
- Commit changed from de597b27c45d0f629698832a4677b92b48472c3a to c6578f47420f1ca835e20d596221c1835b831dcf
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
ad92490 | 19213: clean sage.misc.package
|
8840aa3 | 19213: clean sage-list-packages script
|
383171a | 19213: use list_packages in sage.doctest.control
|
3b97166 | 19213: work around MacOSX (absent) TLS support
|
c91197c | 19213: a warning with tests
|
c6578f4 | Trac 19213: remove X_PACKAGES from the doc + deprecations
|
comment:54 follow-up: ↓ 56 Changed 6 years ago by
- Status changed from needs_work to needs_review
Indeed... I was wrong the variables X_PACKAGES
where hardcoded (and unmaintained).
I rebased and added some deprecations to all of standard_packages
, optional_packages
, experimental_packages
and package_versions
(since all the information is now available from list_packages
).
comment:55 Changed 6 years ago by
- Description modified (diff)
comment:56 in reply to: ↑ 54 Changed 6 years ago by
Replying to vdelecroix:
Indeed... I was wrong the variables
X_PACKAGES
where hardcoded
Yes, that's what I meant.
comment:57 Changed 6 years ago by
I don't see why standard_packages()
and friends should be deprecated. They offer a user-friendly interface for list_packages()
.
comment:58 Changed 6 years ago by
Anyway, I don't really care much about this.
comment:59 Changed 6 years ago by
- Commit changed from c6578f47420f1ca835e20d596221c1835b831dcf to f0549980f6951f0fe6638afcdb309cdf50aa1378
Branch pushed to git repo; I updated commit sha1. New commits:
f054998 | Trac 19213: remove deprecations
|
comment:60 Changed 6 years ago by
I don't care either. But a pair of lists is not a lot more friendly than a dictionary.
comment:61 Changed 6 years ago by
- Description modified (diff)
comment:62 Changed 6 years ago by
- Status changed from needs_review to positive_review
comment:63 Changed 6 years ago by
- Branch changed from u/vdelecroix/19213 to f0549980f6951f0fe6638afcdb309cdf50aa1378
- Resolution set to fixed
- Status changed from positive_review to closed
comment:64 Changed 5 years ago by
- Commit f0549980f6951f0fe6638afcdb309cdf50aa1378 deleted
It seems that the deprecation warnings were removed in the latest commit f054998 , but the documentation still says they are deprecated. I modified the documentation accordingly in #23849 .
To my mind, it would nice to also list non installed pip packages as well as the last version available. The pip version can be obtained from Python with the json API
The last version available might differ from what is installed. And it is worth mentioning that an update is available.
New commits:
Change sage -p to always install a package
Better help
Remove -f option to sage-spkg when running sage -i
Use PKG-clean target to implement sage -f PKG
Merge tag '6.9.beta6' into t/19119/ticket/19119
Add rules for installing packages with pip
Add mercurial and sqlalchemy (ex-standard packages)
trac 19213: include packages from build/pkgs/piprules in 'sage --optional'