Opened 6 years ago
Closed 5 years ago
#16759 closed enhancement (fixed)
install_package() is obsolete
Reported by:  jdemeyer  Owned by:  

Priority:  major  Milestone:  sage6.9 
Component:  misc  Keywords:  
Cc:  was  Merged in:  
Authors:  Jeroen Demeyer  Reviewers:  John Palmieri 
Report Upstream:  N/A  Work issues:  
Branch:  da03be1 (Commits)  Commit:  da03be13988e41fc483ee2001654a55d4e45ee95 
Dependencies:  Stopgaps:  #16760 
Description (last modified by )
Packages should be installed using sage i
from a shell, not with the install_package()
Sage command.
For consistency in error messages, we introduce the exception PackageNotFoundError
which should be used to signal that a package is not installed. This idea was taken from an existing but unused exception OptionalPackageNotFoundError
(which is removed).
Change History (40)
comment:1 followup: ↓ 2 Changed 6 years ago by
comment:2 in reply to: ↑ 1 Changed 6 years ago by
Replying to jhpalmieri:
It also looks like
sage standard
is broken: it is using old package information.
I would say that sage standard
is doing the right thing, but somehow http://www.sagemath.org/packages/standard/
is no longer updated. The right solution is updating the website script to take into account the new gitstyle packages (which is certainly a different issue than the one this ticket is about).
comment:3 Changed 6 years ago by
...that being said, it makes no sense to work on this ticket if the underlying sage standard
command isn't fixed.
comment:4 Changed 6 years ago by
Along similar lines, does sage i
work properly with installing optional packages that have both "old" and "new" package versions?
But I agree with William that this is important. I'm working at a Sage Days prep right now with exactly the kind of people William was talking about in the thread that led to this  people who know quite a bit of math and might want to use some specialized optional package (say, polymake) but would need an awful lot of handholding to make it through some of this weird behavior.
comment:5 Changed 6 years ago by
 Stopgaps set to #16760
Here's an option: turn off install_package
until we can fix this. See #16760.
comment:6 Changed 6 years ago by
 Milestone changed from sage6.3 to sage6.4
comment:7 Changed 6 years ago by
This might be related to the case of the tarball : you use Pillow2.2.2.tar.gz
. it turns out that sage sh sagefixpkgchecksums
, that computes the checksumes, will treat only lowercasenamed tarballs ... except on Macs, whose filesystems will find Pillow2.2.2.tar.gz
when asked for pillow2.2.2.tar.gz
. See #18229 for discussion.
This is documented in the Developer's guide (http://www.sagemath.org/doc/developer/packaging.html, paragraph "Directory structure"), that states :
"The build scripts and associated files are in a subdirectory SAGE_ROOT/build/pkgs/package, where you replace package with a lowercase version of the upstream project name."
I suppose that the original Pillow porter might have been a Mac user, and that nobody caught the problem.
HTH,
comment:8 followup: ↓ 11 Changed 6 years ago by
From the sagefixpkgchecksums
script, it looks like the case for the directory name (in SAGE_ROOT/build/pkgs/
) must match the case for the tarball (except on OS X). Why not change the policy that currently requires the directory to be all lowercase? Would it cause upgrade problems on OS X if we renamed "pillow" to "Pillow", given the stupid way it deals with case?
comment:9 Changed 6 years ago by
Or rewrite sagefixpkgchecksums
to convert the tarball to lowercase to determine the directory name. That might be a better choice.

src/bin/sagefixpkgchecksums
diff git a/src/bin/sagefixpkgchecksums b/src/bin/sagefixpkgchecksums index 65542b9..b42463d 100755
a b fi 11 11 for upstream in "$@" 12 12 do 13 13 tarball=`basename "$upstream"` 14 pkg_name= ${tarball%%*}14 pkg_name=`echo ${tarball%%*}  tr '[:upper:]' '[:lower:]'` 15 15 pkg_compression=${tarball#*.tar} # gz or bz2 16 16 if [ d "$SAGE_ROOT/build/pkgs/$pkg_name" ]; then 17 17 sage_version=`cat "$SAGE_ROOT/build/pkgs/$pkg_name/packageversion.txt"  sed 's/\.p[09][09]*$//'`
Maybe print a warning whenever the case is changed, just to be safe?
comment:10 Changed 6 years ago by
Oops, there are more changes required: we need to keep track of pkg_name
for the directory, and also the version with the original case for use in
echo "tarball=$original_pkg_nameVERSION.tar$pkg_compression" > $checksums
comment:11 in reply to: ↑ 8 ; followup: ↓ 12 Changed 6 years ago by
Replying to jhpalmieri:
Would you mind joining this thead on sagedevel ? There is not only a (small) technical problem, but also a policy problem, and this thread i probably not the right place to discuss it. In other words, we know that at least the sagefixspkgchecksums
script won't work on mixedCase tarballs, and I wonder what will be broken if this script is fixed to accept mixedcase tarball names. Leif thinks that's only a convention, but I am not so sure...
Update : Just saw your patch. It fixes the technical problem, but what other holes does it open ?
From the
sagefixpkgchecksums
script, it looks like the case for the directory name (inSAGE_ROOT/build/pkgs/
) must match the case for the tarball (except on OS X). Why not change the policy that currently requires the directory to be all lowercase? Would it cause upgrade problems on OS X if we renamed "pillow" to "Pillow", given the stupid way it deals with case?
comment:12 in reply to: ↑ 11 Changed 6 years ago by
Replying to charpent:
Update : Just saw your patch. It fixes the technical problem, but what other holes does it open ?
Good question. Since we want to support OS X, we cannot allow different files whose names are the same except for case differences, so we cannot allow different directories build/Pillow
and build/PiLlOW
. So I hope it doesn't open any holes. As I suggested, maybe the script should print a warning if there are case differences between the tarball and the directory: "Note: case mismatch between upstream/Pillow2.2.2.tar.gz and build/pkgs/pillow."
comment:13 Changed 6 years ago by
See #18344.
comment:14 followup: ↓ 15 Changed 6 years ago by
I'd keep all (Sageinternal) "package names" (i.e., folders) in build/pkgs/
lowercase.
We should IMHO simply be more flexible w.r.t. upstream tarballs (their names and also their toplevel folders).
The easiest way to achieve this is to record the upstream tarball's name as well; currently we only have packageversion.txt
, which doesn't contain anything but what we consider the version of the package.
In the long run, we wouldn't have to rename or modify upstream tarballs (which was the goal anyway), and could even provide the original URL(s) to obtain them, at least as an alternative, and also for documentation purposes.
(There's more data we could move/replicate from SPKG.txt
into more machinereadable form, finally creating SPKG.txt
from, or augmenting it with data from other metadata files, such as authors, copyright, upstream contact/address for bug reports, etc.)
comment:15 in reply to: ↑ 14 ; followup: ↓ 16 Changed 6 years ago by
Replying to leif:
I'd keep all (Sageinternal) "package names" (i.e., folders) in
build/pkgs/
lowercase.
Yes.
We should IMHO simply be more flexible w.r.t. upstream tarballs (their names and also their toplevel folders).
Yes.
The easiest way to achieve this is to record the upstream tarball's name as well; currently we only have
packageversion.txt
, which doesn't contain anything but what we consider the version of the package.
The tarball name (without the version) is stored in checksums.ini
already.
See the proposal at #18344. I think that the tarball and the directory name should agree except possibly for case differences. I guess we could allow completely different names for the two, but that would make various parts of the system more complicated: sagefixpkgchecksums
, sagespkg
, etc., not to mention making everything more opaque: why should a file foo.tar.gz
be the right tarball for build/pkgs/bar
?
comment:16 in reply to: ↑ 15 Changed 6 years ago by
Replying to jhpalmieri:
The tarball name (without the version) is stored in
checksums.ini
already.
I disregarded M$ Windows files... ;)
comment:17 Changed 6 years ago by
Related:
> I tried to install pyopenssl in order to run sagenb in sage 6.6 with > secure=True. It fails with Error 404, see below. > > Regards, > > CH > > I get: > $ ./sage i pyopenssl > Attempting to download package pyopenssl >>>> Checking online list of optional packages. > [.] >>>> Found pyopenssl0.13.p0 >>>> Trying to download http://www.sagemath.org/spkg/optional/pyopenssl0.13.p0.spkg > .... > IOError: [Errno 404] Not Found: > '//www.sagemath.org/spkg/optional/pyopenssl0.13.p0.spkg' > Error: failed to download package pyopenssl0.13.p0 > > $ LANG=C wget http://www.sagemath.org/spkg/optional/pyopenssl0.13.p0.spkg > 20150503 16:02:09 > http://www.sagemath.org/spkg/optional/pyopenssl0.13.p0.spkg > Resolving www.sagemath.org (www.sagemath.org)... 128.208.178.250 > Connecting to www.sagemath.org (www.sagemath.org)128.208.178.250:80... connected. > HTTP request sent, awaiting response... 404 Not Found > 20150503 16:02:10 ERROR 404: Not Found. Unfortunately setting SAGE_SERVER is borked as well (because the mirrors still have 'spkg/', not 'packages/', which the scripts simply ignore), but you could use a mirror "manually": wget http://mirror.switch.ch/mirror/sagemath/spkg/optional/pyopenssl0.13.p0.spkg ./sage i pyopenssl0.13.p0.spkg # Note the extension
Setting SAGE_SERVER
leads into an endless loop: ;)
SAGE_SERVER="http://sage.scipy.org/sage/" ./sage optional Using Sage Server http://sage.scipy.org/sage/packages HTTP Error 404: Not Found ******************************************************************************** Error contacting http://sage.scipy.org/sage/packages/optional/list. Try using an alternative server. For example, from the bash prompt try typing export SAGE_SERVER=http://sage.scipy.org/sage/ then try again. ********************************************************************************
The above fails for a different reason, but try yourself with some other mirror...
comment:18 Changed 5 years ago by
 Dependencies set to #15642
comment:19 Changed 5 years ago by
 Milestone changed from sage6.4 to sage6.7
comment:20 Changed 5 years ago by
 Dependencies changed from #15642 to #15642, #18407
comment:21 Changed 5 years ago by
 Description modified (diff)
comment:22 Changed 5 years ago by
 Dependencies #15642, #18407 deleted
 Description modified (diff)
 Summary changed from Fix install_package() to install_package() is obsolete
I vote for removing the install_package()
command. Installing packages from within Sage can never work properly anyway:
 for some packages, you need to rebuild the Sage library, which cannot be done in Sage.
 some packages might need a change in environment variables (e.g.
ccache
) and that can only be done ifsageenv
is rerun after installing the package.
comment:23 Changed 5 years ago by
 Branch set to u/jdemeyer/install_package___is_obsolete
comment:24 Changed 5 years ago by
 Commit set to ec6c23a895a72e46b4920ee08a5de73a3d269791
 Milestone changed from sage6.7 to sage6.9
 Status changed from new to needs_review
 Type changed from defect to enhancement
New commits:
ec6c23a  Remove install_package(), document sage i

comment:25 Changed 5 years ago by
 Description modified (diff)
comment:26 Changed 5 years ago by
 Description modified (diff)
comment:27 Changed 5 years ago by
 Description modified (diff)
comment:28 followups: ↓ 29 ↓ 31 ↓ 32 Changed 5 years ago by
Channeling Nathann: regarding changes like sage i cryptominisat && make
, do we need to say something about being in SAGE_ROOT
first?
This is no longer a stopgap, so in misc.package.install_package
, should we be returning some other error message? Maybe NotImplementedError
? I agree that install_package('blah')
won't work much of the time for the reasons you give, so a deprecation isn't the right course of action, but a stopgap implies that the situation is temporary and is going to be fixed. This situation may be temporary, but only in the sense that install_package()
may be completely deleted eventually. Also, with a stopgap, you only get the message the first time you run it. If we raise an error, you will get it every time, which might be better.
I'm not sure if the changes to trac.py
belong here or on a separate ticket. I don't care that much, though.
Also, please make the following change (or something similar):

src/sage/misc/package.py
diff git a/src/sage/misc/package.py b/src/sage/misc/package.py index de4d206..9d727d9 100644
a b components: 19 19 Use the :func:``optional_packages`` command to list all 20 20 optional packages available on the central Sage server. 21 21 22 Actually installing the packages should be done via the Sagecommand22 Actually installing the packages should be done via the command 23 23 line, using the following commands: 24 24 25 25  ``sage i PACKAGE_NAME``  install the given package
Also, the sentences at the top of that file describing Sage packages are outdated and should probably just be replaced with a pointer to the Developer's Guide. If you are willing to do that, too, that would be nice.
comment:29 in reply to: ↑ 28 Changed 5 years ago by
Replying to jhpalmieri:
Channeling Nathann: regarding changes like
sage i cryptominisat && make
, do we need to say something about being inSAGE_ROOT
first?
Good point. I'll replace this by sage i cryptominisat sagelib
which doesn't have the SAGE_ROOT
problem.
comment:30 Changed 5 years ago by
 Commit changed from ec6c23a895a72e46b4920ee08a5de73a3d269791 to cbf315d4d1623a2200937f4556388de5214566ce
Branch pushed to git repo; I updated commit sha1. New commits:
cbf315d  Replace "make" by "sage i sagelib"

comment:31 in reply to: ↑ 28 Changed 5 years ago by
Replying to jhpalmieri:
replaced with a pointer to the Developer's Guide.
I don't think it's technically possible to have an actual link to the Developer's Guide, but I will change the wording.
comment:32 in reply to: ↑ 28 Changed 5 years ago by
Replying to jhpalmieri:
I'm not sure if the changes to
trac.py
belong here or on a separate ticket.
It's really just a deprecation that I added. Pretty innocent.
comment:33 Changed 5 years ago by
 Commit changed from cbf315d4d1623a2200937f4556388de5214566ce to da03be13988e41fc483ee2001654a55d4e45ee95
Branch pushed to git repo; I updated commit sha1. New commits:
da03be1  Replace install_package() by installed_packages()

comment:34 Changed 5 years ago by
John, I think I made all the changes you proposed.
comment:35 followup: ↓ 36 Changed 5 years ago by
Okay, looks good. What do we do about the files
src/doc/en/thematic_tutorials/numerical_sage/installation_linux.rst src/doc/en/thematic_tutorials/numerical_sage/installation_osx.rst src/doc/en/thematic_tutorials/numerical_sage/mpi4py.rst
Maybe I can handle the last one:

src/doc/en/thematic_tutorials/numerical_sage/mpi4py.rst
diff git a/src/doc/en/thematic_tutorials/numerical_sage/mpi4py.rst b/src/doc/en/thematic_tutorials/numerical_sage/mpi4py.rst index 93afc85..ef129b1 100644
a b MPI which stands for message passing interface is a common library 5 5 for parallel programming. There is a package mpi4py that builds on 6 6 the top of mpi, and lets arbitrary python objects be passed between 7 7 different processes. These packages are not part of the default 8 sage install. To install them do 9 10 .. skip 11 12 :: 13 14 sage: optional_packages() 15 16 Find the package name openmpi\* and mpi4py\*and do 17 18 .. skip 8 sage install. To install them, from a shell prompt, run 19 9 20 10 :: 21 11 22 sage: install_package('openmpi*') 23 sage: install_package('mpi4py*') 12 sage i openmpi mpi4py 24 13 25 14 Note that openmpi takes a while to compile (1520 minutes or so). 26 15 Openmpi can be run on a cluster, however this requires some set up
But the other two files refer to packages which are now archived (vtk
) or are archived and almost certainly a bad idea to install (python2.5.1framework
). We can change each instance of install_package
to sage i ...
anyway; I'm not sure what else to suggest.
comment:36 in reply to: ↑ 35 Changed 5 years ago by
Replying to jhpalmieri:
Okay, looks good. What do we do about the files
src/doc/en/thematic_tutorials/numerical_sage/installation_linux.rst src/doc/en/thematic_tutorials/numerical_sage/installation_osx.rst src/doc/en/thematic_tutorials/numerical_sage/mpi4py.rst
I know.... it's difficult. Since I see no sensible way to fix those files, I just left them alone.
The problem is that those files have a laundry list of oldstyle packages which are no longer supported. I see no sensible way of updating them, there is no point to replace install_package("foo")
by ./sage i foo
if the latter doesn't actually work.
comment:37 followup: ↓ 38 Changed 5 years ago by
How about this:

src/doc/en/thematic_tutorials/numerical_sage/installation_linux.rst
diff git a/src/doc/en/thematic_tutorials/numerical_sage/installation_linux.rst b/src/doc/en/thematic_tutorials/numerical_sage/installation_linux.rst index 16df05f..01ee37f 100644
a b 1 1 Installing Visualization Tools on Linux 2 2 ======================================= 3 3 4 .. warning:: 5 6 The following instructions are outdated (2015Sep). 7 4 8 This section assumes you are running linux. You may need 5 9 administrator rights to complete this section. First try 6 10 
src/doc/en/thematic_tutorials/numerical_sage/installation_osx.rst
diff git a/src/doc/en/thematic_tutorials/numerical_sage/installation_osx.rst b/src/doc/en/thematic_tutorials/numerical_sage/installation_osx.rst index 05c27e8..2c6cec8 100644
a b 1 1 Installing Visualization Software on OS X 2 2 ========================================= 3 3 4 5 .. warning:: 6 7 The following instructions are outdated (2015Sep). 8 4 9 The first thing we need to do is rebuild Python to use OSX's 5 10 frameworks, so that it can create graphical windows. To do this 6 11 first from the terminal do 
src/doc/en/thematic_tutorials/numerical_sage/mpi4py.rst
diff git a/src/doc/en/thematic_tutorials/numerical_sage/mpi4py.rst b/src/doc/en/thematic_tutorials/numerical_sage/mpi4py.rst index 93afc85..98d0edb 100644
a b 1 1 mpi4py 2 2 ====== 3 3 4 5 .. warning:: 6 7 The following instructions are outdated (2015Sep). 8 4 9 MPI which stands for message passing interface is a common library 5 10 for parallel programming. There is a package mpi4py that builds on 6 11 the top of mpi, and lets arbitrary python objects be passed between
We can also add a note to contact sagesupport for help. What do you think?
comment:38 in reply to: ↑ 37 Changed 5 years ago by
Replying to jhpalmieri:
We can also add a note to contact sagesupport for help. What do you think?
In order to move forward with this ticket, I made it a separate ticket: #19198
comment:39 Changed 5 years ago by
 Reviewers set to John Palmieri
 Status changed from needs_review to positive_review
comment:40 Changed 5 years ago by
 Branch changed from u/jdemeyer/install_package___is_obsolete to da03be13988e41fc483ee2001654a55d4e45ee95
 Resolution set to fixed
 Status changed from positive_review to closed
It also looks like
sage standard
is broken: it is using old package information. For example, it thinks that the current version of Sage is 5.13. Also,pillow
is now a standard package, replacingpil
, butpil
shows up.I'm not sure where to get a list of all of the current standard packages, but once we had that, we might be able to get the version numbers from
http://www.sagemath.org/packages/upstream/
.