Opened 9 years ago

Closed 9 years ago

#13373 closed enhancement (fixed)

Make sage -i http://.../foo.spkg work immediately

Reported by: jdemeyer Owned by: leif
Priority: critical Milestone: sage-5.3
Component: scripts Keywords:
Cc: Merged in: sage-5.3.rc0
Authors: Leif Leonhardy, Jeroen Demeyer Reviewers: John Palmieri
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by jdemeyer)

The following works right after extracting the Sage source tarball:

./sage -i /path/to/foo.spkg

But this doesn't work for remote packages:

./sage -i http://example.com/foo.spkg

or

./sage -i foo

Using either wget, curl or system Python, we should also make this work.

Apply:

  1. 13373_sage-i.patch to SAGE_ROOT.
  2. 13373_remove_scripts.patch to SAGE_LOCAL/bin.
  3. 13373_doc.patch to the Sage library.

Attachments (4)

latest-online-spkg.sh (5.7 KB) - added by leif 9 years ago.
Tentative / preliminary "stand-alone" shell script to replace sage-latest-online-package; doesn't use/support Python (even if available) [yet].
13373_remove_scripts.patch (6.4 KB) - added by jdemeyer 9 years ago.
13373_doc.patch (1.4 KB) - added by jdemeyer 9 years ago.
13373_sage-i.patch (6.4 KB) - added by jdemeyer 9 years ago.

Download all attachments as: .zip

Change History (27)

comment:1 Changed 9 years ago by leif

So we need some sage-latest-online-package and sage-download_package [replacement / wrapper] scripts in spkg/bin/?

(In case there's some capable system-wide Python in the user's PATH, the original scripts [copied to spkg/bin/] should already work, right?)

To not have different files with the same name (in local/bin/ and spkg/bin/), we might rename the current [Python] versions to *.py and make sage-{latest-online-package,download_package} plain shell scripts (then moved to spkg/bin/), which just call the original Python implementation if appropriate (Sage's or some other Python available), or otherwise mimic the behavior using wget or curl.

We could probably drop the Python implementations completely in favor of Bash versions (although in theory neither wget nor curl [nor perl], but some python may be available), but as the features of these scripts are IMHO unlikely to change, we can keep parallel implementations without hassle as well.

Changed 9 years ago by leif

Tentative / preliminary "stand-alone" shell script to replace sage-latest-online-package; doesn't use/support Python (even if available) [yet].

comment:2 Changed 9 years ago by leif

Ok, couldn't resist to write some shell script to replace sage-latest-online-package, using curl, wget or $URL_GRABBER instead of Python (which the shell script currently doesn't support at all), and not requiring any "Sage environment".

Do what you want with it... ;-)

Replacing sage-download_package like that should be even easier, I just haven't [yet] done so...

comment:3 Changed 9 years ago by leif

P.S.: Also hardly tested (especially not using it from sage-spkg / sage -i ...; would need it to be renamed anyway).

comment:4 follow-up: Changed 9 years ago by leif

P.P.S.:

Since the scripts are / will be very similar, I'd write a single script which operates depending on basename $0, i.e., on the name it is invoked with, sage-latest-online-package or sage-download_package (one being just a link to the other). Don't know whether Mercurial would like hard links, so one could be a symbolic link to the other.

comment:5 in reply to: ↑ 4 Changed 9 years ago by jdemeyer

Replying to leif:

Since the scripts are / will be very similar, I'd write a single script which operates depending on basename $0, i.e., on the name it is invoked with, sage-latest-online-package or sage-download_package (one being just a link to the other). Don't know whether Mercurial would like hard links, so one could be a symbolic link to the other.

Or better: why not create a separate script download_file.sh which simply downloads a given URL and nothing more and which is called by the other scripts?

comment:6 Changed 9 years ago by jdemeyer

Concerning verbosity of these scripts: The current Python scripts also show something when downloading, so I wouldn't mind having wget or curl work in its full verbosity.

comment:7 Changed 9 years ago by jdemeyer

  • Authors set to Leif Leonhardy, Jeroen Demeyer
  • Description modified (diff)

I expanded on Leif's idea and made a self-contained patch. This no longer uses neither sage-latest-online-package nor sage-download_package.

Changed 9 years ago by jdemeyer

Changed 9 years ago by jdemeyer

comment:8 Changed 9 years ago by jdemeyer

  • Description modified (diff)

comment:9 Changed 9 years ago by jdemeyer

  • Status changed from new to needs_review

comment:10 Changed 9 years ago by jhpalmieri

It mostly seems to work, although I would like other people to review this as well. One problem: if curl, wget, and python are not available, running sage -i without a full URL won't print the relevant error message, because stderr is redirected to /dev/null when sage-download-file is called in sage-spkg:

pkg=`sage-download-file 2>/dev/null "$URL_BASE/$repo/list" |grep "^${PKG_NAME}\(\$\|-\)"`

comment:11 Changed 9 years ago by jdemeyer

I was already doubting whether to use 2>/dev/null there, so I removed that.

comment:12 Changed 9 years ago by jdemeyer

  • Status changed from needs_review to needs_work

This doesn't work with Solaris grep :-(

comment:13 Changed 9 years ago by jhpalmieri

Now if the programs are not available, sage-download-file is executed four times (for repo in optional experimental standard archive; do ...), so the error message prints four times.

comment:14 Changed 9 years ago by jdemeyer

Portability problems fixed by using slightly contrived sed commands instead of grep.

comment:15 Changed 9 years ago by jdemeyer

  • Status changed from needs_work to needs_review

New version, fails immediately if downloading one of the package lists failed.

comment:16 Changed 9 years ago by jhpalmieri

I was wondering if you could use egrep instead of grep. sed is fine, though...

Why do you use >&2 on line 280 in echo >&2 "Found $PKG_NAME"?

I think it might be clearer with some extra output, say like this:

  • spkg/bin/sage-spkg

    diff --git a/spkg/bin/sage-spkg b/spkg/bin/sage-spkg
    a b if [ ! -f "$PKG_SRC" ]; then 
    258258        URL_BASE="${SAGE_SERVER}packages"
    259259        for repo in optional experimental standard archive; do
    260260            # Download the list of packages.
     261            echo
     262            echo "Checking list of $repo packages."
    261263            repolist="${repo}.list"
    262264            sage-download-file "$URL_BASE/$repo/list" >$repolist
    263265            if [ $? -ne 0 ]; then
    if [ ! -f "$PKG_SRC" ]; then 
    282284            echo >&2 "Error: Could not find a package matching $PKG_NAME in $URL_BASE"
    283285            exit 1
    284286        fi
    285         echo >&2 "Found $PKG_NAME"
     287        echo
     288        echo "Found $PKG_NAME."
    286289    fi
    287290
    288291    PKG_SRC="`pwd`/$PKG_NAME.spkg"
     292    echo "Downloading $PKG_NAME.spkg."
    289293    sage-download-file "$PKG_URL" >"$PKG_SRC"
    290294    if [ $? -ne 0 ]; then
    291295        # Delete failed download

Changed 9 years ago by jdemeyer

comment:17 Changed 9 years ago by jdemeyer

Some further cosmetic changes.

Also one important functional change: first download to a temporary file, such that we don't get a corrupted download.

comment:18 Changed 9 years ago by jdemeyer

I am also using curl by default instead of wget because I like the output while downloading better.

comment:19 Changed 9 years ago by jhpalmieri

I just ran into a tangentially related problem: if you unpack the Sage tarball and run ./sage -i openssl..... On a system like mark on skynet (Solaris), the system version of patch is stupid, so it can't apply the patches in the spkg, and the build fails. Is it worth trying to address this somehow (on another ticket, not here)? Or should we not worry about it? I'm inclined to not worry about it, personally.

comment:20 Changed 9 years ago by jhpalmieri

With Sage's Python, hitting ctrl-c not only interrupts the download but also causes the file to be deleted: the if block in the following gets executed:

    sage-download-file "$PKG_URL" >"$PKG_TMP"
    if [ $? -ne 0 ]; then
        # Delete failed download
        rm -f "$PKG_TMP"
        echo >&2 "Error: failed to download package $PKG_NAME"
        exit 1
    fi

But with curl, and I think wget, hitting ctrl-c dumps me out of the script entirely, so I still end up with the temporary file. What do you think about something like this:

  • spkg/bin/sage-spkg

    diff --git a/spkg/bin/sage-spkg b/spkg/bin/sage-spkg
    a b if [ ! -f "$PKG_SRC" ]; then 
    284284    # corrupted .spkg file).
    285285    PKG_TMP="$PKG_NAME.tmp"
    286286    echo ">>> Downloading $PKG_NAME.spkg."
     287    trap 'echo' INT
    287288    sage-download-file "$PKG_URL" >"$PKG_TMP"
    288289    if [ $? -ne 0 ]; then
    289290        # Delete failed download
    if [ ! -f "$PKG_SRC" ]; then 
    291292        echo >&2 "Error: failed to download package $PKG_NAME"
    292293        exit 1
    293294    fi
     295    trap - INT
    294296
    295297    PKG_SRC="`pwd`/$PKG_NAME.spkg"
    296298    mv -f "$PKG_TMP" "$PKG_SRC"

(This seems to work. Is it the right way to catch ctrl-c?)

comment:21 Changed 9 years ago by jdemeyer

Personally, I wouldn't worry about the non-deleting of that temporary file and just leave things as they are. That temporary file will be overwritten anyway if the user tries to download that package again.

Concerning patch: one can always do

./sage -i patch
./sage -i openssl

That should work, so I wouldn't worry too much either.

comment:22 Changed 9 years ago by jhpalmieri

  • Reviewers set to John Palmieri
  • Status changed from needs_review to positive_review

Okay, looks good. Works on linux, OS X 10.7, Solaris, OpenSolaris.

comment:23 Changed 9 years ago by jdemeyer

  • Merged in set to sage-5.3.rc0
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.