Use mirrors for tarballs

[vbraun]

This ticket rewrites the sage-download-file script in pure Python. It is, first of all, a drop-in replacement of the current script. But it is now also aware of our tarballs, so you can just write e.g.

sage-download-file sympy-0.7.4.tar.gz

and it will download that tarball from the fastest mirror and verify its checksum.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​3967968	Merge tag '6.1.beta4' into 'u/vbraun/use_mirrors_for_tarballs'
​dd4abcb	py3 fix, also try testing packages

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​b1ef0b6

prefer SAGE_DISTFILES for upstream directory

[ohanar]

I would replace the regex in _parse_name with something like

(?P<base>(?:[^-]|-(?!\d))*)-(?P<version>.*)\.(?P<ext>tar.*|zip|tgz)

(We may as well support more general package names, even if our restrictive naming convention doesn't allow it at the moment.)

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​cd155cf

more general tarball regex

[ohanar]

I added a few extra commits to resolve a couple of small bugs I noticed. Pending someone signing off on those, and testing on various versions of python, this looks good to me.

---

New commits:

​f647251	remove commented out old code
​905df09	rework environment stuff a bit
​991de26	don't actually write unsorted mirror list to disk

[jdemeyer]

Why remove the possibility to use the system curl or wget programs? That seems like an unneeded regression to me.

[vbraun]

We only need one way to download files. There is no point in having untested (hence broken) code laying around "just in case". There is no use case, just clutter laying around.

[jdemeyer]

Replying to vbraun:

There is no use case

The use case is building from git on systems without Python installed.

[vbraun]

Per the discussion on sage-devel, Python is a requirement. Sooner or later Sage won't build without Python with or without this ticket.

[jdemeyer]

Replying to vbraun:

Per the discussion on sage-devel, Python is a requirement.

That would require at least a change to the documentation.

[jdemeyer]

Replying to vbraun:

Sooner or later Sage won't build without Python with or without this ticket.

Sure, but why break building on systems without Python on this ticket? It is absolutely trivial to keep supporting without Python for now (just keep the shell part of the script). It seems you're breaking stuff for no reason.

[vbraun]

So would it make you feel better if we wait with this ticket and merge it only after another change broke building without Python?

[jdemeyer]

Replying to vbraun:

So would it make you feel better if we wait with this ticket and merge it only after another change broke building without Python?

Essentially yes. I agree with breaking building without Python, but not breaking just for the sake of it.

[novoselt]

I would say this is not breaking building without Python for the sake of breaking, but for the sake of moving the build system to a language that all Sage developers understand (and therefore can make further improvements).

[jdemeyer]

Currently in Sage, we have a Python script to download stuff with a shell-script wrapper to handle the case where Python in Sage is not yet installed. I would rather keep the shell-script wrapper and only modify the Python script.

[vbraun]

Why have a shell script here-document a fortran program which runs a python script? Just so that we can build Sage without Python for a few more months? The decision is made, Python is a built-time requirement.

[jdemeyer]

Replying to vbraun:

The decision is made

I don't recall that discussion...

Let me repeat what I said above:

Why break building on systems without Python on this ticket? It is absolutely trivial to keep support for building without Python for now (just keep the shell part of the script). It seems you're breaking stuff for no reason. I don't mind breaking building without Python if there is a good reason, but not breaking just for the sake of it.

[vbraun]

You are going to say the same thing all the time, why rely on a Python script here when you could just as well have an additional shell script to provide a fall back. The answer is: fewer moving parts = less bugs.

[tmonteil]

This is not about python as a requirement nor the curl and wget fallbacks, but i see the benefit of being able to set a URL_GRABBER variable that calls some fetcher with more robust options, as discussed on ​this sage-devel thread.

[vbraun]

We must use the mirrors very soon, the centrally-hosted packages will go away.

The downloading script still needs to be called from sage-spkg, when that is finished we must merge this ticket asap.

---

New commits:

​74ac922

Merge 6.7.beta3 into #15642

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​d9c0318	Merge 6.7.beta4 into #15642
​a4f9892	Ignore the single ftp-only mirror for now
​33bf83f	Use the new sage-download-file script to download all packages

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a7afa48

Add a SAGE_MIRROR environment variable

[vbraun]

The buildbot will set SAGE_MIRROR to its own so it doesn't have to wait for the mirrors to catch up.

[leif]

"Usage" doesn't match implementation: --fastest-mirror vs. --print-fastest-mirror.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​9a06850

fix usage doc

[vbraun]

Thanks, fixed

[leif]

Not entirely clear to me why you use .startswith() though... ;-)

(Ok, it does support --print-fastest-mirror-and-drop-me-a-beer, but I'm still waiting...)

[leif]

Could we please suppress the tracebacks on errors like 404?

[vbraun]

As you said, using startswith() is a cheap emulation of argparse so --quiet=... works. The point is to be compatible with Python 2.6, of course, so we can't use argparse.

[leif]

Mirror layout fixes, part I:

src/bin/sage-spkg

@@ -367 +367 @@
                 echo ">>> Checking online list of $repo packages."
                 # File inside DOT_SAGE should be writable
                 repolist="${DOT_SAGE}/${repo}.list"
-                sage-download-file "$MIRROR/$repo/list" $repolist
+                sage-download-file "$MIRROR/spkg/$repo/list" $repolist
                 if [ $? -ne 0 ]; then
-                    echo >&2 "Error: failed to download $MIRROR/$repo/list, aborting"
+                    echo >&2 "Error: failed to download $MIRROR/spkg/$repo/list, aborting"
                     rm -f $repolist
                     exit 1
                 fi
@@ -386 +386 @@
 
                     # If INFO is set, try downloading only the .txt file
                     if [ $INFO -eq 1 ]; then
-                        PKG_URL="$MIRROR/$repo/$pkg.txt"
+                        PKG_URL="$MIRROR/spkg/$repo/$pkg.txt"
                         sage-download-file "$PKG_URL" && exit 0
                         # If the download failed (for whatever reason),
                         # fall through and use the .spkg file.
@@ -416 +416 @@
                             esac
                         fi
                     fi
-                    PKG_URL="$MIRROR/$repo/$pkg.spkg"
+                    PKG_URL="$MIRROR/spkg/$repo/$pkg.spkg"
                     break
                 fi
             done

(Try e.g. ./sage -i foobar.)

[leif]

... we'd better use a variable like MIRROR_PACKAGE_FOLDER or whatever there.

[vbraun]

how about you just push that to trac, I'm not going to type that in for you

[leif]

There are still issues we have to fix w.r.t. MIRROR vs. SAGE_SERVER.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d9e66ac

no backtrace on IOError

[vbraun]

SAGE_SERVER is used elsewhere, its where the docs are hosted etc.

[leif]

Replying to vbraun:

how about you just push that to trac, I'm not going to type that in for you

That's not something I expect you to merge; it just shows errors in the current code. As said, we should IMHO use a variable for that. (I'm not doing your work either... ;-) )

[leif]

Replying to vbraun:

SAGE_SERVER is used elsewhere, its where the docs are hosted etc.

It's still in an error message in sage-spkg, although MIRROR was used.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​30b1647

fix mirror uri

[vbraun]

Fixed

[leif]

Replying to vbraun:

Fixed

Ok, but there's an extra slash (MIRROR already has a trailing one).

[vbraun]

the official mirror list may or may not have a slash at the end

[leif]

FTP doesn't seem to work (hangs after receiving the file).

Try for example

./sage --sh -c 'sage-download-file ftp://ftp.fu-berlin.de/unix/misc/sage/spkg/optional/list'

[vbraun]

Correct, FTP is not allowed

[leif]

Replying to vbraun:

the official mirror list may or may not have a slash at the end

A matter of what your script outputs...

So, what about SAGE_SERVER to specify a mirror? While $SOMEBODY partially broke that anyway by hardcoding the incompatible packages/ subfolder, we should IMHO still (or again) support that. (At least sage-list-packages also needs to get fixed, preferably on a separate ticket I'd say.)

[leif]

Replying to vbraun:

Correct, FTP is not allowed

The single mandatory argument can be a http:// or ftp:// url or a
tarball filename.

    if url.startswith('http://') or url.startswith('https://') or url.startswith('ftp://'):
        http_download(url, destination, progress=progress, ignore_errors=True)

[vbraun]

I don't really like the SAGE_SERVER name (could refer anything, sage cell server), but historically that was where the official mirror list was and thats what I wanted to keep

[vbraun]

Also, IMHO we should just phase out old-style spkgs. As long as we have to drag along that technical debt its very hard to clean up scripts.

[leif]

Replying to vbraun:

I don't really like the SAGE_SERVER name (could refer anything, sage cell server), but historically that was where the official mirror list was and thats what I wanted to keep

? Before the breakage, setting SAGE_SERVER to e.g. http://mirror.switch.ch/mirror/sagemath/ worked, for example with sage --optional and -- long time ago -- with sage --upgrade.

Deprecate it, later use SAGE_MIRROR for package download purposes?

[leif]

Replying to vbraun:

Also, IMHO we should just phase out old-style spkgs.

I'm all against that. The worst decision was to hardcode package metadata (solely!) into specific Sage releases, while that data doesn't even show whether a package is optional, experimental, or whatever. (It's pretty easy to create "old-style" packages from that though... ;-) )

In my opinion the main reason we should (still) support real spkgs is that anybody can easily create and offer such self-contained packages to others, and installing and using them doesn't artificially depend on the Sage version. (Whether they'll work of course depends on a couple of things.)

They're also often much more convenient for testing.

As long as we have to drag along that technical debt its very hard to clean up scripts.

Perhaps with the exception of left-over old-style standard packages, I don't see a problem.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​27b71e5	remove ftp from docs for now
​a103ab5	Use SAGE_SERVER instead of SAGE_MIRROR

[vbraun]

Replying to leif:

Whether they'll work of course depends on a couple of things.

Even if they work in principle you can't do that much with them as there is no dependency management. Its just an all-out shitty solution for everyone.

[vbraun]

SAGE_SERVER must of course then not be set by default...

[leif]

Does any mirror have packages/? Will they at some point?

Otherwise s|SAGE_SERVER/packages/|SAGE_SERVER/spkg/| in the docs.

[vbraun]

AFAIK there will only ever be /spkg/ on the mirrors

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a259f27

rename SAGE_SERVER/packages to SAGE_SERVER/spkg

[leif]

Replying to leif:

Does any mirror have packages/? Will they at some point?

Otherwise s|SAGE_SERVER/packages/|SAGE_SERVER/spkg/| in the docs.

Should we fix this here, since we now touch sage-list-packages anyway?

SAGE_SERVER="http://mirror.switch.ch/mirror/sagemath/" ./sage --optional
Using Sage Server http://mirror.switch.ch/mirror/sagemath/packages
HTTP Error 404: Not Found



********************************************************************************



Error contacting http://mirror.switch.ch/mirror/sagemath/packages/optional/list. Try using an alternative server.
For example, from the bash prompt try typing

   export SAGE_SERVER=http://sage.scipy.org/sage/

then try again.



********************************************************************************

(​http://sage.scipy.org/sage/ is btw. a joke...)

[leif]

Concurrent posting (luckily not pushing)...

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d79364b

fix sage-list-packages

[vbraun]

The whole listing of packages is also made hideously complicated by mashing old and new style packages together...

[leif]

I thought of just adding a further try: ... except HTTPError: ..., in case sagemath.org/packages/ remains for a while. (IIRC it had both for a while.)

PKG_SERVER is superfluous anyway; I'd simply use os.environ['SAGE_SERVER'] there.

[leif]

Replying to vbraun:

The whole listing of packages is also made hideously complicated by mashing old and new style packages together...

As I said, in principle only the old standard packages disturb a bit; otherwise it's just a matter of syncing.

I was actually working on that a few days ago (#16759).

[leif]

Quoting jdemeyer:

Replying to jhpalmieri:

It also looks like sage --standard is broken: it is using old package information.

I would say that sage --standard is doing the right thing, but somehow http://www.sagemath.org/packages/standard/ is no longer updated. The right solution is updating the website script to take into account the new git-style packages (which is certainly a different issue than the one this ticket is about).

[schilly]

Since I'm CCed: There is currently a move of the whole infrastructure going on. Before that I tried to keep everything compatible with the original system from a long time ago. That's no longer possible. Long story short: the website and the binary-files need to be untangled and that means there are no longer chains of symlink-aliases. Hence, don't use the /mirror subdirectory of the website and I hope the essence of this ticket is to get rid of this old dependence. The only mirror related aspect is the mirror_list file for the base URLs.

[leif]

Replying to schilly:

Since I'm CCed: There is currently a move of the whole infrastructure going on. Before that I tried to keep everything compatible with the original system from a long time ago. That's no longer possible. Long story short: the website and the binary-files need to be untangled and that means there are no longer chains of symlink-aliases. Hence, don't use the /mirror subdirectory of the website and I hope the essence of this ticket is to get rid of this old dependence. The only mirror related aspect is the mirror_list file for the base URLs.

Well, the Sage scripts never used the mirror/ folder. The problematic part here is the structure below the base URL; spkg/ was changed to packages on sagemath.org, but the mirrors all have the classical structure (plus upstream/ below spkg/).

The new thing on this ticket is also that we now use http://www.sagemath.org/mirror_list (currently hardcoded) again, which is periodically generated by a script on the server (or at least used to be).

[leif]

Harald, also note Jeroen's comment I quoted above.

Meanwhile, it's not just standard packages, but also optional packages, of which (only) some now have the "new style", i.e., the data of newer ones is hardcoded into the Sage releases, while obsolete "old-style" packages remain in spkg/optional/list (and spkg/optional/).

./sage --optional shows some examples of optional packages which became standard:

Optional packages:

...
git-1.7.12.2.p0 ........................ installed version: 2.3.0
...
pyzmq-2.1.11.p1 ........................ installed version: 14.3.0
...
zeromq-3.2.0.p0 ........................ installed version: 4.0.5

There are other still optional packages where sage -i ... would install newer versions than listed by sage --optional because the "local metadata" ($SAGE_ROOT/build/pkgs/<package>/package-version.txt) differs from what spkg/optional/list gives. (Same issue as with our current standard packages.)

A more funny one is TOPCOM:

./sage -i TOPCOM installs the "legacy" spkg, while ./sage -i topcom installs the "new-style" spkg. (The package versions are currently still the same though, modulo a .p0 for the "new-style" one.)

[vbraun]

We definitely should not check sagemath.org/packages any more, I've already changed the release management scripts to no longer put the tarballs there.

[leif]

Replying to vbraun:

We definitely should not check sagemath.org/packages any more, I've already changed the release management scripts to no longer put the tarballs there.

Ok, sagemath.org/spkg/* gave a 404 for a while, but apparently works again.

[leif]

Note for other reviewers: So far I'm ok with the changes/corrections made, but haven't yet tested thoroughly enough (I think).

While I don't like completely changing the previously generic character of sage-download-file (nor removing the "shell fallback" with e.g. wget or curl if Python isn't available or functional), I strongly doubt we could convince the author or the RM to change that back to how it was.

FTP support (probably with Python3) and e.g. usage of a static mirror list (perhaps to be shipped with Sage) is something for follow-up tickets; the broken behaviour of sage-list-packages is (or will be) dealt with at #16759, which depends on this ticket.

W.r.t. the threat of losing sagemath.org for downloads: ​According to William, "soon" is not that soon.

[vbraun]

I don't mind if you rewrite the functionality in bash fwiw. I would consider it utterly retarded and backward, but as long as it works and you are committed to maintaining that solution then that would be fine with me.

[schilly]

Replying to leif:

Ok, sagemath.org/spkg/* gave a 404 for a while, but apparently works again.

Uhm, to reduce that confusion a little bit: The source of this inconsistency is that right now most of the old infrastructure is still running, including the currently "old" website. I would suggest to wait until the "new" website is operational. Then, any bugs in scripts relying on the website will be obvious (and fixes to the new website visible immediately).

The reason why this is a bit slow is, because there are many links assuming certain subdirectories (for documentation and e.g. /mirror/) and my time is limited. I could report back once the DNS entry has switched to the new page.

PS: "new" just means, that there are minor changes like an additional mirror, publications etc. The current "old" website is an older fork of the website with some other patches. Hence this diverged and there is no need to support the old one any more.

[leif]

Replying to schilly:

I could report back once the DNS entry has switched to the new page.

ETA Hausnummer?

PS: "new" just means, that there are minor changes like an additional mirror, publications etc. The current "old" website is an older fork of the website with some other patches. Hence this diverged and there is no need to support the old one any more.

Could we already test the new one (using its unofficial address) w.r.t. download stuff / package (and mirror) lists?

[vbraun]

The attached branch will only use www.sagemath.org to download the mirror list, so once we have merged it there are a lot fewer things that could break.

[schilly]

Replying to leif:

ETA Hausnummer?

Friday. It's tedious stupid work, so maybe I'm already whining more about it than it will actually take to work on.

Could we already test the new one (using its unofficial address) w.r.t. download stuff / package (and mirror) lists?

No, Github resolves the inofficial URL to the official one and I don't want to mess with this around even more.

[schilly]

Replying to vbraun:

The attached branch will only use www.sagemath.org to download the mirror list ...

Yes, it's just one file, and as already mentioned: the full list could easily be hardcoded as a backup into Sage to remove that SPOF. Ticket for later please :-)

[leif]

Replying to vbraun:

once we have merged it there are a lot fewer things that could break.

Provided the mirrors can still sync...

In principle we could already include a snapshot of mirror_list into the git tree (in $SAGE_ROOT/? in more human-readable form?). The dynamic one should rather be located in $SAGE_ROOT/local/var/{lib,tmp}/sage/, say, not $SAGE_DIST_FILES/ (defaults to $SAGE_ROOT/upstream/).

[vbraun]

Nothing else needs the mirror list so why put it in a separate file? In any case, this question should be dealt with in a separate ticket.

[leif]

Replying to vbraun:

Nothing else needs the mirror list

Human users perhaps?

so why put it in a separate file?

You're right, all package metadata should be moved into sage-spkg as well.

[vbraun]

There are many packages but only one mirror list. Your argument is invalid.

Not to mention the question of what file format, custom = crap, json = not exactly human-readable, yaml = not necessarily available in the System python, is it supposed to be readable from a shell script, ...

[leif]

Plain text, one URL per line, preferably in the top-level folder. What's the problem?

Any human and any kind of script can easily read such. (If you want to add further data such as organization, location etc., just put that after each URL, seperated by whitespace, but I wouldn't even do that. We don't have to process that. Blank lines and comment lines starting with # at the first position would IMHO be ok, too, but that's not really needed. Order of the mirrors doesn't matter for our purposes either, although we could keep them sorted by continent, say.)

I thought we had postponed that to a follow-up ticket.

[vbraun]

Replying to leif:

I thought we had postponed that to a follow-up ticket.

We have, but you keep bringing it up.

[leif]

Replying to vbraun:

you keep bringing it up.

Not me, Harald did.

[schilly]

Replying to leif:

Replying to schilly:

I could report back once the DNS entry has switched to the new page.

ETA Hausnummer?

Yay! 2 days ahead of schedule! I fixed all the obviously broken links and there are just two minor details left.

[leif]

Replying to schilly:

Replying to leif:

Replying to schilly:

I could report back once the DNS entry has switched to the new page.

ETA Hausnummer?

Yay! 2 days ahead of schedule! I fixed all the obviously broken links and there are just two minor details left.

sagemath.org no longer serves packages (especially not the upstream tarballs); is that intentional, or will it again? (Cf. ​sage-release.)