Opened 6 years ago

Closed 4 years ago

#15642 closed enhancement (fixed)

Use mirrors for tarballs

Reported by: vbraun Owned by:
Priority: blocker Milestone: sage-6.7
Component: build Keywords:
Cc: ohanar, tmonteil, schilly Merged in:
Authors: Volker Braun Reviewers: Volker Braun
Report Upstream: N/A Work issues:
Branch: d79364b (Commits) Commit: d79364b9fc5131199eaed14340582715f2f937fb
Dependencies: Stopgaps:

Description (last modified by vbraun)

This ticket rewrites the sage-download-file script in pure Python. It is, first of all, a drop-in replacement of the current script. But it is now also aware of our tarballs, so you can just write e.g.

sage-download-file sympy-0.7.4.tar.gz

and it will download that tarball from the fastest mirror and verify its checksum.

Change History (95)

comment:1 Changed 6 years ago by vbraun

  • Branch set to u/vbraun/use_mirrors_for_tarballs

comment:2 Changed 6 years ago by git

  • Commit set to dd4abcb7757c17e5a7bae9cf347c9d34e154503d

Branch pushed to git repo; I updated commit sha1. New commits:

3967968Merge tag '6.1.beta4' into 'u/vbraun/use_mirrors_for_tarballs'
dd4abcbpy3 fix, also try testing packages

comment:3 Changed 6 years ago by vbraun

  • Component changed from PLEASE CHANGE to build
  • Status changed from new to needs_review
  • Type changed from PLEASE CHANGE to enhancement

comment:4 Changed 6 years ago by vbraun

  • Cc ohanar added
  • Description modified (diff)

comment:5 Changed 6 years ago by git

  • Commit changed from dd4abcb7757c17e5a7bae9cf347c9d34e154503d to b1ef0b663cff6fdbcb2ae9cefad7d72c14605b49

Branch pushed to git repo; I updated commit sha1. New commits:

b1ef0b6prefer SAGE_DISTFILES for upstream directory

comment:6 Changed 6 years ago by ohanar

I would replace the regex in _parse_name with something like

(?P<base>(?:[^-]|-(?!\d))*)-(?P<version>.*)\.(?P<ext>tar.*|zip|tgz)

(We may as well support more general package names, even if our restrictive naming convention doesn't allow it at the moment.)

comment:7 Changed 6 years ago by git

  • Commit changed from b1ef0b663cff6fdbcb2ae9cefad7d72c14605b49 to cd155cf944bcdede7c6489fc489511703c10c8ad

Branch pushed to git repo; I updated commit sha1. New commits:

cd155cfmore general tarball regex

comment:8 Changed 6 years ago by ohanar

  • Authors set to Volker Braun
  • Branch changed from u/vbraun/use_mirrors_for_tarballs to u/ohanar/use_mirrors_for_tarballs
  • Commit changed from cd155cf944bcdede7c6489fc489511703c10c8ad to 991de268a9fd9bc0ff4bd94295d8b6cfbe78015e

I added a few extra commits to resolve a couple of small bugs I noticed. Pending someone signing off on those, and testing on various versions of python, this looks good to me.


New commits:

f647251remove commented out old code
905df09rework environment stuff a bit
991de26don't actually write unsorted mirror list to disk

comment:9 Changed 6 years ago by vbraun_spam

  • Milestone changed from sage-6.1 to sage-6.2

comment:10 Changed 6 years ago by jdemeyer

  • Status changed from needs_review to needs_info

Why remove the possibility to use the system curl or wget programs? That seems like an unneeded regression to me.

comment:11 follow-up: Changed 6 years ago by vbraun

We only need one way to download files. There is no point in having untested (hence broken) code laying around "just in case". There is no use case, just clutter laying around.

comment:12 in reply to: ↑ 11 Changed 6 years ago by jdemeyer

Replying to vbraun:

There is no use case

The use case is building from git on systems without Python installed.

comment:13 follow-ups: Changed 6 years ago by vbraun

Per the discussion on sage-devel, Python is a requirement. Sooner or later Sage won't build without Python with or without this ticket.

comment:14 in reply to: ↑ 13 Changed 6 years ago by jdemeyer

Replying to vbraun:

Per the discussion on sage-devel, Python is a requirement.

That would require at least a change to the documentation.

comment:15 in reply to: ↑ 13 Changed 6 years ago by jdemeyer

Replying to vbraun:

Sooner or later Sage won't build without Python with or without this ticket.

Sure, but why break building on systems without Python on this ticket? It is absolutely trivial to keep supporting without Python for now (just keep the shell part of the script). It seems you're breaking stuff for no reason.

comment:16 follow-up: Changed 6 years ago by vbraun

So would it make you feel better if we wait with this ticket and merge it only after another change broke building without Python?

comment:17 in reply to: ↑ 16 Changed 6 years ago by jdemeyer

Replying to vbraun:

So would it make you feel better if we wait with this ticket and merge it only after another change broke building without Python?

Essentially yes. I agree with breaking building without Python, but not breaking just for the sake of it.

comment:18 Changed 5 years ago by vbraun_spam

  • Milestone changed from sage-6.2 to sage-6.3

comment:19 Changed 5 years ago by vbraun_spam

  • Milestone changed from sage-6.3 to sage-6.4

comment:20 Changed 5 years ago by tmonteil

  • Cc tmonteil added

comment:21 Changed 5 years ago by novoselt

I would say this is not breaking building without Python for the sake of breaking, but for the sake of moving the build system to a language that all Sage developers understand (and therefore can make further improvements).

comment:22 Changed 5 years ago by jdemeyer

Currently in Sage, we have a Python script to download stuff with a shell-script wrapper to handle the case where Python in Sage is not yet installed. I would rather keep the shell-script wrapper and only modify the Python script.

comment:23 follow-up: Changed 5 years ago by vbraun

Why have a shell script here-document a fortran program which runs a python script? Just so that we can build Sage without Python for a few more months? The decision is made, Python is a built-time requirement.

comment:24 in reply to: ↑ 23 Changed 5 years ago by jdemeyer

Replying to vbraun:

The decision is made

I don't recall that discussion...

Let me repeat what I said above:

Why break building on systems without Python on this ticket? It is absolutely trivial to keep support for building without Python for now (just keep the shell part of the script). It seems you're breaking stuff for no reason. I don't mind breaking building without Python if there is a good reason, but not breaking just for the sake of it.

comment:25 Changed 5 years ago by vbraun

You are going to say the same thing all the time, why rely on a Python script here when you could just as well have an additional shell script to provide a fall back. The answer is: fewer moving parts = less bugs.

comment:26 Changed 5 years ago by tmonteil

This is not about python as a requirement nor the curl and wget fallbacks, but i see the benefit of being able to set a URL_GRABBER variable that calls some fetcher with more robust options, as discussed on this sage-devel thread.

comment:27 Changed 4 years ago by vbraun

  • Branch changed from u/ohanar/use_mirrors_for_tarballs to u/vbraun/use_mirrors_for_tarballs

comment:28 Changed 4 years ago by vbraun

  • Commit changed from 991de268a9fd9bc0ff4bd94295d8b6cfbe78015e to 74ac9229132583c3636e551e36aa0a53a23dbc8b
  • Priority changed from major to blocker
  • Status changed from needs_info to needs_work

We must use the mirrors very soon, the centrally-hosted packages will go away.

The downloading script still needs to be called from sage-spkg, when that is finished we must merge this ticket asap.


New commits:

74ac922Merge 6.7.beta3 into #15642

comment:29 Changed 4 years ago by git

  • Commit changed from 74ac9229132583c3636e551e36aa0a53a23dbc8b to 33bf83f60c6332bd06d19c177638b00e0e918354

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

d9c0318Merge 6.7.beta4 into #15642
a4f9892Ignore the single ftp-only mirror for now
33bf83fUse the new sage-download-file script to download all packages

comment:30 Changed 4 years ago by vbraun

  • Status changed from needs_work to needs_review

comment:31 Changed 4 years ago by git

  • Commit changed from 33bf83f60c6332bd06d19c177638b00e0e918354 to a7afa48370ccba8b406a3f820b76ff522bea600c

Branch pushed to git repo; I updated commit sha1. New commits:

a7afa48Add a SAGE_MIRROR environment variable

comment:32 Changed 4 years ago by vbraun

The buildbot will set SAGE_MIRROR to its own so it doesn't have to wait for the mirrors to catch up.

comment:33 Changed 4 years ago by leif

  • Milestone changed from sage-6.4 to sage-6.7

comment:34 Changed 4 years ago by leif

  • Status changed from needs_review to needs_work

"Usage" doesn't match implementation: --fastest-mirror vs. --print-fastest-mirror.

comment:35 Changed 4 years ago by git

  • Commit changed from a7afa48370ccba8b406a3f820b76ff522bea600c to 9a06850aa7e90080cd1841ce7a4aea46da25ccc1

Branch pushed to git repo; I updated commit sha1. New commits:

9a06850fix usage doc

comment:36 Changed 4 years ago by vbraun

  • Status changed from needs_work to needs_review

Thanks, fixed

comment:37 Changed 4 years ago by leif

Not entirely clear to me why you use .startswith() though... ;-)

(Ok, it does support --print-fastest-mirror-and-drop-me-a-beer, but I'm still waiting...)

comment:38 Changed 4 years ago by leif

Could we please suppress the tracebacks on errors like 404?

comment:39 Changed 4 years ago by vbraun

As you said, using startswith() is a cheap emulation of argparse so --quiet=... works. The point is to be compatible with Python 2.6, of course, so we can't use argparse.

comment:40 Changed 4 years ago by leif

Mirror layout fixes, part I:

  • src/bin/sage-spkg

    diff --git a/src/bin/sage-spkg b/src/bin/sage-spkg
    index d25542c..0eb2f8a 100755
    a b if [ ! -f "$PKG_SRC" ]; then 
    367367                echo ">>> Checking online list of $repo packages."
    368368                # File inside DOT_SAGE should be writable
    369369                repolist="${DOT_SAGE}/${repo}.list"
    370                 sage-download-file "$MIRROR/$repo/list" $repolist
     370                sage-download-file "$MIRROR/spkg/$repo/list" $repolist
    371371                if [ $? -ne 0 ]; then
    372                     echo >&2 "Error: failed to download $MIRROR/$repo/list, aborting"
     372                    echo >&2 "Error: failed to download $MIRROR/spkg/$repo/list, aborting"
    373373                    rm -f $repolist
    374374                    exit 1
    375375                fi
    if [ ! -f "$PKG_SRC" ]; then 
    386386
    387387                    # If INFO is set, try downloading only the .txt file
    388388                    if [ $INFO -eq 1 ]; then
    389                         PKG_URL="$MIRROR/$repo/$pkg.txt"
     389                        PKG_URL="$MIRROR/spkg/$repo/$pkg.txt"
    390390                        sage-download-file "$PKG_URL" && exit 0
    391391                        # If the download failed (for whatever reason),
    392392                        # fall through and use the .spkg file.
    if [ ! -f "$PKG_SRC" ]; then 
    416416                            esac
    417417                        fi
    418418                    fi
    419                     PKG_URL="$MIRROR/$repo/$pkg.spkg"
     419                    PKG_URL="$MIRROR/spkg/$repo/$pkg.spkg"
    420420                    break
    421421                fi
    422422            done

(Try e.g. ./sage -i foobar.)

comment:41 Changed 4 years ago by leif

... we'd better use a variable like MIRROR_PACKAGE_FOLDER or whatever there.

comment:42 follow-up: Changed 4 years ago by vbraun

how about you just push that to trac, I'm not going to type that in for you

comment:43 Changed 4 years ago by leif

There are still issues we have to fix w.r.t. MIRROR vs. SAGE_SERVER.

comment:44 Changed 4 years ago by git

  • Commit changed from 9a06850aa7e90080cd1841ce7a4aea46da25ccc1 to d9e66acabedfe0ae55b961482bfa4ec3515257ec

Branch pushed to git repo; I updated commit sha1. New commits:

d9e66acno backtrace on IOError

comment:45 follow-up: Changed 4 years ago by vbraun

SAGE_SERVER is used elsewhere, its where the docs are hosted etc.

comment:46 in reply to: ↑ 42 Changed 4 years ago by leif

Replying to vbraun:

how about you just push that to trac, I'm not going to type that in for you

That's not something I expect you to merge; it just shows errors in the current code. As said, we should IMHO use a variable for that. (I'm not doing your work either... ;-) )

comment:47 in reply to: ↑ 45 Changed 4 years ago by leif

Replying to vbraun:

SAGE_SERVER is used elsewhere, its where the docs are hosted etc.

It's still in an error message in sage-spkg, although MIRROR was used.

comment:48 Changed 4 years ago by git

  • Commit changed from d9e66acabedfe0ae55b961482bfa4ec3515257ec to 30b1647e7b6a527e72daa7835f26a710306a6099

Branch pushed to git repo; I updated commit sha1. New commits:

30b1647fix mirror uri

comment:49 follow-up: Changed 4 years ago by vbraun

Fixed

comment:50 in reply to: ↑ 49 Changed 4 years ago by leif

Replying to vbraun:

Fixed

Ok, but there's an extra slash (MIRROR already has a trailing one).

comment:51 follow-up: Changed 4 years ago by vbraun

the official mirror list may or may not have a slash at the end

comment:52 Changed 4 years ago by leif

FTP doesn't seem to work (hangs after receiving the file).

Try for example

./sage --sh -c 'sage-download-file ftp://ftp.fu-berlin.de/unix/misc/sage/spkg/optional/list'

comment:53 follow-up: Changed 4 years ago by vbraun

Correct, FTP is not allowed

comment:54 in reply to: ↑ 51 Changed 4 years ago by leif

Replying to vbraun:

the official mirror list may or may not have a slash at the end

A matter of what your script outputs...


So, what about SAGE_SERVER to specify a mirror? While $SOMEBODY partially broke that anyway by hardcoding the incompatible packages/ subfolder, we should IMHO still (or again) support that. (At least sage-list-packages also needs to get fixed, preferably on a separate ticket I'd say.)

comment:55 in reply to: ↑ 53 Changed 4 years ago by leif

Replying to vbraun:

Correct, FTP is not allowed

The single mandatory argument can be a http:// or ftp:// url or a
tarball filename.
    if url.startswith('http://') or url.startswith('https://') or url.startswith('ftp://'):
        http_download(url, destination, progress=progress, ignore_errors=True)
Last edited 4 years ago by leif (previous) (diff)

comment:56 follow-up: Changed 4 years ago by vbraun

I don't really like the SAGE_SERVER name (could refer anything, sage cell server), but historically that was where the official mirror list was and thats what I wanted to keep

comment:57 follow-up: Changed 4 years ago by vbraun

Also, IMHO we should just phase out old-style spkgs. As long as we have to drag along that technical debt its very hard to clean up scripts.

comment:58 in reply to: ↑ 56 Changed 4 years ago by leif

Replying to vbraun:

I don't really like the SAGE_SERVER name (could refer anything, sage cell server), but historically that was where the official mirror list was and thats what I wanted to keep

? Before the breakage, setting SAGE_SERVER to e.g. http://mirror.switch.ch/mirror/sagemath/ worked, for example with sage --optional and -- long time ago -- with sage --upgrade.

Deprecate it, later use SAGE_MIRROR for package download purposes?

comment:59 in reply to: ↑ 57 ; follow-up: Changed 4 years ago by leif

Replying to vbraun:

Also, IMHO we should just phase out old-style spkgs.

I'm all against that. The worst decision was to hardcode package metadata (solely!) into specific Sage releases, while that data doesn't even show whether a package is optional, experimental, or whatever. (It's pretty easy to create "old-style" packages from that though... ;-) )

In my opinion the main reason we should (still) support real spkgs is that anybody can easily create and offer such self-contained packages to others, and installing and using them doesn't artificially depend on the Sage version. (Whether they'll work of course depends on a couple of things.)

They're also often much more convenient for testing.

As long as we have to drag along that technical debt its very hard to clean up scripts.

Perhaps with the exception of left-over old-style standard packages, I don't see a problem.

comment:60 Changed 4 years ago by git

  • Commit changed from 30b1647e7b6a527e72daa7835f26a710306a6099 to a103ab5dd842e5a6f24970a42649e4df8935a9da

Branch pushed to git repo; I updated commit sha1. New commits:

27b71e5remove ftp from docs for now
a103ab5Use SAGE_SERVER instead of SAGE_MIRROR

comment:61 in reply to: ↑ 59 Changed 4 years ago by vbraun

Replying to leif:

Whether they'll work of course depends on a couple of things.

Even if they work in principle you can't do that much with them as there is no dependency management. Its just an all-out shitty solution for everyone.

comment:62 Changed 4 years ago by vbraun

SAGE_SERVER must of course then not be set by default...

comment:63 follow-up: Changed 4 years ago by leif

Does any mirror have packages/? Will they at some point?

Otherwise s|SAGE_SERVER/packages/|SAGE_SERVER/spkg/| in the docs.

comment:64 Changed 4 years ago by vbraun

  • Cc schilly added

AFAIK there will only ever be /spkg/ on the mirrors

comment:65 Changed 4 years ago by git

  • Commit changed from a103ab5dd842e5a6f24970a42649e4df8935a9da to a259f27b95e1b69fbdde1a4b0e8496996cd427fe

Branch pushed to git repo; I updated commit sha1. New commits:

a259f27rename SAGE_SERVER/packages to SAGE_SERVER/spkg

comment:66 in reply to: ↑ 63 Changed 4 years ago by leif

Replying to leif:

Does any mirror have packages/? Will they at some point?

Otherwise s|SAGE_SERVER/packages/|SAGE_SERVER/spkg/| in the docs.

Should we fix this here, since we now touch sage-list-packages anyway?

SAGE_SERVER="http://mirror.switch.ch/mirror/sagemath/" ./sage --optional
Using Sage Server http://mirror.switch.ch/mirror/sagemath/packages
HTTP Error 404: Not Found



********************************************************************************



Error contacting http://mirror.switch.ch/mirror/sagemath/packages/optional/list. Try using an alternative server.
For example, from the bash prompt try typing

   export SAGE_SERVER=http://sage.scipy.org/sage/

then try again.



********************************************************************************

(http://sage.scipy.org/sage/ is btw. a joke...)

comment:67 Changed 4 years ago by leif

Concurrent posting (luckily not pushing)...

comment:68 Changed 4 years ago by git

  • Commit changed from a259f27b95e1b69fbdde1a4b0e8496996cd427fe to d79364b9fc5131199eaed14340582715f2f937fb

Branch pushed to git repo; I updated commit sha1. New commits:

d79364bfix sage-list-packages

comment:69 follow-up: Changed 4 years ago by vbraun

The whole listing of packages is also made hideously complicated by mashing old and new style packages together...

comment:70 Changed 4 years ago by leif

I thought of just adding a further try: ... except HTTPError: ..., in case sagemath.org/packages/ remains for a while. (IIRC it had both for a while.)

PKG_SERVER is superfluous anyway; I'd simply use os.environ['SAGE_SERVER'] there.

comment:71 in reply to: ↑ 69 Changed 4 years ago by leif

Replying to vbraun:

The whole listing of packages is also made hideously complicated by mashing old and new style packages together...

As I said, in principle only the old standard packages disturb a bit; otherwise it's just a matter of syncing.

I was actually working on that a few days ago (#16759).

comment:72 Changed 4 years ago by leif

Quoting jdemeyer:

Replying to jhpalmieri:

It also looks like sage --standard is broken: it is using old package information.

I would say that sage --standard is doing the right thing, but somehow http://www.sagemath.org/packages/standard/ is no longer updated. The right solution is updating the website script to take into account the new git-style packages (which is certainly a different issue than the one this ticket is about).

comment:73 follow-up: Changed 4 years ago by schilly

Since I'm CCed: There is currently a move of the whole infrastructure going on. Before that I tried to keep everything compatible with the original system from a long time ago. That's no longer possible. Long story short: the website and the binary-files need to be untangled and that means there are no longer chains of symlink-aliases. Hence, don't use the /mirror subdirectory of the website and I hope the essence of this ticket is to get rid of this old dependence. The only mirror related aspect is the mirror_list file for the base URLs.

comment:74 in reply to: ↑ 73 Changed 4 years ago by leif

Replying to schilly:

Since I'm CCed: There is currently a move of the whole infrastructure going on. Before that I tried to keep everything compatible with the original system from a long time ago. That's no longer possible. Long story short: the website and the binary-files need to be untangled and that means there are no longer chains of symlink-aliases. Hence, don't use the /mirror subdirectory of the website and I hope the essence of this ticket is to get rid of this old dependence. The only mirror related aspect is the mirror_list file for the base URLs.

Well, the Sage scripts never used the mirror/ folder. The problematic part here is the structure below the base URL; spkg/ was changed to packages on sagemath.org, but the mirrors all have the classical structure (plus upstream/ below spkg/).

The new thing on this ticket is also that we now use http://www.sagemath.org/mirror_list (currently hardcoded) again, which is periodically generated by a script on the server (or at least used to be).

comment:75 Changed 4 years ago by leif

Harald, also note Jeroen's comment I quoted above.

Meanwhile, it's not just standard packages, but also optional packages, of which (only) some now have the "new style", i.e., the data of newer ones is hardcoded into the Sage releases, while obsolete "old-style" packages remain in spkg/optional/list (and spkg/optional/).

./sage --optional shows some examples of optional packages which became standard:

Optional packages:

...
git-1.7.12.2.p0 ........................ installed version: 2.3.0
...
pyzmq-2.1.11.p1 ........................ installed version: 14.3.0
...
zeromq-3.2.0.p0 ........................ installed version: 4.0.5

There are other still optional packages where sage -i ... would install newer versions than listed by sage --optional because the "local metadata" ($SAGE_ROOT/build/pkgs/<package>/package-version.txt) differs from what spkg/optional/list gives. (Same issue as with our current standard packages.)

A more funny one is TOPCOM:

./sage -i TOPCOM installs the "legacy" spkg, while ./sage -i topcom installs the "new-style" spkg. (The package versions are currently still the same though, modulo a .p0 for the "new-style" one.)

comment:76 follow-up: Changed 4 years ago by vbraun

We definitely should not check sagemath.org/packages any more, I've already changed the release management scripts to no longer put the tarballs there.

comment:77 in reply to: ↑ 76 ; follow-up: Changed 4 years ago by leif

Replying to vbraun:

We definitely should not check sagemath.org/packages any more, I've already changed the release management scripts to no longer put the tarballs there.

Ok, sagemath.org/spkg/* gave a 404 for a while, but apparently works again.

comment:78 Changed 4 years ago by leif

Note for other reviewers: So far I'm ok with the changes/corrections made, but haven't yet tested thoroughly enough (I think).

While I don't like completely changing the previously generic character of sage-download-file (nor removing the "shell fallback" with e.g. wget or curl if Python isn't available or functional), I strongly doubt we could convince the author or the RM to change that back to how it was.

FTP support (probably with Python3) and e.g. usage of a static mirror list (perhaps to be shipped with Sage) is something for follow-up tickets; the broken behaviour of sage-list-packages is (or will be) dealt with at #16759, which depends on this ticket.


W.r.t. the threat of losing sagemath.org for downloads: According to William, "soon" is not that soon.

comment:79 Changed 4 years ago by vbraun

I don't mind if you rewrite the functionality in bash fwiw. I would consider it utterly retarded and backward, but as long as it works and you are committed to maintaining that solution then that would be fine with me.

comment:80 in reply to: ↑ 77 ; follow-up: Changed 4 years ago by schilly

Replying to leif:

Ok, sagemath.org/spkg/* gave a 404 for a while, but apparently works again.

Uhm, to reduce that confusion a little bit: The source of this inconsistency is that right now most of the old infrastructure is still running, including the currently "old" website. I would suggest to wait until the "new" website is operational. Then, any bugs in scripts relying on the website will be obvious (and fixes to the new website visible immediately).

The reason why this is a bit slow is, because there are many links assuming certain subdirectories (for documentation and e.g. /mirror/) and my time is limited. I could report back once the DNS entry has switched to the new page.

PS: "new" just means, that there are minor changes like an additional mirror, publications etc. The current "old" website is an older fork of the website with some other patches. Hence this diverged and there is no need to support the old one any more.

comment:81 in reply to: ↑ 80 ; follow-ups: Changed 4 years ago by leif

Replying to schilly:

I could report back once the DNS entry has switched to the new page.

ETA Hausnummer?

PS: "new" just means, that there are minor changes like an additional mirror, publications etc. The current "old" website is an older fork of the website with some other patches. Hence this diverged and there is no need to support the old one any more.

Could we already test the new one (using its unofficial address) w.r.t. download stuff / package (and mirror) lists?

comment:82 follow-ups: Changed 4 years ago by vbraun

The attached branch will only use www.sagemath.org to download the mirror list, so once we have merged it there are a lot fewer things that could break.

comment:83 in reply to: ↑ 81 Changed 4 years ago by schilly

Replying to leif:

ETA Hausnummer?

Friday. It's tedious stupid work, so maybe I'm already whining more about it than it will actually take to work on.

Could we already test the new one (using its unofficial address) w.r.t. download stuff / package (and mirror) lists?

No, Github resolves the inofficial URL to the official one and I don't want to mess with this around even more.

comment:84 in reply to: ↑ 82 Changed 4 years ago by schilly

Replying to vbraun:

The attached branch will only use www.sagemath.org to download the mirror list ...

Yes, it's just one file, and as already mentioned: the full list could easily be hardcoded as a backup into Sage to remove that SPOF. Ticket for later please :-)

comment:85 in reply to: ↑ 82 Changed 4 years ago by leif

Replying to vbraun:

once we have merged it there are a lot fewer things that could break.

Provided the mirrors can still sync...


In principle we could already include a snapshot of mirror_list into the git tree (in $SAGE_ROOT/? in more human-readable form?). The dynamic one should rather be located in $SAGE_ROOT/local/var/{lib,tmp}/sage/, say, not $SAGE_DIST_FILES/ (defaults to $SAGE_ROOT/upstream/).

comment:86 follow-up: Changed 4 years ago by vbraun

Nothing else needs the mirror list so why put it in a separate file? In any case, this question should be dealt with in a separate ticket.

comment:87 in reply to: ↑ 86 Changed 4 years ago by leif

Replying to vbraun:

Nothing else needs the mirror list

Human users perhaps?

so why put it in a separate file?

You're right, all package metadata should be moved into sage-spkg as well.

comment:88 Changed 4 years ago by vbraun

There are many packages but only one mirror list. Your argument is invalid.

Not to mention the question of what file format, custom = crap, json = not exactly human-readable, yaml = not necessarily available in the System python, is it supposed to be readable from a shell script, ...

comment:89 follow-up: Changed 4 years ago by leif

Plain text, one URL per line, preferably in the top-level folder. What's the problem?

Any human and any kind of script can easily read such. (If you want to add further data such as organization, location etc., just put that after each URL, seperated by whitespace, but I wouldn't even do that. We don't have to process that. Blank lines and comment lines starting with # at the first position would IMHO be ok, too, but that's not really needed. Order of the mirrors doesn't matter for our purposes either, although we could keep them sorted by continent, say.)

I thought we had postponed that to a follow-up ticket.

comment:90 in reply to: ↑ 89 ; follow-up: Changed 4 years ago by vbraun

Replying to leif:

I thought we had postponed that to a follow-up ticket.

We have, but you keep bringing it up.

comment:91 in reply to: ↑ 90 Changed 4 years ago by leif

Replying to vbraun:

you keep bringing it up.

Not me, Harald did.

comment:92 in reply to: ↑ 81 ; follow-up: Changed 4 years ago by schilly

Replying to leif:

Replying to schilly:

I could report back once the DNS entry has switched to the new page.

ETA Hausnummer?

Yay! 2 days ahead of schedule! I fixed all the obviously broken links and there are just two minor details left.

comment:93 Changed 4 years ago by vbraun

  • Reviewers set to Volker Braun
  • Status changed from needs_review to positive_review

comment:94 in reply to: ↑ 92 Changed 4 years ago by leif

Replying to schilly:

Replying to leif:

Replying to schilly:

I could report back once the DNS entry has switched to the new page.

ETA Hausnummer?

Yay! 2 days ahead of schedule! I fixed all the obviously broken links and there are just two minor details left.

sagemath.org no longer serves packages (especially not the upstream tarballs); is that intentional, or will it again? (Cf. sage-release.)

comment:95 Changed 4 years ago by vbraun

  • Branch changed from u/vbraun/use_mirrors_for_tarballs to d79364b9fc5131199eaed14340582715f2f937fb
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.