Opened 2 years ago

Closed 2 years ago

Last modified 2 years ago

#27978 closed enhancement (fixed)

sha1-based configure versioning

Reported by: dimpase Owned by:
Priority: major Milestone: sage-8.8
Component: build: configure Keywords:
Cc: embray, vbraun Merged in:
Authors: Dima Pasechnik Reviewers: Volker Braun
Report Upstream: N/A Work issues:
Branch: 1cab029 (Commits, GitHub, GitLab) Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by dimpase)

as proposed by Volker, extend the bootstrap script to save/load based on the git head sha1, e.g.:

    ./bootstrap -s

saves upstream/configure-<sha1>.tag.gz

    ./bootstrap  -u http://path/to/upstream/

loads http://path/to/upstream/configure-<sha1>.tag.gz

./bootstrap -u <URL> loads the tarball to upstream/ and checks that the checksum matches, as usual, so this is basically an extenion of -D option.

We also remove -i as not needed any more.

Change History (32)

comment:1 Changed 2 years ago by dimpase

Is there any need to keep the numeric version numbers?

How about simply switching from version numbers for configure to sha1 versions?

Then -i should be just removed, and -s would mean --save-head, as in the proposal ? So then the main change would be implementing --load-head (perhaps, --load-from-url is a better name).

comment:2 Changed 2 years ago by vbraun

Yes, we don't need consecutive numeric version numbers at all

comment:3 Changed 2 years ago by dimpase

  • Description modified (diff)
  • Summary changed from implement ./bootstrap --save-head to sha1-based configure versioning

comment:4 Changed 2 years ago by embray

This was a good suggestion, thanks. Would we still need to distribute all the tarball on the mirror though? I'm not sure exactly how this will work.

comment:5 Changed 2 years ago by dimpase

  • Authors set to Dima Pasechnik
  • Branch set to public/build/sha1bootstrap
  • Commit set to 69d274d6f6d887ee0cabf93e9717faa1fa03de34

New commits:

df844b6switch to sha1 for version "number", remove -i
69d274duse new interface for checksum computation

comment:6 Changed 2 years ago by dimpase

So far, the branch just does the switch to sha1 from "number".

The url to download the configure tarball may be stored somewhere, maybe in build/pkgs/configure/checksum.ini ?

comment:7 Changed 2 years ago by vbraun

In my proposal the download base url is a parameter to bootstrap, since the buildbot and patchbots may want to use different servers. And its then easy enough to supply the location in the build step configuration, and you don't need to distribute the confballs on the mirror network.

comment:8 follow-up: Changed 2 years ago by embray

But what would be that location? Say for the patchbots you'd want to have them get the configure tarball from a ticket attachment. That could be done, but we'd have to modify the patchbot code to do that (doesn't sound too hard, I just want to make sure I understand properly). What would be the standard location for the buildbots to check?

comment:9 Changed 2 years ago by git

  • Commit changed from 69d274d6f6d887ee0cabf93e9717faa1fa03de34 to 8367d565e2157574e3ed51150ebf02f35bafe608

Branch pushed to git repo; I updated commit sha1. New commits:

8367d56implement fetching configure tarball from an URL

comment:10 Changed 2 years ago by dimpase

  • Description modified (diff)
  • Status changed from new to needs_review

OK, so here it is.

comment:11 Changed 2 years ago by dimpase

basically, -u <URL> triggers uploading the URL to upstream/, and it implies -D.

if the resulting tarball does not match the present branch, too bad, it will fail (unless it succeeds to download from the "usual" Sage mirrors)

comment:12 in reply to: ↑ 8 Changed 2 years ago by dimpase

Replying to embray:

But what would be that location? Say for the patchbots you'd want to have them get the configure tarball from a ticket attachment. That could be done, but we'd have to modify the patchbot code to do that (doesn't sound too hard, I just want to make sure I understand properly). What would be the standard location for the buildbots to check?

a package tarball is a package tarball. Is there a special-casing for configure package in patch/buildbot configs?

I presume mentioning its URL in the ticket description should be equivalent to what we do with the "normal" packages (for this the new -u feature doesn't seem to be needed). And release tools can make use of it in some way too.

comment:13 Changed 2 years ago by vbraun

I've asked Harald & Wiliam for what a good "default" serving url is in the current webserver, but we don't necessarily have to implement that fallback here.

comment:15 Changed 2 years ago by dimpase

  • Description modified (diff)

comment:16 Changed 2 years ago by vbraun

  • Reviewers set to Volker Braun
  • Status changed from needs_review to positive_review

comment:17 Changed 2 years ago by vbraun

  • Status changed from positive_review to needs_work

/src/bin/sage-update-version calls bootstrap -i which is now an illegal argument error

comment:18 Changed 2 years ago by dimpase

would just removing that “-i” there be good enough?

comment:19 Changed 2 years ago by vbraun

Yes, just needs to be removed I think

Also:

    if [ $CONFTARBALL_URL != "" ]; then
        [...]
    fi
    bootstrap-download || exit $?

always ends up calling bootstrap-download, should be in an else branch

comment:20 Changed 2 years ago by git

  • Commit changed from 8367d565e2157574e3ed51150ebf02f35bafe608 to e58bab67b00e8cce7f5ce0646672ae228fe20928

Branch pushed to git repo; I updated commit sha1. New commits:

f0340eeno more '-i'
e58bab6move bootstrap-download call

comment:21 Changed 2 years ago by dimpase

  • Status changed from needs_work to needs_review

comment:22 Changed 2 years ago by git

  • Commit changed from e58bab67b00e8cce7f5ce0646672ae228fe20928 to 1cab029ccaafc6158ece9187e7cedc0ba6bdab37

Branch pushed to git repo; I updated commit sha1. New commits:

1cab029Use the stored instead of the current sha1 when loading.

comment:23 Changed 2 years ago by vbraun

There was also the following logic error: The confball checksum needs to be checked in, changing the sha1. So when restoring we need to use the sha1 from package-version.txt and not the current HEAD sha1.

Also, sage-download-file used to ignore error.

comment:24 Changed 2 years ago by dimpase

I merely assumed that with CONFTARBALL_URL one can install an arbitrary configure verrsion, not one in any way related to the current repo state. Certainly if your need the latter, this is fine too.

Last edited 2 years ago by dimpase (previous) (diff)

comment:25 Changed 2 years ago by vbraun

I think its better to have bootstrap figure out the confball to download, otherwise the buildbot needs to rummage around in the sources which is all rather complicated.

comment:26 Changed 2 years ago by vbraun

  • Branch changed from public/build/sha1bootstrap to 1cab029ccaafc6158ece9187e7cedc0ba6bdab37
  • Resolution set to fixed
  • Status changed from needs_review to closed

comment:27 Changed 2 years ago by embray

  • Commit 1cab029ccaafc6158ece9187e7cedc0ba6bdab37 deleted

So now what are we supposed to do for building in a fresh Sage repository when we need the configure tarball? Just running make configure blows up with an error that the relevant tarball can't be found.

comment:28 follow-up: Changed 2 years ago by dimpase

Are you talking about the transition from old versioning to the new one?

comment:29 in reply to: ↑ 28 Changed 2 years ago by embray

Replying to dimpase:

Are you talking about the transition from old versioning to the new one?

I don't know. Am I? On a fresh VM in a fresh build of Sage I just get sage-download blowing up that it can't find configure-<sha1>.tar.gz.

comment:30 Changed 2 years ago by embray

I think I must be seeing #28099. Likewise on my buildbot:

[xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
ERROR [transfer|run:135]: [Errno 404] Not Found: '//old.files.sagemath.org/configure//configure-6fd12b83387b4658062ce7af6ddf23d68d9da17e.tar.gz'
************************************************************************
Traceback (most recent call last):
  File "/home/Admin/sage-buildbot-worker/sage_git/build/build/bin/../sage_bootstrap/download/cmdline.py", line 118, in run_safe
    run()
  File "/home/Admin/sage-buildbot-worker/sage_git/build/build/bin/../sage_bootstrap/download/cmdline.py", line 98, in run
    app.download_url(args.url_or_tarball, args.destination)
  File "/home/Admin/sage-buildbot-worker/sage_git/build/build/bin/../sage_bootstrap/download/app.py", line 39, in download_url
    Download(url, destination, progress=not self.quiet, ignore_errors=False).run()
  File "/home/Admin/sage-buildbot-worker/sage_git/build/build/bin/../sage_bootstrap/download/transfer.py", line 137, in run
    raise error
DownloadError: [Errno 404] Not Found: '//old.files.sagemath.org/configure//configure-6fd12b83387b4658062ce7af6ddf23d68d9da17e.tar.gz'
************************************************************************
Error: downloading configure-6fd12b83387b4658062ce7af6ddf23d68d9da17e.tar.gz from http://old.files.sagemath.org/configure/ failed

comment:31 Changed 2 years ago by dimpase

could it be that Volker did not upload the configure tarball to mirrors yet?

comment:32 Changed 2 years ago by dimpase

yeah I guess #28099 will fix this.

Note: See TracTickets for help on using tickets.