Opened 9 years ago

Last modified 9 years ago

#9433 closed enhancement

Put more files under revision control. — at Version 72

Reported by: jhpalmieri Owned by: tbd
Priority: blocker Milestone: sage-4.7
Component: distribution Keywords:
Cc: was, ddrake, kcrisman, leif Merged in:
Authors: John Palmieri Reviewers: Leif Leonhardy
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description (last modified by jhpalmieri)

Put the text files in $SAGE_ROOT, and also the text files in spkg, under revision control. (See the discussion at the end of #9351.)

Here are the instructions:

  • apply the patches trac_9433-sage-repo.2.patch and trac_9433-scripts.v5.patch
  • move the attached file "hgignore" to SAGE_ROOT/.hgignore
  • move the attached file "root-spkg-install.v2" to SAGE_ROOT/spkg/root-spkg-install
  • move the attached file "install" to SAGE_ROOT/spkg/install
  • move the attached file "deps" to SAGE_ROOT/spkg/standard/deps

Then from $SAGE_ROOT, run the attached script "hg_script" to create the Mercurial repository.

To test upgrading, try

./sage -upgrade http://sage.math.washington.edu/home/palmieri/misc/9433/sage-4.6.1.9433.alpha0/

You can then try

./sage -upgrade http://sage.math.washington.edu/home/palmieri/misc/9433/sage-4.6.1.9433.alpha1/

to test an upgrade which changes an existing root repo: it is supposed to upgrade without error, changing just the beginning of the file SAGE_ROOT/README.txt.

Change History (90)

comment:1 Changed 9 years ago by jhpalmieri

  • Status changed from new to needs_review

I'm marking this as "needs review", but consider this patch experimental.

comment:2 Changed 9 years ago by jhpalmieri

A little explanation: this patch creates a directory "other-scripts" in SAGE_ROOT/local/bin. This new directory contains a brief README.txt and also subdirectories "root" and "spkg". "root" contains the text files from SAGE_ROOT. The only one with any changes is README.txt which explains how these files are under revision control. Similarly, "spkg" contains various text files from SAGE_ROOT/spkg, and the only one with any changes is README.txt.

comment:3 Changed 9 years ago by jhpalmieri

This probably needs to be rebased. When people are ready to look at it, let me know and I'll see what I can do.

Changed 9 years ago by jhpalmieri

rebased against 4.5.alpha4 + #9456

comment:4 Changed 9 years ago by jhpalmieri

New approach, after a discussion on sage-devel: create a new repo at the top level tracking the appropriate files. I'm attaching a new version of the patch for the scripts repo. Someone -- the release manager, I guess -- also needs to create the top level repo, because I don't know how to do this in such a way that it can be posted on a ticket. Here are the instructions:

  • move the attached file "hgignore" to SAGE_ROOT/.hgignore
  • cd $SAGE_ROOT
  • hg init .
  • hg add .hgignore COPYING.txt README.txt makefile sage sage-python
  • cd spkg
  • hg add README.txt gen_html install pipestatus
  • cd standard
  • hg add README.txt deps libdist_filelist newest_version
  • hg add notes.txt numeric-24.2.txt

(I don't know if we really need these last two files, but this is probably not the ticket for making such decisions.) Finally, do

  • hg commit

When you run "sage -sdist ..." it should add a tag for the new version of Sage.

This does not create a new spkg for the files in SAGE_ROOT, since those files have to be in place when you unpack the sage tar file. But it creates the repository so that people can post patches to the trac server, etc.

comment:5 follow-up: Changed 9 years ago by was

  • Cc was added

Looking with my eyes, this looks good. I don't have time to test right now. The test would be to take a clean Sage, do the above, then do "sage -sdist ..." and make sure that in the sdist the above is all still there.

Changed 9 years ago by jhpalmieri

main repo: add "hg_root" command to Sage

comment:6 in reply to: ↑ 5 Changed 9 years ago by jhpalmieri

Replying to was:

The test would be to take a clean Sage, do the above, then do "sage -sdist ..." and make sure that in the sdist the above is all still there.

This works for me, but other people should definitely look at it carefully.

comment:7 Changed 9 years ago by jhpalmieri

This probably needs work: how will it work with "sage -upgrade"?

comment:8 Changed 9 years ago by jhpalmieri

  • Description modified (diff)

Here's a new version of the patch for the scripts repo. I think this should deal with upgrading: the script "sage-upgrade" now runs "sage --hg branch" from SAGE_ROOT, and if this fails, it assumes that there is no Mercurial repository and creates it.

comment:9 Changed 9 years ago by ddrake

  • Cc ddrake added

comment:10 Changed 9 years ago by jhpalmieri

This may need to be changed if #9609 is merged: the directions here, and also the modified "sage-upgrade" script, refer to files which would be deleted by #9609.

comment:11 Changed 9 years ago by jhpalmieri

  • Description modified (diff)

New version, rebased against #9609.

comment:12 Changed 9 years ago by ddrake

In sage-sdist, where you have # copy sage root repo over, why not just clone the repo? That will take care of copying all the necessary files, and if we add or remove files tracked by the repo, we won't need to mess with sage-sdist. I'm thinking that something like

cd $SAGE_ROOT 
hg clone --pull . DEST_DIR

Using --pull means that it doesn't use hardlinks in the clone; I *think* there would be no problem with using hardlinks, but it's unlikely to make a big difference. The clone will include a hgrc file that points to where it came from: it would look something like this:

[paths]
default = /home/foo/sage-whatever

We could simply delete the file, or just leave it, since it would not negatively affect anything (except running hg pull from SAGE_ROOT, which you wouldn't do anyway).

So, all the cp -p lines could be just

hg clone --pull . $TMP
rm $TMP/.hg/hgrc

and files added or removed to the repo would get copied correctly without changing any scripts.

Changed 9 years ago by jhpalmieri

new version, using "hg clone". apply to scripts repo.

Changed 9 years ago by jhpalmieri

new version, using "hg clone". apply to scripts repo.

comment:13 Changed 9 years ago by jhpalmieri

Note that if you use "hg clone ..." to copy the repo, you have to do it earlier in the process: it apparently needs to be done with an empty directory, so in sage-sdist it is now done right after creating $TMP.

comment:14 Changed 9 years ago by kcrisman

  • Cc kcrisman added

comment:15 follow-up: Changed 9 years ago by mpatel

  • Authors set to John Palmieri
  • Milestone changed from sage-4.5.2 to sage-4.5.3
  • Status changed from needs_review to needs_work

If I

  • Build the forthcoming Sage 4.5.2 (which is just 4.5.2.rc1 + #9226) from source.
  • Follow the steps in the description.
  • ./sage -sdist 4.5.99

hg log in the "root" repo now gives

changeset:   1:0fea58e94942
tag:         tip
user:        Mitesh Patel <qed777@gmail.com>
date:        Fri Aug 06 21:40:23 2010 -0700
summary:     Added tag 4.5.99 for changeset 4c1f4320f743

changeset:   0:4c1f4320f743
tag:         4.5.99
user:        Mitesh Patel <qed777@gmail.com>
date:        Fri Aug 06 21:33:45 2010 -0700
summary:     Initial Sage "root" repository

The new 4.5.99 builds successfully from source and passes the long doctests. But hg log in the root repo for 4.5.99 lists just revision 0, and the root repo is missing from a binary distribution made with ./sage -bdist 4.5.99.

comment:16 Changed 9 years ago by mpatel

I also noticed

  • SAGE_ROOT/ipython and SAGE_ROOT/sage-README-osx.txt are missing from the new source and binary distributions.
  • An extra SAGE_ROOT/sage-python after unpacking sage-4.5.99.tar.

comment:17 in reply to: ↑ 15 ; follow-up: Changed 9 years ago by jhpalmieri

  • Status changed from needs_work to needs_review

Replying to mpatel:

The new 4.5.99 builds successfully from source and passes the long doctests. But hg log in the root repo for 4.5.99 lists just revision 0, and the root repo is missing from a binary distribution made with ./sage -bdist 4.5.99.

Right, I didn't do this right. In the new patch, the root repo is not modified at all by sage-make_devel_packages; instead, sage-sdist clones it, tags it, and commits the new tag, while sage-bdist just clones it.

I also noticed

  • SAGE_ROOT/ipython and SAGE_ROOT/sage-README-osx.txt are missing from the new source and binary distributions.

The missing ipython directory was an oversight. I think I've fixed it. The missing sage-README-osx.txt was intentional: this should only be included for binary distributions on OS X, and its presence there is taken care of by sage-bdist:

if [ "$UNAME" = "Darwin" ]; then
    ...
    cp sage/local/bin/sage-README-osx.txt README.txt
    ...

Perhaps we can close #6938 if this gets merged?

  • An extra SAGE_ROOT/sage-python after unpacking sage-4.5.99.tar.

This was intentional. Before this, the file sage-python was stored in the scripts repo and then unpacked to the top level. I'm not sure why this was done, but I wanted to keep the end result as close as possible to what it was before.

comment:18 Changed 9 years ago by jhpalmieri

  • Description modified (diff)

comment:19 in reply to: ↑ 17 Changed 9 years ago by kcrisman

I also noticed

  • SAGE_ROOT/ipython and SAGE_ROOT/sage-README-osx.txt are missing from the new source and binary distributions.

The missing ipython directory was an oversight. I think I've fixed it. The missing sage-README-osx.txt was intentional: this should only be included for binary distributions on OS X, and its presence there is taken care of by sage-bdist:

if [ "$UNAME" = "Darwin" ]; then
    ...
    cp sage/local/bin/sage-README-osx.txt README.txt
    ...

Perhaps we can close #6938 if this gets merged?

As one of the people involved on that ticket, that is fine. The problem is that #6938 does not currently have positive review! So I think that would be necessary first, or something else indicating that the solution proposed there is correct. Maybe 'merge' that ticket at the same time as this one, for whatever it's worth.

Sounds like you agree :) In fact, notice that once that is removed, that file will only appear ABOVE the SAGE_ROOT directory, in the place a normal README would occur in a dmg or bundle, so it does work properly (I've tested this numerous times

comment:20 Changed 9 years ago by jhpalmieri

  • Description modified (diff)

I realized there was another problem with the previous patch: while it would work when upgrading from a Sage build with no root repo, it wouldn't do anything when upgrading a Sage build with an existing root repo. The new patch constructs a sage_root spkg file, but this file only gets used during upgrading. So it is installed in the script "sage-upgrade", but it does not appear in spkg/standard/deps.

comment:21 Changed 9 years ago by jhpalmieri

  • Status changed from needs_review to needs_work

This still doesn't deal with upgrading very well. I need to think about how to do this and work on it for a while in a place where I have a better internet connection. I may have something in a few days. Meanwhile, if anyone has suggestions for how to deal with upgrades, I would like to hear them. The issues:

  • currently, I don't even know what happens with "sage -upgrade ..." if SAGE_ROOT/makefile is changed, for example. Does anything happen?
  • does it matter whether we install the new sage_root spkg before or after the sage_scripts spkg? The patch here affects the sage-upgrade script, but the new version won't get run during an upgrade anyway.
  • is it good enough to just install the new sage_root spkg? I think it might be, but I might be confused. Anyway, I think I need time to play with it and test it out.

comment:22 Changed 9 years ago by jhpalmieri

  • Description modified (diff)
  • Status changed from needs_work to needs_review

This seems to work for me, with one slight glitch: if I run "sage -upgrade" on a copy of sage which does not yet include the root repo, it lists the spkg's to be upgraded and asks me "do you really want to continue", then it does some stuff, and then it lists just the root_repo spkg and asks me again if I want to continue. The issue is that, before it has installed some of the upgraded packages, it doesn't know what to do with the root_repo spkg, so it doesn't get installed the first time through. I don't see any way around this, but it's a one-time problem.

comment:23 Changed 9 years ago by leif

  • Cc leif added

Ooops, I once had been aware of this ticket, but forgot to cc me...

Before doing this, can we rename makefile to Makefile?

comment:24 Changed 9 years ago by jhpalmieri

Before doing this, can we rename makefile to Makefile?

I have no problems with that. Are there ever any (good) reasons for using makefile instead of Makefile?

comment:25 follow-up: Changed 9 years ago by leif

Should sage-README-osx.txt be ignored (i.e. not be under revision control)?

Also a candidate for renaming (README.MacOS_X.txt or alike).

From the attached hg_script:

...
$hg add .hgignore .hgtags COPYING.txt README.txt makefile sage sage-python
cd ipython
$hg add *.py ipythonrc*
...

sage-python and ipython/ should IMHO be in .hgignore, since these are not in $SAGE_ROOT until Sage is built.

comment:26 in reply to: ↑ 25 ; follow-ups: Changed 9 years ago by jhpalmieri

Replying to leif:

Should sage-README-osx.txt be ignored (i.e. not be under revision control)?

It shouldn't be present at all, and with the patches here, it isn't. See #6938 and also my comment above. (It should only appear in the parent directory of SAGE_ROOT when you make a dmg file on a Mac.)

sage-python and ipython/ should IMHO be in .hgignore, since these are not in $SAGE_ROOT until Sage is built.

This is another change on this ticket: these files are now tracked in this repo, not in sage_scripts anymore. Actually, I don't know if ipython was tracked anywhere, but now it is. I don't know who chose to include sage-python in the top-level directory, but since it's there, it should be tracked properly. (It's a simple enough script I don't mind having two copies of it.) I personally don't see the point of it and think it should be removed, but that should probably happen on another ticket. Or maybe it should be a link to local/bin/sage-python?

comment:27 in reply to: ↑ 26 Changed 9 years ago by kcrisman

Replying to jhpalmieri:

Replying to leif:

Should sage-README-osx.txt be ignored (i.e. not be under revision control)?

It shouldn't be present at all, and with the patches here, it isn't. See #6938 and also my comment above. (It should only appear in the parent directory of SAGE_ROOT when you make a dmg file on a Mac.)

Correct.

comment:28 in reply to: ↑ 26 Changed 9 years ago by leif

Replying to jhpalmieri:

Replying to leif:

Should sage-README-osx.txt be ignored (i.e. not be under revision control)?

It shouldn't be present at all, and with the patches here, it isn't. See #6938 and also my comment above. (It should only appear in the parent directory of SAGE_ROOT when you make a dmg file on a Mac.)

Missed that.

sage-python and ipython/ should IMHO be in .hgignore, since these are not in $SAGE_ROOT until Sage is built.

This is another change on this ticket: these files are now tracked in this repo, not in sage_scripts anymore.

That, too... :/

Actually, I don't know if ipython was tracked anywhere, but now it is. I don't know who chose to include sage-python in the top-level directory, but since it's there, it should be tracked properly. (It's a simple enough script I don't mind having two copies of it.) I personally don't see the point of it and think it should be removed, but that should probably happen on another ticket. Or maybe it should be a link to local/bin/sage-python?

Perhaps...

Sorry for the noise.

comment:29 follow-up: Changed 9 years ago by leif

I'd also like to have a VERSION.txt in SAGE_ROOT/, best containing just the "plain" version number (perhaps only with also the release date added); cf. #9434 (the number is a coincidence!).

What about release notes (in the top-level directory)?

comment:30 in reply to: ↑ 29 ; follow-up: Changed 9 years ago by kcrisman

Replying to leif:

I'd also like to have a VERSION.txt in SAGE_ROOT/, best containing just the "plain" version number (perhaps only with also the release date added); cf. #9434 (the number is a coincidence!).

That's a good idea. Be sure not to hold this one up too long ;)

What about release notes (in the top-level directory)?

That definitely used to be in this directory, filename HISTORY.txt. Maybe it was getting long? It certainly got out of date quickly. It might be an onus on the release manager to create the notes *before* the release is made, though - usually that takes a little time, and then it somehow gets added to the official version at sagemath.org (and a few other places?). The problem is that this isn't fully automated yet.

comment:31 in reply to: ↑ 30 ; follow-up: Changed 9 years ago by leif

Replying to kcrisman:

Replying to leif:

I'd also like to have a VERSION.txt in SAGE_ROOT/, best containing just the "plain" version number (perhaps only with also the release date added); cf. #9434 (the number is a coincidence!).

That's a good idea. Be sure not to hold this one up too long ;)

I don't think adding this needs great effort... We already have devel/sage/sage/version.py, which looks like this:

"""nodoctests"""
version='4.6.alpha1'; date='2010-09-15'

What about release notes (in the top-level directory)?

That definitely used to be in this directory, filename HISTORY.txt. Maybe it was getting long? It certainly got out of date quickly. It might be an onus on the release manager to create the notes *before* the release is made, though

Doesn't have to be under revision control (i.e. could be in .hgignore); it IMHO shouldn't be too lengthy, perhaps just contain the most recent changes (e.g. tickets merged since last final, with a reference to a complete version elsewhere).

  • usually that takes a little time, and then it somehow gets added to the official version at sagemath.org (and a few other places?). The problem is that this isn't fully automated yet.

I think the release notes as in the announcements are meanwhile fully automatically generated by a script. I just wonder if these are really "human"-(means user-)readable, if we intend that.

comment:32 in reply to: ↑ 31 Changed 9 years ago by kcrisman

Those ideas seem reasonable, but probably I'm not the one to make the call, since I'm not doing the work.

  • usually that takes a little time, and then it somehow gets added to the official version at sagemath.org (and a few other places?). The problem is that this isn't fully automated yet.

I think the release notes as in the announcements are meanwhile fully automatically generated by a script. I just wonder if these are really "human"-(means user-)readable, if we intend that.

The hard part is, but at least in theory the "known issues" and "new features" sections and things like that are supposed to be human-generated. mvngu used to make great categorized ones, but likely hasn't had the time lately. I think they are more human-readable than some I've seen in other programs, though :)

comment:33 Changed 9 years ago by jhpalmieri

I think that for release notes, we could just have a document which says something like "Please see http://sagemath.org/mirror/src/changelogs/sage-$VERSION.txt for a list of recent changes." Or we could use the link "http://sagemath.org/mirror/src/changelogs/" and then we wouldn't have to update it. This also provides easy access to older changelogs (which the $VERSION.txt doesn't, and notice it's just a text file -- no links to the parent directory or older changelogs). Opinions? We could also add the link "http://wiki.sagemath.org/ReleaseTours/", although this hasn't been updated in a while.

I like the generic option better. But notice that this file won't be auto-generated by "sage -sdist" or any other script, and it should probably be under revision control. This is not the right ticket for adding new files to the top directory, or for modifying existing files, so I think this should go elsewhere.

I also agree that a VERSION.txt file is a good idea. Since we can automatically generate this, and since it shouldn't be under revision control, I think we can do it on this ticket. It should just require modifying sage-sdist and sage-bdist. I'll try towork on this some time soon.

comment:34 Changed 9 years ago by leif

As a first step, just add VERSION.txt to .hgignore (here). ;-)

My intention (perhaps as a developer) regarding "release notes" (whatever the file is called) also was not to have to search trac or some web page(s) (or query Mercurial) just to see which tickets have recently been merged...

For users, of course a more abstract description would be better (bugs fixed, packages newly included or upgraded, new features etc.)

comment:35 Changed 9 years ago by jhpalmieri

I've opened up #9922 for adding release notes. For VERSION.txt, I think we (meaning me) should do the whole thing on this ticket: add it to .hgignore, create it in sage-sdist, and make sure it gets copied by sage-bdist, or else the whole thing should go on another ticket. If I don't get to it on this one, then I'll open a ticket describing the steps (including adding it to .hgignore) -- I don't really see the point in doing one piece of it here.

comment:36 Changed 9 years ago by mpatel

Should we update VERSION.txt after successful upgrades? Or maybe have separate fields for the original version (and whether it was a binary) and the current version? Or even keep a brief upgrade history?

comment:37 Changed 9 years ago by jhpalmieri

I've attached a new version of hg_script which rename 'makefile' to 'Makefile' if it hasn't already been done. (I thought I also had to modify root-spkg-install, but I don't think I do: once the repo has been made via hg_script, it will have 'Makefile' in it, not 'makefile'.)

I also think mpatel has a good point about VERSION.txt. I think there are possible design decisions to be made about how to create that file, what should go in it, how to modify it, etc., so I think it should go another ticket. It could piggy-back on #9922, I think. I'm going to change the description of that ticket to include this.

comment:38 Changed 9 years ago by ddrake

Regarding all this VERSION.txt stuff, I think we should stay on task here. This ticket is about putting existing files under version control. Adding new files "while we're at it" is, IMHO, an unnecessary distraction.

Once SAGE_ROOT is in a Mercurial repo, we can add new files by just creating a ticket, attaching a patch, and getting a positive review. Let's get that repo there first!

Changed 9 years ago by jhpalmieri

fixes problems found by mpatel. apply to scripts repo

comment:39 Changed 9 years ago by jhpalmieri

  • Description modified (diff)

comment:40 Changed 9 years ago by leif

A lot of environment variables should be quoted ($SAGE_ROOT, $TMP, $OPT etc. in most commands).

Needs rebasing in case #9896 gets merged... (or the other way around)

comment:41 Changed 9 years ago by leif

  • Status changed from needs_review to needs_work

root-spkg-install also needs work:

...
    if [ -e makefile ]; then
        cp makefile "$TARGET"/Makefile
    else
        cp Makefile "$TARGET"
    fi
    if [ ! -d "$TARGET/skpg" ]; then
        mkdir "$TARGET/spkg"
    fi
    if [ ! -d "$TARGET/skpg/standard" ]; then
        mkdir "$TARGET/spkg/standard"
...

We'd better use -f than -e, s/skpg/spkg/g.

comment:42 Changed 9 years ago by leif

P.S.: Instead of

if [ ! -d foo ]; then
    mkdir foo
fi

you can simply do

mkdir -p foo

comment:43 Changed 9 years ago by jhpalmieri

  • Description modified (diff)
  • Status changed from needs_work to needs_review

Okay, I've fixed root-spkg-install, although I could probably just have one line

mkdir -p "$TARGET/spkg/standard"

instead of two

mkdir -p "$TARGET/spkg"
mkdir -p "$TARGET/spkg/standard"

I've also quoted every variable in sight in the scripts patch.

comment:44 Changed 9 years ago by leif

Okay, test -e shouldn't be a problem with bash, only (despite being POSIX) with some Solaris' /bin/sh.

comment:45 Changed 9 years ago by leif

You don't have to quote variable expansions in normal assignments like foo=$BAR or echos, but never mind.

Also, test -d foo is superfluous if you rm -rf foo. ("force" also implies not raising an error if the directory doesn't exist.)

I'll have to take a closer look regarding (quoting) $OPT; this might be wrong in some cases.

comment:46 Changed 9 years ago by leif

There are some quotes missing at the end of sage-make_devel_packages (in the newly included part.)

Quoting $OPT currently works, since it is

OPT="pPR"

but it is a bad idea to omit the dash(es) in OPT and prepend it to the expansion.

I.e., it should be

OPT="-pPR"

...

cp $OPT ... # NOT quoted

...

cp -L $OPT ... # also NOT quoted

And I'd suggest renaming OPT to CP_OPTS.

comment:47 Changed 9 years ago by jhpalmieri

There are some quotes missing at the end of sage-make_devel_packages (in the newly included part.)

Oops, I don't know how I missed those. Anyway, here are new versions. This changes "OPT" to "CP_OPT" and includes the hyphen.

comment:48 Changed 9 years ago by jhpalmieri

I'm attaching new versions of root-spkg-install and hg_script which omit the file SAGE_ROOT/sage-python. I asked about this on sage-devel and no one demanded that it be kept.

Changed 9 years ago by jhpalmieri

script to initialize repository. for use by the release manager.

Changed 9 years ago by jhpalmieri

the file SAGE_ROOT/spkg/root-spkg-install

comment:49 follow-up: Changed 9 years ago by leif

Ok, except that hg_script won't work if $SAGE_ROOT contains spaces, and "sage" should be "Sage" in the messages, the patches and attached files now look fine (with the one exception below).

In my opinion more exit codes should be checked (of hg, tar and python), but most of these omissions have been in before, so it is at least "consistent". ;-) (And some tar operations are verbose, while others are not. I also think the release date should be UTC or at least contain the [time and] time zone / UTC offset.)

But sage-upgrade should in any case check that

./pipestatus "sage-spkg $ROOT_REPO 2>&1" "tee -a $SAGE_ROOT/spkg/logs/$ROOT_REPO.log"

worked before calling ./install.

I also wonder if this shouldn't (yet) be

./pipestatus "sage-spkg $ROOT_REPO 2>&1" "tee -a \"$SAGE_ROOT\"/spkg/logs/$ROOT_REPO.log"

(Not tested; the side effects of pipestatus are quite weird.)

I haven't yet applied the patches or fully checked the functionality; at least I didn't find errors in the latest attachments. :)

Btw, why aren't base packages subject to upgrading? (I would have expected the root spkg there.)

comment:50 in reply to: ↑ 49 ; follow-up: Changed 9 years ago by leif

Replying to leif:

I also wonder if this shouldn't (yet) be

./pipestatus "sage-spkg $ROOT_REPO 2>&1" "tee -a \"$SAGE_ROOT\"/spkg/logs/$ROOT_REPO.log"

(Not tested; the side effects of pipestatus are quite weird.)

The above doesn't work; it should be

./pipestatus "sage-spkg $ROOT_REPO 2>&1" "tee -a $$SAGE_ROOT/spkg/logs/$ROOT_REPO.log"

(since SAGE_ROOT is in the environment anyway).

comment:51 Changed 9 years ago by leif

Ouch. Forget my last comment... (i.e. the "solution") :D

comment:52 in reply to: ↑ 50 Changed 9 years ago by leif

Finally :) it should be:

./pipestatus "sage-spkg $ROOT_REPO 2>&1" "tee -a '$SAGE_ROOT'/spkg/logs/$ROOT_REPO.log"

comment:53 follow-up: Changed 9 years ago by jhpalmieri

Replying to leif:

Ok, except that hg_script won't work if $SAGE_ROOT contains spaces,

Since this script only gets run once, by the release manager when this ticket is merged, and since it doesn't get distributed with Sage, I'm not going to worry about it. When Sage works with spaces in the path, if this hasn't been merged yet, I can fix it then.

and "sage" should be "Sage" in the messages,

I'll fix that. You mean messages like "Error building the sage source code package" or "Error copying sage root repository", right?

In my opinion more exit codes should be checked (of hg, tar and python), but most of these omissions have been in before, so it is at least "consistent". ;-) (And some tar operations are verbose, while others are not. I also think the release date should be UTC or at least contain the [time and] time zone / UTC offset.)

But sage-upgrade should in any case check that

./pipestatus "sage-spkg $ROOT_REPO 2>&1" "tee -a $SAGE_ROOT/spkg/logs/$ROOT_REPO.log"

worked before calling ./install.

You're right, I thought about this before, but didn't do it.

Btw, why aren't base packages subject to upgrading? (I would have expected the root spkg there.)

I'm not sure what you mean by "base packages". The sage-upgrade script will certainly upgrade the sage-root spkg. Let's see, the sage-update script will download any new spkg, including any sage-root spkg. Then it gets installed in sage-upgrade before the other spkgs, in case it changes deps or spkg/install.

Should I create an upgrade path on sage.math? I may wait a day or two for the disk situation to settle down...

Re the "pipestatus" command, I'm going to leave it as is and see what happens during testing. (When I tested a month ago or so, the sage-root spkg got upgraded, and I vaguely remember checking that the log file had the right name, but I'll test again.)

comment:54 in reply to: ↑ 53 Changed 9 years ago by leif

Replying to jhpalmieri:

Replying to leif:

Ok, except that hg_script won't work if $SAGE_ROOT contains spaces,

Since this script only gets run once, by the release manager when this ticket is merged, and since it doesn't get distributed with Sage, I'm not going to worry about it.

Yes, this is (and was meant) a non-issue, I only mentioned it for completeness. ;-)

and "sage" should be "Sage" in the messages,

I'll fix that. You mean messages like "Error building the sage source code package" or "Error copying sage root repository", right?

Yes.

Btw, why aren't base packages subject to upgrading? (I would have expected the root spkg there.)

I'm not sure what you mean by "base packages".

The packages /files in spkg/base, also referred to by $(BASE) in the Makefile (deps).

The sage-upgrade script will certainly upgrade the sage-root spkg.

Yes, of course. (sage-update currently doesn't attempt to upgrade anything from spkg/base.)

Should I create an upgrade path on sage.math? I may wait a day or two for the disk situation to settle down...

I have no opinion on that.

Re the "pipestatus" command, I'm going to leave it as is and see what happens during testing. (When I tested a month ago or so, the sage-root spkg got upgraded, and I vaguely remember checking that the log file had the right name, but I'll test again.)

This again just referred to spaces in SAGE_ROOT. (My last suggestions would work with such.)

Changed 9 years ago by jhpalmieri

apply to scripts repo

Changed 9 years ago by jhpalmieri

for reference only: diff between v2 and v3 patches.

comment:55 Changed 9 years ago by jhpalmieri

Here are new versions; the only difference with any content (i.e., other than capitalization, I think) is from sage-upgrade:

-./pipestatus "sage-spkg $ROOT_REPO 2>&1" "tee -a $SAGE_ROOT/spkg/logs/$ROOT_REPO.log"
+./pipestatus "sage-spkg $ROOT_REPO 2>&1" "tee -a '$SAGE_ROOT'/spkg/logs/$ROOT_REPO.log"
+
+if [ $? -ne 0 ]; then
+    echo "Error installing Sage root repository."
+    exit 1
+fi

I don't know why base packages don't get upgraded. That's for another ticket.

Should I create an upgrade path on sage.math?

I meant so that people could test "sage -upgrade". I'll do this pretty soon.

comment:56 Changed 9 years ago by jdemeyer

  • Milestone changed from sage-4.6 to sage-4.6.1
  • Reviewers set to Leif Leonhardy

Changed 9 years ago by ddrake

rebased for 4.6.1.alpha0

Changed 9 years ago by ddrake

rebased for 4.6.1.alpha0

comment:57 Changed 9 years ago by ddrake

I rebased the above two patches. The main Sage repo one is fine, but the scripts patch had rejects on some huge hunks and it took a while to manually put them in. Someone should check over the "-rebased" patch and make sure I did everything correctly.

comment:58 Changed 9 years ago by jhpalmieri

Hi Dan,

Thanks for working on this. The Sage repo patch looks good. Skimming the scripts patch, the only issue I see with the rebasing is that I think $PKGDIR should be quoted in the following (this is lines 39-41 in sage-sdist):

cp "$SAGE_ROOT"/local/bin/sage-spkg $PKGDIR/base/ 
cp "$SAGE_ROOT"/local/bin/sage-env  $PKGDIR/base/ 
cp "$SAGE_ROOT"/local/bin/sage-make_relative $PKGDIR/base/ 

I'll look at this more carefully when I have time.

comment:59 Changed 9 years ago by jhpalmieri

  • Description modified (diff)

I also found the following corrections to be added to the part of the patch for sage-bdist:

  • sage-bdist

    diff -r fa7bc24587ef sage-bdist
    a b  
    4646if [ $? -ne 0 ]; then
    4747    echo "Error copying Sage root repository."
    4848    exit 1
     49fi
    4950
    5051rm "$TMP"/.hg/hgrc
    5152echo "Done copying root repository."
    5253
    53 mkdir "$TMP"
    54 
    5554cd "$SAGE_ROOT"
    5655
    5756echo "Copying files over to tmp directory"
    58 cp -$CP_OPT examples local data "$TMP"/
     57cp $CP_OPT examples local data "$TMP"/
    5958
    6059if [ -d devel/sage-main ]; then
    6160   echo "Copying Sage library over"
     
    6362   cp -L $CP_OPT devel/sagenb-main "$TMP"/devel/sagenb-main
    6463   cp -L $CP_OPT devel/sage-main "$TMP"/devel/sage-main
    6564   cd "$TMP"/devel
    66    cd $TMP/devel
    6765   ln -s sage-main sage
    6866   ln -s sagenb-main sagenb
    6967   cd sage

I'm posting a "v4" patch including this and the above change in sage-sdist (quoting "$PKGDIR").

Changed 9 years ago by jhpalmieri

rebased for 4.6.1.alpha0

comment:60 Changed 9 years ago by ddrake

I used my rebased patches and 4.6.1.alpha0 to initialize the root repo; I sdisted that and the resulting thing has the correct root repo and built fine -- so everything here seems to work "going forward"; I haven't tested any upgrading yet.

comment:61 Changed 9 years ago by jhpalmieri

I'm working on providing an upgrade path (or two) for testing. I'll post here when I have links.

comment:62 follow-up: Changed 9 years ago by jhpalmieri

Okay for testing upgrading:

./sage -upgrade http://sage.math.washington.edu/home/palmieri/misc/9433/sage-4.6.1.9433.alpha0/

This first upgrade path is just a vanilla version of 4.6.1.alpha0, plus the new root repo.

./sage -upgrade http://sage.math.washington.edu/home/palmieri/misc/9433/sage-4.6.1.9433.alpha1/

The second upgrade path contains everything from the first one, plus a change to the top-level README.txt file. This is so we can test upgrading to a plain root repo, then from there to a modified version. So upgrading to "...9433.alpha0" followed by "...9433.alpha1" should work, as well as just upgrading straight to "...9433.alpha1".

I think that if you upgrade from a version with a Sage root repo to either of these, you may be asked twice if you want to upgrade. I mentioned this issue above.

comment:63 follow-up: Changed 9 years ago by jhpalmieri

One issue: if you upgrade a version of Sage containing "makefile", then you end up with both "makefile" and "Makefile" when you're done. We can just delete "$SAGE_ROOT/makefile" when running the root-spkg-install file. That's easy enough to do; is it the right solution?

comment:64 Changed 9 years ago by leif

Hmmm, this doesn't work for upgrades from older versions (<=4.5), and as you mentioned, behaves strange on newer versions >=4.5.1.

We have to download spkg/install (and I'd prefer the rest, too) in sage-update (the Python script).

What's currently newly made in sage-upgrade should be moved to that file, which is in any case the fresh, downloaded one. There we have to make a distinction on if we are upgrading.

If so, spkg/install should then install the root repo (preferably without overwriting itself, but the root repo should, i.e. must, contain exactly the same file), since our "real" Makefile spkg/standard/deps doesn't (though I don't know why you don't want to put the root spkg there).

Btw, it would be better to also have sage-spkg in the root rather than the scripts repo; or download an identical copy (from the scripts repo) along with install, deps etc.

This way we would always do the whole build with the new one, and not switch the version during the installation (after the new scripts spkg has been installed).

Another reason to keep the repos separate, and not as Robert B. suggested on sage-devel, merge them all into an even larger, monolithic one.

(To avoid trouble with getting overwritten, a script can always clone itself and then exec this copy, passing it a parameter such that it knows if it's the clone or the original, similar to fork(). We should IMHO do this for a few scripts involved in upgrading and the build process. Therefore, the chain or stack of called rather than exec'ed scripts shouldn't be deep.)

comment:65 in reply to: ↑ 63 ; follow-up: Changed 9 years ago by leif

Replying to jhpalmieri:

One issue: if you upgrade a version of Sage containing "makefile", then you end up with both "makefile" and "Makefile" when you're done. We can just delete "$SAGE_ROOT/makefile" when running the root-spkg-install file. That's easy enough to do; is it the right solution?

Rename it to Makefile.old?

comment:66 in reply to: ↑ 62 Changed 9 years ago by leif

Replying to jhpalmieri:

I think that if you upgrade from a version with a Sage root repo to either of these, you may be asked twice if you want to upgrade. I mentioned this issue above.

According to the comment you refer to, this should read "upgrade from a version without a Sage root repo".

comment:67 in reply to: ↑ 65 ; follow-up: Changed 9 years ago by jhpalmieri

Replying to leif:

We have to download spkg/install (and I'd prefer the rest, too) in sage-update (the Python script).

I'm not sure what "the rest" refers to here.

What's currently newly made in sage-upgrade should be moved to that file, which is in any case the fresh, downloaded one. There we have to make a distinction on if we are upgrading.

Okay, I think I can do that.

If so, spkg/install should then install the root repo (preferably without overwriting itself, but the root repo should, i.e. must, contain exactly the same file), since our "real" Makefile spkg/standard/deps doesn't (though I don't know why you don't want to put the root spkg there).

I don't want to put it there because in a non-upgrade, it's already installed as part of the download.

Btw, it would be better to also have sage-spkg in the root rather than the scripts repo; or download an identical copy (from the scripts repo) along with install, deps etc.

That's not a bad idea, but it should go on another ticket. What about the other scripts in spkg/base, for example sage-env?

Replying to leif:

Replying to jhpalmieri:

One issue: if you upgrade a version of Sage containing "makefile", then you end up with both "makefile" and "Makefile" when you're done. We can just delete "$SAGE_ROOT/makefile" when running the root-spkg-install file. That's easy enough to do; is it the right solution?

Rename it to Makefile.old?

That sounds good to me.

Replying to leif:

Replying to jhpalmieri:

I think that if you upgrade from a version with a Sage root repo to either of these, you may be asked twice if you want to upgrade. I mentioned this issue above.

According to the comment you refer to, this should read "upgrade from a version without a Sage root repo".

Yes, that's what I meant to say. Sorry for any confusion.

comment:68 in reply to: ↑ 67 ; follow-up: Changed 9 years ago by leif

Replying to jhpalmieri:

Replying to leif:

We have to download spkg/install (and I'd prefer the rest, too) in sage-update (the Python script).

I'm not sure what "the rest" refers to here.

Well, the files that were previously downloaded (deps and newest_version, too). But also downloading sage-env (and e.g. sage-spkg) is not a bad idea.

(I also plan to download upgrade-notes.txt and pre-upgrade-script.sh first, such that the user (and we) can make an informed choice.)


What's currently newly made in sage-upgrade should be moved to that file, which is in any case the fresh, downloaded one. There we have to make a distinction on if we are upgrading.

Okay, I think I can do that.

If so, spkg/install should then install the root repo (preferably without overwriting itself, but the root repo should, i.e. must, contain exactly the same file), since our "real" Makefile spkg/standard/deps doesn't (though I don't know why you don't want to put the root spkg there).

I don't want to put it there because in a non-upgrade, it's already installed as part of the download.

Well, then hg pull would simply be a no-op. But we should check the exit codes of the commands in root-spkg-install (hg and cp) anyway.

hg incoming <source repo> returns 1 if there's nothing to pull, 0 if there are changes to pull, other values on errors, so you could change it to:

...
if [ -d "$TARGET"/.hg ]; then

    # Merge the repository, rather than overwrite changes that the
    # user may have made.
    cd "$TARGET"
    hg ci  # Don't know if we should check in unconditionally,
           # so perhaps move this below the if-elif-fi.
    # should check for errors here
    
    hg incoming "$CUR"  # perhaps redirect output
    if [ $? -eq 1 ]; then
        # No changes to pull
        exit 0
    elif [ $? -ne 0 ]; then
        # Some error, report...
        exit 1
    fi
    # $? = 0: Changes to pull...
    hg pull "$CUR"
    hg merge tip
    hg ci -m "Check-in during upgrade of Sage."
    hg update
    # Add error checks above, too.

else
    ...

(Or you could add a rule to deps that tests its presence / the need to pull first, before calling sage-spkg. But if we one day generate deps, this would be less optimal.)


Btw, it would be better to also have sage-spkg in the root rather than the scripts repo; or download an identical copy (from the scripts repo) along with install, deps etc.

That's not a bad idea, but it should go on another ticket. What about the other scripts in spkg/base, for example sage-env?

Yes, see above. Once we have this ticket in, it will be easier (or at least safer) to make such changes. :-)

P.S.: I wonder if the presence of $SAGE_ROOT/.hg guarantees us a functional Mercurial... ;-)

comment:69 in reply to: ↑ 68 Changed 9 years ago by jhpalmieri

Replying to leif:

Downloading these various files in sage-update seems to at least partly defeat the purpose of the new repo, but anyway.

Replying to jhpalmieri:

I don't want to put it [deps] because in a non-upgrade, it's already installed as part of the download.

Well, then hg pull would simply be a no-op. But we should check the exit codes of the commands in root-spkg-install (hg and cp) anyway.

If we put it in deps, I guess it gets installed at the end? I was tempted to make it part of BASE, but then during an upgrade, any changes to the Sage root repo would mean that everything would be rebuilt, which would be annoying. The only issue is if there is an upgrade to a script like "pipestatus" which is in the root repo, is used in the upgrade process, and is not downloaded in sage-update.

P.S.: I wonder if the presence of $SAGE_ROOT/.hg guarantees us a functional Mercurial... ;-)

I think I'll switch to running "hg verify".

comment:70 Changed 9 years ago by jhpalmieri

  • Description modified (diff)

Okay, here's a new scripts patch (with no changes to sage-update or sage-upgrade), along with new "deps" and "install". I've updated the upgrade paths I listed above; I'm adding them to the ticket description. I am still building Sage 4.4; when that's done, I'll test upgrading it.

comment:71 Changed 9 years ago by jhpalmieri

  • Description modified (diff)

Changed 9 years ago by jhpalmieri

the file SAGE_ROOT/spkg/install

Changed 9 years ago by jhpalmieri

diff between current install and new one; for reference only

Changed 9 years ago by jhpalmieri

the file SAGE_ROOT/spkg/root-spkg-install

Changed 9 years ago by jhpalmieri

patch for scripts repo

comment:72 Changed 9 years ago by jhpalmieri

  • Description modified (diff)

Changed 9 years ago by jhpalmieri

the file SAGE_ROOT/spkg/standard/deps

Changed 9 years ago by jhpalmieri

diff between current deps and new one; for reference only

Note: See TracTickets for help on using tickets.