Opened 9 years ago
Last modified 9 years ago
#9433 closed enhancement
Put more files under revision control. — at Version 135
Reported by:  jhpalmieri  Owned by:  tbd 

Priority:  blocker  Milestone:  sage4.7 
Component:  distribution  Keywords:  
Cc:  was, ddrake, kcrisman, leif  Merged in:  
Authors:  John Palmieri  Reviewers:  Leif Leonhardy, Volker Braun 
Report Upstream:  N/A  Work issues:  
Branch:  Commit:  
Dependencies:  Stopgaps: 
Description (last modified by )
Put the text files in $SAGE_ROOT
, and also the text files in spkg, under revision control. (See the discussion at the end of #9351.)
Here are the instructions:
 apply trac_9433sagerepo.2.patch
 apply trac_9433scripts.v8.patch to the scripts repository
 move the attached file hgignore to
$SAGE_ROOT/.hgignore
(note that this is a new file)  move the attached file rootspkginstall.v3 to
$SAGE_ROOT/spkg/rootspkginstall
(note that this is a new file)  apply 9433_install.diff to
$SAGE_ROOT/spkg/install
 apply 9433_deps.diff to
$SAGE_ROOT/spkg/standard/deps
Then from $SAGE_ROOT, run the attached script 9433_hg_script.2.sh to create the Mercurial repository.
Testing: see http://sage.math.washington.edu/home/release/sage4.7.alpha0/
Change History (166)
comment:1 Changed 9 years ago by
 Status changed from new to needs_review
comment:2 Changed 9 years ago by
A little explanation: this patch creates a directory "otherscripts" in SAGE_ROOT/local/bin. This new directory contains a brief README.txt and also subdirectories "root" and "spkg". "root" contains the text files from SAGE_ROOT. The only one with any changes is README.txt which explains how these files are under revision control. Similarly, "spkg" contains various text files from SAGE_ROOT/spkg, and the only one with any changes is README.txt.
comment:3 Changed 9 years ago by
This probably needs to be rebased. When people are ready to look at it, let me know and I'll see what I can do.
comment:4 Changed 9 years ago by
New approach, after a discussion on sagedevel: create a new repo at the top level tracking the appropriate files. I'm attaching a new version of the patch for the scripts repo. Someone  the release manager, I guess  also needs to create the top level repo, because I don't know how to do this in such a way that it can be posted on a ticket. Here are the instructions:
 move the attached file "hgignore" to SAGE_ROOT/.hgignore
 cd $SAGE_ROOT
 hg init .
 hg add .hgignore COPYING.txt README.txt makefile sage sagepython
 cd spkg
 hg add README.txt gen_html install pipestatus
 cd standard
 hg add README.txt deps libdist_filelist newest_version
 hg add notes.txt numeric24.2.txt
(I don't know if we really need these last two files, but this is probably not the ticket for making such decisions.) Finally, do
 hg commit
When you run "sage sdist ..." it should add a tag for the new version of Sage.
This does not create a new spkg for the files in SAGE_ROOT, since those files have to be in place when you unpack the sage tar file. But it creates the repository so that people can post patches to the trac server, etc.
comment:5 followup: ↓ 6 Changed 9 years ago by
 Cc was added
Looking with my eyes, this looks good. I don't have time to test right now. The test would be to take a clean Sage, do the above, then do "sage sdist ..." and make sure that in the sdist the above is all still there.
comment:6 in reply to: ↑ 5 Changed 9 years ago by
Replying to was:
The test would be to take a clean Sage, do the above, then do "sage sdist ..." and make sure that in the sdist the above is all still there.
This works for me, but other people should definitely look at it carefully.
comment:7 Changed 9 years ago by
This probably needs work: how will it work with "sage upgrade"?
comment:8 Changed 9 years ago by
 Description modified (diff)
Here's a new version of the patch for the scripts repo. I think this should deal with upgrading: the script "sageupgrade" now runs "sage hg branch" from SAGE_ROOT, and if this fails, it assumes that there is no Mercurial repository and creates it.
comment:9 Changed 9 years ago by
 Cc ddrake added
comment:10 Changed 9 years ago by
comment:12 Changed 9 years ago by
In sagesdist, where you have # copy sage root repo over
, why not just clone the repo? That will take care of copying all the necessary files, and if we add or remove files tracked by the repo, we won't need to mess with sagesdist. I'm thinking that something like
cd $SAGE_ROOT hg clone pull . DEST_DIR
Using pull means that it doesn't use hardlinks in the clone; I *think* there would be no problem with using hardlinks, but it's unlikely to make a big difference. The clone will include a hgrc file that points to where it came from: it would look something like this:
[paths] default = /home/foo/sagewhatever
We could simply delete the file, or just leave it, since it would not negatively affect anything (except running hg pull
from SAGE_ROOT, which you wouldn't do anyway).
So, all the cp p
lines could be just
hg clone pull . $TMP rm $TMP/.hg/hgrc
and files added or removed to the repo would get copied correctly without changing any scripts.
comment:13 Changed 9 years ago by
Note that if you use "hg clone ..." to copy the repo, you have to do it earlier in the process: it apparently needs to be done with an empty directory, so in sagesdist it is now done right after creating $TMP.
comment:14 Changed 9 years ago by
 Cc kcrisman added
comment:15 followup: ↓ 17 Changed 9 years ago by
 Milestone changed from sage4.5.2 to sage4.5.3
 Status changed from needs_review to needs_work
If I
 Build the forthcoming Sage 4.5.2 (which is just 4.5.2.rc1 + #9226) from source.
 Follow the steps in the description.
./sage sdist 4.5.99
hg log
in the "root" repo now gives
changeset: 1:0fea58e94942 tag: tip user: Mitesh Patel <qed777@gmail.com> date: Fri Aug 06 21:40:23 2010 0700 summary: Added tag 4.5.99 for changeset 4c1f4320f743 changeset: 0:4c1f4320f743 tag: 4.5.99 user: Mitesh Patel <qed777@gmail.com> date: Fri Aug 06 21:33:45 2010 0700 summary: Initial Sage "root" repository
The new 4.5.99 builds successfully from source and passes the long doctests. But hg log
in the root repo for 4.5.99 lists just revision 0, and the root repo is missing from a binary distribution made with ./sage bdist 4.5.99
.
comment:16 Changed 9 years ago by
I also noticed
SAGE_ROOT/ipython
andSAGE_ROOT/sageREADMEosx.txt
are missing from the new source and binary distributions. An extra
SAGE_ROOT/sagepython
after unpackingsage4.5.99.tar
.
comment:17 in reply to: ↑ 15 ; followup: ↓ 19 Changed 9 years ago by
 Status changed from needs_work to needs_review
Replying to mpatel:
The new 4.5.99 builds successfully from source and passes the long doctests. But
hg log
in the root repo for 4.5.99 lists just revision 0, and the root repo is missing from a binary distribution made with./sage bdist 4.5.99
.
Right, I didn't do this right. In the new patch, the root repo is not modified at all by sagemake_devel_packages; instead, sagesdist clones it, tags it, and commits the new tag, while sagebdist just clones it.
I also noticed
 SAGE_ROOT/ipython and SAGE_ROOT/sageREADMEosx.txt are missing from the new source and binary distributions.
The missing ipython directory was an oversight. I think I've fixed it. The missing sageREADMEosx.txt was intentional: this should only be included for binary distributions on OS X, and its presence there is taken care of by sagebdist:
if [ "$UNAME" = "Darwin" ]; then ... cp sage/local/bin/sageREADMEosx.txt README.txt ...
Perhaps we can close #6938 if this gets merged?
 An extra SAGE_ROOT/sagepython after unpacking sage4.5.99.tar.
This was intentional. Before this, the file sagepython was stored in the scripts repo and then unpacked to the top level. I'm not sure why this was done, but I wanted to keep the end result as close as possible to what it was before.
comment:18 Changed 9 years ago by
 Description modified (diff)
comment:19 in reply to: ↑ 17 Changed 9 years ago by
I also noticed
 SAGE_ROOT/ipython and SAGE_ROOT/sageREADMEosx.txt are missing from the new source and binary distributions.
The missing ipython directory was an oversight. I think I've fixed it. The missing sageREADMEosx.txt was intentional: this should only be included for binary distributions on OS X, and its presence there is taken care of by sagebdist:
if [ "$UNAME" = "Darwin" ]; then ... cp sage/local/bin/sageREADMEosx.txt README.txt ...Perhaps we can close #6938 if this gets merged?
As one of the people involved on that ticket, that is fine. The problem is that #6938 does not currently have positive review! So I think that would be necessary first, or something else indicating that the solution proposed there is correct. Maybe 'merge' that ticket at the same time as this one, for whatever it's worth.
Sounds like you agree :) In fact, notice that once that is removed, that file will only appear ABOVE the SAGE_ROOT directory, in the place a normal README would occur in a dmg or bundle, so it does work properly (I've tested this numerous times
comment:20 Changed 9 years ago by
 Description modified (diff)
I realized there was another problem with the previous patch: while it would work when upgrading from a Sage build with no root repo, it wouldn't do anything when upgrading a Sage build with an existing root repo. The new patch constructs a sage_root spkg file, but this file only gets used during upgrading. So it is installed in the script "sageupgrade", but it does not appear in spkg/standard/deps.
comment:21 Changed 9 years ago by
 Status changed from needs_review to needs_work
This still doesn't deal with upgrading very well. I need to think about how to do this and work on it for a while in a place where I have a better internet connection. I may have something in a few days. Meanwhile, if anyone has suggestions for how to deal with upgrades, I would like to hear them. The issues:
 currently, I don't even know what happens with "sage upgrade ..." if SAGE_ROOT/makefile is changed, for example. Does anything happen?
 does it matter whether we install the new sage_root spkg before or after the sage_scripts spkg? The patch here affects the sageupgrade script, but the new version won't get run during an upgrade anyway.
 is it good enough to just install the new sage_root spkg? I think it might be, but I might be confused. Anyway, I think I need time to play with it and test it out.
comment:22 Changed 9 years ago by
 Description modified (diff)
 Status changed from needs_work to needs_review
This seems to work for me, with one slight glitch: if I run "sage upgrade" on a copy of sage which does not yet include the root repo, it lists the spkg's to be upgraded and asks me "do you really want to continue", then it does some stuff, and then it lists just the root_repo spkg and asks me again if I want to continue. The issue is that, before it has installed some of the upgraded packages, it doesn't know what to do with the root_repo spkg, so it doesn't get installed the first time through. I don't see any way around this, but it's a onetime problem.
comment:23 Changed 9 years ago by
 Cc leif added
Ooops, I once had been aware of this ticket, but forgot to cc me...
Before doing this, can we rename makefile
to Makefile
?
comment:24 Changed 9 years ago by
Before doing this, can we rename makefile to Makefile?
I have no problems with that. Are there ever any (good) reasons for using makefile
instead of Makefile
?
comment:25 followup: ↓ 26 Changed 9 years ago by
Should sageREADMEosx.txt
be ignored (i.e. not be under revision control)?
Also a candidate for renaming (README.MacOS_X.txt
or alike).
From the attached hg_script
:
... $hg add .hgignore .hgtags COPYING.txt README.txt makefile sage sagepython cd ipython $hg add *.py ipythonrc* ...
sagepython
and ipython/
should IMHO be in .hgignore
, since these are not in $SAGE_ROOT
until Sage is built.
comment:26 in reply to: ↑ 25 ; followups: ↓ 27 ↓ 28 Changed 9 years ago by
Replying to leif:
Should
sageREADMEosx.txt
be ignored (i.e. not be under revision control)?
It shouldn't be present at all, and with the patches here, it isn't. See #6938 and also my comment above. (It should only appear in the parent directory of SAGE_ROOT when you make a dmg file on a Mac.)
sagepython
andipython/
should IMHO be in.hgignore
, since these are not in$SAGE_ROOT
until Sage is built.
This is another change on this ticket: these files are now tracked in this repo, not in sage_scripts anymore. Actually, I don't know if ipython was tracked anywhere, but now it is. I don't know who chose to include sagepython in the toplevel directory, but since it's there, it should be tracked properly. (It's a simple enough script I don't mind having two copies of it.) I personally don't see the point of it and think it should be removed, but that should probably happen on another ticket. Or maybe it should be a link to local/bin/sagepython?
comment:27 in reply to: ↑ 26 Changed 9 years ago by
Replying to jhpalmieri:
Replying to leif:
Should
sageREADMEosx.txt
be ignored (i.e. not be under revision control)?It shouldn't be present at all, and with the patches here, it isn't. See #6938 and also my comment above. (It should only appear in the parent directory of SAGE_ROOT when you make a dmg file on a Mac.)
Correct.
comment:28 in reply to: ↑ 26 Changed 9 years ago by
Replying to jhpalmieri:
Replying to leif:
Should
sageREADMEosx.txt
be ignored (i.e. not be under revision control)?It shouldn't be present at all, and with the patches here, it isn't. See #6938 and also my comment above. (It should only appear in the parent directory of SAGE_ROOT when you make a dmg file on a Mac.)
Missed that.
sagepython
andipython/
should IMHO be in.hgignore
, since these are not in$SAGE_ROOT
until Sage is built.This is another change on this ticket: these files are now tracked in this repo, not in sage_scripts anymore.
That, too... :/
Actually, I don't know if ipython was tracked anywhere, but now it is. I don't know who chose to include sagepython in the toplevel directory, but since it's there, it should be tracked properly. (It's a simple enough script I don't mind having two copies of it.) I personally don't see the point of it and think it should be removed, but that should probably happen on another ticket. Or maybe it should be a link to local/bin/sagepython?
Perhaps...
Sorry for the noise.
comment:29 followup: ↓ 30 Changed 9 years ago by
I'd also like to have a VERSION.txt
in SAGE_ROOT/
, best containing just the "plain" version number (perhaps only with also the release date added); cf. #9434 (the number is a coincidence!).
What about release notes (in the toplevel directory)?
comment:30 in reply to: ↑ 29 ; followup: ↓ 31 Changed 9 years ago by
Replying to leif:
I'd also like to have a
VERSION.txt
inSAGE_ROOT/
, best containing just the "plain" version number (perhaps only with also the release date added); cf. #9434 (the number is a coincidence!).
That's a good idea. Be sure not to hold this one up too long ;)
What about release notes (in the toplevel directory)?
That definitely used to be in this directory, filename HISTORY.txt
. Maybe it was getting long? It certainly got out of date quickly. It might be an onus on the release manager to create the notes *before* the release is made, though  usually that takes a little time, and then it somehow gets added to the official version at sagemath.org (and a few other places?). The problem is that this isn't fully automated yet.
comment:31 in reply to: ↑ 30 ; followup: ↓ 32 Changed 9 years ago by
Replying to kcrisman:
Replying to leif:
I'd also like to have a
VERSION.txt
inSAGE_ROOT/
, best containing just the "plain" version number (perhaps only with also the release date added); cf. #9434 (the number is a coincidence!).That's a good idea. Be sure not to hold this one up too long ;)
I don't think adding this needs great effort... We already have devel/sage/sage/version.py
, which looks like this:
"""nodoctests""" version='4.6.alpha1'; date='20100915'
What about release notes (in the toplevel directory)?
That definitely used to be in this directory, filename
HISTORY.txt
. Maybe it was getting long? It certainly got out of date quickly. It might be an onus on the release manager to create the notes *before* the release is made, though
Doesn't have to be under revision control (i.e. could be in .hgignore
); it IMHO shouldn't be too lengthy, perhaps just contain the most recent changes (e.g. tickets merged since last final, with a reference to a complete version elsewhere).
 usually that takes a little time, and then it somehow gets added to the official version at sagemath.org (and a few other places?). The problem is that this isn't fully automated yet.
I think the release notes as in the announcements are meanwhile fully automatically generated by a script. I just wonder if these are really "human"(means user)readable, if we intend that.
comment:32 in reply to: ↑ 31 Changed 9 years ago by
Those ideas seem reasonable, but probably I'm not the one to make the call, since I'm not doing the work.
 usually that takes a little time, and then it somehow gets added to the official version at sagemath.org (and a few other places?). The problem is that this isn't fully automated yet.
I think the release notes as in the announcements are meanwhile fully automatically generated by a script. I just wonder if these are really "human"(means user)readable, if we intend that.
The hard part is, but at least in theory the "known issues" and "new features" sections and things like that are supposed to be humangenerated. mvngu used to make great categorized ones, but likely hasn't had the time lately. I think they are more humanreadable than some I've seen in other programs, though :)
comment:33 Changed 9 years ago by
I think that for release notes, we could just have a document which says something like "Please see http://sagemath.org/mirror/src/changelogs/sage$VERSION.txt for a list of recent changes." Or we could use the link "http://sagemath.org/mirror/src/changelogs/" and then we wouldn't have to update it. This also provides easy access to older changelogs (which the $VERSION.txt doesn't, and notice it's just a text file  no links to the parent directory or older changelogs). Opinions? We could also add the link "http://wiki.sagemath.org/ReleaseTours/", although this hasn't been updated in a while.
I like the generic option better. But notice that this file won't be autogenerated by "sage sdist" or any other script, and it should probably be under revision control. This is not the right ticket for adding new files to the top directory, or for modifying existing files, so I think this should go elsewhere.
I also agree that a VERSION.txt file is a good idea. Since we can automatically generate this, and since it shouldn't be under revision control, I think we can do it on this ticket. It should just require modifying sagesdist and sagebdist. I'll try towork on this some time soon.
comment:34 Changed 9 years ago by
As a first step, just add VERSION.txt
to .hgignore
(here). ;)
My intention (perhaps as a developer) regarding "release notes" (whatever the file is called) also was not to have to search trac or some web page(s) (or query Mercurial) just to see which tickets have recently been merged...
For users, of course a more abstract description would be better (bugs fixed, packages newly included or upgraded, new features etc.)
comment:35 Changed 9 years ago by
I've opened up #9922 for adding release notes. For VERSION.txt, I think we (meaning me) should do the whole thing on this ticket: add it to .hgignore
, create it in sagesdist
, and make sure it gets copied by sagebdist
, or else the whole thing should go on another ticket. If I don't get to it on this one, then I'll open a ticket describing the steps (including adding it to .hgignore
)  I don't really see the point in doing one piece of it here.
comment:36 Changed 9 years ago by
Should we update VERSION.txt
after successful upgrades? Or maybe have separate fields for the original version (and whether it was a binary) and the current version? Or even keep a brief upgrade history?
comment:37 Changed 9 years ago by
I've attached a new version of hg_script which rename 'makefile' to 'Makefile' if it hasn't already been done. (I thought I also had to modify rootspkginstall, but I don't think I do: once the repo has been made via hg_script, it will have 'Makefile' in it, not 'makefile'.)
I also think mpatel has a good point about VERSION.txt. I think there are possible design decisions to be made about how to create that file, what should go in it, how to modify it, etc., so I think it should go another ticket. It could piggyback on #9922, I think. I'm going to change the description of that ticket to include this.
comment:38 Changed 9 years ago by
Regarding all this VERSION.txt stuff, I think we should stay on task here. This ticket is about putting existing files under version control. Adding new files "while we're at it" is, IMHO, an unnecessary distraction.
Once SAGE_ROOT is in a Mercurial repo, we can add new files by just creating a ticket, attaching a patch, and getting a positive review. Let's get that repo there first!
comment:39 Changed 9 years ago by
 Description modified (diff)
comment:40 Changed 9 years ago by
A lot of environment variables should be quoted ($SAGE_ROOT
, $TMP
, $OPT
etc. in most commands).
Needs rebasing in case #9896 gets merged... (or the other way around)
comment:41 Changed 9 years ago by
 Status changed from needs_review to needs_work
rootspkginstall
also needs work:
... if [ e makefile ]; then cp makefile "$TARGET"/Makefile else cp Makefile "$TARGET" fi if [ ! d "$TARGET/skpg" ]; then mkdir "$TARGET/spkg" fi if [ ! d "$TARGET/skpg/standard" ]; then mkdir "$TARGET/spkg/standard" ...
We'd better use f
than e
, s/skpg/spkg/g
.
comment:42 Changed 9 years ago by
P.S.: Instead of
if [ ! d foo ]; then mkdir foo fi
you can simply do
mkdir p foo
comment:43 Changed 9 years ago by
 Description modified (diff)
 Status changed from needs_work to needs_review
Okay, I've fixed rootspkginstall, although I could probably just have one line
mkdir p "$TARGET/spkg/standard"
instead of two
mkdir p "$TARGET/spkg" mkdir p "$TARGET/spkg/standard"
I've also quoted every variable in sight in the scripts patch.
comment:44 Changed 9 years ago by
Okay, test e
shouldn't be a problem with bash
, only (despite being POSIX) with some Solaris' /bin/sh
.
comment:45 Changed 9 years ago by
You don't have to quote variable expansions in normal assignments like foo=$BAR
or echo
s, but never mind.
Also, test d foo
is superfluous if you rm rf foo
. ("force" also implies not raising an error if the directory doesn't exist.)
I'll have to take a closer look regarding (quoting) $OPT
; this might be wrong in some cases.
comment:46 Changed 9 years ago by
There are some quotes missing at the end of sagemake_devel_packages
(in the newly included part.)
Quoting $OPT
currently works, since it is
OPT="pPR"
but it is a bad idea to omit the dash(es) in OPT
and prepend it to the expansion.
I.e., it should be
OPT="pPR" ... cp $OPT ... # NOT quoted ... cp L $OPT ... # also NOT quoted
And I'd suggest renaming OPT
to CP_OPTS
.
comment:47 Changed 9 years ago by
There are some quotes missing at the end of sagemake_devel_packages (in the newly included part.)
Oops, I don't know how I missed those. Anyway, here are new versions. This changes "OPT" to "CP_OPT" and includes the hyphen.
comment:48 Changed 9 years ago by
I'm attaching new versions of rootspkginstall
and hg_script
which omit the file SAGE_ROOT/sagepython
. I asked about this on sagedevel and no one demanded that it be kept.
comment:49 followup: ↓ 50 Changed 9 years ago by
Ok, except that hg_script
won't work if $SAGE_ROOT
contains spaces, and "sage" should be "Sage" in the messages, the patches and attached files now look fine (with the one exception below).
In my opinion more exit codes should be checked (of hg
, tar
and python
), but most of these omissions have been in before, so it is at least "consistent". ;) (And some tar
operations are verbose, while others are not. I also think the release date should be UTC or at least contain the [time and] time zone / UTC offset.)
But sageupgrade
should in any case check that
./pipestatus "sagespkg $ROOT_REPO 2>&1" "tee a $SAGE_ROOT/spkg/logs/$ROOT_REPO.log"
worked before calling ./install
.
I also wonder if this shouldn't (yet) be
./pipestatus "sagespkg $ROOT_REPO 2>&1" "tee a \"$SAGE_ROOT\"/spkg/logs/$ROOT_REPO.log"
(Not tested; the side effects of pipestatus
are quite weird.)
I haven't yet applied the patches or fully checked the functionality; at least I didn't find errors in the latest attachments. :)
Btw, why aren't base packages subject to upgrading? (I would have expected the root spkg there.)
comment:50 in reply to: ↑ 49 ; followup: ↓ 52 Changed 9 years ago by
Replying to leif:
I also wonder if this shouldn't (yet) be
./pipestatus "sagespkg $ROOT_REPO 2>&1" "tee a \"$SAGE_ROOT\"/spkg/logs/$ROOT_REPO.log"
(Not tested; the side effects of pipestatus
are quite weird.)
The above doesn't work; it should be
./pipestatus "sagespkg $ROOT_REPO 2>&1" "tee a $$SAGE_ROOT/spkg/logs/$ROOT_REPO.log"
(since SAGE_ROOT
is in the environment anyway).
comment:51 Changed 9 years ago by
Ouch. Forget my last comment... (i.e. the "solution") :D
comment:52 in reply to: ↑ 50 Changed 9 years ago by
Finally :) it should be:
./pipestatus "sagespkg $ROOT_REPO 2>&1" "tee a '$SAGE_ROOT'/spkg/logs/$ROOT_REPO.log"
comment:53 followup: ↓ 54 Changed 9 years ago by
Replying to leif:
Ok, except that
hg_script
won't work if$SAGE_ROOT
contains spaces,
Since this script only gets run once, by the release manager when this ticket is merged, and since it doesn't get distributed with Sage, I'm not going to worry about it. When Sage works with spaces in the path, if this hasn't been merged yet, I can fix it then.
and "sage" should be "Sage" in the messages,
I'll fix that. You mean messages like "Error building the sage source code package" or "Error copying sage root repository", right?
In my opinion more exit codes should be checked (of
hg
,tar
andpython
), but most of these omissions have been in before, so it is at least "consistent". ;) (And sometar
operations are verbose, while others are not. I also think the release date should be UTC or at least contain the [time and] time zone / UTC offset.)But
sageupgrade
should in any case check that
./pipestatus "sagespkg $ROOT_REPO 2>&1" "tee a $SAGE_ROOT/spkg/logs/$ROOT_REPO.log"
worked before calling
./install
.
You're right, I thought about this before, but didn't do it.
Btw, why aren't base packages subject to upgrading? (I would have expected the root spkg there.)
I'm not sure what you mean by "base packages". The sageupgrade
script will certainly upgrade the sageroot spkg. Let's see, the sageupdate script will download any new spkg, including any sageroot spkg. Then it gets installed in sageupgrade before the other spkgs, in case it changes deps or spkg/install.
Should I create an upgrade path on sage.math? I may wait a day or two for the disk situation to settle down...
Re the "pipestatus" command, I'm going to leave it as is and see what happens during testing. (When I tested a month ago or so, the sageroot spkg got upgraded, and I vaguely remember checking that the log file had the right name, but I'll test again.)
comment:54 in reply to: ↑ 53 Changed 9 years ago by
Replying to jhpalmieri:
Replying to leif:
Ok, except that
hg_script
won't work if$SAGE_ROOT
contains spaces,Since this script only gets run once, by the release manager when this ticket is merged, and since it doesn't get distributed with Sage, I'm not going to worry about it.
Yes, this is (and was meant) a nonissue, I only mentioned it for completeness. ;)
and "sage" should be "Sage" in the messages,
I'll fix that. You mean messages like "Error building the sage source code package" or "Error copying sage root repository", right?
Yes.
Btw, why aren't base packages subject to upgrading? (I would have expected the root spkg there.)
I'm not sure what you mean by "base packages".
The packages /files in spkg/base
, also referred to by $(BASE)
in the Makefile (deps
).
The
sageupgrade
script will certainly upgrade the sageroot spkg.
Yes, of course. (sageupdate
currently doesn't attempt to upgrade anything from spkg/base
.)
Should I create an upgrade path on sage.math? I may wait a day or two for the disk situation to settle down...
I have no opinion on that.
Re the "pipestatus" command, I'm going to leave it as is and see what happens during testing. (When I tested a month ago or so, the sageroot spkg got upgraded, and I vaguely remember checking that the log file had the right name, but I'll test again.)
This again just referred to spaces in SAGE_ROOT. (My last suggestions would work with such.)
comment:55 Changed 9 years ago by
Here are new versions; the only difference with any content (i.e., other than capitalization, I think) is from sageupgrade:
./pipestatus "sagespkg $ROOT_REPO 2>&1" "tee a $SAGE_ROOT/spkg/logs/$ROOT_REPO.log" +./pipestatus "sagespkg $ROOT_REPO 2>&1" "tee a '$SAGE_ROOT'/spkg/logs/$ROOT_REPO.log" + +if [ $? ne 0 ]; then + echo "Error installing Sage root repository." + exit 1 +fi
I don't know why base packages don't get upgraded. That's for another ticket.
Should I create an upgrade path on sage.math?
I meant so that people could test "sage upgrade". I'll do this pretty soon.
comment:56 Changed 9 years ago by
 Milestone changed from sage4.6 to sage4.6.1
 Reviewers set to Leif Leonhardy
comment:57 Changed 9 years ago by
I rebased the above two patches. The main Sage repo one is fine, but the scripts patch had rejects on some huge hunks and it took a while to manually put them in. Someone should check over the "rebased" patch and make sure I did everything correctly.
comment:58 Changed 9 years ago by
Hi Dan,
Thanks for working on this. The Sage repo patch looks good. Skimming the scripts patch, the only issue I see with the rebasing is that I think $PKGDIR should be quoted in the following (this is lines 3941 in sagesdist):
cp "$SAGE_ROOT"/local/bin/sagespkg $PKGDIR/base/ cp "$SAGE_ROOT"/local/bin/sageenv $PKGDIR/base/ cp "$SAGE_ROOT"/local/bin/sagemake_relative $PKGDIR/base/
I'll look at this more carefully when I have time.
comment:59 Changed 9 years ago by
 Description modified (diff)
I also found the following corrections to be added to the part of the patch for sagebdist:

sagebdist
diff r fa7bc24587ef sagebdist
a b 46 46 if [ $? ne 0 ]; then 47 47 echo "Error copying Sage root repository." 48 48 exit 1 49 fi 49 50 50 51 rm "$TMP"/.hg/hgrc 51 52 echo "Done copying root repository." 52 53 53 mkdir "$TMP"54 55 54 cd "$SAGE_ROOT" 56 55 57 56 echo "Copying files over to tmp directory" 58 cp $CP_OPT examples local data "$TMP"/57 cp $CP_OPT examples local data "$TMP"/ 59 58 60 59 if [ d devel/sagemain ]; then 61 60 echo "Copying Sage library over" … … 63 62 cp L $CP_OPT devel/sagenbmain "$TMP"/devel/sagenbmain 64 63 cp L $CP_OPT devel/sagemain "$TMP"/devel/sagemain 65 64 cd "$TMP"/devel 66 cd $TMP/devel67 65 ln s sagemain sage 68 66 ln s sagenbmain sagenb 69 67 cd sage
I'm posting a "v4" patch including this and the above change in sagesdist (quoting "$PKGDIR").
comment:60 Changed 9 years ago by
I used my rebased patches and 4.6.1.alpha0 to initialize the root repo; I sdisted that and the resulting thing has the correct root repo and built fine  so everything here seems to work "going forward"; I haven't tested any upgrading yet.
comment:61 Changed 9 years ago by
I'm working on providing an upgrade path (or two) for testing. I'll post here when I have links.
comment:62 followups: ↓ 66 ↓ 81 ↓ 84 Changed 9 years ago by
Okay for testing upgrading:
./sage upgrade http://sage.math.washington.edu/home/palmieri/misc/9433/sage4.6.1.9433.alpha0/
This first upgrade path is just a vanilla version of 4.6.1.alpha0, plus the new root repo.
./sage upgrade http://sage.math.washington.edu/home/palmieri/misc/9433/sage4.6.1.9433.alpha1/
The second upgrade path contains everything from the first one, plus a change to the toplevel README.txt file. This is so we can test upgrading to a plain root repo, then from there to a modified version. So upgrading to "...9433.alpha0" followed by "...9433.alpha1" should work, as well as just upgrading straight to "...9433.alpha1".
I think that if you upgrade from a version with a Sage root repo to either of these, you may be asked twice if you want to upgrade. I mentioned this issue above.
comment:63 followup: ↓ 65 Changed 9 years ago by
One issue: if you upgrade a version of Sage containing "makefile", then you end up with both "makefile" and "Makefile" when you're done. We can just delete "$SAGE_ROOT/makefile" when running the rootspkginstall file. That's easy enough to do; is it the right solution?
comment:64 Changed 9 years ago by
Hmmm, this doesn't work for upgrades from older versions (<=4.5), and as you mentioned, behaves strange on newer versions >=4.5.1.
We have to download spkg/install
(and I'd prefer the rest, too) in sageupdate
(the Python script).
What's currently newly made in sageupgrade
should be moved to that file, which is in any case the fresh, downloaded one. There we have to make a distinction on if we are upgrading.
If so, spkg/install
should then install the root repo (preferably without overwriting itself, but the root repo should, i.e. must, contain exactly the same file), since our "real" Makefile spkg/standard/deps
doesn't (though I don't know why you don't want to put the root spkg there).
Btw, it would be better to also have sagespkg
in the root rather than the scripts repo; or download an identical copy (from the scripts repo) along with install
, deps
etc.
This way we would always do the whole build with the new one, and not switch the version during the installation (after the new scripts spkg has been installed).
Another reason to keep the repos separate, and not as Robert B. suggested on sagedevel, merge them all into an even larger, monolithic one.
(To avoid trouble with getting overwritten, a script can always clone itself and then exec
this copy, passing it a parameter such that it knows if it's the clone or the original, similar to fork()
. We should IMHO do this for a few scripts involved in upgrading and the build process. Therefore, the chain or stack of called rather than exec
'ed scripts shouldn't be deep.)
comment:65 in reply to: ↑ 63 ; followup: ↓ 67 Changed 9 years ago by
Replying to jhpalmieri:
One issue: if you upgrade a version of Sage containing "makefile", then you end up with both "makefile" and "Makefile" when you're done. We can just delete "$SAGE_ROOT/makefile" when running the rootspkginstall file. That's easy enough to do; is it the right solution?
Rename it to Makefile.old
?
comment:66 in reply to: ↑ 62 Changed 9 years ago by
Replying to jhpalmieri:
I think that if you upgrade from a version with a Sage root repo to either of these, you may be asked twice if you want to upgrade. I mentioned this issue above.
According to the comment you refer to, this should read "upgrade from a version without a Sage root repo".
comment:67 in reply to: ↑ 65 ; followup: ↓ 68 Changed 9 years ago by
Replying to leif:
We have to download
spkg/install
(and I'd prefer the rest, too) insageupdate
(the Python script).
I'm not sure what "the rest" refers to here.
What's currently newly made in
sageupgrade
should be moved to that file, which is in any case the fresh, downloaded one. There we have to make a distinction on if we are upgrading.
Okay, I think I can do that.
If so,
spkg/install
should then install the root repo (preferably without overwriting itself, but the root repo should, i.e. must, contain exactly the same file), since our "real" Makefilespkg/standard/deps
doesn't (though I don't know why you don't want to put the root spkg there).
I don't want to put it there because in a nonupgrade, it's already installed as part of the download.
Btw, it would be better to also have
sagespkg
in the root rather than the scripts repo; or download an identical copy (from the scripts repo) along withinstall
,deps
etc.
That's not a bad idea, but it should go on another ticket. What about the other scripts in spkg/base, for example sageenv?
Replying to leif:
Replying to jhpalmieri:
One issue: if you upgrade a version of Sage containing "makefile", then you end up with both "makefile" and "Makefile" when you're done. We can just delete "$SAGE_ROOT/makefile" when running the rootspkginstall file. That's easy enough to do; is it the right solution?
Rename it to
Makefile.old
?
That sounds good to me.
Replying to leif:
Replying to jhpalmieri:
I think that if you upgrade from a version with a Sage root repo to either of these, you may be asked twice if you want to upgrade. I mentioned this issue above.
According to the comment you refer to, this should read "upgrade from a version without a Sage root repo".
Yes, that's what I meant to say. Sorry for any confusion.
comment:68 in reply to: ↑ 67 ; followup: ↓ 69 Changed 9 years ago by
Replying to jhpalmieri:
Replying to leif:
We have to download
spkg/install
(and I'd prefer the rest, too) insageupdate
(the Python script).I'm not sure what "the rest" refers to here.
Well, the files that were previously downloaded (deps
and newest_version
, too). But also downloading sageenv
(and e.g. sagespkg
) is not a bad idea.
(I also plan to download upgradenotes.txt
and preupgradescript.sh
first, such that the user (and we) can make an informed choice.)
What's currently newly made in
sageupgrade
should be moved to that file, which is in any case the fresh, downloaded one. There we have to make a distinction on if we are upgrading.Okay, I think I can do that.
If so,
spkg/install
should then install the root repo (preferably without overwriting itself, but the root repo should, i.e. must, contain exactly the same file), since our "real" Makefilespkg/standard/deps
doesn't (though I don't know why you don't want to put the root spkg there).I don't want to put it there because in a nonupgrade, it's already installed as part of the download.
Well, then hg pull
would simply be a noop. But we should check the exit codes of the commands in rootspkginstall
(hg
and cp
) anyway.
hg incoming <source repo>
returns 1 if there's nothing to pull, 0 if there are changes to pull, other values on errors, so you could change it to:
... if [ d "$TARGET"/.hg ]; then # Merge the repository, rather than overwrite changes that the # user may have made. cd "$TARGET" hg ci # Don't know if we should check in unconditionally, # so perhaps move this below the ifeliffi. # should check for errors here hg incoming "$CUR" # perhaps redirect output if [ $? eq 1 ]; then # No changes to pull exit 0 elif [ $? ne 0 ]; then # Some error, report... exit 1 fi # $? = 0: Changes to pull... hg pull "$CUR" hg merge tip hg ci m "Checkin during upgrade of Sage." hg update # Add error checks above, too. else ...
(Or you could add a rule to deps
that tests its presence / the need to pull first, before calling sagespkg
. But if we one day generate deps
, this would be less optimal.)
Btw, it would be better to also have
sagespkg
in the root rather than the scripts repo; or download an identical copy (from the scripts repo) along withinstall
,deps
etc.That's not a bad idea, but it should go on another ticket. What about the other scripts in spkg/base, for example sageenv?
Yes, see above. Once we have this ticket in, it will be easier (or at least safer) to make such changes. :)
P.S.: I wonder if the presence of $SAGE_ROOT/.hg
guarantees us a functional Mercurial... ;)
comment:69 in reply to: ↑ 68 Changed 9 years ago by
Replying to leif:
Downloading these various files in sageupdate seems to at least partly defeat the purpose of the new repo, but anyway.
Replying to jhpalmieri:
I don't want to put it [deps] because in a nonupgrade, it's already installed as part of the download.
Well, then
hg pull
would simply be a noop. But we should check the exit codes of the commands inrootspkginstall
(hg
andcp
) anyway.
If we put it in deps, I guess it gets installed at the end? I was tempted to make it part of BASE, but then during an upgrade, any changes to the Sage root repo would mean that everything would be rebuilt, which would be annoying. The only issue is if there is an upgrade to a script like "pipestatus" which is in the root repo, is used in the upgrade process, and is not downloaded in sageupdate.
P.S.: I wonder if the presence of
$SAGE_ROOT/.hg
guarantees us a functional Mercurial... ;)
I think I'll switch to running "hg verify".
comment:70 Changed 9 years ago by
 Description modified (diff)
Okay, here's a new scripts patch (with no changes to sageupdate or sageupgrade), along with new "deps" and "install". I've updated the upgrade paths I listed above; I'm adding them to the ticket description. I am still building Sage 4.4; when that's done, I'll test upgrading it.
comment:71 Changed 9 years ago by
 Description modified (diff)
comment:72 Changed 9 years ago by
 Description modified (diff)
comment:73 Changed 9 years ago by
For what it's worth, I've done the following successfully with the current versions:
 build from scratch ("./sage sdist ..." produced the tar file http://sage.math.washington.edu/home/palmieri/misc/9433/sage4.6.1.9433.alpha0.tar)
 upgrade from 4.6.1.alpha0, and then upgrade again to the version with a modified root repo
 same, but started from 4.4
This was all on sage.math. I also tested 4.6.1.alpha0 on OS X 10.6 with no problems.
I'm testing "./sage bdist ...", which I forgot to do earlier. I'm also testing a build from scratch on another platform.
comment:74 Changed 9 years ago by
Is this ticket now really ready for review? (if not, please change status to needs_work).
comment:75 followup: ↓ 77 Changed 9 years ago by
 Status changed from needs_review to needs_work
Please also add a rootspkginstall.patch
. I prefer using the patches, not the patched files.
comment:76 Changed 9 years ago by
comment:77 in reply to: ↑ 75 ; followups: ↓ 78 ↓ 79 Changed 9 years ago by
 Status changed from needs_work to needs_review
Replying to jdemeyer:
Please also add a
rootspkginstall.patch
. I prefer using the patches, not the patched files.
I'm not sure what you mean: the file "rootspkginstall" is completely new to this ticket, and is not part of any preexisting repo. So there's nothing to patch. I'm flipping this back to "needs review".
Regarding #10231, I'd like to see more discussion on sagedevel about it. I know that leif disagrees, but I'm also interested in the idea of combining the various nonroot repos (scripts, examples, extcode, and the main Sage library) into one. It seems like the Sage community should decide about both of these issues in sagedevel rather than on a trac ticket.
comment:78 in reply to: ↑ 77 Changed 9 years ago by
 Description modified (diff)
Replying to jhpalmieri:
Replying to jdemeyer:
Please also add a
rootspkginstall.patch
. I prefer using the patches, not the patched files.I'm not sure what you mean: the file "rootspkginstall" is completely new to this ticket, and is not part of any preexisting repo.
In that case, sorry for the noise.
comment:79 in reply to: ↑ 77 ; followup: ↓ 80 Changed 9 years ago by
Replying to jhpalmieri:
Regarding #10231, I'd like to see more discussion on sagedevel about it. I know that leif disagrees, but I'm also interested in the idea of combining the various nonroot repos (scripts, examples, extcode, and the main Sage library) into one. It seems like the Sage community should decide about both of these issues in sagedevel rather than on a trac ticket.
Well, there was some discussion at http://groups.google.com/group/sagedevel/browse_thread/thread/cc30eaf87b283881/c2290991827fec9c?hl=en_US&#c2290991827fec9c.
Personally, I simply don't see the need for merging sage, sage_scripts, extcode, examples,... into one big spkg. So there is some disagreement on this. But this disagreement should not stop us from doing something to which most people seemed to agree: not repackaging extcode and examples with every new Sage version (which is exactly what I want to accomplish with #10231).
comment:80 in reply to: ↑ 79 Changed 9 years ago by
Replying to jdemeyer:
Well, there was some discussion at http://groups.google.com/group/sagedevel/browse_thread/thread/cc30eaf87b283881/c2290991827fec9c?hl=en_US&#c2290991827fec9c.
Not too much discussion of that proposal. You suggested it, and a few people liked the idea. On the other hand, a few people made the counterproposal to merge all of the repositories. If we end up merging everything, then doing #10231 first will make more work: whatever is involved in that, and then the merge, as opposed to just the merge. So I'd like to see more of a consensus about which direction to go before putting much effort into #10231.
This ticket seems like a separate issue, and regardless of everything else, the Sage root repo should remain separate from the others, because it should already be installed when you unpack the Sage tar ball. Perhaps its role should increase, including files like sageenv and sagespkg, but that's for another ticket. I hope this ticket can get a positive review soon, and then if anyone feels like working on #10231 (or on a possible merge of the repos), they can base their work on this ticket and other tickets which touch sagesdist and related files.
comment:81 in reply to: ↑ 62 Changed 9 years ago by
Replying to jhpalmieri:
Okay for testing upgrading:
./sage upgrade http://sage.math.washington.edu/home/palmieri/misc/9433/sage4.6.1.9433.alpha0/
I've tried twice with 4.6 using the above command, and despite the upgrade appearing to work, I get this when I start Sage:
  Sage Version 4.6.1.9433.alpha0, Release Date: 20101106   Type notebook() for the GUI, and license() for information.   ********************************************************************** * * * Warning: this is a prerelease version, and it may be unstable. * * * ********************************************************************** /home/drake/s/9433/sage4.6/local/lib/python2.6/sitepackages/sage/plot/plot3d/implicit_plot3d.py:5: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility from implicit_surface import ImplicitSurface /home/drake/s/9433/sage4.6/local/lib/python2.6/sitepackages/sage/plot/plot3d/implicit_plot3d.py:5: RuntimeWarning: numpy.flatiter size changed, may indicate binary incompatibility from implicit_surface import ImplicitSurface /home/drake/s/9433/sage4.6/local/lib/python2.6/sitepackages/sage/calculus/all.py:20: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility from interpolators import polygon_spline, complex_cubic_spline /home/drake/s/9433/sage4.6/local/lib/python2.6/sitepackages/sage/calculus/all.py:20: RuntimeWarning: numpy.flatiter size changed, may indicate binary incompatibility from interpolators import polygon_spline, complex_cubic_spline /home/drake/s/9433/sage4.6/local/lib/python2.6/sitepackages/sage/stats/hmm/all.py:8: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility from hmm import DiscreteHiddenMarkovModel /home/drake/s/9433/sage4.6/local/lib/python2.6/sitepackages/sage/stats/hmm/all.py:8: RuntimeWarning: numpy.flatiter size changed, may indicate binary incompatibility from hmm import DiscreteHiddenMarkovModel sage:
comment:82 followup: ↓ 83 Changed 9 years ago by
Dan, you have to touch the Cython include file and then do ./sage b
to build the dependencies  see #9808 for details. Or you could do ./sage ba
and wait a long time.
comment:83 in reply to: ↑ 82 Changed 9 years ago by
comment:84 in reply to: ↑ 62 ; followup: ↓ 85 Changed 9 years ago by
Replying to jhpalmieri:
Okay for testing upgrading:
I started with 4.6, and both upgrade paths work fine: first to "9433.alpha0", then to 9433.alpha1, and also directly to 9433.alpha1. I'm currently testing with upgrades from 4.3.5.
comment:85 in reply to: ↑ 84 Changed 9 years ago by
Replying to ddrake:
Replying to jhpalmieri:
Okay for testing upgrading:
I started with 4.6, and both upgrade paths work fine: first to "9433.alpha0", then to 9433.alpha1, and also directly to 9433.alpha1. I'm currently testing with upgrades from 4.3.5.
Both upgrade paths also work when upgrading from 4.3.5. (I'm using 64bit Ubuntu 10.04.)
comment:86 Changed 9 years ago by
I just added "VERSION.txt" to the .hgignore file, for compatibility with #9434.
comment:87 Changed 9 years ago by
 Description modified (diff)
 Milestone changed from sage4.6.1 to sage4.6.2
 Reviewers changed from Leif Leonhardy to Leif Leonhardy, Volker Braun
 Status changed from needs_review to positive_review
After some rediffing I built it successfully on top of Sage4.6.1.rc1 (identical to the Sage4.6.1 release) with the following updated files:
trac_9433scripts.v5.patch
>trac_9433scripts.v5.2.patch
install
>install.2
deps
>deps.2
I've changed the main ticket documentation accordingly.
For reference, here is a list of files in the root repository:
[vbraun@volkertwo sage4.6.1.vb2]$ hg st all  grep v '^I' C .hgignore C .hgtags C COPYING.txt C Makefile C README.txt C ipython/ipy_profile_sh.py C ipython/ipy_user_conf.py C ipython/ipythonrc C ipython/ipythonrcmath C ipython/ipythonrcnumeric C ipython/ipythonrcphysics C ipython/ipythonrcpysh C ipython/ipythonrcscipy C ipython/ipythonrctutorial C sage C spkg/README.txt C spkg/gen_html C spkg/install C spkg/pipestatus C spkg/rootspkginstall C spkg/standard/README.txt C spkg/standard/deps C spkg/standard/libdist_filelist C spkg/standard/newest_version
Really, any kind of root repository would be better than the caveman technology we have in place right now. I read through all the scripts and they do make sense to me. I built my private release using sage sdist <version>
and it compiled fine. You guys put a lot of effort into this ticket to make sure that nothing breaks, and I think it really is the time to integrate this with Sage.
Positive review.
Jeroen, can you plug this into 4.6.2.alpha as soon as possible to give it as much exposure as possible?
comment:88 followup: ↓ 92 Changed 9 years ago by
 Status changed from positive_review to needs_info
Why do we need a sage_root.spkg
? I don't see a reason for a "root repo" to ever exist in tarball form.
comment:89 Changed 9 years ago by
 Description modified (diff)
Changed 9 years ago by
comment:90 Changed 9 years ago by
 Description modified (diff)
comment:91 Changed 9 years ago by
 Status changed from needs_info to needs_work
If we do this, we really should also take care of the following duplicate files (files both in sage_scripts and in another repo):
README.txt spkg/install spkg/base/sagespkg spkg/base/sageenv spkg/base/sagemake_relative spkg/base/sagecheck64 (added by the notyetmerged #10303)
Also, I think we should get rid of the spkg/base
repo and merge it with the new root repo.
comment:92 in reply to: ↑ 88 ; followup: ↓ 93 Changed 9 years ago by
 Status changed from needs_work to needs_review
Replying to jdemeyer:
Why do we need a
sage_root.spkg
? I don't see a reason for a "root repo" to ever exist in tarball form.
The way things are set up right now, if you upgrade an existing sage installation then the updated SAGE_ROOT_REPO
will merge updates to the $SAGE_ROOT
repository. As long as we don't have an official online repository to pull changes from I don't see any better way to do the upgrade. Am I missing something?
I agree that we need to disentangle the sage_root
repository more from sage_scripts
. Basically, everything that sage_scripts/spkginstall
manually copies into $SAGE_ROOT
should be part of the root repo, like README.txt
. But I think this can wait until we actually do have a sage_root
repository. Then it'll be easy to write complimentary patches for the two repositories that clean this up.
I'm also totally in favour of merging the spkg/base repo. Since that has only 26 log entries (with the most recent one from July), I think we can live without preserving its history. Right now we don't use this repository during upgrades as far as I know.
If you agree with this then I'll make a followup ticket...
comment:93 in reply to: ↑ 92 ; followup: ↓ 94 Changed 9 years ago by
 Status changed from needs_review to needs_work
Replying to vbraun:
Replying to jdemeyer:
Why do we need a
sage_root.spkg
? I don't see a reason for a "root repo" to ever exist in tarball form.The way things are set up right now, if you upgrade an existing sage installation then the updated
SAGE_ROOT_REPO
will merge updates to the$SAGE_ROOT
repository. As long as we don't have an official online repository to pull changes from I don't see any better way to do the upgrade.
I have to admit I know nothing about this. If you really think we need a sage_root.spkg
then I believe you...
I agree that we need to disentangle the
sage_root
repository more fromsage_scripts
. Basically, everything thatsage_scripts/spkginstall
manually copies into$SAGE_ROOT
should be part of the root repo, likeREADME.txt
. But I think this can wait until we actually do have asage_root
repository. Then it'll be easy to write complimentary patches for the two repositories that clean this up.
Personally, I would rather like to clean this up as part of this ticket. Keep in mind that adding or changing the working of a SAGE_ROOT repo will require some changes to the process of merging Sage releases. I would prefer to have to do this only once, not once for this ticket and once for every followup ticket.
I'm also totally in favour of merging the spkg/base repo. Since that has only 26 log entries (with the most recent one from July), I think we can live without preserving its history. Right now we don't use this repository during upgrades as far as I know.
If you agree with this then I'll make a followup ticket...
Same answer as before: I prefer to do it in this ticket.
comment:94 in reply to: ↑ 93 ; followup: ↓ 96 Changed 9 years ago by
Replying to jdemeyer:
Replying to vbraun:
Replying to jdemeyer:
Why do we need a
sage_root.spkg
? I don't see a reason for a "root repo" to ever exist in tarball form.The way things are set up right now, if you upgrade an existing sage installation then the updated
SAGE_ROOT_REPO
will merge updates to the$SAGE_ROOT
repository. As long as we don't have an official online repository to pull changes from I don't see any better way to do the upgrade.I have to admit I know nothing about this. If you really think we need a
sage_root.spkg
then I believe you...
Yes, that's the reason for its existence.
I agree that we need to disentangle the
sage_root
repository more fromsage_scripts
. Basically, everything thatsage_scripts/spkginstall
manually copies into$SAGE_ROOT
should be part of the root repo, likeREADME.txt
. But I think this can wait until we actually do have asage_root
repository. Then it'll be easy to write complimentary patches for the two repositories that clean this up.Personally, I would rather like to clean this up as part of this ticket. Keep in mind that adding or changing the working of a SAGE_ROOT repo will require some changes to the process of merging Sage releases. I would prefer to have to do this only once, not once for this ticket and once for every followup ticket.
I think I don't understand. Once there is a SAGE_ROOT repo, then any followup ticket can just have a patch which needs to be applied to that repo, as opposed to manually patching spkg/install
or Makefile
or whatever.
For the particular change you have in mind, it requires modifying various scripts (like spkginstall, but maybe others?) in local/bin, and it could possibly be complicated. It seems safer to do things incrementally, first setting up the new repo, then on a later ticket making further changes.
(I also have a heavy teaching load right now, so I won't have much time to work on this. I can try to fix bugs with the current implementation when anyone finds them, but I don't think I can work on any major modifications, like dealing with the spkg/base repo.)
comment:95 Changed 9 years ago by
I'm with John here: Its easy to generate a patch that adds a new file to the sage_root
repository. You don't have to do anything else for updates, the change will automatically make it into the next sage_root
package. No more manually digging around to make a new source distribution :)
comment:96 in reply to: ↑ 94 ; followup: ↓ 97 Changed 9 years ago by
For the particular change you have in mind, it requires modifying various scripts (like spkginstall, but maybe others?) in local/bin, and it could possibly be complicated. It seems safer to do things incrementally, first setting up the new repo, then on a later ticket making further changes.
And indeed, this is the Sage way of doing things  perfect being the enemy of things ever happening in the open development, open source model.
I'd also like to put in a plug for sageREADMEosx.txt finally being removed from $SAGE_ROOT. Apparently #6938 never got 'merged'...> (I also have a heavy teaching load right now, so I won't have much time to work on this. I can try to fix bugs with the current implementation when anyone finds them, but I don't think I can work on any major modifications, like dealing with the spkg/base repo.)
comment:97 in reply to: ↑ 96 ; followups: ↓ 98 ↓ 99 Changed 9 years ago by
Replying to kcrisman:
And indeed, this is the Sage way of doing things  perfect being the enemy of things ever happening in the open development, open source model.
In general, I completely agree with your sentiment: too many times a patch has not been finished because it was not the "optimal" solution. However, this only applies if the halfwayimplemented patch makes the situation better.
For this particular ticket, I feel that having a halfwayimplemented sage_root repository is strictly worse than having no sage_root repository at all. The situation for merging SAGE_ROOT patches is already complicated enough (but I can manage) and I'm afraid the current patch on this ticket will make it only worse.
This is not a ticket I want to "rush" into Sage.
comment:98 in reply to: ↑ 97 Changed 9 years ago by
Replying to jdemeyer:
This is not a ticket I want to "rush" into Sage.
This ticket has been worked on for 7 months and has almost 100 comments ;)
And every time I want to change something on this ticket I need half a day to rebuild sage, manually make half a dozen changes, test the source distribution.
I can make the changes you wanted in comment:91 but there are a bunch of other files I want to move to the sage_root
repository as well. Some of them might need some discussion on sagedevel first. How many more months should we wait until the main makefile is under revision control?
comment:99 in reply to: ↑ 97 ; followup: ↓ 100 Changed 9 years ago by
Replying to jdemeyer:
For this particular ticket, I feel that having a halfwayimplemented sage_root repository is strictly worse than having no sage_root repository at all.
In what way is the SAGE_ROOT repo "halfwayimplemented"? With these scripts and patches, the repo will exist and work for upgrades and new builds.
The situation for merging SAGE_ROOT patches is already complicated enough (but I can manage)
Maybe you can manage  but there's plenty of evidence that other release managers can't. Changing files in SAGE_ROOT currently takes a long time and is an extremely brittle process; why continue in this way?
And even if there's no trouble actually changing files, we still have no version control for our basic makefile, README, and executable script. The only way to track changes is to download tarballs, unpack them, and compare files. Think about it: neither the basic file with which we build Sage (the makefile) nor the basic file with which one runs Sage (SAGE_ROOT/sage) are under any version control! That is crazy. Imagine if Firefox shipped with their "firefox" script not under version control.
and I'm afraid the current patch on this ticket will make it only worse.
Right now to change the README:
 make a backup copy
 make your changes
 manually run
diff u
to produce a patch  upload both the new README and the patch to a ticket
 upon positive review, hope that the release manager replaces the README file correctly and does
sage sdist
in the right directory
With a SAGE_ROOT repo:
 use the same procedure that one uses in the Sage library to produce a patch.
 release manager uses same procedure as for the Sage library to merge the patch.
Perhaps it's only me, but the current situation seems far worse.
Please, let's create the SAGE_ROOT repo and argue about merging repos after they actually exist.
comment:100 in reply to: ↑ 99 Changed 9 years ago by
Replying to ddrake:
 release manager uses same procedure as for the Sage library to merge the patch.
Wrong, because the README
is actually already in a different repository, namely sage_scripts
. This is the issue of duplicate files that I mentioned above.
comment:101 Changed 9 years ago by
After applying the patches here, README.txt will be in the SAGE_ROOT repo, not the scripts repo. The full list of files which will be tracked:
.hgignore .hgtags COPYING.txt README.txt Makefile sage
 in the ipython directory: *.py, ipythonrc*
 in the spkg directory: README.txt gen_html install pipestatus rootspkginstall
 in the spkg/standard directory: README.txt deps libdist_filelist newest_version
comment:102 followup: ↓ 105 Changed 9 years ago by
The SAGE_ROOT/ipython
directory is created during the Sage build process, so I don't think it should be part of the SAGE_ROOT
repository.
comment:103 Changed 9 years ago by
 Description modified (diff)
comment:104 Changed 9 years ago by
What's the reason for removing the quoting of SAGE_ROOT
in sagespkginstall
?
comment:105 in reply to: ↑ 102 Changed 9 years ago by
Replying to jdemeyer:
The
SAGE_ROOT/ipython
directory is created during the Sage build process, so I don't think it should be part of theSAGE_ROOT
repository.
It's just copied from whatever was there before. Why not track those files in a repository? Otherwise they're not tracked anywhere: what if we want to change any of them? Also, if you decide to not include them in the repo, then you'll need to work on the scripts patch: put back the parts (in sagesdist for example) which create those files.
What's the reason for removing the quoting of SAGE_ROOT in sagespkginstall?
That was presumably just a mistake.
comment:106 followup: ↓ 107 Changed 9 years ago by
The ipython directory is in the sage_scripts
repository and sage_scripts/spkginstall
manually copies it into $SAGE_ROOT
. The patch on this ticket removes that part from sage_scripts/spkginstall
, so it will no longer be copied over.
I'm not sure if the removal of the quotes has any deeper meaning. But righthandsides of variable assignments need not be quoted in shell script:
[vbraun@volkertwo ~]$ x="a b" [vbraun@volkertwo ~]$ y=$x/c [vbraun@volkertwo ~]$ echo $y a b/c
comment:107 in reply to: ↑ 106 Changed 9 years ago by
Replying to vbraun:
The ipython directory is in the
sage_scripts
repository andsage_scripts/spkginstall
manually copies it into$SAGE_ROOT
. The patch on this ticket removes that part fromsage_scripts/spkginstall
, so it will no longer be copied over.
Okay, I see. I got confused by the renaming of sagespkginstall
to spkginstall
, so I didn't realize what the file sagespkginstall
was all about.
I'm not sure if the removal of the quotes has any deeper meaning. But righthandsides of variable assignments need not be quoted in shell script:
[vbraun@volkertwo ~]$ x="a b" [vbraun@volkertwo ~]$ y=$x/c [vbraun@volkertwo ~]$ echo $y a b/c
I agree, but I think that having the quotes is clearer anyway.
This is most certainly a bug (in sagemake_devel_packages
):
+if [ ! f "$PKG/sage_scripts$SAGE_VERSION.spkg" ]; then
comment:108 Changed 9 years ago by
Upon closer investigation I found that the ipython directory is in the sage_scripts
spkg, but ignored in the sage_scripts
mercurial repository. So its even worse :)
comment:109 Changed 9 years ago by
 Description modified (diff)
comment:110 Changed 9 years ago by
Ha, I beat you by 13 seconds! :)
comment:111 Changed 9 years ago by
 Description modified (diff)
comment:112 Changed 9 years ago by
That was probably my typo in rebasing the patch to Sage4.6.1 ;)
comment:113 Changed 9 years ago by
 Description modified (diff)
Changed 9 years ago by
Changed 9 years ago by
comment:114 Changed 9 years ago by
 Status changed from needs_work to positive_review
I'm fine with all changes.
comment:115 followup: ↓ 116 Changed 9 years ago by
 Status changed from positive_review to needs_work
Upgrading from earlier versions of Sage doesn't work because spkg/standard/VERSION.txt
is gone:
$ ./sage upgrade http://sage.math.washington.edu/home/jdemeyer/release/sage4.6.2.sage_root/sage4.6.2.sage_root/ Downloading packages from 'http://sage.math.washington.edu/home/jdemeyer/release/sage4.6.2.sage_root/sage4.6.2.sage_root//spkg'. Reading package lists... Done! The following packages will be upgraded: examples4.6.2.sage_root extcode4.6.2.sage_root sage4.6.2.sage_root sage_root4.6.2.sage_root sage_scripts4.6.2.sage_root ** WARNING: This is a sourcebased upgrade, which could take hours, ** fail, and render your Sage install useless!! Do you want to continue [y/N]? y http://sage.math.washington.edu/home/jdemeyer/release/sage4.6.2.sage_root/sage4.6.2.sage_root//spkg/standard/examples4.6.2.sage_root.spkg > examples4.6.2.sage_root.spkg [..................................................] Deleting old spkg 'examples4.6.1.spkg'... http://sage.math.washington.edu/home/jdemeyer/release/sage4.6.2.sage_root/sage4.6.2.sage_root//spkg/standard/extcode4.6.2.sage_root.spkg > extcode4.6.2.sage_root.spkg [..................................................] Deleting old spkg 'extcode4.6.1.spkg'... http://sage.math.washington.edu/home/jdemeyer/release/sage4.6.2.sage_root/sage4.6.2.sage_root//spkg/standard/sage4.6.2.sage_root.spkg > sage4.6.2.sage_root.spkg [..................................................] Deleting old spkg 'sage4.6.1.spkg'... http://sage.math.washington.edu/home/jdemeyer/release/sage4.6.2.sage_root/sage4.6.2.sage_root//spkg/standard/sage_root4.6.2.sage_root.spkg > sage_root4.6.2.sage_root.spkg [..............] http://sage.math.washington.edu/home/jdemeyer/release/sage4.6.2.sage_root/sage4.6.2.sage_root//spkg/standard/sage_scripts4.6.2.sage_root.spkg > sage_scripts4.6.2.sage_root.spkg [..................................................] Deleting old spkg 'sage_scripts4.6.1.spkg'... http://sage.math.washington.edu/home/jdemeyer/release/sage4.6.2.sage_root/sage4.6.2.sage_root//spkg/standard/VERSION.txt > VERSION.txt [.] Failed to download 'http://sage.math.washington.edu/home/jdemeyer/release/sage4.6.2.sage_root/sage4.6.2.sage_root//spkg/standard/VERSION.txt'. Abort.
comment:116 in reply to: ↑ 115 Changed 9 years ago by
Replying to jdemeyer:
Upgrading from earlier versions of Sage doesn't work because
spkg/standard/VERSION.txt
is gone:
That's because when the scripts patch was rebased, lines like these were removed:
# Put VERSION.txt in a directory available for download during the # update process. (See sageupdate.) cp p VERSION.txt $TMP/$PKGDIR/$STD/
I think that just restoring these lines should fix it .
comment:117 Changed 9 years ago by
 Description modified (diff)
 Status changed from needs_work to needs_review
Here's a new patch for the scripts repo.
comment:118 Changed 9 years ago by
 Status changed from needs_review to positive_review
Yes that fixes it!
comment:119 Changed 9 years ago by
For the record: I successfully upgraded a Sage4.6.1 installation to Sage4.6.2.alpha1.
comment:120 Changed 9 years ago by
 Milestone changed from sage4.6.2 to sage4.7
comment:121 Changed 9 years ago by
 Priority changed from major to blocker
comment:122 Changed 9 years ago by
 Description modified (diff)
Changed 9 years ago by
comment:123 Changed 9 years ago by
 Description modified (diff)
comment:124 Changed 9 years ago by
 Description modified (diff)
 Status changed from positive_review to needs_work
comment:125 Changed 9 years ago by
Testing changes to rootspkginstall...
comment:126 Changed 9 years ago by
It seems that the SAGE_ROOT
repository requires a very recent of Mercurial. With Mercurial version 1.6.4, I get
$ hg version Mercurial Distributed SCM (version 1.6.4) Copyright (C) 20052010 Matt Mackall <mpm@selenic.com> and others This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. $ hg commit abort: requirement 'dotencode' not supported!
I agree that a sufficiently recent version of Mercurial is shipped with Sage, but unless there is a good reason for this dotencode
requirement, I would prefer if the root repo worked with Mercurial 1.3.1 like the other repos.
comment:127 followup: ↓ 132 Changed 9 years ago by
The file spkginstall
from the root repo is not under revision control. Is there a reason for this?
comment:128 Changed 9 years ago by
I'm pretty sure that the rootspkginstall
ought to be under revision control. I'm travelling right now, so I can't test it out myself.
comment:129 Changed 9 years ago by
I don't think there is anything in the repository which should require a new version of Mercurial. According to this page from the Mercurial wiki, if you create a repo with a newer version and then try to access it with an older version, you can see this message. So I don't think it's anything specific about the root repo.
I'll try to look into the rootspkginstall file issue.
comment:130 Changed 9 years ago by
Okay, line 24 in 9433_hg_script.sh
is
( cd spkg && hg add README.txt gen_html install pipestatus rootspkginstall )
So it looks like rootspkginstall should be tracked. Can you explain why you think it's not?
Also, I see that vbraun has a new version of this script which might avoid the dotencode issue...
comment:131 Changed 9 years ago by
Yes, I tripped over the dotencode thing before when working with mercurial. We don't really need that, so its ok to switch it off for backward compatibility, at least for the next year or so.
comment:132 in reply to: ↑ 127 Changed 9 years ago by
 Description modified (diff)
 Status changed from needs_work to needs_review
Replying to jdemeyer:
The file
spkginstall
from the root repo is not under revision control. Is there a reason for this?
I now realize that you're talking about the file spkginstall
in the actual spkg file. This is just a copy of rootspkginstall
made by sagemake_devel_packages
, so we don't need to track it. I'm going to add it to the .hgignore
file (by adding ^spkginstall$
, so it only matches a file with exactly that name at the top level). This change requires review.
Meanwhile, I'm giving Volker's change to the hg_script a positive review: for me, it makes any errors about dotencode go away when I use an older version of Mercurial to access the repo.
comment:133 Changed 9 years ago by
 Status changed from needs_review to needs_work
About the dotencode
requirement: the change in the shell script indeed "fixes" the actual root repository. However, the created sageroot
spkg still has a mercurial repository which requires dotencode
(so, after sdist and make, you need a new version of hg
). Since none of the other spkgs have this behaviour, I think this still needs work.
comment:134 Changed 9 years ago by
 Status changed from needs_work to needs_review
I don't really understand this complaint. After all, Sage is distributed with a version of Mercurial, and it is completely reasonable to require that version for repositories which are part of Sage. It's easy enough to type "sage hg" instead of "hg", or make an alias, or put the Sage version of hg in your $PATH. I think this is more a problem with Mercurial  why is it not backwards compatible? Who merged this version of Mercurial into Sage? Did the spkg maintainers or release manager notice or discuss this incompatibility issue?
Also, if you're going to start inventing rules about how an spkg should be prepared, you might consider putting those rules in the developer's guide, and perhaps discussing them and getting approval for them on sagedevel before imposing them.
We can add config format.dotencode=0
at various places in sagesdist and sagemake_devel_packages, and I'm attaching a new version of the scripts patch to do this, but this is not a viable longterm solution: the version requirements of Mercurial for anyone who wants to make a new spkg for Sage need to be made public, not imposed at anyone's whim.
comment:135 Changed 9 years ago by
 Description modified (diff)
I'm marking this as "needs review", but consider this patch experimental.