Upgrade Mercurial to 1.8.x

[ryan]

Upgrade the Mercurial spkg to 1.8.4 (probably the final 1.8.x release).

spkg: ​http://sage.math.washington.edu/home/keshav/files/mercurial-1.8.4.spkg

[dimpase]

it seems to work for me, but how does one really test it?

[ddrake]

Replying to dimpase:

it seems to work for me, but how does one really test it?

In a Sage shell ("sage -sh"), unpack the spkg and go into the src directory. There, you can build the package (just do make all) and then run Mercurial's built-in tests: make tests.

Interestingly, on a regular Ubuntu machine (on which Mercurial works perfectly fine) some of the tests don't pass; it seems like their tests want to have the unzip utility available.

The .spkg installs fine in Linux, and on t2.math. I'm going to take a look at OS X. Mercurial is very well tested and cross-platform, so I would be very surprised if we have any problems.

[dimpase]

Replying to ddrake:

Replying to dimpase:

it seems to work for me, but how does one really test it?

In a Sage shell ("sage -sh"), unpack the spkg and go into the src directory. There, you can build the package (just do make all) and then run Mercurial's built-in tests: make tests.

Interestingly, on a regular Ubuntu machine (on which Mercurial works perfectly fine) some of the tests don't pass; it seems like their tests want to have the unzip utility available.

The .spkg installs fine in Linux, and on t2.math. I'm going to take a look at OS X. Mercurial is very well tested and cross-platform, so I would be very surprised if we have any problems.

one should at least test that the combinat stuff works. This is one of the most immediate and tricky uses of hg in Sage.

[ddrake]

The spkg builds and installs on t2.math and bsd.math. I ran tests on those two machines and on an Ubuntu machine, and I got some failures on all three...but the failures seem to be the same as what we got with 1.6.4.

I tried the combinat patch queue, and it works the same as it does in 4.6.1.rc1 with Mercurial 1.6.4. The ​upgrade notes don't mention anything that affects our use of Mercurial, so positive review.

[jhpalmieri]

After some issues on #9433, I wonder if this should be reopened. There are some incompatibilities between Mercurial 1.7.3 and earlier versions; see ​this web page. I think the issues is this:

• any new repositories created with 1.7.3 (without including a flag like --config format.dotencode=False) will not be usable by versions before 1.7.

I think a reasonable argument could be made for merging 1.7.3 anyway: when dealing with Sage, it's not a huge requirement to ask people to use the Mercurial distributed with Sage instead of some older version installed elsewhere on their machine.

But perhaps a reasonable argument could be made on the other side: why create repositories which can't be read by Mercurial 1.6.4 or earlier? Should we back out 1.7.3 altogether?

We could also keep 1.7.3 but require all repos to be created with the flag --config format.dotencode=False. I personally think that this is unwieldy.

Any opinions?

[vbraun]

Just to reiterate my point from #9433, I think we should keep the new version. Current distributions (e.g. Fedora 14) have already mercurial 1.7+. If you have a way old mercurial install, on the other extreme, then you are probably forced to use Sage's mercurial already. This is precisely why mercurial is included in Sage.

[jdemeyer]

Volker, I see your point but I have a small preference for delaying this new version of Mercurial until it's more widespread. I also don't see a compelling reason for including Mercurial 1.7.3 (as opposed to 1.6.4).

[jdemeyer]

hg repositories made with 1.7.3 cannot be manipulated using versions of mercurial <1.7. Because of this backwards incompatible change and because I see no compelling reason to upgrade Mercurial, I want to be on the safe side and unmerge #10594 (i.e. revert to Mercurial version 1.6.4) in the sage-4.6.2 release.

In some later Sage release, we can still decide to upgrade Mercurial.

[kini]

Can we upgrade yet, Jeroen? What is probably the final version of Mercurial 1.8.x, i.e. 1.8.4, is already out. Soon we will be three versions behind the latest version. Do we really need a compelling reason to upgrade packages we ship, when possible? As Volker said, anyone who is having trouble keeping up with Mercurial versions is probably using sage -hg anyway.

I'm working on an SPKG which will upgrade Mercurial to 1.8.4 and resolve #11120 and #11121 as well. Since 1.7.3 is already old, I hope you don't mind if I change around the ticket description...

[kini]

The spkg is available ​here. It also fixes #11120 and #11121, as mentioned. I'm CCing William since we talked about this in person and I don't see him on this ticket.

[jdemeyer]

Replying to kini:

Do we really need a compelling reason to upgrade packages we ship, when possible?

I think we do, I don't think we should simply upgrade packages for the sake of upgrading. I think the incompatibility with older versions of Mercurial is a negative point for upgrading Mercurial. Only if there is a particular feature of Mercurial 1.8.4 which is really useful, then I think we should upgrade.

[jhpalmieri]

On first glance, this looks okay. There are a few extraneous files in the spkg:

$ hg status
? .SPKG.txt.swp
? .SPKG.txt.un~
? .spkg-install.un~

So of course they should be cleaned up.

If someone upgrades from an earlier version of mercurial on OS X, should we check for and delete the old file SAGE/local/bin/hgmerge?

[kini]

Replying to jdemeyer:

Replying to kini:

Do we really need a compelling reason to upgrade packages we ship, when possible?

I think we do, I don't think we should simply upgrade packages for the sake of upgrading. I think the incompatibility with older versions of Mercurial is a negative point for upgrading Mercurial. Only if there is a particular feature of Mercurial 1.8.4 which is really useful, then I think we should upgrade.

This is not a backwards-incompatible change in 1.7 -- it is a forward-incompatibility in 1.6.x. In the current state of affairs, the sage scripts cannot properly handle packages created by the user's system Mercurial, for many users whose system Mercurial is "too new" for Sage (and this proportion will only increase with time). If we upgrade Mercurial in Sage, the user's system Mercurial may not be able to handle packages created by the sage scripts, but this is not a problem because in those cases the user can do echo alias hg="sage -hg" >> ~/.bashrc and go on with their life. The proportion of users for which this will be a problem will only decrease with time. Also, rather than being an inconvenience, this will actually improve the user's life, since they have access to a newer version of Mercurial than they would otherwise. If we don't upgrade, we run into situations where you need to clone an entire repository with --config format.dotencode=False before you can run Sage's packaging tools on it, because they all use Sage's Mercurial. The proportion of users who encounter this problem will only increase with time, and I suspect that developers (i.e. the usual users of Mercurial) number in this population with more likelihood than average users.

So upgrading Mercurial would help more than it would hurt. But even if it helped exactly as much as it hurt in terms of this repository format issue, software isn't perfect. Newer versions fix bugs that existed in older versions. While newer versions may include new bugs, those too will be fixed in the future. Sitting tight on an old version of software is not safer than following the latest stable releases. I'm not advocating updating to bleeding-edge nightly builds or anything; 1.8.4 was released on June 1 and will be the final bugfix release before the feature release 1.9 (which will be released on July 1, according to Mercurial's release schedule). This makes it as stable as one could like, I think.

edit: yikes, let me clean out that junk from the spkg, hang on...

[kini]

OK, should be fixed. Sorry about that, that was some vim detritus...

[ryan]

Reviewing.... If anyone else wnats to test this package, by all means help review.

[ryan]

The package install fine and none of the hg self-tests fail. Positive review

[kini]

A full review of this spkg should probably include a runthrough of sage -bdist, sage -sdist, and friends. I can easily believe that hg's self-tests are successful and that Mercurial builds properly from my spkg; what is less clear is whether the newer version of Mercurial breaks any of Sage's various shell scripts that currently use it to do various mysterious things during release management and other such activities. It looks like Jeroen has the spkg in 4.7.1.alpha4 as of right now, though, so I'm guessing he's testing these things at the moment.

[ryan]

setting back to needs review. If jdemeyer is reviewing this, he can set it back to positive review when he is finished.

As I was testing this, my sage installation got corrupted (not related to this review). So I wasn't able finish testing -bdist and -sdist

[kini]

Replying to ryan:

As I was testing this, my sage installation got corrupted (not related to this review). So I wasn't able finish testing -bdist and -sdist

Why not do it on sage.math with a cleanly unpacked tarball? :)

[ryan]

Replying to kini:

Replying to ryan:

As I was testing this, my sage installation got corrupted (not related to this review). So I wasn't able finish testing -bdist and -sdist

Why not do it on sage.math with a cleanly unpacked tarball? :)

Good idea! I will tomorrow :)

[jdemeyer]

Replying to ryan:

setting back to needs review. If jdemeyer is reviewing this, he can set it back to positive review when he is finished.

I don't plan to formally review this.

[ryan]

-bdist and -sdist both run without errors on my machine with the updated spkg.

Any other tests that should be run?

[kini]

Other than those, maybe -merge and -combinat if you're familiar with them. Also, this spkg resolves #11121 and #11120 as well, so those need to be reviewed as well...

[jhpalmieri]

This seems to work, from scratch and in an upgrade, although I've asked some questions on #11120 and #11121.

[jhpalmieri]

I'm happy with this, so I'm setting it to "positive review".

[jhpalmieri]

By the way, I just had the opposite issue to that raised in comment 13: someone prepared an spkg for Sage using their system version of mercurial, and I couldn't check the repo with the version in Sage because the one in Sage is too old. I don't know enough about Mercurial to give instructions about how to repair their repo, so I could only check it by installing the spkg on this ticket.

So I think we should merge this in 4.7.2, not delay it further.

[jdemeyer]

Replying to jhpalmieri:

So I think we should merge this in 4.7.2, not delay it further.

Sure!

[kcrisman]

In a Sage shell ("sage -sh"), unpack the spkg and go into the src directory. There, you can build the package (just do make all) and then run Mercurial's built-in tests: make tests.

There is currently no spkg-check here. If they have tests, we might as well add one. This is #11701.

[jdemeyer]

Could any of the authors here please have a look at #12058: Mercurial should not enable pager by default.