Opened 11 years ago

Closed 8 years ago

Last modified 8 years ago

#3052 closed defect (fixed)

mercurial --> plain text --> mercurial

Reported by: was Owned by: mabshoff
Priority: major Milestone: sage-4.7.2
Component: distribution Keywords:
Cc: Merged in: sage-4.7.2.alpha3
Authors: Keshav Kini Reviewers: Volker Braun
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: #10594 Stopgaps:

Description (last modified by jdemeyer)

Robert Bradshaw has mostly solved this:

I've looked into this some more and it looks like we can completely
reconstruct a repository from the export of all its keywords. The
trick is to use the --exact keyword when importing. This forces it to
apply the given patch to the correct parent (sometimes creating a new
head) and will also correctly import merge patches (removing heads).
Some scripts to do this are up at

http://sage.math.washington.edu/home/robertwb/hg/

I've successfully exported and re-created simple repositories (with
branching) with these scripts, and it works great and preserves all
the history. The only issue is that I can't seem to get it to work
with any repositories older than a certain date. I think the issue is
that mercurial changed the way nodeid's are calculated (and I keep
getting an error "abort: patch is damaged or loses information" which
is thrown when the newly computed nodeid does not match the one in
the patch (command.py:1632 in 0.9.5)). Matt Mackall, any suggestions
on how to cleanly get around this/get the old node-id numbers instead?

- Robert Bradshaw

But there are issues. See the complete thread here:

http://groups.google.com/group/sage-devel/browse_thread/thread/79da4852b8e20851/c4b8e87260f08f96?#c4b8e87260f08f96


Apply trac_3052-makefile.patch or trac_3052-makefile-rebased.patch to the Sage root repository (in $SAGE_ROOT/).

Apply trac_3052-textify.patch to the base repository (in $SAGE_ROOT/spkg/base/).

Attachments (3)

trac_3052-makefile.patch (596 bytes) - added by kini 8 years ago.
apply to $SAGE_ROOT
trac_3052-textify.patch (9.9 KB) - added by kini 8 years ago.
apply to $SAGE_ROOT/spkg/base
trac_3052-makefile-rebased.patch (686 bytes) - added by jdemeyer 8 years ago.

Download all attachments as: .zip

Change History (19)

comment:1 Changed 9 years ago by kini

  • Report Upstream set to N/A

Is this ticket still valid? I'm not sure I understand what exactly needs to be done with this, but it seems to apply to ancient mercurial versions and sage 2.x...

comment:2 Changed 8 years ago by kini

OK, William explained it to me simply - some users (especially, say, corporate environments) might be afraid of viruses in Mercurial's binary history data. This ticket is looking for a way to make our source distribution consist entirely of text files only.

comment:3 Changed 8 years ago by kini

I've written a python program that converts Mercurial 1.0+ bundles to a JSON representation and back. This doesn't produce a series of patch files, but it is a fully 7-bit-clean text format which is human-readable (though not as easy to grasp as a git diff or unified diff). It is also a bit-identically reversible computation and preserves all node IDs. The transformation from repository to bundle is also fully reversible as it is what Mercurial itself uses for pushes and pulls.

Changed 8 years ago by kini

apply to $SAGE_ROOT

Changed 8 years ago by kini

apply to $SAGE_ROOT/spkg/base

comment:4 Changed 8 years ago by kini

  • Status changed from new to needs_review

So here are a couple of patches. Apply as indicated. The python file is not doctested, but most of the functions are pretty side-effectful so I don't see how exactly to do this. It at least does fail the doctester if it can't find the correct things to import, so that much is at least safeguarded (in case Mercurial is too old or too new, for example). Speaking of which, this patch depends on #10594 (merged in 4.7.2.alpha0).

Test this patch by unpacking a source distro tarball, doing make text-expand, make text-collapse, make, and make ptestlong (in that order, of course). Works for me on sage.math.washington.edu .

comment:5 Changed 8 years ago by leif

  • Dependencies set to #10594

comment:6 Changed 8 years ago by vbraun

  • Authors set to Keshav Kini
  • Reviewers set to Volker Braun
  • Status changed from needs_review to positive_review

Sounds good, positive review.

Of course there is still a 35MB spkg containing various OSX Fortran compilers, thats a lot of binary code ;) Hopefully apple will fix their toolchain at one point...

comment:7 Changed 8 years ago by vbraun

  • Description modified (diff)

comment:8 Changed 8 years ago by kini

Thanks for the review! :)

comment:9 Changed 8 years ago by leif

  • Description modified (diff)

comment:10 Changed 8 years ago by leif

Looks like Jeroen's merger doesn't yet support the base repository... :(

comment:11 Changed 8 years ago by leif

  • Merged in set to sage-4.7.2.alpha3
  • Resolution set to fixed
  • Status changed from positive_review to closed

Changed 8 years ago by jdemeyer

comment:12 Changed 8 years ago by jdemeyer

  • Description modified (diff)

comment:13 Changed 8 years ago by leif

I rebased all patches with fuzz 2, but didn't bother about this one because it was so trivial (or obvious). ;-)

comment:14 Changed 8 years ago by kini

BTW this code imports stuff from Mercurial internals so it may need to be updated when we next upgrade Mercurial. Any ideas on how to make a doctest that will break when Mercurial is upgraded so that this will be noticed when the time comes? I guess there's always the trivial

>>> import mercurial.__version__.version
>>> mercurial.__version__.version
'1.8.4'

But even if we use this silly doctest, where should it go? $SAGE_ROOT/spkg/base is not doctested by make ptestlong.

comment:15 Changed 8 years ago by was

Odd tests go in SAGE_ROOT/devel/sage/sage/tests.

comment:16 Changed 8 years ago by leif

Perhaps we could tag such tests # optional - release such they get only run by developers and the release manager(s) or release scripts. Then we could really test whether text-expand and text-collapse are still functional (and not just test for a Mercurial version, which is a bit odd).

Note: See TracTickets for help on using tickets.