Opened 11 years ago

Closed 5 years ago

Last modified 16 months ago

#11720 closed enhancement (duplicate)

regularly refresh the pickle jar

Reported by: boothby Owned by: was
Priority: major Milestone: sage-duplicate/invalid/wontfix
Component: pickling Keywords:
Cc: vbraun Merged in:
Authors: Reviewers:
Report Upstream: N/A Work issues:
Branch: public/11720 (Commits, GitHub, GitLab) Commit:
Dependencies: Stopgaps:

Status badges


The pickle jar hasn't been refreshed in a very long time. We need a system in place that continually adds pickles to the jar, and keeps the old pickles around.

Change History (8)

comment:1 Changed 9 years ago by jdemeyer

  • Milestone changed from sage-5.11 to sage-5.12

comment:2 Changed 8 years ago by vbraun_spam

  • Milestone changed from sage-6.1 to sage-6.2

comment:3 Changed 8 years ago by SimonKing

  • Cc vbraun added

I guess one question that needs to be addressed is: How can we keep track of the pickle jar with git? IICTC, the pickle jar is not under version control, and being a binary file (.tar.bz2) it shouldn't be under version control.

Each item in the pickle jar corresponds to a pair of files. One file is pure text, describing the Sage version used to create the pickle, the class whose instance is pickled, and the expected string representation of the unpickled object. The second file is the actual pickle, .sobj.

Is git able to look into bzipped tar files? If this was the case, then we could at least let git track the text files associated with the pickles.

Another question that needs to be addressed: How to post changes of the pickle jar on a trac ticket? Shall one attach the new version of the pickle jar, leaving a comment for the release manager asking him/her to replace the old jar with the new jar? That's why I put Volker on Cc (Hi Volker...).

And perhaps it would be nice to have a function that automatically creates a new pickle in the jar. Yes, I know, there is a function that puts a "text file"-"sobj"-pair for a given object into a given directory. However, some manual work is currently involved to move this into SAGE_EXTCODE/pickle_jar/pickle_jar.tar.bz2, and this should easily be automated.

Concerning the topic of this trac ticket: If I understand correctly, there is an option to automatically create a pickle jar during sage -t, containing pickles for all objects that are subject to a loads/dumps doc test. But perhaps this is overzealous, and in addition the result would currently not be automatically put into the standard pickle jar.

However, this could be used as a starting point, to be modified so that

  1. the resulting pickles would automatically be put into the standard bzipped pickle jar in SAGE_EXTCODE
  2. only new pickles will be added: If an sobj created in version 5.9 is bitwise identical with an sobj created in version 6.1, then we should only have one copy of it in the pickle jar.

comment:4 Changed 8 years ago by vbraun

Git doesn't do anything special for bzipped files, in particular space usage will grow with the size of the pickle jar.

How about we distribute the pickle jar like any other database, as a tarball with a dummy build/pkg/pickle_jar?

comment:5 Changed 8 years ago by vbraun_spam

  • Milestone changed from sage-6.2 to sage-6.3

comment:6 Changed 8 years ago by vbraun_spam

  • Milestone changed from sage-6.3 to sage-6.4

comment:7 Changed 5 years ago by jdemeyer

  • Milestone changed from sage-6.4 to sage-duplicate/invalid/wontfix
  • Resolution set to duplicate
  • Status changed from new to closed

Duplicate of #10768.

comment:8 Changed 16 months ago by gh-DaveWitteMorris

  • Branch set to public/11720

Sorry, this branch was intended for a ticket #17720 (typo).

Last edited 16 months ago by gh-DaveWitteMorris (previous) (diff)
Note: See TracTickets for help on using tickets.