Opened 11 years ago

Closed 11 years ago

#7706 closed defect (fixed)

palp (lattice polytopes): replace the pickle-based database of lattice polytopes by a non-pickle database format

Reported by: was Owned by: novoselt
Priority: major Milestone: sage-4.3.3
Component: geometry Keywords:
Cc: Merged in: sage-4.3.3.alpha0
Authors: Andrey Novoseltsev Reviewers: Marshall Hampton, Mitesh Patel
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description

The Sage source distribution must ship with a bare minimum of opaque potentially dangerous binary files. Pickles (i.e. sobjs) are opaque binary files that can invoke arbitrary code when being unpickled. Also, sobj's have the drawback that they can someday break, and can be very hard to update and extend later. They are also hard to scan for virus. There are currently three places in the Sage source code that includes pickles:

  • the pickle jar,
  • the database of lattice polytopes
  • the world map graph

For this ticket, please find a way to replace the lattice polytopes database spkg with something that contains no pickles. One solution would be to put plain text files in polytopes_db-*.spkg that described the 2d and 3d lattice polytopes. Then make the sobj's only when the spkg is installed. This would require making the spkg depend on the sage library (which is very reasonable).

Another possibility would be to change your code so that the first time the lattice polytope table is needed, a plain text file is parsed (so there is never an sobj).

Attachments (4)

trac_7706_change_format_of_reflexive_polytope_databases.patch (9.0 KB) - added by novoselt 11 years ago.
Based on 4.3.rc0
reflexive_polytopes_2d (448 bytes) - added by novoselt 11 years ago.
Must be placed in DB_HOME/reflexive_polytopes
reflexive_polytopes_3d (269.1 KB) - added by novoselt 11 years ago.
Must be placed in DB_HOME/reflexive_polytopes
polytopes_db-20100210.spkg (31.4 KB) - added by novoselt 11 years ago.
Should be used instead of two data files attached earlier

Download all attachments as: .zip

Change History (13)

comment:1 Changed 11 years ago by was

See also #7705.

comment:2 Changed 11 years ago by novoselt

  • Owner changed from mhampton to novoselt

Changed 11 years ago by novoselt

Based on 4.3.rc0

Changed 11 years ago by novoselt

Must be placed in DB_HOME/reflexive_polytopes

Changed 11 years ago by novoselt

Must be placed in DB_HOME/reflexive_polytopes

comment:3 Changed 11 years ago by novoselt

  • Status changed from new to needs_review

4 sobj files should be removed and replaced with two attached text files.

After some thinking and adjusting internal functions it turned out to be possible to reduce computing time for databases from 15 minutes to about 5 seconds (by avoiding extra checks and using initial polytopes in normal form), which is about 10 times longer than it was taking to load pickled files, but still seems quite reasonable to me as a once-per-session computation. It also now has the advantage of cached points (which are dropped during pickling for faster unpickling).

Timing (on sage.math):

Before (with sobj's):

sage: time len(ReflexivePolytopes(3))
CPU times: user 0.51 s, sys: 0.02 s, total: 0.53 s
Wall time: 0.54 s
4319
sage: time len(ReflexivePolytopes(3))
CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s
Wall time: 0.00 s
4319

After (with text data files):

sage: time len(ReflexivePolytopes(3))
CPU times: user 5.04 s, sys: 0.25 s, total: 5.29 s
Wall time: 5.90 s
4319
sage: time len(ReflexivePolytopes(3))
CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s
Wall time: 0.00 s
4319

comment:4 Changed 11 years ago by mhampton

  • Reviewers set to mhampton

comment:5 Changed 11 years ago by was

  • Status changed from needs_review to positive_review

comment:6 Changed 11 years ago by mpatel

  • Status changed from positive_review to needs_work
  • Work issues set to Missing spkg?

Is there a new polytopes_db-*.spkg?

Changed 11 years ago by novoselt

Should be used instead of two data files attached earlier

comment:7 Changed 11 years ago by novoselt

  • Status changed from needs_work to needs_review
  • Work issues Missing spkg? deleted

comment:8 Changed 11 years ago by mpatel

  • Authors set to Andrey Novoseltsev
  • Reviewers changed from mhampton to Marshall Hampton, Mitesh Patel
  • Status changed from needs_review to positive_review
 Please remember to update the relevant ticket fields --- the release
 managers use an automated script to generate lists of merged tickets.

comment:9 Changed 11 years ago by mpatel

  • Merged in set to sage-4.3.3.alpha0
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.