Opened 11 years ago
Closed 11 years ago
#7706 closed defect (fixed)
palp (lattice polytopes): replace the pickle-based database of lattice polytopes by a non-pickle database format
Reported by: | was | Owned by: | novoselt |
---|---|---|---|
Priority: | major | Milestone: | sage-4.3.3 |
Component: | geometry | Keywords: | |
Cc: | Merged in: | sage-4.3.3.alpha0 | |
Authors: | Andrey Novoseltsev | Reviewers: | Marshall Hampton, Mitesh Patel |
Report Upstream: | N/A | Work issues: | |
Branch: | Commit: | ||
Dependencies: | Stopgaps: |
Description
The Sage source distribution must ship with a bare minimum of opaque potentially dangerous binary files. Pickles (i.e. sobjs) are opaque binary files that can invoke arbitrary code when being unpickled. Also, sobj's have the drawback that they can someday break, and can be very hard to update and extend later. They are also hard to scan for virus. There are currently three places in the Sage source code that includes pickles:
- the pickle jar,
- the database of lattice polytopes
- the world map graph
For this ticket, please find a way to replace the lattice polytopes database spkg with something that contains no pickles. One solution would be to put plain text files in polytopes_db-*.spkg that described the 2d and 3d lattice polytopes. Then make the sobj's only when the spkg is installed. This would require making the spkg depend on the sage library (which is very reasonable).
Another possibility would be to change your code so that the first time the lattice polytope table is needed, a plain text file is parsed (so there is never an sobj).
Attachments (4)
Change History (13)
comment:1 Changed 11 years ago by
comment:2 Changed 11 years ago by
- Owner changed from mhampton to novoselt
comment:3 Changed 11 years ago by
- Status changed from new to needs_review
4 sobj files should be removed and replaced with two attached text files.
After some thinking and adjusting internal functions it turned out to be possible to reduce computing time for databases from 15 minutes to about 5 seconds (by avoiding extra checks and using initial polytopes in normal form), which is about 10 times longer than it was taking to load pickled files, but still seems quite reasonable to me as a once-per-session computation. It also now has the advantage of cached points (which are dropped during pickling for faster unpickling).
Timing (on sage.math):
Before (with sobj's):
sage: time len(ReflexivePolytopes(3)) CPU times: user 0.51 s, sys: 0.02 s, total: 0.53 s Wall time: 0.54 s 4319 sage: time len(ReflexivePolytopes(3)) CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s Wall time: 0.00 s 4319
After (with text data files):
sage: time len(ReflexivePolytopes(3)) CPU times: user 5.04 s, sys: 0.25 s, total: 5.29 s Wall time: 5.90 s 4319 sage: time len(ReflexivePolytopes(3)) CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s Wall time: 0.00 s 4319
comment:4 Changed 11 years ago by
- Reviewers set to mhampton
comment:5 Changed 11 years ago by
- Status changed from needs_review to positive_review
comment:6 Changed 11 years ago by
- Status changed from positive_review to needs_work
- Work issues set to Missing spkg?
Is there a new polytopes_db-*.spkg
?
comment:7 Changed 11 years ago by
- Status changed from needs_work to needs_review
- Work issues Missing spkg? deleted
comment:8 Changed 11 years ago by
- Reviewers changed from mhampton to Marshall Hampton, Mitesh Patel
- Status changed from needs_review to positive_review
Please remember to update the relevant ticket fields --- the release managers use an automated script to generate lists of merged tickets.
comment:9 Changed 11 years ago by
- Merged in set to sage-4.3.3.alpha0
- Resolution set to fixed
- Status changed from positive_review to closed
See also #7705.