Opened 6 years ago

Closed 6 years ago

#15556 closed defect (fixed)

Convert Stein-Watkins database to git spkg format

Reported by: cremona Owned by:
Priority: minor Milestone: sage-6.1
Component: packages: optional Keywords: elliptic curves, databases
Cc: Merged in:
Authors: John Cremona, Jeroen Demeyer Reviewers: Jeroen Demeyer, John Cremona
Report Upstream: N/A Work issues:
Branch: u/jdemeyer/ticket/15556 (Commits) Commit: 9936f24af676e509a6ec06665a4f164c3cccd3ea
Dependencies: Stopgaps:

Description (last modified by cremona)

There are two optional spkgs containing small and full version of the Stein-Watkins database of elliptic curves, namely database_stein_watkins_mini.p0.spkg (the small version, approximately 20mb) and stein-watkins-ecdb.p0.spkg (the huge version, approximately 2.7gb).

Both should be converted to the new git format for spkgs, at the same time making their names more consistent and completing their SPKG.txt information. The interface in sage/databases/stein-watkins.py will need to be tested.

New upstream tarballs:

  1. http://boxen.math.washington.edu/home/cremona/database_stein_watkins_mini-20030423.tar.bz2
  2. http://boxen.math.washington.edu/home/cremona/database_stein_watkins-20030423.tar.bz2

Change History (45)

comment:1 Changed 6 years ago by cremona

  • Branch set to u/cremona/ticket/15556
  • Created changed from 12/20/13 12:13:39 to 12/20/13 12:13:39
  • Modified changed from 12/20/13 12:13:39 to 12/20/13 12:13:39

comment:2 Changed 6 years ago by cremona

Done both.

The new package files have been uploaded to sagemath in /home/cremona:

-rw-r--r-- 1 cremona cremona   19935714 Dec 20 09:19 database_stein_watkins_mini-20131220.tar.bz2
-rw-r--r-- 1 cremona cremona 2784729140 Dec 20 10:48 database_stein_watkins-20131220.tar.bz2

Installing even the small one is not very quick, and the long one quite slow. The optional doctests in sage/da\ tabases/stein_watkins.py will work with only the mini package; remember to add --optional=stein_watkins_databas\ e to the command line or you'll not be checking much!

comment:3 Changed 6 years ago by cremona

  • Modified changed from 12/20/13 20:05:32 to 12/20/13 20:05:32
  • Status changed from new to needs_review

comment:4 Changed 6 years ago by cremona

  • Authors set to John Cremona
  • Description modified (diff)

comment:5 Changed 6 years ago by cremona

  • Modified changed from 12/20/13 20:08:08 to 12/20/13 20:08:08

comment:6 Changed 6 years ago by cremona

  • Commit set to 747a385d4ae1a844df7efffc6555f1767e8594ac

comment:7 follow-ups: Changed 6 years ago by jdemeyer

Various details:

  1. use cp -R instead of cp -rf
  2. omit the version check (you're adding a check for version 6.x to a file which only exists in version 6.x)
  3. replace $SAGE_LOCAL/share by $SAGE_SHARE
  4. replace
    if [ "x${SAGE_LOCAL}" = x ]; then
       echo "SAGE_LOCAL undefined ... exiting"
       echo "Maybe run 'sage -sh'?"
       exit 1
    fi
    

by

if [ -z "$SAGE_SHARE" ]; then
    echo >&2 "SAGE_SHARE undefined ... exiting"
    echo >&2 "Maybe run 'sage --sh'?"
    exit 1
fi

comment:8 Changed 6 years ago by git

  • Commit changed from 747a385d4ae1a844df7efffc6555f1767e8594ac to 38ef1d19c616f00c9edeee9d8fb55fa4d758370f

Branch pushed to git repo; I updated commit sha1. New commits:

38ef1d1trac 15556: revised spkg-install following review

comment:9 in reply to: ↑ 7 Changed 6 years ago by cremona

  • Reviewers set to Jeroen Demeyer

Replying to jdemeyer:

Various details:

  1. use cp -R instead of cp -rf

done

  1. omit the version check (you're adding a check for version 6.x to a file which only exists in version 6.x)

done

  1. replace $SAGE_LOCAL/share by $SAGE_SHARE

done

  1. replace
    if [ "x${SAGE_LOCAL}" = x ]; then
       echo "SAGE_LOCAL undefined ... exiting"
       echo "Maybe run 'sage -sh'?"
       exit 1
    fi
    

by

if [ -z "$SAGE_SHARE" ]; then
    echo >&2 "SAGE_SHARE undefined ... exiting"
    echo >&2 "Maybe run 'sage --sh'?"
    exit 1
fi

done (all in both database_stein_watkins and ...mini). New commit pushed.


New commits:

38ef1d1trac 15556: revised spkg-install following review

comment:10 in reply to: ↑ 7 Changed 6 years ago by cremona

Replying to jdemeyer:

Various details:

  1. use cp -R instead of cp -rf

done

  1. omit the version check (you're adding a check for version 6.x to a file which only exists in version 6.x)

done

  1. replace $SAGE_LOCAL/share by $SAGE_SHARE

done

  1. replace
    if [ "x${SAGE_LOCAL}" = x ]; then
       echo "SAGE_LOCAL undefined ... exiting"
       echo "Maybe run 'sage -sh'?"
       exit 1
    fi
    

by

if [ -z "$SAGE_SHARE" ]; then
    echo >&2 "SAGE_SHARE undefined ... exiting"
    echo >&2 "Maybe run 'sage --sh'?"
    exit 1
fi

done (all in both database_stein_watkins and ...mini). New commit pushed. Wait though since there is a typo and a new one is coming up.


New commits:

38ef1d1trac 15556: revised spkg-install following review

New commits:

38ef1d1trac 15556: revised spkg-install following review

comment:11 Changed 6 years ago by git

  • Commit changed from 38ef1d19c616f00c9edeee9d8fb55fa4d758370f to 6a35c8a76d8796ffbe1ddcd6597db4eac0b0f1ee

Branch pushed to git repo; I updated commit sha1. New commits:

6a35c8atrac 15556: correct typo in previous commit

comment:12 follow-up: Changed 6 years ago by jdemeyer

For the VERSION of the package, would it not make more sense to use the date of the database itself, as supposed of the date of the packaging? So the version number would be 20030423.p1

comment:13 in reply to: ↑ 12 ; follow-up: Changed 6 years ago by cremona

Replying to jdemeyer:

For the VERSION of the package, would it not make more sense to use the date of the database itself, as supposed of the date of the packaging? So the version number would be 20030423.p1

Possibly. The people who should really decide that are S&W themselves. Rather more important, there were some changes to the 2003 data made in 2007 and I am not sure whether the files in this version include those changes or not. I did *not* intend to change the data in the spkg at all with this ticket, only the packaging of it. So I used a data which related to the packaging, not the data itself.

If in fact the data in here is the original 2003 version (something I will check, but not today) then one possibility would be to use the 2003 date for the version now, and change it to a 2007 date for a revision. Then it would be clear exactly what is what.

It would still be helpful to know if the new packages do install and test correctly, even if we change the version before committing to this.

comment:14 follow-up: Changed 6 years ago by jdemeyer

My comment about the version check applies also to database_cremona_ellcurve so you can remove that also.

comment:15 in reply to: ↑ 13 ; follow-up: Changed 6 years ago by jdemeyer

Replying to cremona:

The people who should really decide that are S&W themselves.

I don't see why. They are welcome to have an opinion, but I don't think we need to ask every upstream project how it should be included in Sage.

Rather more important, there were some changes to the 2003 data made in 2007 and I am not sure whether the files in this version include those changes or not.

I checked and those changes are not included, the data is from 2003.

If in fact the data in here is the original 2003 version (something I will check, but not today) then one possibility would be to use the 2003 date for the version now, and change it to a 2007 date for a revision. Then it would be clear exactly what is what.

I totally agree with this. That's another reason why the version number should refer to the date of the database, to reduce such confusion. The date of today when you packaged the database is not relevant for anything.

It would still be helpful to know if the new packages do install and test correctly, even if we change the version before committing to this.

Of course, I will check that.

comment:16 in reply to: ↑ 14 ; follow-up: Changed 6 years ago by cremona

Replying to jdemeyer:

My comment about the version check applies also to database_cremona_ellcurve so you can remove that also.

I think that should be on another ticket.

I did use that one as a model for this one (I was not the author of it!)

comment:17 in reply to: ↑ 15 Changed 6 years ago by cremona

Replying to jdemeyer:

Replying to cremona:

The people who should really decide that are S&W themselves.

I don't see why. They are welcome to have an opinion, but I don't think we need to ask every upstream project how it should be included in Sage.

Rather more important, there were some changes to the 2003 data made in 2007 and I am not sure whether the files in this version include those changes or not.

I checked and those changes are not included, the data is from 2003.

If in fact the data in here is the original 2003 version (something I will check, but not today) then one possibility would be to use the 2003 date for the version now, and change it to a 2007 date for a revision. Then it would be clear exactly what is what.

I totally agree with this. That's another reason why the version number should refer to the date of the database, to reduce such confusion. The date of today when you packaged the database is not relevant for anything.

ok, I will change it to what you suggested. This will mean renamed upstream tarballs and so possibly also checksums. And a second ticket to update both to include the 2007 fixes.

It would still be helpful to know if the new packages do install and test correctly, even if we change the version before committing to this.

Of course, I will check that.

comment:18 in reply to: ↑ 16 ; follow-up: Changed 6 years ago by jdemeyer

Replying to cremona:

Replying to jdemeyer:

My comment about the version check applies also to database_cremona_ellcurve so you can remove that also.

I think that should be on another ticket.

Sorry, but I disagree. You're already making modifications to build/pkgs/database_cremona_ellcurve/spkg-install on this ticket. It's pointless for me to review these changes in code which a new ticket will probably remove.

comment:19 Changed 6 years ago by jdemeyer

  • Description modified (diff)
  • Status changed from needs_review to needs_work

comment:20 in reply to: ↑ 18 Changed 6 years ago by cremona

Replying to jdemeyer:

My comment about the version check applies also to database_cremona_ellcurve so you can remove that also.

I think that should be on another ticket.

Sorry, but I disagree. You're already making modifications to build/pkgs/database_cremona_ellcurve/spkg-install on this ticket. It's pointless for me to review these changes in code which a new ticket will probably remove.

Of course -- I had forgotten that. It Will Be Done.

comment:21 follow-up: Changed 6 years ago by jdemeyer

The files inside the upstream tarballs must all have permission 644 (world-readable), otherwise they will not work for users besides the one who installed the database.

comment:22 Changed 6 years ago by cremona

  • Description modified (diff)

comment:23 in reply to: ↑ 21 Changed 6 years ago by cremona

Replying to jdemeyer:

The files inside the upstream tarballs must all have permission 644 (world-readable), otherwise they will not work for users besides the one who installed the database.

OK, have done that (and changed the names of these in the ticket description).

comment:24 Changed 6 years ago by cremona

  • Description modified (diff)

comment:25 follow-up: Changed 6 years ago by jdemeyer

  • Work issues set to HTTP 403

The second link gives

Forbidden

You don't have permission to access /home/cremona/database_stein_watkins-20030423.tar.bz2 on this server.

comment:26 in reply to: ↑ 25 Changed 6 years ago by cremona

Replying to jdemeyer:

The second link gives

Forbidden

You don't have permission to access /home/cremona/database_stein_watkins-20030423.tar.bz2 on this server.

I fixed that and it is again ready for re-review, I have pushed anew commit, but will not respond until after the holiday so there is no rush!

Last edited 6 years ago by cremona (previous) (diff)

comment:27 Changed 6 years ago by git

  • Commit changed from 6a35c8a76d8796ffbe1ddcd6597db4eac0b0f1ee to 2b283c7b8e59d4f80758c82e2c93dc5d0c450757

Branch pushed to git repo; I updated commit sha1. New commits:

2b283c7trac 15556: correct package date and upstream tarball name

comment:28 Changed 6 years ago by jdemeyer

  • Status changed from needs_work to needs_review
  • Work issues HTTP 403 deleted

comment:29 follow-up: Changed 6 years ago by vbraun

The tarball link in the ticket description is probably incorrect, it uses a package date from 2003 instead of 2013?

comment:30 in reply to: ↑ 29 Changed 6 years ago by jdemeyer

Replying to vbraun:

The tarball link in the ticket description is probably incorrect, it uses a package date from 2003 instead of 2013?

2003 is correct. The contents of the database in the tarball (which is what matters) are from the day in the version number.

comment:31 follow-ups: Changed 6 years ago by vbraun

OK, thanks. The implementation looks good. We must change the license to something other than "None", as no license technically means all right reserved and no permission to distribute whatsoever. Per William's post this should be "Public domain"

comment:32 in reply to: ↑ 31 Changed 6 years ago by jdemeyer

Replying to vbraun:

OK, thanks. The implementation looks good. We must change the license to something other than "None", as no license technically means all right reserved and no permission to distribute whatsoever. Per William's post this should be "Public domain"

IANAL, but I thought that pure data is not copyrighted and therefore is automatically public domain...

comment:33 Changed 6 years ago by vbraun

There is no copyright for facts, but there is for how they are presented. And the bar is really low for what constitutes an original presentation. E.g. the information in the yellow pages is not protected, but the layout and arrangement of the printed yellow pages is.

comment:34 in reply to: ↑ 31 Changed 6 years ago by jdemeyer

Replying to vbraun:

Per William's post this should be "Public domain"

Link please.

comment:35 Changed 6 years ago by jdemeyer

  • Branch changed from u/cremona/ticket/15556 to u/jdemeyer/ticket/15556
  • Modified changed from 12/27/13 10:12:46 to 12/27/13 10:12:46

comment:36 Changed 6 years ago by jdemeyer

  • Authors changed from John Cremona to John Cremona, Jeroen Demeyer
  • Commit changed from 2b283c7b8e59d4f80758c82e2c93dc5d0c450757 to 3f48d25582785bf6b92ed6d92255657f09175347

Additional commit needs review.


New commits:

3f48d25Stein-Watkins database: reviewer patch

comment:37 Changed 6 years ago by git

  • Commit changed from 3f48d25582785bf6b92ed6d92255657f09175347 to 9936f24af676e509a6ec06665a4f164c3cccd3ea

Branch pushed to git repo; I updated commit sha1. New commits:

9936f24Where possible, remove optional - database_stein_watkins

comment:38 Changed 6 years ago by vbraun

https://groups.google.com/d/msg/sage-devel/onWFAVz-LDA/wj7FAsabIpkJ

anybody can use them however they want = public domain

comment:39 follow-up: Changed 6 years ago by cremona

Thanks for tidying this up, Jeroen. I can see the point in removing the optional tags and also renaming the optional flag to match the spkg name. The new version works for me, so shall we mutually approve each other's commits? [I did need to remember to do "--optional=sage,database_stein_watkins" to do the testing.]

My intention had been to go straight on to revising this to include the (very slightly different) 2007 data, building on this branch. I am not sure whether we should ask for the commits so far to be merged before doing that. It will only require: making new versions of the upstream tarballs with version date being 2007???? (the fixed files are dated June 16, June 25, and Aug 25 and 27, so perhaps we should take the latest date) and changing both packa-version.txt files to the same date.

comment:40 in reply to: ↑ 39 Changed 6 years ago by jdemeyer

Replying to cremona:

My intention had been to go straight on to revising this to include the (very slightly different) 2007 data, building on this branch. I am not sure whether we should ask for the commits so far to be merged before doing that. It will only require: making new versions of the upstream tarballs with version date being 2007???? (the fixed files are dated June 16, June 25, and Aug 25 and 27, so perhaps we should take the latest date) and changing both packa-version.txt files to the same date.

Do as you wish, for me it's fine either way.

comment:41 Changed 6 years ago by jdemeyer

  • Reviewers changed from Jeroen Demeyer to Jeroen Demeyer, John Cremona

comment:42 Changed 6 years ago by cremona

I'm going to make the 2007 upgrade into another ticket (dependent on this one), which I will cross-ref here when it has been created. So this one can have a positive review.

comment:43 Changed 6 years ago by jdemeyer

  • Status changed from needs_review to positive_review

comment:44 Changed 6 years ago by cremona

See #15607 for the follow-up ticket.

comment:45 Changed 6 years ago by vbraun

  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.