Opened 6 years ago

Closed 6 years ago

Last modified 5 years ago

#14962 closed task (fixed)

make sure all optional packages work in the git repository

Reported by: ohanar Owned by:
Priority: major Milestone: sage-6.0
Component: packages: optional Keywords:
Cc: schilly Merged in:
Authors: Volker Braun Reviewers: Volker Braun
Report Upstream: N/A Work issues:
Branch: u/vbraun/optional_databases (Commits) Commit: 5b72359a8c98b14002e8a4e35600d9252280c7c7
Dependencies: #15172, #15261, #15263, #15370 #15302, #15814 Stopgaps:

Description (last modified by vbraun)

we support them, so we need to make sure they will continue to work

Script to systematically try out all optional spkgs: http://boxen.math.washington.edu/home/vbraun/Sage/Testing/test_optional_packages.sh

The optional databases will be dealt with here, other packages should make separate tickets. Use the following new "upstream" tarballs:

Leave for later, don't work and no SPKG.txt:

  • cunningham_tables
  • database_jones_numfield
  • database_kohel
  • database_odlyzko_zeta
  • database_sloane_oeis
  • database_stein_watkins_mini.p0

Move to experimental (broken with or without git, probably for a long time):

  • fricas (doesn't build with our ecl)
  • fricas-aldor (dito)
  • libcocoa (doesn't build with our gmp/mpir)

Delete for other reasons:

  • jsmath_image_fonts (we switched to mathjax)

Change History (34)

comment:1 Changed 6 years ago by ohanar

  • Branch set to u/ohanar/optional

comment:2 Changed 6 years ago by git

  • Commit set to acbbe0dce4ad570153083146a34e660dd620ab6f

Branch pushed to git repo; I updated commit sha1. This was a forced push. Recent commits:

[changeset:acbbe0d]Merge branch 'build_system' into optional
[changeset:2535903][FIXUP] optional: add git specific files
[changeset:c463e93][CLEANUP] remove hgignores
[changeset:8bb933a]Merge in optional packages
[changeset:4ea72f0]Trac #14909: Add new dependencies of the HAP package

comment:3 Changed 6 years ago by ohanar

  • Branch changed from u/ohanar/optional to u/ohanar/test

comment:4 Changed 6 years ago by git

  • Commit changed from acbbe0dce4ad570153083146a34e660dd620ab6f to cfd9dd10cf3ff3dab9d10f61f82c418723b0e395
Last edited 6 years ago by ohanar (previous) (diff)

comment:5 Changed 6 years ago by git

  • Commit changed from cfd9dd10cf3ff3dab9d10f61f82c418723b0e395 to cf45f4588bbc4e8ec59cc11b96dd4afc1abc7500

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:cf45f45]blah3
Version 0, edited 6 years ago by git (next)

comment:6 Changed 6 years ago by git

  • Commit changed from cf45f4588bbc4e8ec59cc11b96dd4afc1abc7500 to 728e542453259b59b7e012f382e141652671d425

Branch pushed to git repo; I updated commit sha1. New commits:

comment:7 Changed 6 years ago by git

  • Commit changed from 728e542453259b59b7e012f382e141652671d425 to cf45f4588bbc4e8ec59cc11b96dd4afc1abc7500
Last edited 6 years ago by ohanar (previous) (diff)

comment:8 Changed 6 years ago by vbraun

  • Commit cf45f4588bbc4e8ec59cc11b96dd4afc1abc7500 deleted
  • Dependencies set to #15172
  • Description modified (diff)

4ti2 is currently broken (not related to the git transition) but works with the update at #15172.

comment:9 Changed 6 years ago by vbraun

  • Branch u/ohanar/test deleted

comment:10 Changed 6 years ago by vbraun

  • Dependencies changed from #15172 to #15172, #15258

#15258 fails, because it implements a dependency check in the spkg-install using newest-version. How should optional spkgs handle dependencies when they transition to the new layout, by adding to deps? Or put the dependency rule in a <packagename>/deps file that is included from deps? Or no dependency management?

comment:11 Changed 6 years ago by ohanar

For right now I would say no dependency management -- any other solution at this point in time would require messing with the build system, which I don't want to do at the moment (so that we can get through this git transition).

On top of our current faux package manager I would say the best solution would be to replace (at least most of) deps with <packagename>/deps, but I'm not particularly interested in continuing the development of the current build system.

comment:12 Changed 6 years ago by vbraun

The following spkgs failed to install:

brian
cunningham_tables
database_cremona_ellcurve
database_jones_numfield
database_kohel
database_odlyzko_zeta
database_sloane_oeis
database_stein_watkins_mini.p0
database_symbolic_data
fricas
fricasaldor
jsmath_image_fonts
libcocoa
libtheora
mpi4py
openssl
p_group_cohomology
PyQt_x11
TOPCOM
valgrind

Logfiles are here: http://boxen.math.washington.edu/home/vbraun/Sage/Testing/optional/

Last edited 6 years ago by vbraun (previous) (diff)

comment:13 Changed 6 years ago by vbraun

  • Authors set to Volker Braun
  • Branch set to u/vbraun/optional_databases
  • Commit set to 5b72359a8c98b14002e8a4e35600d9252280c7c7

New commits:

[changeset:5b72359]switched the large Cremona ellcurve database to the new git directory layout

comment:14 Changed 6 years ago by vbraun

SAGE_DATA is now SAGE_LOCAL/share, for the record.

comment:15 Changed 6 years ago by vbraun

  • Description modified (diff)

comment:16 Changed 6 years ago by vbraun

Also, extra_docs barfs the extra docs into SAGE_ROOT.

comment:17 Changed 6 years ago by kcrisman

I am not online with the git layout yet (need to attend a tutorial, at the least, though Volker's video was a helpful start) so I can't help with this writing or reviewing yet BUT I have diagnosed the brian problem.

./spkg-install: line 10: cd: /mnt/storage2TB/vbraun-data/Sage/spkg/standard/: No such file or directory
./spkg-install: line 10: ./newest_version: No such file or directory
Failed to find numpy.  Please install the numpy spkg.

There are four tests of the following kind:

NUMPY=`cd $SAGE_ROOT/spkg/standard/; ./newest_version numpy`

So where would we check the "newest version" now? That's all one would need to do, or even check "presence" if the spkg placeholder txt files are no longer there. (That said, I'm not sure if those checks are even really necessary, which would also fix it, I'm sure. But if the fix is easy, then we should try that first.)

comment:18 follow-up: Changed 6 years ago by vbraun

The brian spkg just needs to strip out all the dependency checking, then it should work. You can still make spkgs as in the old workflow. Git installation instructions are, by now, easy:

git clone git://github.com/sagemath/sage.git
cd sage
make

comment:19 in reply to: ↑ 18 ; follow-up: Changed 6 years ago by kcrisman

The brian spkg just needs to strip out all the dependency checking, then it should work.

As I said:

That said, I'm not sure if those checks are even really necessary, which would also fix it, I'm sure.

I just don't remember why that was introduced. #9675 doesn't make it clear, so when I open this ticket, I'll cc: the original author. Otherwise I agree we should get rid of this. I'll update the patch situation as well and perhaps upgrade Brian if I find time for all this- should be able to in the next couple days.

You can still make spkgs as in the old workflow. Git installation instructions are, by now, easy:

Good to know! And that includes all the spkg stuff.

comment:20 in reply to: ↑ 19 Changed 6 years ago by kcrisman

I just don't remember why that was introduced. #9675 doesn't make it clear, so when I open this ticket, I'll cc: the original author. Otherwise I agree we should get rid of this. I'll update the patch situation as well and perhaps upgrade Brian if I find time for all this- should be able to in the next couple days.

This is now #15261.

comment:21 Changed 6 years ago by vbraun

  • Dependencies changed from #15172, #15258 to #15172, #15261

comment:22 Changed 6 years ago by vbraun

  • Description modified (diff)
  • mpi4py actually works, just needs openmpi libraries/compiler wrapper installed

comment:23 Changed 6 years ago by vbraun

  • Dependencies changed from #15172, #15261 to #15172, #15261, #15263, #15302

comment:24 Changed 6 years ago by ohanar

  • Dependencies changed from #15172, #15261, #15263, #15302 to #15172, #15261, #15263, #15370 #15302

comment:25 Changed 6 years ago by vbraun

  • Description modified (diff)
  • Status changed from new to needs_review

comment:26 follow-up: Changed 6 years ago by vbraun

  • Cc schilly added
  • Description modified (diff)
  • Reviewers set to Volker Braun
  • Status changed from needs_review to positive_review

I'm setting this to positive review since clearly nobody else has the guts to review this ticket.

The database_cremona_ellcurve-20121022.tar.bz2 tarball is already in the upstream dir, apparently.

Harald, can you delete/move to trash: fricas, fricas-aldor, libcocoa, and jsmath_image_fonts? They are (and have been for a while) useless with or without git.

comment:27 Changed 6 years ago by vbraun

  • Status changed from positive_review to closed

comment:28 Changed 6 years ago by vbraun

  • Resolution set to fixed

comment:29 in reply to: ↑ 26 Changed 6 years ago by schilly

Replying to vbraun:

Harald, can you delete/move to trash: fricas, fricas-aldor, libcocoa, and jsmath_image_fonts?

ok.

comment:30 Changed 5 years ago by cremona

My copy of database_stein_watkins_mini.p0.spkg does contain SPKG.txt:

jec@atkin:~/spkg/database_stein_watkins_mini.p0$ ls -l
total 12
-rwx------ 1 jec jec   49 Mar 12  2010 spkg-install
-rw------- 1 jec jec  389 Mar 13  2010 SPKG.txt
drwx------ 2 jec jec 4096 Nov  7  2005 stein-watkins-ecdb

so I think this one should be easy -- no compilaton, it just copies recurseively the directory to SAGE_DATA:

jec@atkin:~/spkg/database_stein_watkins_mini.p0$ cat spkg-install 
#!/bin/sh

cp -rf stein-watkins-ecdb $SAGE_DATA/

comment:31 Changed 5 years ago by cremona

If I was going to work on converting stein_watkins_mini, should I open a new ticket to hold the branch?

comment:32 follow-up: Changed 5 years ago by vbraun

Please open a new ticket, this one is closed.

stein_watkins_mini contains a skeleton SPKG.txt, but no description (what is it anyways?) and no license (=potentially not legal to distribute). William indicated in the ensuing thread on sage-devel that it is public domain afair.

comment:33 in reply to: ↑ 32 Changed 5 years ago by cremona

Replying to vbraun:

Please open a new ticket, this one is closed.

See #15556.

stein_watkins_mini contains a skeleton SPKG.txt, but no description (what is it anyways?) and no license (=potentially not legal to distribute). William indicated in the ensuing thread on sage-devel that it is public domain afair.

It's a large collection of elliptic curves compiled by William Stein and Mark Watkins about 10 years ago, stored in simple compressed text files. I'll put a reference to their paper in the SPKG.txt (it is already cited properly in the docstrings).

comment:34 Changed 5 years ago by rws

  • Dependencies changed from #15172, #15261, #15263, #15370 #15302 to #15172, #15261, #15263, #15370 #15302, #15814

See also #15814

Note: See TracTickets for help on using tickets.