Opened 10 years ago

Closed 8 years ago

Last modified 6 years ago

#10352 closed enhancement (fixed)

upgrade ZODB to version 3.10.5

Reported by: was Owned by: tbd
Priority: major Milestone: sage-5.6
Component: packages: standard Keywords:
Cc: Merged in: sage-5.6.beta3
Authors: William Stein, Nils Bruin, Volker Braun Reviewers: Jeroen Demeyer
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description (last modified by vbraun)

Upgrade ZODB to the latest upstream version.

The new spkg is here:

http://www.stp.dias.ie/~vbraun/Sage/spkg/zodb3-3.10.5.spkg

See also: #10353.

Change History (33)

comment:1 Changed 10 years ago by was

NOTE ABOUT spkg-check

I tried adding an spkg-check, but the fact is that the ZODB test suite fails miserably all over the place. This is true both of the version in Sage (3.7.0) and the version posted here. The failures start with:

(sage subshell) deep:src wstein$ python test.py
Running tests from build/lib.macosx-10.6-i386-2.6
test.py:44: DeprecationWarning: zope.testing.testrunner is deprecated in favour of zope.testrunner.

but there is no zope.testrunner so the DeprecationWarning? is wrong!! And it gets worse from there. Tons of key errors, other failures, then a total hang.

Regarding spkg-check, here's what it *should* look like:

#!/usr/bin/env bash
cd src/
python test.py

comment:2 Changed 10 years ago by was

  • Description modified (diff)
  • Status changed from new to needs_review

ZODB is used by the database_cremona optional spkg.  Thus a way to test this package is to install that optional package, then do:

sage: EllipticCurve("30001a1")
Elliptic Curve defined by y^2 + y = x^3 + x^2 - 171*x - 6408 over Rational Field
sage: CremonaDatabase().largest_conductor()
130000

and not get an error. Just that is enough to show that the ZODB is doing pretty much all we use it for in Sage. Running the test suite for ZODB is unfortunately not possible :-(.

Also, be sure to test the "devel/sage/sage/databases" directory, and you of course should do "make test" in SAGE_ROOT to test the whole library.

comment:3 Changed 10 years ago by was

  • Description modified (diff)

comment:4 Changed 10 years ago by was

  • Status changed from needs_review to needs_work

I just tried installing this and at the end got:

Finished processing dependencies for ZODB3==3.7.0

real	0m20.027s
user	0m14.322s
sys	0m2.886s
Successfully installed zodb3-3.10.1

so somehow setuptools is still thinking partly that it needs to install ZODB 3.7.0. So screw it. I'm changing this to needs work and not messing with it anymore.

comment:5 Changed 10 years ago by was

  • Description modified (diff)

comment:6 Changed 9 years ago by cschwan

Ticket #1634 should be marked as a duplicate.

comment:7 Changed 8 years ago by SimonKing

  • Priority changed from minor to major

I am bumping it up to "major", since it turns out that zodb screws up refcounts, which makes the debug version of Sage (#13864) fail.

comment:8 follow-up: Changed 8 years ago by SimonKing

Can we change this ticket and update to zodb3-3.10.5? Namely, in contrast to 3.10.1, version 3.10.5 does fix one of the problems from #13864.

comment:9 in reply to: ↑ 8 Changed 8 years ago by nbruin

Replying to SimonKing:

Can we change this ticket and update to zodb3-3.10.5? Namely, in contrast to 3.10.1, version 3.10.5 does fix one of the problems from #13864.

Will do. Got an spkg ready here.

comment:10 Changed 8 years ago by nbruin

  • Description modified (diff)
  • Status changed from needs_work to needs_review

For review purposes, I've just taken William's original upgrade to 3.10.1 and dumped 3.10.5 in instead. http://sage.math.washington.edu/home/wstein/patches/zodb3-3.10.1.spkg

As it turns out, our present ZODB3 version does not support Python 2.7, so upgrading is very much recommended.

comment:11 Changed 8 years ago by nbruin

  • Authors set to William Stein, Nils Bruin

comment:12 Changed 8 years ago by nbruin

  • Summary changed from upgrade ZODB to version 3.10.1 to upgrade ZODB to version 3.10.5

comment:13 Changed 8 years ago by fbissey

Updating could be done but some of us would rather ditch zodb altogether if someone with the know how would work on converting conway polynomial at #12205 which may have been done already and that I missed. See also #10353.

comment:14 Changed 8 years ago by jdemeyer

  • Status changed from needs_review to needs_work

There are some version mismatches, this needs

http://pypi.python.org/packages/source/z/zope.interface/zope.interface-4.0.2.tar.gz
http://pypi.python.org/packages/source/z/zope.exceptions/zope.exceptions-4.0.4.zip

comment:15 Changed 8 years ago by nbruin

  • Status changed from needs_work to needs_review

Thanks for catching! spkg updated.

comment:16 Changed 8 years ago by jpflori

I cannot install the last version of the spkg (the previous one seemed fine). It fails during the installation of the dependencies:

  • it install transaction 1.1.1,
  • and then fails with zope.interface.

comment:17 Changed 8 years ago by jpflori

And remark that twisted in sagenb spkg will also need zope.interface, so if #10353 gets merged I guess we have to provide it somewhere else than in zodb3 spkg.

comment:18 follow-up: Changed 8 years ago by vbraun

How about

  • zope.interface will be included in the sagenb spkg
  • We fix up the conway polynomials (#12205)
  • The fixed zodb spkg from this ticket will be an optional spkg
  • The zodb spkg will be a dependency for any optional spkg that needs it

comment:19 in reply to: ↑ 18 Changed 8 years ago by SimonKing

Replying to vbraun:

How about

  • zope.interface will be included in the sagenb spkg

Why should an independent module only be shipped as part of another independent module?

Even if zope.interface was currently only used by sagenb, I'd still think zope.interface should be an separate spkg.

  • We fix up the conway polynomials (#12205)
  • The fixed zodb spkg from this ticket will be an optional spkg
  • The zodb spkg will be a dependency for any optional spkg that needs it

That sounds good to me.

comment:20 follow-up: Changed 8 years ago by vbraun

The sagenb spkg contains already a dozen of python dependencies:

sagenb-0.10.2/src/Flask-OpenID-1.0.1.tar.gz
sagenb-0.10.2/src/Werkzeug-0.8.3.tar.gz
sagenb-0.10.2/src/Flask-AutoIndex-0.4.1.tar.gz
sagenb-0.10.2/src/Twisted-12.1.0.tar.bz2
sagenb-0.10.2/src/Babel-0.9.6.zip
sagenb-0.10.2/src/Flask-Silk-0.1.1.tar.gz
sagenb-0.10.2/src/pytz-2012d.zip
sagenb-0.10.2/src/python-openid-2.2.5.zip
sagenb-0.10.2/src/Flask-Babel-0.8.tar.gz
sagenb-0.10.2/src/Flask-0.9.tar.gz
sagenb-0.10.2/src/speaklater-1.3.tar.gz
sagenb-0.10.2/src/webassets-0.7.1.tar.gz

comment:21 follow-up: Changed 8 years ago by jdemeyer

  • Status changed from needs_review to needs_work

Not sure exactly what changed in the new spkg, but there are still version mismatches:

http://pypi.python.org/packages/source/z/zope.interface/zope.interface-4.0.3.tar.gz
http://pypi.python.org/packages/source/z/zope.exceptions/zope.exceptions-4.0.5.tar.gz

comment:22 in reply to: ↑ 20 ; follow-up: Changed 8 years ago by SimonKing

Replying to vbraun:

The sagenb spkg contains already a dozen of python dependencies:

Is sagenb a system that should be able to run independent of another system, or is sagenb supposed to be installed on top of another system (such as Sage)? In the former case, I could understand why sagenb comes with batteries included. But if sagenb is intended to be a small brick in a large wall, then it seems to me that a small brick shouldn't attempt to contain other small bricks...

But I am not a software engineer. Anyway, the fact that sagenb contains a lot of dependencies (taking the risk that the same dependency is provided twice, perhaps even in different versions) should not force us to add yet another module to sagenb.

comment:23 in reply to: ↑ 22 Changed 8 years ago by was

Replying to SimonKing:

Replying to vbraun:

The sagenb spkg contains already a dozen of python dependencies:

Is sagenb a system that should be able to run independent of another system?

Yes, it is supposed to be something that can run without Sage (it is supposed to only require python). I don't know to what extent that is currently the case.

William

comment:24 in reply to: ↑ 21 ; follow-up: Changed 8 years ago by nbruin

Replying to jdemeyer:

Not sure exactly what changed in the new spkg, but there are still version mismatches:

http://pypi.python.org/packages/source/z/zope.interface/zope.interface-4.0.3.tar.gz
http://pypi.python.org/packages/source/z/zope.exceptions/zope.exceptions-4.0.5.tar.gz

These are different versions than you reported before. I put the files in there you requested last time.

to any zope version expert: please put versions in there that make Jeroen happy.

comment:25 in reply to: ↑ 24 Changed 8 years ago by jdemeyer

Perhaps the problem is simply the order of packages installed. transaction is installed before the zope stuff, but transaction requires zope.interface.

comment:26 follow-up: Changed 8 years ago by vbraun

Can somebody actually post a log from a failed install attempt? It works for me. The version mismatches shouldn't matter either, distutils is checking that each package's dependencies are satisfied. You don't need matching versions of all zope* packages (indeed, they are not in sync).

Also I don't understand what Jeroen posted, the spkg from the ticket issue contains zope.exceptions-4.0.4.zip and zope.interface-4.0.2.tar.gz and not the versions you mention.

comment:27 in reply to: ↑ 26 Changed 8 years ago by jdemeyer

Replying to vbraun:

Can somebody actually post a log from a failed install attempt? It works for me. The version mismatches shouldn't matter either, distutils is checking that each package's dependencies are satisfied. You don't need matching versions of all zope* packages (indeed, they are not in sync).

The problem is this:

Processing transaction-1.1.1.tar.gz
Running transaction-1.1.1/setup.py -q bdist_egg --dist-dir /tmp/release/easy_install-FMO44V/transaction-1.1.1/egg-dist-tmp-B7YtEa
Adding transaction 1.1.1 to easy-install.pth file

Installed /release/merger/sage-5.6.beta2-debug/local/lib/python2.7/site-packages/transaction-1.1.1-py2.7.egg
Processing dependencies for transaction==1.1.1
Searching for zope.interface
Reading http://pypi.python.org/simple/zope.interface/
Best match: zope.interface 4.0.3
Downloading http://pypi.python.org/packages/source/z/zope.interface/zope.interface-4.0.3.tar.gz#md5=1ddd308f2c83703accd1696158c300eb
Processing zope.interface-4.0.3.tar.gz
Running zope.interface-4.0.3/setup.py -q bdist_egg --dist-dir /tmp/release/easy_install-86Axhl/zope.interface-4.0.3/egg-dist-tmp-2mKFn_
Adding zope.interface 4.0.3 to easy-install.pth file

Installed /release/merger/sage-5.6.beta2-debug/local/lib/python2.7/site-packages/zope.interface-4.0.3-py2.7-linux-x86_64.egg
Finished processing dependencies for transaction==1.1.1

When transaction is installing, zope.interface hasn't been installed yet and must be downloaded from the internet.

comment:28 Changed 8 years ago by jpflori

I guess you can indeed easily reproduce that by shutting down your internet connection and making sure no previous installation of zope.interface is available (e.g. do not upgrade the zodb spkg on an existing sage install).

comment:29 Changed 8 years ago by vbraun

  • Authors changed from William Stein, Nils Bruin to William Stein, Nils Bruin, Volker Braun
  • Description modified (diff)
  • Status changed from needs_work to needs_review
  • upgraded to zope.interface-4.0.3
  • using easy_install --index-url dependencies <package> to use the tarballs from the dependencies directory

I've built sage-5.6.beta2 from scratch with the new spkg and verified that zodb is not downloading anything (that is, no "Downloading" or urls in the spkg log).

comment:30 Changed 8 years ago by jdemeyer

  • Reviewers set to Jeroen Demeyer
  • Status changed from needs_review to positive_review

Looks good to me (but it's going to be removed anyway, right?)

comment:31 Changed 8 years ago by vbraun

Its probably premature to remove it in sage-5.6, let's first fix the sagenb dependency. I've created https://github.com/sagemath/sagenb/issues/126 to raise awareness on the sagenb side.

comment:32 Changed 8 years ago by jdemeyer

  • Merged in set to sage-5.6.beta3
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:33 Changed 6 years ago by kcrisman

Just as a point of information, this was indeed added to upstream, see also #10353 where that was confirmed.

Note: See TracTickets for help on using tickets.