upgrade ZODB to version 3.10.5

[was]

Upgrade ZODB to the latest upstream version.

The new spkg is here:

​http://www.stp.dias.ie/~vbraun/Sage/spkg/zodb3-3.10.5.spkg

See also: #10353.

[was]

NOTE ABOUT spkg-check

I tried adding an spkg-check, but the fact is that the ZODB test suite fails miserably all over the place. This is true both of the version in Sage (3.7.0) and the version posted here. The failures start with:

(sage subshell) deep:src wstein$ python test.py
Running tests from build/lib.macosx-10.6-i386-2.6
test.py:44: DeprecationWarning: zope.testing.testrunner is deprecated in favour of zope.testrunner.

but there is no zope.testrunner so the DeprecationWarning? is wrong!! And it gets worse from there. Tons of key errors, other failures, then a total hang.

Regarding spkg-check, here's what it *should* look like:

#!/usr/bin/env bash
cd src/
python test.py

[was]

ZODB is used by the database_cremona optional spkg.  Thus a way to test this package is to install that optional package, then do:

sage: EllipticCurve("30001a1")
Elliptic Curve defined by y^2 + y = x^3 + x^2 - 171*x - 6408 over Rational Field
sage: CremonaDatabase().largest_conductor()
130000

and not get an error. Just that is enough to show that the ZODB is doing pretty much all we use it for in Sage. Running the test suite for ZODB is unfortunately not possible :-(.

Also, be sure to test the "devel/sage/sage/databases" directory, and you of course should do "make test" in SAGE_ROOT to test the whole library.

[was]

I just tried installing this and at the end got:

Finished processing dependencies for ZODB3==3.7.0

real	0m20.027s
user	0m14.322s
sys	0m2.886s
Successfully installed zodb3-3.10.1

so somehow setuptools is still thinking partly that it needs to install ZODB 3.7.0. So screw it. I'm changing this to needs work and not messing with it anymore.

[cschwan]

Ticket #1634 should be marked as a duplicate.

[SimonKing]

I am bumping it up to "major", since it turns out that zodb screws up refcounts, which makes the debug version of Sage (#13864) fail.

[SimonKing]

Can we change this ticket and update to zodb3-3.10.5? Namely, in contrast to 3.10.1, version 3.10.5 does fix one of the problems from #13864.

[nbruin]

Replying to SimonKing:

Can we change this ticket and update to zodb3-3.10.5? Namely, in contrast to 3.10.1, version 3.10.5 does fix one of the problems from #13864.

Will do. Got an spkg ready here.

[nbruin]

For review purposes, I've just taken William's original upgrade to 3.10.1 and dumped 3.10.5 in instead. ​http://sage.math.washington.edu/home/wstein/patches/zodb3-3.10.1.spkg

As it turns out, our present ZODB3 version does not support Python 2.7, so upgrading is very much recommended.

[fbissey]

Updating could be done but some of us would rather ditch zodb altogether if someone with the know how would work on converting conway polynomial at #12205 which may have been done already and that I missed. See also #10353.

[jdemeyer]

There are some version mismatches, this needs

http://pypi.python.org/packages/source/z/zope.interface/zope.interface-4.0.2.tar.gz
http://pypi.python.org/packages/source/z/zope.exceptions/zope.exceptions-4.0.4.zip

[nbruin]

Thanks for catching! spkg updated.

[jpflori]

I cannot install the last version of the spkg (the previous one seemed fine). It fails during the installation of the dependencies:

• it install transaction 1.1.1,
• and then fails with zope.interface.

[jpflori]

And remark that twisted in sagenb spkg will also need zope.interface, so if #10353 gets merged I guess we have to provide it somewhere else than in zodb3 spkg.

[vbraun]

How about

• zope.interface will be included in the sagenb spkg
• We fix up the conway polynomials (#12205)
• The fixed zodb spkg from this ticket will be an optional spkg
• The zodb spkg will be a dependency for any optional spkg that needs it

[SimonKing]

Replying to vbraun:

How about

• zope.interface will be included in the sagenb spkg

Why should an independent module only be shipped as part of another independent module?

Even if zope.interface was currently only used by sagenb, I'd still think zope.interface should be an separate spkg.

• We fix up the conway polynomials (#12205)
• The fixed zodb spkg from this ticket will be an optional spkg
• The zodb spkg will be a dependency for any optional spkg that needs it

That sounds good to me.

[vbraun]

The sagenb spkg contains already a dozen of python dependencies:

sagenb-0.10.2/src/Flask-OpenID-1.0.1.tar.gz
sagenb-0.10.2/src/Werkzeug-0.8.3.tar.gz
sagenb-0.10.2/src/Flask-AutoIndex-0.4.1.tar.gz
sagenb-0.10.2/src/Twisted-12.1.0.tar.bz2
sagenb-0.10.2/src/Babel-0.9.6.zip
sagenb-0.10.2/src/Flask-Silk-0.1.1.tar.gz
sagenb-0.10.2/src/pytz-2012d.zip
sagenb-0.10.2/src/python-openid-2.2.5.zip
sagenb-0.10.2/src/Flask-Babel-0.8.tar.gz
sagenb-0.10.2/src/Flask-0.9.tar.gz
sagenb-0.10.2/src/speaklater-1.3.tar.gz
sagenb-0.10.2/src/webassets-0.7.1.tar.gz

[jdemeyer]

Not sure exactly what changed in the new spkg, but there are still version mismatches:

http://pypi.python.org/packages/source/z/zope.interface/zope.interface-4.0.3.tar.gz
http://pypi.python.org/packages/source/z/zope.exceptions/zope.exceptions-4.0.5.tar.gz

[SimonKing]

Replying to vbraun:

The sagenb spkg contains already a dozen of python dependencies:

Is sagenb a system that should be able to run independent of another system, or is sagenb supposed to be installed on top of another system (such as Sage)? In the former case, I could understand why sagenb comes with batteries included. But if sagenb is intended to be a small brick in a large wall, then it seems to me that a small brick shouldn't attempt to contain other small bricks...

But I am not a software engineer. Anyway, the fact that sagenb contains a lot of dependencies (taking the risk that the same dependency is provided twice, perhaps even in different versions) should not force us to add yet another module to sagenb.

[was]

Replying to SimonKing:

Replying to vbraun:

The sagenb spkg contains already a dozen of python dependencies:

Is sagenb a system that should be able to run independent of another system?

Yes, it is supposed to be something that can run without Sage (it is supposed to only require python). I don't know to what extent that is currently the case.

William

[nbruin]

Replying to jdemeyer:

Not sure exactly what changed in the new spkg, but there are still version mismatches:

http://pypi.python.org/packages/source/z/zope.interface/zope.interface-4.0.3.tar.gz
http://pypi.python.org/packages/source/z/zope.exceptions/zope.exceptions-4.0.5.tar.gz

These are different versions than you reported before. I put the files in there you requested last time.

to any zope version expert: please put versions in there that make Jeroen happy.

[jdemeyer]

Perhaps the problem is simply the order of packages installed. transaction is installed before the zope stuff, but transaction requires zope.interface.

[vbraun]

Can somebody actually post a log from a failed install attempt? It works for me. The version mismatches shouldn't matter either, distutils is checking that each package's dependencies are satisfied. You don't need matching versions of all zope* packages (indeed, they are not in sync).

Also I don't understand what Jeroen posted, the spkg from the ticket issue contains zope.exceptions-4.0.4.zip and zope.interface-4.0.2.tar.gz and not the versions you mention.

[jdemeyer]

Replying to vbraun:

Can somebody actually post a log from a failed install attempt? It works for me. The version mismatches shouldn't matter either, distutils is checking that each package's dependencies are satisfied. You don't need matching versions of all zope* packages (indeed, they are not in sync).

The problem is this:

Processing transaction-1.1.1.tar.gz
Running transaction-1.1.1/setup.py -q bdist_egg --dist-dir /tmp/release/easy_install-FMO44V/transaction-1.1.1/egg-dist-tmp-B7YtEa
Adding transaction 1.1.1 to easy-install.pth file

Installed /release/merger/sage-5.6.beta2-debug/local/lib/python2.7/site-packages/transaction-1.1.1-py2.7.egg
Processing dependencies for transaction==1.1.1
Searching for zope.interface
Reading http://pypi.python.org/simple/zope.interface/
Best match: zope.interface 4.0.3
Downloading http://pypi.python.org/packages/source/z/zope.interface/zope.interface-4.0.3.tar.gz#md5=1ddd308f2c83703accd1696158c300eb
Processing zope.interface-4.0.3.tar.gz
Running zope.interface-4.0.3/setup.py -q bdist_egg --dist-dir /tmp/release/easy_install-86Axhl/zope.interface-4.0.3/egg-dist-tmp-2mKFn_
Adding zope.interface 4.0.3 to easy-install.pth file

Installed /release/merger/sage-5.6.beta2-debug/local/lib/python2.7/site-packages/zope.interface-4.0.3-py2.7-linux-x86_64.egg
Finished processing dependencies for transaction==1.1.1

When transaction is installing, zope.interface hasn't been installed yet and must be downloaded from the internet.

[jpflori]

I guess you can indeed easily reproduce that by shutting down your internet connection and making sure no previous installation of zope.interface is available (e.g. do not upgrade the zodb spkg on an existing sage install).

[vbraun]

• upgraded to zope.interface-4.0.3
• using easy_install --index-url dependencies <package> to use the tarballs from the dependencies directory

I've built sage-5.6.beta2 from scratch with the new spkg and verified that zodb is not downloading anything (that is, no "Downloading" or urls in the spkg log).

[jdemeyer]

Looks good to me (but it's going to be removed anyway, right?)

[vbraun]

Its probably premature to remove it in sage-5.6, let's first fix the sagenb dependency. I've created ​https://github.com/sagemath/sagenb/issues/126 to raise awareness on the sagenb side.

[kcrisman]

Just as a point of information, this was indeed added to upstream, see also #10353 where that was confirmed.