Upgrade: Flint 2.7.1

[slelievre]

This is to upgrade to Flint 2.7.1.

Our last upgrade was to Flint 2.6.3 in #29719.

Changes in FLINT 2.7.0 (from NEWS file)

2.7.0 -- 11 Dec 2020

• Multivariate factorisation
• Square root and square testing for finite fields
• Square root and square testing for multivariates
• Zassenhaus factoring speedups (incl. degree pruning)
• Fast factorisation of cubic univariate polynomials
• Add context objects to fmpz_mod_poly functions
• Use BLAS for matrix multiplication over Z/nZ (small n)
• Linear solving for non-square/singular matrices (can_solve)
• Speed up factorisation over Z/nZ (for multiprecision n)

From the packaging point of view the main changes will be

• configuring BLAS
• pthread support
• context object in fmpz_mod_poly function (API change)

Related:

• upstream e-antic issue ​https://github.com/flatsurf/e-antic/issues/154
• #31408: Meta-ticket: Use new features from FLINT 2.x

[slelievre]

Flint 2.7-rc2 is out. Can someone test with Sage?

• ​https://github.com/wbhart/flint2/releases
• ​sage-devel announcement: flint 2.7.0 rc1, rc2

[slelievre]

Flint 2.7.0 was released today (2020-12-18).

[arojas]

singular and e-antic fail to build due to the API changes in fmpz_mod_poly

[arojas]

Actually, Singular is fixed in the (just released) 4.2.0 version. Unfortunately doesn't help much since the Singular upgrade has been stuck for years in #25993

[arojas]

sagelib itself builds fine with flint 2.7 and passes doctests, modulo the singular upgrade.

[kedlaya]

For the record, we are now up to Flint 2.7.1. As the timing and numbering suggest, this is mainly a bugfix release.

[gh-thierry-FreeBSD]

e-antic has been fixed (in head, to be 1.0.0).

[vdelecroix]

New commits:

​73bd1a7	31069: flint version upgrade
​29c0477	31069: flint declarations

[vdelecroix]

I did not have any trouble to build singular. What was the issue?

[arojas]

Replying to vdelecroix:

I did not have any trouble to build singular. What was the issue?

Which version of singular?

[vdelecroix]

Replying to arojas:

Replying to vdelecroix:

I did not have any trouble to build singular. What was the issue?

Which version of singular?

4.1.1p2.p0 shipped with sage 9.3.beta7

[arojas]

Replying to vdelecroix:

Replying to arojas:

Replying to vdelecroix:

I did not have any trouble to build singular. What was the issue?

Which version of singular?

4.1.1p2.p0 shipped with sage 9.3.beta7

It builds because it's not using flint

checking for FLINT >= 2.4... problem
Sorry, your FLINT version is too old. Disabling.
configure: WARNING: Unable to find FLINT (which is strongly recommended) on your machine: please use --with-flint=PATH_TO_DIR_CONTAINING_LIB_AND_INCLUDE (see also ./configure --help if you do not understand what we are talking about)

[mkoeppe]

Can't we patch up the failing configure check in Singular to avoid the dependency on #25993?

[vdelecroix]

Replying to mkoeppe:

Can't we patch up the failing configure check in Singular to avoid the dependency on #25993?

That would be an alternative. But because of the API change I believe it is not enough. Let me give it a try.

[arojas]

Replying to mkoeppe:

Can't we patch up the failing configure check in Singular to avoid the dependency on #25993?

Then it will fail to build with real compilation errors, unless you also backport the flint 2.7 support patches for singular

[mkoeppe]

perhaps just Singular commit 2e7e3dc89e04ca1ef4175816cd23db6780ad5a38?

[mkoeppe]

Does anyone already have e-antic 0.1.x patches to work with FLINT 2.7?

[arojas]

Replying to mkoeppe:

Does anyone already have e-antic 0.1.x patches to work with FLINT 2.7?

Yes, ​https://github.com/archlinux/svntogit-community/blob/packages/e-antic/trunk/e-antic-flint-2.7.patch

Needs massive ifdef'ing if you still want to support flint <2.7

[mkoeppe]

Replying to arojas:

Replying to mkoeppe:

Does anyone already have e-antic 0.1.x patches to work with FLINT 2.7?

Yes, ​https://github.com/archlinux/svntogit-community/blob/packages/e-antic/trunk/e-antic-flint-2.7.patch

Needs massive ifdef'ing if you still want to support flint <2.7

Thanks! I've prepared this in ​https://github.com/mkoeppe/e-antic/tree/flint-2.7, testing currently at ​https://github.com/mkoeppe/e-antic/actions/runs/593227357

[mkoeppe]

Opened #31430 for e-antic patch (or ideally new 0.1.x release)

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​871fdbd	31069: flint version upgrade
​dbc6477	31069: flint declarations

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​797bacd	src/sage/rings/polynomial/multi_polynomial_ideal.py: Use sorted() to make doctests more stable
​31fe1eb	build/pkgs/singular/spkg-install.in: Build the documentation if the Singular tarball does not contain it
​6c01591	Merge #31439
​f3119b1	Merge tag '9.3.beta9' into t/25993/public/singular4-1-3
​49de2ff	build/pkgs/singular: Use 4.2.0p1+2021-03-13+sage
​f816f07	build/pkgs/singular/spkg-install.in: Use --disable-static on all platforms
​b0d7748	build/pkgs/singular/spkg-install.in: Build subdirs in parallel
​c33ca2b	src/sage/rings/polynomial/{laurent,multi}_polynomial_ideal: Use sorted(..., key=str) to make doctests stable
​ec471e0	build/pkgs/singular/SPKG.rst: Add link to branch
​8e3040c	Merge #25993

[mkoeppe]

There are still problems with e_antic (0.1.9):

ubuntu-trusty-maximal (​https://github.com/mkoeppe/sage/runs/2145415921 - uses FLINT from SPKG):

  [e_antic-0.1.9]   checking for gmp.h... yes
  [e_antic-0.1.9]   checking for __gmpz_add in -lgmp... yes
  [e_antic-0.1.9]   checking for flint/flint.h... yes
  [e_antic-0.1.9]   checking for flint/fmpz.h... no
  [e_antic-0.1.9]   configure: error: flint header not found

cygwin-standard (​https://github.com/mkoeppe/sage/runs/2145195263 - uses system FLINT 2.7)

libtool: compile:  gcc -DHAVE_CONFIG_H -I. -fopenmp -O2 -g -march=native -c poly_extra/fmpz_nextprime.c -o poly_extra/fmpz_nextprime.o >/dev/null 2>&1
poly_extra/fmpz_randprime.c:17:6: error: conflicting types for 'fmpz_randprime'
   17 | void fmpz_randprime(fmpz_t f, flint_rand_t state, mp_bitcnt_t bits, int proved)
      |      ^~~~~~~~~~~~~~
In file included from /usr/include/flint/fmpz_poly.h:33,
                 from ./e-antic/poly_extra.h:17,
                 from poly_extra/fmpz_randprime.c:13:
/usr/include/flint/fmpz.h:188:16: note: previous declaration of 'fmpz_randprime' was here
  188 | FLINT_DLL void fmpz_randprime(fmpz_t f, flint_rand_t state,
      |                ^~~~~~~~~~~~~~
make[4]: *** [Makefile:2699: poly_extra/fmpz_randprime.lo] Error 1

[mkoeppe]

OTOH, on ubuntu-xenial-maximal (​https://github.com/mkoeppe/sage/runs/2145415962)and ubuntu-bionic-maximal (​https://github.com/mkoeppe/sage/runs/2145416018), e_antic installs OK. We can just write off ubuntu-trusty - Normaliz does not compile with the ancient compiler there either.

[mkoeppe]

A serious issue: On ubuntu-focal-standard (​https://github.com/mkoeppe/sage/runs/2145416039)

  [sagelib-9.3.beta9]   In file included from build/cythonized/sage/data_structures/bounded_integer_sequences.c:3535:
  [sagelib-9.3.beta9]   sage/libs/flint/flint_wrap.h: At top level:
  [sagelib-9.3.beta9]   sage/libs/flint/flint_wrap.h:39:10: fatal error: flint/fmpz_mod.h: No such file or directory
  [sagelib-9.3.beta9]      39 | #include <flint/fmpz_mod.h>
  [sagelib-9.3.beta9]         |          ^~~~~~~~~~~~~~~~~~
  [sagelib-9.3.beta9]   compilation terminated.

This is using system flint 2.5.2 (see ​https://repology.org/project/flint/versions)

[mkoeppe]

(fmpz_mod.h was introduced in 2.6.0 according to ​https://github.com/wbhart/flint2/blob/trunk/NEWS)

[mkoeppe]

I think dropping support for system FLINT 2.5.x is too disruptive for Sage 9.3, given that the new features are not used yet by anything.

So I propose to split the ticket into just the upgrade and put the Cython interface to the new FLINT functions on a follow-up ticket for Sage 9.4.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​8c71a1c

31069: flint version upgrade

[mkoeppe]

Replying to mkoeppe:

I propose to split the ticket into just the upgrade and put the Cython interface to the new FLINT functions on a follow-up ticket for Sage 9.4.

This is now #31525.

[mjo]

I think this is OK. The main issues were in other packages that have already been fixed. With system flint-2.7.1, I see only one doctest failure due to a timeout in pari that probably cannot be blamed on flint.

I see "unrelated" failures in the CI runs as well. If you are convinced of flint's innocence, we can set this to positive review.

[dimpase]

lgtm too

[mkoeppe]

Thanks!

[mkoeppe]

Setting priority to blocker to bring this ticket to the attention of the release bot.

[roed]

After merging this ticket into #31548 (since I want to use new functionality from FLINT 2.7.1 there) I'm getting errors starting Sage. I don't usually update spkgs outside of beta releases, so I could be doing something wrong (I'm running OS X 10.15.2).

$ git trac fetch 31069 && git merge
$ make build
...
Sage build/upgrade complete!
$ sage
ImportError Python 3.9.1: /Users/roed/sage/sage-9.3.beta7/local/bin/python3
...
ImportError: dlopen(/Users/roed/sage/sage-9.3.beta7/local/lib/python3.9/site-packages/sage/matrix/matrix_mpolynomial_dense.cpython-39-darwin.so, 2): Library not loaded: /Users/roed/sage/sage-9.3.beta7/local/lib/libSingular-4.1.1.dylib
  Referenced from: /Users/roed/sage/sage-9.3.beta7/local/lib/python3.9/site-packages/sage/matrix/matrix_mpolynomial_dense.cpython-39-darwin.so
  Reason: image not found

Full install.log available ​here.

[fbissey]

I believe you need singular 4.2.0p1 (9.3.beta8 if I am not mistaken) with that. Singular uses flint so does need rebuilding and 4.2.0 is the first one that is compatible with flint-2.7.1.

[roed]

Despite the folder name, this is based on sage-9.3.beta9, and when I did make build it appeared to be building singular-4.2.0p1+2021-03-13+sage.

[roed]

I feel like I'm missing something basic here: when I try running sage -f flint it gives me lots of errors saying that flint-2.7.1 isn't found:

[flint-2.7.1] Attempting to download package flint-2.7.1.tar.gz from mirrors
[flint-2.7.1] http://mirrors.mit.edu/sage/spkg/upstream/flint/flint-2.7.1.tar.gz
[flint-2.7.1] [xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
[flint-2.7.1] ERROR [transfer|run:135]: [Errno 404] Not Found: '//mirrors.mit.edu/sage/spkg/upstream/flint/flint-2.7.1.tar.gz'
[flint-2.7.1] http://www.mirrorservice.org/sites/www.sagemath.org/spkg/upstream/flint/flint-2.7.1.tar.gz
...
[flint-2.7.1] FileNotMirroredError: tarball does not exist on mirror network
[flint-2.7.1] ************************************************************************
[flint-2.7.1] ************************************************************************
[flint-2.7.1] Error downloading flint-2.7.1.tar.gz

Do I need to download a tarball manually from somewhere?

[fbissey]

Replying to roed:

Despite the folder name, this is based on sage-9.3.beta9, and when I did make build it appeared to be building singular-4.2.0p1+2021-03-13+sage.

OK, then it appears that somehow sage/matrix/matrix_mpolynomial_dense.cpython-39-darwin.so has not be rebuilt because it is still looking for libSingular-4.1.1 when it should look for libSingular-4.2.0.

[fbissey]

Replying to roed:

I feel like I'm missing something basic here: when I try running sage -f flint it gives me lots of errors saying that flint-2.7.1 isn't found:

[flint-2.7.1] Attempting to download package flint-2.7.1.tar.gz from mirrors
[flint-2.7.1] http://mirrors.mit.edu/sage/spkg/upstream/flint/flint-2.7.1.tar.gz
[flint-2.7.1] [xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
[flint-2.7.1] ERROR [transfer|run:135]: [Errno 404] Not Found: '//mirrors.mit.edu/sage/spkg/upstream/flint/flint-2.7.1.tar.gz'
[flint-2.7.1] http://www.mirrorservice.org/sites/www.sagemath.org/spkg/upstream/flint/flint-2.7.1.tar.gz
...
[flint-2.7.1] FileNotMirroredError: tarball does not exist on mirror network
[flint-2.7.1] ************************************************************************
[flint-2.7.1] ************************************************************************
[flint-2.7.1] Error downloading flint-2.7.1.tar.gz

Do I need to download a tarball manually from somewhere?

This ticket is not merged, tarball are not usually mirrored before Volker attempt to merge the tickets. You'll have to fetch the source a different way. Matthias created a handy way of using a string called upstream_url in the file checksum.ini to do the fetching from upstream but I cannot remember how it works.

[slelievre]

Do this:

$ ./bootstrap -q
$ ./configure --enable-download-from-upstream-url -q
$ make -s V=0

[roed]

Thanks! That seems to have worked.

[roed]

For anyone else looking at this, I also had to do sage -f flint after the commands Samuel gave.

[slelievre]

This should do everything at once:

$ ./bootstrap -q
$ ./configure --enable-download-from-upstream-url -q
$ make -s V=0 flint-clean
$ make -s V=0

[mkoeppe]

Looks unrealistic to squeeze this into 9.3