list of projective points fails to pickle in p_iter_fork call

[bhutz]

The following is a toy example showing that a list of projective points is failing to pickle. Replace the point return with just the list of coordinates works just fine.

def func(Z, p):
    Xp = Z.change_ring(GF(p))
    return Xp.rational_points()

from sage.parallel.ncpus import ncpus
from sage.parallel.use_fork import p_iter_fork

P.<x,y,z,q>=ProjectiveSpace(QQ,3)
Y=P.subscheme([x^2-3^2*y^2+z*q,x+z+4*q])

normalized_input = []
for q in primes(1,10):
    normalized_input.append(((Y, q, ), {}))

p_iter = p_iter_fork(ncpus(), 10)
points_pair = list(p_iter(func, normalized_input))

[saraedum]

a hacky script to produce the problematic pickle graph

[saraedum]

the pickle graph

[saraedum]

So, I wrote a little script (attached) that creates a graph of objects that fail to pickle here. You can look at the attached dot-file with xdot. Each node is an object that fails to pickle. Each edge points to another node that is part of that object and that also fails to pickle. (more specifically, it points to the items in __reduce_ex__(2) that fail to pickle.) Nodes that have edges going out of them are usually not problematic. They most probably fail to pickle because something inside them fails to pickle. The problem are most likely the sinks, i.e., the nodes with zero out-degree at the bottom of the graph. From my experience these are likely decorated functions/methods or creative caching constructions.

[bhutz]

I installed xdot and looked at the .dot file. I see the 4 sinks. They are all named fairly obscure things like <type Type>. Is there a way to figure out what those actually are?

[bhutz]

It almost looks like it is the defining polynomials of the subscheme that is failing to pickle not the points. And if you change the example to have

return Xp.defining_ideal()

you get the same pickling error.

[saraedum]

It's a bit tricky to find out what these are. You can run my script and add arbitrary print statements where the sinks are printed. It might be possible to connect a debugger somehow, but it's complicated because it's in a forked process. I think that all four are methods of some kind.

Btw, if the code complains about some package not being found, you can install it with sage -pip PACKAGE.

[saraedum]

Sorry, sage -pip install PACKAGE.

[raghukul01]

This bug is not specific to projective points, if we change P to AffineSpace, error is same.