tab completion crashes Sage on (console) Linux

[dimpase]

As observed on #23700, the following

sage: from sage.libs.ecl import *
sage: from sage.interfaces.maxima_lib import <TAB>

(i.e. hit <TAB> to complete the 2nd line) crashes Sage on some configs. Worse, it happens on Sage 8.0 (say, on CoCalc?), on a clean build of 8.1.beta6 on Fedora 26 (using gcc 6.4), as well as I see this on a semi-current Gentoo system (using gcc 6.3) for some time, since 8.0 at least.

See also #22766 for a different way to trigger essentially the same crash.

[dimpase]

One can either force IPython to somehow be single-thread; they rely on prompt_toolkit for tab completion, and the latter ATM always runs in a separate thread(sic!), or make ECL extension thread-safe.

[jhpalmieri]

On OS X (10.12.6, Xcode 8.3.3, Sage 8.1.beta6), I see the crash here but not the crash at #22766.

[nbruin]

ecllib quite fundamentally has problems with multiple threads, due to the fact that ECL and sage/python both have features for dealing with multiple threads, but in incompatible ways.

I think the best fix is to insert code to ensure that rebuilding the tab completion list is not built in the non-main thread (so, basically tie ecllib to one thread). I have no clue how to do that.

The simplest solution would be to not build the tab completion list on-demand, but build the list at build-time instead. The list doesn't need to be stored in user directories either, so it would improve performance characteristic on multi-user installs.

[jhpalmieri]

I notice that my .sage directory also contains giac_commandlist_cache.sobj. Should this also be constructed at build time? (I don't know if the bug for maxima would also apply to giac, probably not, but why not improve performance for multi-user installs for this case, too?)

[dimpase]

Replying to nbruin:

ecllib quite fundamentally has problems with multiple threads, due to the fact that ECL and sage/python both have features for dealing with multiple threads, but in incompatible ways.

I don't understand what kind of incompatibility you mean.

In fact, we do not need ECLs multithreading capabilities, neither as a compiler capable of dealing with Lisp threads, nor having its signals running in a separate thread. All we need is its ability to inform its GC about all the threads it has to handle (these are the ones ECL was ever invoked), this is what ecl_import_current_thread() is for (not to be called in the main thread). I suppose we don't need to call ecl_release_current_thread(), as IPython's threads persist during the session (IMHO).

I think the best fix is to insert code to ensure that rebuilding the tab completion list is not built in the non-main thread (so, basically tie ecllib to one thread). I have no clue how to do that.

In threading there is threading.current_thread() which tells you where you are; one needs to figure out how to check whether it's the main one (in Python 3 it is simple, as there is threading.main_thread() to compare with).

The simplest solution would be to not build the tab completion list on-demand, but build the list at build-time instead. The list doesn't need to be stored in user directories either, so it would improve performance characteristic on multi-user installs.

Yes, but this does not look very attractive. (Although libGAP does it even simpler, it hardcodes the names of GAP functions in a Python module...)

[jdemeyer]

It seems to me that either this ticket or #22766 should be closed as duplicate.

[dimpase]

Let us continue on ##22766 - it is about the same bug.