Opened 8 years ago

Closed 8 years ago

Last modified 7 years ago

#14405 closed defect (fixed)

Replace termcap with ncurses

Reported by: vbraun Owned by: jdemeyer
Priority: major Milestone: sage-5.12
Component: packages: standard Keywords:
Cc: jdemeyer, jpflori, leif Merged in: sage-5.12.beta2
Authors: Volker Braun, Jean-Pierre Flori Reviewers: Volker Braun, Jean-Pierre Flori, Jeroen Demeyer
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by jdemeyer)

The readline in IPython is broken if ncurses-devel is not installed in various modern Linux distributions. Installing the ncurses headers from the distribution (yum install ncurses-devel on Fedora) fixes this.

Example video: http://youtu.be/dxp1SIXbIyY

The problem seems to be that the ancient libtermcap (no new releases since March 2002) is not enough anymore. I suggest to replace the termcap spkg with ncurses. Sage-5.9.beta3 builds fine. The only downside is that it takes ~2mb instead of ~300k, but on the upside the terminal is no longer broken.

Attachments (5)

log-python-2.7.3.p5.diff (15.3 KB) - added by vbraun 8 years ago.
Diff between broken and fixed python install
trac_14405_replace_termcap_with_ncurses.patch (1.7 KB) - added by vbraun 8 years ago.
Updated patch
readline-p3-to-p4.patch (2.1 KB) - added by vbraun 8 years ago.
for review only
14405_hgignore.patch (439 bytes) - added by jdemeyer 8 years ago.
readline-tinfo.diff (10.5 KB) - added by vbraun 8 years ago.
for review only

Download all attachments as: .zip

Change History (101)

comment:1 Changed 8 years ago by leif

  • Cc leif added

comment:2 Changed 8 years ago by jdemeyer

Does Python's readline work correctly with the old termcap spkg (http://boxen.math.washington.edu/home/release/sage-5.6/sage-5.6/spkg/standard/termcap-1.3.1.p2.spkg)? Because if it worked neither before #12725, nor after #12725, there is no bug.

If Python's readline did work before #12725 and #12725 broke it: please attach logs of the Python compilation, before and after (with the same Sage version, serial install).

comment:3 follow-up: Changed 8 years ago by jpflori

I've tested an old 4.8 install which works fine whereas I found the same problem as Volker on a recent 5.9.beta2. This is on two different Ubuntu 12.04.1 installs where libncurses5 and libncursesw5 packages are installed but no -dev packages are.

comment:4 in reply to: ↑ 3 Changed 8 years ago by jdemeyer

Replying to jpflori:

I've tested an old 4.8 install which works fine whereas I found the same problem as Volker on a recent 5.9.beta2. This is on two different Ubuntu 12.04.1 installs where libncurses5 and libncursesw5 packages are installed but no -dev packages are.

OK, then please try the things I asked to Volker: try without #12725 and show the Python logs so we can see what the problem is.

comment:5 Changed 8 years ago by jpflori

As I already typed that, here is a bit of info:

  • On the 4.8 install, I have the usual "Python build finished, but the necessary bits to build these modules were not found" and the modules listed include _curses and _curses_panel.
  • On the 5.9.beta2 install, it only lists the _curses_panel module and I have a new worrying line "Failed to build these modules" which only lists _curses. And indeed before in the log, there's an error:
    In file included from .../_cursesmodule.c:114:0:
    Include/py_curses.h:57:20: erreur fatale: curses.h : Aucun fichier ou dossier de ce type
    

comment:6 follow-up: Changed 8 years ago by jpflori

(And installing the latest Python spkg on the 4.8 Sage install gives the same result as the old 2.6.* spkg which it included, i.e. missing bits for _curses and _curses_panel.)

comment:7 in reply to: ↑ 6 Changed 8 years ago by jdemeyer

Replying to jpflori:

(And installing the latest Python spkg on the 4.8 Sage install gives the same result as the old 2.6.* spkg which it included, i.e. missing bits for _curses and _curses_panel.)

But the end result is the same: the _curses module doesn't build (either due to "missing bits" or "failed to build"), neither with #12725, nor without. So I still doubt that #12725 is a problem.

What happens with other packages using readline, PARI/GP for example? How is the line editing in

$ ./sage --gp

And what is the header message when starting GP, it indicates whether or not readline is used.

comment:8 follow-up: Changed 8 years ago by jpflori

And after removing the libncurses.a symlink and rebuilding the Python 2.7 spkg on the 5.9.beta2, the log looks similar to the one on the 4.8 install. But Sage command line still seems broken (I've rebuilt ipython and run ./sage -b).

I could not test the new python on the 4.8 install as it does not like it that much and fails to rebuild.

No such problems on 5.6 and 5.7, so I guess that #12725 is not the culprit for the bug (although the error message about _curses I get shows it may not be that useful). Maybe new ipython?

comment:9 follow-up: Changed 8 years ago by vbraun

The standard Fedora install contains ncurses and ncurses-libs rpms, which install libncurses.so.5 but no libncurses.so. Only the ncurses-devel package, which is not installed by default, brings libncurses.so.

comment:10 in reply to: ↑ 8 Changed 8 years ago by jpflori

Replying to jpflori:

And after removing the libncurses.a symlink and rebuilding the Python 2.7 spkg on the 5.9.beta2, the log looks similar to the one on the 4.8 install. But Sage command line still seems broken (I've rebuilt ipython and run ./sage -b).

I could not test the new python on the 4.8 install as it does not like it that much and fails to rebuild.

No such problems on 5.6 and 5.7, so I guess that #12725 is not the culprit for the bug (although the error message about _curses I get shows it may not be that useful). Maybe new ipython?

Ipython was updated in 5.7, so that's not it.

comment:11 Changed 8 years ago by vbraun

I've removed ncurses-devel again and compiled sage-5.9.beta2 with the old termcap-1.3.1.p2.spkg. The python command line is still borked in the same way. But no libncurses symlink. So I guess the bug is somewhere else.

comment:12 Changed 8 years ago by jpflori

Replying to jdemeyer:

Replying to jpflori:

(And installing the latest Python spkg on the 4.8 Sage install gives the same result as the old 2.6.* spkg which it included, i.e. missing bits for _curses and _curses_panel.)

But the end result is the same: the _curses module doesn't build (either due to "missing bits" or "failed to build"), neither with #12725, nor without. So I still doubt that #12725 is a problem.

What happens with other packages using readline, PARI/GP for example? How is the line editing in

$ ./sage --gp

And what is the header message when starting GP, it indicates whether or not readline is used.

I've now rebuilt readline, ipython, pari as well, still without the libncurses.a symlink. ./sage -gp works fine (and shows realine 6.2 is enabled), and sage command line still broken.

comment:13 Changed 8 years ago by jpflori

Note the ./sage -python command line is fine, whereas ./sage -ipython is not.

comment:14 Changed 8 years ago by jpflori

In Sage 5.7, the sage command line is fine whereas the ipython one is already (in color) and broken.

comment:15 Changed 8 years ago by jpflori

In fact with the old ipython 0.10 from sage 5.6, the command line is already broken.

comment:16 Changed 8 years ago by vbraun

Last edited 8 years ago by jdemeyer (previous) (diff)

comment:17 in reply to: ↑ 9 Changed 8 years ago by leif

Replying to vbraun:

The standard Fedora install contains ncurses and ncurses-libs rpms, which install libncurses.so.5 but no libncurses.so. Only the ncurses-devel package, which is not installed by default, brings libncurses.so.

Which is pretty correct. No system should have foo.so (nor foo.a!) unless foo.h (i.e., the related headers) are also present.

comment:18 follow-up: Changed 8 years ago by vbraun

To go from broken -> fixed: Install ncurses-devel, recompile readline, recompile python.

comment:19 follow-up: Changed 8 years ago by vbraun

The sage -gp readline also is different between the working and broken sage commandline. If I compile the working readline, sage -gp has correct multi-line readline (input is continued on subsequent lines. With the broken readline, sage -gp falls back to single-line readline (input scrolls horizontally if necessary).

comment:20 Changed 8 years ago by vbraun

Curiously, the log file for readline is identical (apart from the timing stats). But it fails to link to tinfo in the broken version

[vbraun@volker-desktop sage-5.9.beta2]$ ldd local/lib/libreadline.so
	linux-vdso.so.1 =>  (0x00007fff365fe000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f2866ee3000)
	/lib64/ld-linux-x86-64.so.2 (0x0000003c14800000)

vs the fixed version:

[vbraun@volker-desktop sage-5.9.beta2]$ ldd local/lib/libreadline.so
	linux-vdso.so.1 =>  (0x00007fffef3fe000)
	libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00007ffb0c10a000)
	libc.so.6 => /lib64/libc.so.6 (0x00007ffb0bd51000)
	/lib64/ld-linux-x86-64.so.2 (0x0000003c14800000)
[vbraun@volker-desktop sage-5.9.beta2]$ cat /usr/lib64/libncurses.so
INPUT(libncurses.so.5 -ltinfo)

Changed 8 years ago by vbraun

Diff between broken and fixed python install

comment:21 in reply to: ↑ 19 Changed 8 years ago by jpflori

Replying to vbraun:

The sage -gp readline also is different between the working and broken sage commandline. If I compile the working readline, sage -gp has correct multi-line readline (input is continued on subsequent lines. With the broken readline, sage -gp falls back to single-line readline (input scrolls horizontally if necessary).

With all versions I have, I get horizontla scrolling indeed (but not borken as with the ipython command prompt).

comment:22 Changed 8 years ago by jdemeyer

Replying to vbraun:

Deleting (renaming) $SAGE_LOCAL/share fixes this, even without recompiling.

So the issue is IPython configuration then?

comment:23 in reply to: ↑ 18 ; follow-up: Changed 8 years ago by jpflori

Replying to vbraun:

To go from broken -> fixed: Install ncurses-devel, recompile readline, recompile python.

But I guess this "fixed" readline links to libncurses, doesn't it? We don't want that... We want to use the libtermcap we provide.

comment:24 Changed 8 years ago by vbraun

In fact, I don't need to recompile python. Just install ncurses-devel and recompile readline = working. Uninstall ncurses-devel and recompile readline = broken.

comment:25 Changed 8 years ago by jdemeyer

  • Description modified (diff)
  • Summary changed from The libncurses hack breaks Python/readline to IPython readline broken if ncurses-devel is not installed

comment:26 follow-up: Changed 8 years ago by vbraun

The real reason is libtinfo. In fact I can just export LD_PRELOAD=/usr/lib64/libtinfo.so.5 and this fixes it.

comment:27 follow-up: Changed 8 years ago by vbraun

So what should we do, make a tinfo spkg or require ncurses-devel?

comment:28 in reply to: ↑ 23 ; follow-up: Changed 8 years ago by leif

Replying to jpflori:

Replying to vbraun:

To go from broken -> fixed: Install ncurses-devel, recompile readline, recompile python.

But I guess this "fixed" readline links to libncurses, doesn't it? We don't want that... We want to use the libtermcap we provide.

Not me. We shouldn't build libtermcap at all on systems that have libncurses / libtinfo [-dev].

comment:29 in reply to: ↑ 27 Changed 8 years ago by jdemeyer

Replying to vbraun:

So what should we do, make a tinfo spkg or require ncurses-devel?

Fix IPython?

comment:30 in reply to: ↑ 28 Changed 8 years ago by jpflori

Replying to leif:

Replying to jpflori:

Replying to vbraun:

To go from broken -> fixed: Install ncurses-devel, recompile readline, recompile python.

But I guess this "fixed" readline links to libncurses, doesn't it? We don't want that... We want to use the libtermcap we provide.

Not me. We shouldn't build libtermcap at all on systems that have libncurses / libtinfo [-dev].

And what about systems not having any -dev packages?

comment:31 in reply to: ↑ 26 Changed 8 years ago by jpflori

Replying to vbraun:

The real reason is libtinfo. In fact I can just export LD_PRELOAD=/usr/lib64/libtinfo.so.5 and this fixes it.

I don't have libtinfo on my systems :)

comment:32 Changed 8 years ago by jpflori

I guess Ubuntu packages it inside libncurses: https://bugs.launchpad.net/ubuntu/+source/ncurses/+bug/259139

comment:33 follow-up: Changed 8 years ago by vbraun

Fedora packages libtinfo also as part of ncurses.

Jeroen, please read my edits on comment:16

comment:34 follow-up: Changed 8 years ago by jpflori

If I run LD_PRELOAD=...libncurses.so.5 ./sage -ipython then long command lines are ok. Notice that the same experiment with ./sage -gp still gives me horizontal scrolling.

comment:35 in reply to: ↑ 33 Changed 8 years ago by jdemeyer

Replying to vbraun:

Jeroen, please read my edits on comment:16

OK, I didn't understand your edit, I just removed the whole comment, which is actually more clear.

comment:36 Changed 8 years ago by jpflori

What's strange is that iPython runs "import curses" which fails (whether lincurses is preloaded or not) and so should behave correctly without curses installed.

comment:37 in reply to: ↑ 34 Changed 8 years ago by vbraun

Replying to jpflori:

If I run LD_PRELOAD=...libncurses.so.5 ./sage -ipython then long command lines are ok. Notice that the same experiment with ./sage -gp still gives me horizontal scrolling.

Hmm, strange. Works for me: with LD_PRELOAD, sage -gp has multi-line readline. Without it, only single-line.

comment:38 follow-up: Changed 8 years ago by vbraun

libtinfo is actually part of ncurses (distributed inside the ncurses tarball)

comment:39 in reply to: ↑ 38 Changed 8 years ago by jpflori

Replying to vbraun:

libtinfo is actually part of ncurses (distributed inside the ncurses tarball)

Yup, and on Ubuntu, there is even no separate libtinfo.so file (except for the libncurses-dbg debug package). The functions it should provided are actually provided by libncurses.so directly.

comment:40 Changed 8 years ago by vbraun

Ubuntu 12.04 has /lib/i386-linux-gnu/libtinfo.so.5 and /lib/x86_64-linux-gnu/libtinfo.so.5.

comment:41 follow-up: Changed 8 years ago by jpflori

Oh yes it has... Please disregard all my rambling then.

comment:42 in reply to: ↑ 41 Changed 8 years ago by leif

Replying to jpflori:

Oh yes it has...

Otherwise

INPUT(libncurses.so.5 -ltinfo)

wouldn't make sense... ;-)

Please disregard all my rambling then.

comment:43 Changed 8 years ago by vbraun

I made some experimental spkgs for

comment:44 Changed 8 years ago by vbraun

  • Authors set to Volker Braun
  • Description modified (diff)
  • Summary changed from IPython readline broken if ncurses-devel is not installed to Replace termcap with ncurses

Changed 8 years ago by vbraun

Updated patch

comment:45 Changed 8 years ago by vbraun

  • Description modified (diff)

Changed 8 years ago by vbraun

for review only

comment:46 Changed 8 years ago by vbraun

  • Status changed from new to needs_review

comment:47 Changed 8 years ago by jpflori

Just for completeness, I'd like to see if its possible to fix iPython so its not broken when using termcap only. I'll have some time tonight or tomorrow hopefully.

comment:48 follow-up: Changed 8 years ago by vbraun

Its really readline that breaks, as demonstrated by the fact that it falls back to single-line editing. Ihmo its pointless to improve support for the dead libtermcap, but then its your time ;-)

comment:49 Changed 8 years ago by jdemeyer

  • Milestone changed from sage-5.9 to sage-5.10

comment:50 in reply to: ↑ 48 Changed 8 years ago by jpflori

Replying to vbraun:

Its really readline that breaks, as demonstrated by the fact that it falls back to single-line editing. Ihmo its pointless to improve support for the dead libtermcap, but then its your time ;-)

Can we consider it is broken because it uses single-line editing mode? Isn't that to be expected with termcap? (I really have no clue...)

Note that this single-line mode is ok in gp and python, it's only in ipython its broken, so there is a problem with ipython (and its been there since at least 0.10, we just encountered it recently in Sage because the way we use ipython must have changed).

Anyway, I agree that multi-line editing looks nicer so just ignoring this ipython/readline/termcap problem and using ipython/readlines/ncurses is a sensible workaround/fix.

comment:51 Changed 8 years ago by jpflori

I see you disabled static libs, any reason for that except for the fact "shared libraries are better" (or "static ones sucks" :)).

comment:52 Changed 8 years ago by vbraun

Only because static libraries suck.

comment:53 Changed 8 years ago by vbraun

About the horizontal-scroll-mode, that just shows that readline couldn't figure out the UP capability. Its pretty clear that this is because we only installed libtermcap, but not an actual termcap data file. So libtermcap used to fall back to /etc/termcap, but newer Linux distributions don't include that any more since libtermcap is being phased out in favor of ncurses.

comment:54 Changed 8 years ago by jpflori

Good point. Ubuntu 12.04 does not ship /etc/termcap indeed. Although that does not clearly explain the discrepancy between the behavior of ipython vs. python and gp.

comment:55 Changed 8 years ago by jhpalmieri

The ncurses spkg fails to build on OS X 10.8.3. log file here.

Last edited 8 years ago by jhpalmieri (previous) (diff)

comment:56 Changed 8 years ago by vbraun

Fixed, new spkg at same place. I checked that it compiles on bsd.math.

comment:57 Changed 8 years ago by jpflori

Just a reminder for myself: HAVE TO FIX THIS. Either find out how ipython breaks with termcap only or be lazy and use Volker spkgs.

The broken command line edit is just driving me crazy, I cannot understand why the google groups are not flooded by rants about that. Does every one use the notebook?

comment:58 Changed 8 years ago by vbraun

We should switch to ncurses anyways, I don't see a point in adding workarounds for termcap.

I guess most users already have ncurses headers in their OS install, if you install any development packages then its very likely to be pulled in.

comment:59 Changed 8 years ago by jpflori

Agreed, I'll try to review it tonight.

If by any chance there have been upgrades to curses and readline, could you upload new spkgs?

comment:60 Changed 8 years ago by vbraun

Neither readline nor ncurses has been updated in the last 3 months... neither of them is exactly a moving target ;-)

comment:61 Changed 8 years ago by jpflori

Yup I saw that.

Cleaned up spkg are up.

I'm testing them right now.

comment:62 follow-up: Changed 8 years ago by jpflori

I've removed obsolete comments, homogenized spkg scripts and so on. The only thing Im not sure about is the LIB_TERMCAP patch which was added in #11970 is needed anymore.

The only thing you did I changed is to not disable static libs :) Please let keep that change for later and for a more general clean up of spkgs regarding static libs.

comment:63 Changed 8 years ago by jpflori

Spkgs at:

termcap patch was needed to finish building Sage. Looks to be working. ldd'ing $SAGE_LOCAL/lib shows only ncurses in the same dir.

comment:64 Changed 8 years ago by vbraun

spkg urls don't work

comment:66 Changed 8 years ago by jpflori

Strange, yesterday the doc failed to build because of not finding some UP stuff in readline, and I thought it was solved by readding the termcap patch. But now on another machine it fails though I used the patch.

comment:67 Changed 8 years ago by jpflori

Maybe that was because of old termcap libs being around on this setup were I. Removing them and rebuilding everything was fine...

comment:68 Changed 8 years ago by vbraun

And this is why we shouldn't have static libraries (sorry couldn't resist ;-)

comment:69 follow-up: Changed 8 years ago by vbraun

Whats the point of

    [ -r "$patch" ] || continue  # Skip non-existing or non-readable patches

IMHO we should error out if the spkg permissions are not set correctly instead of silently not applying the patch.

comment:70 Changed 8 years ago by vbraun

The rest looks good to me (modulo building static libraries, though I don't mind disabling them globally after we switch to git and the new repo layout)

comment:71 in reply to: ↑ 69 Changed 8 years ago by leif

Replying to vbraun:

Whats the point of

    [ -r "$patch" ] || continue  # Skip non-existing or non-readable patches

IMHO we should error out if the spkg permissions are not set correctly instead of silently not applying the patch.

That's Jeroen's invention, the idea being to be able to (temporarily) "disable" a patch by chmod -r ... without having to change spkg-install (which usually unconditionally applies all [readable] patches in a loop).

And if there are no patches at all, patch doesn't bail out because the shell doesn't expand .../*.patch (where [ -f ... ] would be sufficient).

comment:72 in reply to: ↑ 62 Changed 8 years ago by leif

Replying to jpflori:

The only thing Im not sure about is the LIB_TERMCAP patch which was added in #11970 is needed anymore.

Did you finally remove it? (Too lazy to look at the spkg now.)

If so, please make sure R still builds on $DISTRO...

comment:73 Changed 8 years ago by jpflori

I did at first, but in a build from scratch the doc failed to build complaining about UP missing and after rebuilding readline with it, it was ok so I've left it in.

Another installation of just the two spkg after that on a system where Sage was already installed, with the patch in, also failed with the same error. But it seems it was because a previous libtermcap.a was already in $SAGE_LOCAL/lib when the spkgs were built. At least removing libtermcap.a and rebuilding once more let the doc build. Note I've not looked deeply into this and this is mostly experimental. But I seem to remember that in this latter case, the first libreadline.so produced was not linked with libtinfo (that ncurses should now provide, but I did not look to see if it was there during that try) and that the second one was. Not completely sure about that though.

comment:74 Changed 8 years ago by vbraun

  • Authors changed from Volker Braun to Volker Braun, Jean-Pierre Flori
  • Reviewers set to Volker Braun, Jean-Pierre Flori
  • Status changed from needs_review to positive_review

R builds for me, even has working readline in the standalone R binary.

comment:75 Changed 8 years ago by jdemeyer

  • Milestone changed from sage-5.11 to sage-5.12

comment:76 Changed 8 years ago by jdemeyer

  • Status changed from positive_review to needs_work

The spkg needs some trivial work:

In SPKG.txt, the license should be GPL v3+ and the following paragraph needs to be updated:

== Dependencies ==

 * GNU patch

 * libtermcap (or libncurses or libtinfo), but not Sage's one,
   since we only build a static version. I.e., the system has
   to provide one of these.

Also, the file src/support/mkinstalldirs is not user-writable, which is strange.

comment:77 Changed 8 years ago by vbraun

  • Description modified (diff)
  • Status changed from needs_work to positive_review

I've updated the readline spkg documentation.

The permissions of mkinstalldirs are what upstream uses; since its not copied anywhere it doesn't really matter if it is user-writeable or not.

comment:78 Changed 8 years ago by jdemeyer

  • Status changed from positive_review to needs_work

Some files in $SAGE_LOCAL/bin need to be added to .hgignore:

? captoinfo
? clear
? infocmp
? infotocap
? ncurses5-config
? reset
? tabs
? tic
? toe
? tput
? tset

Changed 8 years ago by jdemeyer

comment:79 Changed 8 years ago by jdemeyer

  • Description modified (diff)
  • Status changed from needs_work to needs_review

comment:80 Changed 8 years ago by vbraun

  • Status changed from needs_review to positive_review

Thanks!

comment:81 Changed 8 years ago by jdemeyer

  • Status changed from positive_review to needs_work

Buildbot failure on snapperkob (Ubuntu 12.04 x86_64) while building Python:

building 'readline' extension
gcc -pthread -fPIC -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -I/home/buildbot/build/sage/snapperkob/snapperkob_full/build/sage-5.12.beta1/local/include -I. -IInclude -I./Include -I/usr/include/x86_64-linux-gnu -I/usr/local/include -I/home/buildbot/build/sage/snapperkob/snapperkob_full/build/sage-5.12.beta1/spkg/build/python-2.7.5.p1/src/Include -I/home/buildbot/build/sage/snapperkob/snapperkob_full/build/sage-5.12.beta1/spkg/build/python-2.7.5.p1/src -c /home/buildbot/build/sage/snapperkob/snapperkob_full/build/sage-5.12.beta1/spkg/build/python-2.7.5.p1/src/Modules/readline.c -o build/temp.linux-x86_64-2.7/home/buildbot/build/sage/snapperkob/snapperkob_full/build/sage-5.12.beta1/spkg/build/python-2.7.5.p1/src/Modules/readline.o
gcc -pthread -shared -L/home/buildbot/build/sage/snapperkob/snapperkob_full/build/sage-5.12.beta1/local/lib -L/home/buildbot/build/sage/snapperkob/snapperkob_full/build/sage-5.12.beta1/local/lib build/temp.linux-x86_64-2.7/home/buildbot/build/sage/snapperkob/snapperkob_full/build/sage-5.12.beta1/spkg/build/python-2.7.5.p1/src/Modules/readline.o -L/usr/lib/termcap -L/home/buildbot/build/sage/snapperkob/snapperkob_full/build/sage-5.12.beta1/local/lib -L/usr/lib/x86_64-linux-gnu -L/usr/local/lib -L. -lreadline -lncurses -lpython2.7 -o build/lib.linux-x86_64-2.7/readline.so
*** WARNING: renaming "readline" since importing it failed: /home/buildbot/build/sage/snapperkob/snapperkob_full/build/sage-5.12.beta1/local/lib/libreadline.so.6: undefined symbol: UP

This causes in the end

Testing importing of various modules...
ctypes module imported OK
math module imported OK
hashlib module imported OK
crypt module imported OK
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named readline
readline module failed to import
socket module imported OK
Error: One or more modules failed to import.
Last edited 8 years ago by jdemeyer (previous) (diff)

comment:82 Changed 8 years ago by jdemeyer

Exact same failure on arando (Ubuntu 13.04 32-bit).

The problem might be that libreadline links against -lcurses while Python uses -lncurses.

Version 0, edited 8 years ago by jdemeyer (next)

comment:83 Changed 8 years ago by jdemeyer

No, the issue that, despite the -lncurses in the gcc invocation, Python's readline.so does not actually link against `libncurses.so:

(sage-sh) buildbot@arando:src$ ldd build/lib.linux-i686-2.7/readline.so
        linux-gate.so.1 =>  (0xb778b000)
        libreadline.so.6 => /var/lib/buildbot/build/sage/arando-1/arando_full/build/sage-5.12.beta1/local/lib/libreadline.so.6 (0xb7746000)
        libpython2.7.so.1.0 => /var/lib/buildbot/build/sage/arando-1/arando_full/build/sage-5.12.beta1/local/lib/libpython2.7.so.1.0 (0xb758e000)
        libpthread.so.0 => /lib/i386-linux-gnu/libpthread.so.0 (0xb7564000)
        libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xb73b1000)
        libdl.so.2 => /lib/i386-linux-gnu/libdl.so.2 (0xb73ac000)
        libutil.so.1 => /lib/i386-linux-gnu/libutil.so.1 (0xb73a8000)
        libm.so.6 => /lib/i386-linux-gnu/libm.so.6 (0xb7365000)
        /lib/ld-linux.so.2 (0xb778c000)

Some dynamic linking expert is needed here.

Running Python with LD_PRELOAD="$SAGE_LOCAL/lib/libncurses.so" solves the problem.

Last edited 8 years ago by jdemeyer (previous) (diff)

comment:84 Changed 8 years ago by jdemeyer

The issue is https://lists.ubuntu.com/archives/ubuntu-devel/2010-November/031991.html

Adding -Wl,--no-as-needed to the linker invocation solves the problem.

gcc -pthread -shared -L/var/lib/buildbot/build/sage/arando-1/arando_full/build/sage-5.12.beta1/local/lib -L/var/lib/buildbot/build/sage/arando-1/arando_full/build/sage-5.12.beta1/local/lib build/temp.linux-i686-2.7/var/lib/buildbot/build/sage/arando-1/arando_full/build/sage-5.12.beta1/spkg/build/python-2.7.5.p1/src/Modules/readline.o -L/usr/lib/termcap -L/var/lib/buildbot/build/sage/arando-1/arando_full/build/sage-5.12.beta1/local/lib -L/usr/lib/i386-linux-gnu -L/usr/local/lib -L. -lreadline -Wl,--no-as-needed -lncurses -lpython2.7 -o build/lib.linux-i686-2.7/readline.so

comment:85 Changed 8 years ago by vbraun

libcurses is just a symlink to libncurses. I don't have an account on any of those machines. Can you post the ldd output for libncurses.so and libreadline.so? I would guess that the latter is underlinked, and since python readline.so doesn't use UP directly we end up not pulling in ncurses.

comment:86 Changed 8 years ago by jdemeyer

(sage-sh) buildbot@arando:sage$ ldd $SAGE_LOCAL/lib/libncurses.so
        linux-gate.so.1 =>  (0xb7759000)
        libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xb7573000)
        libtinfo.so.5 => /var/lib/buildbot/build/sage/arando-1/arando_full/build/sage-5.12.beta1/local/lib/libtinfo.so.5 (0xb7549000)
        /lib/ld-linux.so.2 (0xb775a000)
(sage-sh) buildbot@arando:sage$ ldd $SAGE_LOCAL/lib/libreadline.so
        linux-gate.so.1 =>  (0xb773a000)
        libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xb753a000)
        /lib/ld-linux.so.2 (0xb773b000)

comment:87 Changed 8 years ago by jdemeyer

Contrast with boxen.math (Ubuntu 8.04):

(sage-sh) jdemeyer@boxen:sage-5.12.beta1$ ldd $SAGE_LOCAL/lib/libncurses.so
        linux-vdso.so.1 =>  (0x00007fff2fffe000)
        libc.so.6 => /lib/libc.so.6 (0x00007ffc27946000)
        libtinfo.so.5 => /mazur/release/merger/sage-5.12.beta1/local/lib/libtinfo.so.5 (0x00007ffc27712000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ffc27ee2000)
(sage-sh) jdemeyer@boxen:sage-5.12.beta1$ ldd $SAGE_LOCAL/lib/libreadline.so
        linux-vdso.so.1 =>  (0x00007fff099fe000)
        libncurses.so.5 => /mazur/release/merger/sage-5.12.beta1/local/lib/libncurses.so.5 (0x00007f8b01386000)
        libc.so.6 => /lib/libc.so.6 (0x00007f8b0100f000)
        libtinfo.so.5 => /mazur/release/merger/sage-5.12.beta1/local/lib/libtinfo.so.5 (0x00007f8b00ddc000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f8b017f0000)

comment:88 Changed 8 years ago by vbraun

  • Status changed from needs_work to needs_review

Yes, it is exactly as I suspected: libreadline does not link against libncurses. In fact, the terminal handling stuff is in libtinfo (part of ncurses), this is why the -lcurses is a no-op when linking libreadline.

I've updated the readline spkg to link with -ltinfo, this is the proper thing to do. While I was at it I removed various old cruft.

Changed 8 years ago by vbraun

for review only

comment:89 Changed 8 years ago by jdemeyer

Looks good on first sight, still have to test it though...

comment:90 Changed 8 years ago by jdemeyer

  • Status changed from needs_review to needs_work

ncurses always fails its testsuite (spkg-check) because of

Running the test suite for ncurses-5.9...
make[3]: *** No rule to make target `check'.

comment:91 Changed 8 years ago by jdemeyer

ncurses fails to build on mark (Solaris SPARC) because of

/home/buildbot/build/sage/mark-1/mark_full/build/sage-5.12.beta1/local/bin/g++ -I../c++ -I../include -I. -DHAVE_CONFIG_H   -D__EXTENSIONS__ -D_XOPEN_SOURCE=500 -D_FILE_OFFSET_BITS=64  -DNDEBUG -I. -I../include -I/home/buildbot/build/sage/mark-1/mark_full/build/sage-5.12.beta1/local/include  -fPIC -c ../c++/cursesm.cc -o ../obj_s/cursesm.o
<command-line>:0:0: warning: "_XOPEN_SOURCE" redefined [enabled by default]
../c++/cursesm.cc:1:0: note: this is the location of the previous definition
<command-line>:0:0: warning: "_XOPEN_SOURCE" redefined [enabled by default]
../c++/cursesf.cc:1:0: note: this is the location of the previous definition
In file included from /home/buildbot/build/sage/mark-1/mark_full/build/sage-5.12.beta1/local/lib/gcc/sparc-sun-solaris2.10/4.7.3/include-fixed/iso/stdlib_iso.h:39:0,
                 from /usr/include/stdlib.h:18,
                 from ../c++/internal.h:53,
                 from ../c++/cursesm.cc:34:
/home/buildbot/build/sage/mark-1/mark_full/build/sage-5.12.beta1/local/lib/gcc/sparc-sun-solaris2.10/4.7.3/include-fixed/sys/feature_tests.h:341:2: error: #error "Compiler or options invalid for pre-UNIX 03 X/Open applications 	and pre-2001 POSIX applications"

comment:92 Changed 8 years ago by vbraun

  • Status changed from needs_work to needs_review

I've deleted spkg-check and added the patch from http://stackoverflow.com/questions/6774738/compiling-ncurses-on-solaris-compiler-or-options-invalid-for-pre-unix-03-x-op

Updated ncurses spkg at the same place.

comment:93 Changed 8 years ago by jdemeyer

  • Reviewers changed from Volker Braun, Jean-Pierre Flori to Volker Braun, Jean-Pierre Flori, Jeroen Demeyer
  • Status changed from needs_review to positive_review

Haven't tested all systems yet, but looks good so far.

comment:94 Changed 8 years ago by jdemeyer

  • Merged in set to sage-5.12.beta2
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:95 Changed 7 years ago by jpflori

Just a random rant on this closed ticket: not building the readline static library makes python fails to build on Cygwin. Ok this is without doubt because of python build system as it does not manage to find the specially named shared libraries on cygwin (note that when shared and static are installed, gcc will be smart enough to pick the shared one first, so this really is a python issue which fails to detect the shared one...).

comment:96 Changed 7 years ago by jpflori

Note: See TracTickets for help on using tickets.