Opened 6 years ago

Closed 6 years ago

#15317 closed defect (fixed)

Troubles with Python and ncurses on Cygwin

Reported by: jpflori Owned by:
Priority: major Milestone: sage-6.2
Component: porting: Cygwin Keywords: cygwin spkg ncurses
Cc: dimpase, vbraun, jdemeyer Merged in:
Authors: Jean-Pierre Flori Reviewers: Travis Scrimshaw
Report Upstream: Workaround found; Bug reported upstream. Work issues:
Branch: c0a09c5 (Commits) Commit: c0a09c5122ffba794a121ad3bdf8fdf5cb96f2c8
Dependencies: Stopgaps:

Description (last modified by jpflori)

Python currently fails/is suboptimal on Cygwin because:

  • it does not properly detect readline which is only installed as a shared library, patch "2.7.3-dylib.patch" from Cygwin package fixes that;
  • the curses module is not built because of undefined refs, just passing -lcurses when linking is not enough, one should add -ltinfo; in fact the real problem is that setup.py tries to run ldd on the import library (dll.a) which fails to detect what libraries are already linked and anyway would decide to discard -ltinfo as its already linked to readline, but on Cygwin you have to explicitely pass everything...
  • it segfault at startup when loading the readline module, not sure why. It looks exactly as what is reported at http://trac.macports.org/ticket/29496 . Rebuilding ncurses with debug info (including CFLAGS="-O0 -g" which cannot easily be passed right now) does not give much info. The offending line pointed by GCC is "char *result = exit_attribute_mode;"; maybe some dark magic going on as reported in http://lists.gnu.org/archive/html/bug-ncurses/2006-10/msg00002.html ; please not that on Cygwin the stack is small by default but playing a little bit with that did not really help. Further info: ncurses 5.7 is fine, ncurses 5.8 fails in the same way. Upstream bu report: http://lists.gnu.org/archive/html/bug-ncurses/2013-10/msg00014.html. Should be fixed with http://lists.gnu.org/archive/html/bug-ncurses/2013-10/msg00017.html

This ticket fixes the Python part. The ncurses part is #15617

Attachments (2)

scrimshaw_cygwin.log (52.1 KB) - added by tscrim 6 years ago.
scrimshaw_cywgin64.log (588.6 KB) - added by tscrim 6 years ago.

Download all attachments as: .zip

Change History (50)

comment:1 Changed 6 years ago by jpflori

Had no time to investigate the diff between 5.7 and 5.8, especially around the offending lines, function calls. Any help/clues welcomed.

comment:2 Changed 6 years ago by jpflori

Similar problem on different archs: http://lists.busybox.net/pipermail/buildroot/2011-April/042350.html No solution though...

comment:3 Changed 6 years ago by jpflori

  • Cc jdemeyer added

comment:4 Changed 6 years ago by jpflori

Playing a little bit around, it seems that setting TERM to something non-existing (e.g. "" or "blblbblblbl") makes the segfault disappear.

comment:5 Changed 6 years ago by jpflori

  • Description modified (diff)
  • Report Upstream changed from N/A to Workaround found; Bug reported upstream.

Upstream bug report: http://lists.gnu.org/archive/html/bug-ncurses/2013-10/msg00014.html

The problem is that when the config BROKEN_LINKER is set, then a pointer CurTerm? is used but never set to something else than zero. Setting it as in other cases seems to fix the problem. Or unsetting BROKEN_LINKER on Cygwin.

comment:6 Changed 6 years ago by jpflori

  • Description modified (diff)

The BROKEN_LINKER stuff should be fixed in ncurses:

comment:7 Changed 6 years ago by jpflori

  • Report Upstream changed from Workaround found; Bug reported upstream. to Fixed upstream, but not in a stable release.

comment:8 Changed 6 years ago by jpflori

There were further problem with the threading code in the Cygwin dll itself. Fixed in the > 1.7.25 releases (currently only snpashots). See http://cygwin.com/ml/cygwin/2013-10/msg00371.html.

comment:9 Changed 6 years ago by jpflori

As far as not linking to tinfo is concerned, I'm putting this here for reference: http://bugs.python.org/issue7384.

In particular, python uses ldd for linking detection but applies it on import libraries on cygwin which just fails. (The situation is more or less the same on OS X but OS X seems less picky about providing explicitely all libraries to be linked in.)

comment:10 follow-up: Changed 6 years ago by jpflori

Here is how to find the dll corresponding to an import lib:

(dlltool --identify [implib])

comment:11 in reply to: ↑ 10 Changed 6 years ago by dimpase

Replying to jpflori:

Here is how to find the dll corresponding to an import lib:

(dlltool --identify [implib])

Fixed typo in the url.

comment:12 Changed 6 years ago by jpflori

I've opened #15617 for the ncurses update.

comment:13 Changed 6 years ago by jpflori

I've put a dirty but working python spkg at:

for those wanting to try a Cygwin(32) build.

comment:14 follow-up: Changed 6 years ago by tscrim

I installed the new version of ncurses and created a new branch with the python spkg at u/tscrim/cygwin_python-15317 (with #15617 merged in). Now I get somewhere else than I did before (with cygwin32) with it failing at importing the module _socket...

comment:15 in reply to: ↑ 14 Changed 6 years ago by jpflori

Replying to tscrim:

I installed the new version of ncurses and created a new branch with the python spkg at u/tscrim/cygwin_python-15317 (with #15617 merged in). Now I get somewhere else than I did before (with cygwin32) with it failing at importing the module _socket...

What is in that branch exactly? From the git log I assume you just updated the tarball with what's within the spkg? Note that's only for testing purpose and a clean patch should be produced :) FYI, I think we temporarly could just diff what's in the tarball with the vanilla tarball and ship it as a patch in Sage, but that's definitely not for upstream inclusion. Right know it basically adds "-ltinfo" all the time which is right for Sage as we build our own ncurses which makes tinfo a separate lib.

The _socket thing reminds me of rebasing issues. Try to run "find . -name "*.dll" > dlls && rebase -O -T dlls && make" in Sage's root. And I did not test on Cygwin64 lately, so that might be another problem.

comment:16 Changed 6 years ago by tscrim

Tried that and it didn't work. I also pushed a patch to the branch.

comment:17 Changed 6 years ago by jpflori

  • Authors set to Jean-Pierre Flori
  • Branch set to u/jpflori/ticket/15317
  • Commit set to 79166403ba2f72a9cda9a6641a8914293d3cd2d1
  • Description modified (diff)
  • Report Upstream changed from Fixed upstream, but not in a stable release. to Workaround found; Bug reported upstream.
  • Status changed from new to needs_review

The branch I pushed is cleaner and should be good to go.


New commits:

7916640Make sure tinfo is correctly linked in when needed on Cygwin.
9acea9bActually add the NTL patch.

comment:18 Changed 6 years ago by jpflori

Groumpf, wrong branch, I'm pushing the right one shortly.

Travis: Could you post the log of the failed build?

comment:19 Changed 6 years ago by git

  • Commit changed from 79166403ba2f72a9cda9a6641a8914293d3cd2d1 to c0a09c5122ffba794a121ad3bdf8fdf5cb96f2c8

Branch pushed to git repo; I updated commit sha1. New commits:

c0a09c5Make sure tinfo is correctly linked in when needed on Cygwin.

comment:20 Changed 6 years ago by tscrim

I think with the new branch it worked...I'm now building atlas. Let me try on my Cygwin64 and Ubuntu machines.

Version 0, edited 6 years ago by tscrim (next)

comment:21 Changed 6 years ago by jpflori

I don't promise it works on Cygwin64... though I don't see any reason why it shouldn't.

comment:22 Changed 6 years ago by jpflori

For ATLAS you will need #15615 as I forgot to update the autotools project within the tarball after #14410... and so the changes introduced in the patches/ATLAS-lib dir are not reflected in the build system actually used (src/ATLAS-lib).

Changed 6 years ago by tscrim

comment:23 follow-up: Changed 6 years ago by tscrim

ATLAS built fine for me, but it's failing on pillow. I've attached the log. I'm trying the Cygwin64 now.

Changed 6 years ago by tscrim

comment:24 follow-up: Changed 6 years ago by tscrim

FTR It doesn't work for me on Cygwin64. Here's the log.

comment:25 in reply to: ↑ 23 Changed 6 years ago by jpflori

Replying to tscrim:

ATLAS built fine for me, but it's failing on pillow. I've attached the log. I'm trying the Cygwin64 now.

The pillow failure has noting to do with the cahnges here. It is quite easy to fix and is an upstream bug:

  • pillow wants to use some png functions but does not explicitely link to libpng and that's a no go on cygwin (but ok on most other systems).
  • in the log see line 314 "building 'PIL._imagingft' extension" and the few next ones

comment:26 in reply to: ↑ 24 ; follow-up: Changed 6 years ago by jpflori

Replying to tscrim:

FTR It doesn't work for me on Cygwin64. Here's the log.

It seems the readline module is not built:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named readline

and indeed readline is not found:

checking how to link readline libs... none
checking for rl_callback_handler_install in -lreadline... no
checking for rl_pre_input_hook in -lreadline... no
checking for rl_completion_display_matches_hook in -lreadline... no
checking for rl_completion_matches in -lreadline... no

Could you post your readline log?

Please let's go one step at a time, at least now installing python on Cygwin32 works, I'll try to go with the pillow and installing python on Cygwin64 issues on follow up tickets quickly.

comment:27 in reply to: ↑ 26 Changed 6 years ago by jpflori

Replying to jpflori:

Replying to tscrim:

FTR It doesn't work for me on Cygwin64. Here's the log.

It seems the readline module is not built:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named readline

and indeed readline is not found:

checking how to link readline libs... none
checking for rl_callback_handler_install in -lreadline... no
checking for rl_pre_input_hook in -lreadline... no
checking for rl_completion_display_matches_hook in -lreadline... no
checking for rl_completion_matches in -lreadline... no

Could you post your readline log?Atickets quickly.

Also note this failure surel has nothing to do with the changes here as readline is not detected during the autotools based configuration phase, not the setup.py one which I modified.

comment:28 Changed 6 years ago by jpflori

Strangely on my setup, readline seems to have built fine, is detected, but the python fails because of an ICE in GCC.

comment:29 Changed 6 years ago by jpflori

(I mean my Cygwin64 setup, with Cygwin's GCC 4.8.2.)

comment:30 Changed 6 years ago by jpflori

After fixing my ICE on Cygwin64 (using #10572), I oculd build pytohn.

comment:31 Changed 6 years ago by vbraun_spam

  • Milestone changed from sage-6.1 to sage-6.2

comment:32 Changed 6 years ago by tscrim

  • Reviewers set to Travis Scrimshaw
  • Status changed from needs_review to positive_review

Sorry for the delays, let's get this in.

comment:34 Changed 6 years ago by jpflori

Any chance to get the getbuildinfo.c file? Doesn't look obvously related, so needs further info.

comment:35 Changed 6 years ago by vbraun

Sorry, the buildbot has already moved on. You'll have to build it yourself.

comment:36 Changed 6 years ago by jpflori

I just did so ... and the build succeeded.

I merged u/jpflori/ticket/15317 with develop, exported MAKE to something sensible and ran make. Does the buildbot export anything strange before building?

comment:37 Changed 6 years ago by jpflori

Hum, here is waht the pytho log says:

gcc -pthread -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall   -I. -IIn
clude -I./Include  -fPIC -DPy_BUILD_CORE -o Python/graminit.o Python/graminit.c
gcc -pthread -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall   -I. -IIn
clude -I./Include  -fPIC -DPy_BUILD_CORE \
              -DSVNVERSION="\"`LC_ALL=C svnversion .`\"" \
              -DHGVERSION="\"`LC_ALL=C hg id -i .`\"" \
              -DHGTAG="\"`LC_ALL=C hg id -t .`\"" \
              -DHGBRANCH="\"`LC_ALL=C hg id -b .`\"" \
              -o Modules/getbuildinfo.o ./Modules/getbuildinfo.c
************************************************************************
It seems that you are attempting to run Sage from an unpacked source
tarball, but you have not compiled it yet (or maybe the build has not
finished). You should run `make` in the Sage root directory first.
If you did not intend to build Sage from source, you should download
a binary tarball instead. Read README.txt for more information.
************************************************************************
************************************************************************
It seems that you are attempting to run Sage from an unpacked source
tarball, but you have not compiled it yet (or maybe the build has not
finished). You should run `make` in the Sage root directory first.
If you did not intend to build Sage from source, you should download
a binary tarball instead. Read README.txt for more information.
************************************************************************
************************************************************************
It seems that you are attempting to run Sage from an unpacked source
tarball, but you have not compiled it yet (or maybe the build has not
finished). You should run `make` in the Sage root directory first.
If you did not intend to build Sage from source, you should download
a binary tarball instead. Read README.txt for more information.
************************************************************************

comment:38 Changed 6 years ago by jpflori

Trying to reinstall python then fails:

gcc -pthread -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall   -I. -IInclude -I./Include  -fPIC -DPy_BUILD_CORE \
              -DSVNVERSION="\"`LC_ALL=C svnversion .`\"" \
              -DHGVERSION="\"`LC_ALL=C hg id -i .`\"" \
              -DHGTAG="\"`LC_ALL=C hg id -t .`\"" \
              -DHGBRANCH="\"`LC_ALL=C hg id -b .`\"" \
              -o Modules/getbuildinfo.o ./Modules/getbuildinfo.c
<command-line>:0:11: warning: missing terminating " character [enabled by default]
<command-line>:1:7: warning: missing terminating " character [enabled by default]
<command-line>:2:10: warning: missing terminating " character [enabled by default]
./Modules/getbuildinfo.c: In function 'Py_GetBuildInfo':
./Modules/getbuildinfo.c:45:5: error: missing terminating " character
./Modules/getbuildinfo.c:45:48: error: expected expression before ')' token
./Modules/getbuildinfo.c:46:27: error: missing terminating " character
./Modules/getbuildinfo.c:46:27: error: missing terminating " character
./Modules/getbuildinfo.c:47:28: error: missing terminating " character
./Modules/getbuildinfo.c:47:28: error: missing terminating " character
./Modules/getbuildinfo.c:53:19: error: 'buildinfo' undeclared (first use in this function)
./Modules/getbuildinfo.c:53:19: note: each undeclared identifier is reported only once for each function it appears in
./Modules/getbuildinfo.c: In function '_Py_hgversion':
./Modules/getbuildinfo.c:72:5: error: missing terminating " character
./Modules/getbuildinfo.c:72:5: warning: 'return' with no value, in function returning non-void [-Wreturn-type]
./Modules/getbuildinfo.c: In function '_Py_hgidentifier':
./Modules/getbuildinfo.c:79:5: error: missing terminating " character
./Modules/getbuildinfo.c:79:18: error: expected expression before ';' token
./Modules/getbuildinfo.c:83:9: error: missing terminating " character
./Modules/getbuildinfo.c:83:24: error: expected expression before ';' token
./Modules/getbuildinfo.c: In function 'Py_GetBuildInfo':
./Modules/getbuildinfo.c:57:1: warning: control reaches end of non-void function [-Wreturn-type]
make: *** [Modules/getbuildinfo.o] Error 1

comment:39 Changed 6 years ago by jpflori

/usr/local/bin/hg is broken on mod:

#!/bin/sh
sage -hg $*

whence all the above.

In particular, once sage(.git) is built, one get:

(sage-sh) jpflori@mod:src$ which hg
/usr/local/bin/hg
(sage-sh) jpflori@mod:src$ LC_ALL=C hg id -i .
sage-run received unknown option: -hg 
usage: sage [options]
Try 'sage -h' for more information.

comment:40 Changed 6 years ago by jdemeyer

This ticket is not broken, the hg command on mod is broken.

comment:41 follow-up: Changed 6 years ago by vbraun

How about

./configure HAS_HG=no SVNVERSION=no [...]

comment:42 in reply to: ↑ 41 Changed 6 years ago by jpflori

Replying to vbraun:

How about

./configure HAS_HG=no SVNVERSION=no [...]

Why not, but let's do this somewhere else. And maybe only if SAGE_FAT_BINARY is set or something like that. We cannot deal with every broken systems users may come up with.

comment:43 Changed 6 years ago by vbraun

Fair enough, but this ticket can't be merged until either mod is un-broken or a workaround is implemented. Or both.

Last edited 6 years ago by vbraun (previous) (diff)

comment:44 Changed 6 years ago by jpflori

  • Cc was added

Maybe William can fix mod (i.e. delete the harmful /usr/local/bin/hg)?

comment:45 Changed 6 years ago by jpflori

(And isn't the failure triggered even without this ticket? I mean: just trying to recompile python should fail.)

comment:46 Changed 6 years ago by vbraun

  • Cc was removed

Can you post to the sage.math users mailing list, this is for admin issues.

comment:48 Changed 6 years ago by vbraun

  • Branch changed from u/jpflori/ticket/15317 to c0a09c5122ffba794a121ad3bdf8fdf5cb96f2c8
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.