wiki:CygwinPort

Version 230 (modified by jpflori, 8 years ago) (diff)

--

Information for the port to Cygwin

  • Below are the list of prerequisites, bug fixes/upgrades needed for Cygwin to build, and doctest tickets for Cygwin
  • This is as of Sage 5.2. See bottom of this page for more information with Sage 5.4 and 5.5.
  • Closed tickets and some other information, like how to mark Cygwin-specific parts in C, is here as well
  • Although below it compares Windows 7 and Windows XP, it is conceivable that the Cygwin version also sometimes has something to do with the problems listed

Cygwin Notes

Prerequisites

These Cygwin packages need to be installed in order for the Sage build to complete. Eventually, these should be added to prereq, but that would be for when Cygwin is actually supported.

Also, many of these require things like zlib and other included Sage packages, so it would probably be possible to remove a lot of those from a Sage Cygwin tarball (with effort).

  • file, liblapack, liblapack0, liblapack-devel, patch
  • also libiconv and openssl, openssl-devel
  • Windows 7 additional requirements
    • libncurses-devel (to build Singular)
  • Reminders for compilers
    • Don't forget fortran
    • Make sure you download all gcc and g++ so the versions match
  • Useful other things to install, especially on a minimal install of Cygwin
    • gcc, make, m4, and perl are needed
    • not strictly needed but very useful are wget, ssh and scp
    • similarly and nano or vim or something to edit files with
  • Make sure you uninstall the applications interfering with Cygwin, see BLODA (Big List Of Dodgy Apps), including:
    • antivirus programs
    • antispyware programs
    • etc etc etc

Rebasing

At any point in the build process (or after), one can get forking problems, related to the fact that Windows can relocate a dynamic library that just called fork() to another place in memory, causing fork() failure (this is a "feature" of the current Cygwin fork() implementation on Win32). To decrease the probability of this happening, you may need to rebase all the Cygwin and Sage DLLs you have so far, i.e. allocate each its own space address to load in (otherwise they compete for the space).

  • See these build notes of William's for a fairly effective strategy to deal with this. (But ignore the parts (4) and (5) about editing the list of Sage dlls, this is no longer necessary since #12096.)
  • In order to use this, you will need to run the 'ash' shell (or 'dash' shell) from the Windows command line.
    • This will require closing all Cygwin applications, and doing C:\cygwin\bin\ash.exe or something similar instead of just ash.
    • For ash to actually find any commands, you may need to follow the tips at this blog post for setting the CYGWIN_HOME variable and prepending the Cygwin binary directory (usually C:\cygwin\bin) to the Path variable.
    • You may also have problems with permissions (getting a message about "is not writable"). This cyg-apt thread seems to have several solutions for this.

Porting!

Also, don't forget to export SAGE_PORT=yes before starting.

Build Tickets

  • #6743: A metaticket for exactly this, which also points here. Can be closed when Sage builds out of the box with the prerequisites above. Be aware that exactly what tickets/spkgs are needed depends on which alpha or release you use; read carefully to make sure the spkgs coincide.

Things without tickets

  • in some cases, untarring the Sage source needs to be done with options xopf, not just xf (observed on Win7)
  • Issue with Python 2.6.4 building on Win7 reported. Relevant comments:
    • Comment 0 - Original report, from March 2010, at http://groups.google.com/group/sage-windows/browse_thread/thread/88499b995a22ecc1 - but note the fix should be __CYGWIN__, not CYGWIN
    • Comment 1 - I had no problems with that thus far in Vista Premium 32-bit - Alec Mihailovs
    • Comment 2 - Alec - Win7 has a newer codebase, so this is not really a proof that we need not worry --- I think we really need to. Dmitrii
    • Comment 3 (from somebody) "obsolete, as Cygwin now has Python 2.6.5 as the standard" - true, but apparently still a problem for us
    • Comment 4 - Python spkg builds fine on XP - kcrisman
    • Comment 5 - Rebasing everything in $SAGE_ROOT seemed to fix it on Windows 7. See the section about rebasing above.
    • Comment 6 - Sometimes sage-gdb does not work. Here is the relevant message.
      Failed to find the necessary bits to build these modules:
      _bsddb             _tkinter           bsddb185
      gdbm               linuxaudiodev      nis
      ossaudiodev        spwd               sunaudiodev
      To find the necessary bits, look in setup.py in detect_modules() for the module's name.
      
      Failed to build these modules:
      _curses            _curses_panel
      

Getting Sage to start

  • #11635: Also copy "libntl.dll.a -> libntl.dll" (see ticket for why this was split off from the previous one) - #12104 is a dup, as it turns out

Doctest Tickets

  • #9165 -- lcalc does not work for elliptic curves on cygwin
  • #9167 -- importing sage.libs.ecl yields a "no such process" error
  • #9168 -- ratpoints does not work correctly
  • #9169 -- a cachefunc.py doctest hangs seemingly forever
  • #9170 -- get_memory_usage isn't implemented, e.g., because there's no top
  • #9171 -- some test failures in sagedoc.py
  • #9172 -- numerical noise in sage/rings/integer.pyx
  • #9173 -- BSD.py tests behave differently on cygwin, so need to be written to reflect that
  • #9174 -- robert miller's 2-descent is completely broken on cygwin
  • #9176 -- various heegner_index errors involving interval arithmetic on cygwin

Closed

  • #11260: ECL-11.1.1 fails to compile maxima on Windows 7 / Cygwin 1.7.9
  • #11502: Maxima fails to build on Cygwin 1.7.3 on XP (sometimes)
  • #11884: ECL doesn't build on OS X Lion - this upgrade to ECL changes and avoids forking on ECL
  • #9162 -- pynac.pyx use double precision special functions instead of long double (REVISITED)
  • #11119: ECL 11.1.1 fails on Cygwin
  • #11497: Twisted sometimes doesn't build
  • #11245: on Win7 tar does not unpack cddlib and networkx spkg's right.
  • #11504: Tachyon fails to build due to same failure that was fixed in #7335. See also #11706.
  • #11744: real interval absolute needs gmp as a dependency
  • #11727: Flint needs a very minor Cygwin fix due to a carriage return mistakenly placed in it
  • #11499: sage/rings/factorint needs gmp as a dependency
  • #11550: Singular does not build
  • #11547: The following files need to be copied "libcliquer.dll -> libcliquer.so", "libflint.dll -> libflint.so"
  • #8269: Maxima doesn't build on XP
  • #10240: pari-2.4.3.svn-12577.p9 incorrectly checks for the shared library on Cygwin (should be fixed by #11130)
  • #11494: libblas.dll.a is not copied to $SAGE_LOCAL/lib when ATLAS is "installed" (breaks linbox) (should be fixed by #10226)
  • #11232: patch spkg should not be installed, use Cygwin's
  • #11246: flint-1.5.0.p5 extraneous #includes break typedef ulong
  • #9845: lcalc doesn't build on cygwin due to missing time.h include
  • #9776: Cygwin: GLPK extension module needs to be more explicit in its linking (positive review)
  • #9146: cygwin: gd doesn't correctly link in libpng with sage-4.4.3.
  • #8257: gd needs a better version of expr
  • #7319: gdmodule fails
  • #7321: numpy fails to build on cygwin due to not using the correct fortran compiler
  • #8780: Add cephes spkg for Cygwin (provides complex.h amongst other things)
  • #8192: PolyBoRi needs png12 and z to build on Cygwin.
  • #8782: pari does not work under Cygwin
  • #8907: Python should be build with enable-shared
  • #9050: libntl.dll should be libntl.dll.a
  • #8542 / #8903: Pynac needs to use a function pointer table.
  • #7337: PolyBoRi fails to build on cygwin
  • libflint.dll needs to be libflint.so, or polynomial_integer_dense_flint.pyx needs to know to use the .dll (maybe not a real issue, let's wait an see on this one).
  • #8642: port detection for the notebook does not work in Cygwin
  • #8855: sage-env needs to be updated for Cygwin
  • #8853: fix M_PI_4 in complex_double on Cygwin
  • #8852: minor doctest fix in fast_eval.pyx for Cygwin
  • #8850: Cython should link against BLAS instead of ATLAS on Cygwin
  • #8849: the palp interface throws a (harmless?) OSError on Cygwin
  • #8848, #8847, #8846, #8845, #8844, #8843
  • #9067: boehm_gc (still) fails to build on Cygwin
  • #9166 -- sympow does not work on cygwin (This should be fixed by the patches made for Solaris x86 - see #9703)
  • #9154: BoehmGC again!
  • #10247: Sage 4.6 has PARI problems on Cygwin
  • #11167: Sage 4.7.alpha3 library doesn't build under Cygwin
  • #9163 -- output of a subtle test in expect.py differs slightly on cygwin

how to mark CYGWIN-specific parts in code

gcc on CYGWIN defines a variable __CYGWIN__. So you can use it to #if(n)def stuff.

old Cygwin binaries

(Old) Cygwin prebuilt sage binaries for cygwin are here: http://sage.math.washington.edu/home/wstein/binaries/cygwin/

Testing Sage 5.1.rc1 on Windows 7 64 bits with Cygwin 1.7.15

Let's put here reports of my WIP. When this is finished, I'll reformat everything in a more proper way.

  • The version of gcc is hardcoded in /bin/libtool, if you happen to use 4.5.3 rather than 4.3.4 you'll have to edit /bin/libtool
  • As noted above, you should surely rebase (your system once, and) the libraries build by Sage from time to time when fork errors occurs before restarting the compilation
  • I also got quite often cannot allocate memory errors, which could (only) be solved by restarting my computer (memleak somewhere in cygwin?)
  • #12115 : MPIR cannot be built both as a static and a shared library, lets try only shared for the time being, although that may not be the most sensible choice
  • ABI=32 must be set to build MPIR on 64 bits systems
  • Python 2.7.3 does not compile, fixes from issues 9665, 14437, 14438 in Python issue tracker are needed
  • ECL does not build with recent Cygwin version because fd_set does not get defined. See https://groups.google.com/forum/?fromgroups#!topic/sage-devel/7e-CCbRLpto
  • eclib fails with undefined references to gmp -> -lgmp should come at the end of the command line for gcc to correctly take it into account
  • eclib fails in make tests -> disable as a workaround in Makefile.in (somehow eclib/*.h are not found because AM_CPPFLAGS is not used and so the include path are not set...)
  • flint-1.5.2.p0 fails to apply the patch for longlong.h because of bad line breaks in the patch file, same problem in mpn_extras.patch
  • the libntl.a hack in flint spkg-install is not needed anymore since #9050 (which is not a really good solution, gcc should be told to generate a proper .dll.a file)
  • polybori from #13124 is needed
  • singular fails with ld not finding -lkernel and -lhtmlhelp, see http://wstein.org/home/wstein/www/home/keshav/irclogs/%23sagemath.2011-11-26.log, this can be fixed by modifying LDFLAGS in Singular/Makefile?.in to include -L../kernel even on Windows (and not -L/bin which seems useless) (this is #12089)
  • libpari.dll should be moved to libpari.dll.a to be linked with (rather than libpari.a, which anyway looks dysfunctional) so that the Sage library can be built
  • gmp should be added as a dependency to ecl.pyx in modules_list to avoid undefined references
  • matrix_modn_dense_float fails with ndefined references in givaro. Change order of dependencies in module_list.
  • in modules_list.pyx, farey_symbol.pyx should include gmpxx AND gmp; but that's not enough yet: undefined reference to convert_to_long and things defined in farey_symbol.pyx, to solve this second problem, one can add a extra_compiler_flags="-Wl,--enable-auto-import" to the farey stuff in module_list. A better way could be to let Cytohn add _declspec(dll[im|exp]port) things in farey_symbol.h
  • complex_double linking fails because of -lmc not found -> indeed, this was not built (don't know what it should be), solution: remove it from module_list
  • expression.pyx fails, seems related to http://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg163112.html. Somehow g++ does not realize that the infinity class used in the templates is the one from pynac. Maybe some name clash with Python/Cython/Sage? things in sage.rings.infinity[.infinity .infinity]. So letting g++ know that infinity is GiNaC::infinity should do the trick (I did that by modifying the cpp file produced by Cython which was hackish but quick), explicitly writing GiNaC::infinity in ginac.h in sage.libs solves the problem
  • undefined references in stl_vectors solved by adding the correct library dependencies in modules_list
  • undefined references in wrapper_rdf because of a functions defined in interp_rdf and seemingly correctly shown by nm in both files. Changing the order of the files in the linking command, and adding -no-undefined flag, did not solve the problem yet. It seems that there are incorrect __imp__ prefix added to the symbols of the interp_* functions when wrapper_*.o is built. Because of that, the functions are looked to be imported from some .dll rather .o file. Indeed, building an independent .dll for interp_* and then linking wrapper_* to it to produce wrapper_*.dll works. To only have one wrapper_*.dll I've added an header file for interp_*.c and included it as cdef extern from "interp_*.h" in wrapper_*.pyx rather than without the from part as before. This seems functional, but needs non-trivial modifications to gen_interpreter.py.
  • Apart from that, one of the modules built by gen_interpreter.py linked to the mc library which is not available on my system (also had that problem for some module build in module_list.py).
  • gap fails because an old ln trick which used to be needed now fails.
  • maxima was a pain to build, not sure if that's a local problem, but after the "make" stage, the "make install" stage always failed with fork errors (cannot commit memory...). So I just let the "make" stage finish, rebase my system, and go on with the process, which seems ok (I can ./sage -maxima and compute 1+1)
  • finally sage finished to build but would not start. Just copying libntl.dll.a to libntl.dll was enough (this is #11635)
  • some elliptic curve stuff worked ok, but the notebook did not start in the first session (something about an alarm set at 1 second, maybe it was just too slow) and then was ok!

Testing sage 5.2 on cygwin 1.7.16 on Windows 7 64 bits

Let's do the same thing as before with updated and freshly installed cygwin/sage, but opening tickets this time in addition to posting the progress here.

Recall that the procedure to rebase is to:

  • kill all cygwin processes (i.e. close all cygwin terminal windows, and shut down cygwin services, if any)
  • launch dash.exe (in .../cygwin/bin) as admin (sometimes you might need to use ash.exe instead, as C:\cygwin\bin\ash.exe)
  • cd $SAGE_ROOT (i.e. to the root of the currently building Sage install, or just cd / which should be a parent of the former)
  • /bin/find . -name *.dll > sage-dlls.lst (warning: on some systems to get this to actually find the right stuff you may need /bin/find . -name "*.dll" > sage-dlls.lst)
  • /bin/rebaseall -T sage-dlls.lst (you add -v flag to get verbose output and check that the currently building Sage dlls are rebased)

I still had problems with libtool (which I installed when I wanted to test an updated NTL spkg using the system libtool). I think it is because I installed libtool after installing the gcc packages. Therefore the postinstall scripts included in the gcc packages did not have the chance to modify the hardcoded path in libtool (unless there is also some such script in the libtool package itself). Anyway, reinstalling gcc packages did not solve the problem. Paths in sys_lib_search_path_spec where updated, but not in compiler_lib_search_dirs, predep_objects, postdep_objects, compiler_lib_search_path. The /usr/sbin/fix... script with is run be gcc4-core package (at least) does not update these paths. The other package do not seem to update any path by themselves. I'll file a Cygwin bug report.

Avoid to build Sage in a path involving uppercase letters or ECL might get upset! See #13343.

Here follow the problem encountered while building Sage:

  • The python-2.7.3.p0.spkg fails as before. This is now #13319. The python-2.7.3.p1.spkg from there installs fine.
  • The mercurial-2.2.2.p0.spkg fails because of a fork errors. Rebasing solves the problem.
  • The mpir-2.4.0.p6.spkg fails as before. This is #12115. The mpir-2.4.0.p7.spkg from there installs fine.
  • Second rebase needed to be able to build matplotlib.
  • The ecl-[... ...].p1.spkg failed as above. This is now #13324. The ecl-[... ...].p2.spkg from there installs fine.
  • Install PARI spkg from #13333, to make sure that the libpari.dll.a import file gets installed.
  • The eclib spkg fails as above. This is now #13325. The *.p0.spkg from there installs fine after installing PARI spkg from #13333.
  • The flint-1.5.2.p0.spkg fails as above. This is now #13330. The p1 spkg from there installs fine.
  • Singular fails as above. See #12089. Updated spkg from #13237 works fine.
  • Got segfaults during the tuning phase of zn_poly-0.9.p9.spkg. Rebooting solved the problem...
  • Now building the Sage library itself:
    • Failed for ecl.pyx as above. This is now #13334.
    • Failed for matrix_modn_dense_float.pyx and matrix_mod_dense_double.pyx as above. This is now #13335.
    • Failed for farey.pyx as above. This is now #13336.
    • Failed for expression.pyx as above. This is now #13337.
    • Failed for stl_vector.pyx as above. This is now #13338.
    • Failed for wrapper_*.pyx as above. This is now #13339.
  • The gap-4.4.12.p7.spkg fails. This is now #13341. The *.p8.spkg from there installs fine.
  • The sagenb-0.9.1.spkg fails while installing pyOpenSSL-0.12.tar.gz because of Singular's file Singular/LIB/crypto.lib which is installed in local/lib/crypto.lib. See #13344. Updated spkg from #13237 works fine.
  • No fork problems with Maxima this time! (but beware of #13343 !)
  • Build finished ok, but "Testing that Sage starts..." failed with fork errors. Rebasing solved the problem, although I got "Warning: Could not import sage.interfaces.maxima_lib No such process". This is #9167. Need NTL trick as well, this is #11635.
  • While building the reference manual, I got "from sage.libs.ecl import * ImportError?: No such process". This is #9167 as well.
  • Once again, got some alarm problems when starting the notebook: "KeyboardInterrupt?: computation timed out because alarm was set for 1 seconds", but trying several times let it start.
  • "./sage -gdb" fails with "Reading symbols from /home/jp/sage-5.2/local/bin/python...done. /home/jp/sage-5.2/local/bin/sage-gdb-commands:1: Error in sourced command file: Error creating process /home/jp/sage-5.2/local/bin/python, (error 87)."
  • Now running "make test".

It should be noted that some libraries were only build in static version although building a shared version should not be a problem, but just a matter of passing -no-undefined to libtool when it is invoked in link mode (adding --no-undefined to LDFLAGS might not work, nor adding -Wl,--no-undefined to C[XX]FLAGS flags, it is really libtool which must be convinced it is not an issue and it doesn't care about the previous env variables) so that libtool does not complain, this is now #13354 (see there for current status).

Some worrying cygcheck output:

(sage-sh) jp@THINKPAD:~/sage-5.2$ for d in `find . -name *.dll`; do t=`cygcheck $d 2>&1|grep "not find"`; if [ -n "$t" ]; then echo $d; echo $t; fi; done
./devel/sage-main/build/lib.cygwin-1.7.16-i686-2.7/sage/libs/lcalc/lcalc_Lfunction.dll
cygcheck: track_down: could not find libLfunction.so
./devel/sage-main/build/sage/libs/lcalc/lcalc_Lfunction.dll
cygcheck: track_down: could not find libLfunction.so
./local/lib/python2.7/site-packages/rpy2/rinterface/rinterface.dll
cygcheck: track_down: could not find libR.dll
./local/lib/R/library/class/libs/class.dll
cygcheck: track_down: could not find libR.dll
./local/lib/R/library/cluster/libs/cluster.dll
cygcheck: track_down: could not find libR.dll
./local/lib/R/library/foreign/libs/foreign.dll
cygcheck: track_down: could not find libR.dll
./local/lib/R/library/grDevices/libs/grDevices.dll
cygcheck: track_down: could not find libR.dll
./local/lib/R/library/grid/libs/grid.dll
cygcheck: track_down: could not find libR.dll
./local/lib/R/library/KernSmooth/libs/KernSmooth.dll
cygcheck: track_down: could not find libR.dll
./local/lib/R/library/lattice/libs/lattice.dll
cygcheck: track_down: could not find libR.dll
./local/lib/R/library/MASS/libs/MASS.dll
cygcheck: track_down: could not find libR.dll
./local/lib/R/library/Matrix/libs/Matrix.dll
cygcheck: track_down: could not find libR.dll
./local/lib/R/library/methods/libs/methods.dll
cygcheck: track_down: could not find libR.dll
./local/lib/R/library/mgcv/libs/mgcv.dll
cygcheck: track_down: could not find libR.dll
./local/lib/R/library/nlme/libs/nlme.dll
cygcheck: track_down: could not find libR.dll
./local/lib/R/library/nnet/libs/nnet.dll
cygcheck: track_down: could not find libR.dll
./local/lib/R/library/parallel/libs/parallel.dll
cygcheck: track_down: could not find libR.dll
./local/lib/R/library/rpart/libs/rpart.dll
cygcheck: track_down: could not find libR.dll
./local/lib/R/library/spatial/libs/spatial.dll
cygcheck: track_down: could not find libR.dll
./local/lib/R/library/splines/libs/splines.dll
cygcheck: track_down: could not find libR.dll
./local/lib/R/library/stats/libs/stats.dll
cygcheck: track_down: could not find libR.dll
./local/lib/R/library/survival/libs/survival.dll
cygcheck: track_down: could not find libR.dll
./local/lib/R/library/tools/libs/tools.dll
cygcheck: track_down: could not find libR.dll
./local/lib/R/modules/internet.dll
cygcheck: track_down: could not find libR.dll
./local/lib/R/modules/lapack.dll
cygcheck: track_down: could not find libR.dll
./local/lib/R/modules/vfonts.dll
cygcheck: track_down: could not find libR.dll

Follow-up tickets:

  • the lcalc_Lfunction is now #13351,
  • the rpy2.rinterface.rinterface is now #13350,
  • the R/modules/* errors are harmless.

Doctest failures:

Follow-up tickets:

  • sage.libs.mwrank.mwrank seems dysfunctional. Linking it to a shared version of eclib provided in #13325 solves the problem.

Testing Sage 5.4.rc on Windows XP

So far, following the above instructions, all goes more or less well. Python still reports missing some bits (_bsddb, _tkinter, bsddb185, gdbm, linuxaudiodev, his, osaudiodev, spwd, and sunaudiodev) and at the very beginning it was missing one of the sqrtl() functions, but SAGE_PORT=yes caused it to ignore this. A missing library to link against (gmp) in cvxopt means that one doesn't build (this is now #13799), but everything else does with all the above. And Sage starts, and import ecl works with the appropriate ticket above. A number of doctest errors, at least some of which seem to be due to forking problems, still exist.

Testing Sage 5.5.rc0 on Windows XP

I'm going to try this with the spkgs from:

If all goes well, I'll update #6743 appropriately.

Testing Sage 5.5.rc0 on Windows 7 (64 bits)

I'm going the same with the spkgs from:

Testing Sage 5.5.rc1 on Windows 7 (64 bits)

I'm testing to build Sage-5.5.rc1 on a minimal Cygwin install on 64 bits Windows 7 with:

About Cygwin packages:

  • I had first to install make (to be able to issue make...)
  • Then have to export SAGE_PORT to pass through the prereq spkg. But this disables checking for Sage minimal requirements...
  • So following http://www.sagemath.org/doc/installation/source.html, I installed gcc4-core, m4, perl (make was installed above and tar came by default, ranlib came with some of these).
  • The patch spkg complained I had to have patch installed, so I modified it to try to install itself (and for this I installed the Cygwin nano and emacs packages). See #13844.
  • Now the iconv spkg fails because gcc cannot find /usr/lib/libiconv.la although libiconv was just installed.