Opened 10 years ago

Closed 10 years ago

Last modified 10 years ago

#8440 closed defect (fixed)

Removal of pyprocessing causing problems as _multiprocessing not building on Solaris

Reported by: drkirkby Owned by: drkirkby
Priority: blocker Milestone: sage-4.3.4
Component: porting: Solaris Keywords:
Cc: mhansen, jhpalmieri, was, mvngu, jsp Merged in: sage-4.3.4.alpha1
Authors: David Kirkby Reviewers: Minh Van Nguyen
Report Upstream: Not yet reported upstream; Will do shortly. Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description (last modified by drkirkby)

Since #6503 removed pyprocessing from Sage, multiprocessing (or perhaps _multiprocessing) is needed, but this is not building on Solaris 10 SPARC.

gcc -fPIC -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DHAVE_SEM_OPEN=1 -DHAVE_FD_TRANSFER=1 -DHAVE_SEM_TIMEDWAIT=1 -IModules/_multiprocessing -I. -I/export
/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/./Include -I. -IInclude -I./Include -I/export/home/drkirkby/32/sage-4.3.4.alpha0/local/include -I/usr/local/include
-I/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Include -I/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src -c /export/home/drkirkb
y/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.c -o build/temp.solaris-2.10-sun4u-2.6/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/bu
ild/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.o
/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.c: In function 'multiprocessing_sendfd':
/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.c:100: warning: implicit declaration of function 'CMSG_SPACE'
/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.c:110: error: 'struct msghdr' has no member named 'msg_control'
/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.c:111: error: 'struct msghdr' has no member named 'msg_controllen'
/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.c:114: warning: implicit declaration of function 'CMSG_FIRSTHDR'
/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.c:114: warning: assignment makes pointer from integer without a cast
/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.c:117: warning: implicit declaration of function 'CMSG_LEN'
/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.c:118: error: 'struct msghdr' has no member named 'msg_controllen'
/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.c:119: warning: implicit declaration of function 'CMSG_DATA'
/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.c: In function 'multiprocessing_recvfd':
/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.c:145: error: 'struct msghdr' has no member named 'msg_control'
/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.c:146: error: 'struct msghdr' has no member named 'msg_controllen'
/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.c:149: warning: assignment makes pointer from integer without a cast
/export/home/drkirkby/32/sage-4.3.4.alpha0/spkg/build/python-2.6.4.p7/src/Modules/_multiprocessing/multiprocessing.c:153: error: 'struct msghdr' has no member named 'msg_controllen'

Further down the log, the fact the _multiprocessing module has failed to build is clearly stated:

Failed to find the necessary bits to build these modules:
_bsddb		   _hashlib		_ssl
_tkinter	   bsddb185		gdbm
linuxaudiodev	   ossaudiodev
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


Failed to build these modules:
_curses		   _curses_panel	_multiprocessing

Attachments (1)

8440-python.patch (89.3 KB) - added by drkirkby 10 years ago.
Mercurial patch

Download all attachments as: .zip

Change History (11)

comment:1 Changed 10 years ago by drkirkby

  • Cc jsp added

comment:2 Changed 10 years ago by drkirkby

  • Authors set to David Kirkby
  • Description modified (diff)
  • Priority changed from major to blocker
  • Report Upstream changed from Fixed upstream, but not in a stable release. to Not yet reported upstream; Will do shortly.
  • Status changed from new to needs_review

The solution to this problem is found in the old pyprocessing package, where it says:

#   HAVE_FD_TRANSFER
#     Set this to 1 to compile functions for transferring file
#     descriptors between processes over an AF_UNIX socket using a
#     control message with type SCM_RIGHTS.  On Unix the pickling of 
#     of socket and connection objects depends on this feature.
#
#     If you get errors about missing CMSG_* macros then you should
#     set this to 0.

This is not as well documented in Python's setup.py, but by setting

HAVE_FD_TRANSFER=0

in Python's top-level setup.py, the problem goes away. Python's build then reports:

Failed to build these modules:
_curses		_curses_panel

The _multiprocessing module is now building ok.

The new setup.py has a Solaris-specific section, which I added. However, so reviewers can be even more confident this will only affect Solaris, the patch is only applied on Solaris.

I also took time to address a minor issue at #8356, where '--without-libpng' was used, despite the fact the option is no longer recognised.

Notes for Release Manager

Prerequisites:

  • #7867 (This already has positive review)

This patch should be applied on top of the changes at #7867

Once this ticket is closed, #8356 may be closed too.

It would be appreciated if #8374, #8375, 8391 and #8404 could also be integrated into the next alpha, as that would have a high probability of allowing all doc tests to pass. All these tickets have positive review.

comment:3 Changed 10 years ago by drkirkby

I forgot to put the location of the package

http://boxen.math.washington.edu/home/kirkby/portability/python-2.6.4.p7/python-2.6.4.p7.spkg

Patch will be attached in a minute

Changed 10 years ago by drkirkby

Mercurial patch

comment:4 Changed 10 years ago by mvngu

  • Status changed from needs_review to needs_work

The updated package python-2.6.4.p7.spkg needs a minor clean up:

[mvngu@sage python-2.6.4.p7]$ hg status
? patches/locale.pyc

That is, we don't place Python bytecode under revision control. Nor do we put any binary or Python bytecode under the directory "patches/".

comment:5 Changed 10 years ago by drkirkby

Hi Minh, I need to go out very soon and get a train, so don't have time to fully investigate this, but there appears there may be something wrong on the math's computer setup, as what I am putting there, and what I can see in the browser are not the same. (This could be the fact the ZIL log is disabled - complex story, and one I need to discuss again with William).

There should be a .spkg in the directory http://boxen.math.washington.edu/home/kirkby/portability/python-2.6.4.p7/

which does (for me at least) not have any such file or problem. There should be a directory below that where I extracted the file. But when I look with the browser, I can not see any of it! So in summary.

  • I don't see the problem you do.
  • Changes I am making on the file system do not appear to be reflected in what I can see from my browser.

If you look at this location, you might find the package, but I'm totally confused. I think the file system might be messed up.

kirkby@t2:[/home/kirkby/portability/python-2.6.4.p7] $ ls 
python-2.6.4.p7       python-2.6.4.p7.spkg
kirkby@t2:[/home/kirkby/portability/python-2.6.4.p7] $ 

I will have to look later, as I need to go out now.

Dave

comment:6 Changed 10 years ago by drkirkby

  • Status changed from needs_work to needs_info

I managed to look at this in more detail today. The reason the directory was not visable was a permissions problem, and nothing to do with file system errors.

I still can't understand why you see this odd file, as I don't:

kirkby@sage:~/portability/python-2.6.4.p7/python-2.6.4.p7$ ls patches/locale.pyc
ls: cannot access patches/locale.pyc: No such file or directory
kirkby@sage:~/portability/python-2.6.4.p7/python-2.6.4.p7$ hg status

comment:7 follow-up: Changed 10 years ago by drkirkby

  • Status changed from needs_info to needs_review

Minh,

could you double- check you are using the same package as me, since I simply can't see this spurious file.

comment:8 in reply to: ↑ 7 Changed 10 years ago by mvngu

  • Reviewers set to Minh Van Nguyen
  • Status changed from needs_review to positive_review

Replying to drkirkby:

could you double- check you are using the same package as me, since I simply can't see this spurious file.

I have re-checked python-2.6.4.p7.spkg and indeed it's OK by me. I have no idea why I received the warning about a bytecode file under the directory patches/. What I would usually do is take an spkg that has been compressed or just tarball'd, and unpack it. Then I would go through that unpacked spkg with Mercurial to make sure that Mercurial is happy with the repository under consideration. I have built python-2.6.4.p7.spkg on the following:

  • bsd.math
  • Cygwin (winxp1 on boxen.math)
  • rosemary.math
  • sage.math
  • t2.math

On all systems/platforms that I tested, the said Python package builds as claimed. Where relevant (i.e. bsd.math, rosemary.math, sage.math), all doctests passed.

comment:9 Changed 10 years ago by mhansen

  • Merged in set to sage-4.3.4.alpha1
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:10 Changed 10 years ago by drkirkby

#8356 can be closed too following the inclusion of this updated python package.

dave

Note: See TracTickets for help on using tickets.