The libpng12 spkg should also create symbolic links from `libpng.*` for its libraries

[leif]

Doing so avoids trouble if pkg-config isn't available, and packages look for just libpng.

The header png.h is already installed in $SAGE_LOCAL/include/, and a symbolic link from $SAGE_LOCAL/lib/pkgconfig/libpng.pc to the actual libpng12.pc is also created.

---

Actually the links (and a compatibility libpng shared library) are created by upstream, but spkg-install deleted these.

---

Use spkg at: ​http://boxen.math.washington.edu/home/jpflori/libpng-1.2.35.p5.spkg

---

Original changes:

libpng-1.2.35.p4 (Leif Leonhardy, August 17th 2011)

• #11696: Do *not* delete symbolic links from libpng.* (and the shared library named libpng.so.*) in $SAGE_LOCAL/lib; see the comment in spkg-install for why (cf. also #11686).
• Add -L$SAGE_LOCAL/lib to LDFLAGS, since otherwise Sage's zlib won't be used.
• Fix order of additions to flags, and don't drop user's settings. (Some might still intentionally get overridden though.)
• Use $MAKE, but also install serially since this shouldn't take much time and is certainly safer.
• Quote also $UNAME.

Further changes:

• Let libpng build a correct import on Cygwin using the SYMBOL_PREFIX trick, seehttp://old.nabble.com/Fwd%3A---libpng-Bugs-2981656---Import-library-definitions-missing-in-Windows-td28130513.html.
• Further cleanups.

[leif]

New spkg is up.

Needs review and testing, the latter especially on Darwin.

(Though I see no reason for doing so, we could still delete some of the links / the compatibility shared library on Darwin only.)

[leif]

P.S.:

This is just another instance of where it IMHO doesn't make sense to build a Sage version if the system already provides some sufficient one, since doing so only causes trouble.

The best way would be to make such rather basic, unmodified packages just prerequisites; perhaps also supplying some easy way to build/install them on older systems (i.e., a tarball with some additional install script using reasonable settings). Shipping "standard" spkgs like e.g. also iconv, Fortran or Cephes just blows up the Sage tarball and increases the build time due to useless dependencies.

If we keep building such broadly available packages regardless of whether they're already installed system-wide (like also readline, PPL, GMP/MPIR, MPFR etc.), we should make sure they only get used by Sage components, by "moving them out of LD_LIBRARY_PATH" and using (a relative!) RPATH / RUNPATH in the parts of Sage that use these libraries.

[jhpalmieri]

Notes (from #11686):

• test on OS X (recent)
• test on older versions of OS X
• test on skynet machine iras

For testing on iras, apply the spkg here, and then try building ​this matplotlib spkg. (The standard matplotlib spkg patches the file setupext.py to accommodate the lack of pkg-config on some systems, especially OS X. This modified spkg eliminates that patch.)

[jhpalmieri]

On both an OS X box (OS version 10.6.8) and on iras: before installing the libpng spkg here, both systems failed to build the modified matplotlib spkg referred to in the previous comment. After installing the new libpng, both successfully built it. All doctests passed for the "plot" directory, and this worked:

sage: P = plot(sin(x))
sage: P.save('x.png')

After exiting Sage:

$ file x.png 
x.png: PNG image data, 484 x 316, 8-bit/color RGBA, non-interlaced

From the code, it looks like P.save('x.png') should call matplotlib, and so if libpng is broken or if matplotlib can't find it, I think this should fail. So things look good so far. Any other suggestions for testing this?

The package still needs testing on other systems, and we should run full doctests as well, not just for the 'plot' directory.

[leif]

Replying to jhpalmieri:

Any other suggestions for testing this?

The package still needs testing on other systems, and we should run full doctests as well, not just for the 'plot' directory.

Perhaps also on some MacOS X 10.4 box, and also in general by building Sage from scratch with the new libpng spkg.

The only issues I can imagine to arise happen if you try to start some program dynamically linked to an incompatible (presumably very old) system libpng library from within the Sage environment.

I don't know if there are any use cases of the latter we would have to take care of.

Another problem we would notice when building Sage from scratch could be some "tight" components (shared libraries / extension modules), not using pkg-config in their build, actually linking to the shared libpng.so.3* library, such that we might end up with a Sage binary pulling in both libraries, which could then again lead to name clashes, e.g. on Darwin.

The latter should IMHO never happen (at least not with properly written packages) since $SAGE_ROOT/local/lib/libpng.so is a symbolic link to $SAGE_ROOT/local/lib/libpng12.so, not $SAGE_ROOT/local/lib/libpng.so.3.35.0.

[jhpalmieri]

A full build works successfully on sage.math and another unix box on skynet (taurus). I'm still waiting on the skynet machines fulvia (OpenSolaris), mark (Solaris), and iras ("ia64-Linux-suse").

Meanwhile, this fails on OS X 10.6.8. That is, while the spkg builds, it causes R to fail to install. I've put the full install log ​on-line, but here is the tail end of the log:

building package 'graphics'
mkdir ../../../library/graphics
make[4]: warning: -jN forced in submake: disabling jobserver mode.
mkdir ../../../library/graphics/R
mkdir ../../../library/graphics/po
make[4]: warning: -jN forced in submake: disabling jobserver mode.
Error in dyn.load(file, DLLpath = DLLpath, ...) : 
  unable to load shared library '/Applications/sage_builds/sage-4.7.1.rc2/spkg/build/r-2.10.1.p4/src/library/grDevices/libs/grDevices.so':
  dlopen(/Applications/sage_builds/sage-4.7.1.rc2/spkg/build/r-2.10.1.p4/src/library/grDevices/libs/grDevices.so, 6): Symbol not found: __cg_png_create_info_struct
  Referenced from: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ImageIO.framework/Versions/A/ImageIO
  Expected in: /Applications/sage_builds/sage-4.7.1.rc2/local/lib//libPng.dylib
 in /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ImageIO.framework/Versions/A/ImageIO
Calls: <Anonymous> ... namespaceImport -> loadNamespace -> library.dynam -> dyn.load
Execution halted
make[4]: *** [mklazy] Error 1
make[3]: *** [all] Error 2
make[2]: *** [R] Error 1
make[1]: *** [R] Error 1
make: *** [R] Error 1
Error building R.

On one OS X box, the following line from the R install log switched from "no" to "yes" after installing the new libpng spkg:

checking for png_create_write_struct in -lpng... yes

On another box, it said "yes" for the old spkg as well. For both machines, I get the same failure.

[leif]

Replying to jhpalmieri:

Meanwhile, this fails on OS X 10.6.8. That is, while the spkg builds, it causes R to fail to install. I've put the full install log ​on-line ...

Yes, another system library R links to uses libpng, and that apparently expects other (or additional) symbols to be present, which aren't in the official (Sage's) one.

If only MacOS used proper shared library versioning...

Can you also give the output of

$ ls -l $SAGE_ROOT/local/lib/lib[Pp]ng*

?

[jhpalmieri]

Replying to leif:

Can you also give the output of

$ ls -l $SAGE_ROOT/local/lib/lib[Pp]ng*

?

With the old libpng:

-rwxr-xr-x  1 palmieri  admin  253536 Aug 16 15:27 sage-4.7.1/local/lib/libpng12.0.dylib*
-rw-r--r--  1 palmieri  admin  605960 Aug 16 18:49 sage-4.7.1/local/lib/libpng12.a
lrwxr-xr-x  1 palmieri  admin      16 Aug 16 21:32 sage-4.7.1/local/lib/libpng12.dylib@ -> libpng12.0.dylib
-rwxr-xr-x  1 palmieri  admin     981 Aug 16 18:49 sage-4.7.1/local/lib/libpng12.la*

With the new one:

-rwxr-xr-x  1 palmieri  admin  256112 Aug 17 13:52 sage-4.7.1.rc2/local/lib/libpng.3.dylib*
lrwxr-xr-x  1 palmieri  admin      10 Aug 17 13:52 sage-4.7.1.rc2/local/lib/libpng.a@ -> libpng12.a
lrwxr-xr-x  1 palmieri  admin      14 Aug 17 13:52 sage-4.7.1.rc2/local/lib/libpng.dylib@ -> libpng12.dylib
lrwxr-xr-x  1 palmieri  admin      11 Aug 17 13:52 sage-4.7.1.rc2/local/lib/libpng.la@ -> libpng12.la
-rwxr-xr-x  1 palmieri  admin  253504 Aug 17 13:52 sage-4.7.1.rc2/local/lib/libpng12.0.dylib*
-rw-r--r--  1 palmieri  admin  605864 Aug 17 13:52 sage-4.7.1.rc2/local/lib/libpng12.a
lrwxr-xr-x  1 palmieri  admin      16 Aug 17 13:52 sage-4.7.1.rc2/local/lib/libpng12.dylib@ -> libpng12.0.dylib
-rwxr-xr-x  1 palmieri  admin    1032 Aug 17 13:52 sage-4.7.1.rc2/local/lib/libpng12.la*

[leif]

Ok, removing $SAGE_ROOT/local/lib/libpng.dylib should be sufficient to make R install, though then keeping $SAGE_ROOT/local/lib/libpng.3.dylib doesn't make much sense either. (Just wondering why the mtimes of the libraries of 4.7.1 differ that much.)

I'll update the spkg accordingly later.

Note to myself: Also support CFLAG64.

[jhpalmieri]

For the old libpng spkg, as part of a fresh Sage build:

  -rwxr-xr-x    1 palmieri  admin    253536 Aug 16 15:27 libpng12.0.dylib
  -rw-r--r--    1 palmieri  admin    605960 Aug 16 18:49 libpng12.a
  lrwxr-xr-x    1 palmieri  admin        16 Aug 16 15:27 libpng12.dylib -> libpng12.0.dylib
  -rwxr-xr-x    1 palmieri  admin       981 Aug 16 18:49 libpng12.la

The later times are the same for all of the .a and .la files in SAGE_LOCAL/lib, which is the time when docbuilding finished and doctesting started (as part of "make ptestlong"). For the earlier example I posted, all of the symlinks blah.dylib have the same modification date, and I don't know what it corresponds to. The last time I ran that copy of Sage? The last time I installed an spkg in that copy of Sage?

For the new spkg, since the Sage build crashed, it didn't reach any of these stages.

[leif]

Replying to leif:

Ok, removing $SAGE_ROOT/local/lib/libpng.dylib should be sufficient to make R install, though then keeping $SAGE_ROOT/local/lib/libpng.3.dylib doesn't make much sense either. (Just wondering why the mtimes of the libraries of 4.7.1 differ that much.)

I'll update the spkg accordingly later.

Ooops, completely forgot this ticket.

Deleting the symbolic link libpng.dylib -> libpng12.dylib can also cause trouble, namely if pkg-config is installed on Darwin, we still have a broken / messed-up libpng12.pc file (containing SAGE_ROOT=${SAGE_ROOT}), and there's no system-wide libpng.dylib installed.

Johan Bosman apparently ​ran into this.

So we should perhaps only remove the symbolic link (on Darwin) if there's really a system-wide libpng installed.

Nevertheless, we should still fix sage-location to not "initialize" .pc files more than once, until we handle these files differently such that sage-location won't have to deal with them at all.

[leif]

Replying to leif:

Nevertheless, we should still fix sage-location to not "initialize" .pc files more than once, until we handle these files differently such that sage-location won't have to deal with them at all.

And prepend Sage's pkg-config directory to PKG_CONFIG_PATH if it's already set. Currently it isn't modified at all if it is non-empty.

[leif]

Replying to leif:

Replying to leif:

Nevertheless, we should still fix sage-location to not "initialize" .pc files more than once, until we handle these files differently such that sage-location won't have to deal with them at all.

This is now #11760.

[leif]

Replying to leif:

Deleting the symbolic link libpng.dylib -> libpng12.dylib can also cause trouble, namely if pkg-config is installed on Darwin, we still have a broken / messed-up libpng12.pc file (containing SAGE_ROOT=${SAGE_ROOT}), and there's no system-wide libpng.dylib installed.

Johan Bosman apparently ​ran into this.

The problem on his MacOS X machine is actually caused by something else, namely the way matplotlib tries to detect whether pkg-config is installed, and a broken pkg-config, exiting with a non-zero status when called with --help.

A preliminary matplotlib-1.0.1.p1 spkg working around this (and adding a couple of tests and diagnostic messages) can be found ​here.

[jpflori]

The part about Cygwin in spkg-install is also very fishy and IMHO should be removed. I'm testing a minimalistically modified spkg which does not inlcude this part now.

[jpflori]

In fact it is somehow needed because the import library produced by libpng12 is quite empty and so useless... This was the reason for the hack at #9146.

Maybe it is a good time to update libpng? Or do some other spkgs specifically need version 1.2?

[jpflori]

I've cleaned up everything, removed commented out code, added the trick for Cygwin. Needs review.

[dimpase]

something is not right with the spkg:

$ ../../sage -f http://boxen.math.washington.edu/home/jpflori/libpng-1.2.35.p5.spkg
Attempting to download package libpng-1.2.35.p5
>>> Downloading libpng-1.2.35.p5.spkg.
  File "<stdin>", line 29
SyntaxError: Non-ASCII character '\xc2' in file <stdin> on line 29, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details
Error: failed to download package libpng-1.2.35.p5

[jpflori]

That's strange... Could you try downloading the spkg by hand?

[jdemeyer]

Alternatively, try downloading a different random package to check whether the problem lies with the package or the download script.

[dimpase]

Replying to jpflori:

That's strange... Could you try downloading the spkg by hand?

• wget says "Scheme missing"

• curl -O downloads it quite OK.

looks like a server-side issue to me.

[jpflori]

For info, wget worked from two different places for me (as well as downloading from Firefox). Anyway that's not really the point of this ticket.

[dimpase]

OK, modulo some irrelevant network issue, it works.

[jdemeyer]

On OS X 10.8, Mercurial doesn't work:

running install_egg_info
Removing /Users/dehayebuildbot/build/sage/dehaye/dehaye_full/build/sage-5.8.beta0/local/lib/python/mercurial-2.2.2-py2.7.egg-info
Writing /Users/dehayebuildbot/build/sage/dehaye/dehaye_full/build/sage-5.8.beta0/local/lib/python/mercurial-2.2.2-py2.7.egg-info
Traceback (most recent call last):
  File "/Users/dehayebuildbot/build/sage/dehaye/dehaye_full/build/sage-5.8.beta0/local/bin/hg", line 38, in <module>
    mercurial.dispatch.run()
  File "/Users/dehayebuildbot/build/sage/dehaye/dehaye_full/build/sage-5.8.beta0/local/lib/python/mercurial/dispatch.py", line 27, in run
    sys.exit((dispatch(request(sys.argv[1:])) or 0) & 255)
  File "/Users/dehayebuildbot/build/sage/dehaye/dehaye_full/build/sage-5.8.beta0/local/lib/python/mercurial/dispatch.py", line 40, in dispatch
    req.ui = uimod.ui()
  File "/Users/dehayebuildbot/build/sage/dehaye/dehaye_full/build/sage-5.8.beta0/local/lib/python/mercurial/ui.py", line 43, in __init__
    for f in scmutil.rcpath():
  File "/Users/dehayebuildbot/build/sage/dehaye/dehaye_full/build/sage-5.8.beta0/local/lib/python/mercurial/scmutil.py", line 430, in rcpath
    _rcpath = osrcpath()
  File "/Users/dehayebuildbot/build/sage/dehaye/dehaye_full/build/sage-5.8.beta0/local/lib/python/mercurial/scmutil.py", line 402, in osrcpath
    path = systemrcpath()
  File "/Users/dehayebuildbot/build/sage/dehaye/dehaye_full/build/sage-5.8.beta0/local/lib/python/mercurial/scmutil.py", line 455, in systemrcpath
    path.extend(rcfiles(os.path.join(p, root)))
  File "/Users/dehayebuildbot/build/sage/dehaye/dehaye_full/build/sage-5.8.beta0/local/lib/python/mercurial/scmutil.py", line 440, in rcfiles
    for f, kind in osutil.listdir(rcdir)
  File "/Users/dehayebuildbot/build/sage/dehaye/dehaye_full/build/sage-5.8.beta0/local/lib/python/mercurial/demandimport.py", line 86, in __getattribute__
    self._load()
  File "/Users/dehayebuildbot/build/sage/dehaye/dehaye_full/build/sage-5.8.beta0/local/lib/python/mercurial/demandimport.py", line 58, in _load
    mod = _origimport(head, globals, locals)
ImportError: dlopen(/Users/dehayebuildbot/build/sage/dehaye/dehaye_full/build/sage-5.8.beta0/local/lib/python/mercurial/osutil.so, 2): Symbol not found: __cg_png_create_info_struct
  Referenced from: /System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO
  Expected in: /Users/dehayebuildbot/build/sage/dehaye/dehaye_full/build/sage-5.8.beta0/local/lib/libPng.dylib
 in /System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO
Mercurial installed correctly, but doesn't seem to work properly.
Running 'hg log' in the directory /Users/dehayebuildbot/build/sage/dehaye/dehaye_full/build/sage-5.8.beta0/spkg/build/mercurial-2.2.2.p0 failed.

real	0m9.692s
user	0m2.443s
sys	0m0.917s
************************************************************************
Error installing package mercurial-2.2.2.p0
************************************************************************

[jdemeyer]

In fact, this fails on all OS X systems.

[dimpase]

Replying to jdemeyer:

In fact, this fails on all OS X systems.

The problem is a conflict with the system library, which (on OSX 10.6.8, at least) is /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.dylib

This is probably caused by prepending -L$(SAGE_LOCAL)/lib to LDFLAGS in the new spkg.

[jpflori]

Cannot really work on this one as I don't have access to Mac OS X.

Maybe mercurial links to something outside of Sage which itself links to libpng which at runtime is resolved as our libpng, but unfortunately the libpng we ship is less capable than the system-wide one and the symbol lookup fails.

Could you post the mercurial log?

[dimpase]

Replying to jpflori:

Cannot really work on this one as I don't have access to Mac OS X.

Maybe mercurial links to something outside of Sage which itself links to libpng which at runtime is resolved as our libpng, but unfortunately the libpng we ship is less capable than the system-wide one and the symbol lookup fails.

Could you post the mercurial log?

please see ​http://boxen.math.washington.edu/home/dima/tmp/merc-osx.log

this the install log of this spkg followed by the one of mercurial.

[jpflori]

I think the problem is that somehow loading the mercurial module osutils triggers loading some Mac OS framework mysterious stuff which itself wants libpng, but Apple is so nice they decide to prefix their symbols with cg in their system wide libpng. Unfortunately its ours which gets picked and we do not prefix the symbols with anything fishy. Whence the "fix" to remove libpng.dylib so that when the Mac OS graphic thing looks for libpng it does not find ours.

Hopefully the spkgs we build and which need libpng ling to libpng12 and finds ours rather than the Mac OS X, at least it seems to be the case on Linux.

[jpflori]

Just for fun, could you try building libpng after export SYMBOL_PREFIX=cg (same trick as used on Cygwin but with cg instead of the empty string). But that should probably break our spkgs needing a plain linpng...

For future reference, I just fell on ​http://lists.apple.com/archives/unix-porting/2008/Sep/msg00026.html

[jpflori]

And doesn't it work on Mac OSes with case sensitive filesystems? I think one of Apple's trick is to name their library libPng.dylib, but in the end their filesystem is case insensitive...

[dimpase]

Replying to jpflori:

And doesn't it work on Mac OSes with case sensitive filesystems? I think one of Apple's trick is to name their library libPng.dylib, but in the end their filesystem is case insensitive...

Replying to jpflori:

For future reference, I just fell on ​http://lists.apple.com/archives/unix-porting/2008/Sep/msg00026.html

reading this makes me think that we don't want to mess around with this. The root of the problem is that the *usual* filesystem used by Apple is case-insensitive, and this makes libPNG indistinguishable from libpng. ​http://lists.apple.com/archives/unix-porting/2008/Sep/msg00030.html

[jpflori]

I agree we don't want to deal with ssuch a mess, but the original goal of this ticket is to let Sage install on systems without a system wide libpng. Of course we also want it to install on systems with a system wide conflicting libpng...

By the way it seems mercurial triggers the ImageIO thingy because it uses EDITOR: ​http://trac.sagemath.org/sage_trac/ticket/4678#comment:3

[jpflori]

So maybe the best solution is to do as suggested in the above linked comment and in #4216 and use sage-native-execute to run hg on Mac OS... That would be the more robust solution, unless we play will DYLD_[FALLBACK_]LIBRARY_PATH which I have no idea how Sage uses.

[dimpase]

Replying to jpflori:

So maybe the best solution is to do as suggested in the above linked comment and in #4216 and use sage-native-execute to run hg on Mac OS... That would be the more robust solution, unless we play will DYLD_[FALLBACK_]LIBRARY_PATH which I have no idea how Sage uses.

WIth p4, there is no lib[Pp]ng.* in $SAGE_LOCAL/lib at all on my (curresystem. There are libpng12.* only. (except $SAGE_LOCAL/lib/pkgconfig/libpng.pc, which is a link to $SAGE_LOCAL/lib/pkgconfig/libpng12.pc).

This is why it works.

With p5 (i.e. spkg on the ticket), there are lots of libpng.*

sage-5.6.beta2/local/lib/libpng.3.dylib
sage-5.6.beta2/local/lib/libpng.a -> libpng12.a
sage-5.6.beta2/local/lib/libpng.dylib -> libpng12.dylib
sage-5.6.beta2/local/lib/libpng.la -> libpng12.la

This results in the name clash we see on OSX. I suppose you can tweak the spkg to remove all the libpng.*, they are apparently not needed. (This of course defeats part of the purpose of this ticket :-)).

[jpflori]

Replying to dimpase:

This results in the name clash we see on OSX. I suppose you can tweak the spkg to remove all the libpng.*, they are apparently not needed. (This of course defeats part of the purpose of this ticket :-)).

Exactly, and the solution to delete files is somehow "dirty"... In particular if a package looks for a png rather than png12 it will fail, which may be considered bad. Or good as the problem with R mentioned here shows... if R finds png then it somehow triggers the use of the evil ImageIO which wants a symbol prefixed png, but its our non prefixed version which is loaded.

Here are two different preliminary plans to get this merged. First one:

• delete libpng.* from $SAGE_LOCAL/lib on Darwin

and potentially (although quite unrelated):

• include the fix from John from ​http://sage.math.washington.edu/home/leif/Sage/spkgs/matplotlib-1.0.1.p1.spkg to fix the problem described here ​http://groups.google.com/group/sage-release/msg/ab91298b636b947a caused by broken pkg-config (needs to be rebased on top of #11915).

or for this second point:

• modify the changes introduced in #11686 to matplotlib so that it only directly links to png12 rather than png when pkg-config is not installed and not unconditionally, because I feel it was not really needed (the real problem was that some pc file got broken, that should be now fixed in #11760) but kept because it was cleaner that way, i.e. only patch when needed, although ​http://groups.google.com/group/sage-release/msg/ab91298b636b947a showed it might be beneficial in other situations.

Or second one:

• do not delete libpng.* from $SAGE_LOCAL/lib

and:

• use sage-native-execute to run hg
• find a similar workaround for R and other things which magically trigger the use of ImageIO on Darwin

and potentially:

• get rid of all fixes/hacks which make packages link directly to png12 rather than png as this won't be needed anymore.

The first solution looks infinitely more realistic and simpler, so I'll craft something based on that.

[jpflori]

Spkg diff, for review only.

[jpflori]

Updated spkg which deletes symlinks in $SAGE_LOCAL/lib/ on Darwin, please test there.

The matplotlib/pkg-config problem should be dealt with elsewhere by updating the matplotlib spkg if still present.

[dimpase]

looks ok on OSX. Positive review.