Opened 6 years ago

Closed 17 months ago

#14804 closed enhancement (wontfix)

transition to using install program

Reported by: felixs Owned by: tbd
Priority: critical Milestone: sage-duplicate/invalid/wontfix
Component: distribution Keywords: spkg-install filelist destdir
Cc: leif, jondo Merged in:
Authors: Felix Salfelder Reviewers:
Report Upstream: N/A Work issues:
Branch: u/felixs/sagedist (Commits) Commit: 258434416b495ccf4e81d1c97b244ceec9a26526
Dependencies: Stopgaps:

Description (last modified by felixs)

To keep track of installed files, it's probably most convenient to use a unified install program within all spkg-install scripts, like the one included within #14796.

For transitional purposes, the non-#14796 build-system should provide an install program within $PATH. I propose to place it as sage-dist-install into $SAGE_SRC/bin. It should just ignore -F <filelist> for now.

For python-distutils packages, a setup.py wrapper is included.

Attachments (2)

sage-dist-install (13.7 KB) - added by felixs 6 years ago.
sage-setup.py (442 bytes) - added by felixs 6 years ago.

Download all attachments as: .zip

Change History (27)

Changed 6 years ago by felixs

comment:1 Changed 6 years ago by leif

  • Cc leif added

comment:2 follow-up: Changed 6 years ago by vbraun

Python distutils at least encapsulates the file operations in distutils.file_utils, so it would be a finite task to hook into that as well.

The alternative is to serialize the install phase and diff the SAGE_ROOT tree before and after.

Yet another possibility is to use separate install trees for each spkg in "sage-the-distribution", as in Nix or Stow.

comment:3 in reply to: ↑ 2 ; follow-up: Changed 6 years ago by felixs

Replying to vbraun:

Python distutils at least encapsulates the file operations in distutils.file_utils, so it would be a finite task to hook into that as well.

For example. but i have no idea how to "hook into that" yet without patching distutils. is there any python magic you have in mind?

The alternative is to serialize the install phase and diff the SAGE_ROOT tree before and after.

Won't give me DESTDIR. serialization is one of the uglier hacks (no way). diff takes quadratic time (even less way).

Yet another possibility is to use separate install trees for each spkg in "sage-the-distribution", as in Nix or Stow.

Just complicates the issue. better is supporting no "install" at all (like sage-doc currently does).

comment:4 in reply to: ↑ 3 ; follow-up: Changed 6 years ago by leif

Replying to felixs:

Replying to vbraun:

Python distutils at least encapsulates the file operations in distutils.file_utils, so it would be a finite task to hook into that as well.

For example. but i have no idea how to "hook into that" yet without patching distutils. is there any python magic you have in mind?

You can patch it (Python / setuptools / whatever), but you don't necessarily have to, as most Python objects are mutable, including functions / methods, i.e., you can overwrite them... :-)

In [1]: import distutils.file_util

In [2]: def foo(s,t): print "Hello"
   ...: 

In [3]: distutils.file_util.copy_file=foo

In [4]: distutils.file_util.copy_file("bar","baz")
Hello

And you could e.g. do so in an spkg's setup.py, to use some alternate implementation (provided elsewhere).

comment:5 in reply to: ↑ 4 ; follow-up: Changed 6 years ago by felixs

Replying to leif:

You can patch it (Python / setuptools / whatever), but you don't necessarily have to,

I dont *want* to. I'd just patch upstream if it's completely hosed.

as most Python objects are mutable, including functions / methods, i.e., you can overwrite them... :-)

...

And you could e.g. do so in an spkg's setup.py, to use some alternate implementation (provided elsewhere).

I don't quite understand. Are you proposing to patch the upstream package? (this would be worse, as it involves more than just distutils patching).

I need something of the kind

setup.py install USEMYOWNFILEUTIL

or

IMPORTPATH=myhackedfileutil_path setup.py install

within spkg-install. is this even possible?

comment:6 in reply to: ↑ 5 Changed 6 years ago by leif

Replying to felixs:

Replying to leif:

as most Python objects are mutable, including functions / methods, i.e., you can overwrite them... :-) And you could e.g. do so in an spkg's setup.py, to use some alternate implementation (provided elsewhere).

I don't quite understand. Are you proposing to patch the upstream package? (this would be worse, as it involves more than just distutils patching).

I need something of the kind

setup.py install USEMYOWNFILEUTIL

or

IMPORTPATH=myhackedfileutil_path setup.py install

within spkg-install. is this even possible?

Sure, I was just writing a post scriptum:

And in the latter case, you wouldn't even have to patch / modify the spkg's / upstream's setup.py, but just spkg-install (or whatever that will become), as you can also do things like

python <<EOF
# Do some stuff, e.g. modifying distutils.file_util.*
# (preferably importing an alternate implementation located elsewhere)
import sys
sys.argv = ["setup.py", "install"]
execfile("setup.py")
EOF

Or just create a (shell) script that does similar, and call that instead of doing python setup.py install.

comment:7 follow-up: Changed 6 years ago by leif

P.P.S.:

Of course, as is that relies on upstream only using distutils (at least to copy / install files), just like your custom [sage-dist-]install mechanism relies on upstream respecting autotools' conventions (and exclusively using install[-sh]) for such purposes.

comment:8 Changed 6 years ago by leif

... but you may have to wrap python for some packages that use both (e.g.) autotools and Python's distutils (some support setting PYTHON).

Still, uninstalling an spkg by (just) removing the files it added (probably restoring those it deleted or modified, too) can easily break a Sage installation, but I guess you're aware of the pitfalls (that aren't specific to Sage)...

Changed 6 years ago by felixs

comment:9 in reply to: ↑ 7 ; follow-up: Changed 6 years ago by felixs

Thanks Leif, I wasn't aware of that overriding possibility.

Replying to leif:

Of course, as is that relies on upstream only using distutils (at least to copy / install files), just like your custom [sage-dist-]install mechanism relies on upstream respecting autotools' conventions (and exclusively using install[-sh]) for such purposes.

.. and lets see what else copy_file is abused for. But I think, this approach will finally cut it.

autotools uses $(INSTALL), which can be overriden with anything. also others, otherwise broken make based build systems use $(INSTALL) and $(INSTALLDIRS).

comment:10 in reply to: ↑ 9 ; follow-up: Changed 6 years ago by leif

Replying to felixs:

.. and lets see what else copy_file is abused for. But I think, this approach will finally cut it.

Note that your replacement (or wrapper) function will of course finally have to mimic the full behaviour of the original one, i.e., has to take the same optional arguments (with the same default values), and has to return a tuple.

comment:11 in reply to: ↑ 10 ; follow-up: Changed 6 years ago by felixs

Replying to leif:

Note that your replacement (or wrapper) function will of course finally have to mimic the full behaviour of the original one, i.e., has to take the same optional arguments (with the same default values), and has to return a tuple.

It is supposed to call the original function after it's done with file listing. I'd expect that

def B(*args, **kwds): return B(args, kwds)

works, except for the default values...

comment:12 in reply to: ↑ 11 ; follow-up: Changed 6 years ago by leif

Replying to felixs:

It is supposed to call the original function after it's done with file listing. I'd expect that

def B(*args, **kwds): return B(args, kwds)

works, except for the default values...

... and infinite recursion. ;-)

comment:13 in reply to: ↑ 12 Changed 6 years ago by felixs

Replying to leif:

... and infinite recursion. ;-)

that was typed to fast, also some asterisks are missing. as it seems it even does the right thing wrt default values...

comment:14 Changed 6 years ago by felixs

And indeed, setup.py install takes --record <filename>. I should have tried to patch distutils right away.... :)

comment:15 Changed 6 years ago by leif

It may make sense to wrap the whole distutils.core.setup() function [too]...

(depending on what else you intend to do)

Last edited 6 years ago by leif (previous) (diff)

comment:16 Changed 6 years ago by felixs

There is now sage-dist-install, sage-dist-make, spkg.mk and sage-setup.py within my WIP branch on github.

With these, most spkg-install scripts can be augmented to support both the #14796 build system (together with #14792) as well as the current one.

Please have a look whether this could break anything.

comment:17 Changed 6 years ago by felixs

  • Branch set to u/felixs/sagedist
  • Commit set to fdeefe58eb8cb929e88666df03b5d5f48909b13e

This is now in git. It works for all packages I have tried with.

comment:18 Changed 6 years ago by felixs

  • Description modified (diff)
  • Status changed from new to needs_review

comment:19 Changed 6 years ago by jondo

  • Cc jondo added

comment:20 Changed 6 years ago by git

  • Commit changed from fdeefe58eb8cb929e88666df03b5d5f48909b13e to 258434416b495ccf4e81d1c97b244ceec9a26526

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:2584344]provide SAGE_PREFIX

comment:21 Changed 6 years ago by jdemeyer

  • Status changed from needs_review to needs_work

Needs to be rebased.

comment:22 Changed 6 years ago by vbraun_spam

  • Milestone changed from sage-6.1 to sage-6.2

comment:23 Changed 5 years ago by vbraun_spam

  • Milestone changed from sage-6.2 to sage-6.3

comment:24 Changed 5 years ago by vbraun_spam

  • Milestone changed from sage-6.3 to sage-6.4

comment:25 Changed 17 months ago by embray

  • Milestone changed from sage-6.4 to sage-duplicate/invalid/wontfix
  • Resolution set to wontfix
  • Status changed from needs_work to closed

This was in a way trying to do what is now accomplished (in spirit--not so much in the details) by some combination of #23160 an #22509. I don't think this particular approach really fits in with what we're doing now. But thank you Felix for your early pioneering work on trying to fix Sage's build system...

Note: See TracTickets for help on using tickets.