Opened 3 years ago

Last modified 3 years ago

#21537 needs_work task

If $SAGE_SUDO is set, use it whenever we do "make install" of a package.

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-7.5
Component: build Keywords:
Cc: jdemeyer, fbissey, embray, leif, vbraun, mmezzarobba Merged in:
Authors: Matthias Koeppe Reviewers:
Report Upstream: N/A Work issues: redo patch using the method of 21726
Branch: u/mkoeppe/if__sage_sudo_is_set__use_it_whenever_we_do__make_install__of_a_package_ (Commits) Commit: abd3e96ace7de829396b691e209bb7dd813dd77d
Dependencies: #21726 Stopgaps:

Description (last modified by mkoeppe)

That would be a solution for those who want to install Sage in a prefix that's only writable by root.

In this way, building can take place in unprivileged user account.

It could perhaps also be enabled by ./configure --enable-sudo-install. (not on this ticket)

This ticket makes routine changes to all spkg-install scripts. Whenever installing has more steps than just make install, it moves the install part to a separate script spkg-do-install.

* See #21726 for a slightly different approach *

Change History (23)

comment:1 Changed 3 years ago by mkoeppe

  • Description modified (diff)

comment:2 Changed 3 years ago by mkoeppe

  • Branch set to u/mkoeppe/if__sage_sudo_is_set__use_it_whenever_we_do__make_install__of_a_package_

comment:3 Changed 3 years ago by mkoeppe

  • Cc jdemeyer fbissey added
  • Commit set to 444fd6e910c46e8212e7d10eb160e908e14975c2

New commits:

444fd6eAdd SAGE_SUDO for packages A-L

comment:4 follow-up: Changed 3 years ago by fbissey

Practical matters:

  • documentation
  • If you need a password and that the time between sudo-ing is longer than the time out, you will have to insert the password quite a number of times. In the case of a parallel install, will it block output to screen until you have entered the password?

I don't know enough of the mechanics of sage standard output on screen to know how requesting a password will affect the flow.

comment:5 Changed 3 years ago by fbissey

I guess I should say in the case of parallel make. In which the parallelism of make is translated into installing packages concurrently.

comment:6 Changed 3 years ago by mkoeppe

Yes, these issues need addressing in documentation and/or implementation.

comment:7 Changed 3 years ago by git

  • Commit changed from 444fd6e910c46e8212e7d10eb160e908e14975c2 to 666fb57e41dfcb66c08b1ad8231877b515f17a8e

Branch pushed to git repo; I updated commit sha1. New commits:

666fb57Add SAGE_SUDO for packages L-P

comment:8 in reply to: ↑ 4 Changed 3 years ago by mkoeppe

Replying to fbissey:

In the case of a parallel install, will it block output to screen until you have entered the password?

I don't know enough of the mechanics of sage standard output on screen to know how requesting a password will affect the flow.

The problem of parallel output and prompts has been discussed before in #20884, #21082. Also #21539 could help.

comment:9 Changed 3 years ago by git

  • Commit changed from 666fb57e41dfcb66c08b1ad8231877b515f17a8e to ec6ce246c07221e75fad1c1a6d47983156753540

Branch pushed to git repo; I updated commit sha1. New commits:

ec6ce24spkg-do-install: Make executable

comment:10 Changed 3 years ago by git

  • Commit changed from ec6ce246c07221e75fad1c1a6d47983156753540 to 23e3ddcebbdfbae9d22e05688d2b98acc6c61726

Branch pushed to git repo; I updated commit sha1. New commits:

23e3ddcPackages P-S

comment:11 Changed 3 years ago by git

  • Commit changed from 23e3ddcebbdfbae9d22e05688d2b98acc6c61726 to 74fbeff37dd0fe35ff5d85ade206b4fe9644cc79

Branch pushed to git repo; I updated commit sha1. New commits:

341bdb0Packages M-Z
74fbeffCall pip with SAGE_SUDO

comment:12 Changed 3 years ago by mkoeppe

  • Authors set to Matthias Koeppe

comment:13 Changed 3 years ago by git

  • Commit changed from 74fbeff37dd0fe35ff5d85ade206b4fe9644cc79 to abd3e96ace7de829396b691e209bb7dd813dd77d

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

d8919ccAdd SAGE_SUDO for packages A-L
7f46f6cAdd SAGE_SUDO for packages L-P
ccb3a13spkg-do-install: Make executable
3f957ccPackages P-S
c806e9fPackages M-Z
abd3e96Call pip with SAGE_SUDO

comment:14 Changed 3 years ago by mkoeppe

  • Cc embray leif vbraun mmezzarobba added
  • Description modified (diff)
  • Status changed from new to needs_review

Rebased on 7.4.beta6. Needs review.

comment:15 follow-ups: Changed 3 years ago by embray

Somehow this makes me feel a bit icky, but I'm not -1 on it either. It's a solution for now. I feel icky because if the Sage distribution were like a "normal" autoconf package I should be able to simply run make && sudo make install and not have tell configure that I intend to use sudo when I install. Nor should sudo be invoked in each sub-make. But the Sage distribution is definitely not a normal package since really it's just a big wrapper around a bunch of other packages, combined with some semblance of dependency management.

That said, it would be nice if the Sage distribution did work like make && sudo make install. To me, this points toward another area where sage packages could and probably should be refactored anyways. If packages had separate spkg-build and spkg-install scripts that would go a long way, I think. Then make/make build would run the spkg-build scripts, and only make install would run the actual spkg-install scripts. It would probably also go a long ways towards further eliminating duplication of effort between the many spkg-install scripts.

On another note, for Python packages installed with pip it still might be best to have separate "build" and "install" phases--in particular for Python packages with extension modules, where one might not want to be running a compiler as root. This could be achieved by running pip install without sudo with a temporary target directory, and then copying into the appropriate site-packages (perhaps after an uninstall of previously installed versions of the package). For that to work it might be best if this waited on #21441 which adds a new sage-pip-install script, which might be further modified to support this usage.

comment:16 Changed 3 years ago by mkoeppe

Fine with me to wait for #21441 and its sage-pip-install. Is that ticket moving forward, by the way?

comment:17 in reply to: ↑ 15 Changed 3 years ago by mkoeppe

Replying to embray:

[...] if the Sage distribution were like a "normal" autoconf package I should be able to simply run make && sudo make install and not have tell configure that I intend to use sudo when I install.

Just for reference for other readers (I know that you are aware of that ticket, Erik): Any discussion of separating make and make install for sage-the-distribution should take place on ticket #21495.

comment:18 in reply to: ↑ 15 ; follow-up: Changed 3 years ago by mkoeppe

  • Dependencies set to #21441

Replying to embray:

To me, this points toward another area where sage packages could and probably should be refactored anyways. If packages had separate spkg-build and spkg-install scripts that would go a long way, I think. Then make/make build would run the spkg-build scripts, and only make install would run the actual spkg-install scripts. It would probably also go a long ways towards further eliminating duplication of effort between the many spkg-install scripts.

Right now, in this patch I am calling spkg-do-install scripts with $SAGE_SUDO from various package's spkg-install scripts. Of course, each package's spkg-install script needs to be changed for $SAGE_SUDO to work, and installing packages that have not been prepared for it will cause an error when trying to install them to a prefix that is only root-writable.

An "easier" solution would be to just call spkg-build if it exists, unprivileged, and then call spkg-install using $SAGE_SUDO. This would "just work" without making changes to the -- but it would run a lot of stuff as root until a package moves its build part from spkg-install to spkg-build.

Thoughts?

On another note, for Python packages installed with pip it still might be best to have separate "build" and "install" phases--in particular for Python packages with extension modules, where one might not want to be running a compiler as root. This could be achieved by running pip install without sudo with a temporary target directory, and then copying into the appropriate site-packages (perhaps after an uninstall of previously installed versions of the package).

Are you sure that Python packages with extension modules are relocatable in this way? I have no experience with the specifics of the Python distutils world, but I would imagine all kinds of trouble (rpath/sonames, possible hardcoded paths, ...)

For that to work it might be best if this waited on #21441 which adds a new sage-pip-install script, which might be further modified to support this usage.

Sure, I've made it a dependency.

comment:19 Changed 3 years ago by embray

Right, regardless of how make is invoked, if there are separate spkg-build and spkg-install scripts the former can be run unprivileged. Then, at least in the current system, you would call $SAGE_SUDO spkg-install.

I think most packages would be easy enough to split up their spkg-install into two scripts. Though as I think I wrote in another ticket one might actually want a third spkg-env script that sets environment variables needed for both the build and install steps (fortunately for most autotools packages most relevant environment variables are only needed for the build step, I think).

comment:20 in reply to: ↑ 18 Changed 3 years ago by embray

Replying to mkoeppe:

Replying to embray:

On another note, for Python packages installed with pip it still might be best to have separate "build" and "install" phases--in particular for Python packages with extension modules, where one might not want to be running a compiler as root. This could be achieved by running pip install without sudo with a temporary target directory, and then copying into the appropriate site-packages (perhaps after an uninstall of previously installed versions of the package).

Are you sure that Python packages with extension modules are relocatable in this way? I have no experience with the specifics of the Python distutils world, but I would imagine all kinds of trouble (rpath/sonames, possible hardcoded paths, ...)

In general, yes, it should not be a problem. This does not affect rpath or soname for extension modules--I mean in principle it could but that would be a pathological case the likes of which I've never seen, and that is definitely not supported.

In principle one could also imagine a module that when installed generates some files containing hard-coded paths, but can't think of anything like that. Most well made Python modules are good about keeping such things relative and dynamic. So I'd be surprised if that's a problem anywhere, though if it is obviously exceptions can be made.

(Update: In fact, I should add, this is how distutils builds packages normally anyways--first it builds everything to build/, and then simply copies from there directly to the install location--what I described simply adds one additional copy. All the "install" commands really do in distutils is copy files--all the real work should usually be done in the "build" command (or sub-commands thereof).)

Last edited 3 years ago by embray (previous) (diff)

comment:21 Changed 3 years ago by mkoeppe

  • Status changed from needs_review to needs_work

comment:22 Changed 3 years ago by mkoeppe

  • Description modified (diff)

comment:23 Changed 3 years ago by mkoeppe

  • Dependencies changed from #21441 to #21726
  • Milestone changed from sage-7.4 to sage-7.5
  • Type changed from enhancement to task
  • Work issues set to redo patch using the method of 21726
Note: See TracTickets for help on using tickets.