Opened 12 years ago

Closed 6 years ago

#9536 closed defect (fixed)

python setup.py picks prefix from ~/.pydistutils.cfg

Reported by: Alexander Dreyer Owned by: Alexander Dreyer
Priority: major Milestone: sage-7.6
Component: build Keywords:
Cc: Jeroen Demeyer, Erik Bray, Leif Leonhardy, François Bissey, Volker Braun Merged in:
Authors: Matthias Koeppe Reviewers: Erik Bray
Report Upstream: Reported upstream. Developers deny it's a bug. Work issues:
Branch: 375faba (Commits, GitHub, GitLab) Commit: 375faba4699c44e2be0ecceddfbca29de2770f85
Dependencies: Stopgaps:

Status badges

Description (last modified by Matthias Köppe)

Hi, several packages do use python setup.py. This picks up the customized settings from ~/.pydistutils.cfg. This is bad, because it overrides the prefix setting.

For sage-main, I'll attach a patch: setup.cfg in the corresponding directory overrides the usere settings. Maybe there's a global solution.

Regards,

Alexander Dreyer

......

Update for Sage 7.5. The problem reported on this ancient ticket is still present and affects

  • sagelib,
  • packages that are installed directly using setup.py
  • packages that are installed using pip.

It can be tested by creating the following ~/.pydistutils.cfg:

[install]
root=/does-not-exist

The patch on the ticket uses

  • pip --isolated and
  • setup.py --no-user-cfg

Note there appear to be some issues regarding these option with early versions of Python 3.x: https://github.com/pypa/pip/issues/3509

Attachments (3)

setup_py_issue.patch (174 bytes) - added by Alexander Dreyer 12 years ago.
Patch for sage-main
sage-env.patch (521 bytes) - added by Alexander Dreyer 12 years ago.
patch for sage/spkg/base/sage-env sage/local/bin/sage-env (needs python 2.6.4.p10)
python-2.6.4.p10.patch (231.6 KB) - added by Alexander Dreyer 12 years ago.
Adding handling of environment variable DISTUTILS_NO_USER_CFG to python's distutils

Download all attachments as: .zip

Change History (47)

Changed 12 years ago by Alexander Dreyer

Attachment: setup_py_issue.patch added

Patch for sage-main

comment:1 Changed 12 years ago by Alexander Dreyer

Owner: changed from Alex Ghitza to Alexander Dreyer

comment:2 Changed 12 years ago by Alexander Dreyer

The following packages from the distribution use setup.py and may be affected:

zodb3-3.7.0.p4
scipy_sandbox-20071020.p5
pexpect-2.0.p4
sqlalchemy-0.5.8
glpk-4.44
mpmath-0.15
pil-1.1.6.p2
mercurial-1.3.1.p2
pycrypto-2.0.1.p5
mpir-1.2.2.p1
weave-0.4.9.p0
sympy-0.6.4.p0
scons-1.2.0
numpy-1.3.0.p3
gdmodule-0.56.p7
sagetex-2.2.5
sage-4.5
matplotlib-0.99.3
sage_scripts-4.5
setuptools-0.6c9.p0
python-2.6.4.p9
python_gnutls-1.1.4.p7
sagenb-0.8.1
networkx-1.0.1
jinja-1.2.p0
cython-0.12.1
pygments-0.11.1.p0
scipy-0.7.p5
cvxopt-0.9.p8
jinja2-2.1.1.p0
sphinx-0.6.3.p4
docutils-0.5.p0
twisted-9.0.p2
ipython-0.9.1.p0
moin-1.9.1.p1

The following already have setup.cfg, but it may not have the prefix definition:

zodb3-3.7.0.p4
sqlalchemy-0.5.8
scons-1.2.0
matplotlib-0.99.3
setuptools-0.6c9.p0
sagenb-0.8.1
networkx-1.0.1
jinja-1.2.p0
pygments-0.11.1.p0
jinja2-2.1.1.p0
sphinx-0.6.3.p4
twisted-9.0.p2
moin-1.9.1.p1

comment:3 Changed 12 years ago by PolyBoRi team

Hi! I am not an export for that for that, but did you read http://docs.python.org/install/#location-and-names-of-config-files There seems to be some options: –no-user-cfg

Cheers, Michael

comment:4 Changed 12 years ago by Alexander Dreyer

Hi Michael, indeed, that's a better solution. But both have the problem, that 30+ packages would have to be fixed, and it might be forgotten in the future. (Maybe there some monkey patch for distutils, that all of it can be overwritten by the environment.)

Best regards,

Alexander

comment:5 Changed 12 years ago by PolyBoRi team

maybe, I did not try (just read the docs): http://docs.python.org/install/#inst-config-files

create system distutils.cfg

[global]
no-user-cfg=1

comment:6 Changed 12 years ago by Alexander Dreyer

Hi, unfortunately, this is a Python 2.7 feature and it does not support no-user-cfg in distutils.cfg :-(

Regards,

Alexander

Changed 12 years ago by Alexander Dreyer

Attachment: sage-env.patch added

patch for sage/spkg/base/sage-env sage/local/bin/sage-env (needs python 2.6.4.p10)

comment:7 Changed 12 years ago by Alexander Dreyer

Milestone: sage-4.5.2
Status: newneeds_review

Hi, I backported the handling of setup.py --no-user-cfg from Python 2.7 to Python 2.6.4 and also added the handling of the environment variable DISTUTILS_NO_USER_CFG to python's distutils.

The new spkg can be found here: http://sage.math.washington.edu/home/dreyer/suse101/python-2.6.4.p10.spkg

The last patch adds this variable to sage-env.

Regards,

Alexander

comment:8 Changed 12 years ago by Alexander Dreyer

Report Upstream: Not yet reported upstream; Will do shortly.Reported upstream. Little or no feedback.

comment:9 Changed 12 years ago by Alexander Dreyer

Minor update: http://sage.math.washington.edu/home/dreyer/suse101/python-2.6.4.p10.spkg (useful, if the sage-env patch is not used)

comment:10 in reply to:  9 Changed 12 years ago by David Kirkby

Replying to AlexanderDreyer:

Minor update: http://sage.math.washington.edu/home/dreyer/suse101/python-2.6.4.p10.spkg (useful, if the sage-env patch is not used)

You should also create patches/ACKS.txt.diff as a unified diff file. Although its fairly obvious you can recreate that now, that will not be the case if Python source code is updated.

Would a simpler solution not be for sage-env to set this variable and export it? I don't know much about python, so can't review this myself.

Dave

comment:11 Changed 12 years ago by Alexander Dreyer

Maybe a misunderstanding: the patch of python is necessary to fix that issue anyway. (This first variant didn't work, if the environment variable was not set.)

Changed 12 years ago by Alexander Dreyer

Attachment: python-2.6.4.p10.patch added

Adding handling of environment variable DISTUTILS_NO_USER_CFG to python's distutils

comment:12 Changed 12 years ago by Alexander Dreyer

I've added ACKS.txt.patch to the spkg.

comment:13 Changed 12 years ago by Alexander Dreyer

Component: algebrabuild

comment:14 Changed 12 years ago by Alexander Dreyer

The python community does not like the idea of adding an environment variable, see: http://bugs.python.org/issue9309

comment:15 Changed 12 years ago by Alexander Dreyer

Report Upstream: Reported upstream. Little or no feedback.Reported upstream. Developers deny it's a bug.

comment:16 Changed 10 years ago by Jeroen Demeyer

Please fill in your real name as Author.

comment:17 Changed 9 years ago by Jeroen Demeyer

Milestone: sage-5.11sage-5.12

comment:18 Changed 9 years ago by For batch modifications

Milestone: sage-6.1sage-6.2

comment:19 Changed 9 years ago by For batch modifications

Milestone: sage-6.2sage-6.3

comment:20 Changed 8 years ago by For batch modifications

Milestone: sage-6.3sage-6.4

comment:21 Changed 8 years ago by Karl-Dieter Crisman

Status: needs_reviewneeds_work

Needs a branch. And we have updated Python. And probably other reasons it isn't ready yet. (Or conceivably has been solved in the meantime?)

comment:22 Changed 6 years ago by Matthias Köppe

Cc: Jeroen Demeyer Erik Bray Leif Leonhardy François Bissey added
Description: modified (diff)
Milestone: sage-6.4sage-7.5

comment:23 Changed 6 years ago by Matthias Köppe

Branch: u/mkoeppe/python_setup_py_picks_prefix_from____pydistutils_cfg

comment:24 Changed 6 years ago by Matthias Köppe

Authors: Matthias Koeppe
Cc: Volker Braun added
Commit: 6dcaebb5f95abdc78cce729bcced7dc96ee5cb1f
Description: modified (diff)
Status: needs_workneeds_review

New commits:

6dcaebbUse 'pip --isolated' and 'setup.py --no-user-cfg'

comment:25 Changed 6 years ago by Erik Bray

Reviewers: Erik Bray
Status: needs_reviewpositive_review

This looks like the right approach, thanks.

comment:26 Changed 6 years ago by Volker Braun

Branch: u/mkoeppe/python_setup_py_picks_prefix_from____pydistutils_cfg6dcaebb5f95abdc78cce729bcced7dc96ee5cb1f
Resolution: fixed
Status: positive_reviewclosed

comment:27 Changed 6 years ago by Volker Braun

Commit: 6dcaebb5f95abdc78cce729bcced7dc96ee5cb1f
Resolution: fixed
Status: closednew

Prevents pyzmq from building...

comment:28 Changed 6 years ago by Volker Braun

  Source in /tmp/pip-mTg5Xg-build has version 16.0.0, which satisfies requirement pyzmq==16.0.0 from file:///mnt/disk/home/release/Sage/local/var/tmp/sage/build/pyzmq-16.0.0/src
Installing collected packages: pyzmq
  Running setup.py install for pyzmq: started
    Running command /mnt/disk/home/release/Sage/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-mTg5Xg-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" configure --zmq=/mnt/disk/home/release/Sage/local --no-user-cfg install --record /tmp/pip-o79sR3-record/install-record.txt --single-version-externally-managed --compile
    running configure
    error: error in command line: command 'Configure' has no such option 'no_user_cfg'
    Running setup.py install for pyzmq: finished with status 'error'
Cleaning up...
  Removing source in /tmp/pip-mTg5Xg-build
Command "/mnt/disk/home/release/Sage/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-mTg5Xg-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" configure --zmq=/mnt/disk/home/release/Sage/local --no-user-cfg install --record /tmp/pip-o79sR3-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-mTg5Xg-build/
Exception information:
Traceback (most recent call last):
  File "/mnt/disk/home/release/Sage/local/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/mnt/disk/home/release/Sage/local/lib/python2.7/site-packages/pip/commands/install.py", line 317, in run
    prefix=options.prefix_path,
  File "/mnt/disk/home/release/Sage/local/lib/python2.7/site-packages/pip/req/req_set.py", line 742, in install
    **kwargs
  File "/mnt/disk/home/release/Sage/local/lib/python2.7/site-packages/pip/req/req_install.py", line 880, in install
    spinner=spinner,
  File "/mnt/disk/home/release/Sage/local/lib/python2.7/site-packages/pip/utils/__init__.py", line 718, in call_subprocess
    % (command_desc, proc.returncode, cwd))
InstallationError: Command "/mnt/disk/home/release/Sage/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-mTg5Xg-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" configure --zmq=/mnt/disk/home/release/Sage/local --no-user-cfg install --record /tmp/pip-o79sR3-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-mTg5Xg-build/

comment:29 Changed 6 years ago by Matthias Köppe

I've created an upstream issue for this:

https://github.com/zeromq/pyzmq/issues/937

comment:30 Changed 6 years ago by Erik Bray

That's annoying. For now we should probably remove pip install from pyzmq.

comment:31 Changed 6 years ago by Matthias Köppe

Dependencies: #21835

Upstream pyzmq reports that it works with pip 9.0.1, setuptools 28.8.0.

Let's try with upgrading these packages. #21835 does the upgrade of pip to 9.0.0...

comment:32 Changed 6 years ago by Matthias Köppe

Upgrading setuptools to 28.8.0 does not fix the pyzmq install.

I'll wait for the pip upgrade.

comment:33 Changed 6 years ago by Matthias Köppe

Branch: 6dcaebb5f95abdc78cce729bcced7dc96ee5cb1fu/mkoeppe/6dcaebb5f95abdc78cce729bcced7dc96ee5cb1f

comment:34 Changed 6 years ago by git

Commit: c21829914692175177a5a66775c015719fd450fd

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

bf1b130Use 'pip --isolated' and 'setup.py --no-user-cfg'
c218299pyzmq: Fix pip install command line

comment:35 Changed 6 years ago by Matthias Köppe

Status: newneeds_review

Here's a fix for our pyzmq spkg-install, with help from upstream.

comment:36 Changed 6 years ago by Matthias Köppe

Dependencies: #21835

comment:37 Changed 6 years ago by Jeroen Demeyer

Just a comment: it would make sense to use $PIP_INSTALL for a few more packages where no extra options are needed: fpylll, gambit, sagenb, sagetex.

comment:38 Changed 6 years ago by Matthias Köppe

I agree, but that should be on a different ticket. I've created #21864 for this.

comment:39 Changed 6 years ago by Matthias Köppe

Needs review.

comment:40 Changed 6 years ago by git

Commit: c21829914692175177a5a66775c015719fd450fd375faba4699c44e2be0ecceddfbca29de2770f85

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

1ea8221Use 'pip --isolated' and 'setup.py --no-user-cfg'
375fabapyzmq: Fix pip install command line

comment:41 Changed 6 years ago by Matthias Köppe

Milestone: sage-7.5sage-7.6

Rebased, needs review.

comment:42 Changed 6 years ago by Matthias Köppe

ping?

comment:43 Changed 6 years ago by Erik Bray

Status: needs_reviewpositive_review

The issue with --global-option and --no-user-cfg still seems odd to me. I think that should be considered a bug in pip. But I guess there's nothing to be done for now.

comment:44 Changed 6 years ago by Volker Braun

Branch: u/mkoeppe/6dcaebb5f95abdc78cce729bcced7dc96ee5cb1f375faba4699c44e2be0ecceddfbca29de2770f85
Resolution: fixed
Status: positive_reviewclosed
Note: See TracTickets for help on using tickets.