python setup.py picks prefix from ~/.pydistutils.cfg

[Alexander Dreyer]

Hi, several packages do use python setup.py. This picks up the customized settings from ~/.pydistutils.cfg. This is bad, because it overrides the prefix setting.

For sage-main, I'll attach a patch: setup.cfg in the corresponding directory overrides the usere settings. Maybe there's a global solution.

Regards,

Alexander Dreyer

......

Update for Sage 7.5. The problem reported on this ancient ticket is still present and affects

• sagelib,
• packages that are installed directly using setup.py
• packages that are installed using pip.

It can be tested by creating the following ~/.pydistutils.cfg:

[install]
root=/does-not-exist

The patch on the ticket uses

• pip --isolated and
• setup.py --no-user-cfg

Note there appear to be some issues regarding these option with early versions of Python 3.x: ​https://github.com/pypa/pip/issues/3509

[Alexander Dreyer]

Patch for sage-main

[Alexander Dreyer]

The following packages from the distribution use setup.py and may be affected:

zodb3-3.7.0.p4
scipy_sandbox-20071020.p5
pexpect-2.0.p4
sqlalchemy-0.5.8
glpk-4.44
mpmath-0.15
pil-1.1.6.p2
mercurial-1.3.1.p2
pycrypto-2.0.1.p5
mpir-1.2.2.p1
weave-0.4.9.p0
sympy-0.6.4.p0
scons-1.2.0
numpy-1.3.0.p3
gdmodule-0.56.p7
sagetex-2.2.5
sage-4.5
matplotlib-0.99.3
sage_scripts-4.5
setuptools-0.6c9.p0
python-2.6.4.p9
python_gnutls-1.1.4.p7
sagenb-0.8.1
networkx-1.0.1
jinja-1.2.p0
cython-0.12.1
pygments-0.11.1.p0
scipy-0.7.p5
cvxopt-0.9.p8
jinja2-2.1.1.p0
sphinx-0.6.3.p4
docutils-0.5.p0
twisted-9.0.p2
ipython-0.9.1.p0
moin-1.9.1.p1

The following already have setup.cfg, but it may not have the prefix definition:

zodb3-3.7.0.p4
sqlalchemy-0.5.8
scons-1.2.0
matplotlib-0.99.3
setuptools-0.6c9.p0
sagenb-0.8.1
networkx-1.0.1
jinja-1.2.p0
pygments-0.11.1.p0
jinja2-2.1.1.p0
sphinx-0.6.3.p4
twisted-9.0.p2
moin-1.9.1.p1

[PolyBoRi team]

Hi! I am not an export for that for that, but did you read ​http://docs.python.org/install/#location-and-names-of-config-files There seems to be some options: –no-user-cfg

Cheers, Michael

[Alexander Dreyer]

Hi Michael, indeed, that's a better solution. But both have the problem, that 30+ packages would have to be fixed, and it might be forgotten in the future. (Maybe there some monkey patch for distutils, that all of it can be overwritten by the environment.)

Best regards,

Alexander

[PolyBoRi team]

maybe, I did not try (just read the docs): ​http://docs.python.org/install/#inst-config-files

create system distutils.cfg

[global]
no-user-cfg=1

[Alexander Dreyer]

Hi, unfortunately, this is a Python 2.7 feature and it does not support no-user-cfg in distutils.cfg :-(

Regards,

Alexander

[Alexander Dreyer]

patch for sage/spkg/base/sage-env sage/local/bin/sage-env (needs python 2.6.4.p10)

[Alexander Dreyer]

Hi, I backported the handling of setup.py --no-user-cfg from Python 2.7 to Python 2.6.4 and also added the handling of the environment variable DISTUTILS_NO_USER_CFG to python's distutils.

The new spkg can be found here: ​http://sage.math.washington.edu/home/dreyer/suse101/python-2.6.4.p10.spkg

The last patch adds this variable to sage-env.

Regards,

Alexander

[Alexander Dreyer]

Reported upstream: ​http://bugs.python.org/issue9309

[Alexander Dreyer]

Minor update: ​http://sage.math.washington.edu/home/dreyer/suse101/python-2.6.4.p10.spkg (useful, if the sage-env patch is not used)

[David Kirkby]

Replying to AlexanderDreyer:

Minor update: ​http://sage.math.washington.edu/home/dreyer/suse101/python-2.6.4.p10.spkg (useful, if the sage-env patch is not used)

You should also create patches/ACKS.txt.diff as a unified diff file. Although its fairly obvious you can recreate that now, that will not be the case if Python source code is updated.

Would a simpler solution not be for sage-env to set this variable and export it? I don't know much about python, so can't review this myself.

Dave

[Alexander Dreyer]

Maybe a misunderstanding: the patch of python is necessary to fix that issue anyway. (This first variant didn't work, if the environment variable was not set.)

[Alexander Dreyer]

Adding handling of environment variable DISTUTILS_NO_USER_CFG to python's distutils

[Alexander Dreyer]

I've added ACKS.txt.patch to the spkg.

[Alexander Dreyer]

The python community does not like the idea of adding an environment variable, see: ​http://bugs.python.org/issue9309

[Jeroen Demeyer]

Please fill in your real name as Author.

[Karl-Dieter Crisman]

Needs a branch. And we have updated Python. And probably other reasons it isn't ready yet. (Or conceivably has been solved in the meantime?)

[Matthias Köppe]

New commits:

​6dcaebb

Use 'pip --isolated' and 'setup.py --no-user-cfg'

[Erik Bray]

This looks like the right approach, thanks.

[Volker Braun]

Prevents pyzmq from building...

[Volker Braun]

  Source in /tmp/pip-mTg5Xg-build has version 16.0.0, which satisfies requirement pyzmq==16.0.0 from file:///mnt/disk/home/release/Sage/local/var/tmp/sage/build/pyzmq-16.0.0/src
Installing collected packages: pyzmq
  Running setup.py install for pyzmq: started
    Running command /mnt/disk/home/release/Sage/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-mTg5Xg-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" configure --zmq=/mnt/disk/home/release/Sage/local --no-user-cfg install --record /tmp/pip-o79sR3-record/install-record.txt --single-version-externally-managed --compile
    running configure
    error: error in command line: command 'Configure' has no such option 'no_user_cfg'
    Running setup.py install for pyzmq: finished with status 'error'
Cleaning up...
  Removing source in /tmp/pip-mTg5Xg-build
Command "/mnt/disk/home/release/Sage/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-mTg5Xg-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" configure --zmq=/mnt/disk/home/release/Sage/local --no-user-cfg install --record /tmp/pip-o79sR3-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-mTg5Xg-build/
Exception information:
Traceback (most recent call last):
  File "/mnt/disk/home/release/Sage/local/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/mnt/disk/home/release/Sage/local/lib/python2.7/site-packages/pip/commands/install.py", line 317, in run
    prefix=options.prefix_path,
  File "/mnt/disk/home/release/Sage/local/lib/python2.7/site-packages/pip/req/req_set.py", line 742, in install
    **kwargs
  File "/mnt/disk/home/release/Sage/local/lib/python2.7/site-packages/pip/req/req_install.py", line 880, in install
    spinner=spinner,
  File "/mnt/disk/home/release/Sage/local/lib/python2.7/site-packages/pip/utils/__init__.py", line 718, in call_subprocess
    % (command_desc, proc.returncode, cwd))
InstallationError: Command "/mnt/disk/home/release/Sage/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-mTg5Xg-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" configure --zmq=/mnt/disk/home/release/Sage/local --no-user-cfg install --record /tmp/pip-o79sR3-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-mTg5Xg-build/

[Matthias Köppe]

I've created an upstream issue for this:

​https://github.com/zeromq/pyzmq/issues/937

[Erik Bray]

That's annoying. For now we should probably remove pip install from pyzmq.

[Matthias Köppe]

Upstream pyzmq reports that it works with pip 9.0.1, setuptools 28.8.0.

Let's try with upgrading these packages. #21835 does the upgrade of pip to 9.0.0...

[Matthias Köppe]

Upgrading setuptools to 28.8.0 does not fix the pyzmq install.

I'll wait for the pip upgrade.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​bf1b130	Use 'pip --isolated' and 'setup.py --no-user-cfg'
​c218299	pyzmq: Fix pip install command line

[Matthias Köppe]

Here's a fix for our pyzmq spkg-install, with help from upstream.

[Jeroen Demeyer]

Just a comment: it would make sense to use $PIP_INSTALL for a few more packages where no extra options are needed: fpylll, gambit, sagenb, sagetex.

[Matthias Köppe]

I agree, but that should be on a different ticket. I've created #21864 for this.

[Matthias Köppe]

Needs review.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​1ea8221	Use 'pip --isolated' and 'setup.py --no-user-cfg'
​375faba	pyzmq: Fix pip install command line

[Matthias Köppe]

Rebased, needs review.

[Matthias Köppe]

ping?

[Erik Bray]

The issue with --global-option and --no-user-cfg still seems odd to me. I think that should be considered a bug in pip. But I guess there's nothing to be done for now.