Sage: Ticket #14243: Try not to pick up user versions of things like numpy, mpl
https://trac.sagemath.org/ticket/14243
<p>
At <a class="ext-link" href="http://ask.sagemath.org/question/1449/sage-uses-system-wide-python"><span class="icon"></span>this ask.sagemath.org question</a>, where some users have the system Python being picked up for some reasons, Volker asks
</p>
<pre class="wiki">Can you try
export PYTHONNOUSERSITE=yes
sage
This should prevent Sage from picking up stuff in .local/lib. If that fixes your problem we can add it to the next release.
</pre><p>
Apparently this works, and <a class="ext-link" href="https://groups.google.com/forum/?fromgroups=#!topic/sage-support/Y9bjhb6-dcM"><span class="icon"></span>this sage-support question</a> suggests it works a lot. Maybe we should put this in now.
</p>
en-usSagehttps://trac.sagemath.org/chrome/site/logo_sagemath_trac.png
https://trac.sagemath.org/ticket/14243
Trac 1.1.6jdemeyerTue, 13 Aug 2013 15:34:36 GMTmilestone changed
https://trac.sagemath.org/ticket/14243#comment:1
https://trac.sagemath.org/ticket/14243#comment:1
<ul>
<li><strong>milestone</strong>
changed from <em>sage-5.11</em> to <em>sage-5.12</em>
</li>
</ul>
Ticketvbraun_spamThu, 30 Jan 2014 21:20:52 GMTmilestone changed
https://trac.sagemath.org/ticket/14243#comment:2
https://trac.sagemath.org/ticket/14243#comment:2
<ul>
<li><strong>milestone</strong>
changed from <em>sage-6.1</em> to <em>sage-6.2</em>
</li>
</ul>
Ticketvbraun_spamTue, 06 May 2014 15:19:32 GMTmilestone changed
https://trac.sagemath.org/ticket/14243#comment:3
https://trac.sagemath.org/ticket/14243#comment:3
<ul>
<li><strong>milestone</strong>
changed from <em>sage-6.2</em> to <em>sage-6.3</em>
</li>
</ul>
Ticketvbraun_spamSun, 10 Aug 2014 16:51:03 GMTmilestone changed
https://trac.sagemath.org/ticket/14243#comment:4
https://trac.sagemath.org/ticket/14243#comment:4
<ul>
<li><strong>milestone</strong>
changed from <em>sage-6.3</em> to <em>sage-6.4</em>
</li>
</ul>
TicketjhpalmieriTue, 28 Jul 2015 20:00:54 GMTbranch set
https://trac.sagemath.org/ticket/14243#comment:5
https://trac.sagemath.org/ticket/14243#comment:5
<ul>
<li><strong>branch</strong>
set to <em>u/jhpalmieri/nousersite</em>
</li>
</ul>
TicketjhpalmieriTue, 28 Jul 2015 20:01:32 GMTstatus, cc changed; commit set
https://trac.sagemath.org/ticket/14243#comment:6
https://trac.sagemath.org/ticket/14243#comment:6
<ul>
<li><strong>status</strong>
changed from <em>new</em> to <em>needs_review</em>
</li>
<li><strong>commit</strong>
set to <em>e35ebe9bb554e4f00bcd0c4eabe91cb196997a1a</em>
</li>
<li><strong>cc</strong>
<em>jdemeyer</em> added
</li>
</ul>
<p>
New commits:
</p>
<table class="wiki">
<tr><td><a class="ext-link" href="http://git.sagemath.org/sage.git/commit/?id=e35ebe9bb554e4f00bcd0c4eabe91cb196997a1a"><span class="icon"></span>e35ebe9</a></td><td><code>trac 14243: set PYTHONNOUSERSITE=yes to avoid picking up user's Python files.</code>
</td></tr></table>
TicketbuckTue, 28 Jul 2015 20:07:52 GMTstatus changed; reviewer set
https://trac.sagemath.org/ticket/14243#comment:7
https://trac.sagemath.org/ticket/14243#comment:7
<ul>
<li><strong>status</strong>
changed from <em>needs_review</em> to <em>positive_review</em>
</li>
<li><strong>reviewer</strong>
set to <em>Buck Evan</em>
</li>
</ul>
TicketvbraunTue, 28 Jul 2015 22:48:29 GMTstatus changed
https://trac.sagemath.org/ticket/14243#comment:8
https://trac.sagemath.org/ticket/14243#comment:8
<ul>
<li><strong>status</strong>
changed from <em>positive_review</em> to <em>needs_work</em>
</li>
</ul>
<p>
Author name is missing....
</p>
TicketjhpalmieriTue, 28 Jul 2015 22:50:13 GMTstatus changed; author set
https://trac.sagemath.org/ticket/14243#comment:9
https://trac.sagemath.org/ticket/14243#comment:9
<ul>
<li><strong>status</strong>
changed from <em>needs_work</em> to <em>positive_review</em>
</li>
<li><strong>author</strong>
set to <em>John Palmieri</em>
</li>
</ul>
<p>
Oops.
</p>
TicketvbraunWed, 29 Jul 2015 14:59:59 GMTstatus, branch changed; resolution set
https://trac.sagemath.org/ticket/14243#comment:10
https://trac.sagemath.org/ticket/14243#comment:10
<ul>
<li><strong>status</strong>
changed from <em>positive_review</em> to <em>closed</em>
</li>
<li><strong>resolution</strong>
set to <em>fixed</em>
</li>
<li><strong>branch</strong>
changed from <em>u/jhpalmieri/nousersite</em> to <em>e35ebe9bb554e4f00bcd0c4eabe91cb196997a1a</em>
</li>
</ul>
TicketwasMon, 26 Oct 2015 00:23:29 GMTcommit deleted
https://trac.sagemath.org/ticket/14243#comment:11
https://trac.sagemath.org/ticket/14243#comment:11
<ul>
<li><strong>commit</strong>
<em>e35ebe9bb554e4f00bcd0c4eabe91cb196997a1a</em> deleted
</li>
</ul>
<p>
That we do this is a shame. It only further alienates Sage from standard practices. It makes it so this FAQ entry is no longer valid:
</p>
<p>
<a class="ext-link" href="https://github.com/sagemathinc/smc/wiki/FAQ#-question-how-can-i-install-python-packages-from-httpspypipythonorgpypi--using-pip"><span class="icon"></span>https://github.com/sagemathinc/smc/wiki/FAQ#-question-how-can-i-install-python-packages-from-httpspypipythonorgpypi--using-pip</a>
</p>
<p>
It makes it very, very painful for anybody to ever install Python packages and use them without Sage, without actually modifying a Sage install. This is very much exactly the opposite direction in which we should be going. Big -1.
</p>
TicketjhpalmieriMon, 26 Oct 2015 15:22:59 GMT
https://trac.sagemath.org/ticket/14243#comment:12
https://trac.sagemath.org/ticket/14243#comment:12
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/14243#comment:11" title="Comment 11">was</a>:
</p>
<blockquote class="citation">
<p>
It makes it very, very painful for anybody to ever install Python packages and use them without Sage, without actually modifying a Sage install.
</p>
</blockquote>
<p>
I thought this was exactly what it <em>did</em> accomplish: allow people to install their own Python packages and not have them interfere with Sage, not have Sage interfere with them. Maybe I am misunderstanding what you're saying.
</p>
TicketwasMon, 26 Oct 2015 16:14:29 GMT
https://trac.sagemath.org/ticket/14243#comment:13
https://trac.sagemath.org/ticket/14243#comment:13
<p>
I made a horrible typo. I meant "It makes it very, very painful for anybody to ever install Python packages and use them <strong>with</strong> Sage, without actually modifying a Sage install."
</p>
<p>
This is an absolutely massive problem with <a class="missing wiki">SageMathCloud?</a>, where there is one Sage install and (literally) 15,000 users on a single computer that use that Sage install. They can't all have their own copy of Sage. So when they install packages (often from pip) to use *with* Sage, they have to use the --user flag. My solution is that I now have a fork of Sage that removes the code from this ticket.
</p>
<p>
<a class="ext-link" href="https://github.com/sagemathinc/smc-sage/commit/7c9cfd095463091bcf41e1c6ea6e8eebfb61f00c"><span class="icon"></span>https://github.com/sagemathinc/smc-sage/commit/7c9cfd095463091bcf41e1c6ea6e8eebfb61f00c</a>
</p>
<p>
This works for me, I guess. However, I doubt I'm the only person with this problem.
</p>
<p>
I think the right solution in this case should be better educate people about how to control their Python packaging. For example, for those ask posts, and this problem, we should instead have something in the README about PYTHONNOUSERSITE=yes, and a link to the relevant Python docs. Telling the person with the problem to use that environment variable would have solved the problem for them.
</p>
TicketwasMon, 26 Oct 2015 16:20:48 GMT
https://trac.sagemath.org/ticket/14243#comment:14
https://trac.sagemath.org/ticket/14243#comment:14
<p>
Follow up. I think we absolutely must move to a model where we make it easy for users to create and share packages of code that depend on Sage. Our current spkg system is a complete failure in this regard, and has only got worse with its tight integration into the Sage git repo. In constrast, setuptools and pypi -- the official Python packaging framework -- works very well, is heavily tested, and there's tons of documentation and tutorials. Instead of making it much harder to use pip/setuptools/pypi with Sage, which is what this trac ticket does, we should be making it much easier.
</p>
<p>
There are about 6,000 user-created R packages in Rcran -- I've installed many of these (ok, dozens requested by users), and they all install perfectly. Sage could have hundreds or even thousands of packages on pip right now. We could have an automated testing framework that downloads them all and builds them, and runs their test suites (setuptools has integrated test suite support). We could decide to include some of these packages standard with Sage, and referee them. It's an epic mistake that we aren't doing this with a passion right now. Let's not make it even more difficult.
</p>
TicketvbraunMon, 26 Oct 2015 16:58:25 GMT
https://trac.sagemath.org/ticket/14243#comment:15
https://trac.sagemath.org/ticket/14243#comment:15
<p>
There are two different use cases:
</p>
<ul><li>A private build of Sage (everything user-writable) should not conflict with another private numpy installation. You can use pip just fine without the <code>--user</code> option. Then <code>PYTHONNOUSERSITE=yes</code> is the way to achieve that.
</li><li>A system-wide build of Sage (nothing user-writable, so just <code>pip install foo</code> cannot work) should work with <code>pip --user</code>. This is the SMC problem.
</li></ul><p>
Really this is a special case of distinguishing a build-from-source and a binary install. Sage traditionally pretends that there is no difference, but there is.
</p>
TicketvbraunMon, 26 Oct 2015 17:05:16 GMT
https://trac.sagemath.org/ticket/14243#comment:16
https://trac.sagemath.org/ticket/14243#comment:16
<p>
BTW its now really easy to use pip for optional packages:
</p>
<pre class="wiki">$ ll build/pkgs/trac/
total 4
-rw-rw-r--. 1 vbraun vbraun 4 Sep 27 14:25 type
$ cat build/pkgs/trac/type
pip
</pre><p>
For various non-technical reasons (mainly OSX and OpenSSL) we can't use pip for all our Python packaging (yet).
</p>
TicketwasMon, 26 Oct 2015 17:16:49 GMT
https://trac.sagemath.org/ticket/14243#comment:17
https://trac.sagemath.org/ticket/14243#comment:17
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/14243#comment:15" title="Comment 15">vbraun</a>:
</p>
<blockquote class="citation">
<p>
There are two different use cases:
</p>
<ul><li>A private build of Sage (everything user-writable) should not conflict with another private numpy installation. You can use pip just fine without the <code>--user</code> option. Then <code>PYTHONNOUSERSITE=yes</code> is the way to achieve that.
</li><li>A system-wide build of Sage (nothing user-writable, so just <code>pip install foo</code> cannot work) should work with <code>pip --user</code>. This is the SMC problem.
</li></ul><p>
Really this is a special case of distinguishing a build-from-source and a binary install. Sage traditionally pretends that there is no difference, but there is.
</p>
</blockquote>
<p>
Agreed. I think the terminology "build-from-source" and "binary install" is a little misleading though, since *I* did build the SMC install from source myself. Maybe "system-wide" versus "personal install" would be a better distinction.
</p>
<p>
Any thoughts about how to technically distinguish between these two types of setups? A flag to ./configure on first build? Something else?
</p>
<blockquote class="citation">
<p>
BTW its now really easy to use pip for optional packages
</p>
</blockquote>
<p>
Awesome -- a very good step in the right direction!
</p>
TicketvbraunMon, 26 Oct 2015 17:56:10 GMT
https://trac.sagemath.org/ticket/14243#comment:18
https://trac.sagemath.org/ticket/14243#comment:18
<p>
Of course every software must be built from source at one point, possibly by your distributor ;-)
</p>
<p>
The easiest way would probably be to check if <code>$SAGE_ROOT</code> is writable by the current user.
</p>
<p>
In fact, there are four different cases for the two independent choices:
</p>
<ul><li>Private install (I have write permissions) vs. system-wide (I do not have write permissions)
</li><li>Build from source vs. using a pre-built binary
</li></ul><p>
The issue here only depends on the first choice.
</p>
TicketwasSat, 21 Nov 2015 20:02:19 GMT
https://trac.sagemath.org/ticket/14243#comment:19
https://trac.sagemath.org/ticket/14243#comment:19
<p>
I think we should revert this change. I will always revert it in SMC, and user confusion is happening for sure (e.g., a sage-support message yesterday). I've made <a class="closed ticket" href="https://trac.sagemath.org/ticket/19612" title="enhancement: Use PYTHONUSERBASE (closed: fixed)">#19612</a>, which is revert this.
</p>
<blockquote>
<p>
-- William
</p>
</blockquote>
Ticket