Opened 7 years ago

Closed 5 years ago

Last modified 4 years ago

#14243 closed defect (fixed)

Try not to pick up user versions of things like numpy, mpl

Reported by: kcrisman Owned by: GeorgSWeber
Priority: critical Milestone: sage-6.4
Component: build Keywords:
Cc: vbraun, jdemeyer Merged in:
Authors: John Palmieri Reviewers: Buck Evan
Report Upstream: N/A Work issues:
Branch: e35ebe9 (Commits) Commit:
Dependencies: Stopgaps:

Description

At this ask.sagemath.org question, where some users have the system Python being picked up for some reasons, Volker asks

Can you try
export PYTHONNOUSERSITE=yes
sage
This should prevent Sage from picking up stuff in .local/lib. If that fixes your problem we can add it to the next release.

Apparently this works, and this sage-support question suggests it works a lot. Maybe we should put this in now.

Change History (19)

comment:1 Changed 7 years ago by jdemeyer

  • Milestone changed from sage-5.11 to sage-5.12

comment:2 Changed 6 years ago by vbraun_spam

  • Milestone changed from sage-6.1 to sage-6.2

comment:3 Changed 6 years ago by vbraun_spam

  • Milestone changed from sage-6.2 to sage-6.3

comment:4 Changed 6 years ago by vbraun_spam

  • Milestone changed from sage-6.3 to sage-6.4

comment:5 Changed 5 years ago by jhpalmieri

  • Branch set to u/jhpalmieri/nousersite

comment:6 Changed 5 years ago by jhpalmieri

  • Cc jdemeyer added
  • Commit set to e35ebe9bb554e4f00bcd0c4eabe91cb196997a1a
  • Status changed from new to needs_review

New commits:

e35ebe9trac 14243: set PYTHONNOUSERSITE=yes to avoid picking up user's Python files.

comment:7 Changed 5 years ago by buck

  • Reviewers set to Buck Evan
  • Status changed from needs_review to positive_review

comment:8 Changed 5 years ago by vbraun

  • Status changed from positive_review to needs_work

Author name is missing....

comment:9 Changed 5 years ago by jhpalmieri

  • Authors set to John Palmieri
  • Status changed from needs_work to positive_review

Oops.

comment:10 Changed 5 years ago by vbraun

  • Branch changed from u/jhpalmieri/nousersite to e35ebe9bb554e4f00bcd0c4eabe91cb196997a1a
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:11 follow-up: Changed 4 years ago by was

  • Commit e35ebe9bb554e4f00bcd0c4eabe91cb196997a1a deleted

That we do this is a shame. It only further alienates Sage from standard practices. It makes it so this FAQ entry is no longer valid:

https://github.com/sagemathinc/smc/wiki/FAQ#-question-how-can-i-install-python-packages-from-httpspypipythonorgpypi--using-pip

It makes it very, very painful for anybody to ever install Python packages and use them without Sage, without actually modifying a Sage install. This is very much exactly the opposite direction in which we should be going. Big -1.

comment:12 in reply to: ↑ 11 Changed 4 years ago by jhpalmieri

Replying to was:

It makes it very, very painful for anybody to ever install Python packages and use them without Sage, without actually modifying a Sage install.

I thought this was exactly what it did accomplish: allow people to install their own Python packages and not have them interfere with Sage, not have Sage interfere with them. Maybe I am misunderstanding what you're saying.

comment:13 Changed 4 years ago by was

I made a horrible typo. I meant "It makes it very, very painful for anybody to ever install Python packages and use them with Sage, without actually modifying a Sage install."

This is an absolutely massive problem with SageMathCloud?, where there is one Sage install and (literally) 15,000 users on a single computer that use that Sage install. They can't all have their own copy of Sage. So when they install packages (often from pip) to use *with* Sage, they have to use the --user flag. My solution is that I now have a fork of Sage that removes the code from this ticket.

https://github.com/sagemathinc/smc-sage/commit/7c9cfd095463091bcf41e1c6ea6e8eebfb61f00c

This works for me, I guess. However, I doubt I'm the only person with this problem.

I think the right solution in this case should be better educate people about how to control their Python packaging. For example, for those ask posts, and this problem, we should instead have something in the README about PYTHONNOUSERSITE=yes, and a link to the relevant Python docs. Telling the person with the problem to use that environment variable would have solved the problem for them.

comment:14 Changed 4 years ago by was

Follow up. I think we absolutely must move to a model where we make it easy for users to create and share packages of code that depend on Sage. Our current spkg system is a complete failure in this regard, and has only got worse with its tight integration into the Sage git repo. In constrast, setuptools and pypi -- the official Python packaging framework -- works very well, is heavily tested, and there's tons of documentation and tutorials. Instead of making it much harder to use pip/setuptools/pypi with Sage, which is what this trac ticket does, we should be making it much easier.

There are about 6,000 user-created R packages in Rcran -- I've installed many of these (ok, dozens requested by users), and they all install perfectly. Sage could have hundreds or even thousands of packages on pip right now. We could have an automated testing framework that downloads them all and builds them, and runs their test suites (setuptools has integrated test suite support). We could decide to include some of these packages standard with Sage, and referee them. It's an epic mistake that we aren't doing this with a passion right now. Let's not make it even more difficult.

comment:15 follow-up: Changed 4 years ago by vbraun

There are two different use cases:

  • A private build of Sage (everything user-writable) should not conflict with another private numpy installation. You can use pip just fine without the --user option. Then PYTHONNOUSERSITE=yes is the way to achieve that.
  • A system-wide build of Sage (nothing user-writable, so just pip install foo cannot work) should work with pip --user. This is the SMC problem.

Really this is a special case of distinguishing a build-from-source and a binary install. Sage traditionally pretends that there is no difference, but there is.

comment:16 Changed 4 years ago by vbraun

BTW its now really easy to use pip for optional packages:

$ ll build/pkgs/trac/
total 4
-rw-rw-r--. 1 vbraun vbraun 4 Sep 27 14:25 type
$ cat build/pkgs/trac/type 
pip

For various non-technical reasons (mainly OSX and OpenSSL) we can't use pip for all our Python packaging (yet).

comment:17 in reply to: ↑ 15 Changed 4 years ago by was

Replying to vbraun:

There are two different use cases:

  • A private build of Sage (everything user-writable) should not conflict with another private numpy installation. You can use pip just fine without the --user option. Then PYTHONNOUSERSITE=yes is the way to achieve that.
  • A system-wide build of Sage (nothing user-writable, so just pip install foo cannot work) should work with pip --user. This is the SMC problem.

Really this is a special case of distinguishing a build-from-source and a binary install. Sage traditionally pretends that there is no difference, but there is.

Agreed. I think the terminology "build-from-source" and "binary install" is a little misleading though, since *I* did build the SMC install from source myself. Maybe "system-wide" versus "personal install" would be a better distinction.

Any thoughts about how to technically distinguish between these two types of setups? A flag to ./configure on first build? Something else?

BTW its now really easy to use pip for optional packages

Awesome -- a very good step in the right direction!

comment:18 Changed 4 years ago by vbraun

Of course every software must be built from source at one point, possibly by your distributor ;-)

The easiest way would probably be to check if $SAGE_ROOT is writable by the current user.

In fact, there are four different cases for the two independent choices:

  • Private install (I have write permissions) vs. system-wide (I do not have write permissions)
  • Build from source vs. using a pre-built binary

The issue here only depends on the first choice.

comment:19 Changed 4 years ago by was

I think we should revert this change. I will always revert it in SMC, and user confusion is happening for sure (e.g., a sage-support message yesterday).

-- William

Version 0, edited 4 years ago by was (next)
Note: See TracTickets for help on using tickets.