Opened 6 years ago
Closed 14 months ago
#21507 closed task (invalid)
Meta-ticket: Make sagelib a pip-installable Python source package, listed on PyPI
Reported by: | mkoeppe | Owned by: | |
---|---|---|---|
Priority: | major | Milestone: | sage-duplicate/invalid/wontfix |
Component: | build | Keywords: | pip, PyPI |
Cc: | jdemeyer, was, vbraun, vdelecroix, dimpase, fbissey, embray, leif, aenge, nthiery, mmarco, klee, robertwb, infinity0, thansen, defeo, slelievre, gh-timokau, klui, isuruf, jhpalmieri | Merged in: | |
Authors: | Reviewers: | Dima Pasechnik | |
Report Upstream: | N/A | Work issues: | |
Branch: | Commit: | ||
Dependencies: | Stopgaps: |
Description (last modified by )
This task ticket, organizing the steps necessary to eventually provide sagelib via PyPI, has been superseded by Meta-ticket #29705: Modularize sagelib into separate distributions (distutils packages).
. . . . . . . . . . . . . . . . .
It would be used by a Python user as follows. Let's assume a user has already installed every package that sage-the-distribution provides on their distribution. Then a simple
pip install sagelib
would install sagelib in the user's Python installation. Then the user could do from sage.all import *
.
. . . . . . . . . . . . . . . . .
Here are the first steps:
- #21508: Clean up
src/setup.py
to bring it to standard distutils behavior- #21480: Make
sagelib setup.py
self-contained, independent ofSAGE_ROOT
- #21600: Use custom build_ext to compile Cython code
- #21604: Cleaning up stale installed files in setup()
- #21654: Disentangle cleaning of stale installed files in build directory and in install directory
- #21516: Fix sagelib sdist (
src/setup.py sdist
) - #21535: Make
src/setup.py
respect--build-base
and--inplace
, independent ofSAGE_CYTHONIZED
- #12659: build the sage library in place
- #21573: Make sure
src/setup.py
respects--install-base
and--root
- #21613: Make setup.py not depend on make (follow-up: #22044, #22094, #22106)
- #22061: Don't hard-code the path to thebe.js (dup: #21527: Fix symbolic link to
thebe.js
) - #25546: Install Jupyter kernel in the correct prefix
- #29041: at
./bootstrap
time, generatesrc/requirements.txt
,src/constraints.txt
,src/setup.cfg
[install_requires]
frombuild/pkgs
- #21480: Make
- #21678: Testsuite
for src/setup.py
- #21785: Installation of
SAGE_SRC/ext/
inSAGE_LOCAL/share/sage/ext/
should be done bysetup.py
, notbuild/make/Makefile
- #20108: Use package_data instead of data_files in setup.py
- #21682: Add a separate "cythonize" command to setup.py
- #21569: Install
src/bin/*
scripts viasetup.py
(scripts
,console_scripts
) - #21570: Move non-scripts of
src/bin/
elsewhere (and also move their install location) - #21571: Install
COPYING.txt
inSAGE_LOCAL
and use it frommisc/copying.py
- #21509: Install cython_debug somewhere in SAGE_LOCAL
- #21732: Move runtime documentation python modules into
src/sage
- #29038: Python package
sage_conf
: Provides optional configuration information forsagelib
This defines milestone 1. sagelib
is now a well-behaved Python package. It can be built and installed as follows (without invoking sage -sh
):
export SAGE_LOCAL=/path/to/local/hierarchy/populated/by/sage/distribution export SAGE_PKGS=/path/to/sage/distribution/source/directory/build/pkgs $SAGE_LOCAL/bin/python setup.py install # or pip install .
. . . . . . . . . . . . . . . . .
Next steps:
- Remove the dependency on the environment variable SAGE_PKGS (#20382, ...)
This defines milestone 2. sagelib
can now be built and installed as follows (without invoking sage -sh
):
export SAGE_LOCAL=/path/to/local/hierarchy/populated/by/sage/distribution $SAGE_LOCAL/bin/python setup.py install # or pip install .
. . . . . . . . . . . . . . . . .
Next steps:
- Remove the dependency on the SAGE_LOCAL environment variable
This defines milestone 3. If SAGE_LOCAL is not set, then sagelib will discover system packages and Python packages installed in standard places.
python setup.py install # or pip install .
At this point, sagelib will be a standard pip-installable Python source package, ready for upload to PyPI.
Binary packages (using wheels etc.) is beyond the scope of this ticket.
. . . . . . . . . . . . . . . . .
See also:
- #29133 META-META-TICKET: Build, packaging, testing improvements
. . . . . . . . . . . . . . References . . . . . . . . . . . . . . .
- https://docs.python.org/2/library/distutils.html
- https://setuptools.readthedocs.io/en/latest/setuptools.html#command-reference
- https://docs.python.org/2/distutils/sourcedist.html
- https://docs.python.org/2/install/
- https://pip.pypa.io/en/stable/reference/pip_install/#pip-install-examples
- http://askubuntu.com/questions/802544/is-sudo-pip-install-still-a-broken-practice
- http://stackoverflow.com/questions/33004708/osx-el-capitan-sudo-pip-install-oserror-errno-1-operation-not-permitted/33004920#33004920
- https://packaging.python.org/science/
- https://packaging.python.org/current/
- https://packaging.python.org/key_projects/#setuptools
- https://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode
- https://www.cac.cornell.edu/wiki/index.php?title=Python_Distutils_Tips
- https://blog.ionelmc.ro/2014/05/25/python-packaging/
- http://stackoverflow.com/questions/7522250/how-to-include-package-data-with-setuptools-distribute
Change History (72)
comment:1 Changed 6 years ago by
- Cc was vbraun vdelecroix dimpase added
- Description modified (diff)
comment:2 Changed 6 years ago by
- Cc fbissey embray leif added
- Description modified (diff)
comment:3 Changed 6 years ago by
- Description modified (diff)
comment:4 Changed 6 years ago by
- Milestone changed from sage-7.4 to sage-7.5
comment:5 Changed 6 years ago by
- Description modified (diff)
comment:6 Changed 6 years ago by
- Description modified (diff)
comment:7 Changed 6 years ago by
- Description modified (diff)
comment:8 Changed 6 years ago by
- Cc aenge nthiery added
comment:9 Changed 6 years ago by
- Cc mmarco klee robertwb added
comment:10 follow-up: ↓ 11 Changed 6 years ago by
comment:11 in reply to: ↑ 10 Changed 6 years ago by
Replying to embray:
There's more still be to added to the list of issues, such as providing binary wheels and how that will be handled.
Definitely. I'm hoping that we can use this ticket to organize it.
I think it will be important to also make more progress on making some hard dependencies optional (that is, allow optional features to fail gracefully if a dependency for that feature is not found).
I agree.
comment:12 Changed 6 years ago by
Here's a question for the distutils experts. Is a Python package allowed to install things in bin/
? Do any other Python packages do that?
Where would one install helper scripts? Is there a Python equivalent of $prefix/libexec
?
comment:13 Changed 6 years ago by
- Description modified (diff)
comment:14 Changed 6 years ago by
This Python package, which is part of SMC, installs many things to /usr/local/bin using completely standard approaches: https://github.com/sagemathinc/smc/tree/master/src/smc_pyutil. See, in particular, the console_scripts section of setup.py: https://github.com/sagemathinc/smc/blob/master/src/smc_pyutil/setup.py#L58
comment:15 Changed 6 years ago by
Thanks a lot! Following your pointer, I've found this documentation: http://python-packaging.readthedocs.io/en/latest/command-line-scripts.html, which I will refer to alongside the example that you pointed out.
comment:16 Changed 6 years ago by
- Description modified (diff)
comment:17 Changed 6 years ago by
This now #21569.
comment:18 Changed 6 years ago by
- Description modified (diff)
comment:19 Changed 6 years ago by
- Description modified (diff)
comment:20 Changed 6 years ago by
- Description modified (diff)
comment:21 follow-up: ↓ 23 Changed 6 years ago by
I don't like the phrasing "PyPI package". There's no such thing as a "PyPI package" per se. Let's be more clear about what we want here--should it be buildable from a source package? Or are we providing binary wheels? Both? Neither?
comment:22 follow-up: ↓ 24 Changed 6 years ago by
I wonder if this work might not also benefit from adopting portions of astropy-helpers into Sage's build tools. I don't think it would necessarily be appropriate to use directly--although there's nothing very "astropy" specific in astropy-helpers, a lot of it does assume one is using the astropy project template, or something close to it. But large swaths of it are very generic, and I put a lot of work into developing tools for build sanity of large Python projects.
Some key features that I think might benefit sage include:
- per-subpackage
setup_package.py
modules, which can contain any number of a (mostly) well-documented hooks for building compiled modules in that sub-package. This provides a clear and explicit alternative to a monolithic module_list.py (see the discussion here), and would also make it easier to break off some of Sage's subpackages into standalone packages should the time ever come for that. - one of the less-well-documented features is also that
setup_package.py
can contain hooks for setup.py commands. For example, if a module contains some generated code (a lasage_setup.autogen
), it can include apre_build_ext
hook to make sure the generated code is up to date before running the./setup.py build_ext
command (either explicitly, or implicitly as a sub-command). This makes it relatively easy for individual submodules to put hooks into the build process without having to write complicated command subclasses cluttered with subpackage-specific crud. - smarter handling of Cython modules--for example, when creating a source tarball it will make sure all Cython modules have been Cythonized, and it will include the resulting C/C++ sources in the source tarball. This ensures that the user does not need Cython to be installed in order to build from source (which is good, because even if they do have Cython it will often be the wrong version--and in Sage's case missing important patches as well).
Those are the main ones for now but I could probably think of more.
comment:23 in reply to: ↑ 21 Changed 6 years ago by
Replying to embray:
I don't like the phrasing "PyPI package". There's no such thing as a "PyPI package" per se. Let's be more clear about what we want here--should it be buildable from a source package? Or are we providing binary wheels? Both? Neither?
The scope of this ticket is a source package that is pip-installable and would be listed as such on PyPI.
Binary packaging (about which I don't know anything at the moment) is beyond the scope of this ticket.
comment:24 in reply to: ↑ 22 Changed 6 years ago by
Replying to embray:
I wonder if this work might not also benefit from adopting portions of astropy-helpers into Sage's build tools.
This all sounds great and seems like it could be part of #21508 or be a follow up to it.
comment:25 Changed 6 years ago by
- Description modified (diff)
- Summary changed from Make sagelib a PyPI package to Make sagelib a pip-installable Python source package, listed on PyPI
I've updated the description to explain the scope of the ticket better.
comment:26 Changed 6 years ago by
- Description modified (diff)
- Summary changed from Make sagelib a pip-installable Python source package, listed on PyPI to Task ticket: Make sagelib a pip-installable Python source package, listed on PyPI
comment:27 Changed 6 years ago by
- Description modified (diff)
comment:28 Changed 6 years ago by
- Description modified (diff)
comment:29 Changed 6 years ago by
- Description modified (diff)
comment:30 Changed 6 years ago by
- Cc infinity0 added
- Description modified (diff)
comment:31 follow-up: ↓ 32 Changed 6 years ago by
I have a request: currently the sage kernel for jupyter
is created directly in place on the file system rather than being "installed". Similarly there are 3 links created by the jupyter
kernel installation. That's not ideal from a distro point of view, and I am not sure that's acceptable from pip
point of view.
comment:32 in reply to: ↑ 31 Changed 6 years ago by
Replying to fbissey:
That's not ideal from a distro point of view, and I am not sure that's acceptable from
pip
point of view.
Can you elaborate why that is an issue?
comment:33 follow-ups: ↓ 34 ↓ 36 Changed 6 years ago by
Very simple. From a distro point of view (and good build system), you only build in a sandbox isolated from the main system. Then you stage your installation, if you are using autotools, by using DESTDIR, if using setup.py usually with --root. At no point you do something to the live system before the "merge", moving the staged install to the live file system.
I know we had some discussion when the current mechanic has been put in place as part of the install phase, unfortunately it doesn't obey --root. I will run a new check since it's been a while since I hit this.
Prior to 7.5.beta0 I have been patching to build kernel.json
and then install it with python_package_data. I have decided that the patching to setup.py
would be to invasive now and currently in 7.5.beta0+ I install manually at the package manager level.
comment:34 in reply to: ↑ 33 Changed 6 years ago by
Replying to fbissey:
Very simple. From a distro point of view (and good build system), you only build in a sandbox isolated from the main system. Then you stage your installation, if you are using autotools, by using DESTDIR, if using setup.py usually with --root. At no point you do something to the live system before the "merge", moving the staged install to the live file system.
I know we had some discussion when the current mechanic has been put in place as part of the install phase, unfortunately it doesn't obey --root. I will run a new check since it's been a while since I hit this.
Prior to 7.5.beta0 I have been patching to build
kernel.json
and then install it with python_package_data. I have decided that the patching tosetup.py
would be to invasive now and currently in 7.5.beta0+ I install manually at the package manager level.
Sorry, but I don't see how the above is an answer to my question.
comment:35 Changed 6 years ago by
...or maybe the problem is that the kernel spec is generated during the install phase.
Then the problem is not the fact that it is generated, but when it is generated.
comment:36 in reply to: ↑ 33 Changed 6 years ago by
Replying to fbissey:
Very simple. From a distro point of view (and good build system), you only build in a sandbox isolated from the main system. Then you stage your installation, if you are using autotools, by using DESTDIR, if using setup.py usually with --root.
Just to make me understand better, what are the commands that you would typically run to install a Python package this way?
comment:37 follow-up: ↓ 38 Changed 6 years ago by
Would be worse during the build phase
self.kernel_dir = os.path.join(JUPYTER_PATH, "kernels", self.identifier())
that's straight on the file system.
For the installation, instead of just doing python setup.py install
you do python setup.py install --root=$DESTDIR
. It's supposed to do the same thing than using DESTDIR
in autotool, everything will be installed as it normally should except that all the paths will be prefixed with DESTDIR.
comment:38 in reply to: ↑ 37 Changed 6 years ago by
Replying to fbissey:
For the installation, instead of just doing
python setup.py install
you dopython setup.py install --root=$DESTDIR
.
And why does it make a difference then whether the kernel spec files are generated or copied during that install
step? I am a bit lost here...
comment:39 Changed 6 years ago by
The problem is that it created directly on the file system rather than in DESTDIR
, not a problem for sage the distribution but a pain in the ass for the other people
>>> Install sage-9999 into /scratch2/portage/sci-mathematics/sage-9999/image/ category sci-mathematics * python2_7: running distutils-r1_run_phase distutils-r1_python_install /usr/bin/python2.7 setup.py install --root=/scratch2/portage/sci-mathematics/sage-9999/image/_python2.7 make: Nothing to be done for 'all'. .... .... byte-compiling /scratch2/portage/sci-mathematics/sage-9999/image/_python2.7/usr/lib64/python2.7/site-packages/sage/structure/all.py to all.pyc writing byte-compilation script '/scratch2/portage/sci-mathematics/sage-9999/temp/tmpoErZMu.py' /usr/bin/python2.7 -OO /scratch2/portage/sci-mathematics/sage-9999/temp/tmpoErZMu.py removing /scratch2/portage/sci-mathematics/sage-9999/temp/tmpoErZMu.py running install_egg_info Writing /scratch2/portage/sci-mathematics/sage-9999/image/_python2.7/usr/lib64/python2.7/site-packages/sage-9999-py2.7.egg-info error: [Errno 13] Permission denied: '/usr/share/jupyter/kernels/sagemath/kernel.json'
The correct jargon for this is "sandbox violation", the build process is not supposed to write on the live file system at this stage.
comment:40 Changed 6 years ago by
In that case, I would prefer to fix the problem by generating the files in the correct place. That should not be so hard, feel free to create a ticket for that.
comment:41 Changed 6 years ago by
I wish, --root
is not very well documented as far as I can see. DESTDIR
is quite easy we know the variable name. I am not sure there is a real equivalent variable for setup.py
, otherwise I would have filled a trivial ticket long ago. I'll look again.
comment:42 Changed 6 years ago by
- Description modified (diff)
comment:43 Changed 6 years ago by
- Description modified (diff)
comment:44 Changed 6 years ago by
- Description modified (diff)
comment:45 follow-up: ↓ 46 Changed 6 years ago by
I think the distinction of when or how the file is generated is mostly irrelevant, but fbissey is right that it should be generated in the right location, and not directly to JUPYTER_PATH.
comment:46 in reply to: ↑ 45 Changed 6 years ago by
Replying to embray:
I think the distinction of when or how the file is generated is mostly irrelevant, but fbissey is right that it should be generated in the right location, and not directly to JUPYTER_PATH.
That would be OK if it was done relative to the argument of --root
, you wouldn't know how to get to that Erik? I meant to look it up deeper but stuff is piling on me.
comment:47 Changed 6 years ago by
Yeah, I'd be willing to look into it. There should maybe be yet another ticket opened specifically for this issue.
comment:48 Changed 6 years ago by
- Cc thansen added
comment:49 Changed 6 years ago by
- Description modified (diff)
- Milestone changed from sage-7.5 to sage-7.6
comment:50 Changed 6 years ago by
- Description modified (diff)
comment:51 Changed 6 years ago by
- Description modified (diff)
comment:52 Changed 6 years ago by
- Cc defeo added
comment:53 Changed 4 years ago by
- Cc slelievre added
- Description modified (diff)
- Keywords pip PyPI added
There is interest in this process, see Ask Sage #40947: SageMath as a regular python package.
comment:55 Changed 4 years ago by
- Cc gh-timokau added
comment:56 Changed 3 years ago by
- Cc klui added
comment:57 Changed 3 years ago by
- Description modified (diff)
- Milestone changed from sage-7.6 to sage-9.1
comment:58 Changed 3 years ago by
I propose that we get this task ticket done in 2020.
comment:59 Changed 3 years ago by
- Description modified (diff)
comment:60 Changed 3 years ago by
- Description modified (diff)
comment:61 Changed 2 years ago by
The major problem I feel with making something like Sage into a python package is owing to the enormous complexity of Sagemath, the application itself has such major features , and I am not sure how much would we need to modify the application in order to help the proper incorporation of Sage into ordinary python code.
comment:62 follow-up: ↓ 63 Changed 2 years ago by
Yes, the complexity is why we use a meta-ticket to organize the necessary steps.
If you want to help, you can work on a well-defined ticket such as #21516.
comment:63 in reply to: ↑ 62 Changed 2 years ago by
comment:64 Changed 2 years ago by
- Milestone changed from sage-9.1 to sage-9.2
comment:65 Changed 2 years ago by
- Summary changed from Task ticket: Make sagelib a pip-installable Python source package, listed on PyPI to Meta-ticket: Make sagelib a pip-installable Python source package, listed on PyPI
comment:66 Changed 2 years ago by
- Cc isuruf jhpalmieri added
comment:67 Changed 2 years ago by
- Description modified (diff)
comment:68 Changed 22 months ago by
- Milestone changed from sage-9.2 to sage-9.3
comment:69 Changed 22 months ago by
- Description modified (diff)
- Milestone changed from sage-9.3 to sage-duplicate/invalid/wontfix
- Status changed from new to needs_review
Let's close this ticket. Meta-ticket #29705 describes the updated plan.
comment:70 Changed 22 months ago by
- Status changed from needs_review to positive_review
comment:71 Changed 22 months ago by
- Reviewers set to Dima Pasechnik
comment:72 Changed 14 months ago by
- Resolution set to invalid
- Status changed from positive_review to closed
Thanks for taking the lead on this. There's more still be to added to the list of issues, such as providing binary wheels and how that will be handled. I think it will be important to also make more progress on making some hard dependencies optional (that is, allow optional features to fail gracefully if a dependency for that feature is not found).