Opened 8 years ago

Closed 3 months ago

#13190 closed enhancement (invalid)

make sagelib use setuptools instead of distutils

Reported by: ohanar Owned by: GeorgSWeber
Priority: major Milestone: sage-duplicate/invalid/wontfix
Component: build Keywords:
Cc: kini, jdemeyer, robertwb, kcrisman, mhansen, jhpalmieri, embray, dimpase, fbissey Merged in:
Authors: Reviewers: Dima Pasechnik
Report Upstream: N/A Work issues:
Branch: u/mkoeppe/make_sagelib_use_setuptools_instead_of_distutils (Commits) Commit: 93bbbfe75f162368247a04d0023e95628ef8ec13
Dependencies: #29702, #29803 Stopgaps:

Description (last modified by mkoeppe)

Currently the sage library uses distutils. Switching to setuptools will give us the setup.py develop command, which bypasses the need to rebuild the sage library when modifying python files.

See also: #29845 - PEP 517 buildapi for sage_setup


References:

https://stackoverflow.com/questions/3779915/why-does-python-setup-py-sdist-create-unwanted-project-egg-info-in-project-r

Attachments (3)

trac13190_root.patch (1.1 KB) - added by ohanar 8 years ago.
apply to root repo
trac13190_scripts.patch (2.6 KB) - added by ohanar 8 years ago.
apply to scripts repo
trac13190_library.patch (22.6 KB) - added by ohanar 8 years ago.
apply to sage library

Download all attachments as: .zip

Change History (42)

comment:1 Changed 8 years ago by ohanar

  • Cc kini jdemeyer robertwb kcrisman added
  • Dependencies set to #13031
  • Status changed from new to needs_review

comment:2 Changed 8 years ago by ohanar

  • Cc mhansen added

comment:3 follow-up: Changed 8 years ago by mhansen

I would imagine that you get problems similar to #12659 . Have you tried this out?

comment:4 in reply to: ↑ 3 Changed 8 years ago by ohanar

Replying to mhansen:

I would imagine that you get problems similar to #12659 . Have you tried this out?

Do you mean the sdist issues? No, I have not tried them yet, but I wouldn't doubt that there may be issues with them. Also, as noted in deps, there is an issue with parallel instances of setuptools, so either deps has to be hacked to force sage into the serial build, or we depend on #12994 (my vote is the later, since it is cleaner).

comment:5 Changed 8 years ago by ohanar

I've decided to make #12994 a dependency -- makes it much easier to add setuptool packages to the build system.

comment:6 Changed 8 years ago by ohanar

  • Dependencies changed from #13031 to #12994, #13031

comment:7 Changed 8 years ago by ohanar

  • Dependencies changed from #12994, #13031 to #12994, #13031, #13197

There are some issues with using relative paths with setuptools (which we want for relocation purposes). #13197 fixes these issues.

comment:8 Changed 8 years ago by ohanar

  • Dependencies changed from #12994, #13031, #13197 to #11874, #12994, #13031, #13197

comment:9 follow-up: Changed 8 years ago by jdemeyer

I am a bit lost on the inter-relations between the tickets #12994, #13190, #13201, #13197 (and perhaps #12659). Could you please clarify the dependencies?

Also, I personally think it would be better to split this in two tickets:

  1. Use setuptools
  2. Use setup.py develop
Last edited 8 years ago by jdemeyer (previous) (diff)

comment:10 in reply to: ↑ 9 Changed 8 years ago by ohanar

Replying to jdemeyer:

I am a bit lost on the inter-relations between the tickets #12994, #13190, #13201, #13197 (and perhaps #12659). Could you please clarify the dependencies?

Also, I personally think it would be better to split this in two tickets:

  1. Use setuptools

This should only require #13031.

  1. Use setup.py develop

This is more or less #12659 after switching to setuptools, and is where the other dependencies creep up (they aren't strictly necessary, but the alternative is to hack around setuptools, which is less preferable than patching it IMO).

I'll rework the patch to only do A once we have a 5.4 beta (mainly because #13031 more or less needs to be rebased with every development release).

Changed 8 years ago by ohanar

apply to root repo

Changed 8 years ago by ohanar

apply to scripts repo

comment:11 Changed 8 years ago by ohanar

  • Dependencies changed from #11874, #12994, #13031, #13197 to #11874, #12994, #13031

ok, rebased and only does the switch to setuptools (doesn't switch to setup.py develop anymore). Dependencies are now accurate.

Changed 8 years ago by ohanar

apply to sage library

comment:12 Changed 7 years ago by jdemeyer

  • Milestone changed from sage-5.11 to sage-5.12

comment:13 Changed 7 years ago by roed

  • Status changed from needs_review to needs_info

Are we trying to make this happen before the git transition or afterward?

comment:14 Changed 7 years ago by ohanar

Certainly after, although I'm not currently convinced that we should proceed in this direction.

comment:15 Changed 7 years ago by vbraun_spam

  • Milestone changed from sage-6.1 to sage-6.2

comment:16 Changed 7 years ago by vbraun_spam

  • Milestone changed from sage-6.2 to sage-6.3

comment:17 Changed 6 years ago by vbraun_spam

  • Milestone changed from sage-6.3 to sage-6.4

comment:18 Changed 5 years ago by embray

Any thoughts on this? It hasn't been touched in quite some time despite there being (a likely outdated by now) patch. I can take this issue over and see it to completion if desired.

comment:19 follow-up: Changed 5 years ago by jdemeyer

There is at least one problem: setuptools does not support setup(data_files=...), see setuptools #130. It's a good reason to avoid setuptools.

I'm not sure that we need data_files for the Sage library though. Maybe we could use package_data instead. But then that is something which should be investigated.

comment:20 Changed 5 years ago by jdemeyer

  • Dependencies #11874, #12994, #13031 deleted

comment:21 Changed 5 years ago by embray

I don't know what files need to be installed with the sage package. For the majority of Python-based projects package_data is preferred, but of course Sage is 'special'.

Let me look into what files are installed with the sage package, and I can worry about the details.

comment:22 Changed 5 years ago by jdemeyer

Hang on, I'm looking at it.

comment:23 Changed 5 years ago by jdemeyer

  • Dependencies set to #20108
  • Milestone changed from sage-6.4 to sage-7.2

I created a new ticket for using package_data.

comment:24 in reply to: ↑ 19 Changed 2 years ago by jdemeyer

Replying to jdemeyer:

There is at least one problem: setuptools does not support setup(data_files=...), see setuptools #130. It's a good reason to avoid setuptools.

I recently learned about the (almost undocumented) setuptools options --old-and-unmanageable. It forces a non-egg build which is exactly what we want. So that would make this ticket possible again.

comment:25 Changed 2 years ago by jdemeyer

  • Dependencies #20108 deleted

comment:26 Changed 2 years ago by jdemeyer

Also, it seems that Sage is no longer using either data_files or package_data. So that discussion has also disappeared.

comment:27 follow-ups: Changed 2 years ago by embray

I forget, what is the difference between --old-and-unmanageable and --single-version-externally-managed, where I thought the latter should be sufficient...

Also I do want to use package_data for Sage (see e.g. #21732) but I don't see what the problem is.

comment:28 in reply to: ↑ 27 Changed 2 years ago by jdemeyer

Replying to embray:

Also I do want to use package_data for Sage (see e.g. #21732) but I don't see what the problem is.

There is no problem with package_data if you want to use it in the way that it was intended to be used (which may not align with actual use cases).

There is a problem with data_files: setuptools still doesn't really support that. One of the reasons that this ticket stalled was because Sage was using data_files at the time to install Cython-related files. You suggested that it would be easy to use package_data instead (#20108). That didn't actually work, but now we are using custom code (neither data_files not package_data, see #21600) to install those files.

Last edited 2 years ago by jdemeyer (previous) (diff)

comment:29 in reply to: ↑ 27 Changed 2 years ago by jdemeyer

Replying to embray:

I forget, what is the difference between --old-and-unmanageable and --single-version-externally-managed

Mainly that the latter requires a --record argument also.

comment:30 Changed 6 months ago by mkoeppe

  • Authors R. Andrew Ohana deleted
  • Cc jhpalmieri embray dimpase fbissey added
  • Milestone changed from sage-7.2 to sage-9.2
  • Summary changed from make sage use setuptools instead of distutils to make sagelib use setuptools instead of distutils

Time to revive this ancient ticket.

comment:31 Changed 6 months ago by mkoeppe

  • Status changed from needs_info to needs_work

comment:32 Changed 6 months ago by mkoeppe

  • Dependencies set to #29702

comment:33 Changed 6 months ago by mkoeppe

  • Branch set to u/mkoeppe/make_sagelib_use_setuptools_instead_of_distutils

comment:34 Changed 6 months ago by mkoeppe

  • Commit set to 93bbbfe75f162368247a04d0023e95628ef8ec13
  • Description modified (diff)

Last 10 new commits:

0a61795Trac #29345: don't force SHELL=bash any longer.
5db5318Trac #29345: remove "break" statements from AC_SEARCH_LIBS.
e810ad1Trac #29345: don't use sage's config.status for the lrcalc build.
93c9921Trac #29345: replace the function that populates the CVXOPT_* variables.
0e66a0aTrac #29345: add Dima's SPKG patches for ksh compatibility.
df3f05ebuild/make/Makefile.in [SCRIPT_PACKAGE_templ]: cd into the SPKG directory; adjust spkg-install scripts
5372065Merge branch 't/29793/script_packages_should_cd_into_the_spkg_directory' into t/29411/make_sagelib_a_script_package
c166b97Merge branch 't/29411/make_sagelib_a_script_package' into t/29702/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup
5f33436src/setup.py: from setuptools import setup
93bbbfeAdd src/setup.cfg

comment:35 Changed 6 months ago by mkoeppe

  • Dependencies changed from #29702 to #29702, #29803

comment:36 Changed 6 months ago by mkoeppe

  • Description modified (diff)

comment:37 Changed 5 months ago by mkoeppe

  • Milestone changed from sage-9.2 to sage-duplicate/invalid/wontfix
  • Status changed from needs_work to needs_review

This ticket is superseded by #29950 and can be closed.

comment:38 Changed 5 months ago by dimpase

  • Reviewers set to Dima Pasechnik
  • Status changed from needs_review to positive_review

comment:39 Changed 3 months ago by chapoton

  • Resolution set to invalid
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.