Opened 4 years ago

Closed 24 hours ago

#21559 closed enhancement (fixed)

Install src/bin scripts by sagelib's setup.py, not make

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-9.2
Component: build Keywords:
Cc: jdemeyer, vbraun, embray, isuruf, jhpalmieri, fbissey, kcrisman, mjo, arojas, dimpase Merged in:
Authors: Matthias Koeppe Reviewers: John Palmieri, François Bissey
Report Upstream: N/A Work issues:
Branch: 19d7f67 (Commits) Commit: 19d7f677eab07dc41e3f1e71b9a37f3b58560c39
Dependencies: #29111 Stopgaps:

Description (last modified by mkoeppe)

This ticket cleans up how the contents of src/bin are installed. One of its goals is to make the sagelib installation done by src/setup.py self-contained regarding the scripts that it needs at runtime.

Previous tickets moved some of the scripts to build/bin (no longer installed in SAGE_LOCAL).

In follow-up tickets, src/bin could be cleaned up more.

  • See the list of files added on this ticket to src/setup.py for beginnings of a categorization.
  • #29850: Install src/bin/sage-env-config as part of sage_conf
  • #29847: Install sage_setup with setuptools as a prerequisite for building/installing sagelib, install sage-the-distribution scripts as "scripts"

Change History (71)

comment:1 Changed 4 years ago by mkoeppe

  • Cc embray added
  • Description modified (diff)

comment:2 follow-up: Changed 4 years ago by embray

+1

Also sage-pip-install of #21441, but I might wait until that ticket is merged first, then move that script along with the others.

comment:3 in reply to: ↑ 2 Changed 4 years ago by jdemeyer

Replying to embray:

+1

Also sage-pip-install of #21441

I already moved that in #21441 (which still needs review by the way).

comment:4 Changed 4 years ago by mkoeppe

  • Description modified (diff)

comment:5 Changed 2 years ago by mkoeppe

  • Description modified (diff)
  • Milestone changed from sage-7.5 to sage-8.3

comment:6 Changed 2 years ago by vdelecroix

  • Milestone changed from sage-8.3 to sage-8.4

update milestone 8.3 -> 8.4

comment:7 Changed 14 months ago by mkoeppe

  • Description modified (diff)
  • Milestone changed from sage-8.4 to sage-8.8

comment:8 Changed 13 months ago by embray

  • Milestone sage-8.8 deleted

As the Sage-8.8 release milestone is pending, we should delete the sage-8.8 milestone for tickets that are not actively being worked on or that still require significant work to move forward. If you feel that this ticket should be included in the next Sage release at the soonest please set its milestone to the next release milestone (sage-8.9).

comment:9 Changed 6 months ago by mkoeppe

  • Cc isuruf added

comment:10 Changed 6 months ago by mkoeppe

  • Cc jhpamieri fbissey added

comment:11 Changed 6 months ago by mkoeppe

  • Cc jhpalmieri added; jhpamieri removed

comment:12 Changed 5 months ago by mkoeppe

  • Description modified (diff)
  • Milestone set to sage-9.1

comment:13 Changed 5 months ago by mkoeppe

  • Description modified (diff)

comment:14 Changed 5 months ago by mkoeppe

  • Cc kcrisman added
  • Description modified (diff)

comment:15 Changed 4 months ago by mkoeppe

  • Branch set to u/mkoeppe/change-src-bin-installation

comment:16 Changed 4 months ago by mkoeppe

  • Authors set to Matthias Koeppe
  • Commit set to 428ea11b2a4ac1c6754f9981d0b732e593e02aac
  • Description modified (diff)
  • Summary changed from Move sage-the-distribution scripts from src/bin to build/bin to Move sage-the-distribution scripts from src/bin to build/bin; install remaining src/bin scripts by setup.py, not make

New commits:

1a54a80Move src/bin/testc*.sh to build/bin/, delete src/bin/sage-README-osx.txt
428ea11Install src/bin/* by setup.py [scripts] instead of by build/make/deps

comment:17 Changed 4 months ago by git

  • Commit changed from 428ea11b2a4ac1c6754f9981d0b732e593e02aac to 1b3c860afdb5368545541e989975881a4c57ca1c

Branch pushed to git repo; I updated commit sha1. New commits:

1b3c860Move src/bin/sage-starts to build/bin/

comment:18 Changed 4 months ago by mkoeppe

  • Description modified (diff)
  • Status changed from new to needs_review

comment:19 Changed 4 months ago by mkoeppe

  • Description modified (diff)

comment:20 Changed 4 months ago by git

  • Commit changed from 1b3c860afdb5368545541e989975881a4c57ca1c to 546d1b07856ea236ad6a9021f21109b0082ca82d

Branch pushed to git repo; I updated commit sha1. New commits:

546d1b0Merge tag '9.1.beta7' into t/21559/change-src-bin-installation

comment:21 Changed 4 months ago by mkoeppe

  • Cc mjo added

comment:22 Changed 4 months ago by mkoeppe

This ticket could use some help from distribution packagers...

comment:23 Changed 4 months ago by git

  • Commit changed from 546d1b07856ea236ad6a9021f21109b0082ca82d to dddd4628522f62c7f78315b507116117a580912d

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

2850b50Move src/bin/testc*.sh to build/bin/, delete src/bin/sage-README-osx.txt
4466fd6Install src/bin/* by setup.py [scripts] instead of by build/make/deps
26b7596Move src/bin/sage-starts to build/bin/
dddd462Remove obsolete script sage-pkg (creates old-style spkg archives)

comment:24 Changed 4 months ago by git

  • Commit changed from dddd4628522f62c7f78315b507116117a580912d to 9825799e82c8201359faf2e7db103b9f52ba2aef

Branch pushed to git repo; I updated commit sha1. New commits:

9825799src/bin/sage-unzip: Remove (unused script)

comment:25 Changed 4 months ago by mkoeppe

  • Description modified (diff)
Last edited 4 months ago by mkoeppe (previous) (diff)

comment:26 Changed 4 months ago by mkoeppe

  • Status changed from needs_review to needs_work

comment:27 Changed 4 months ago by mkoeppe

  • Dependencies set to #23769, #29381, #29383

comment:28 Changed 4 months ago by mkoeppe

  • Description modified (diff)

comment:29 Changed 4 months ago by mkoeppe

  • Description modified (diff)

comment:30 Changed 4 months ago by mkoeppe

  • Cc arojas added
  • Description modified (diff)

comment:31 Changed 4 months ago by arojas

This breaks if you try to install sagelib with setup.py without running configure first (as some distros do) since sage-env-config doesn't exist in that case. At least sage-env and sage-env-config should be optional.

comment:32 follow-up: Changed 4 months ago by gh-Shlokatadistance

Can this build also configure a better link or communication between SAGE_LOCAL or SAGE_SCRIPTS_DIR variables?

comment:33 in reply to: ↑ 32 Changed 4 months ago by mkoeppe

Replying to gh-Shlokatadistance:

Can this build also configure a better link or communication between SAGE_LOCAL or SAGE_SCRIPTS_DIR variables?

Take a look at #25486

comment:34 Changed 3 months ago by mkoeppe

  • Milestone changed from sage-9.1 to sage-9.2

comment:35 Changed 4 weeks ago by mkoeppe

  • Dependencies changed from #23769, #29381, #29383 to #29702

comment:36 Changed 4 weeks ago by git

  • Commit changed from 9825799e82c8201359faf2e7db103b9f52ba2aef to 7c8a0304aff5c2f6376fa7c4b4acf56d35766f31

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

5db5318Trac #29345: remove "break" statements from AC_SEARCH_LIBS.
e810ad1Trac #29345: don't use sage's config.status for the lrcalc build.
93c9921Trac #29345: replace the function that populates the CVXOPT_* variables.
0e66a0aTrac #29345: add Dima's SPKG patches for ksh compatibility.
df3f05ebuild/make/Makefile.in [SCRIPT_PACKAGE_templ]: cd into the SPKG directory; adjust spkg-install scripts
5372065Merge branch 't/29793/script_packages_should_cd_into_the_spkg_directory' into t/29411/make_sagelib_a_script_package
c166b97Merge branch 't/29411/make_sagelib_a_script_package' into t/29702/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup
cc30471build/bin/write-dockerfile.sh: Do not ADD removed file src/Makefile.in
8a41326Merge branch 't/29411/make_sagelib_a_script_package' into t/29702/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup
7c8a030Install src/bin/* by setup.py [scripts] instead of by build/make/Makefile.in

comment:37 Changed 4 weeks ago by mkoeppe

  • Description modified (diff)
  • Summary changed from Move sage-the-distribution scripts from src/bin to build/bin; install remaining src/bin scripts by setup.py, not make to Install src/bin scripts by sagelib's setup.py, not make

comment:38 Changed 4 weeks ago by git

  • Commit changed from 7c8a0304aff5c2f6376fa7c4b4acf56d35766f31 to 64ec6ae70a8d10ebec7158eecebff996592d273b

Branch pushed to git repo; I updated commit sha1. New commits:

64ec6aesrc/setup.py [scripts]: Categorize scripts

comment:39 Changed 4 weeks ago by mkoeppe

  • Status changed from needs_work to needs_review

comment:40 Changed 4 weeks ago by git

  • Commit changed from 64ec6ae70a8d10ebec7158eecebff996592d273b to 1ee054c0da2494f6bdbe5d4bd6088e0d39c0d3f8

Branch pushed to git repo; I updated commit sha1. New commits:

1ee054cFixup

comment:41 Changed 3 weeks ago by git

  • Commit changed from 1ee054c0da2494f6bdbe5d4bd6088e0d39c0d3f8 to daa045c01c6ae4767213abf9dcc1820829200999

Branch pushed to git repo; I updated commit sha1. New commits:

daa045csrc/setup.py: Do not install deprecated script bin/sage-fix-pkg-checksums

comment:42 Changed 3 weeks ago by mkoeppe

(this script is deleted in #29111)

comment:43 Changed 3 weeks ago by mkoeppe

  • Cc dimpase added

comment:44 Changed 3 weeks ago by mkoeppe

  • Dependencies changed from #29702 to #29702, #29111, #29920

comment:45 Changed 3 weeks ago by git

  • Commit changed from daa045c01c6ae4767213abf9dcc1820829200999 to e3d9cf23fcce51f679878b86ce462d91d3c41ea5

Branch pushed to git repo; I updated commit sha1. New commits:

e3d9cf2src/setup.py: Do not install deprecated scripts sage-list-{experimental,optional,standard}

comment:46 Changed 3 weeks ago by mkoeppe

(these scripts are deleted in #29920)

comment:47 follow-up: Changed 3 weeks ago by fbissey

A bit much to ask but is there a way to override the scripts variable in setup.py? Apart from patching that is. I and other distros will pick and choose what we install and certainly, even where. We can do things after install is run if needs be.

But building and providing our own list would be a nice to have.

comment:48 in reply to: ↑ 47 Changed 3 weeks ago by mkoeppe

Replying to fbissey:

A bit much to ask but is there a way to override the scripts variable in setup.py? Apart from patching that is. I and other distros will pick and choose what we install and certainly, even where. We can do things after install is run if needs be.

But building and providing our own list would be a nice to have.

Adding hooks for distributors is not a good direction for development. It's better to sort out in sagelib itself which scripts should be considered part of sagelib, and which ones are provided by sage-the-distribution instead.

comment:49 follow-ups: Changed 3 weeks ago by mkoeppe

This can be done on follow-up tickets of this ticket. You can help by categorizing the scripts in the section "Uncategorized scripts in alphabetical order".

comment:50 in reply to: ↑ 49 Changed 3 weeks ago by fbissey

Replying to mkoeppe:

This can be done on follow-up tickets of this ticket. You can help by categorizing the scripts in the section "Uncategorized scripts in alphabetical order".

I'll think about it. Like I said it is a "nice to have", not a must have. But this will mark a transition for distros. Before this, most of us were manually installing a selection of scripts. Suddenly the install won't be manual anymore :)

comment:51 Changed 3 weeks ago by mkoeppe

In fact, the scripts that belong to sage-the-distribution could perhaps be assigned to sage_setup (#29847) -- this distutils package that would be installed by sage-the-distribution but not by distros.

comment:52 in reply to: ↑ 49 ; follow-up: Changed 3 weeks ago by jhpalmieri

Replying to mkoeppe:

This can be done on follow-up tickets of this ticket. You can help by categorizing the scripts in the section "Uncategorized scripts in alphabetical order".

Here are some thoughts:

Perhaps could just be deleted:

  • sage-cython: #29923
  • sage-native-execute: look at the script and see #9386.
  • sage-open: I wonder if the comment in that script is still valid; perhaps it can be deleted, or done purely in Python in sage/misc/viewer.py, the only place it's used. Maybe I'll look into this.
  • sage-pypkg-location: apparently unused
  • sage-unzip is in the listed, but the script itself is gone, isn't it?

Doctesting-related scripts. Where do these go?

  • sage-runtests
  • coverage scripts
  • sage-fix-doctests
  • sage-startuptime

In the libexec group:

  • sage-cleaner?

Only relevant for sage-the-distribution:

  • sage-download
  • sage-clone-source
  • sage-sdist
  • sage-location

comment:53 in reply to: ↑ 52 Changed 3 weeks ago by mkoeppe

Replying to jhpalmieri:

Doctesting-related scripts. Where do these go?

  • sage-runtests
  • coverage scripts
  • sage-fix-doctests

Per #29111, they are sagelib scripts.

  • sage-startuptime

Not sure about this one.

comment:54 Changed 3 weeks ago by git

  • Commit changed from e3d9cf23fcce51f679878b86ce462d91d3c41ea5 to 455c294091537ca92df0bfbf39fae38816c808bf

Branch pushed to git repo; I updated commit sha1. New commits:

455c294src/setup.py: Do not install removed script bin/sage-unzip

comment:55 Changed 3 weeks ago by git

  • Commit changed from 455c294091537ca92df0bfbf39fae38816c808bf to b214410e12972a06e4cab87b3225d3d8aac4db7b

Branch pushed to git repo; I updated commit sha1. New commits:

b214410Classify doctesting-related scripts

comment:56 Changed 3 weeks ago by git

  • Commit changed from b214410e12972a06e4cab87b3225d3d8aac4db7b to 8d10bb99f339795601993cb8b2899f544471fd68

Branch pushed to git repo; I updated commit sha1. New commits:

8d10bb9Categorize some uncategorized scripts

comment:57 Changed 3 weeks ago by git

  • Commit changed from 8d10bb99f339795601993cb8b2899f544471fd68 to 6b04075aca84249a2418a365f31ea4a0978461a8

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

05ba86aMerge branch 't/29884/src_doc_bootstrap__generate_src_doc_en_reference_repl_options_rst' into t/29111/specify_a_subset_of_sage_command_line_options_that_are_supported_by_sagelib___rather_than_sage_the_distribution
72c7e23src/doc/en/reference/repl/options.rst: Replace copypasta by include of generated file options.txt
e9a883dtrac 29111: revising "sage --advanced" message.
5dca421trac 29111: more reorganization of "sage --advanced" message
8f0ee05trac 29111: delete sage-fix-pkg-checksums
21fc231trac 29111: re "sage --advanced" message:
875940dtrac 29111: more tinkering. Fix doctests in tests/cmdline.py.
4a3d36eMove 'sage -app' back to src/bin/sage
3a0193csrc/bin/sage: Remove handling of 'sage -axiom'
6b04075Merge branch 't/29111/specify_a_subset_of_sage_command_line_options_that_are_supported_by_sagelib___rather_than_sage_the_distribution' into t/21559/change-src-bin-installation

comment:58 Changed 3 weeks ago by git

  • Commit changed from 6b04075aca84249a2418a365f31ea4a0978461a8 to 831cc09d32828144a27f7bc8c8b2f70150ed0047

Branch pushed to git repo; I updated commit sha1. New commits:

9c7116bsrc/bin/sage-list-optional, sage-list-experimental, sage-list-standard: Remove deprecated scripts
831cc09Merge branch 't/29920/remove_deprecated_scripts_sage_list_optional__sage_list_experimental__sage_list_standard' into t/21559/change-src-bin-installation

comment:59 Changed 3 weeks ago by git

  • Commit changed from 831cc09d32828144a27f7bc8c8b2f70150ed0047 to e3eca857db71b602cf2c7c16755ab2cdeaca1773

Branch pushed to git repo; I updated commit sha1. New commits:

a56dc35Merge tag '9.2.beta1' into t/29702/public/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup
e3eca85Merge branch 'public/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup' of git://trac.sagemath.org/sage into t/21559/change-src-bin-installation

comment:60 Changed 2 weeks ago by jhpalmieri

I run into problems unless I make this change:

  • src/setup.py

    diff --git a/src/setup.py b/src/setup.py
    index 9970525cf9..85a32a5962 100755
    a b code = setup(name = 'sage', 
    162162                 'bin/sage-rebaseall.sh',
    163163                 'bin/sage-rst2sws',
    164164                 'bin/sage-rst2txt',
    165                  'bin/sage-rsyncdist',
    166165                 'bin/sage-run',
    167166                 'bin/sage-run-cython',
    168167                 'bin/sage-startuptime.py',

comment:61 Changed 2 weeks ago by git

  • Commit changed from e3eca857db71b602cf2c7c16755ab2cdeaca1773 to 7d29141662fe9ffd80c61e8867153244004d0a37

Branch pushed to git repo; I updated commit sha1. New commits:

7d29141src/setup.py: Do not install removed script sage-rsyncdist

comment:62 Changed 2 weeks ago by mkoeppe

  • Description modified (diff)

comment:63 Changed 12 days ago by mkoeppe

Needs review

comment:64 Changed 12 days ago by jhpalmieri

  • Reviewers set to John Palmieri, François Bissey
  • Status changed from needs_review to positive_review

Looks okay to me.

comment:65 Changed 12 days ago by mkoeppe

Thanks!

comment:66 Changed 7 days ago by git

  • Commit changed from 7d29141662fe9ffd80c61e8867153244004d0a37 to c5f7f2fb29d376c6bb103e2f070f92690c689161
  • Status changed from positive_review to needs_review

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

c5f7f2fMerge tag '9.2.beta3' into t/21559/change-src-bin-installation

comment:67 Changed 7 days ago by mkoeppe

  • Status changed from needs_review to positive_review

comment:68 Changed 3 days ago by git

  • Commit changed from c5f7f2fb29d376c6bb103e2f070f92690c689161 to 19d7f677eab07dc41e3f1e71b9a37f3b58560c39
  • Status changed from positive_review to needs_review

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

3953671Merge tag '9.2.beta4' into t/29111/specify_a_subset_of_sage_command_line_options_that_are_supported_by_sagelib___rather_than_sage_the_distribution
19d7f67Merge branch 't/29111/specify_a_subset_of_sage_command_line_options_that_are_supported_by_sagelib___rather_than_sage_the_distribution' into t/21559/change-src-bin-installation

comment:69 Changed 3 days ago by mkoeppe

  • Dependencies changed from #29702, #29111, #29920 to #29111

comment:70 Changed 3 days ago by mkoeppe

  • Status changed from needs_review to positive_review

comment:71 Changed 24 hours ago by vbraun

  • Branch changed from u/mkoeppe/change-src-bin-installation to 19d7f677eab07dc41e3f1e71b9a37f3b58560c39
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.