Opened 2 years ago

Closed 21 months ago

#30383 closed enhancement (fixed)

Add "configure --disable-notebook"; show descriptions of optional packages in "configure --help"

Reported by: Matthias Köppe Owned by:
Priority: critical Milestone: sage-9.3
Component: build Keywords: sd111
Cc: John Palmieri, Michael Orlitzky, Samuel Lelièvre, Sébastien Labbé, Dima Pasechnik, Erik Bray, gh-kliem Merged in:
Authors: Matthias Koeppe Reviewers: John Palmieri, Dima Pasechnik
Report Upstream: N/A Work issues:
Branch: 4916415 (Commits, GitHub, GitLab) Commit: 4916415408a58e4b715143e5c607972822055790
Dependencies: Stopgaps:

Status badges

Description (last modified by Matthias Köppe)

We refactor the code in sage_spkg_enable.m4 and sage_spkg_collect.m4 so that it becomes possible to add configure --disable-SPKG options to disable standard packages. This also simplifies build/make/Makefile.in slightly because it no longer has to make a distinction between standard and optional packages.

We demonstrate this by adding one such option, configure --disable-notebook, which disables building the Jupyter notebook package ... and all of its exclusive dependencies. This is useful for people who want to use the system Jupyter notebook -- the Jupyter kernel is still built and can be installed there.

The new option appears before the --enable... options for optional packages.

We also make the configure help a bit more informative (by including the 1-line descriptions of the optional packages) and prettier.

$ ./configure --help
...
Optional Features:
  --disable-option-checking  ignore unrecognized --enable/--with options
  --disable-FEATURE       do not include FEATURE (same as --enable-FEATURE=no)
  --enable-FEATURE[=ARG]  include FEATURE [ARG=yes]
...
  --enable-experimental-packages
                          allow installing experimental packages (default: no
                          = ask for user confirmation for each package)
  --enable-download-from-upstream-url
                          allow downloading packages from their upstream URL
                          if they cannot be found on the Sage mirrors
  --disable-notebook      disable build of the Jupyter notebook and related
                          packages
  --enable-4ti2={no|if_installed (default)|yes}
                          enable build and use of the optional package 4ti2: Algebraic, geometric
                          and combinatorial problems on linear spaces
                          * package info: ./sage -info 4ti2
  --disable-4ti2          disable build and uninstall if previously installed by Sage in PREFIX;
                          same as --enable-4ti2=no
  --enable-atlas={no|if_installed (default)|yes}
                          enable build and use of the optional package atlas: Automatically Tuned
                          Linear Algebra Software (BLAS implementation)
                          * package info: ./sage -info atlas
  --disable-atlas         disable build and uninstall if previously installed by Sage in PREFIX;
                          same as --enable-atlas=no
  --enable-awali={no|if_installed (default)|yes}
                          enable build and use of the experimental package awali: Computation of/with
                          finite state machines
                          * package info: ./sage -info awali
  --disable-awali         disable build and uninstall if previously installed by Sage in PREFIX;
                          same as --enable-awali=no

This is also preparation for #30556 (packages that will not work without openssl), and for testing modularized installs (#30778, #29864).

Change History (51)

comment:1 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:2 Changed 2 years ago by Matthias Köppe

Cc: Michael Orlitzky Samuel Lelièvre added

comment:3 Changed 2 years ago by Matthias Köppe

Suggestions for a better name for this new package type are welcome

comment:4 Changed 2 years ago by Matthias Köppe

Alternatively, we could also make it possible to disable any standard package.

comment:5 Changed 2 years ago by Matthias Köppe

Description: modified (diff)
Summary: New package type: optional-enabled-by-default"configure --disable-SPKG" for standard packages

comment:6 Changed 2 years ago by Matthias Köppe

Dependencies: #29363

comment:7 Changed 2 years ago by Matthias Köppe

Branch: u/mkoeppe/new_package_type__optional_enabled_by_default

comment:8 Changed 2 years ago by Matthias Köppe

Authors: Matthias Koeppe
Commit: 182c1fd756d75029e2d1885df68af441a56cdac1

New commits:

c73460cAdd options 'configure --disable-SPKG' for standard packages
22d7ae4m4/sage_spkg_*.m4: At the end of configure, indicate which optional/experimental packages are configured to be installed
bd6c109Merge branch 't/29363/at_the_end_of_configure__indicate_which_optional_experimental_packages_are_configured_to_be_installed' into t/30383/new_package_type__optional_enabled_by_default
3c98700Replace SAGE_STANDARD_PACKAGES by use of SAGE_OPTIONAL_INSTALLED_PACKAGES
182c1fdm4/sage_spkg_collect.m4: Reduce verbosity by removing 'does not support check for system package'

comment:9 Changed 2 years ago by Matthias Köppe

Status: newneeds_review

comment:10 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:11 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:12 Changed 2 years ago by Michael Orlitzky

Doesn't this create a bajillion ./configure options that basically do nothing (because the packages are dependencies), and another bajillion that cause sage to function improperly (test suite failures, etc.)?

I think the original approach was more on-track, although I wouldn't have created a separate package type for it (we don't need to proliferate a new package type throughout the build system every time we tweak a bit of metadata). Instead I would have added an "enabled by default" (default: no) feature somewhere to the optional packages. Long-term, a way to control (and depend on) optional features is needed, but that's a much larger undertaking akin to USE flags in Gentoo.

comment:13 in reply to:  12 Changed 2 years ago by Matthias Köppe

Replying to mjo:

Doesn't this create a bajillion ./configure options that basically do nothing (because the packages are dependencies), and another bajillion that cause sage to function improperly (test suite failures, etc.)?

Yes, a gazillion of "advanced options"

comment:14 Changed 2 years ago by Matthias Köppe

Most "standard" packages are not really required in any strict sense other than passing doctests that test particular features. Making it possible to disable them gives users a remedy when the build fails on a system

comment:15 Changed 2 years ago by Matthias Köppe

Keywords: sd111 added

Hoping we can make progress on this ticket this week - https://wiki.sagemath.org/days111

comment:16 Changed 2 years ago by git

Commit: 182c1fd756d75029e2d1885df68af441a56cdac1e64ef1a6c7436dd515c1daf968fe00ed6d1f9f71

Branch pushed to git repo; I updated commit sha1. New commits:

e64ef1aMerge tag '9.3.beta4' into t/30383/new_package_type__optional_enabled_by_default

comment:17 Changed 2 years ago by Matthias Köppe

Cc: Sébastien Labbé added

comment:18 Changed 21 months ago by git

Commit: e64ef1a6c7436dd515c1daf968fe00ed6d1f9f71af2e1fb38eab589e47d3cfcf841054fb9edbd43d

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

ea182d7Copy changes from build/pkgs/sagelib/src to src
a1a10b9src/VERSION.txt: New
5697335src/setup.cfg: Add license_file=LICENSE.txt
deb9eb3Merge tag '9.3.beta3' into t/30912/sagelib__update_metadata_for_pypi_deployment
7ad4c0eMerge tag '9.3.beta4' into t/30912/sagelib__update_metadata_for_pypi_deployment
27b589bMerge branch 't/30912/sagelib__update_metadata_for_pypi_deployment' into t/31362/make_all_sage_local__make_all_sage_venv
8796008Put pynac into SAGE_LOCAL, sagelib into SAGE_VENV
d6831b1build/make/Makefile.in: Add all-build-local, all-build-venv, which include dependency on toolchain
61f6ba6Makefile: Add top-level targets build-local, build-venv
af2e1fbMerge #31362

comment:19 Changed 21 months ago by Matthias Köppe

Dependencies: #29363#31362

comment:20 Changed 21 months ago by Matthias Köppe

Status: needs_reviewneeds_work

comment:21 Changed 21 months ago by git

Commit: af2e1fb38eab589e47d3cfcf841054fb9edbd43d340fbb719da34088eaa3e534a62795121431bdff

Branch pushed to git repo; I updated commit sha1. New commits:

340fbb7bootstrap: Emit SAGE_SPKG_ENABLE calls for standard packages

comment:22 Changed 21 months ago by git

Commit: 340fbb719da34088eaa3e534a62795121431bdff50262a4cba507052b99932813018f291183db856

Branch pushed to git repo; I updated commit sha1. New commits:

47e83d8Move computation of SAGE_OPTIONAL_{INSTALLED,CLEANED}_PACKAGES to sage_spkg_collect.m4
affcad3bootstrap: Emit initialization of SAGE_ENABLE_... variables for all packages; do not call SAGE_SPKG_ENABLE for standard packages
f89c506build/make/Makefile.in: Restore variable OPTIONAL_INSTALLED_PACKAGES
50262a4Include short package description in help string for configure --enable-SPKG, improve formatting

comment:23 Changed 21 months ago by git

Commit: 50262a4cba507052b99932813018f291183db8567e22ff01285b337468802f5af6a45f125db02ac0

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

8397c74Include short package description in help string for configure --enable-SPKG, improve formatting
7e22ff0WIP

comment:24 Changed 21 months ago by git

Commit: 7e22ff01285b337468802f5af6a45f125db02ac00a83740c1b3d12dc27a2e3807b7769e42906a2bf

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

0a83740Initialize SAGE_ENABLE_... properly, add 'configure --disable-notebook'

comment:25 Changed 21 months ago by Matthias Köppe

Dependencies: #31362#31362, #31278
Description: modified (diff)
Status: needs_workneeds_review
Summary: "configure --disable-SPKG" for standard packagesAdd "configure --disable-notebook"

comment:26 Changed 21 months ago by git

Commit: 0a83740c1b3d12dc27a2e3807b7769e42906a2bf82d8030c3c71bb9e5503f77ae42484de47925381

Branch pushed to git repo; I updated commit sha1. New commits:

4146d88configure --disable-notebook: Do not disable widgetsnbextension
a8035aabuild/pkgs/ipywidgets: Upgrade to 7.6.3
5141bc3build/pkgs/widgetsnbextension: Patch out dependency on notebook (backport from widgetsnbextension-4)
fee8379Merge #31278
82d8030m4/sage_spkg_enable.m4: Cosmetic change to help strings

comment:27 Changed 21 months ago by Matthias Köppe

Status: needs_reviewneeds_work

comment:28 Changed 21 months ago by git

Commit: 82d8030c3c71bb9e5503f77ae42484de479253813e929982937143bae88f66917870d39979cc3551

Branch pushed to git repo; I updated commit sha1. New commits:

3e92998configure --disable-notebook: Fix up what packages are disabled

comment:29 Changed 21 months ago by Matthias Köppe

Cc: Dima Pasechnik added
Status: needs_workneeds_review

This version of the ticket no longer offers --disable... options for all standard packages, which was criticized previously.

comment:30 Changed 21 months ago by Matthias Köppe

Description: modified (diff)
Summary: Add "configure --disable-notebook"Add "configure --disable-notebook"; show descriptions of optional packages in "configure --help"

comment:31 Changed 21 months ago by John Palmieri

Why the change to matplotlib dependencies? Also the change in line 66 of m4/sage_spkg_configure.m4 renders very strangely (strange font for "default") when I click on the branch here for example or view the diff on my computer. Any ideas what that's about?

comment:32 Changed 21 months ago by git

Commit: 3e929982937143bae88f66917870d39979cc355174a83fce03916650dd1574ee04c7f3a3dc6d5848

Branch pushed to git repo; I updated commit sha1. New commits:

74a83fcbuild/pkgs/matplotlib/dependencies: Undo removal of tornado

comment:33 in reply to:  31 Changed 21 months ago by Matthias Köppe

Replying to jhpalmieri:

Why the change to matplotlib dependencies?

Thanks for catching this - I had thought I could get rid of tornado at some point but jupyterlab_widgets also needs it as a dependency, so that wouldn't help.

comment:34 in reply to:  31 Changed 21 months ago by Matthias Köppe

Replying to jhpalmieri:

the change in line 66 of m4/sage_spkg_configure.m4 renders very strangely (strange font for "default") when I click on the branch here for example or view the diff on my computer. Any ideas what that's about?

Sorry, I was playing with Unicode superscripts. If it does not render nice, I can just go back to using ascii

comment:35 Changed 21 months ago by git

Commit: 74a83fce03916650dd1574ee04c7f3a3dc6d584893d31b9bfca2f961b490f83cbf51bfa154a9748c

Branch pushed to git repo; I updated commit sha1. New commits:

93d31b9Remove use of unicode superscripts in configure --help

comment:36 Changed 21 months ago by Matthias Köppe

Description: modified (diff)

comment:37 Changed 21 months ago by John Palmieri

Typo in the comments? In m4/sage_spkg_collect.m4, the descriptions for SAGE_OPTIONAL_INSTALLED_PACKAGES and SAGE_OPTIONAL_CLEANED_PACKAGES are the same. Should installed be changed to uninstalled as the last word in the description for the CLEANED version?

Configuring with --disable-notebook does indeed disable the notebook, as advertised. Sage builds, and so does the documentation. I was a little surprised that all doctests passed, too.

I don't know the syntax well enough to be able to thoroughly review this, unfortunately. Anyone else?

comment:38 Changed 21 months ago by git

Commit: 93d31b9bfca2f961b490f83cbf51bfa154a9748c18fbb8598e362f048ebd8a83f963da5b0d9905f3

Branch pushed to git repo; I updated commit sha1. New commits:

18fbb85m4/sage_spkg_collect.m4: Fix description of SAGE_OPTIONAL_CLEANED_PACKAGES

comment:39 in reply to:  37 ; Changed 21 months ago by Matthias Köppe

Replying to jhpalmieri:

Configuring with --disable-notebook does indeed disable the notebook, as advertised. Sage builds, and so does the documentation. I was a little surprised that all doctests passed, too.

This is, in fact, a weakness in our doctest coverage. We have nothing at all that tests whether the Jupyter notebook works.

comment:40 Changed 21 months ago by git

Commit: 18fbb8598e362f048ebd8a83f963da5b0d9905f3c3e4093cd6ed304c271890f35f0075a2bacdf99a

Branch pushed to git repo; I updated commit sha1. New commits:

c3e4093Rename SAGE_OPTIONAL_CLEANED_PACKAGES to SAGE_OPTIONAL_UNINSTALLED_PACKAGES

comment:41 Changed 21 months ago by Matthias Köppe

Cc: Erik Bray added

comment:42 Changed 21 months ago by git

Commit: c3e4093cd6ed304c271890f35f0075a2bacdf99a4916415408a58e4b715143e5c607972822055790

Branch pushed to git repo; I updated commit sha1. New commits:

4916415Merge tag '9.3.beta9' into t/30383/new_package_type__optional_enabled_by_default

comment:43 Changed 21 months ago by Matthias Köppe

Dependencies: #31362, #31278

comment:44 Changed 21 months ago by Matthias Köppe

Priority: majorcritical

comment:45 Changed 21 months ago by Matthias Köppe

Cc: gh-kliem added

comment:46 in reply to:  39 ; Changed 21 months ago by John Palmieri

Replying to mkoeppe:

Replying to jhpalmieri:

Configuring with --disable-notebook does indeed disable the notebook, as advertised. Sage builds, and so does the documentation. I was a little surprised that all doctests passed, too.

This is, in fact, a weakness in our doctest coverage. We have nothing at all that tests whether the Jupyter notebook works.

Maybe related to the fact that you can't start the notebook once you are running Sage from the command line? With SageNB you could run notebook() from within Sage, but this is not possible any more, or at least not in an obvious way, or maybe it just hasn't been implemented by us.

comment:47 Changed 21 months ago by Matthias Köppe

I think it's just that nobody has worked on writing such tests. I guess there must be technologies for automatically testing web apps but I don't know about them or if Jupyter already uses them.

comment:48 in reply to:  46 Changed 21 months ago by Matthias Köppe

Replying to jhpalmieri:

With SageNB you could run notebook() from within Sage, but this is not possible any more, or at least not in an obvious way

This is certainly a feature that I certainly could have used if it was available. Transforming an existing Python process to a Jupyter kernel just amounts to running sage.repl.ipython_kernel.__main__, I think. But to my understanding no feature in the Jupyter notebook allows to connect to an existing running kernel. I found some discussions in this direction, for example https://github.com/jupyter/help/issues/298, https://github.com/ipython/ipython/issues/4066

comment:49 Changed 21 months ago by Dima Pasechnik

Reviewers: Dima Pasechnik
Status: needs_reviewpositive_review

this looks good to go.

comment:50 Changed 21 months ago by Matthias Köppe

Reviewers: Dima PasechnikJohn Palmieri, Dima Pasechnik

Thanks!

comment:51 Changed 21 months ago by Volker Braun

Branch: u/mkoeppe/new_package_type__optional_enabled_by_default4916415408a58e4b715143e5c607972822055790
Resolution: fixed
Status: positive_reviewclosed
Note: See TracTickets for help on using tickets.