Opened 13 months ago

Closed 6 months ago

#30383 closed enhancement (fixed)

Add "configure --disable-notebook"; show descriptions of optional packages in "configure --help"

Reported by: mkoeppe Owned by:
Priority: critical Milestone: sage-9.3
Component: build Keywords: sd111
Cc: jhpalmieri, mjo, slelievre, slabbe, dimpase, embray, gh-kliem Merged in:
Authors: Matthias Koeppe Reviewers: John Palmieri, Dima Pasechnik
Report Upstream: N/A Work issues:
Branch: 4916415 (Commits, GitHub, GitLab) Commit: 4916415408a58e4b715143e5c607972822055790
Dependencies: Stopgaps:

Status badges

Description (last modified by mkoeppe)

We refactor the code in sage_spkg_enable.m4 and sage_spkg_collect.m4 so that it becomes possible to add configure --disable-SPKG options to disable standard packages. This also simplifies build/make/Makefile.in slightly because it no longer has to make a distinction between standard and optional packages.

We demonstrate this by adding one such option, configure --disable-notebook, which disables building the Jupyter notebook package ... and all of its exclusive dependencies. This is useful for people who want to use the system Jupyter notebook -- the Jupyter kernel is still built and can be installed there.

The new option appears before the --enable... options for optional packages.

We also make the configure help a bit more informative (by including the 1-line descriptions of the optional packages) and prettier.

$ ./configure --help
...
Optional Features:
  --disable-option-checking  ignore unrecognized --enable/--with options
  --disable-FEATURE       do not include FEATURE (same as --enable-FEATURE=no)
  --enable-FEATURE[=ARG]  include FEATURE [ARG=yes]
...
  --enable-experimental-packages
                          allow installing experimental packages (default: no
                          = ask for user confirmation for each package)
  --enable-download-from-upstream-url
                          allow downloading packages from their upstream URL
                          if they cannot be found on the Sage mirrors
  --disable-notebook      disable build of the Jupyter notebook and related
                          packages
  --enable-4ti2={no|if_installed (default)|yes}
                          enable build and use of the optional package 4ti2: Algebraic, geometric
                          and combinatorial problems on linear spaces
                          * package info: ./sage -info 4ti2
  --disable-4ti2          disable build and uninstall if previously installed by Sage in PREFIX;
                          same as --enable-4ti2=no
  --enable-atlas={no|if_installed (default)|yes}
                          enable build and use of the optional package atlas: Automatically Tuned
                          Linear Algebra Software (BLAS implementation)
                          * package info: ./sage -info atlas
  --disable-atlas         disable build and uninstall if previously installed by Sage in PREFIX;
                          same as --enable-atlas=no
  --enable-awali={no|if_installed (default)|yes}
                          enable build and use of the experimental package awali: Computation of/with
                          finite state machines
                          * package info: ./sage -info awali
  --disable-awali         disable build and uninstall if previously installed by Sage in PREFIX;
                          same as --enable-awali=no

This is also preparation for #30556 (packages that will not work without openssl), and for testing modularized installs (#30778, #29864).

Change History (51)

comment:1 Changed 12 months ago by mkoeppe

  • Description modified (diff)

comment:2 Changed 12 months ago by mkoeppe

  • Cc mjo slelievre added

comment:3 Changed 12 months ago by mkoeppe

Suggestions for a better name for this new package type are welcome

comment:4 Changed 12 months ago by mkoeppe

Alternatively, we could also make it possible to disable any standard package.

comment:5 Changed 12 months ago by mkoeppe

  • Description modified (diff)
  • Summary changed from New package type: optional-enabled-by-default to "configure --disable-SPKG" for standard packages

comment:6 Changed 12 months ago by mkoeppe

  • Dependencies set to #29363

comment:7 Changed 12 months ago by mkoeppe

  • Branch set to u/mkoeppe/new_package_type__optional_enabled_by_default

comment:8 Changed 12 months ago by mkoeppe

  • Authors set to Matthias Koeppe
  • Commit set to 182c1fd756d75029e2d1885df68af441a56cdac1

New commits:

c73460cAdd options 'configure --disable-SPKG' for standard packages
22d7ae4m4/sage_spkg_*.m4: At the end of configure, indicate which optional/experimental packages are configured to be installed
bd6c109Merge branch 't/29363/at_the_end_of_configure__indicate_which_optional_experimental_packages_are_configured_to_be_installed' into t/30383/new_package_type__optional_enabled_by_default
3c98700Replace SAGE_STANDARD_PACKAGES by use of SAGE_OPTIONAL_INSTALLED_PACKAGES
182c1fdm4/sage_spkg_collect.m4: Reduce verbosity by removing 'does not support check for system package'

comment:9 Changed 12 months ago by mkoeppe

  • Status changed from new to needs_review

comment:10 Changed 12 months ago by mkoeppe

  • Description modified (diff)

comment:11 Changed 12 months ago by mkoeppe

  • Description modified (diff)

comment:12 follow-up: Changed 12 months ago by mjo

Doesn't this create a bajillion ./configure options that basically do nothing (because the packages are dependencies), and another bajillion that cause sage to function improperly (test suite failures, etc.)?

I think the original approach was more on-track, although I wouldn't have created a separate package type for it (we don't need to proliferate a new package type throughout the build system every time we tweak a bit of metadata). Instead I would have added an "enabled by default" (default: no) feature somewhere to the optional packages. Long-term, a way to control (and depend on) optional features is needed, but that's a much larger undertaking akin to USE flags in Gentoo.

comment:13 in reply to: ↑ 12 Changed 12 months ago by mkoeppe

Replying to mjo:

Doesn't this create a bajillion ./configure options that basically do nothing (because the packages are dependencies), and another bajillion that cause sage to function improperly (test suite failures, etc.)?

Yes, a gazillion of "advanced options"

comment:14 Changed 12 months ago by mkoeppe

Most "standard" packages are not really required in any strict sense other than passing doctests that test particular features. Making it possible to disable them gives users a remedy when the build fails on a system

comment:15 Changed 10 months ago by mkoeppe

  • Keywords sd111 added

Hoping we can make progress on this ticket this week - https://wiki.sagemath.org/days111

comment:16 Changed 9 months ago by git

  • Commit changed from 182c1fd756d75029e2d1885df68af441a56cdac1 to e64ef1a6c7436dd515c1daf968fe00ed6d1f9f71

Branch pushed to git repo; I updated commit sha1. New commits:

e64ef1aMerge tag '9.3.beta4' into t/30383/new_package_type__optional_enabled_by_default

comment:17 Changed 9 months ago by mkoeppe

  • Cc slabbe added

comment:18 Changed 7 months ago by git

  • Commit changed from e64ef1a6c7436dd515c1daf968fe00ed6d1f9f71 to af2e1fb38eab589e47d3cfcf841054fb9edbd43d

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

ea182d7Copy changes from build/pkgs/sagelib/src to src
a1a10b9src/VERSION.txt: New
5697335src/setup.cfg: Add license_file=LICENSE.txt
deb9eb3Merge tag '9.3.beta3' into t/30912/sagelib__update_metadata_for_pypi_deployment
7ad4c0eMerge tag '9.3.beta4' into t/30912/sagelib__update_metadata_for_pypi_deployment
27b589bMerge branch 't/30912/sagelib__update_metadata_for_pypi_deployment' into t/31362/make_all_sage_local__make_all_sage_venv
8796008Put pynac into SAGE_LOCAL, sagelib into SAGE_VENV
d6831b1build/make/Makefile.in: Add all-build-local, all-build-venv, which include dependency on toolchain
61f6ba6Makefile: Add top-level targets build-local, build-venv
af2e1fbMerge #31362

comment:19 Changed 7 months ago by mkoeppe

  • Dependencies changed from #29363 to #31362

comment:20 Changed 7 months ago by mkoeppe

  • Status changed from needs_review to needs_work

comment:21 Changed 7 months ago by git

  • Commit changed from af2e1fb38eab589e47d3cfcf841054fb9edbd43d to 340fbb719da34088eaa3e534a62795121431bdff

Branch pushed to git repo; I updated commit sha1. New commits:

340fbb7bootstrap: Emit SAGE_SPKG_ENABLE calls for standard packages

comment:22 Changed 7 months ago by git

  • Commit changed from 340fbb719da34088eaa3e534a62795121431bdff to 50262a4cba507052b99932813018f291183db856

Branch pushed to git repo; I updated commit sha1. New commits:

47e83d8Move computation of SAGE_OPTIONAL_{INSTALLED,CLEANED}_PACKAGES to sage_spkg_collect.m4
affcad3bootstrap: Emit initialization of SAGE_ENABLE_... variables for all packages; do not call SAGE_SPKG_ENABLE for standard packages
f89c506build/make/Makefile.in: Restore variable OPTIONAL_INSTALLED_PACKAGES
50262a4Include short package description in help string for configure --enable-SPKG, improve formatting

comment:23 Changed 7 months ago by git

  • Commit changed from 50262a4cba507052b99932813018f291183db856 to 7e22ff01285b337468802f5af6a45f125db02ac0

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

8397c74Include short package description in help string for configure --enable-SPKG, improve formatting
7e22ff0WIP

comment:24 Changed 6 months ago by git

  • Commit changed from 7e22ff01285b337468802f5af6a45f125db02ac0 to 0a83740c1b3d12dc27a2e3807b7769e42906a2bf

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

0a83740Initialize SAGE_ENABLE_... properly, add 'configure --disable-notebook'

comment:25 Changed 6 months ago by mkoeppe

  • Dependencies changed from #31362 to #31362, #31278
  • Description modified (diff)
  • Status changed from needs_work to needs_review
  • Summary changed from "configure --disable-SPKG" for standard packages to Add "configure --disable-notebook"

comment:26 Changed 6 months ago by git

  • Commit changed from 0a83740c1b3d12dc27a2e3807b7769e42906a2bf to 82d8030c3c71bb9e5503f77ae42484de47925381

Branch pushed to git repo; I updated commit sha1. New commits:

4146d88configure --disable-notebook: Do not disable widgetsnbextension
a8035aabuild/pkgs/ipywidgets: Upgrade to 7.6.3
5141bc3build/pkgs/widgetsnbextension: Patch out dependency on notebook (backport from widgetsnbextension-4)
fee8379Merge #31278
82d8030m4/sage_spkg_enable.m4: Cosmetic change to help strings

comment:27 Changed 6 months ago by mkoeppe

  • Status changed from needs_review to needs_work

comment:28 Changed 6 months ago by git

  • Commit changed from 82d8030c3c71bb9e5503f77ae42484de47925381 to 3e929982937143bae88f66917870d39979cc3551

Branch pushed to git repo; I updated commit sha1. New commits:

3e92998configure --disable-notebook: Fix up what packages are disabled

comment:29 Changed 6 months ago by mkoeppe

  • Cc dimpase added
  • Status changed from needs_work to needs_review

This version of the ticket no longer offers --disable... options for all standard packages, which was criticized previously.

comment:30 Changed 6 months ago by mkoeppe

  • Description modified (diff)
  • Summary changed from Add "configure --disable-notebook" to Add "configure --disable-notebook"; show descriptions of optional packages in "configure --help"

comment:31 follow-ups: Changed 6 months ago by jhpalmieri

Why the change to matplotlib dependencies? Also the change in line 66 of m4/sage_spkg_configure.m4 renders very strangely (strange font for "default") when I click on the branch here for example or view the diff on my computer. Any ideas what that's about?

comment:32 Changed 6 months ago by git

  • Commit changed from 3e929982937143bae88f66917870d39979cc3551 to 74a83fce03916650dd1574ee04c7f3a3dc6d5848

Branch pushed to git repo; I updated commit sha1. New commits:

74a83fcbuild/pkgs/matplotlib/dependencies: Undo removal of tornado

comment:33 in reply to: ↑ 31 Changed 6 months ago by mkoeppe

Replying to jhpalmieri:

Why the change to matplotlib dependencies?

Thanks for catching this - I had thought I could get rid of tornado at some point but jupyterlab_widgets also needs it as a dependency, so that wouldn't help.

comment:34 in reply to: ↑ 31 Changed 6 months ago by mkoeppe

Replying to jhpalmieri:

the change in line 66 of m4/sage_spkg_configure.m4 renders very strangely (strange font for "default") when I click on the branch here for example or view the diff on my computer. Any ideas what that's about?

Sorry, I was playing with Unicode superscripts. If it does not render nice, I can just go back to using ascii

comment:35 Changed 6 months ago by git

  • Commit changed from 74a83fce03916650dd1574ee04c7f3a3dc6d5848 to 93d31b9bfca2f961b490f83cbf51bfa154a9748c

Branch pushed to git repo; I updated commit sha1. New commits:

93d31b9Remove use of unicode superscripts in configure --help

comment:36 Changed 6 months ago by mkoeppe

  • Description modified (diff)

comment:37 follow-up: Changed 6 months ago by jhpalmieri

Typo in the comments? In m4/sage_spkg_collect.m4, the descriptions for SAGE_OPTIONAL_INSTALLED_PACKAGES and SAGE_OPTIONAL_CLEANED_PACKAGES are the same. Should installed be changed to uninstalled as the last word in the description for the CLEANED version?

Configuring with --disable-notebook does indeed disable the notebook, as advertised. Sage builds, and so does the documentation. I was a little surprised that all doctests passed, too.

I don't know the syntax well enough to be able to thoroughly review this, unfortunately. Anyone else?

comment:38 Changed 6 months ago by git

  • Commit changed from 93d31b9bfca2f961b490f83cbf51bfa154a9748c to 18fbb8598e362f048ebd8a83f963da5b0d9905f3

Branch pushed to git repo; I updated commit sha1. New commits:

18fbb85m4/sage_spkg_collect.m4: Fix description of SAGE_OPTIONAL_CLEANED_PACKAGES

comment:39 in reply to: ↑ 37 ; follow-up: Changed 6 months ago by mkoeppe

Replying to jhpalmieri:

Configuring with --disable-notebook does indeed disable the notebook, as advertised. Sage builds, and so does the documentation. I was a little surprised that all doctests passed, too.

This is, in fact, a weakness in our doctest coverage. We have nothing at all that tests whether the Jupyter notebook works.

comment:40 Changed 6 months ago by git

  • Commit changed from 18fbb8598e362f048ebd8a83f963da5b0d9905f3 to c3e4093cd6ed304c271890f35f0075a2bacdf99a

Branch pushed to git repo; I updated commit sha1. New commits:

c3e4093Rename SAGE_OPTIONAL_CLEANED_PACKAGES to SAGE_OPTIONAL_UNINSTALLED_PACKAGES

comment:41 Changed 6 months ago by mkoeppe

  • Cc embray added

comment:42 Changed 6 months ago by git

  • Commit changed from c3e4093cd6ed304c271890f35f0075a2bacdf99a to 4916415408a58e4b715143e5c607972822055790

Branch pushed to git repo; I updated commit sha1. New commits:

4916415Merge tag '9.3.beta9' into t/30383/new_package_type__optional_enabled_by_default

comment:43 Changed 6 months ago by mkoeppe

  • Dependencies #31362, #31278 deleted

comment:44 Changed 6 months ago by mkoeppe

  • Priority changed from major to critical

comment:45 Changed 6 months ago by mkoeppe

  • Cc gh-kliem added

comment:46 in reply to: ↑ 39 ; follow-up: Changed 6 months ago by jhpalmieri

Replying to mkoeppe:

Replying to jhpalmieri:

Configuring with --disable-notebook does indeed disable the notebook, as advertised. Sage builds, and so does the documentation. I was a little surprised that all doctests passed, too.

This is, in fact, a weakness in our doctest coverage. We have nothing at all that tests whether the Jupyter notebook works.

Maybe related to the fact that you can't start the notebook once you are running Sage from the command line? With SageNB you could run notebook() from within Sage, but this is not possible any more, or at least not in an obvious way, or maybe it just hasn't been implemented by us.

comment:47 Changed 6 months ago by mkoeppe

I think it's just that nobody has worked on writing such tests. I guess there must be technologies for automatically testing web apps but I don't know about them or if Jupyter already uses them.

comment:48 in reply to: ↑ 46 Changed 6 months ago by mkoeppe

Replying to jhpalmieri:

With SageNB you could run notebook() from within Sage, but this is not possible any more, or at least not in an obvious way

This is certainly a feature that I certainly could have used if it was available. Transforming an existing Python process to a Jupyter kernel just amounts to running sage.repl.ipython_kernel.__main__, I think. But to my understanding no feature in the Jupyter notebook allows to connect to an existing running kernel. I found some discussions in this direction, for example https://github.com/jupyter/help/issues/298, https://github.com/ipython/ipython/issues/4066

comment:49 Changed 6 months ago by dimpase

  • Reviewers set to Dima Pasechnik
  • Status changed from needs_review to positive_review

this looks good to go.

comment:50 Changed 6 months ago by mkoeppe

  • Reviewers changed from Dima Pasechnik to John Palmieri, Dima Pasechnik

Thanks!

comment:51 Changed 6 months ago by vbraun

  • Branch changed from u/mkoeppe/new_package_type__optional_enabled_by_default to 4916415408a58e4b715143e5c607972822055790
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.