Opened 2 years ago

Last modified 7 months ago

#29854 new enhancement

Install sage_conf using flit instead of setuptools

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-wishlist
Component: build Keywords:
Cc: Merged in:
Authors: Reviewers:
Report Upstream: N/A Work issues:
Branch: u/mkoeppe/install_sage_conf_using_flit (Commits, GitHub, GitLab) Commit: 7ea5fabac7e4f2485a89e93f2e5931c0f9639935
Dependencies: #29846 Stopgaps:

Status badges

Description (last modified by mkoeppe)

So far this is just an illustration of what can be done with flit (#29846) and what cannot:

Not intended for merging at this point.

Change History (18)

comment:1 Changed 2 years ago by mkoeppe

  • Branch set to u/mkoeppe/install_sage_conf_using_flit

comment:2 Changed 2 years ago by mkoeppe

  • Commit set to fd83992029e0eacadd60f2d5bf6c53150c8ce46c
  • Work issues set to Rebase so it is not on top of #29850

Last 10 new commits:

dacea8bWIP: Install sage-env-config with sage_conf
1ee054cFixup
5f7b1ccMerge branch 't/21559/change-src-bin-installation' into t/29850/install_sage_env_config_with_sage_conf
dfb4400build/pkgs/pytoml: New
11b6003build/pkgs/flit, flit_core: New
20e2d59Add dependencies
721e8fdbuild/bin/sage-pip-install: For PEP 517 packages, do not try to uninstall first
5e7155ebuild/pkgs/pytoml/spkg-install.in: New
8f58336Merge branch 't/29846/add_packages_flit_and_dependencies_flit_core__pytoml' into t/29850/install_sage_env_config_with_sage_conf
fd83992build/pkgs/sage_conf: Install with flit instead of setuptools

comment:3 Changed 2 years ago by mkoeppe

This branch does not work because it is trying to use a symlink for sage_conf.py into the SAGE_ROOT/src directory.

[sage_conf-none] Processing /Users/mkoeppe/s/sage/sage-rebasing/build/pkgs/sage_conf/src
[sage_conf-none]   Created temporary directory: /private/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pip-req-build-w7fopzwn
[sage_conf-none]   Added file:///Users/mkoeppe/s/sage/sage-rebasing/build/pkgs/sage_conf/src to build tracker '/private/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pip-req-tracker-jmerw27d'
[sage_conf-none]     Created temporary directory: /private/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pip-modern-metadata-dxzef0bb
[sage_conf-none]     Preparing wheel metadata: started
[sage_conf-none]     Running command /Users/mkoeppe/s/sage/sage-rebasing/local/bin/python3 /Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/tmpb7arkmg5
[sage_conf-none]     Traceback (most recent call last):
[sage_conf-none]       File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
[sage_conf-none]         main()
[sage_conf-none]       File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
[sage_conf-none]         json_out['return_val'] = hook(**hook_input['kwargs'])
[sage_conf-none]       File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 133, in prepare_metadata_for_build_wheel
[sage_conf-none]         return hook(metadata_directory, config_settings)
[sage_conf-none]       File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/flit_core/buildapi.py", line 28, in prepare_metadata_for_build_wheel
[sage_conf-none]         module = Module(ini_info.module, os.getcwd())
[sage_conf-none]       File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/flit_core/common.py", line 55, in __init__
[sage_conf-none]         raise ValueError("No file/folder found for module {}".format(name))
[sage_conf-none]     ValueError: No file/folder found for module sage_conf
[sage_conf-none]     Preparing wheel metadata: finished with status 'error'
[sage_conf-none] ERROR: Command errored out with exit status 1: /Users/mkoeppe/s/sage/sage-rebasing/local/bin/python3 /Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/tmpb7arkmg5 Check the logs for full command output.

comment:4 Changed 2 years ago by git

  • Commit changed from fd83992029e0eacadd60f2d5bf6c53150c8ce46c to e537d48dec34f383c3b66ec48f24b09cc142ae94

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

e537d48build/pkgs/sage_conf: Install with flit instead of setuptools

comment:5 Changed 2 years ago by mkoeppe

  • Dependencies set to #29846
  • Work issues Rebase so it is not on top of #29850 deleted

comment:6 Changed 2 years ago by mkoeppe

Rebased on top of only #29846 (flit)

comment:7 Changed 2 years ago by git

  • Commit changed from e537d48dec34f383c3b66ec48f24b09cc142ae94 to 4d20d4a7c1ed3403644cdd45c7e0afcc4b8c8f2d

Branch pushed to git repo; I updated commit sha1. New commits:

27cf73bbuild/pkgs/sage_conf: Switch to src layout where src is a symlink (also does not work)
4d20d4abuild/pkgs/sage_conf: Back to sage_conf.py[.in] as a file here, not a symlink (works)

comment:8 Changed 2 years ago by mkoeppe

  • Description modified (diff)
  • Summary changed from Install sage_conf using flit to Install sage_conf using flit instead of setuptools

comment:9 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:10 Changed 2 years ago by mkoeppe

Both with this version and the version in 27cf73b, also:

flit build 
Fetching list of valid trove classifiers                                                                                                                                                                      I-flit.validate
Version number normalised: '9.2.beta0' -> '9.2b0' (see PEP 440)                                                                                                                                         W-flit_core.versionno
Found 4 files tracked in git                                                                                                                                                                                     I-flit.sdist
Traceback (most recent call last):
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/bin/flit", line 8, in <module>
    sys.exit(main())
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/flit/__init__.py", line 171, in main
    gen_setup_py=args.setup_py)
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/flit/build.py", line 46, in main
    sdist_file = sb.build(dist_dir, gen_setup_py=gen_setup_py)
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/flit/sdist.py", line 223, in build
    return Path(super().build(str(target_dir), gen_setup_py=gen_setup_py))
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/flit_core/sdist.py", line 181, in build
    files_to_add = self.apply_includes_excludes(self.select_files())
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/flit_core/sdist.py", line 155, in apply_includes_excludes
    .format(", ".join(missing_crucial)))
Exception: Crucial files were excluded from the sdist: src/sage_conf.py

(because sage_conf.py is generated and not tracked in git)

Last edited 2 years ago by mkoeppe (previous) (diff)

comment:11 Changed 2 years ago by git

  • Commit changed from 4d20d4a7c1ed3403644cdd45c7e0afcc4b8c8f2d to 7ea5fabac7e4f2485a89e93f2e5931c0f9639935

Branch pushed to git repo; I updated commit sha1. New commits:

7ea5fabbuild/pkgs/sage_conf/src/pyproject.toml: Add tool.flit.sdist section

comment:12 Changed 2 years ago by mkoeppe

This fixes the sdist.

Same change on top of 27cf73b creates a tar file with a packaged symlink:

 lrwxr-xr-x       0/0             0 sage_conf-9.2b0/src --> ../../../../src

and gives an error afterwards:

Built sdist: dist/sage_conf-9.2b0.tar.gz                                                                                                                                                                    I-flit_core.sdist
Traceback (most recent call last):
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/bin/flit", line 8, in <module>
    sys.exit(main())
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/flit/__init__.py", line 171, in main
    gen_setup_py=args.setup_py)
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/flit/build.py", line 51, in main
    with unpacked_tarball(sdist_file) as tmpdir:
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.7/site-packages/flit/build.py", line 24, in unpacked_tarball
    tf.extractall(tmpdir)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/tarfile.py", line 2000, in extractall
    numeric_owner=numeric_owner)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/tarfile.py", line 2042, in extract
    numeric_owner=numeric_owner)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/tarfile.py", line 2104, in _extract_member
    os.makedirs(upperdirs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/os.py", line 223, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/tmp6jcs8l04/sage_conf-9.2b0/src'
Last edited 2 years ago by mkoeppe (previous) (diff)

comment:13 Changed 2 years ago by mkoeppe

A big part of the issues with symlinks comes, of course, from the behavior of pip, which copies the local source directory to a temporary location, breaking symlinks in the process. Various unresolved pip issues talk about finding a solution for local builds, most recently https://github.com/pypa/pip/issues/7555 (earlier issues: https://github.com/pypa/pip/issues/2195 and within, https://github.com/pypa/pip/pull/7882).

Last edited 2 years ago by mkoeppe (previous) (diff)

comment:14 Changed 2 years ago by mkoeppe

  • Milestone changed from sage-9.2 to sage-9.3

comment:15 Changed 18 months ago by mkoeppe

  • Milestone changed from sage-9.3 to sage-9.4

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

comment:16 Changed 13 months ago by mkoeppe

  • Milestone changed from sage-9.4 to sage-9.5

comment:17 Changed 8 months ago by mkoeppe

  • Milestone changed from sage-9.5 to sage-9.6

comment:18 Changed 7 months ago by mkoeppe

  • Milestone changed from sage-9.6 to sage-wishlist
Note: See TracTickets for help on using tickets.