Opened 5 months ago

Closed 4 months ago

#30195 closed enhancement (fixed)

tox.ini: Add centos-6, docker-nobootstrap, manylinux-*

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-9.2
Component: porting Keywords:
Cc: gh-kliem, slelievre Merged in:
Authors: Matthias Koeppe Reviewers: Jonathan Kliem
Report Upstream: N/A Work issues:
Branch: 14da64d (Commits) Commit: 14da64d27c43ad7f3328da208ee942a54b889b5b
Dependencies: #29929 Stopgaps:

Description (last modified by mkoeppe)

The docker-nobootstrap option uses bootstrap -D; thus, autotools are not required on the host for the build.

This enables building with centos-6 and the manylinux images based on centos-6 (https://github.com/pypa/manylinux).

Examples:

tox -e docker-manylinux-2014-standard
tox -e docker-manylinux-2014-standard-python3.9
tox -e docker-manylinux-2010-standard-i686-python3.8

Change History (28)

comment:1 Changed 5 months ago by mkoeppe

  • Authors set to Matthias Koeppe
  • Cc gh-kliem slelievre added
  • Dependencies set to #29929
  • Description modified (diff)
  • Summary changed from tox.ini: Add manylinux to tox.ini: Add centos-6, docker-nobootstrap, manylinux-*

comment:2 Changed 5 months ago by mkoeppe

  • Branch set to u/mkoeppe/tox_ini__add_manylinux

comment:3 Changed 5 months ago by mkoeppe

  • Commit set to 585bdac80224f6f6daace3d7d8ab663bf309f752
  • Status changed from new to needs_review

Branch is 1 commit on top of #29929, which also needs review


Last 10 new commits:

25393b0Handle SAGE_CHECK_PACKAGES in build/make/Makefile.in, not sage-spkg
5aca238Merge branch 't/30118/handle_sage_check_packages_in_build_make_makefile_in__not_sage_spkg' into t/29929/tox_ini__add_a_macos_environment_without_homebrew__conda
a17c655m4/sage_spkg_collect.m4: Do not include script packages in SAGE_SDIST_PACKAGES
827923abootstrap: Accept 2020s in config.guess version check
c30ac54tox.ini: Add local-nobootstrap
de6464d.github/workflows/tox.yml: Add jobs "dist", "local-macos-nohomebrew
775b7f7Trac #30088: add a few explanatory comments to m4/sage_spkg_collect.m4.
160862fm4/sage_spkg_collect.m4: Mention sagelib in the comment
d1a2cbfMerge branch 't/30088/fix__make_sdist___add_test_run_to_github_actions' into t/29929/tox_ini__add_a_macos_environment_without_homebrew__conda
585bdactox.ini: Add centos-6, docker-nobootstrap, manylinux-*

comment:4 Changed 5 months ago by gh-kliem

  • Reviewers set to Jonathan Kliem

Can you provide an example of this at work?

According to the readme, only CPython up to 3.8 are provided with manylinux. Why have you configured python3.9?

comment:5 Changed 5 months ago by mkoeppe

The readme does not seem to be up to date. On an actual docker image:

[root@4185c8d9310b /]# /opt/python/cp3 TAB
cp35-cp35m/ cp36-cp36m/ cp37-cp37m/ cp38-cp38/  cp39-cp39/  

comment:6 Changed 5 months ago by gh-kliem

That was one of my guesses.

Do the new environments work with github actions? If you haven't tested it yet, I have free capacities and can just test this ticket.

comment:7 Changed 5 months ago by mkoeppe

They should work, just add them to tox.yml for testing. I haven't tested them widely yet, only one run of docker-manylinux-2014-standard on my computer. Best to test together with #30173

comment:8 Changed 5 months ago by gh-kliem

 File "/usr/lib/python3/dist-packages/tox/session.py", line 38, in main
    config = prepare(args)
  File "/usr/lib/python3/dist-packages/tox/session.py", line 26, in prepare
    config = parseconfig(args)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 239, in parseconfig
    parseini(config, inipath)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 760, in __init__
    self.make_envconfig(name, section, reader._subs, config)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 791, in make_envconfig
    res = meth(env_attr.name, env_attr.default)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 964, in getargvlist
    return _ArgvlistReader.getargvlist(self, s)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1134, in getargvlist
    commands.append(cls.processcommand(reader, current_command))
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1164, in processcommand
    new_word = reader._replace(word)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1008, in _replace
    return Replacer(self, crossonly=crossonly).do_replace(value)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1028, in do_replace
    return self.RE_ITEM_REF.sub(self._replace_match, x)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1057, in _replace_match
    return self._replace_env(match)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1077, in _replace_env
    (envkey, envkey))
tox.ConfigError: ConfigError: substitution env:'BOOTSTRAP': unknown environment variable 'BOOTSTRAP'  or recursive definition.
##[error]Process completed with exit code 1.

https://github.com/kliem/sage/runs/897566509?check_suite_focus=true

comment:9 Changed 5 months ago by git

  • Commit changed from 585bdac80224f6f6daace3d7d8ab663bf309f752 to 756f310bce050bd1c4303398638d061a0705feb4

Branch pushed to git repo; I updated commit sha1. New commits:

756f310tox.ini: Initialize the BOOTSTRAP variable

comment:10 Changed 5 months ago by mkoeppe

Thanks for catching this

comment:11 follow-ups: Changed 5 months ago by gh-kliem

Centos 6 seems to work in the way that it fails to buil mpir.

There are many errors with manylinux like

   File "/usr/lib/python3/dist-packages/tox/config.py", line 1008, in _replace
    return Replacer(self, crossonly=crossonly).do_replace(value)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1028, in do_replace
    return self.RE_ITEM_REF.sub(self._replace_match, x)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1057, in _replace_match
    return self._replace_env(match)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1071, in _replace_env
    envvalue = self.reader.get_environ_value(envkey)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 903, in get_environ_value
    return self._setenv.get(name)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 286, in get
    self.resolved[name] = res = self.reader._replace(val)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1008, in _replace
    return Replacer(self, crossonly=crossonly).do_replace(value)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1028, in do_replace
    return self.RE_ITEM_REF.sub(self._replace_match, x)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1057, in _replace_match
    return self._replace_env(match)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1077, in _replace_env
    (envkey, envkey))
tox.ConfigError: ConfigError: substitution env:'BASE_IMAGE': unknown environment variable 'BASE_IMAGE'  or recursive definition.

https://github.com/kliem/sage/runs/899346008?check_suite_focus=true

and

 Step 14/18 : RUN ${BOOTSTRAP}
 ---> Running in d5c9ccb6ceb3
rm -rf config configure build/make/Makefile-auto.in
rm -f src/doc/en/installation/*.txt
rm -rf src/doc/en/reference/spkg/*.rst
rm -f src/doc/en/reference/repl/*.txt
Traceback (most recent call last):
  File "/sage/build/bin/sage-download-file", line 27, in ?
    from sage_bootstrap.download.cmdline import run_safe
  File "/sage/build/bin/../sage_bootstrap/download/__init__.py", line 7, in ?
    from sage_bootstrap.download.transfer import Download
  File "/sage/build/bin/../sage_bootstrap/download/transfer.py", line 43
    n = 0 if chunks_so_far == 0 else self.length // 2
           ^
SyntaxError: invalid syntax
Error: downloading configure-339b4599590f3dc3acec5eace4b7a82f210d3219.tar.gz failed

https://github.com/kliem/sage/runs/899346035?check_suite_focus=true

Should they be fixed here?

comment:12 Changed 5 months ago by mkoeppe

  • Status changed from needs_review to needs_work

I'll look into the tox config error first.

comment:13 in reply to: ↑ 11 Changed 4 months ago by mkoeppe

Replying to gh-kliem:

Centos 6 seems to work in the way that it fails to buil mpir.

Right... errors from an ancient assembler version.

tmp-mul_1.s: Assembler messages:
tmp-mul_1.s:94: Error: no such instruction: `mulx (%rsi),%r9,%r8'
tmp-mul_1.s:95: Error: no such instruction: `mulx 8(%rsi),%r11,%r10'
tmp-mul_1.s:96: Error: no such instruction: `mulx 16(%rsi),%rcx,%r12'

We'll definitely not try to support centos-6 and shouldn't be adding it to tox.yml.

comment:14 in reply to: ↑ 11 Changed 4 months ago by mkoeppe

Replying to gh-kliem:

Centos 6 seems to work in the way that it fails to buil mpir.

There are many errors with manylinux like

tox.ConfigError: ConfigError: substitution env:'BASE_IMAGE': unknown environment variable 'BASE_IMAGE'  or recursive definition.

This is just caused by me not adding documentation and leaving you to guess what is supported. manylinux is not a configuration - only manylinux-1, manylinux-2010, manylinux-2014 are.

I'll make manylinux an alias for manylinux-2014 and add some documentation.

comment:15 Changed 4 months ago by git

  • Commit changed from 756f310bce050bd1c4303398638d061a0705feb4 to 8c13ac7bef0b07070f761035064c6920384ca46b

Branch pushed to git repo; I updated commit sha1. New commits:

8c13ac7tox.ini: Make manylinux default to manylinux-2014, add documentation

comment:16 in reply to: ↑ 11 Changed 4 months ago by mkoeppe

Replying to gh-kliem:

  File "/sage/build/bin/../sage_bootstrap/download/transfer.py", line 43
    n = 0 if chunks_so_far == 0 else self.length // 2
           ^
SyntaxError: invalid syntax
Error: downloading configure-339b4599590f3dc3acec5eace4b7a82f210d3219.tar.gz failed

https://github.com/kliem/sage/runs/899346035?check_suite_focus=true

The added comments also explain this one -- manylinux-1 is too old, after all, and sage cannot build on it

comment:17 Changed 4 months ago by mkoeppe

On manylinux-2010-minimal (https://github.com/kliem/sage/runs/899346056), building python3 fails with ./python: error while loading shared libraries: libcrypt.so.2: cannot open shared object file: No such file or directory (related to #29012?) ... but actually it was intended to use the installed python3 in /opt. Looks like I need to make adjustments for -2010.

comment:18 Changed 4 months ago by git

  • Commit changed from 8c13ac7bef0b07070f761035064c6920384ca46b to 2747c379b26310253b4a5996b9dd6e74858d3323

Branch pushed to git repo; I updated commit sha1. New commits:

4c670d3build/pkgs/xz/spkg-configure.m4: Slightly lower the xz version bound set in #26286, for manylinux-2010
2f43164tox.ini [manylinux]: Use --with-system-python3=force
2747c37tox.ini [manylinux]: Patch the downloaded configure file so that system xz is accepted

comment:19 Changed 4 months ago by mkoeppe

  • Status changed from needs_work to needs_review

comment:20 Changed 4 months ago by mkoeppe

When you have a chance, please test this version again. See above on my comments on the supported tox environments. Also note the images for non-x86 archs can't work on Linux Docker (and thus on GH Actions) because they are not multi-arch images. They work on macOS Docker Desktop because they are running inside of an emulator.

comment:21 Changed 4 months ago by gh-kliem

https://github.com/kliem/sage/actions/runs/180807074

That is what I'm testing for now:

 tox_system_factor: [ubuntu-focal, debian-buster, centos-8, manylinux-1, manylinux-2010, manylinux-2014, manylinux-2014-python3.7]

manylinux-1 will of course fail.

comment:22 Changed 4 months ago by gh-kliem

I'm getting configure failures with minimal. Will this be a subject of this ticket or of another one?

Otherwise it's looking fine.

comment:23 Changed 4 months ago by git

  • Commit changed from 2747c379b26310253b4a5996b9dd6e74858d3323 to 14da64d27c43ad7f3328da208ee942a54b889b5b

Branch pushed to git repo; I updated commit sha1. New commits:

14da64dtox.ini [manylinux]: Do not use --with-system-python3=force for -minimal

comment:24 Changed 4 months ago by gh-kliem

manylinux-2014-python3.7, minimal still fails at configuration.

https://github.com/kliem/sage/runs/908012032?check_suite_focus=true

Maybe I just got the name scheme wrong.

comment:25 Changed 4 months ago by mkoeppe

minimal with python3.7 cannot work because python3 dependencies such as sqlite3 are not available from the system and therefore it cannot use system python3.

comment:26 Changed 4 months ago by gh-kliem

https://github.com/kliem/sage/actions/runs/181590963

This seems to work well. At least according to the timings we are well in the build process. Any failures are probably subject to different tickets now.

If the run turns out to your satisfaction, you can put it on positive review on my behalf.

comment:27 Changed 4 months ago by mkoeppe

  • Status changed from needs_review to positive_review

Thank you!

comment:28 Changed 4 months ago by vbraun

  • Branch changed from u/mkoeppe/tox_ini__add_manylinux to 14da64d27c43ad7f3328da208ee942a54b889b5b
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.