Opened 3 months ago

Last modified 3 weeks ago

#34081 needs_work enhancement

Upgrade scipy to 1.9.x, add meson toolchain

Reported by: Matthias Köppe Owned by:
Priority: major Milestone: sage-9.8
Component: packages: standard Keywords:
Cc: Dima Pasechnik, Antonio Rojas, François Bissey, Michael Orlitzky Merged in:
Authors: Matthias Koeppe Reviewers:
Report Upstream: N/A Work issues:
Branch: u/mkoeppe/numpy_1_23_x__scipy_1_9_x (Commits, GitHub, GitLab) Commit: df0fb1590d7a1916ad4f5ad7b002e807517b1802
Dependencies: #34228, #34115, #34246, #32992, #34449, #34344 Stopgaps:

Status badges

Description (last modified by Matthias Köppe)

Previous update:

  • #32423 Update numpy to 1.22.x, scipy 1.8.x, networkx 2.8.x

See also:

Change History (79)

comment:1 Changed 3 months ago by Matthias Köppe

Dependencies: #32423

comment:2 Changed 3 months ago by Matthias Köppe

Branch: u/mkoeppe/numpy_1_23_x__scipy_1_9_x

comment:3 Changed 3 months ago by Matthias Köppe

Commit: d87212c623914f1e63baef913bdac4f444e20c97
Dependencies: #32423#32423, #33789

Last 10 new commits:

e1998daMerge #32423
507fc2ebuild/pkgs/numpy: Update to 1.23.0
58d62f4build/bin/write-dockerfile.sh: ADD src/VERSION.txt
99f3617Merge #34017
3fab415build/pkgs/numpy: Switch to using .tar.gz
443b3d3build/pkgs/numpy/spkg-install.in: Fix 32bit build on x86_64
9cd5ea6Merge #34017
ee79c2ebuild/pkgs/numpy/spkg-install.in: Fix 32bit build on x86_64 (fixup)
2534f5abuild/pkgs/numpy/patches/21891.patch: New
d87212cMerge #32423

comment:4 Changed 3 months ago by git

Commit: d87212c623914f1e63baef913bdac4f444e20c97bf62474026511a5ee20f9beec9f96b3ce3b29820

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

738ab26build/bin/sage-dist-helpers (sdh_pip_install): Change default to --build-isolation
79c6002build/bin/sage-dist-helpers (sdh_pip_install): Fall back to --no-build-isolation on error
f6ba526build/pkgs/tomli/spkg-install.in: Build it with --no-build-isolation
0b744beMerge #33789
10aaf61build/pkgs/meson: New
3ca53c9build/pkgs/pyproject_metadata: New
be09d5abuild/pkgs/ninja: New
3b92d3fbuild/pkgs/ninja_build/type: Change to standard
563ba19build/pkgs/pyproject_metadata/dependencies: Update
bf62474build/pkgs/meson_python: Update dependencies, spkg-install

comment:5 Changed 3 months ago by Matthias Köppe

Dependencies: #32423, #33789#32423, #33789, #33866

comment:6 Changed 3 months ago by Matthias Köppe

[scipy-1.9.0rc1]       hook = backend.prepare_metadata_for_build_wheel
[scipy-1.9.0rc1]   AttributeError: module 'mesonpy' has no attribute 'prepare_metadata_for_build_wheel'
[scipy-1.9.0rc1] 
[scipy-1.9.0rc1]   During handling of the above exception, another exception occurred:
[scipy-1.9.0rc1] 
[}}}

comment:7 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:8 Changed 3 months ago by git

Commit: bf62474026511a5ee20f9beec9f96b3ce3b2982022f91fdaf314fd40f9cfaccdb6861f667da4d0b9

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

5f29fb6build/pkgs/poetry_core: New (tomlkit build dep)
0804b62build/pkgs/poetry_core/patches/5547.diff_bin: New
0c4d0ebbuild/pkgs/hatchling/dependencies: Add pluggy, packaging
3e54474build/pkgs/pip: Update to 22.1.2
7e9722fbuild/pkgs/setuptools_scm: Update to 7.0.2, add typing_extensions as dep
6f72e8cbuild/pkgs/wheel: Update to 0.37.1
3e9d4a8Revert "build/pkgs/setuptools_scm: Update to 7.0.2, add typing_extensions as dep"
5d8111fbuild/pkgs/hatchling/SPKG.rst: Add license info
6607401src/sage/misc/package.py (pip_installed_packages): Make doctests more flexible
22f91fdMerge #33866

comment:9 Changed 3 months ago by Matthias Köppe

Opened https://github.com/scipy/scipy/issues/16536 (BUG: 1.9.0rc1 fails to build on macOS when CXX="g++ -std=gnu++11")

comment:10 Changed 3 months ago by git

Commit: 22f91fdaf314fd40f9cfaccdb6861f667da4d0b902d968523bd638e4faaa4f219a6e2ca7721fbb14

Branch pushed to git repo; I updated commit sha1. New commits:

02d9685build/pkgs/scipy/spkg-install.in: Remove all old compiler/linker settings, add workaround for https://github.com/scipy/scipy/issues/16536

comment:11 Changed 3 months ago by git

Commit: 02d968523bd638e4faaa4f219a6e2ca7721fbb1448e6783e1a20b27f2e6bf202a5d1b38a9b20c672

Branch pushed to git repo; I updated commit sha1. New commits:

48e6783build/pkgs/scipy/spkg-install.in: Relax version pinning of numpy in pyproject.toml

comment:12 Changed 3 months ago by Matthias Köppe

Build goes almost through but ends with:

[scipy-1.9.0rc1] [1569/1573] Linking target scipy/spatial/_qhull.cpython-38-darwin.so
[scipy-1.9.0rc1] [1570/1573] Compiling C++ object scipy/interpolate/_rbfinterp_pythran.cpython-38-darwin.so.p/meson-generated_..__rbfinterp_pythran.cpp.o
[scipy-1.9.0rc1] [1571/1573] Linking target scipy/interpolate/_rbfinterp_pythran.cpython-38-darwin.so
[scipy-1.9.0rc1] [1572/1573] Compiling C++ object scipy/fft/_pocketfft/pypocketfft.cpython-38-darwin.so.p/pypocketfft.cxx.o
[scipy-1.9.0rc1] [1573/1573] Linking target scipy/fft/_pocketfft/pypocketfft.cpython-38-darwin.so
[scipy-1.9.0rc1] ninja: Entering directory `/Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/var/tmp/sage/build/scipy-1.9.0rc1/src/.mesonpy-bo77bwb4/build'
[scipy-1.9.0rc1] [1/2] Generating scipy/generate-version with a custom command
[scipy-1.9.0rc1] [2/2] Generating scipy/generate-config with a custom command
[scipy-1.9.0rc1] Installing scipy/__config__.py to /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/var/tmp/sage/build/scipy-1.9.0rc1/src/.mesonpy-bo77bwb4/install/usr/local/Cellar/python@3.8/3.8.13_1/bin/../Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages//scipy

<--- these directories are wrong

[scipy-1.9.0rc1] Installing scipy/version.py to /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/var/tmp/sage/build/scipy-1.9.0rc1/src/.mesonpy-bo77bwb4/install/usr/local/Cellar/python@3.8/3.8.13_1/bin/../Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages//scipy
[scipy-1.9.0rc1] Installing scipy/_lib/_ccallback_c.cpython-38-darwin.so to /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/var/tmp/sage/build/scipy-1.9.0rc1/src/.mesonpy-bo77bwb4/install/usr/local/Cellar/python@3.8/3.8.13_1/bin/../Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/scipy/_lib
[scipy-1.9.0rc1] Stripping target 'scipy/_lib/_ccallback_c.cpython-38-darwin.so'.
[scipy-1.9.0rc1] Could not strip file.
[scipy-1.9.0rc1] 
[scipy-1.9.0rc1] Stdout:
[scipy-1.9.0rc1] 
[scipy-1.9.0rc1] 
[scipy-1.9.0rc1] Stderr:
[scipy-1.9.0rc1] /Library/Developer/CommandLineTools/usr/bin/strip: error: symbols referenced by indirect symbol table entries that can't be stripped in: /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/var/tmp/sage/build/scipy-1.9.0rc1/src/.mesonpy-bo77bwb4/install/usr/local/Cellar/python@3.8/3.8.13_1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/scipy/_lib/_ccallback_c.cpython-38-darwin.so
[scipy-1.9.0rc1] _PyBytes_AsStringAndSize
[scipy-1.9.0rc1] _PyBytes_FromString
[...]
[scipy-1.9.0rc1] __Py_NoneStruct
[scipy-1.9.0rc1] __Py_TrueStruct
[scipy-1.9.0rc1] ___stack_chk_guard
[scipy-1.9.0rc1] 
[scipy-1.9.0rc1] 
[scipy-1.9.0rc1] + meson setup --native-file=/Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/var/tmp/sage/build/scipy-1.9.0rc1/src/.mesonpy-native-file.ini -Ddebug=false -Dstrip=true -Doptimization=2 --prefix=/usr/local/Cellar/python@3.8/3.8.13_1/bin/../Frameworks/Python.framework/Versions/3.8 /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/var/tmp/sage/build/scipy-1.9.0rc1/src /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/var/tmp/sage/build/scipy-1.9.0rc1/src/.mesonpy-bo77bwb4/build
[scipy-1.9.0rc1] + meson compile
[scipy-1.9.0rc1] + meson install --destdir /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/var/tmp/sage/build/scipy-1.9.0rc1/src/.mesonpy-bo77bwb4/install
[scipy-1.9.0rc1] Traceback (most recent call last):
[scipy-1.9.0rc1]   File "/Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/lib/python3.8/site-packages/pep517/in_process/_in_process.py", line 363, in <module>
[scipy-1.9.0rc1]     main()
[scipy-1.9.0rc1]   File "/Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/lib/python3.8/site-packages/pep517/in_process/_in_process.py", line 345, in main
[scipy-1.9.0rc1]     json_out['return_val'] = hook(**hook_input['kwargs'])
[scipy-1.9.0rc1]   File "/Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/lib/python3.8/site-packages/pep517/in_process/_in_process.py", line 261, in build_wheel
[scipy-1.9.0rc1]     return _build_backend().build_wheel(wheel_directory, config_settings,
[scipy-1.9.0rc1]   File "/Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/lib/python3.8/site-packages/mesonpy/__init__.py", line 837, in build_wheel
[scipy-1.9.0rc1]     return project.wheel(out).name
[scipy-1.9.0rc1]   File "/Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/lib/python3.8/site-packages/mesonpy/__init__.py", line 786, in wheel
[scipy-1.9.0rc1]     wheel = _WheelBuilder(self).build(self._install_plan, self._copy_files, self._build_dir)
[scipy-1.9.0rc1]   File "/Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/lib/python3.8/site-packages/mesonpy/__init__.py", line 318, in build
[scipy-1.9.0rc1]     self._project.build()  # ensure project is built
[scipy-1.9.0rc1]   File "/Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/lib/python3.8/site-packages/mesonpy/__init__.py", line 482, in build
[scipy-1.9.0rc1]     self._meson('install', '--destdir', os.fspath(self._install_dir))
[scipy-1.9.0rc1]   File "/Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/lib/python3.8/site-packages/mesonpy/__init__.py", line 430, in _meson
[scipy-1.9.0rc1]     return self._proc('meson', *args)
[scipy-1.9.0rc1]   File "/Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/lib/python3.8/site-packages/mesonpy/__init__.py", line 426, in _proc
[scipy-1.9.0rc1]     subprocess.check_call(list(args))
[scipy-1.9.0rc1]   File "/usr/local/Cellar/python@3.8/3.8.13_1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 364, in check_call
[scipy-1.9.0rc1]     raise CalledProcessError(retcode, cmd)
[scipy-1.9.0rc1] subprocess.CalledProcessError: Command '['meson', 'install', '--destdir', '/Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/var/tmp/sage/build/scipy-1.9.0rc1/src/.mesonpy-bo77bwb4/install']' returned non-zero exit status 1.
[scipy-1.9.0rc1] * Getting dependencies for wheel...
[scipy-1.9.0rc1] * Building wheel...
[scipy-1.9.0rc1] 
[scipy-1.9.0rc1] ERROR Backend subprocess exited when trying to invoke build_wheel

comment:14 Changed 3 months ago by git

Commit: 48e6783e1a20b27f2e6bf202a5d1b38a9b20c6729b4115490ddde5704d4fe56a601a3cde3b56fb96

Branch pushed to git repo; I updated commit sha1. New commits:

9b41154build/pkgs/meson_python/patches: Add https://github.com/FFY00/meson-python/pull/90

comment:15 Changed 3 months ago by Matthias Köppe

That fixes it, and now I get

[scipy-1.9.0rc1] Successfully built scipy-1.9.0rc1-cp38-cp38-macosx_12_x86_64.whl
[scipy-1.9.0rc1] WARNING: Skipping scipy as it is not installed.
[scipy-1.9.0rc1] Using pip 22.1.2 from /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/lib/python3.8/site-packages/pip (python 3.8)
[scipy-1.9.0rc1] Looking in links: /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.8/var/lib/sage/wheels
[scipy-1.9.0rc1] ERROR: scipy-1.9.0rc1-cp38-cp38-macosx_12_x86_64.whl is not a supported wheel on this platform.

comment:16 Changed 3 months ago by Matthias Köppe

After renaming it from ...macosx_12... to ...macosx_12_0..., the wheel can be installed

comment:17 Changed 3 months ago by Matthias Köppe

comment:18 Changed 3 months ago by Matthias Köppe

Authors: Matthias Koeppe
Dependencies: #32423, #33789, #33866#32423, #33789, #33866, #34110
Description: modified (diff)
Summary: numpy 1.23.x, scipy 1.9.xscipy 1.9.x

comment:19 Changed 3 months ago by git

Commit: 9b4115490ddde5704d4fe56a601a3cde3b56fb96a4b03dd5a3ad7ff498ce0dc5606044a2696b9a4a

Branch pushed to git repo; I updated commit sha1. New commits:

a4b03ddbuild/pkgs/scipy/spkg-install.in: Back to sdh_pip_install

comment:20 Changed 3 months ago by Matthias Köppe

Dependencies: #32423, #33789, #33866, #34110#32423, #33789, #33866, #34110, #34119

comment:21 in reply to:  9 Changed 3 months ago by Matthias Köppe

Replying to mkoeppe:

Opened https://github.com/scipy/scipy/issues/16536 (BUG: 1.9.0rc1 fails to build on macOS when CXX="g++ -std=gnu++11")

Opened #34119 (src/bin/sage-env: Export CXXCPP)

comment:22 Changed 3 months ago by Matthias Köppe

Milestone: sage-9.7sage-9.8

comment:23 Changed 3 months ago by git

Commit: a4b03dd5a3ad7ff498ce0dc5606044a2696b9a4af83327a201e3182a0b43a83e5b3cd27d8f79cbd6

Branch pushed to git repo; I updated commit sha1. New commits:

4cce3efMerge tag '9.7.beta4' into t/34110/numpy_1_23
8f5860abuild/pkgs/numpy: Update to 1.23.1
5f6e21bMerge #34110
f83327abuild/pkgs/scipy: Use 1.9.0rc2

comment:24 in reply to:  17 Changed 2 months ago by Matthias Köppe

Replying to mkoeppe:

That's now https://github.com/FFY00/meson-python/issues/91

Reported as resolved in meson-python 0.8.0

comment:25 Changed 2 months ago by git

Commit: f83327a201e3182a0b43a83e5b3cd27d8f79cbd6eab320fea7d3db4ea103e17bd46f46dc91060c2d

Branch pushed to git repo; I updated commit sha1. New commits:

df36af4Merge tag '9.7.beta6' into t/34081/numpy_1_23_x__scipy_1_9_x
69045e3build/pkgs/meson_python: Update to 0.8.0
c51bdedbuild/pkgs/scipy: Update to 1.9.0rc3
eab320fbuild/pkgs/meson_python/patches/90.patch: Remove (upstreamed)

comment:26 Changed 2 months ago by Matthias Köppe

Indeed it creates a wheel with an acceptable filename now:

[scipy-1.9.0rc3]   Created wheel for scipy: filename=scipy-1.9.0rc3-cp39-cp39-macosx_12_0_x86_64.whl size=22657115 sha256=d2a44150f7bd42caf6610a06c9f4282430e4e8a119273fe42e9bea02d0aa0d10

comment:27 Changed 2 months ago by Matthias Köppe

Reviewers: https://github.com/mkoeppe/sage/actions/runs/2741873215, https://github.com/mkoeppe/sage/actions/runs/2741873210

comment:28 Changed 2 months ago by Matthias Köppe

Dependencies: #32423, #33789, #33866, #34110, #34119#34110

comment:29 Changed 2 months ago by Matthias Köppe

comment:30 Changed 2 months ago by git

Commit: eab320fea7d3db4ea103e17bd46f46dc91060c2d0223e55d5a128de83f90478528c8e6613cb8fc8b

Branch pushed to git repo; I updated commit sha1. New commits:

ee16112build/pkgs/patchelf: New (scipy/meson_py dependency)
a9eaae5tox.ini, build/bin/write-dockerfile.sh: Add 'tox -e docker-...-incremental'
b9bfbf9tox.ini: Add comment
a0f0f2eMerge #34228
4e88370build/pkgs/meson_python/dependencies: fixup
a830dfbbuild/pkgs/distro: New (skbuild dep)
0223e55build/pkgs/scikit_build: New (patchelf build dependency)

comment:31 Changed 2 months ago by Matthias Köppe

Dependencies: #34110#34110, #34228

comment:32 Changed 2 months ago by git

Commit: 0223e55d5a128de83f90478528c8e6613cb8fc8bc59f6d8cf6eccf5eb4e18d335e0fa17af97596e5

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

c59f6d8build/pkgs/patchelf: Switch to using non-Python package

comment:33 Changed 2 months ago by Matthias Köppe

Reviewers: https://github.com/mkoeppe/sage/actions/runs/2741873215, https://github.com/mkoeppe/sage/actions/runs/2741873210https://github.com/mkoeppe/sage/actions/runs/2743631300, https://github.com/mkoeppe/sage/actions/runs/2743631299

comment:34 Changed 2 months ago by Matthias Köppe

On debian-stretch-standard (https://github.com/mkoeppe/sage/runs/7532141718?check_suite_focus=true)

  [patchelf-0.15.0]   g++ -std=gnu++11 -DPACKAGE_NAME=\"patchelf\" -DPACKAGE_TARNAME=\"patchelf\" -DPACKAGE_VERSION=\"0.15.0\" -DPACKAGE_STRING=\"patchelf\ 0.15.0\" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -DPACKAGE=\"patchelf\" -DVERSION=\"0.15.0\" -I.    -Wall -std=c++17 -D_FILE_OFFSET_BITS=64     -g -O2 -c -o patchelf.o patchelf.cc
  [patchelf-0.15.0]   patchelf.cc:29:20: fatal error: optional: No such file or directory
  [patchelf-0.15.0]    #include <optional>
  [patchelf-0.15.0]                       ^

comment:35 Changed 2 months ago by Matthias Köppe

This is a C++17 feature, https://stackoverflow.com/questions/38253971/std-optional-no-such-file-or-directory. On this platform, gcc is 6.3.0-18+deb9u1 and apparently does not have sufficient C++17 support

comment:37 Changed 2 months ago by Matthias Köppe

Dependencies: #34110, #34228#34110, #34228, #34115

comment:38 Changed 2 months ago by git

Commit: c59f6d8cf6eccf5eb4e18d335e0fa17af97596e5b4e6615bff21507371fbd5afc8cafb9e0e8ad281

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

71948ba.github/workflows/docker.yml: Add opensuse-15.4, centos-stream-9
5b5c76e.github/workflows/ci-linux.yml: Change name to CI Linux
fb80dceMerge #33791
7ddad62.github/workflows/docker.yml: Fix input type
0ee5b2e.github/workflows/docker.yml: Pass inputs.target* to tox
7d4dd69.github/workflows/docker.yml: Use upstream artifact only if: inputs.upstream_artifact
c161c5cMerge tag '9.7.beta5' into t/34115/tox_yml__refactor_using_reusable_workflows
c75c499Merge tag '9.7.beta6' into t/34115/tox_yml__refactor_using_reusable_workflows
5cb365a.github/workflows/docker.yml: Reduce NUMPROC to 4
b4e6615Merge #34115

comment:39 Changed 2 months ago by Matthias Köppe

On debian-stretch-standard (https://github.com/mkoeppe/sage/runs/7545914147?check_suite_focus=true)

Unpacking ninja-build (1.7.2-1) ...

 ...
 ERROR: Could not detect Ninja v1.8.2 or newer

  A full log can be found at /sage/local/var/lib/sage/venv-python3.10.5/var/tmp/sage/build/meson_python-0.8.0/src/.mesonpy-7_1ejlan/build/meson-logs/meson-log.txt

comment:40 Changed 2 months ago by git

Commit: b4e6615bff21507371fbd5afc8cafb9e0e8ad2818f9135ea659fc59db1166a42a31bdccc2654ded4

Branch pushed to git repo; I updated commit sha1. New commits:

85797abbuild/pkgs/ninja_build: Update to 1.11.0
8f9135ebuild/pkgs/ninja_build/spkg-configure.m4: Set lower version bound to 1.8.2

comment:41 Changed 2 months ago by Matthias Köppe

Summary: scipy 1.9.xUpgrade scipy to 1.9.x, ninja_build to 1.11.0, add meson toolchain

comment:42 Changed 2 months ago by Matthias Köppe

Reviewers: https://github.com/mkoeppe/sage/actions/runs/2743631300, https://github.com/mkoeppe/sage/actions/runs/2743631299https://github.com/mkoeppe/sage/actions/runs/2748565810, https://github.com/mkoeppe/sage/actions/runs/2743631299

comment:43 Changed 2 months ago by Antonio Rojas

Dependencies: #34110, #34228, #34115#34110, #34228, #34115, #34246

comment:44 Changed 2 months ago by Matthias Köppe

Description: modified (diff)
Milestone: sage-9.8sage-9.7
Summary: Upgrade scipy to 1.9.x, ninja_build to 1.11.0, add meson toolchainUpgrade scipy to 1.9.0, ninja_build to 1.11.0, add meson toolchain

I'll rebase this away from the numpy upgrade (#34110)

comment:45 Changed 2 months ago by git

Commit: 8f9135ea659fc59db1166a42a31bdccc2654ded44fb3a63847487c0b205a4a02eed24a069f84513c

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

fd0eb69build/pkgs/scipy: Use 1.9.0rc2
f517441build/pkgs/meson_python: Update to 0.8.0
ee75343build/pkgs/scipy: Update to 1.9.0rc3
bfd5ff7build/pkgs/meson_python/patches/90.patch: Remove (upstreamed)
30c2c79build/pkgs/patchelf: New (scipy/meson_py dependency)
66d989abuild/pkgs/meson_python/dependencies: fixup
b8e8ad7build/pkgs/patchelf: Switch to using non-Python package
270789ebuild/pkgs/patchelf: Downgrade to 0.13.1
1c53962build/pkgs/ninja_build: Update to 1.11.0
4fb3a63build/pkgs/ninja_build/spkg-configure.m4: Set lower version bound to 1.8.2

comment:46 Changed 2 months ago by Matthias Köppe

Dependencies: #34110, #34228, #34115, #34246#34228, #34115, #34246
Description: modified (diff)
Status: newneeds_review

comment:47 Changed 2 months ago by Matthias Köppe

On 32bit, it builds a wheel that is named wrong

  [scipy-1.9.0rc3]   Using pip 22.1.2 from /sage/local/var/lib/sage/venv-python3.10.5/lib/python3.10/site-packages/pip (python 3.10)
  [scipy-1.9.0rc3]   Looking in links: /sage/local/var/lib/sage/venv-python3.10.5/var/lib/sage/wheels
  [scipy-1.9.0rc3]   ERROR: scipy-1.9.0rc3-cp310-cp310-linux_x86_64.whl is not a supported wheel on this platform.

https://github.com/mkoeppe/sage/runs/7599750782?check_suite_focus=true

comment:49 Changed 2 months ago by Matthias Köppe

After renaming the wheel file to scipy-1.9.0rc3-cp310-cp310-linux_i686.whl, it can be installed with pip, and

$ file venv/lib/python3.10/site-packages/scipy/**/*.so 
venv/lib/python3.10/site-packages/scipy/_lib/_ccallback_c.cpython-310-i386-linux-gnu.so:              ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=7ea4f88eb1db30a2264fa6414e073f9d42cb90c4, not stripped
venv/lib/python3.10/site-packages/scipy/_lib/_fpumode.cpython-310-i386-linux-gnu.so:                  ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=f9ae2041983e2b111529320c4b6b25d6fc1cd4d1, not stripped
...

confirms that this is a correct 32-bit build.

comment:50 Changed 2 months ago by Matthias Köppe

Status: needs_reviewneeds_work

comment:51 in reply to:  29 Changed 8 weeks ago by Matthias Köppe

Replying to mkoeppe:

mesonpy introduces a hidden dependency on patchelf on Linux systems https://github.com/FFY00/meson-python/blob/865183e46a3ed434c0c347c1ca752aec1ce6fce9/mesonpy/__init__.py#L927

see https://github.com/FFY00/meson-python/issues/125 (Only use and depend on patchelf if it is actually needed)

comment:52 Changed 8 weeks ago by git

Commit: 4fb3a63847487c0b205a4a02eed24a069f84513c90b07deb6a1f6f3b328cc77f84c71e56d07e04c0

Branch pushed to git repo; I updated commit sha1. New commits:

bcd9443Merge tag '9.7.beta7' into t/34081/numpy_1_23_x__scipy_1_9_x
a4c3821build/pkgs/meson_python/patches: Add https://github.com/FFY00/meson-python/pull/126
b354a38tox.ini, .github/workflows/docker.yml: Reimplement -incremental via SKIP_SYSTEM_PACKAGES
5decaddbuild/bin/write-dockerfile.sh: Do not use persistent env var PACKAGES (except on nix)
90b07deMerge #34228

comment:53 Changed 8 weeks ago by Matthias Köppe

Tested successfully with tox -e docker-ubuntu-bionic-i386-standard-incremental -- scipy

comment:54 Changed 8 weeks ago by Matthias Köppe

Status: needs_workneeds_review

comment:55 Changed 8 weeks ago by Matthias Köppe

The build errors are all fixed, let's get this in

comment:56 Changed 8 weeks ago by Matthias Köppe

Cc: Dima Pasechnik added

comment:57 Changed 8 weeks ago by Matthias Köppe

Cc: Antonio Rojas added

On archlinux-standard (https://github.com/mkoeppe/sage/runs/7659424080?check_suite_focus=true) I see

 [sagemath_doc_html-none]   [plotting ]   File "/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/scipy/sparse/csgraph/__init__.py", line 182, in <module>
  [sagemath_doc_html-none]   [plotting ]     from ._laplacian import laplacian
  [sagemath_doc_html-none]   [plotting ]   File "/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/scipy/sparse/csgraph/_laplacian.py", line 7, in <module>
  [sagemath_doc_html-none]   [plotting ]     from scipy.sparse.linalg import LinearOperator
  [sagemath_doc_html-none]   [plotting ]   File "/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/scipy/sparse/linalg/__init__.py", line 120, in <module>
  [sagemath_doc_html-none]   [plotting ]     from ._isolve import *
  [sagemath_doc_html-none]   [plotting ]   File "/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/scipy/sparse/linalg/_isolve/__init__.py", line 4, in <module>
  [sagemath_doc_html-none]   [plotting ]     from .iterative import *
  [sagemath_doc_html-none]   [plotting ]   File "/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/scipy/sparse/linalg/_isolve/iterative.py", line 9, in <module>
  [sagemath_doc_html-none]   [plotting ]     from . import _iterative
  [sagemath_doc_html-none]   [plotting ] ImportError: /sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/scipy/sparse/linalg/_isolve/_iterative.cpython-310-x86_64-linux-gnu.so: undefined symbol: slamch_
  [sagemath_doc_html-none]   [plotting ] /sage/src/sage/plot/plot.py:docstring of sage.plot.plot.multi_graphics:59: WARNING: Exception occurred in plotting plot-50

comment:58 Changed 8 weeks ago by Matthias Köppe

Cc: François Bissey Michael Orlitzky added

Also on gentoo-python3.9-standard (https://github.com/mkoeppe/sage/runs/7659423834?check_suite_focus=true) during the docbuild I see these strange errors:

  [sagemath_doc_html-none]   [plotting ] /sage/src/sage/plot/plot.py:docstring of sage.plot.plot.multi_graphics:59: WARNING: Exception occurred in plotting plot-50
  [sagemath_doc_html-none]   [plotting ]  from /sage/src/doc/en/reference/plotting/sage/plot/plot.rst:
  [sagemath_doc_html-none]   [plotting ] Traceback (most recent call last):
  [sagemath_doc_html-none]   [plotting ]   File "/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/scipy/__init__.py", line 134, in <module>
  [sagemath_doc_html-none]   [plotting ]     from scipy.__config__ import show as show_config
  [sagemath_doc_html-none]   [plotting ] ModuleNotFoundError: No module named 'scipy.__config__'
  [sagemath_doc_html-none]   [plotting ] The above exception was the direct cause of the following exception:
  [sagemath_doc_html-none]   [plotting ] Traceback (most recent call last):
  [sagemath_doc_html-none]   [plotting ]   File "/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/matplotlib/sphinxext/plot_directive.py", line 517, in _run_code
  [sagemath_doc_html-none]   [plotting ]     exec(code, ns)
  [sagemath_doc_html-none]   [plotting ]   File "<string>", line 2, in <module>
  [sagemath_doc_html-none]   [plotting ]   File "/sage/src/sage/misc/decorators.py", line 410, in wrapper
  [sagemath_doc_html-none]   [plotting ]     return func(*args, **kwds)
  [sagemath_doc_html-none]   [plotting ]   File "/sage/src/sage/misc/decorators.py", line 410, in wrapper
  [sagemath_doc_html-none]   [plotting ]     return func(*args, **kwds)
  [sagemath_doc_html-none]   [plotting ]   File "/sage/src/sage/misc/decorators.py", line 491, in wrapper
  [sagemath_doc_html-none]   [plotting ]     return func(*args, **options)
  [sagemath_doc_html-none]   [plotting ]   File "/sage/src/sage/plot/matrix_plot.py", line 565, in matrix_plot
  [sagemath_doc_html-none]   [plotting ]     import scipy.sparse as scipysparse
  [sagemath_doc_html-none]   [plotting ]   File "/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/scipy/__init__.py", line 139, in <module>
  [sagemath_doc_html-none]   [plotting ]     raise ImportError(msg) from e
  [sagemath_doc_html-none]   [plotting ] ImportError: Error importing SciPy: you cannot import SciPy while
  [sagemath_doc_html-none]   [plotting ]         being in scipy source directory; please exit the SciPy source
  [sagemath_doc_html-none]   [plotting ]         tree first and relaunch your Python interpreter.

comment:59 Changed 8 weeks ago by Matthias Köppe

I've opened https://github.com/FFY00/meson-python/issues/127 for this failure on gentoo

comment:60 Changed 8 weeks ago by git

Commit: 90b07deb6a1f6f3b328cc77f84c71e56d07e04c012698ea05dcf048574dbac4650900f8dc77a775f

Branch pushed to git repo; I updated commit sha1. New commits:

12698eabuild/pkgs/scipy: Update to 1.9.0

comment:61 in reply to:  57 ; Changed 8 weeks ago by Antonio Rojas

Replying to mkoeppe:

[sagemath_doc_html-none] [plotting ] ImportError?: /sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/scipy/sparse/linalg/_isolve/_iterative.cpython-310-x86_64-linux-gnu.so: undefined symbol: slamch_

I suppose this is caused by our openblas not including lapack symbols. Our distro package is fine since we haven't switched to meson (yet)

Last edited 8 weeks ago by Antonio Rojas (previous) (diff)

comment:62 in reply to:  61 Changed 8 weeks ago by Antonio Rojas

Replying to arojas:

I suppose this is caused by our openblas not including lapack symbols. Our distro package is fine since we haven't switched to meson (yet)

Indeed, building with -Dlapack=lapack fixes this when building the distro package with meson. Not sure how to fix this in sage.

comment:63 Changed 8 weeks ago by git

Commit: 12698ea05dcf048574dbac4650900f8dc77a775ff896ae19d78b17476dbe2079251832fa3b4f21cb

Branch pushed to git repo; I updated commit sha1. New commits:

f896ae1build/pkgs/meson_python: Update to 0.8.1

comment:64 Changed 8 weeks ago by git

Commit: f896ae19d78b17476dbe2079251832fa3b4f21cb9e64ba4905de7d5548e175dc6ee8a5f3a541a9aa

Branch pushed to git repo; I updated commit sha1. New commits:

9e64ba4build/pkgs/meson: Update to 0.63.0

comment:65 Changed 8 weeks ago by Matthias Köppe

Milestone: sage-9.7sage-9.8
Status: needs_reviewneeds_work

comment:66 in reply to:  58 Changed 8 weeks ago by François Bissey

Replying to mkoeppe:

Also on gentoo-python3.9-standard (https://github.com/mkoeppe/sage/runs/7659423834?check_suite_focus=true) during the docbuild I see these strange errors:

  [sagemath_doc_html-none]   [plotting ] /sage/src/sage/plot/plot.py:docstring of sage.plot.plot.multi_graphics:59: WARNING: Exception occurred in plotting plot-50
  [sagemath_doc_html-none]   [plotting ]  from /sage/src/doc/en/reference/plotting/sage/plot/plot.rst:
  [sagemath_doc_html-none]   [plotting ] Traceback (most recent call last):
  [sagemath_doc_html-none]   [plotting ]   File "/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/scipy/__init__.py", line 134, in <module>
  [sagemath_doc_html-none]   [plotting ]     from scipy.__config__ import show as show_config
  [sagemath_doc_html-none]   [plotting ] ModuleNotFoundError: No module named 'scipy.__config__'
  [sagemath_doc_html-none]   [plotting ] The above exception was the direct cause of the following exception:
  [sagemath_doc_html-none]   [plotting ] Traceback (most recent call last):
  [sagemath_doc_html-none]   [plotting ]   File "/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/matplotlib/sphinxext/plot_directive.py", line 517, in _run_code
  [sagemath_doc_html-none]   [plotting ]     exec(code, ns)
  [sagemath_doc_html-none]   [plotting ]   File "<string>", line 2, in <module>
  [sagemath_doc_html-none]   [plotting ]   File "/sage/src/sage/misc/decorators.py", line 410, in wrapper
  [sagemath_doc_html-none]   [plotting ]     return func(*args, **kwds)
  [sagemath_doc_html-none]   [plotting ]   File "/sage/src/sage/misc/decorators.py", line 410, in wrapper
  [sagemath_doc_html-none]   [plotting ]     return func(*args, **kwds)
  [sagemath_doc_html-none]   [plotting ]   File "/sage/src/sage/misc/decorators.py", line 491, in wrapper
  [sagemath_doc_html-none]   [plotting ]     return func(*args, **options)
  [sagemath_doc_html-none]   [plotting ]   File "/sage/src/sage/plot/matrix_plot.py", line 565, in matrix_plot
  [sagemath_doc_html-none]   [plotting ]     import scipy.sparse as scipysparse
  [sagemath_doc_html-none]   [plotting ]   File "/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/scipy/__init__.py", line 139, in <module>
  [sagemath_doc_html-none]   [plotting ]     raise ImportError(msg) from e
  [sagemath_doc_html-none]   [plotting ] ImportError: Error importing SciPy: you cannot import SciPy while
  [sagemath_doc_html-none]   [plotting ]         being in scipy source directory; please exit the SciPy source
  [sagemath_doc_html-none]   [plotting ]         tree first and relaunch your Python interpreter.

I just built sage-on-gentoo with scipy 1.9.0 without issue. It was with python 3.10 and ninja_build 1.10.2 though.

comment:67 Changed 8 weeks ago by François Bissey

Getting ninja to 1.11.0 didn't bring any issues in sage-on-gentoo.

comment:68 Changed 7 weeks ago by Matthias Köppe

Dependencies: #34228, #34115, #34246#34228, #34115, #34246, #32992

comment:69 Changed 7 weeks ago by Matthias Köppe

Summary: Upgrade scipy to 1.9.0, ninja_build to 1.11.0, add meson toolchainUpgrade scipy to 1.9.0, add meson toolchain

comment:70 Changed 4 weeks ago by Matthias Köppe

Dependencies: #34228, #34115, #34246, #32992#34228, #34115, #34246, #32992, #34449

comment:71 Changed 4 weeks ago by Matthias Köppe

New versions: scipy 1.9.1, meson 0.63.1, per update in https://github.com/FFY00/meson-python/issues/127#issuecomment-1235798227

comment:72 Changed 4 weeks ago by git

Commit: 9e64ba4905de7d5548e175dc6ee8a5f3a541a9aa3d7229e42a416b7f376ffe2a42da0b0941134d8f

Branch pushed to git repo; I updated commit sha1. New commits:

2253e81Merge tag '9.7.rc0' into t/34081/numpy_1_23_x__scipy_1_9_x
66e1f10build/pkgs/scipy: Update to 1.9.1
b85c2d6build/pkgs/meson: Update to 0.63.1
adddf8eMerge tag '9.7.beta8' into t/34228/tox__e_docker_____incremental
a90608fbuild/bin/write-dockerfile.sh: Invoke sage-package directly, do not go through ./sage
85ce9a6Merge tag '9.7.rc0' into t/34228/tox__e_docker_____incremental
3d7229eMerge #34228

comment:73 Changed 4 weeks ago by Matthias Köppe

The issue in comment:58 persists after these updates

comment:74 Changed 4 weeks ago by Matthias Köppe

Reviewers: https://github.com/mkoeppe/sage/actions/runs/2748565810, https://github.com/mkoeppe/sage/actions/runs/2743631299
Summary: Upgrade scipy to 1.9.0, add meson toolchainUpgrade scipy to 1.9.x, add meson toolchain

comment:75 Changed 3 weeks ago by Matthias Köppe

Dependencies: #34228, #34115, #34246, #32992, #34449#34228, #34115, #34246, #32992, #34449, #34344

comment:76 Changed 3 weeks ago by git

Commit: 3d7229e42a416b7f376ffe2a42da0b0941134d8faff3fcaf34e789eeef49b887576286dc66676464

Branch pushed to git repo; I updated commit sha1. New commits:

a2d8f2fbuild/pkgs/scipy/spkg-check.in: New
aff3fca.github/workflows/docker.yml: Lowercase for DOCKER_PUSH_REPOSITORY

comment:77 Changed 3 weeks ago by git

Commit: aff3fcaf34e789eeef49b887576286dc66676464a10bd2965e4dfdcbdb4f25d77932330ac981a80a

Branch pushed to git repo; I updated commit sha1. New commits:

a10bd29build/pkgs/scipy/spkg-check.in: Use scipy.test()

comment:78 Changed 3 weeks ago by git

Commit: a10bd2965e4dfdcbdb4f25d77932330ac981a80a074984448b625aaab4b334732a481d52bdcf3c0f

Branch pushed to git repo; I updated commit sha1. New commits:

0749844build/pkgs/scipy/dependencies_check: New - need pytest

comment:79 Changed 3 weeks ago by git

Commit: 074984448b625aaab4b334732a481d52bdcf3c0fdf0fb1590d7a1916ad4f5ad7b002e807517b1802

Branch pushed to git repo; I updated commit sha1. New commits:

df0fb15Merge tag '9.7.rc1' into t/34081/numpy_1_23_x__scipy_1_9_x
Note: See TracTickets for help on using tickets.