Opened 2 years ago

Last modified 2 months ago

#30371 needs_review enhancement

Use pipenv to create venv for development

Reported by: Matthias Köppe Owned by:
Priority: major Milestone: sage-9.8
Component: build Keywords: sd111
Cc: Tobias Diez, François Bissey, John Palmieri, Isuru Fernando Merged in:
Authors: Tobias Diez Reviewers:
Report Upstream: N/A Work issues:
Branch: public/build/inplace (Commits, GitHub, GitLab) Commit: c736e0cbccc8ddb9cc8359a48094aa3181b881fa
Dependencies: #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029, #30912, #31357, #31362, #31365, #31377 Stopgaps:

Status badges

Description (last modified by Tobias Diez)

We modify src/setup.py to support editable installs via pipenv (pipenv install -e ., which invokes pip install --editable). Since distribution continues using build/pkgs/sagelib/src/setup.py, the changes to src/setup.py made in this ticket do not affect the existing build process of the Sage library.

With this ticket, the following code works:

./bootstrap
./configure --enable-editable
make build-local
cd src
mkdir .venv
pipenv install --dev --ignore-pipfile
pipenv run pip install -e . --upgrade --exists-action=i --no-build-isolation
pipenv run sage-runtests --all

Instead of the first three commands, one can of course also use tox -e local-sudo-standard build-local. It generates a virtual env in src/.venv that contains all necessary Python packages, and an editable install of sage. Moreover, the installation of some packages might fail (depending on your system configuration). In this case one can manually build it using sage and then install the sage-built wheel in the pipenv. For example,

make fpylll
cd src && pipenv run pip install <sage>/local/var/lib/sage/venv-python3.8/var/lib/sage/wheels/fpylll-0.5.6-cp38-cp38-linux_x86_64.whl

The output of the added GH workflow is available here: https://github.com/sagemath/sagetrac-mirror/actions?query=workflow%3A%22Build+%26+Test+using+pipenv%22+branch%3Apublic%2Fbuild%2Finplace Most tests are working, but a few are failing. Only a few tests are failing:

sage -t --random-seed=0 sage/misc/sagedoc.py  # 4 doctests failed
sage -t --random-seed=0 sage/docs/conf.py  # 1 doctest failed
sage -t --random-seed=0 sage/matrix/matrix_integer_dense.pyx  # 1 doctest failed
sage -t --random-seed=0 sage/cpython/dict_del_by_value.pyx  # 2 doctests failed
sage -t --random-seed=0 sage/tests/cmdline.py  # 8 doctests failed

TODO (as followup tickets):

Previous tickets on in-place builds, editable installs:

Change History (489)

comment:1 Changed 2 years ago by Tobias Diez

Branch: public/build/inplace
Commit: bc6e2726cb87273de0c92297156dccf4b5107f06

I've worked a bit on it. The current commit is a semi-working prototype, in the sense that all cython files are successfully compiled to C files if invoked via python src/setup.py build_ext --inplace. The second compilation of the c files does not yet work.

I had problems that none of the source cython files were correctly found on my system (wsl) and that a few dependencies in the current build files were not met ('linbox', 'Singular', 'm4ri', 'zlib', 'cblas'). For this reason, I started from scratch and only included what was needed so far. Any pointers to how to install these dependencies (I think they are needed for the second compilation).


New commits:

bc6e272Semi-working prototype for inplace compilation

comment:2 Changed 2 years ago by Matthias Köppe

I don't really have time to look into this before the 9.2 release. But if you are looking for where include directories are determined, take a look at sage_setup.command.sage_build_cython

comment:3 Changed 2 years ago by git

Commit: bc6e2726cb87273de0c92297156dccf4b5107f0658452ce79c2d8863f139e72d75ec7b0c5904a989

Branch pushed to git repo; I updated commit sha1. New commits:

58452ceMake in-place compilation work

comment:4 Changed 2 years ago by Tobias Diez

The command ./sage src/setup.py build_ext --inplace runs now fine for me. I had to change quite a few things to make it work, and found it easier to completely start over using cythonize instead of using the existing build_cython code. The resulting code is a minimal working version. Please let me know which features of the existing build_cython file you would like to keep (to be honest, I don't really understand what some of the code there is doing, and why its needed).

The code definitely needs cleanup, but otherwise is ready for a first round of review.

comment:5 Changed 2 years ago by Tobias Diez

Status: newneeds_review

comment:6 Changed 2 years ago by git

Commit: 58452ce79c2d8863f139e72d75ec7b0c5904a9891a99228dafea2b7b676e1754b3d4b43f794ec7a7

Branch pushed to git repo; I updated commit sha1. New commits:

1a99228Update gitignore

comment:7 Changed 2 years ago by Matthias Köppe

--- a/.gitignore
+++ b/.gitignore
@@ -99,7 +99,18 @@ gitlab-build-docker.log
 /src/build
 /src/Makefile
 /src/bin/sage-env-config
-/build/bin/sage-build-env-config
+/build

this is not good. Our /build/ contains important files of the sage distribution

comment:8 Changed 2 years ago by Matthias Köppe

And the changes removing old python2 code really need to go on a separate ticket

comment:9 Changed 2 years ago by git

Commit: 1a99228dafea2b7b676e1754b3d4b43f794ec7a722b27a051b7f1a429b7f91f965b33154ee776aea

Branch pushed to git repo; I updated commit sha1. New commits:

22b27a0Remove global gitignore of build folder

comment:10 Changed 2 years ago by Tobias Diez

Ok, I was under the assumption that the build folder is for build artifacts. If this would be the case, then I suppose it would be better to whitelist those files in the build folder that needs to be checked-in. I've now replaced the global build ignore, by a more fine-grained statement which only ignores the "temporary" files.

Concerning the changes about old Python2 code, I've created #30397.

comment:11 in reply to:  10 Changed 2 years ago by Matthias Köppe

Replying to gh-tobiasdiez:

I was under the assumption that the build folder is for build artifacts.

No, not at all.

comment:12 Changed 2 years ago by Matthias Köppe

The changes I see in setup.py don't seem to fit so well into the plan I have regarding modularization of sagelib. See #29705 and #28925 (which needs help).

comment:13 Changed 2 years ago by Tobias Diez

I wasn't aware of your changes in #28925. I guess what I'm proposing is a different strategy for the source discovery. The approach in #28925 is to change the source files themselves and enrich them by some form of metadata that decides in which distribution / package they land. Here I was proposing to specify the distribution of the files in the setup.py file. I guess both strategies have pros and cons. The main reason for me was that it seems the responsibility of the setup.py file to construct the distribution; why should the source file need to care about where they are used/included? I.e classical example of separation of concerns. I think, it's also the more flexible approach since it's easy to include a source file into multiple distributions etc. Moreover, you also gain a bit of performance at build time since you don't need to parse the files for some magic comments. What do you think?

comment:14 in reply to:  13 ; Changed 2 years ago by Matthias Köppe

Replying to gh-tobiasdiez:

[...] Here I was proposing to specify the distribution of the files in the setup.py file. I guess both strategies have pros and cons. The main reason for me was that it seems the responsibility of the setup.py file to construct the distribution; why should the source file need to care about where they are used/included? I.e classical example of separation of concerns. [...]

One more thing to be aware of is that (1) setup.py by itself actually is not able to control what goes into the source distribution. In #29865 (where I prepare some experimental subset distributions), I additionally use MANIFEST.in to control that.

Separation of concerns is, of course, desirable. But so is locality: (2) My plan for splitting up the distribution is driven to a large extent by the dependencies on different sets of C/C++ libraries. And these are already encoded in Cython distutils directives.

Also I have explored a little bit the packaging options beyond using setuptools (in part motivated because of the shortcoming (1)). See #29854 (flit), #29810 (poetry). Unfortunately currently there does not seem to be a proper mainstream solution for the source layout that we want to keep (a monolithic source tree in src/sage). For example, flit (and even pip out-of-directory installs) do not handle symbolic links well. In the long term, I was hoping to transform sage_setup to implement the PEP 517 API directly (see #29845). That's another reason why I would be reluctant to hard-code distribution information in setup.py.

In any case, I am hoping to have a broader discussion about all of this on sage-devel at the beginning of the 9.3 cycle.

comment:15 in reply to:  14 ; Changed 2 years ago by Tobias Diez

Separation of concerns is, of course, desirable. But so is locality: (2) My plan for splitting up the distribution is driven to a large extent by the dependencies on different sets of C/C++ libraries. And these are already encoded in Cython distutils directives.

One could also take this as an argument to remove the distutil dependency directives and put them into the setup.py, giving you locality and separation of concerns.

Anyway, my goal with this PR is not to redesign or modularize the build. I just wanted to have a working inplace build. For this moving the distribution declaration to setup.py was the easiest option, and I quite liked this also from a conceptual point of view. But if the current way is preferred, I change it back of course. Anything else that needs to be changed?

comment:16 in reply to:  15 Changed 2 years ago by Matthias Köppe

Replying to gh-tobiasdiez:

One could also take this as an argument to remove the distutil dependency directives and put them into the setup.py, giving you locality and separation of concerns.

Ha, no, we just moved them there. Getting rid of the messy file modules_list.py was an achievement in the present cycle. See #29706

comment:17 in reply to:  15 Changed 2 years ago by Matthias Köppe

Replying to gh-tobiasdiez:

Anything else that needs to be changed?

I haven't really had a chance to look at it in more detail yet, sorry. Soon...

comment:18 Changed 2 years ago by git

Commit: 22b27a051b7f1a429b7f91f965b33154ee776aea55d00d7f055628848bbdeef9075b8574c44316b2

Branch pushed to git repo; I updated commit sha1. New commits:

55d00d7Merge branch 'develop' of git://trac.sagemath.org/sage into public/build/inplace

comment:19 Changed 2 years ago by Tobias Diez

Dependencies: #30397

comment:20 Changed 2 years ago by git

Commit: 55d00d7f055628848bbdeef9075b8574c44316b240f35033ce241f8a6bf7fe2307a8c8b946c981e1

Branch pushed to git repo; I updated commit sha1. New commits:

8404a5dRemove old python 2 code from cython files
e9c06e3Readd constants in build script
811b78cRemove unused import
32daec5Merge branch 'public/build/removeOldCode' of git://trac.sagemath.org/sage into public/build/inplace
40f3503Fix merge problem

comment:21 Changed 2 years ago by git

Commit: 40f35033ce241f8a6bf7fe2307a8c8b946c981e1851c557ccb33c61c78b005a20b908721deaf31a0

Branch pushed to git repo; I updated commit sha1. New commits:

851c557Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace

comment:22 Changed 2 years ago by git

Commit: 851c557ccb33c61c78b005a20b908721deaf31a01899dda1def19c3d7d518a5b4ccce67738415611

Branch pushed to git repo; I updated commit sha1. New commits:

1899ddaRemove unnessary changes

comment:23 Changed 2 years ago by git

Commit: 1899dda1def19c3d7d518a5b4ccce6773841561174c9f7d2a328df4a2f436d3523daa12dddf655a6

Branch pushed to git repo; I updated commit sha1. New commits:

74c9f7dCleanup code

comment:24 Changed 2 years ago by Tobias Diez

I've now changed the code to use the % sage_setup: distribution = xyz way. Moreover, I've cleanup up the code a bit.

Before I continue removing the code that is no longer needed, I would like to ask you which features of the current cythonize build script are missing from my new minimalistic implementation and should be reimplemented. (As I've said above, I don't really understand some of the code in build_cython.

comment:25 Changed 2 years ago by Matthias Köppe

These changes:

-# distutils: sources = sage/libs/ppl_shim.cc
+# distutils: sources = src/sage/libs/ppl_shim.cc

are not good. The root of the sagelib source tree is src, and paths are relative to this - this is where setup.py lives, after all.

As a way forward, I would suggest to actually work on #29864 instead of the present ticket. #29864 is a step in the modularization task (#29705), creating two new distributions - sage-core and sage-tdlib. sage-core is like sagelib minus sage-tdlib and without the fancy incremental installation tricks.

comment:26 Changed 2 years ago by Matthias Köppe

Status: needs_reviewneeds_work

comment:27 Changed 2 years ago by Tobias Diez

Thanks for the feedback. I'll try to find a way to specify the paths relative to src. I think my problem there was that setup.py is invoked from the central sage script, which sits in the root and not in the src folder.

Can you please explain the connection to #29864 in detail? For my the main point of this ticket is that I can compile the cython files to obtain compiled so files in the same folder as the original cython file (e.g. src/sage/rings). This is needed so that VS code can pick up these dependencies, resolve the imports, provide intellisense for it etc. My longterm goal would be to have a normal python installation (central or virtual env), point VS towards this runtime, and be able to use the sage code (in /src) without the need to use make at any point. How are #29864 and #29705 helping for this?

Last edited 2 years ago by Tobias Diez (previous) (diff)

comment:28 in reply to:  27 Changed 2 years ago by Matthias Köppe

Replying to gh-tobiasdiez:

Can you please explain the connection to #29864 in detail?

Package sage-core created in #29864 will be a version of sagelib with a plain setuptools build system, intended for (modular) deployment of the Sage library on PyPI. Since setuptools supports setup.py develop, this has synergy with your goal.

And we will be able to work on the build system of sage-core, across several tickets, without having to worry about breaking sagelib builds - or what current users/developers are expecting of it.

comment:29 Changed 2 years ago by Tobias Diez

That make sense, thanks for the explanation.

The only thing that I still don't understand is how the modularization will help with the compilation of cython files. If if everything is modularized, I guess the setup.py will still be used for this. And the point of this ticket is to improve the cyhonization...

comment:30 in reply to:  29 Changed 2 years ago by Matthias Köppe

Replying to gh-tobiasdiez:

The only thing that I still don't understand is how the modularization will help with the compilation of cython files. If if everything is modularized, I guess the setup.py will still be used for this.

Simply by having a separate setup.py in build/pkgs/sage-core/src.

comment:31 Changed 2 years ago by Matthias Köppe

Dependencies: #30397#29500, #30527
Description: modified (diff)
Summary: In-place installs with "./sage -pip install --editable src/"In-place (editable) installs of sagelib in a venv

comment:32 Changed 2 years ago by Matthias Köppe

Slight change of plans, forget about sage-core.

comment:33 Changed 2 years ago by git

Commit: 74c9f7d2a328df4a2f436d3523daa12dddf655a65bda3321b88289cbd3f0dfe97308b07dd9dd1459

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

0a64674build/pkgs/gambit/spkg-install.in: Install via bdist_wheel
ca58693build/pkgs/pillow/spkg-install.in: Install via bdist_wheel
207d80fbuild/pkgs/pip2pi: New
2555e19build/make/install: At the end, update the repository index
5a747c4build/bin/sage-pip-{install,uninstall}: Remove pip2 support
e66243fMerge branch 't/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels' into t/30527/pep-503-simple-repository-for-wheels
9937126Merge branch 't/30527/pep-503-simple-repository-for-wheels' into t/30371/public/build/inplace
1a39442build/pkgs/sagelib/src/setup.py: Replace symlink to SAGE_SRC/setup.py by a copy
54fe0c7src/tox.ini, src/MANIFEST.in: Move here from build/pkgs/sagelib/src, leave symlinks behind
5bda332In-place (editable) installs of sagelib (#30371 squashed)

comment:34 Changed 2 years ago by Matthias Köppe

TODO:

  • The changes to src/sage/cpython/debugimpl.c need to go on a separate ticket
  • The changes to src/sage/misc/persist.pyx need to go on a separate ticket
  • Don't change find_python_sources because build/pkgs/sagelib/src/setup.py continues to use it

comment:35 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:36 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:37 Changed 2 years ago by Matthias Köppe

Dependencies: #29500, #30527#29500, #30527, #22731

comment:38 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:39 Changed 2 years ago by git

Commit: 5bda3321b88289cbd3f0dfe97308b07dd9dd1459274538190984ea38145ffdb4d7d1defc88784124

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

67c56a6src/requirements.txt, src/MANIFEST.in: Move here from build/pkgs/sagelib/src, leave symlinks behind
2745381In-place (editable) installs of sagelib (#30371 squashed)

comment:40 Changed 2 years ago by Matthias Köppe

Dependencies: #29500, #30527, #22731#29500, #30527, #22731, #30578
Description: modified (diff)

comment:41 in reply to:  34 Changed 2 years ago by Matthias Köppe

Replying to mkoeppe:

TODO:

  • The changes to src/sage/cpython/debugimpl.c need to go on a separate ticket

This is now #30590

comment:42 Changed 2 years ago by git

Commit: 274538190984ea38145ffdb4d7d1defc88784124a493e6e2a972ab5344fdd3a953376278e807cf3b

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

851c557Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace
1899ddaRemove unnessary changes
74c9f7dCleanup code
81fdcf3Make distutils relative to src again
faf6777Fail gracefully for missing packages
8af6e23Add minimal pipfile
456589cIgnore build files
59e8361Make it compile by ignoring a lot of errors
054fedcAdd psutil and pexpect
a493e6eMerge branch 'public/build/inplace' of git://trac.sagemath.org/sage into public/build/inplace

comment:43 Changed 2 years ago by git

Commit: a493e6e2a972ab5344fdd3a953376278e807cf3bc5f3873cc7ddb158646051861fc10c72c6be22f5

Branch pushed to git repo; I updated commit sha1. New commits:

c5f3873Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace

comment:44 Changed 2 years ago by Tobias Diez

I have now added a minimal pipfile including sage as an editable install. So running pipenv install results in a semi-workable virtual environment. "Semi" because not all pyx files could be compiled successfully since they rely on external dependencies (e.g. singular, giac, ecl, ratpoints, ...).

For now, I've excluded the relevant pyx files from the compilation. Any ideas how to handle these dependencies are welcome! One approach would be to treat them similar to the the optional packages by putting a "sage_setup distribution" tag to the pyx file.

I also tried to run pipenv install directly under Windows, but that's not possible at the moment as some of the dependencies cannot be installed on Windows currently: https://github.com/sagemath/cypari2/issues/19 https://github.com/sagemath/cysignals/pull/104

Last edited 2 years ago by Tobias Diez (previous) (diff)

comment:45 Changed 2 years ago by git

Commit: c5f3873cc7ddb158646051861fc10c72c6be22f5b7fb5a9d46d8acb5068cde9a94009adb800d4783

Branch pushed to git repo; I updated commit sha1. New commits:

8a7ec88Revert format changes in debug.pyx
ba58ffaMake distutil imports relative to src (again?)
b7fb5a9Incorporate upstream changes

comment:46 in reply to:  44 Changed 2 years ago by Matthias Köppe

Replying to gh-tobiasdiez:

[...] not all pyx files could be compiled successfully since they rely on external dependencies (e.g. singular, giac, ecl, ratpoints, ...). One approach would be to treat them similar to the the optional packages by putting a "sage_setup distribution" tag to the pyx file.

Yes, exactly this approach is part of the "modularization" that I have in mind. See #29705 under the items titled "Deploy mildly modularized distributions", "Further modularization" for a sketch of such distributions. Basically, one distribution for each major C/C++ library (such as sage-ntl) - but some libraries come in packs that would not make much sense to separate (such as sage-flint-arb-e_antic).

comment:47 in reply to:  44 Changed 2 years ago by Matthias Köppe

Replying to gh-tobiasdiez:

I also tried to run pipenv install directly under Windows, but that's not possible at the moment as some of the dependencies cannot be installed on Windows currently: https://github.com/sagemath/cypari2/issues/19 https://github.com/sagemath/cysignals/pull/104

Right, I hope that when modularization is done, a native Windows port of a tiny Sage subset would be an actionable project.

Last edited 2 years ago by Matthias Köppe (previous) (diff)

comment:48 Changed 2 years ago by Matthias Köppe

As in #30578 (where I am trying the same with requirements.txt and plain venv), I would like to get the following to work:

  1. Build the sage distribution (actually all prerequisites of sagelib, using make sagelib-build-deps)
  1. Then within sage -sh (where all the libraries installed in $SAGE_LOCAL are available) do the in-place install into a new venv created by pipenv - using the wheels built in $SAGE_LOCAL/var/lib/sage/wheels.
Last edited 2 years ago by Matthias Köppe (previous) (diff)

comment:49 Changed 2 years ago by Tobias Diez

Thanks for the quick feedback.

I agree the modularization is a good way to approach these problems. Sadly, I feel like this requires too much in-depth knowledge of sage for me to contribute directly. But after skimming the tickets, I suggest you have a look at my changes here to the way distributions are excluded. This seems to be somewhat opposite to say #29864 where this is handled by adding a new setup-tdlib/setup.py. For the in-place installation it is important that the src/setup.py results in a working installation.

What do you suggest as the way forward to align this ticket with the modularization effort?

As for #30578, my motivation here is a bit different: I wanted to have a prototype for sagelib with pipenv that works without ever invoking sage/make and directly uses the versions of the dependencies found on pypi instead of (re)building them locally. This seems to work in principle, and might lead to a workable solution for most purposes of development and ci testing, where the advantages of building the dependencies (like optimization) are not that important. That being said, I'm not sure if we really need to have a requirements.txt and a pipfile. I would opt for the more modern version and only keep the pipfile.

After installing singular via apt-get install libsingular4-dev libpynac-dev, a few more pyx files could be compiled. Should I add this in build\pkgs\singular\distros\debian ?

One more thing: Currently sage/rings/number_field/number_field_element_quadratic.pyx relies on the arb lib, but is also (indirectly) imported in sage.all. What's the best way to handle these imports that depend on external C/C++ libs? For now I've uncommented some of these all-imports.

comment:50 Changed 2 years ago by git

Commit: b7fb5a9d46d8acb5068cde9a94009adb800d478312e6d6711ca76bee221336093fa03e162962a896

Branch pushed to git repo; I updated commit sha1. New commits:

cc2d6ffUncomment non-working all imports
9d9bf05Reenable pyx compilation depending on singular
12e6d67Fix imports

comment:51 Changed 2 years ago by Tobias Diez

Yet another question: when/how is src/sage/ext/interpreters generated?

comment:52 Changed 2 years ago by git

Commit: 12e6d6711ca76bee221336093fa03e162962a8967a7e5764d619f0207c9d57d1c24b6198f944b799

Branch pushed to git repo; I updated commit sha1. New commits:

7a7e576Make import sage.manifolds.all work

comment:53 in reply to:  49 Changed 2 years ago by Matthias Köppe

Replying to gh-tobiasdiez:

As for #30578, my motivation here is a bit different: I wanted to have a prototype for sagelib with pipenv that works without ever invoking sage/make and directly uses the versions of the dependencies found on pypi instead of (re)building them locally.

Yes, I agree with this goal. But I would strongly suggest to focus on a smaller scope for this ticket: Make the in-place install work when all the dependencies are available. At the same time, make sure that the normal Sage build process does not get broken (e.g. take care of comment 34 "Don't change find_python_sources because build/pkgs/sagelib/src/setup.py continues to use it"). This ensures that the ticket can be merged in Sage, thus, a way forward. This is better than preparing one ticket with huge changes.

comment:54 in reply to:  49 ; Changed 2 years ago by Matthias Köppe

Replying to gh-tobiasdiez:

I suggest you have a look at my changes here to the way distributions are excluded. This seems to be somewhat opposite to say #29864 where this is handled by adding a new setup-tdlib/setup.py.

As discussed, adding these sage_setup: distribution tags to Cython source files is a way forward that aligns with the modularization effort.

comment:55 in reply to:  49 ; Changed 2 years ago by Matthias Köppe

Replying to gh-tobiasdiez:

Currently sage/rings/number_field/number_field_element_quadratic.pyx relies on the arb lib, but is also (indirectly) imported in sage.all. What's the best way to handle these imports that depend on external C/C++ libs?

Try lazy imports with PythonModule features, from #30616/#30647.

comment:56 in reply to:  49 Changed 2 years ago by Matthias Köppe

Replying to gh-tobiasdiez:

I suggest you have a look at my changes here to the way distributions are excluded.

Well, it's the same as before in sagelib's setup.py, filtering by distributions, plus some hardcoded ones. Except that you have changed the interface of the function that does the filtering (see above - this change is not mergeable because build/pkgs/sagelib/src/setup.py needs to keep working!)

comment:57 in reply to:  54 Changed 2 years ago by Matthias Köppe

Replying to mkoeppe:

As discussed, adding these sage_setup: distribution tags to Cython source files is a way forward that aligns with the modularization effort.

I've opened #30666 for this. (The new distribution names have to be added to setup.py, of course, so that the files continue to be built in the Sage distribution.)

comment:58 Changed 2 years ago by Matthias Köppe

Dependencies: #29500, #30527, #22731, #30578#29500, #30527, #22731, #30578, #30666

comment:59 in reply to:  55 Changed 2 years ago by Matthias Köppe

Replying to mkoeppe:

Replying to gh-tobiasdiez:

Currently sage/rings/number_field/number_field_element_quadratic.pyx relies on the arb lib, but is also (indirectly) imported in sage.all. What's the best way to handle these imports that depend on external C/C++ libs?

Try lazy imports with PythonModule features, from #30616/#30647.

For another approach regarding the ".all" imports, see #29865. Probably we will need a combination of this and lazy imports.

comment:60 Changed 2 years ago by git

Commit: 7a7e5764d619f0207c9d57d1c24b6198f944b7994afd411dd86c37e90ca32c8f77de89a91c57a667

Branch pushed to git repo; I updated commit sha1. New commits:

4afd411Readd old find_python_sources

comment:61 Changed 2 years ago by Tobias Diez

I've now restored the old find_python_sources method. Is there something else that needs to be done, except waiting for the other tickets to be finished?

comment:62 Changed 2 years ago by Tobias Diez

Description: modified (diff)

comment:63 Changed 2 years ago by Matthias Köppe

Nothing immediate. This is going in a good direction

comment:64 Changed 2 years ago by git

Commit: 4afd411dd86c37e90ca32c8f77de89a91c57a667c82bbd56e5ab828b73dfe09ad1c32c9b7d96f866

Branch pushed to git repo; I updated commit sha1. New commits:

849350cFix missing ccobject
ee8df98Update gitignore and add vs code config
c82bbd5Add pycodestyle dep

comment:65 in reply to:  64 Changed 2 years ago by Matthias Köppe

Replying to git:

Branch pushed to git repo; I updated commit sha1. New commits:

849350cFix missing ccobject

Created #30672 (Remove sage/ext from sage_include_directories) for this

comment:66 Changed 2 years ago by Matthias Köppe

Dependencies: #29500, #30527, #22731, #30578, #30666#29500, #30527, #22731, #30578, #30666, #30672

comment:67 Changed 2 years ago by Tobias Diez

Dependencies: #29500, #30527, #22731, #30578, #30666, #30672#29500, #30527, #22731, #30578, #30666, #30672, #30673

comment:68 Changed 2 years ago by Tobias Diez

Description: modified (diff)

comment:69 Changed 2 years ago by git

Commit: c82bbd56e5ab828b73dfe09ad1c32c9b7d96f86662930ee7ada36664e0fc2264e5395f8b8337b650

Branch pushed to git repo; I updated commit sha1. New commits:

62930eeAdd ipykernel and tox as dev packages

comment:70 Changed 2 years ago by Tobias Diez

Description: modified (diff)

New commits:

62930eeAdd ipykernel and tox as dev packages

comment:71 Changed 2 years ago by Tobias Diez

Dependencies: #29500, #30527, #22731, #30578, #30666, #30672, #30673#29500, #30527, #22731, #30578, #30672, #30673

comment:72 Changed 2 years ago by git

Commit: 62930ee7ada36664e0fc2264e5395f8b8337b650cd196a0773e36e7e6c1fbe21962ef5b1b1485cf5

Branch pushed to git repo; I updated commit sha1. New commits:

3cbb457Reenable compilation of most files relying on external libs
8174e1aReenable imports
724223fDisable non-working imports
cd196a0Disable things in ecl integration (probably because of too old ecl lib)

comment:73 Changed 2 years ago by Tobias Diez

Description: modified (diff)

comment:74 Changed 2 years ago by Matthias Köppe

This ticket is way too big. I really strong recommend to get a working in-place installation of sagelib in the situation when its dependencies have been built using the Sage distribution.

comment:75 Changed 2 years ago by Tobias Diez

I've now followed your advice, and tried to have a prototype that works if all library dependencies are fulfilled (instead of adding exceptions / distribution annotations everywhere). And I was successful indeed, import sage.all works without error and I can do basic calculations with the manifold package. This required tinkering with a lot of system-installed packages (instructions added in the ticket).

The following is not yet working, and I would like to get input on how to resolve this:

  • sage/ext/interpreters is empty. Where is the code generated? Can this easily be extracted to a script that can be invoked separately?
  • The code relying on the arb library still doesn't compile although I've libflint-arb-dev installed. The compilation fails with ld larb was not found. Why is this, and what's the best way to solve it?
  • The sage.doctest package conflicted with the sympy (I believe), i.e. sympy tried to load classes from sage.doctest for some reason, and couldn't locate them. I've fixed this by renaming src/sage/doctest to sage-doctest. Is the doctest package really required under the sage folder, or could it be e.g moved up to src/doctest?

I agree the ticket is too big at the moment. Once I found a solution to the above points, I'll clean the code. I sadly cannot work with the dependencies created by the sage distribution, as the build process fails due to compilation errors in the pip library...

comment:76 Changed 2 years ago by git

Commit: cd196a0773e36e7e6c1fbe21962ef5b1b1485cf51422cf461178919c2ccf1a30b6a82e1ac5dee396

Branch pushed to git repo; I updated commit sha1. New commits:

1422cf4Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace

comment:77 in reply to:  75 Changed 2 years ago by Matthias Köppe

Replying to gh-tobiasdiez:

I've now followed your advice, and tried to have a prototype that works if all library dependencies are fulfilled (instead of adding exceptions / distribution annotations everywhere). And I was successful indeed, import sage.all works without error and I can do basic calculations with the manifold package. This required tinkering with a lot of system-installed packages (instructions added in the ticket).

Great.

The following is not yet working, and I would like to get input on how to resolve this:

  • sage/ext/interpreters is empty. Where is the code generated? Can this easily be extracted to a script that can be invoked separately?

sage_setup.command.sage_build

  • The code relying on the arb library still doesn't compile although I've libflint-arb-dev installed. The compilation fails with ld larb was not found. Why is this, and what's the best way to solve it?

Use sage.env.cython_aliases. In particular ARB_LIBRARY gives you the name. (The Debian installation uses a different name from upstream.)

  • The sage.doctest package conflicted with the sympy (I believe), i.e. sympy tried to load classes from sage.doctest for some reason, and couldn't locate them. I've fixed this by renaming src/sage/doctest to sage-doctest. Is the doctest package really required under the sage folder, or could it be e.g moved up to src/doctest?

Sounds like a bad PYTHONPATH. src/sage should not be in it.

I agree the ticket is too big at the moment. Once I found a solution to the above points, I'll clean the code. I sadly cannot work with the dependencies created by the sage distribution, as the build process fails due to compilation errors in the pip library...

Hm, that's still happening? Sorry I lost track of this since various other things seemed to break on WSL. Have you tried with downgrading/upgrading the pip package?

comment:78 Changed 2 years ago by git

Commit: 1422cf461178919c2ccf1a30b6a82e1ac5dee39605713b87fff7948a68562358a5d32a29701bfa2f

Branch pushed to git repo; I updated commit sha1. New commits:

05713b8Remove dependency on #30527

comment:79 Changed 2 years ago by Matthias Köppe

By the way, for dev purposes until the WSL issues are resolved, you could try running Sage in Docker instead (bind-mount the source tree into it)

comment:80 Changed 2 years ago by Tobias Diez

Dependencies: #29500, #30527, #22731, #30578, #30672, #30673#29500, #22731, #30672, #30673

New commits:

05713b8Remove dependency on #30527

New commits:

05713b8Remove dependency on #30527

comment:81 Changed 2 years ago by Tobias Diez

Dependencies: #29500, #22731, #30672, #30673#29500, #30672, #30673

comment:82 Changed 2 years ago by git

Commit: 05713b87fff7948a68562358a5d32a29701bfa2fac07cc0d005de5e067e29c22f154521537f5906d

Branch pushed to git repo; I updated commit sha1. New commits:

ac07cc0Move requirements file back

comment:83 in reply to:  description Changed 2 years ago by Matthias Köppe

  • Fix installation of ecl:

After apt-get install ecl ecl is broken under WSL. To fix this, first run apt-get install libtool autoconf, then follow https://github.com/rdp/ffmpeg-windows-build-helpers/issues/452#issuecomment-638639182.

Could you try the related #30629 on WSL please?

comment:84 Changed 2 years ago by Tobias Diez

Dependencies: #29500, #30672, #30673#29500, #30672, #30673, #30590
Description: modified (diff)

comment:85 Changed 2 years ago by git

Commit: ac07cc0d005de5e067e29c22f154521537f5906d3023260e2c36e80113daf0af69e95d6091de461e

Branch pushed to git repo; I updated commit sha1. New commits:

9141ae9Reenable doctests
cb50197Fix compilation of interpreters
3023260Renable all imports

comment:86 Changed 2 years ago by git

Commit: 3023260e2c36e80113daf0af69e95d6091de461e5b12a924507cf85a2eca749993835927b9398668

Branch pushed to git repo; I updated commit sha1. New commits:

e620323Remove unnessary files
6502495Reenable ppl
5b12a92Cleanup setup script

comment:87 Changed 2 years ago by Tobias Diez

Description: modified (diff)
Status: needs_workneeds_review

comment:88 Changed 2 years ago by Tobias Diez

Perfect, thanks a lot Matthias! I got now a working installation of all libraries, and could import sage.all without the need to uncomment any imports. Yeaaaah!

I've also cleaned the code and extended the ticket description, so this ticket is now ready for review.

comment:89 Changed 2 years ago by Matthias Köppe

Could you put the changes to sage_setup/autogen/interpreters/ on a separate ticket please?

comment:90 Changed 2 years ago by Matthias Köppe

The changes regarding imports (mostly around sage.rings) should also be on a separate ticket.

comment:91 Changed 2 years ago by Matthias Köppe

Also for the changes to src/sage/libs/eclsig.h -- which will certainly need discussion

comment:92 Changed 2 years ago by Matthias Köppe

+compile_time_env = dict(
+    PY_PLATFORM=sys.platform
+)
+cython_directives = dict(
+    language_level="3str",
+    cdivision=True,
+)

This duplicates settings defined in sage_setup.command.sage_build_cython. Could you refactor this so that these directives are defined in a reusable way? Perhaps in a new module sage_setup.cython_options

Last edited 2 years ago by Matthias Köppe (previous) (diff)

comment:93 Changed 2 years ago by Matthias Köppe

Authors: Tobias Diez

comment:94 in reply to:  92 Changed 2 years ago by Matthias Köppe

Replying to mkoeppe:

Could you refactor this so that these directives are defined in a reusable way? Perhaps in a new module sage_setup.cython_options

... in a separate ticket of course.

comment:95 Changed 2 years ago by Tobias Diez

Will do so. For the cython options, they are actually only a small subset of the ones in sage_build_cython. Should I still refactor? (Feels a bit strange as there are more differences than similarities.) But I'm also not sure if the other cython options are needed...

comment:96 in reply to:  95 Changed 2 years ago by Matthias Köppe

Replying to gh-tobiasdiez:

Will do so. For the cython options, they are actually only a small subset of the ones in sage_build_cython. Should I still refactor? (Feels a bit strange as there are more differences than similarities.) But I'm also not sure if the other cython options are needed...

Well, the idea would be to see whether you can just build with the options that Sage uses instead of using your own set of options.

comment:97 Changed 2 years ago by Matthias Köppe

Whether other Cython build options can/should be used - should be orthogonal to whether an in-place or regular build is done. So it should not be done on this ticket.

comment:98 in reply to:  description Changed 2 years ago by Matthias Köppe

Some comments on items in the ticket description:

TODO (as followup tickets):

  • Automatically generate the apt-get install command with the necessary libraries (add them as distros/debian.txt?)

./configure takes care of all of this. Do you plan to do system interface configuration and system package advice without using ./configure (or, more generally, Unix shell scripts)?

A way forward is to reimplement (cleaner version of) some of the simple shell code for system package advice as Python code in sage_bootstrap. See #29146.

  • Remove ARB_LIBRARY env variable, and move check from arb/config.m4 to env.py

My suggestion: Instead of discussing to remove this variable (or other configuration variables that are set by ./configure), rather discuss different ways of generating the sage_conf module / or providing different implementations of the interface defined by the sage_conf module for different use cases (such as supporting a platform that does not have a Unix shell).

  • Move package dependencies from pipfile to install_requires in src/setup.py.

Best to put them in src/setup.cfg instead of src/setup.py (see #30578)

comment:99 Changed 2 years ago by git

Commit: 5b12a924507cf85a2eca749993835927b93986688abab5dfb588a3b6e6fe0c1d32525656ddc13cf6

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

82e8b7dNarrow down to essenital changes
8abab5dAdd copy of setup.py

comment:100 Changed 2 years ago by Matthias Köppe

Thanks for preparing this cleaned up branch.

Trying to test it, I seem to be running into something that looks like https://discuss.python.org/t/pip-19-1-and-installing-in-editable-mode-with-pyproject-toml/1553 : Something is generating a pyproject.toml in src/ and then it dies with

Writing supplied requirement line to temporary file: '-e .'
Installing 'e1839a8'
$ ['/Users/mkoeppe/.local/share/virtualenvs/src-BXov0SKG/bin/pip', 'install', '--verbose', '--upgrade', '--no-use-pep517', '--no-deps', '--exists-action=i', '-r', '/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pipenv-_yopr9l3-requirements/pipenv-7uv8tb0m-requirement.txt', '-i', 'https://pypi.org/simple']
Using source directory: '/Users/mkoeppe/.local/share/virtualenvs/src-BXov0SKG/src'
Using pip 20.2.1 from /Users/mkoeppe/.local/share/virtualenvs/src-BXov0SKG/lib/python3.8/site-packages/pip (python 3.8)
Non-user install by explicit request
...
  File "/Users/mkoeppe/.local/share/virtualenvs/src-BXov0SKG/lib/python3.8/site-packages/pip/_internal/distributions/sdist.py", line 33, in prepare_distribution_metadata
    self.req.load_pyproject_toml()
  File "/Users/mkoeppe/.local/share/virtualenvs/src-BXov0SKG/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 503, in load_pyproject_toml
    pyproject_toml_data = load_pyproject_toml(
  File "/Users/mkoeppe/.local/share/virtualenvs/src-BXov0SKG/lib/python3.8/site-packages/pip/_internal/pyproject.py", line 95, in load_pyproject_toml
    raise InstallationError(
pip._internal.exceptions.InstallationError: Disabling PEP 517 processing is invalid: project specifies a build backend of setuptools.build_meta in pyproject.toml
Removed file:///Users/mkoeppe/s/sage/sage-rebasing/src (from -r /var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pipenv-_yopr9l3-requirements/pipenv-7uv8tb0m-requirement.txt (line 1)) from build tracker '/private/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pip-req-tracker-lxrv7bcd'
Removed build tracker: '/private/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pip-req-tracker-lxrv7bcd'

comment:101 Changed 2 years ago by Tobias Diez

Strange. For some reason, for you pip is invoked with --no-use-pep517. For me it is not:

$ "/mnt/d/Programming/Projects/sage/src/.venv/bin/pip" install   --verbose  --no-deps  -e "." -i https://pypi.org/simple --exists-action w...

What version of pipenv do you use? For me: pipenv 11.9.0 (and the pip in the virtual env also differs: pip 20.0.2)

comment:102 Changed 2 years ago by Matthias Köppe

I'm using pipenv from homebrew, which is version 2020.8.13

comment:103 Changed 2 years ago by Tobias Diez

I will try to update my pipenv later. For now you can have a look if setting the environment variable PIP_USE_PEP517 to true resolves the issue.

comment:104 Changed 2 years ago by Matthias Köppe

I'll try, but I think editable installs are not compatible with PEP517

comment:105 Changed 2 years ago by Matthias Köppe

No change after setting this environment variable

comment:106 Changed 2 years ago by Tobias Diez

Dependencies: #29500, #30672, #30673, #30590#29500, #30672, #30673, #30590, #30709

comment:107 Changed 2 years ago by Tobias Diez

I had a closer look at this, and it turns out that the first time pip is correctly called without any --no-use-pep517 argument, only on the retry this argument is used. Thus, the error actually shadows another error that occurs during the initial pip install.

Can you thus run the following pipenv run pip install -e . --no-build-isolation after pipenv install. This should point to the real issue why the editable install fails.

I've created a github workflow because I don't have a mac. https://github.com/tobiasdiez/sage/blob/public/build/inplace/.github/workflows/ci-pipenv.yml It shows the same problem, the pipenv install fails apparently due to a pep517 issue, but the editable install actually fails due to missing interpreters and other dependencies (which requires manual installations as in the ticket description).

comment:108 Changed 2 years ago by Matthias Köppe

Hm...

(sage-sh) mkoeppe@egret:src$ pipenv run pip install -e . --no-build-isolation
Obtaining file:///Users/mkoeppe/s/sage/sage-rebasing/src
    Preparing wheel metadata ... error
    ERROR: Command errored out with exit status 1:
     command: /Users/mkoeppe/.local/share/virtualenvs/src-BXov0SKG/bin/python /Users/mkoeppe/.local/share/virtualenvs/src-BXov0SKG/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/tmpvt150u93
         cwd: /Users/mkoeppe/s/sage/sage-rebasing/src
    Complete output (14 lines):
    Traceback (most recent call last):
      File "/Users/mkoeppe/.local/share/virtualenvs/src-BXov0SKG/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
        main()
      File "/Users/mkoeppe/.local/share/virtualenvs/src-BXov0SKG/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/Users/mkoeppe/.local/share/virtualenvs/src-BXov0SKG/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 133, in prepare_metadata_for_build_wheel
        return hook(metadata_directory, config_settings)
      File "/Users/mkoeppe/.local/share/virtualenvs/src-BXov0SKG/lib/python3.8/site-packages/setuptools/build_meta.py", line 161, in prepare_metadata_for_build_wheel
        self.run_setup()
      File "/Users/mkoeppe/.local/share/virtualenvs/src-BXov0SKG/lib/python3.8/site-packages/setuptools/build_meta.py", line 145, in run_setup
        exec(compile(code, __file__, 'exec'), locals())
      File "setup.py", line 23, in <module>
        import sage.env
    ModuleNotFoundError: No module named 'sage'
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/mkoeppe/.local/share/virtualenvs/src-BXov0SKG/bin/python /Users/mkoeppe/.local/share/virtualenvs/src-BXov0SKG/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/tmpvt150u93 Check the logs for full command output.

comment:109 Changed 2 years ago by Tobias Diez

No idea....for the github action this is not a problem: https://github.com/tobiasdiez/sage/runs/1205452529?check_suite_focus=true. Maybe pipenv run pip install -e . --verbose --upgrade --exists-action=i --no-build-isolation works, or you need to clean the venv first (pipenv -rm).

comment:110 Changed 2 years ago by Matthias Köppe

Sure, I'll play with it more in the next days.

comment:111 Changed 2 years ago by Tobias Diez

Description: modified (diff)

comment:112 Changed 2 years ago by git

Commit: 8abab5dfb588a3b6e6fe0c1d32525656ddc13cf6edde293052d7e4660df11ce13513f08edf44c5e5

Branch pushed to git repo; I updated commit sha1. New commits:

edde293Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace

comment:113 Changed 2 years ago by git

Commit: edde293052d7e4660df11ce13513f08edf44c5e574ecddbe6a71f133458cdb018f45e36ba0aaf818

Branch pushed to git repo; I updated commit sha1. New commits:

74ecddbUpdate dependencies

comment:114 Changed 2 years ago by git

Commit: 74ecddbe6a71f133458cdb018f45e36ba0aaf8187110841f2c816001c14205c1426088fb873499b3

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

4f56517Duplicate src/setup.py
18a091bNarrow down to essenital changes
7110841Update dependencies

comment:115 Changed 2 years ago by Matthias Köppe

Dependencies: #29500, #30672, #30673, #30590, #30709#30779, #30672, #30673, #30709

comment:116 Changed 2 years ago by git

Commit: 7110841f2c816001c14205c1426088fb873499b30b5403699f9472f102b4781e3fd9ebae80e6bb8a

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

52e2107sage_include_directories: Remove sage/ext
6ccce6bFix missing ccobject
1402129Merge branch 'u/mkoeppe/remove__sage_ext__from__sage_include_directories_' of git://trac.sagemath.org/sage into t/30371/public/build/inplace
6eb3845Extend gitignore and add VS code config
bbe73eeRemove VS code settings, and add exceptions for already checked-in files
22b7cacMerge branch 'public/build/enhancegitignore' of git://trac.sagemath.org/sage into t/30371/public/build/inplace
1aef376Extract cython options to new module
68ab47bMerge branch 'public/build/cythonConfig' of git://trac.sagemath.org/sage into t/30371/public/build/inplace
d1b1288Narrow down to essenital changes
0b54036Update dependencies

comment:117 Changed 2 years ago by Matthias Köppe

Rebased on top of the dependencies

comment:118 Changed 2 years ago by Matthias Köppe

Status: needs_reviewneeds_work
Work issues: Use the cython options

comment:119 Changed 2 years ago by git

Commit: 0b5403699f9472f102b4781e3fd9ebae80e6bb8a55930254f941314d9996bfc6a9a4d5f9767c2e6d

Branch pushed to git repo; I updated commit sha1. New commits:

82e8b7dNarrow down to essenital changes
8abab5dAdd copy of setup.py
edde293Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace
74ecddbUpdate dependencies
1286fcbMerge branch 'public/build/inplace' of git://trac.sagemath.org/sage into public/build/inplace
946737eUse new cython option modules
5593025Update dependencies

comment:120 Changed 2 years ago by Tobias Diez

Status: needs_workneeds_review
Work issues: Use the cython options

Thanks for the rebase. Next time please try to not force-push to a branch that others have checked out because it messes up their git history.

I've updated the code to use the new cython options module.

comment:121 Changed 2 years ago by Matthias Köppe

Sorry if the push made it harder for you to continue on this branch. But rebasing it on top of the branches of the dependency tickets was a necessary cleanup

comment:122 Changed 2 years ago by Tobias Diez

Description: modified (diff)

comment:123 Changed 2 years ago by Matthias Köppe

What's the status of this branch? Does it work for you? Can you run doctests in the in-place installation and does it pass them?

comment:124 Changed 2 years ago by Tobias Diez

It's working for me. pipenv install -dev gives a fully functional virtual environment (so the cython compilation works). Moreover, import sage.all works as well, and one can then use it to run sage-code. So far, I've tested it with some code from the manifolds package. Not yet tried doctests, as I've no idea how to run them from the virtual environment.

comment:125 in reply to:  124 Changed 2 years ago by Matthias Köppe

Replying to gh-tobiasdiez:

Not yet tried doctests, as I've no idea how to run them from the virtual environment.

Well, the virtual environment has a bin/sage script, so sage -t "should" work. If it does not, this needs to be fixed. Likely some work in the direction of #22731 is necessary.

comment:126 Changed 2 years ago by Tobias Diez

Yeah, sage -t still uses the python in sage/local. So this does not work, and a mechanism is needed to use the python of the activated virtual env (without changing env variables which is too inflexible in my opinion).

Directly running bin/sage-runtests --all also leads to errors:

no stored timings available
Running doctests with ID 2020-10-20-21-18-29-b0a05b3d.
Using --optional=dochtml,memlimit,sage,sage_spkg
Doctesting entire Sage library.
Traceback (most recent call last):
  File "bin/sage-runtests", line 182, in <module>
    err = DC.run()
  File "/mnt/d/Programming/Projects/sage/src/sage/doctest/control.py", line 1234, in run
    self.expand_files_into_sources()
  File "/mnt/d/Programming/Projects/sage/src/sage/doctest/control.py", line 817, in expand_files_into_sources
    self.sources = [FileDocTestSource(path, self.options) for path in expand()]
  File "/mnt/d/Programming/Projects/sage/src/sage/doctest/control.py", line 817, in <listcomp>
    self.sources = [FileDocTestSource(path, self.options) for path in expand()]
  File "/mnt/d/Programming/Projects/sage/src/sage/doctest/sources.py", line 528, in __init__
    raise ValueError("unknown file extension %r"%ext)
ValueError: unknown file extension ''

(I guess some files are not found.)

Anyway, that's probably a topic for another ticket.

comment:127 in reply to:  126 Changed 2 years ago by Matthias Köppe

Replying to gh-tobiasdiez:

Yeah, sage -t still uses the python in sage/local. So this does not work, and a mechanism is needed to use the python of the activated virtual env (without changing env variables which is too inflexible in my opinion).

Yes, this is precisely #22731

comment:128 Changed 2 years ago by git

Commit: 55930254f941314d9996bfc6a9a4d5f9767c2e6d9dc8a2b0bcd6ff1cada426c60184b322312618e4

Branch pushed to git repo; I updated commit sha1. New commits:

9dc8a2bGenerate code automatically

comment:129 Changed 2 years ago by git

Commit: 9dc8a2b0bcd6ff1cada426c60184b322312618e419482048d4fffe52e94fcd94b81fa8ef59000558

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

fcef2b6Use pip to install pipenv
4160c9fUpdate ci-pipenv.yml
d0e01f2Update ci-pipenv.yml
711a2a6Update ci-pipenv.yml
341be72Update ci-pipenv.yml
918e1a5Update ci-pipenv.yml
573ec51Try with only one python
961abc3Update ci-pipenv.yml
5a9b803Update ci-pipenv.yml
1948204Merge branch 'public/build/inplace' of https://github.com/tobiasdiez/sage into public/build/inplace

comment:130 Changed 2 years ago by git

Commit: 19482048d4fffe52e94fcd94b81fa8ef59000558fffea7a7a50dd8c79827a07d6ef4eb3568fc1817

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

0016538Merge branch 'public/build/inplace' of https://github.com/tobiasdiez/sage into public/build/inplace
d65d85eUpdate ci-pipenv.yml
c1b8aefMerge branch 'public/build/inplace' of https://github.com/tobiasdiez/sage into public/build/inplace
23d17adRename cblas to blas
3cfe1c9Merge branch 'develop' of git://github.com/sagemath/sage into public/build/cblas
ea4a44cSupport different blas libraries
75966fcPut cblas first
273bb93Make cblas pc module search configurable via sage_conf
e6ff9baMerge branch 'public/build/cblas' of git://trac.sagemath.org/sage into public/build/inplace
fffea7aInstall more dependencies

comment:131 Changed 2 years ago by Tobias Diez

Dependencies: #30779, #30672, #30673, #30709#30779, #30672, #30673, #30709, #30706

comment:132 Changed 2 years ago by git

Commit: fffea7a7a50dd8c79827a07d6ef4eb3568fc1817062746cad45c3c3edc9f0d905de8312df1c020ac

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

ed74c46Fix powershell script
88f4529Make powershell run
4171062Cleanup
bfdc1c0Fix problem with libhomfly
3bda21eFix arb lib
3d476d5Install more deps for macos (as recommended by configure)
4302ac9Also install ecl
e34a6afAlso install pynac
8679770Disable copying of sage-env-config
062746cTry with hard-coded version

comment:133 Changed 2 years ago by Tobias Diez

I've now added github actions to have a reproducible test in a clean environment. The compilation is working now (under linux), but the singular library cannot be found during runtime. https://github.com/tobiasdiez/sage/runs/1287237342?check_suite_focus=true

The problem appears to be that sage is looking for singular in sysconfig.get_config_var('LIBDIR') (e.g. /usr/local/lib) but it is installed as /usr/lib/x86_64-linux-gnu/libsingular-Singular.so. Is there a reason why LIBDIR is used in env.py instead of the approach using pkgconfig as during compilation time?

comment:134 Changed 2 years ago by Matthias Köppe

_get_shared_lib_filename actually also looks in MULTILIB and is supposed to find this Debian-specific install location via this mechanism. But it is possible that this is broken because we currently actually do not use system singular (nor gap) at all (see #29644, #29024, meta-ticket #27330). This needs more investigation.

comment:135 in reply to:  133 Changed 2 years ago by Matthias Köppe

Replying to gh-tobiasdiez:

Is there a reason why LIBDIR is used in env.py instead of the approach using pkgconfig as during compilation time?

pkgconfig does not really reveal the libname directly, so one would need to parse pkgconfig --libs.

Another reason is that the .pc files are usually only available in "dev" packages. So the runtime of Sage should not depend on it.

#29024 intends to make the location configurable (at build time!) via sage_conf.

comment:136 Changed 2 years ago by git

Commit: 062746cad45c3c3edc9f0d905de8312df1c020ac7523baa68b46732cc381387e58ea1a9559c11e96

Branch pushed to git repo; I updated commit sha1. New commits:

38acc87Fix _get_shared_lib_filename to find singular
94787d1Try to run without build-isolation (manual)
8afec5aTry to set PIP_NO_BUILD_ISOLATION:
a321431Strange convention for PIP_NO variables
7523baaCleanup

comment:137 Changed 2 years ago by Tobias Diez

Thanks for the input. I had a closer look at _get_shared_lib_filename and the problem was that libsingular-Singular.so is in the subfolder /usr/lib/x86_64-linux-gnu/ while _get_shared_lib_filename only searched the top folder /usr/lib. I've now changed it to also look in subfolders. That should be reasonable fix until #29024 is implemented.

With these changes, the github action workflow passes successful on linux: https://github.com/tobiasdiez/sage/runs/1303324520?check_suite_focus=true It's nice to see that it only takes about 1 hour to have a fully workable sage environment.

This is now really ready for review.

TODOs (for further tickets):

  • Investigate what needs to be installed on macOS to have a successful build, and add these steps to the github action. (I don't have a mac, so I'll not work on this.)
  • Run doctests in the github action as soon as they no longer rely on SAGE_LOCAL (#22731)

comment:138 Changed 2 years ago by Tobias Diez

Description: modified (diff)

comment:139 in reply to:  137 Changed 2 years ago by Matthias Köppe

Replying to gh-tobiasdiez:

Thanks for the input. I had a closer look at _get_shared_lib_filename and the problem was that libsingular-Singular.so is in the subfolder /usr/lib/x86_64-linux-gnu/ while _get_shared_lib_filename only searched the top folder /usr/lib. I've now changed it to also look in subfolders.

This should have been taking care of by MULTILIB. Is this not configured in your system?

Last edited 2 years ago by Matthias Köppe (previous) (diff)

comment:140 Changed 2 years ago by Tobias Diez

MULTILIB is empty on my system (WSL Ubuntu 20.04), nor was it working for the github actions (Ubuntu 20.10).

comment:141 Changed 2 years ago by Matthias Köppe

Can you show the full output of python3 -m sysconfig on this system?

comment:142 Changed 2 years ago by Tobias Diez

Platform: "linux-x86_64"
Python version: "3.8"
Current installation scheme: "posix_prefix"

Paths: 
        data = "/mnt/d/Programming/Projects/sage/src/.venv"
        include = "/usr/include/python3.8"
        platinclude = "/usr/include/python3.8"
        platlib = "/mnt/d/Programming/Projects/sage/src/.venv/lib/python3.8/site-packages"
        platstdlib = "/mnt/d/Programming/Projects/sage/src/.venv/lib/python3.8"
        purelib = "/mnt/d/Programming/Projects/sage/src/.venv/lib/python3.8/site-packages"
        scripts = "/mnt/d/Programming/Projects/sage/src/.venv/bin"
        stdlib = "/usr/lib/python3.8"

Variables: 
        ABIFLAGS = ""
        AC_APPLE_UNIVERSAL_BUILD = "0"
        AIX_GENUINE_CPLUSPLUS = "0"
        ALT_SOABI = "0"
        ANDROID_API_LEVEL = "0"
        AR = "x86_64-linux-gnu-gcc-ar"
        ARFLAGS = "rcs"
        BASECFLAGS = "-Wno-unused-result -Wsign-compare"
        BASECPPFLAGS = "-IObjects -IInclude -IPython"
        BASEMODLIBS = ""
        BINDIR = "/usr/bin"
        BINLIBDEST = "/usr/lib/python3.8"
        BLDLIBRARY = "-lpython3.8"
        BLDSHARED = "x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions  -Wl,-z,relro -g -fwrapv -O2   "
        BUILDEXE = ""
        BUILDPYTHON = "python"
        BUILD_GNU_TYPE = "x86_64-pc-linux-gnu"
        BYTESTR_DEPS = "\"
        CC = "x86_64-linux-gnu-gcc -pthread"
        CCSHARED = "-fPIC"
        CFLAGS = "-Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g   -fstack-protector-strong -Wformat -Werror=format-security  -g -fwrapv -O2   "
        CFLAGSFORSHARED = "-fPIC"
        CFLAGS_ALIASING = ""
        CFLAGS_NODIST = ""
        CONFIGFILES = "configure configure.ac acconfig.h pyconfig.h.in Makefile.pre.in"
        CONFIGURE_CFLAGS = "-g   -fstack-protector-strong -Wformat -Werror=format-security"
        CONFIGURE_CFLAGS_NODIST = "-std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration"
        CONFIGURE_CPPFLAGS = "-Wdate-time -D_FORTIFY_SOURCE=2"
        CONFIGURE_LDFLAGS = "-Wl,-Bsymbolic-functions  -Wl,-z,relro -g -fwrapv -O2   "
        CONFIGURE_LDFLAGS_NODIST = ""
        CONFIG_ARGS = "'--enable-shared' '--prefix=/usr' '--enable-ipv6' '--enable-loadable-sqlite-extensions' '--with-dbmliborder=bdb:gdbm' '--with-computed-gotos' '--without-ensurepip' '--with-system-expat' '--with-system-libmpdec' '--with-dtrace' '--with-system-ffi' 'CC=x86_64-linux-gnu-gcc' 'CFLAGS=-g   -fstack-protector-strong -Wformat -Werror=format-security ' 'LDFLAGS=-Wl,-Bsymbolic-functions  -Wl,-z,relro -g -fwrapv -O2   ' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2'"
        CONFINCLUDEDIR = "/usr/include"
        CONFINCLUDEPY = "/usr/include/python3.8"
        COREPYTHONPATH = ""
        COVERAGE_INFO = "/build/python3.8-6QL2k7/python3.8-3.8.2/build-shared/coverage.info"
        COVERAGE_REPORT = "/build/python3.8-6QL2k7/python3.8-3.8.2/build-shared/lcov-report"
        COVERAGE_REPORT_OPTIONS = "--no-branch-coverage --title "CPython lcov report""
        CPPFLAGS = "-IObjects -IInclude -IPython -I. -I../Include -Wdate-time -D_FORTIFY_SOURCE=2"
        CXX = "x86_64-linux-gnu-g++ -pthread"
        DESTDIRS = "/usr /usr/lib /usr/lib/python3.8 /usr/lib/python3.8/lib-dynload"
        DESTLIB = "/usr/lib/python3.8"
        DESTPATH = ""
        DESTSHARED = "/usr/lib/python3.8/lib-dynload"
        DFLAGS = ""
        DIRMODE = "755"
        DIST = "README.rst ChangeLog configure configure.ac acconfig.h pyconfig.h.in Makefile.pre.in Include Lib Misc Ext-dummy"
        DISTDIRS = "Include Lib Misc Ext-dummy"
        DISTFILES = "README.rst ChangeLog configure configure.ac acconfig.h pyconfig.h.in Makefile.pre.in"
        DLINCLDIR = "."
        DLLLIBRARY = ""
        DOUBLE_IS_ARM_MIXED_ENDIAN_IEEE754 = "0"
        DOUBLE_IS_BIG_ENDIAN_IEEE754 = "0"
        DOUBLE_IS_LITTLE_ENDIAN_IEEE754 = "1"
        DTRACE = "/usr/bin/dtrace"
        DTRACE_DEPS = "\"
        DTRACE_HEADERS = "Include/pydtrace_probes.h"
        DTRACE_OBJS = "Python/pydtrace.o"
        DYNLOADFILE = "dynload_shlib.o"
        ENABLE_IPV6 = "1"
        ENSUREPIP = "no"
        EXE = ""
        EXEMODE = "755"
        EXTRATESTOPTS = ""
        EXT_SUFFIX = ".cpython-38-x86_64-linux-gnu.so"
        FILEMODE = "644"
        FLOAT_WORDS_BIGENDIAN = "0"
        FLOCK_NEEDS_LIBBSD = "0"
        GETPGRP_HAVE_ARG = "0"
        GETTIMEOFDAY_NO_TZ = "0"
        GITBRANCH = ""
        GITTAG = ""
        GITVERSION = ""
        GNULD = "yes"
        HAVE_ACCEPT4 = "1"
        HAVE_ACOSH = "1"
        HAVE_ADDRINFO = "1"
        HAVE_ALARM = "1"
        HAVE_ALIGNED_REQUIRED = "0"
        HAVE_ALLOCA_H = "1"
        HAVE_ALTZONE = "0"
        HAVE_ASINH = "1"
        HAVE_ASM_TYPES_H = "1"
        HAVE_ATANH = "1"
        HAVE_BIND_TEXTDOMAIN_CODESET = "1"
        HAVE_BLUETOOTH_BLUETOOTH_H = "1"
        HAVE_BLUETOOTH_H = "0"
        HAVE_BROKEN_MBSTOWCS = "0"
        HAVE_BROKEN_NICE = "0"
        HAVE_BROKEN_PIPE_BUF = "0"
        HAVE_BROKEN_POLL = "0"
        HAVE_BROKEN_POSIX_SEMAPHORES = "0"
        HAVE_BROKEN_PTHREAD_SIGMASK = "0"
        HAVE_BROKEN_SEM_GETVALUE = "0"
        HAVE_BROKEN_UNSETENV = "0"
        HAVE_BUILTIN_ATOMIC = "1"
        HAVE_CHFLAGS = "0"
        HAVE_CHOWN = "1"
        HAVE_CHROOT = "1"
        HAVE_CLOCK = "1"
        HAVE_CLOCK_GETRES = "1"
        HAVE_CLOCK_GETTIME = "1"
        HAVE_CLOCK_SETTIME = "1"
        HAVE_COMPUTED_GOTOS = "1"
        HAVE_CONFSTR = "1"
        HAVE_CONIO_H = "0"
        HAVE_COPYSIGN = "1"
        HAVE_COPY_FILE_RANGE = "1"
        HAVE_CRYPT_H = "1"
        HAVE_CRYPT_R = "1"
        HAVE_CTERMID = "1"
        HAVE_CTERMID_R = "0"
        HAVE_CURSES_FILTER = "1"
        HAVE_CURSES_H = "1"
        HAVE_CURSES_HAS_KEY = "1"
        HAVE_CURSES_IMMEDOK = "1"
        HAVE_CURSES_IS_PAD = "1"
        HAVE_CURSES_IS_TERM_RESIZED = "1"
        HAVE_CURSES_RESIZETERM = "1"
        HAVE_CURSES_RESIZE_TERM = "1"
        HAVE_CURSES_SYNCOK = "1"
        HAVE_CURSES_TYPEAHEAD = "1"
        HAVE_CURSES_USE_ENV = "1"
        HAVE_CURSES_WCHGAT = "1"
        HAVE_DECL_ISFINITE = "1"
        HAVE_DECL_ISINF = "1"
        HAVE_DECL_ISNAN = "1"
        HAVE_DECL_RTLD_DEEPBIND = "1"
        HAVE_DECL_RTLD_GLOBAL = "1"
        HAVE_DECL_RTLD_LAZY = "1"
        HAVE_DECL_RTLD_LOCAL = "1"
        HAVE_DECL_RTLD_MEMBER = "0"
        HAVE_DECL_RTLD_NODELETE = "1"
        HAVE_DECL_RTLD_NOLOAD = "1"
        HAVE_DECL_RTLD_NOW = "1"
        HAVE_DECL_TZNAME = "0"
        HAVE_DEVICE_MACROS = "1"
        HAVE_DEV_PTC = "0"
        HAVE_DEV_PTMX = "1"
        HAVE_DIRECT_H = "0"
        HAVE_DIRENT_D_TYPE = "1"
        HAVE_DIRENT_H = "1"
        HAVE_DIRFD = "1"
        HAVE_DLFCN_H = "1"
        HAVE_DLOPEN = "1"
        HAVE_DUP2 = "1"
        HAVE_DUP3 = "1"
        HAVE_DYNAMIC_LOADING = "1"
        HAVE_ENDIAN_H = "1"
        HAVE_EPOLL = "1"
        HAVE_EPOLL_CREATE1 = "1"
        HAVE_ERF = "1"
        HAVE_ERFC = "1"
        HAVE_ERRNO_H = "1"
        HAVE_EXECV = "1"
        HAVE_EXPLICIT_BZERO = "1"
        HAVE_EXPLICIT_MEMSET = "0"
        HAVE_EXPM1 = "1"
        HAVE_FACCESSAT = "1"
        HAVE_FCHDIR = "1"
        HAVE_FCHMOD = "1"
        HAVE_FCHMODAT = "1"
        HAVE_FCHOWN = "1"
        HAVE_FCHOWNAT = "1"
        HAVE_FCNTL_H = "1"
        HAVE_FDATASYNC = "1"
        HAVE_FDOPENDIR = "1"
        HAVE_FDWALK = "0"
        HAVE_FEXECVE = "1"
        HAVE_FINITE = "1"
        HAVE_FLOCK = "1"
        HAVE_FORK = "1"
        HAVE_FORKPTY = "1"
        HAVE_FPATHCONF = "1"
        HAVE_FSEEK64 = "0"
        HAVE_FSEEKO = "1"
        HAVE_FSTATAT = "1"
        HAVE_FSTATVFS = "1"
        HAVE_FSYNC = "1"
        HAVE_FTELL64 = "0"
        HAVE_FTELLO = "1"
        HAVE_FTIME = "1"
        HAVE_FTRUNCATE = "1"
        HAVE_FUTIMENS = "1"
        HAVE_FUTIMES = "1"
        HAVE_FUTIMESAT = "1"
        HAVE_GAI_STRERROR = "1"
        HAVE_GAMMA = "1"
        HAVE_GCC_ASM_FOR_MC68881 = "0"
        HAVE_GCC_ASM_FOR_X64 = "1"
        HAVE_GCC_ASM_FOR_X87 = "1"
        HAVE_GCC_UINT128_T = "1"
        HAVE_GETADDRINFO = "1"
        HAVE_GETC_UNLOCKED = "1"
        HAVE_GETENTROPY = "1"
        HAVE_GETGRGID_R = "1"
        HAVE_GETGRNAM_R = "1"
        HAVE_GETGROUPLIST = "1"
        HAVE_GETGROUPS = "1"
        HAVE_GETHOSTBYNAME = "0"
        HAVE_GETHOSTBYNAME_R = "1"
        HAVE_GETHOSTBYNAME_R_3_ARG = "0"
        HAVE_GETHOSTBYNAME_R_5_ARG = "0"
        HAVE_GETHOSTBYNAME_R_6_ARG = "1"
        HAVE_GETITIMER = "1"
        HAVE_GETLOADAVG = "1"
        HAVE_GETLOGIN = "1"
        HAVE_GETNAMEINFO = "1"
        HAVE_GETPAGESIZE = "1"
        HAVE_GETPEERNAME = "1"
        HAVE_GETPGID = "1"
        HAVE_GETPGRP = "1"
        HAVE_GETPID = "1"
        HAVE_GETPRIORITY = "1"
        HAVE_GETPWENT = "1"
        HAVE_GETPWNAM_R = "1"
        HAVE_GETPWUID_R = "1"
        HAVE_GETRANDOM = "1"
        HAVE_GETRANDOM_SYSCALL = "1"
        HAVE_GETRESGID = "1"
        HAVE_GETRESUID = "1"
        HAVE_GETSID = "1"
        HAVE_GETSPENT = "1"
        HAVE_GETSPNAM = "1"
        HAVE_GETTIMEOFDAY = "1"
        HAVE_GETWD = "1"
        HAVE_GLIBC_MEMMOVE_BUG = "0"
        HAVE_GRP_H = "1"
        HAVE_HSTRERROR = "1"
        HAVE_HTOLE64 = "1"
        HAVE_HYPOT = "1"
        HAVE_IEEEFP_H = "0"
        HAVE_IF_NAMEINDEX = "1"
        HAVE_INET_ATON = "1"
        HAVE_INET_PTON = "1"
        HAVE_INITGROUPS = "1"
        HAVE_INTTYPES_H = "1"
        HAVE_IO_H = "0"
        HAVE_IPA_PURE_CONST_BUG = "0"
        HAVE_KILL = "1"
        HAVE_KILLPG = "1"
        HAVE_KQUEUE = "0"
        HAVE_LANGINFO_H = "1"
        HAVE_LARGEFILE_SUPPORT = "0"
        HAVE_LCHFLAGS = "0"
        HAVE_LCHMOD = "0"
        HAVE_LCHOWN = "1"
        HAVE_LGAMMA = "1"
        HAVE_LIBDL = "1"
        HAVE_LIBDLD = "0"
        HAVE_LIBIEEE = "0"
        HAVE_LIBINTL_H = "1"
        HAVE_LIBREADLINE = "1"
        HAVE_LIBRESOLV = "0"
        HAVE_LIBSENDFILE = "0"
        HAVE_LIBUTIL_H = "0"
        HAVE_LINK = "1"
        HAVE_LINKAT = "1"
        HAVE_LINUX_CAN_BCM_H = "1"
        HAVE_LINUX_CAN_H = "1"
        HAVE_LINUX_CAN_RAW_FD_FRAMES = "1"
        HAVE_LINUX_CAN_RAW_H = "1"
        HAVE_LINUX_MEMFD_H = "1"
        HAVE_LINUX_NETLINK_H = "1"
        HAVE_LINUX_QRTR_H = "1"
        HAVE_LINUX_RANDOM_H = "1"
        HAVE_LINUX_TIPC_H = "1"
        HAVE_LINUX_VM_SOCKETS_H = "1"
        HAVE_LOCKF = "1"
        HAVE_LOG1P = "1"
        HAVE_LOG2 = "1"
        HAVE_LONG_DOUBLE = "1"
        HAVE_LSTAT = "1"
        HAVE_LUTIMES = "1"
        HAVE_MADVISE = "1"
        HAVE_MAKEDEV = "1"
        HAVE_MBRTOWC = "1"
        HAVE_MEMFD_CREATE = "1"
        HAVE_MEMORY_H = "1"
        HAVE_MEMRCHR = "1"
        HAVE_MKDIRAT = "1"
        HAVE_MKFIFO = "1"
        HAVE_MKFIFOAT = "1"
        HAVE_MKNOD = "1"
        HAVE_MKNODAT = "1"
        HAVE_MKTIME = "1"
        HAVE_MMAP = "1"
        HAVE_MREMAP = "1"
        HAVE_NCURSES_H = "1"
        HAVE_NDIR_H = "0"
        HAVE_NETPACKET_PACKET_H = "1"
        HAVE_NET_IF_H = "1"
        HAVE_NICE = "1"
        HAVE_OPENAT = "1"
        HAVE_OPENPTY = "1"
        HAVE_PATHCONF = "1"
        HAVE_PAUSE = "1"
        HAVE_PIPE2 = "1"
        HAVE_PLOCK = "0"
        HAVE_POLL = "1"
        HAVE_POLL_H = "1"
        HAVE_POSIX_FADVISE = "1"
        HAVE_POSIX_FALLOCATE = "1"
        HAVE_POSIX_SPAWN = "1"
        HAVE_POSIX_SPAWNP = "1"
        HAVE_PREAD = "1"
        HAVE_PREADV = "1"
        HAVE_PREADV2 = "1"
        HAVE_PRLIMIT = "1"
        HAVE_PROCESS_H = "0"
        HAVE_PROTOTYPES = "1"
        HAVE_PTHREAD_CONDATTR_SETCLOCK = "1"
        HAVE_PTHREAD_DESTRUCTOR = "0"
        HAVE_PTHREAD_GETCPUCLOCKID = "1"
        HAVE_PTHREAD_H = "1"
        HAVE_PTHREAD_INIT = "0"
        HAVE_PTHREAD_KILL = "1"
        HAVE_PTHREAD_SIGMASK = "1"
        HAVE_PTY_H = "1"
        HAVE_PUTENV = "1"
        HAVE_PWRITE = "1"
        HAVE_PWRITEV = "1"
        HAVE_PWRITEV2 = "1"
        HAVE_READLINK = "1"
        HAVE_READLINKAT = "1"
        HAVE_READV = "1"
        HAVE_REALPATH = "1"
        HAVE_RENAMEAT = "1"
        HAVE_RL_APPEND_HISTORY = "1"
        HAVE_RL_CATCH_SIGNAL = "1"
        HAVE_RL_COMPLETION_APPEND_CHARACTER = "1"
        HAVE_RL_COMPLETION_DISPLAY_MATCHES_HOOK = "1"
        HAVE_RL_COMPLETION_MATCHES = "1"
        HAVE_RL_COMPLETION_SUPPRESS_APPEND = "1"
        HAVE_RL_PRE_INPUT_HOOK = "1"
        HAVE_RL_RESIZE_TERMINAL = "1"
        HAVE_ROUND = "1"
        HAVE_RTPSPAWN = "0"
        HAVE_SCHED_GET_PRIORITY_MAX = "1"
        HAVE_SCHED_H = "1"
        HAVE_SCHED_RR_GET_INTERVAL = "1"
        HAVE_SCHED_SETAFFINITY = "1"
        HAVE_SCHED_SETPARAM = "1"
        HAVE_SCHED_SETSCHEDULER = "1"
        HAVE_SEM_GETVALUE = "1"
        HAVE_SEM_OPEN = "1"
        HAVE_SEM_TIMEDWAIT = "1"
        HAVE_SEM_UNLINK = "1"
        HAVE_SENDFILE = "1"
        HAVE_SETEGID = "1"
        HAVE_SETEUID = "1"
        HAVE_SETGID = "1"
        HAVE_SETGROUPS = "1"
        HAVE_SETHOSTNAME = "1"
        HAVE_SETITIMER = "1"
        HAVE_SETLOCALE = "1"
        HAVE_SETPGID = "1"
        HAVE_SETPGRP = "1"
        HAVE_SETPRIORITY = "1"
        HAVE_SETREGID = "1"
        HAVE_SETRESGID = "1"
        HAVE_SETRESUID = "1"
        HAVE_SETREUID = "1"
        HAVE_SETSID = "1"
        HAVE_SETUID = "1"
        HAVE_SETVBUF = "1"
        HAVE_SHADOW_H = "1"
        HAVE_SHM_OPEN = "1"
        HAVE_SHM_UNLINK = "1"
        HAVE_SIGACTION = "1"
        HAVE_SIGALTSTACK = "1"
        HAVE_SIGFILLSET = "1"
        HAVE_SIGINFO_T_SI_BAND = "1"
        HAVE_SIGINTERRUPT = "1"
        HAVE_SIGNAL_H = "1"
        HAVE_SIGPENDING = "1"
        HAVE_SIGRELSE = "1"
        HAVE_SIGTIMEDWAIT = "1"
        HAVE_SIGWAIT = "1"
        HAVE_SIGWAITINFO = "1"
        HAVE_SNPRINTF = "1"
        HAVE_SOCKADDR_ALG = "1"
        HAVE_SOCKADDR_SA_LEN = "0"
        HAVE_SOCKADDR_STORAGE = "1"
        HAVE_SOCKETPAIR = "1"
        HAVE_SPAWN_H = "1"
        HAVE_SSIZE_T = "1"
        HAVE_STATVFS = "1"
        HAVE_STAT_TV_NSEC = "1"
        HAVE_STAT_TV_NSEC2 = "0"
        HAVE_STDARG_PROTOTYPES = "1"
        HAVE_STDINT_H = "1"
        HAVE_STDLIB_H = "1"
        HAVE_STD_ATOMIC = "1"
        HAVE_STRDUP = "1"
        HAVE_STRFTIME = "1"
        HAVE_STRINGS_H = "1"
        HAVE_STRING_H = "1"
        HAVE_STRLCPY = "0"
        HAVE_STROPTS_H = "0"
        HAVE_STRSIGNAL = "1"
        HAVE_STRUCT_PASSWD_PW_GECOS = "1"
        HAVE_STRUCT_PASSWD_PW_PASSWD = "1"
        HAVE_STRUCT_STAT_ST_BIRTHTIME = "0"
        HAVE_STRUCT_STAT_ST_BLKSIZE = "1"
        HAVE_STRUCT_STAT_ST_BLOCKS = "1"
        HAVE_STRUCT_STAT_ST_FLAGS = "0"
        HAVE_STRUCT_STAT_ST_GEN = "0"
        HAVE_STRUCT_STAT_ST_RDEV = "1"
        HAVE_STRUCT_TM_TM_ZONE = "1"
        HAVE_SYMLINK = "1"
        HAVE_SYMLINKAT = "1"
        HAVE_SYNC = "1"
        HAVE_SYSCONF = "1"
        HAVE_SYSEXITS_H = "1"
        HAVE_SYS_AUDIOIO_H = "0"
        HAVE_SYS_BSDTTY_H = "0"
        HAVE_SYS_DEVPOLL_H = "0"
        HAVE_SYS_DIR_H = "0"
        HAVE_SYS_ENDIAN_H = "0"
        HAVE_SYS_EPOLL_H = "1"
        HAVE_SYS_EVENT_H = "0"
        HAVE_SYS_FILE_H = "1"
        HAVE_SYS_IOCTL_H = "1"
        HAVE_SYS_KERN_CONTROL_H = "0"
        HAVE_SYS_LOADAVG_H = "0"
        HAVE_SYS_LOCK_H = "0"
        HAVE_SYS_MEMFD_H = "0"
        HAVE_SYS_MKDEV_H = "0"
        HAVE_SYS_MMAN_H = "1"
        HAVE_SYS_MODEM_H = "0"
        HAVE_SYS_NDIR_H = "0"
        HAVE_SYS_PARAM_H = "1"
        HAVE_SYS_POLL_H = "1"
        HAVE_SYS_RANDOM_H = "1"
        HAVE_SYS_RESOURCE_H = "1"
        HAVE_SYS_SELECT_H = "1"
        HAVE_SYS_SENDFILE_H = "1"
        HAVE_SYS_SOCKET_H = "1"
        HAVE_SYS_STATVFS_H = "1"
        HAVE_SYS_STAT_H = "1"
        HAVE_SYS_SYSCALL_H = "1"
        HAVE_SYS_SYSMACROS_H = "1"
        HAVE_SYS_SYS_DOMAIN_H = "0"
        HAVE_SYS_TERMIO_H = "0"
        HAVE_SYS_TIMES_H = "1"
        HAVE_SYS_TIME_H = "1"
        HAVE_SYS_TYPES_H = "1"
        HAVE_SYS_UIO_H = "1"
        HAVE_SYS_UN_H = "1"
        HAVE_SYS_UTSNAME_H = "1"
        HAVE_SYS_WAIT_H = "1"
        HAVE_SYS_XATTR_H = "1"
        HAVE_TCGETPGRP = "1"
        HAVE_TCSETPGRP = "1"
        HAVE_TEMPNAM = "1"
        HAVE_TERMIOS_H = "1"
        HAVE_TERM_H = "1"
        HAVE_TGAMMA = "1"
        HAVE_TIMEGM = "1"
        HAVE_TIMES = "1"
        HAVE_TMPFILE = "1"
        HAVE_TMPNAM = "1"
        HAVE_TMPNAM_R = "1"
        HAVE_TM_ZONE = "1"
        HAVE_TRUNCATE = "1"
        HAVE_TZNAME = "0"
        HAVE_UCS4_TCL = "0"
        HAVE_UNAME = "1"
        HAVE_UNISTD_H = "1"
        HAVE_UNLINKAT = "1"
        HAVE_UNSETENV = "1"
        HAVE_USABLE_WCHAR_T = "0"
        HAVE_UTIL_H = "0"
        HAVE_UTIMENSAT = "1"
        HAVE_UTIMES = "1"
        HAVE_UTIME_H = "1"
        HAVE_UUID_CREATE = "0"
        HAVE_UUID_ENC_BE = "0"
        HAVE_UUID_GENERATE_TIME_SAFE = "1"
        HAVE_UUID_H = "0"
        HAVE_UUID_UUID_H = "1"
        HAVE_WAIT3 = "1"
        HAVE_WAIT4 = "1"
        HAVE_WAITID = "1"
        HAVE_WAITPID = "1"
        HAVE_WCHAR_H = "1"
        HAVE_WCSCOLL = "1"
        HAVE_WCSFTIME = "1"
        HAVE_WCSXFRM = "1"
        HAVE_WMEMCMP = "1"
        HAVE_WORKING_TZSET = "1"
        HAVE_WRITEV = "1"
        HAVE_X509_VERIFY_PARAM_SET1_HOST = "1"
        HAVE_ZLIB_COPY = "1"
        HAVE__GETPTY = "0"
        HOST_GNU_TYPE = "x86_64-pc-linux-gnu"
        INCLDIRSTOMAKE = "/usr/include /usr/include /usr/include/python3.8 /usr/include/python3.8"
        INCLUDEDIR = "/usr/include"
        INCLUDEPY = "/usr/include/python3.8"
        INSTALL = "/usr/bin/install -c"
        INSTALL_DATA = "/usr/bin/install -c -m 644"
        INSTALL_PROGRAM = "/usr/bin/install -c"
        INSTALL_SCRIPT = "/usr/bin/install -c"
        INSTALL_SHARED = "/usr/bin/install -c -m 755"
        INSTSONAME = "libpython3.8.so.1.0"
        IO_H = "Modules/_io/_iomodule.h"
        IO_OBJS = "\"
        LDCXXSHARED = "x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions"
        LDFLAGS = "-Wl,-Bsymbolic-functions  -Wl,-z,relro -g -fwrapv -O2   "
        LDFLAGS_NODIST = ""
        LDLIBRARY = "libpython3.8.so"
        LDLIBRARYDIR = ""
        LDSHARED = "x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions  -Wl,-z,relro -g -fwrapv -O2   "
        LDVERSION = "3.8"
        LIBC = ""
        LIBDEST = "/usr/lib/python3.8"
        LIBDIR = "/usr/lib"
        LIBFFI_INCLUDEDIR = ""
        LIBM = "-lm"
        LIBOBJDIR = "Python/"
        LIBOBJS = ""
        LIBPC = "/usr/lib/x86_64-linux-gnu/pkgconfig"
        LIBPL = "/usr/lib/python3.8/config-3.8-x86_64-linux-gnu"
        LIBPYTHON = ""
        LIBRARY = "libpython3.8.a"
        LIBRARY_OBJS = "\"
        LIBRARY_OBJS_OMIT_FROZEN = "\"
        LIBS = "-lcrypt -lpthread -ldl  -lutil -lm"
        LIBSUBDIRS = "tkinter tkinter/test tkinter/test/test_tkinter \"
        LINKCC = "x86_64-linux-gnu-gcc -pthread"
        LINKFORSHARED = "-Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions"
        LIPO_32BIT_FLAGS = ""
        LLVM_PROF_ERR = "no"
        LLVM_PROF_FILE = ""
        LLVM_PROF_MERGER = "true"
        LN = "ln"
        LOCALMODLIBS = "-lexpat                       -L/usr/lib -lz                       -lexpat"
        MACHDEP = "linux"
        MACHDEP_OBJS = ""
        MACHDESTLIB = "/usr/lib/python3.8"
        MACOSX_DEPLOYMENT_TARGET = ""
        MAINCC = "x86_64-linux-gnu-gcc -pthread"
        MAJOR_IN_MKDEV = "0"
        MAJOR_IN_SYSMACROS = "1"
        MAKESETUP = "../Modules/makesetup"
        MANDIR = "/usr/share/man"
        MKDIR_P = "/bin/mkdir -p"
        MODBUILT_NAMES = "array  cmath  math  _struct  _random  _elementtree  _pickle  _datetime  _bisect  _heapq  _statistics  unicodedata  fcntl  spwd  grp  select  _csv  _socket  _posixsubprocess  _md5  _sha1  _sha256  _sha512  _sha3  _blake2  syslog  binascii  zlib  posix  errno  pwd  _sre  _codecs  _weakref  _functools  _operator  _collections  _abc  itertools  atexit  _signal  _stat  time  _thread  _locale  _io  faulthandler  _tracemalloc  _symtable  pyexpat  xxsubtype"
        MODDISABLED_NAMES = ""
        MODLIBS = "-lexpat                       -L/usr/lib -lz                       -lexpat"
        MODOBJS = "$(sort   Modules/arraymodule.o  Modules/cmathmodule.o Modules/_math.o  Modules/mathmodule.o Modules/_math.o  Modules/_struct.o  Modules/_randommodule.o  Modules/_elementtree.o  Modules/_pickle.o  Modules/_datetimemodule.o  Modules/_bisectmodule.o  Modules/_heapqmodule.o  Modules/_statisticsmodule.o  Modules/unicodedata.o  Modules/fcntlmodule.o  Modules/spwdmodule.o  Modules/grpmodule.o  Modules/selectmodule.o  Modules/_csv.o  Modules/socketmodule.o  Modules/_posixsubprocess.o  Modules/md5module.o  Modules/sha1module.o  Modules/sha256module.o  Modules/sha512module.o  Modules/sha3module.o  Modules/blake2module.o Modules/blake2b_impl.o Modules/blake2s_impl.o  Modules/syslogmodule.o  Modules/binascii.o  Modules/zlibmodule.o  Modules/posixmodule.o  Modules/errnomodule.o  Modules/pwdmodule.o  Modules/_sre.o  Modules/_codecsmodule.o  Modules/_weakref.o  Modules/_functoolsmodule.o  Modules/_operator.o  Modules/_collectionsmodule.o  Modules/_abc.o  Modules/itertoolsmodule.o  Modules/atexitmodule.o  Modules/signalmodule.o  Modules/_stat.o  Modules/timemodule.o  Modules/_threadmodule.o  Modules/_localemodule.o  Modules/_iomodule.o Modules/iobase.o Modules/fileio.o Modules/bytesio.o Modules/bufferedio.o Modules/textio.o Modules/stringio.o  Modules/faulthandler.o  Modules/_tracemalloc.o Modules/hashtable.o  Modules/symtablemodule.o  Modules/pyexpat.o  Modules/xxsubtype.o)"
        MODULE_OBJS = "\"
        MULTIARCH = "x86_64-linux-gnu"
        MULTIARCH_CPPFLAGS = "-DMULTIARCH=\"x86_64-linux-gnu\""
        MVWDELCH_IS_EXPRESSION = "1"
        NO_AS_NEEDED = "-Wl,--no-as-needed"
        OBJECT_OBJS = "\"
        OPENSSL_INCLUDES = ""
        OPENSSL_LDFLAGS = ""
        OPENSSL_LIBS = "-lssl -lcrypto"
        OPT = "-DNDEBUG -g -fwrapv -O2 -Wall"
        OTHER_LIBTOOL_OPT = ""
        PACKAGE_BUGREPORT = "0"
        PACKAGE_NAME = "0"
        PACKAGE_STRING = "0"
        PACKAGE_TARNAME = "0"
        PACKAGE_URL = "0"
        PACKAGE_VERSION = "0"
        PARSER_HEADERS = "\"
        PARSER_OBJS = "\ Parser/myreadline.o Parser/parsetok.o Parser/tokenizer.o"
        PGO_PROF_GEN_FLAG = "-fprofile-generate"
        PGO_PROF_USE_FLAG = ""
        POBJS = "\"
        POSIX_SEMAPHORES_NOT_ENABLED = "0"
        PROFILE_TASK = "-m test --pgo"
        PTHREAD_KEY_T_IS_COMPATIBLE_WITH_INT = "1"
        PTHREAD_SYSTEM_SCHED_SUPPORTED = "1"
        PURIFY = ""
        PY3LIBRARY = "libpython3.so"
        PYLONG_BITS_IN_DIGIT = "0"
        PYTHON = "python"
        PYTHONFRAMEWORK = ""
        PYTHONFRAMEWORKDIR = "no-framework"
        PYTHONFRAMEWORKINSTALLDIR = ""
        PYTHONFRAMEWORKPREFIX = ""
        PYTHONPATH = ""
        PYTHON_FOR_BUILD = "./python -E"
        PYTHON_FOR_REGEN = "python3.8"
        PYTHON_HEADERS = "\"
        PYTHON_OBJS = "\"
        PY_BUILTIN_MODULE_CFLAGS = "-Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g   -fstack-protector-strong -Wformat -Werror=format-security  -g -fwrapv -O2    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration  -I../Include/internal -IObjects -IInclude -IPython -I. -I../Include -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DPy_BUILD_CORE_BUILTIN"
        PY_CFLAGS = "-Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g   -fstack-protector-strong -Wformat -Werror=format-security  -g -fwrapv -O2   "
        PY_CFLAGS_NODIST = "-std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration  -I../Include/internal"
        PY_COERCE_C_LOCALE = "1"
        PY_CORE_CFLAGS = "-Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g   -fstack-protector-strong -Wformat -Werror=format-security  -g -fwrapv -O2    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration  -I../Include/internal -IObjects -IInclude -IPython -I. -I../Include -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DPy_BUILD_CORE"
        PY_CORE_LDFLAGS = "-Wl,-Bsymbolic-functions  -Wl,-z,relro -g -fwrapv -O2   "
        PY_CPPFLAGS = "-IObjects -IInclude -IPython -I. -I../Include -Wdate-time -D_FORTIFY_SOURCE=2"
        PY_FORMAT_SIZE_T = ""z""
        PY_LDFLAGS = "-Wl,-Bsymbolic-functions  -Wl,-z,relro -g -fwrapv -O2   "
        PY_LDFLAGS_NODIST = ""
        PY_SSL_DEFAULT_CIPHERS = "1"
        PY_SSL_DEFAULT_CIPHER_STRING = "0"
        PY_STDMODULE_CFLAGS = "-Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g   -fstack-protector-strong -Wformat -Werror=format-security  -g -fwrapv -O2    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration  -I../Include/internal -IObjects -IInclude -IPython -I. -I../Include -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC"
        Py_DEBUG = "0"
        Py_ENABLE_SHARED = "1"
        Py_HASH_ALGORITHM = "0"
        Py_TRACE_REFS = "0"
        QUICKTESTOPTS = "-x test_subprocess test_io test_lib2to3 \"
        READELF = "readelf"
        RESSRCDIR = "Mac/Resources/framework"
        RETSIGTYPE = "void"
        RUNSHARED = "LD_LIBRARY_PATH=/build/python3.8-6QL2k7/python3.8-3.8.2/build-shared"
        SCRIPTDIR = "/usr/lib"
        SETPGRP_HAVE_ARG = "0"
        SGI_ABI = "@SGI_ABI@"
        SHELL = "/bin/sh"
        SHLIBS = "-lcrypt -lpthread -ldl  -lutil -lm"
        SHLIB_SUFFIX = ".so"
        SHM_NEEDS_LIBRT = "1"
        SIGNED_RIGHT_SHIFT_ZERO_FILLS = "0"
        SITEPATH = ""
        SIZEOF_DOUBLE = "8"
        SIZEOF_FLOAT = "4"
        SIZEOF_FPOS_T = "16"
        SIZEOF_INT = "4"
        SIZEOF_LONG = "8"
        SIZEOF_LONG_DOUBLE = "16"
        SIZEOF_LONG_LONG = "8"
        SIZEOF_OFF_T = "8"
        SIZEOF_PID_T = "4"
        SIZEOF_PTHREAD_KEY_T = "4"
        SIZEOF_PTHREAD_T = "8"
        SIZEOF_SHORT = "2"
        SIZEOF_SIZE_T = "8"
        SIZEOF_TIME_T = "8"
        SIZEOF_UINTPTR_T = "8"
        SIZEOF_VOID_P = "8"
        SIZEOF_WCHAR_T = "4"
        SIZEOF__BOOL = "1"
        SO = ".cpython-38-x86_64-linux-gnu.so"
        SOABI = "cpython-38-x86_64-linux-gnu"
        SRCDIRS = "Parser Objects Python Modules Modules/_io Programs"
        SRC_GDB_HOOKS = "../Tools/gdb/libpython.py"
        STDC_HEADERS = "1"
        STRICT_SYSV_CURSES = "/* Don't use ncurses extensions */"
        STRIPFLAG = "-s"
        SUBDIRS = ""
        SUBDIRSTOO = "Include Lib Misc"
        SYSLIBS = "-lm"
        SYS_SELECT_WITH_SYS_TIME = "1"
        TCLTK_INCLUDES = ""
        TCLTK_LIBS = ""
        TESTOPTS = ""
        TESTPATH = ""
        TESTPYTHON = "LD_LIBRARY_PATH=/build/python3.8-6QL2k7/python3.8-3.8.2/build-shared ./python"
        TESTPYTHONOPTS = ""
        TESTRUNNER = "LD_LIBRARY_PATH=/build/python3.8-6QL2k7/python3.8-3.8.2/build-shared ./python ../Tools/scripts/run_tests.py"
        TESTTIMEOUT = "1200"
        TIMEMODULE_LIB = "0"
        TIME_WITH_SYS_TIME = "1"
        TM_IN_SYS_TIME = "0"
        UNICODE_DEPS = "\"
        UNIVERSALSDK = ""
        UPDATE_FILE = "python3.8 ../Tools/scripts/update_file.py"
        USE_COMPUTED_GOTOS = "1"
        VERSION = "3.8"
        VPATH = ".."
        WINDOW_HAS_FLAGS = "1"
        WITH_DOC_STRINGS = "1"
        WITH_DTRACE = "1"
        WITH_DYLD = "0"
        WITH_LIBINTL = "0"
        WITH_NEXT_FRAMEWORK = "0"
        WITH_PYMALLOC = "1"
        WITH_VALGRIND = "0"
        X87_DOUBLE_ROUNDING = "0"
        XMLLIBSUBDIRS = "xml xml/dom xml/etree xml/parsers xml/sax"
        abiflags = ""
        abs_builddir = "/build/python3.8-6QL2k7/python3.8-3.8.2/build-shared"
        abs_srcdir = "/build/python3.8-6QL2k7/python3.8-3.8.2/build-shared/.."
        base = "/mnt/d/Programming/Projects/sage/src/.venv"
        datarootdir = "/usr/share"
        exec_prefix = "/usr"
        installed_base = "/usr"
        installed_platbase = "/usr"
        multiarchsubdir = "/x86_64-linux-gnu"
        platbase = "/mnt/d/Programming/Projects/sage/src/.venv"
        prefix = "/usr"
        projectbase = "/usr/bin"
        py_version = "3.8.2"
        py_version_nodot = "38"
        py_version_short = "3.8"
        srcdir = "/usr/lib/python3.8/config-3.8-x86_64-linux-gnu"
        userbase = "/home/tobias/.local"

comment:143 Changed 2 years ago by Matthias Köppe

I see MULTIARCH = "x86_64-linux-gnu" and multiarchsubdir = "/x86_64-linux-gnu".

comment:144 Changed 2 years ago by Matthias Köppe

One of these variables should be used in _get_shared_lib_filename. Why it is instead looking for MULTILIB, which indeed does not exist, should be investigated...

comment:145 Changed 2 years ago by Matthias Köppe

Let's make this a separate ticket please

comment:146 Changed 2 years ago by Matthias Köppe

This is now #30833

comment:147 Changed 2 years ago by Tobias Diez

Dependencies: #30779, #30672, #30673, #30709, #30706#30779, #30672, #30673, #30709, #30706, #30833

comment:148 Changed 2 years ago by git

Commit: 7523baa68b46732cc381387e58ea1a9559c11e96443424cf878ac7fa9c57a1bad313f1b4d84ed031

Branch pushed to git repo; I updated commit sha1. New commits:

289c3efImprove _get_shared_lib_filename
443424cCleanup code

comment:149 Changed 2 years ago by Tobias Diez

Dependencies: #30779, #30672, #30673, #30709, #30706, #30833#30779, #30672, #30673, #30709, #30706, #30833, #30748

comment:150 Changed 2 years ago by git

Commit: 443424cf878ac7fa9c57a1bad313f1b4d84ed031b3045684c751317f252ff61f59ab53782df76177

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

2351ae3Update, extend and reorder dependencies
99ae00cReplace startup exception by warning
11882e5Use context manager
d858e46Merge branch 'public/build/startupWarning' of git://trac.sagemath.org/sage into public/build/inplace
46ac365Update numpy
3fcd71fMerge tag '9.3.beta0' into t/30709/public/build/cythonConfig
a646a24src/sage_setup/cython_options.py: Restore auto_pickle=False
8eae4bcMerge branch 'public/build/cythonConfig' of git://trac.sagemath.org/sage into public/build/inplace
5ac3a29Add matlibplot library
b304568Run doctests in github action

comment:151 Changed 2 years ago by git

Commit: b3045684c751317f252ff61f59ab53782df761779a94a29bc1c6179282dbd94a7afd19247884e525

Branch pushed to git repo; I updated commit sha1. New commits:

9a94a29Don't install texlive-generic-extra

comment:152 Changed 2 years ago by git

Commit: 9a94a29bc1c6179282dbd94a7afd19247884e52516674ba8da9daa53709c9ad4bfc6fca6bdba13ee

Branch pushed to git repo; I updated commit sha1. New commits:

16674baRemove lazy import finish startup

comment:153 Changed 2 years ago by git

Commit: 16674ba8da9daa53709c9ad4bfc6fca6bdba13ee511775ec71b3d2efef8f6e9ca2ea95f234c566ad

Branch pushed to git repo; I updated commit sha1. New commits:

f078aa0Speedup a bit
3eac0afRefactor
dba18d0Add none check for IPython
511775eAdd caching

comment:154 Changed 2 years ago by Tobias Diez

I've now added the doctest run to the github workflow. It works in principle although there are quite a lot of failing tests (mainly because some of the system-installed libs are not discovered). In order to keep the focus of this ticket narrow, fixing this should better be done in follow-up tickets.

comment:155 Changed 2 years ago by Tobias Diez

Description: modified (diff)

comment:156 Changed 2 years ago by Matthias Köppe

It has been reported that the current src/setup.py (in Sage 9.2) is able to install a working sagelib on top of conda (see #28745).

I'll create a test for this in #30845, and then we can test the present ticket to make sure it does not break anything.

comment:157 Changed 2 years ago by git

Commit: 511775ec71b3d2efef8f6e9ca2ea95f234c566ad373d1c12e77647541fa7fc8b315aae98ee0298cd

Branch pushed to git repo; I updated commit sha1. New commits:

373d1c1Add rope as dev dependency

comment:158 in reply to:  description Changed 2 years ago by Matthias Köppe

Replying to mkoeppe:

  • A few libraries are not contained in distros/debian.txt files. Should they be added? In particular, this concerns libsingular4-dev, liblinbox-dev, libratpoints-dev, libgap-dev, ecl, libpynac-dev.

Yes, let's add them in #30859.

Some of the information that you put in the ticket description above could go as comments into the debian.txt files.

comment:159 Changed 2 years ago by Tobias Diez

Thanks for opening a new ticket for this!

Concerning the conda integration, this ticket here might actually help with this. You can use pipenv on top of conda, see https://pipenv.pypa.io/en/latest/advanced/#pipenv-and-other-python-distributions.

comment:160 Changed 2 years ago by Tobias Diez

Description: modified (diff)

comment:161 Changed 2 years ago by git

Commit: 373d1c12e77647541fa7fc8b315aae98ee0298cd096df15a1253e6cb0fa75c92e29458f29d5ad77d

Branch pushed to git repo; I updated commit sha1. New commits:

096df15Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace

comment:162 Changed 2 years ago by git

Commit: 096df15a1253e6cb0fa75c92e29458f29d5ad77d8eb2f8ec11943c8f7175ce4d8a293673ab18b774

Branch pushed to git repo; I updated commit sha1. New commits:

8eb2f8eRemove homfly workaround

comment:163 Changed 2 years ago by git

Commit: 8eb2f8ec11943c8f7175ce4d8a293673ab18b77417f367eebf0e329275773e0c048f3009b2de19ed

Branch pushed to git repo; I updated commit sha1. New commits:

17f367eRefactor shared library discovery

comment:164 Changed 2 years ago by Tobias Diez

Dependencies: #30779, #30672, #30673, #30709, #30706, #30833, #30748#30779, #30672, #30673, #30709, #30706, #30901, #30748

comment:165 Changed 2 years ago by git

Commit: 17f367eebf0e329275773e0c048f3009b2de19ed680f85d72cf88ae96e0bc4bf7b422e694f353437

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

c47c4bfCorrect indent
23b6336Merge branch 'public/build/multiarchsimple' of git://trac.sagemath.org/sage into public/build/inplace
6a52fbfRemove lazy import finish startup
f96025eMerge branch 'develop' of git://github.com/sagemath/sage into public/build/startupWarning
b952bb5Fix doctests
c6857ddHopefully fix doctest
23da14bWrap each sage.all import in startup guard
2f4055eSpecify that exit was succesful
d4d94a8Indeed replace exception by print statement
680f85dMerge branch 'public/build/startupWarning' of git://trac.sagemath.org/sage into public/build/inplace

comment:166 Changed 2 years ago by git

Commit: 680f85d72cf88ae96e0bc4bf7b422e694f353437ddcf2e3ba09cb3731042c1372b93e4db84f93b13

Branch pushed to git repo; I updated commit sha1. New commits:

ddcf2e3Cleanup code

comment:167 Changed 2 years ago by git

Commit: ddcf2e3ba09cb3731042c1372b93e4db84f93b13772baf208259ff75edebdbf6f340c700be0681ab

Branch pushed to git repo; I updated commit sha1. New commits:

2088ffdUpgrade to dotnet 5.0 rc2
1b1750dUpgrade to dotnet 5.0
638ecbeInstall wget before using it
772baf2Install tzdata earlier

comment:168 Changed 2 years ago by Tobias Diez

I've now merged the latest version of the dependencies, and cleaned the code.

Matthias (and others, of course), it would be nice if you could review the code (and its dependencies). I have a hard time working on other tickets as long as this ticket is not merged. So I would be grateful if that happens as soon as possible. Thanks!

comment:169 Changed 2 years ago by Matthias Köppe

See if you can get doctesting to work on this branch - by merging #30578.

comment:170 Changed 2 years ago by Matthias Köppe

Also removing sage-env-config from the list of installed scripts is not good. Instead please review and merge #29850 - which reassigns this script to the sage_conf package.

comment:171 in reply to:  169 ; Changed 2 years ago by Matthias Köppe

Replying to mkoeppe:

See if you can get doctesting to work on this branch - by merging #30578.

In #30578 I had to add a few more packages to requirements.txt that you don't have in Pipfile yet, so that more doctests pass.

comment:172 in reply to:  170 Changed 2 years ago by Tobias Diez

Replying to mkoeppe:

Also removing sage-env-config from the list of installed scripts is not good. Instead please review and merge #29850 - which reassigns this script to the sage_conf package.

I agree #29850 is a better solution. I had a look at that ticket, but don't really feel comfortable enough with the code to review it. Sorry! However, I've now added it as a dependency of this branch, and will merge as soon as it is reviewed.

comment:173 Changed 2 years ago by Tobias Diez

Dependencies: #30779, #30672, #30673, #30709, #30706, #30901, #30748#30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850

comment:174 in reply to:  171 ; Changed 2 years ago by Tobias Diez

Replying to mkoeppe:

Replying to mkoeppe:

See if you can get doctesting to work on this branch - by merging #30578.

In #30578 I had to add a few more packages to requirements.txt that you don't have in Pipfile yet, so that more doctests pass.

Thanks for the pointer! I've now added these packages also to the pipfile. However, it is way beyond the scope of this ticket to make the doctest pass in the GH pipenv workflow. There are just so many things that need to be changed in order for the doctests to find the correct dependencies. For example, here are a few relevant error messages:

FileNotFoundError: [Errno 2] No such file or directory: '/mnt/d/Programming/Projects/sage/src/.venv/bin/gap'

RuntimeError: unable to start singular because the command 'Singular -t --ticks-per-sec 1000 --cntrlc=a' failed: The command was not found or was not executable: Singular.

RuntimeError: In order to initialize the database, /__w/sage/sage/src/.venv/share/conway_polynomials/conway_polynomials.p must exist.

I know that some of these errors would disappear if one use your recent progress towards installation of sage-build wheels.

Anyway, the aim of this ticket is to create a mostly-working editable install of sagelib. Making the doctests pass relative to this editable install in a venv should be subject of a follow-up ticket.

comment:175 Changed 2 years ago by git

Commit: 772baf208259ff75edebdbf6f340c700be0681abad0f33cd4c67d2eb31cbf864e8112caeb8958990

Branch pushed to git repo; I updated commit sha1. New commits:

d3174ffFix merge problems
7e926c7Run compilation in parallel
488e6dcUpdate and extend dependencies
68c98d3Fix merge error for gap
f1b1f01Install singular package as well
ad0f33cRemove parallelization again (doesn't work)

comment:176 Changed 2 years ago by git

Commit: ad0f33cd4c67d2eb31cbf864e8112caeb895899077e1af16bcfce183bfe072cfb7324e82ee95a478

Branch pushed to git repo; I updated commit sha1. New commits:

20885b6MacOS: Find openblas
7f11466Correctly specify arb dependency not flint
223c2e4Mac: add flint as dependency
77e1af1Mac: Add arb

comment:177 in reply to:  174 Changed 2 years ago by Matthias Köppe

Replying to gh-tobiasdiez:

Anyway, the aim of this ticket is to create a mostly-working editable install of sagelib.

Most of the changes that you are doing in this ticket have really nothing to do with this goal. As I have pointed out previously, you are trying to solve the orthogonal problem of making sagelib work outside of the Sage distribution, i.e., without SAGE_LOCAL.

comment:178 Changed 2 years ago by Tobias Diez

I agree. The thing is that it is currently my only way to work with sage, and thus test the editable install. But it's good that you are pushing from the other side and work on the editable install based on a working sage distribution.

So, is there something that I can still do here? I see that verifying and testing that the editable install works is important. It's just not very realistic to have a fully functional doctest without SAGE_LOCAL at the moment. However, I can also not very actively contribute to the editable install with SAGE_LOCAL.

comment:179 in reply to:  178 ; Changed 2 years ago by Matthias Köppe

Replying to gh-tobiasdiez:

I agree. The thing is that it is currently my only way to work with sage, and thus test the editable install.

Is the WSL build of the Sage distribution still failing for you? Have you tried a different Linux distribution? So far the WSL runs on GH Actions have not been very conclusive - they have failed for unrelated reasons - you could work on #30910 to improve it

comment:180 Changed 2 years ago by Matthias Köppe

Also, I am interested in a GH Actions workflow for the various new installations (venv based on requirements on top of SAGE_LOCAL, venv based on Pipfile on top of SAGE_LOCAL, conda without SAGE_LOCAL, etc.) This ticket adds something like this, but I don't like that it does Ubuntu-specific things again. We have a better way of doing such things already so that everything can be tested on all supported distributions, not just some Ubuntu. I would hope that we can add such workflows in #30845. #30859 will help with this. The powershell bits to work with system packages could also be valuable - see #30865.

comment:181 in reply to:  179 Changed 2 years ago by Tobias Diez

Replying to mkoeppe:

Is the WSL build of the Sage distribution still failing for you? Have you tried a different Linux distribution?

Yes, it's still not working. In a few weeks I'll get a new PC and then will try again afresh using WSL 2.0.

comment:182 in reply to:  180 Changed 2 years ago by Tobias Diez

Replying to mkoeppe:

This ticket adds something like this, but I don't like that it does Ubuntu-specific things again. We have a better way of doing such things already so that everything can be tested on all supported distributions, not just some Ubuntu.

Yes and no. Since the whole workflow only takes about 2 hours, my hope was that one can use it in the longterm to run it on each push to each branch, replacing/complementing the patchbot. But of course it could also serve as the blueprint for a complete matrix integration test suite. Anyway, for the moment it serves as a record of the things I did locally on my system, and proofs that the editable install without SAGE_LOCAL works to a large degree.

comment:183 Changed 2 years ago by Tobias Diez

Dependencies: #30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850#30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850, #29314

After playing around with it further I was able to make most of the tests pass. There are still a few hundreds that fail (see below for details), most of which are related to issues with maxima (TypeError?: unable to start maxima: End Of File (EOF). Exception style platform.) or fpylll (blocked due to https://github.com/fplll/fpylll/issues/185). Since most test pass and there are many subpackages for which all tests pass, I'd say that's enough "proof of concept" for this ticket and would leave the rest for follow-up tickets.

2020-11-17T19:31:00.7800025Z sage -t --random-seed=0 /__w/sage/sage/src/sage/interacts/debugger.py  # 2 doctests failed
2020-11-17T19:31:00.7801032Z sage -t --random-seed=0 /__w/sage/sage/src/sage/interfaces/singular.py  # 5 doctests failed
2020-11-17T19:31:00.7802028Z sage -t --random-seed=0 /__w/sage/sage/src/sage/interfaces/maxima_lib.py  # 1 doctest failed
2020-11-17T19:31:00.7803005Z sage -t --random-seed=0 /__w/sage/sage/src/sage/interfaces/quit.py  # 6 doctests failed
2020-11-17T19:31:00.7803974Z sage -t --random-seed=0 /__w/sage/sage/src/sage/interfaces/maxima.py  # 144 doctests failed
2020-11-17T19:31:00.7804998Z sage -t --random-seed=0 /__w/sage/sage/src/sage/interfaces/maxima_abstract.py  # 168 doctests failed
2020-11-17T19:31:00.7806039Z sage -t --random-seed=0 /__w/sage/sage/src/sage/interfaces/interface.py  # 90 doctests failed
2020-11-17T19:31:00.7807034Z sage -t --random-seed=0 /__w/sage/sage/src/sage/interfaces/expect.py  # 6 doctests failed
2020-11-17T19:31:00.7808001Z sage -t --random-seed=0 /__w/sage/sage/src/sage/interfaces/tests.py  # 3 doctests failed
2020-11-17T19:31:00.7809010Z sage -t --random-seed=0 /__w/sage/sage/src/sage/interfaces/gap_workspace.py  # 2 doctests failed
2020-11-17T19:31:00.7810009Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/rational.pyx  # 2 doctests failed
2020-11-17T19:31:00.7810921Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/ring.pyx  # 1 doctest failed
2020-11-17T19:31:00.7811830Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/infinity.py  # 1 doctest failed
2020-11-17T19:31:00.7812754Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/real_mpfr.pyx  # 9 doctests failed
2020-11-17T19:31:00.7813669Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/real_mpfi.pyx  # 3 doctests failed
2020-11-17T19:31:00.7814600Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/complex_mpfr.pyx  # 2 doctests failed
2020-11-17T19:31:00.7815569Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/localization.py  # 1 doctest failed
2020-11-17T19:31:00.7816785Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/tests.py  # 2 doctests failed
2020-11-17T19:31:00.7817714Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/integer_ring.pyx  # 5 doctests failed
2020-11-17T19:31:00.7818771Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/polynomial/polynomial_zmod_flint.pyx  # 1 doctest failed
2020-11-17T19:31:00.7819934Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/polynomial/polynomial_quotient_ring.py  # 1 doctest failed
2020-11-17T19:31:00.7821093Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/polynomial/polynomial_modn_dense_ntl.pyx  # 6 doctests failed
2020-11-17T19:31:00.7822245Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/polynomial/polynomial_element.pyx  # 13 doctests failed
2020-11-17T19:31:00.7823414Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/polynomial/polynomial_number_field.pyx  # 19 doctests failed
2020-11-17T19:31:00.7824560Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/polynomial/multi_polynomial_ring.py  # 2 doctests failed
2020-11-17T19:31:00.7825670Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/finite_rings/residue_field.pyx  # 14 doctests failed
2020-11-17T19:31:00.7826727Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/number_field/S_unit_solver.py  # 18 doctests failed
2020-11-17T19:31:00.7827787Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/number_field/number_field_base.pyx  # 15 doctests failed
2020-11-17T19:31:00.7828969Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/number_field/class_group.py  # 2 doctests failed
2020-11-17T19:31:00.7830020Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/number_field/number_field_ideal.py  # 1 doctest failed
2020-11-17T19:31:00.7831068Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/number_field/galois_group.py  # 58 doctests failed
2020-11-17T19:31:00.7832134Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/number_field/splitting_field.py  # 28 doctests failed
2020-11-17T19:31:00.7833172Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/number_field/order.py  # 6 doctests failed
2020-11-17T19:31:00.7834186Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/number_field/number_field.py  # 33 doctests failed
2020-11-17T19:31:00.7835200Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/number_field/unit_group.py  # 1 doctest failed
2020-11-17T19:31:00.7836234Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/number_field/number_field_rel.py  # 11 doctests failed
2020-11-17T19:31:00.7837488Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/number_field/number_field_element.pyx  # 27 doctests failed
2020-11-17T19:31:00.7838576Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/number_field/bdd_height.py  # 2 doctests failed
2020-11-17T19:31:00.7839663Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/valuation/inductive_valuation.py  # 4 doctests failed
2020-11-17T19:31:00.7840725Z sage -t --random-seed=0 /__w/sage/sage/src/sage/cpython/dict_del_by_value.pyx  # 2 doctests failed
2020-11-17T19:31:00.7841710Z sage -t --random-seed=0 /__w/sage/sage/src/sage/databases/cremona.py  # 2 doctests failed
2020-11-17T19:31:00.7842673Z sage -t --random-seed=0 /__w/sage/sage/src/sage/groups/cubic_braid.py  # 6 doctests failed
2020-11-17T19:31:00.7843635Z sage -t --random-seed=0 /__w/sage/sage/src/sage/sandpiles/sandpile.py  # 5 doctests failed
2020-11-17T19:31:00.7844616Z sage -t --random-seed=0 /__w/sage/sage/src/sage/coding/gabidulin_code.py  # 1 doctest failed
2020-11-17T19:31:00.7845566Z sage -t --random-seed=0 /__w/sage/sage/src/sage/tests/cmdline.py  # 59 doctests failed
2020-11-17T19:31:00.7846502Z sage -t --random-seed=0 /__w/sage/sage/src/sage/tests/benchmark.py  # 4 doctests failed
2020-11-17T19:31:00.7847448Z sage -t --random-seed=0 /__w/sage/sage/src/sage/tests/gap_packages.py  # 1 doctest failed
2020-11-17T19:31:00.7848644Z sage -t --random-seed=0 /__w/sage/sage/src/sage/tests/books/judson-abstract-algebra/galois-sage.py  # 9 doctests failed
2020-11-17T19:31:00.7850157Z sage -t --random-seed=0 /__w/sage/sage/src/sage/tests/books/judson-abstract-algebra/fields-sage.py  # 17 doctests failed
2020-11-17T19:31:00.7851577Z sage -t --random-seed=0 /__w/sage/sage/src/sage/tests/books/judson-abstract-algebra/domains-sage.py  # 5 doctests failed
2020-11-17T19:31:00.7853321Z sage -t --random-seed=0 /__w/sage/sage/src/sage/tests/books/computational-mathematics-with-sagemath/polynomes_doctest.py  # 1 doctest failed
2020-11-17T19:31:00.7855341Z sage -t --random-seed=0 /__w/sage/sage/src/sage/tests/books/computational-mathematics-with-sagemath/domaines_doctest.py  # 3 doctests failed
2020-11-17T19:31:00.7857340Z sage -t --random-seed=0 /__w/sage/sage/src/sage/tests/books/computational-mathematics-with-sagemath/combinat_doctest.py  # 1 doctest failed
2020-11-17T19:31:00.7859360Z sage -t --random-seed=0 /__w/sage/sage/src/sage/tests/books/computational-mathematics-with-sagemath/integration_doctest.py  # 1 doctest failed
2020-11-17T19:31:00.7861360Z sage -t --random-seed=0 /__w/sage/sage/src/sage/tests/books/computational-mathematics-with-sagemath/mpoly_doctest.py  # 2 doctests failed
2020-11-17T19:31:00.7863355Z sage -t --random-seed=0 /__w/sage/sage/src/sage/tests/books/computational-mathematics-with-sagemath/calculus_doctest.py  # 3 doctests failed
2020-11-17T19:31:00.7865352Z sage -t --random-seed=0 /__w/sage/sage/src/sage/tests/books/computational-mathematics-with-sagemath/sol/mpoly_doctest.py  # 2 doctests failed
2020-11-17T19:31:00.7867492Z sage -t --random-seed=0 /__w/sage/sage/src/sage/tests/books/computational-mathematics-with-sagemath/sol/graphique_doctest.py  # 8 doctests failed
2020-11-17T19:31:00.7868993Z sage -t --random-seed=0 /__w/sage/sage/src/sage/repl/configuration.py  # 1 doctest failed
2020-11-17T19:31:00.7869942Z sage -t --random-seed=0 /__w/sage/sage/src/sage/repl/image.py  # 4 doctests failed
2020-11-17T19:31:00.7870875Z sage -t --random-seed=0 /__w/sage/sage/src/sage/repl/interpreter.py  # 7 doctests failed
2020-11-17T19:31:00.7871818Z sage -t --random-seed=0 /__w/sage/sage/src/sage/repl/preparse.py  # 1 doctest failed
2020-11-17T19:31:00.7872750Z sage -t --random-seed=0 /__w/sage/sage/src/sage/repl/ipython_tests.py  # 1 doctest failed
2020-11-17T19:31:00.7873726Z sage -t --random-seed=0 /__w/sage/sage/src/sage/repl/rich_output/pretty_print.py  # 3 doctests failed
2020-11-17T19:31:00.7874739Z sage -t --random-seed=0 /__w/sage/sage/src/sage/repl/ipython_kernel/install.py  # 1 doctest failed
2020-11-17T19:31:00.7875703Z sage -t --random-seed=0 /__w/sage/sage/src/sage/doctest/forker.py  # 1 doctest failed
2020-11-17T19:31:00.7876633Z sage -t --random-seed=0 /__w/sage/sage/src/sage/doctest/control.py  # 2 doctests failed
2020-11-17T19:31:00.7883133Z sage -t --random-seed=0 /__w/sage/sage/src/sage/doctest/test.py  # 3 doctests failed
2020-11-17T19:31:00.7884402Z sage -t --random-seed=0 /__w/sage/sage/src/sage/libs/singular/function.pyx  # 1 doctest failed
2020-11-17T19:31:00.7885406Z sage -t --random-seed=0 /__w/sage/sage/src/sage/libs/pari/convert_sage.pyx  # 10 doctests failed
2020-11-17T19:31:00.7886362Z sage -t --random-seed=0 /__w/sage/sage/src/sage/libs/pari/__init__.py  # 12 doctests failed
2020-11-17T19:31:00.7887265Z sage -t --random-seed=0 /__w/sage/sage/src/sage/libs/pari/tests.py  # 9 doctests failed
2020-11-17T19:31:00.7888179Z sage -t --random-seed=0 /__w/sage/sage/src/sage/libs/pynac/pynac.pyx  # 1 doctest failed
2020-11-17T19:31:00.7889240Z sage -t --random-seed=0 /__w/sage/sage/src/sage/dynamics/arithmetic_dynamics/generic_ds.py  # 10 doctests failed
2020-11-17T19:31:00.7890445Z sage -t --random-seed=0 /__w/sage/sage/src/sage/dynamics/arithmetic_dynamics/projective_ds.py  # 25 doctests failed
2020-11-17T19:31:00.7891469Z sage -t --random-seed=0 /__w/sage/sage/src/sage/arith/misc.py  # 20 doctests failed
2020-11-17T19:31:00.7892598Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/hyperelliptic_curves/hyperelliptic_generic.py  # 3 doctests failed
2020-11-17T19:31:00.7893789Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/product_projective/space.py  # 1 doctest failed
2020-11-17T19:31:00.7895096Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/product_projective/rational_point.py  # 1 doctest failed
2020-11-17T19:31:00.7896170Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/affine/affine_space.py  # 1 doctest failed
2020-11-17T19:31:00.7897209Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/generic/algebraic_scheme.py  # 1 doctest failed
2020-11-17T19:31:00.7898340Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/riemann_surfaces/riemann_surface.py  # 31 doctests failed
2020-11-17T19:31:00.7899522Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/projective/projective_rational_point.py  # 4 doctests failed
2020-11-17T19:31:00.7900680Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/projective/projective_space.py  # 1 doctest failed
2020-11-17T19:31:00.7901771Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/toric/toric_subscheme.py  # 11 doctests failed
2020-11-17T19:31:00.7902793Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/toric/divisor.py  # 2 doctests failed
2020-11-17T19:31:00.7903768Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/toric/points.py  # 1 doctest failed
2020-11-17T19:31:00.7904739Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/toric/variety.py  # 4 doctests failed
2020-11-17T19:31:00.7905707Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/toric/ideal.py  # 43 doctests failed
2020-11-17T19:31:00.7906825Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/heegner.py  # 33 doctests failed
2020-11-17T19:31:00.7907884Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/kraus.py  # 4 doctests failed
2020-11-17T19:31:00.7908978Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/constructor.py  # 4 doctests failed
2020-11-17T19:31:00.7910128Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/gal_reps_number_field.py  # 2 doctests failed
2020-11-17T19:31:00.7911278Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/ell_rational_field.py  # 14 doctests failed
2020-11-17T19:31:00.7912397Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/isogeny_class.py  # 2 doctests failed
2020-11-17T19:31:00.7913509Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/ell_number_field.py  # 26 doctests failed
2020-11-17T19:31:00.7914601Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/gal_reps.py  # 2 doctests failed
2020-11-17T19:31:00.7915708Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/isogeny_small_degree.py  # 1 doctest failed
2020-11-17T19:31:00.7916803Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/ell_egros.py  # 4 doctests failed
2020-11-17T19:31:00.7918998Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/ell_curve_isogeny.py  # 1 doctest failed
2020-11-17T19:31:00.7920093Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/ell_field.py  # 2 doctests failed
2020-11-17T19:31:00.7921146Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/plane_conics/con_field.py  # 7 doctests failed
2020-11-17T19:31:00.7922203Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/plane_conics/con_number_field.py  # 3 doctests failed
2020-11-17T19:31:00.7923188Z sage -t --random-seed=0 /__w/sage/sage/src/sage/features/__init__.py  # 1 doctest failed
2020-11-17T19:31:00.7924137Z sage -t --random-seed=0 /__w/sage/sage/src/sage/calculus/desolvers.py  # 8 doctests failed
2020-11-17T19:31:00.7925128Z sage -t --random-seed=0 /__w/sage/sage/src/sage/calculus/calculus.py  # 25 doctests failed
2020-11-17T19:31:00.7926071Z sage -t --random-seed=0 /__w/sage/sage/src/sage/crypto/lattice.py  # 1 doctest failed
2020-11-17T19:31:00.7927011Z sage -t --random-seed=0 /__w/sage/sage/src/sage/combinat/tutorial.py  # 1 doctest failed
2020-11-17T19:31:00.7927975Z sage -t --random-seed=0 /__w/sage/sage/src/sage/combinat/combinat.py  # 5 doctests failed
2020-11-17T19:31:00.7929092Z sage -t --random-seed=0 /__w/sage/sage/src/sage/combinat/degree_sequences.pyx  # 1 doctest failed
2020-11-17T19:31:00.7930148Z sage -t --random-seed=0 /__w/sage/sage/src/sage/categories/algebra_functor.py  # 5 doctests failed
2020-11-17T19:31:00.7931175Z sage -t --random-seed=0 /__w/sage/sage/src/sage/categories/number_fields.py  # 1 doctest failed
2020-11-17T19:31:00.7932168Z sage -t --random-seed=0 /__w/sage/sage/src/sage/categories/rings.py  # 12 doctests failed
2020-11-17T19:31:00.7933122Z sage -t --random-seed=0 /__w/sage/sage/src/sage/categories/primer.py  # 1 doctest failed
2020-11-17T19:31:00.7934130Z sage -t --random-seed=0 /__w/sage/sage/src/sage/categories/quotient_fields.py  # 7 doctests failed
2020-11-17T19:31:00.7935102Z sage -t --random-seed=0 /__w/sage/sage/src/sage/modular/cusps_nf.py  # 1 doctest failed
2020-11-17T19:31:00.7936063Z sage -t --random-seed=0 /__w/sage/sage/src/sage/modular/etaproducts.py  # 10 doctests failed
2020-11-17T19:31:00.7937057Z sage -t --random-seed=0 /__w/sage/sage/src/sage/modular/modsym/p1list_nf.py  # 1 doctest failed
2020-11-17T19:31:00.7938011Z sage -t --random-seed=0 /__w/sage/sage/src/sage/structure/formal_sum.py  # 1 doctest failed
2020-11-17T19:31:00.7938985Z sage -t --random-seed=0 /__w/sage/sage/src/sage/structure/element.pyx  # 5 doctests failed
2020-11-17T19:31:00.7940062Z sage -t --random-seed=0 /__w/sage/sage/src/sage/structure/dynamic_class.py  # 1 doctest failed
2020-11-17T19:31:00.7941022Z sage -t --random-seed=0 /__w/sage/sage/src/sage/misc/nested_class.pyx  # 1 doctest failed
2020-11-17T19:31:00.7941915Z sage -t --random-seed=0 /__w/sage/sage/src/sage/misc/trace.py  # 3 doctests failed
2020-11-17T19:31:00.7942828Z sage -t --random-seed=0 /__w/sage/sage/src/sage/misc/sage_ostools.pyx  # 1 doctest failed
2020-11-17T19:31:00.7943764Z sage -t --random-seed=0 /__w/sage/sage/src/sage/misc/functional.py  # 1 doctest failed
2020-11-17T19:31:00.7944692Z sage -t --random-seed=0 /__w/sage/sage/src/sage/misc/edit_module.py  # 1 doctest failed
2020-11-17T19:31:00.7945625Z sage -t --random-seed=0 /__w/sage/sage/src/sage/misc/temporary_file.py  # 1 doctest failed
2020-11-17T19:31:00.7946578Z sage -t --random-seed=0 /__w/sage/sage/src/sage/misc/lazy_attribute.pyx  # 1 doctest failed
2020-11-17T19:31:00.7947551Z sage -t --random-seed=0 /__w/sage/sage/src/sage/misc/sageinspect.py  # 13 doctests failed
2020-11-17T19:31:00.7948497Z sage -t --random-seed=0 /__w/sage/sage/src/sage/misc/sagedoc.py  # 3 doctests failed
2020-11-17T19:31:00.7949390Z sage -t --random-seed=0 /__w/sage/sage/src/sage/misc/compat.py  # 1 doctest failed
2020-11-17T19:31:00.7950304Z sage -t --random-seed=0 /__w/sage/sage/src/sage/misc/sphinxify.py  # 4 doctests failed
2020-11-17T19:31:00.7951326Z sage -t --random-seed=0 /__w/sage/sage/src/sage/functions/hypergeometric.py  # 7 doctests failed
2020-11-17T19:31:00.7952357Z sage -t --random-seed=0 /__w/sage/sage/src/sage/functions/bessel.py  # 7 doctests failed
2020-11-17T19:31:00.7953305Z sage -t --random-seed=0 /__w/sage/sage/src/sage/functions/gamma.py  # 2 doctests failed
2020-11-17T19:31:00.7954306Z sage -t --random-seed=0 /__w/sage/sage/src/sage/functions/special.py  # 1 doctest failed
2020-11-17T19:31:00.7955238Z sage -t --random-seed=0 /__w/sage/sage/src/sage/functions/trig.py  # 1 doctest failed
2020-11-17T19:31:00.7956168Z sage -t --random-seed=0 /__w/sage/sage/src/sage/functions/error.py  # 7 doctests failed
2020-11-17T19:31:00.7957101Z sage -t --random-seed=0 /__w/sage/sage/src/sage/functions/log.py  # 6 doctests failed
2020-11-17T19:31:00.7958207Z sage -t --random-seed=0 /__w/sage/sage/src/sage/functions/other.py  # 15 doctests failed
2020-11-17T19:31:00.7959216Z sage -t --random-seed=0 /__w/sage/sage/src/sage/functions/orthogonal_polys.py  # 11 doctests failed
2020-11-17T19:31:00.7960272Z sage -t --random-seed=0 /__w/sage/sage/src/sage/geometry/polyhedron/parent.py  # 2 doctests failed
2020-11-17T19:31:00.7961352Z sage -t --random-seed=0 /__w/sage/sage/src/sage/geometry/polyhedron/palp_database.py  # 1 doctest failed
2020-11-17T19:31:00.7962568Z sage -t --random-seed=0 /__w/sage/sage/src/sage/algebras/quaternion_algebra_element.py  # 4 doctests failed
2020-11-17T19:31:00.7963697Z sage -t --random-seed=0 /__w/sage/sage/src/sage/algebras/quatalg/quaternion_algebra.py  # 6 doctests failed
2020-11-17T19:31:00.7964873Z sage -t --random-seed=0 /__w/sage/sage/src/sage/algebras/quatalg/quaternion_algebra_element.pyx  # 38 doctests failed
2020-11-17T19:31:00.7965949Z sage -t --random-seed=0 /__w/sage/sage/src/sage/modules/free_module.py  # 5 doctests failed
2020-11-17T19:31:00.7966956Z sage -t --random-seed=0 /__w/sage/sage/src/sage/modules/free_module_morphism.py  # 3 doctests failed
2020-11-17T19:31:00.7967988Z sage -t --random-seed=0 /__w/sage/sage/src/sage/modules/free_module_integer.py  # 57 doctests failed
2020-11-17T19:31:00.7969027Z sage -t --random-seed=0 /__w/sage/sage/src/sage/modules/vector_space_morphism.py  # 3 doctests failed
2020-11-17T19:31:00.7970041Z sage -t --random-seed=0 /__w/sage/sage/src/sage/modules/fg_pid/fgp_module.py  # 3 doctests failed
2020-11-17T19:31:00.7971051Z sage -t --random-seed=0 /__w/sage/sage/src/sage/symbolic/expression.pyx  # 17 doctests failed
2020-11-17T19:31:00.7972071Z sage -t --random-seed=0 /__w/sage/sage/src/sage/symbolic/assumptions.py  # 3 doctests failed
2020-11-17T19:31:00.7973064Z sage -t --random-seed=0 /__w/sage/sage/src/sage/symbolic/constants_c.pyx  # 1 doctest failed
2020-11-17T19:31:00.7974140Z sage -t --random-seed=0 /__w/sage/sage/src/sage/symbolic/maxima_wrapper.py  # 2 doctests failed
2020-11-17T19:31:00.7975123Z sage -t --random-seed=0 /__w/sage/sage/src/sage/symbolic/relation.py  # 4 doctests failed
2020-11-17T19:31:00.7976102Z sage -t --random-seed=0 /__w/sage/sage/src/sage/symbolic/constants.py  # 17 doctests failed
2020-11-17T19:31:00.7977173Z sage -t --random-seed=0 /__w/sage/sage/src/sage/symbolic/expression_conversions.py  # 10 doctests failed
2020-11-17T19:31:00.7978261Z sage -t --random-seed=0 /__w/sage/sage/src/sage/graphs/comparability.pyx  # 1 doctest failed
2020-11-17T19:31:00.7979313Z sage -t --random-seed=0 /__w/sage/sage/src/sage/graphs/digraph_generators.py  # 8 doctests failed
2020-11-17T19:31:00.7980236Z sage -t --random-seed=0 /__w/sage/sage/src/sage/graphs/graph.py  # Timed out
2020-11-17T19:31:00.7981184Z sage -t --random-seed=0 /__w/sage/sage/src/sage/graphs/connectivity.pyx  # 2 doctests failed
2020-11-17T19:31:00.7982192Z sage -t --random-seed=0 /__w/sage/sage/src/sage/graphs/graph_generators.py  # Timed out
2020-11-17T19:31:00.7983217Z sage -t --random-seed=0 /__w/sage/sage/src/sage/graphs/hypergraph_generators.py  # 9 doctests failed
2020-11-17T19:31:00.7984259Z sage -t --random-seed=0 /__w/sage/sage/src/sage/graphs/graph_coloring.pyx  # 2 doctests failed
2020-11-17T19:31:00.7985206Z sage -t --random-seed=0 /__w/sage/sage/src/sage/graphs/genus.pyx  # 1 doctest failed
2020-11-17T19:31:00.7986147Z sage -t --random-seed=0 /__w/sage/sage/src/sage/graphs/generic_graph_pyx.pyx  # 2 doctests failed
2020-11-17T19:31:00.7987202Z sage -t --random-seed=0 /__w/sage/sage/src/sage/sets/set_from_iterator.py  # Timed out (and interrupt failed)
2020-11-17T19:31:00.7988200Z sage -t --random-seed=0 /__w/sage/sage/src/sage/matrix/matrix1.pyx  # 3 doctests failed
2020-11-17T19:31:00.7989131Z sage -t --random-seed=0 /__w/sage/sage/src/sage/matrix/matrix2.pyx  # 3 doctests failed
2020-11-17T19:31:00.7990123Z sage -t --random-seed=0 /__w/sage/sage/src/sage/matrix/matrix_rational_dense.pyx  # 1 doctest failed
2020-11-17T19:31:00.7991189Z sage -t --random-seed=0 /__w/sage/sage/src/sage/matrix/matrix_integer_dense.pyx  # 14 doctests failed
2020-11-17T19:31:00.7992217Z sage -t --random-seed=0 /__w/sage/sage/src/sage/matrix/matrix_symbolic_dense.pyx  # 1 doctest failed

https://github.com/tobiasdiez/sage/runs/1413136300?check_suite_focus=true

comment:184 Changed 2 years ago by git

Commit: 77e1af16bcfce183bfe072cfb7324e82ee95a478d1341b0ab3b8bbc47a25beea2a8392632c981bfb

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

9a5a145version bump
d4c4453remove old workaround from #22626
ebe9780this file should exist - leave just comments there
a507fb9trac 29314: fix version numbers in patches for gap_packages
9836f79move GAP's PackageManager to gap from gap_packages
435d82fMerge branch 'u/dimpase/packages/gap411' of git://trac.sagemath.org/sage into public/build/inplace
2fc8bf5Hangs with maxima, so remove again
f6c1d87Try to install maxima again
13e96adFix workflow syntax
d1341b0Can we now activate cysignals again?

comment:185 Changed 2 years ago by Tobias Diez

Dependencies: #30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850, #29314#30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850, #29314, #30937, #30936, #30935, #30934

comment:186 Changed 2 years ago by Tobias Diez

Dependencies: #30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850, #29314, #30937, #30936, #30935, #30934#30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850, #29314, #30937, #30936, #30935, #30934, #30859, #30923, #30910

comment:187 Changed 2 years ago by Tobias Diez

Description: modified (diff)

comment:188 Changed 2 years ago by Tobias Diez

Dependencies: #30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850, #29314, #30937, #30936, #30935, #30934, #30859, #30923, #30910#30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850, #29314, #30937, #30936, #30935, #30934, #30859, #30923, #30910, #30944

comment:189 Changed 2 years ago by Tobias Diez

Dependencies: #30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850, #29314, #30937, #30936, #30935, #30934, #30859, #30923, #30910, #30944#30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850, #29314, #30937, #30936, #30935, #30731 , #30859, #30923, #30910, #30944

comment:190 Changed 2 years ago by Matthias Köppe

Keywords: sd111 added

Hoping we can make progress on this ticket this week - https://wiki.sagemath.org/days111

comment:191 Changed 2 years ago by Matthias Köppe

Status: needs_reviewneeds_work

comment:192 Changed 2 years ago by git

Commit: d1341b0ab3b8bbc47a25beea2a8392632c981bfb17e06daef346efc2f83f7b5304fc598c181079de

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

4393801Build all of sage deps
2027ebbRun make in parallel
405e8d2Install dependencies on mac with tox as well
894b9d5Fix indent
4cefc89Always run manually to see what's going on
ee6aaebDon't require spkg-configure.m4 file
8cc37e7Cleanup mac workflow
c571a07Fix maxima install
8ee33f2Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace
17e06daUpgrade packages

comment:193 Changed 2 years ago by Tobias Diez

From my side, this is actually ready for review. I've been using it now constantly the last few weeks and it is working. Or do you have something that I should be improving/working on?

comment:194 Changed 2 years ago by Dima Pasechnik

Description: modified (diff)

comment:195 Changed 2 years ago by git

Commit: 17e06daef346efc2f83f7b5304fc598c181079de5c719fe14a0fa6a0aa966031786cf6f3c437b239

Branch pushed to git repo; I updated commit sha1. New commits:

3fcaf5fMerge branch 'develop' of git://github.com/sagemath/sage into public/build/multiarchsimple
090e6f1Simplify code
fa4556aRemove _get_sage_local
7cc2005Merge branch 'public/build/multiarchsimple' of git://trac.sagemath.org/sage into public/build/inplace
a590924Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace
afa0518Make autogen more stable
5c719feFix functional.py:56: DeprecationWarning: invalid escape sequence \d

comment:196 Changed 2 years ago by Tobias Diez

Status: needs_workneeds_review

Merged the latest develop branch.

comment:197 Changed 2 years ago by Tobias Diez

Description: modified (diff)

comment:198 Changed 2 years ago by git

Commit: 5c719fe14a0fa6a0aa966031786cf6f3c437b2397f0335de97d12b1bd5e6cc56879ab47f7423665a

Branch pushed to git repo; I updated commit sha1. New commits:

5e4f607Change import exception to print message
a5eb21dFix doctest
7f0335dMerge branch 'public/build/startupWarningMinimal' of git://trac.sagemath.org/sage into public/build/inplace

comment:199 Changed 2 years ago by Tobias Diez

Dependencies: #30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850, #29314, #30937, #30936, #30935, #30731 , #30859, #30923, #30910, #30944#30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30936, #30935, #30731 , #30859, #30923, #30910, #30944

New commits:

5e4f607Change import exception to print message
a5eb21dFix doctest
7f0335dMerge branch 'public/build/startupWarningMinimal' of git://trac.sagemath.org/sage into public/build/inplace

comment:200 in reply to:  156 Changed 2 years ago by Matthias Köppe

Replying to mkoeppe:

It has been reported that the current src/setup.py (in Sage 9.2) is able to install a working sagelib on top of conda (see #28745).

I'll create a test for this in #30845, and then we can test the present ticket to make sure it does not break anything.

Ensuring that the feature of installing sagelib on top of conda is not broken is a prerequisite for merging this ticket.

To move this ticket forward, could you help me with this GH Actions test? (This should be on top of #30944, which still needs review)

I think there should be one new workflow for all tests that do not build the Sage distribution (no build/pkgs).

comment:201 Changed 2 years ago by Tobias Diez

Sure, I will have a look at this. Could you by extending #30845 to include in more detail how the manual workflow with conda looks like (I've no experience with conda) and why #30944 is needed.

Maybe it's a good idea to review how the GH workflows are structured and revise this if necessary. For example, focus more on the purpose vs how it done, e.g. "test that building packages in build/pgks" works, "test that sagelib installed on top of various setups works". For the latter, my hope with pipenv was actually that one can easily run it on every push on every branch as a gateway in addition to the patchbot (since the time it takes is very minimal, time of setup of < 1h, + time for doctests).

comment:202 in reply to:  201 Changed 2 years ago by Matthias Köppe

Replying to gh-tobiasdiez:

Sure, I will have a look at this. Could you by extending #30845 to include in more detail how the manual workflow with conda looks like (I've no experience with conda) and

Done.

why #30944 is needed.

Well, it will be testing something that currently happens to work but is not automatically tested.

Maybe it's a good idea to review how the GH workflows are structured and revise this if necessary.

Sure, I'll be happy about more eyes on the workflows.

For example, focus more on the purpose vs how it done

All workflows test from-scratch builds of the Sage distribution. Two workflows separately test optional / experimental packages. That's really all.

comment:203 in reply to:  201 Changed 2 years ago by Matthias Köppe

Replying to gh-tobiasdiez:

"test that sagelib installed on top of various setups works". For the latter, my hope with pipenv was actually that one can easily run it on every push on every branch as a gateway in addition to the patchbot (since the time it takes is very minimal, time of setup of < 1h, + time for doctests).

An incremental build setup based on Docker has already been developed, which drives the current GitLab workflow. I'd recommend to start from there (it needs attention) instead of developing another thing from scratch... Take a look at #29536

comment:204 Changed 2 years ago by git

Commit: 7f0335de97d12b1bd5e6cc56879ab47f7423665af96b923fd5c342c5401f4e8f993546b523fe2cc0

Branch pushed to git repo; I updated commit sha1. New commits:

f96b923Use sage_conf

comment:205 Changed 2 years ago by Tobias Diez

Dependencies: #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30936, #30935, #30731 , #30859, #30923, #30910, #30944#30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944

comment:206 Changed 2 years ago by git

Commit: f96b923fd5c342c5401f4e8f993546b523fe2cc08c97e7c9290c067af0e0bd974f60f4b3966a516c

Branch pushed to git repo; I updated commit sha1. New commits:

8c97e7cCleanup code

comment:207 Changed 23 months ago by Tobias Diez

Since now most dependencies are merged in the develop branch, this ticket is now definitely ready for review.

comment:208 Changed 23 months ago by git

Commit: 8c97e7c9290c067af0e0bd974f60f4b3966a516cf871fc77be3b6ec719dc3f64a033ca78e6deea67

Branch pushed to git repo; I updated commit sha1. New commits:

71b4c06Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace
f871fc7Add rstcheck

comment:209 Changed 23 months ago by Tobias Diez

Matthias, I have a question: the current code works very well for system-installed packages. However, there are a few packages that are required but for which system-installed versions are not currently not supported (e.g. ecl). How could I use the sage-built versions of these packages for the built process of sagelib, i.e. for pip install -e .?

Take for example the import cdef extern from "ecl/ecl.h" in libs/ecl.pxd. I guess one needs to pass the path of local/include/ecl as argument to the C++ compiler. How is currently done?

comment:210 in reply to:  209 ; Changed 23 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

Take for example the import cdef extern from "ecl/ecl.h" in libs/ecl.pxd. I guess one needs to pass the path of local/include/ecl as argument to the C++ compiler. How is currently done?

We install all packages that are not accepted from the system into $SAGE_LOCAL, and scripts such as sage-env and sage-build-env set up environment variables such as LIBRARY_PATH and CPATH so that compilers find stuff. Configuration for Cython is somewhat separate - you are already familiar with the cython_aliases function.

For ECL, there's ticket #30770 to make this configuration more flexible.

comment:211 Changed 23 months ago by Matthias Köppe

Also, if you are able to use system packages for which we do not have system package detection yet (spkg-configure.m4), you can join the effort in #27330.

comment:212 in reply to:  209 ; Changed 23 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

How could I use the sage-built versions of these packages for the built process of sagelib, i.e. for pip install -e .?

One part of the answer to this involves the sage_conf package: Installing it into your virtual environment provides sage-env-config and the sage_conf module. So you may want to add it to Pipfile. (See also #30913).

comment:213 in reply to:  210 Changed 23 months ago by Tobias Diez

Replying to mkoeppe:

For ECL, there's ticket #30770 to make this configuration more flexible.

Thanks! I'll see if I can make progress towards this.

comment:214 in reply to:  212 Changed 23 months ago by Tobias Diez

Replying to mkoeppe:

One part of the answer to this involves the sage_conf package: Installing it into your virtual environment provides sage-env-config and the sage_conf module. So you may want to add it to Pipfile. (See also #30913).

That's already done:

+[dev-packages.8c46424]
+path = "./../build/pkgs/sage_conf/src"
+editable = true

comment:215 Changed 23 months ago by Tobias Diez

Dependencies: #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944#30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770

comment:216 Changed 23 months ago by git

Commit: f871fc77be3b6ec719dc3f64a033ca78e6deea67deeeb75c316c0e06c40e36ec65f472d4d3b90972

Branch pushed to git repo; I updated commit sha1. New commits:

8920ad5Update dependencies
3049e53Use ecl-config to determine compiler/linker flags for ecl
deeeb75Merge branch 'public/build/ecl-config' of git://trac.sagemath.org/sage into public/build/inplace

comment:217 Changed 23 months ago by Tobias Diez

Description: modified (diff)

comment:218 Changed 23 months ago by git

Commit: deeeb75c316c0e06c40e36ec65f472d4d3b909722ea4935dbae81bf1f2c9f9dc4e29f2feeb1a733f

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

9988c5fci-cygwin*.yml: Adjust to new script packages _bootstrap, _prereq
ab19133Merge branch 't/29124/script-packages-prereq-toolchain-bootstrap' into t/30944/tox__improve_local_sudo_ubuntu_standard
c6ad06eMerge branch 't/30944/tox__improve_local_sudo_ubuntu_standard' into t/31216/tox_ini__local___add_environment_variables_to_skip_system_package_installs__mechanism_for_a_local_interactive_shell
13f58fctox.ini (local): Run an interactive shell if posargs are bash
ace4ef6tox.ini (local): Add option SKIP_SYSTEM_PKG_INSTALL=1
ddc4f0btox.ini (local): Handle environment variables SKIP_BOOTSTRAP, SKIP_CONFIGURE
bb6aec4src/doc/en/developer/portability_testing.rst: Updates
6253bd8src/doc/en/developer/portability_testing.rst: New section on SKIP_... environment variables
a52e899Merge branch 'u/mkoeppe/tox_ini__local___add_environment_variables_to_skip_system_package_installs__mechanism_for_a_local_interactive_shell' of git://trac.sagemath.org/sage into public/build/inplace
2ea4935Split workflow in install and build

comment:219 Changed 23 months ago by Tobias Diez

Dependencies: #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770#30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216

comment:220 Changed 23 months ago by git

Commit: 2ea4935dbae81bf1f2c9f9dc4e29f2feeb1a733f6bcc5c34991dc4b81362b2427ff8d38b50f32c7e

Branch pushed to git repo; I updated commit sha1. New commits:

6bcc5c3Add esac

comment:221 Changed 23 months ago by git

Commit: 6bcc5c34991dc4b81362b2427ff8d38b50f32c7ed32b22b7091051522180ea58ea861b022e16e4d7

Branch pushed to git repo; I updated commit sha1. New commits:

d32b22bTry with config

comment:222 Changed 23 months ago by git

Commit: d32b22b7091051522180ea58ea861b022e16e4d773c99f3b7f797a7ab61507392c40268bb50e0a00

Branch pushed to git repo; I updated commit sha1. New commits:

73c99f3Ignore errors for now

comment:223 Changed 23 months ago by git

Commit: 73c99f3b7f797a7ab61507392c40268bb50e0a005cb58cf579b41780791b9bf0cb46563713efd1e5

Branch pushed to git repo; I updated commit sha1. New commits:

5cb58cfEnable build as root

comment:224 Changed 23 months ago by git

Commit: 5cb58cf579b41780791b9bf0cb46563713efd1e502542220efcfc3989a682ed082092c7f6e9a1c30

Branch pushed to git repo; I updated commit sha1. New commits:

0254222Cleanup github workflow

comment:225 Changed 23 months ago by git

Commit: 02542220efcfc3989a682ed082092c7f6e9a1c304dcd083615e7eb7218b595c61fd9b9a4754222cb

Branch pushed to git repo; I updated commit sha1. New commits:

4dcd083Use --all for the tests

comment:226 Changed 23 months ago by git

Commit: 4dcd083615e7eb7218b595c61fd9b9a4754222cbd30fda0a1bd9682d624feed871c3c94fa53921df

Branch pushed to git repo; I updated commit sha1. New commits:

d30fda0Also install maxima

comment:227 Changed 23 months ago by git

Commit: d30fda0a1bd9682d624feed871c3c94fa53921dfb17e6faef915267735b95135e326ffc74fd558b6

Branch pushed to git repo; I updated commit sha1. New commits:

b17e6faUse configure.ac as make target

comment:228 in reply to:  210 ; Changed 23 months ago by Tobias Diez

Replying to mkoeppe:

For ECL, there's ticket #30770 to make this configuration more flexible.

This seems to work now. Thanks for you help!

Two follow-up questions:

  1. Is there a make target similar to sagelib-build-deps that doesn't install pythonic packages such as cython.
  2. Why doesn't sagelib-build-deps install maxima?

comment:229 Changed 23 months ago by git

Commit: b17e6faef915267735b95135e326ffc74fd558b65d64c5a44b44e36c260eca75c60e179093f0b67f

Branch pushed to git repo; I updated commit sha1. New commits:

5d64c5aInstall/fix other packages

comment:230 Changed 23 months ago by git

Commit: 5d64c5a44b44e36c260eca75c60e179093f0b67fdbf2df2f53f7b1b7e7760f0fd578f86a44c9df4b

Branch pushed to git repo; I updated commit sha1. New commits:

dbf2df2Install more dependencies manually

comment:231 in reply to:  228 Changed 23 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

  1. Is there a make target similar to sagelib-build-deps that doesn't install pythonic packages such as cython.

Not yet. #30896 will go into this direction

  1. Why doesn't sagelib-build-deps install maxima?

maxima is only needed at runtime of Sage, not while installing sagelib.

comment:232 Changed 23 months ago by git

Commit: dbf2df2f53f7b1b7e7760f0fd578f86a44c9df4bb66df86359a2ee8be8ab481d0aea20b8960d0ef9

Branch pushed to git repo; I updated commit sha1. New commits:

b66df86Run tests in parallel

comment:233 Changed 23 months ago by git

Commit: b66df86359a2ee8be8ab481d0aea20b8960d0ef91fdfe680752f2878c76234733dc34780e3107715

Branch pushed to git repo; I updated commit sha1. New commits:

5458776Disable timeout in tests
1fdfe68Install fpylll

comment:234 Changed 23 months ago by git

Commit: 1fdfe680752f2878c76234733dc34780e3107715f0bd45cf7ff1ac240bfd10af8844df187c1f97f4

Branch pushed to git repo; I updated commit sha1. New commits:

f0bd45cInstall singular-data to find gftables?

comment:235 Changed 23 months ago by git

Commit: f0bd45cf7ff1ac240bfd10af8844df187c1f97f48a401efbb30b1049b05af3b4e756087d64968505

Branch pushed to git repo; I updated commit sha1. New commits:

0b18fd031029: upgrade cypari2 to 2.1.2
b9aadfd31029: adapt sage library code to cypari2 version 2.1.2
8a401efMerge branch 'u/vdelecroix/31029' of git://trac.sagemath.org/sage into public/build/inplace

comment:236 Changed 23 months ago by git

Commit: 8a401efbb30b1049b05af3b4e756087d6496850522e8fe7c5ab241996824216396b9025997c939be

Branch pushed to git repo; I updated commit sha1. New commits:

f6c3285Workflow cleanup
22e8fe7Set high timeout

comment:237 Changed 23 months ago by Tobias Diez

Dependencies: #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216#30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029

Thanks for the help! I've now improved the github workflow and the doctest pass now (except a handful). So I would appreciate if this ticket could be review!

https://github.com/sagemath/sagetrac-mirror/actions?query=workflow%3A%22Build+%26+Test+using+pipenv%22

Last edited 23 months ago by Tobias Diez (previous) (diff)

comment:238 Changed 23 months ago by git

Commit: 22e8fe7c5ab241996824216396b9025997c939be3d071819a98f2d8f8e50935223d38042e04923e1

Branch pushed to git repo; I updated commit sha1. New commits:

3d07181Run tests in serial again

comment:239 Changed 23 months ago by git

Commit: 3d071819a98f2d8f8e50935223d38042e04923e1cd38e05d1b88d4879bd65aa32eb1614356569802

Branch pushed to git repo; I updated commit sha1. New commits:

cd38e05Don't set SAGE_LOCAL in workflow

comment:240 Changed 23 months ago by git

Commit: cd38e05d1b88d4879bd65aa32eb1614356569802e42e1249042e31381f40889cef835e5947add799

Branch pushed to git repo; I updated commit sha1. New commits:

e42e124Update deps

comment:241 Changed 23 months ago by git

Commit: e42e1249042e31381f40889cef835e5947add799d1079de570eb56f0031153286ed3ff79e273986c

Branch pushed to git repo; I updated commit sha1. New commits:

2d84b5atox.ini: Add missing esac
3fd16f4Merge branch 'u/mkoeppe/tox_ini__local___add_environment_variables_to_skip_system_package_installs__mechanism_for_a_local_interactive_shell' of git://trac.sagemath.org/sage into public/build/inplace
d1079deMerge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace

comment:242 Changed 23 months ago by Matthias Köppe

Instead of introducing a new environment variable SAGE_NUM_BUILD_JOBS, probably best to use SAGE_NUM_THREADS, which is used already in the same source tree.

comment:243 in reply to:  242 Changed 22 months ago by Tobias Diez

Replying to mkoeppe:

Instead of introducing a new environment variable SAGE_NUM_BUILD_JOBS, probably best to use SAGE_NUM_THREADS, which is used already in the same source tree.

Ok, done!

comment:244 Changed 22 months ago by git

Commit: d1079de570eb56f0031153286ed3ff79e273986cd798925a1bb48a96087c3db03950e90059c9c37b

Branch pushed to git repo; I updated commit sha1. New commits:

54ec549Reuse SAGE_NUM_THREADS
d798925Update dependencies

comment:245 Changed 22 months ago by git

Commit: d798925a1bb48a96087c3db03950e90059c9c37bcd8a8d19ea80b99357c465b4e1ff3f285a8b26b5

Branch pushed to git repo; I updated commit sha1. New commits:

cd8a8d1Move pgkconfig to normal package

comment:246 Changed 22 months ago by git

Commit: cd8a8d19ea80b99357c465b4e1ff3f285a8b26b563c7997e078bb7b3a1ab39ef1a2c29d210376f71

Branch pushed to git repo; I updated commit sha1. New commits:

63c7997Make cython and sage_conf also normal packages

comment:247 Changed 22 months ago by git

Commit: 63c7997e078bb7b3a1ab39ef1a2c29d210376f71a18b3c59eff2940159fdb930eea4af04b444e828

Branch pushed to git repo; I updated commit sha1. New commits:

87e2992Move ipywidgets to normal package
a18b3c5Install dev packages as well on ci

comment:248 Changed 22 months ago by git

Commit: a18b3c59eff2940159fdb930eea4af04b444e828f88cfc6a6382cf5cf35a602f83b7bedc06fc67eb

Branch pushed to git repo; I updated commit sha1. New commits:

f88cfc6Set SAGE_LOCAL in workflow for gap

comment:249 Changed 22 months ago by git

Commit: f88cfc6a6382cf5cf35a602f83b7bedc06fc67ebd42fbfc8fe52e06a7c6a54cf6c0a7ea94c6e3a29

Branch pushed to git repo; I updated commit sha1. New commits:

d42fbfcSee if error comes from memory problems

comment:250 Changed 22 months ago by git

Commit: d42fbfc8fe52e06a7c6a54cf6c0a7ea94c6e3a298580789c62f04a49ea893c53bf28d20a80a1aa73

Branch pushed to git repo; I updated commit sha1. New commits:

8580789Fix indent

comment:251 Changed 22 months ago by git

Commit: 8580789c62f04a49ea893c53bf28d20a80a1aa731145360fdc51ee246c27715ce298c6005fd4b281

Branch pushed to git repo; I updated commit sha1. New commits:

1145360Fix paths to memory usage

comment:252 Changed 22 months ago by git

Commit: 1145360fdc51ee246c27715ce298c6005fd4b2814bdfef1b21da9f376cfd3c096cb9932fe5334e4e

Branch pushed to git repo; I updated commit sha1. New commits:

4bdfef1Try with sudo

comment:253 Changed 22 months ago by Tobias Diez

Description: modified (diff)

comment:254 Changed 22 months ago by Matthias Köppe

I think it should be done on top of #30913

comment:255 Changed 22 months ago by Tobias Diez

Since the pipfile in this ticket is more uptodate and is welltested (manually and through the added github action), I would actually argue that #30913 should be on top of this ticket.

Moreover, #30770 is the only open dependency of this ticket, so I guess its also more realistic to get this in first.

comment:256 Changed 22 months ago by Matthias Köppe

This approach would also work. But I am skeptical about checking in the Pipfile.lock. Stuff for nightmare merge conflicts for everyone. I think it would be better if this was an ignored file.

Last edited 22 months ago by Matthias Köppe (previous) (diff)

comment:257 Changed 22 months ago by Matthias Köppe

Cc: Isuru Fernando added

comment:258 Changed 22 months ago by Matthias Köppe

When the next beta comes out (hopefully soon), let's rebase it on top of it (and #30770). The commit history is a bit too wild

comment:259 Changed 22 months ago by Matthias Köppe

Priority: majorcritical

Also, could you please:

  • clean up the ticket description
  • move documentation items from there to the manuals (installation and/or developer's manual)
  • move the remaining issues to new or existing tickets

comment:260 in reply to:  256 ; Changed 22 months ago by Tobias Diez

Replying to mkoeppe:

But I am skeptical about checking in the Pipfile.lock. Stuff for nightmare merge conflicts for everyone. I think it would be better if this was an ignored file.

Without the lock file you wouldn't get a reproducible virtual environment, since pipenv would always use the latest versions (that are compatible with each other). Moreover, the lock file is normally not changed, except if you call pipenv update. The workflow to update Python packages would then look as follows:

  • Run pipenv update
  • Test
  • If everything is fine, create ticket updating pipfile.lock and create a ticket for each updated dependency that is also listed in sage's build/pgks folder.

Will have a look at the other todo's you mention.

comment:261 Changed 22 months ago by Tobias Diez

Description: modified (diff)

comment:262 Changed 22 months ago by Tobias Diez

Cleaned up the ticket description and added a bit of documentation. Since this is still somewhat experimental, I've only put it in the coding basics of the development doc. I guess with the recent changes and support of virtual environments, it might soon be time to update the recommended way of setting up a development platform.

comment:263 Changed 22 months ago by git

Commit: 4bdfef1b21da9f376cfd3c096cb9932fe5334e4e543b6d4e55879dee44f0e3d482b1ca1ebeda1d04

Branch pushed to git repo; I updated commit sha1. New commits:

543b6d4Add basic documentation

comment:264 in reply to:  260 Changed 22 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

The workflow to update Python packages would then look as follows [...]

I would say it's not a good idea to try to change the developer workflow regarding package upgrades as part of this ticket.

comment:265 Changed 22 months ago by Tobias Diez

I agree, that was more meant to illustrate possible applications of the new pipenv environment (and the lockfile for that purpose).

comment:266 Changed 22 months ago by Matthias Köppe

OK. I have created #31346 as a place for this.

comment:267 Changed 22 months ago by Matthias Köppe

Regarding the documentation: "After following the normal procedure of setting up your local development environment" -- I think we should be explicit about requiring the full Sage distribution to be built first. We cannot support experimental workflows as you (still) describe them in the ticket description.

comment:268 Changed 22 months ago by git

Commit: 543b6d4e55879dee44f0e3d482b1ca1ebeda1d04510a3bb5344b544e70b12cd53dd7c48e19701963

Branch pushed to git repo; I updated commit sha1. New commits:

510a3bbClarify point in the docs

comment:269 Changed 22 months ago by Tobias Diez

Sure!

comment:270 Changed 22 months ago by git

Commit: 510a3bb5344b544e70b12cd53dd7c48e19701963c21cc76386b75dbf07f01ae822c120ef9f29c9e9

Branch pushed to git repo; I updated commit sha1. New commits:

c21cc76Update deps

comment:271 Changed 22 months ago by Matthias Köppe

Description: modified (diff)

I've moved the part of the ticket description describing the experimental workflow on bare ubuntu to #31347.

comment:272 Changed 22 months ago by Matthias Köppe

9.3.beta7 is out, time for rebasing

comment:273 Changed 22 months ago by git

Commit: c21cc76386b75dbf07f01ae822c120ef9f29c9e9cf068dfaf42a32f6bbd1ebd55376006e56fb6d8a

Branch pushed to git repo; I updated commit sha1. New commits:

cf068dfMerge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace

comment:274 Changed 22 months ago by Tobias Diez

Nice, thanks for letting me know. I've merged the latest develop branch. The history of this branch still needs to be cleanup up, but let's do this after positive review.

comment:275 Changed 22 months ago by Matthias Köppe

Could you:

  • remove the change to tox.ini please?
  • Rename sage_build_ext and move it to a new module of sage_setup.command (and document what is different to the existing sage_build_ext)

comment:276 Changed 22 months ago by Matthias Köppe

In the new CI workflows:

  • For mac: I do not understand the line: brew install pari
  • For mac: Instead of setting PKG_CONFIG_PATH manually, perhaps use .homebrew-build-env because this is what configure suggests users to use
  • For linux: Why make maxima conway_polynomials jmol conway_polynomials elliptic_curves combinatorial_designs polytopes_db graphs threejs (it is not done in the mac workflow). Should we create a make target that corresponds to what is needed?
  • For linux: Could you elaborate on "# Set SAGE_LOCAL (required for gap)"

Could you review and add comments to the file please?

comment:277 Changed 22 months ago by Matthias Köppe

Also, as #30589 has been merged, probably best to switch from python 3.8 to 3.9 here too

comment:278 Changed 22 months ago by Matthias Köppe

Also #30912 will bring merge conflicts that are best to resolve now

comment:279 Changed 22 months ago by Matthias Köppe

Another conflict from the current version of #30770.

comment:280 Changed 22 months ago by Matthias Köppe

Also create_extension duplicates some of the code in sage_build_cython.create_extension. In particular, it should not use numpy.get_include() directly but rather use sage.env.sage_include_directories (note #31333 for a related change).

comment:281 Changed 22 months ago by Matthias Köppe

I have tested this ticket (on top of #31335, which fixes macOS-specific problems, and resolving the mentioned merge conflicts); following the instructions from the added documentation, pipenv install leads to ERROR: Couldn't install package: cypari2.

The first error leading to this is:

  [1/7] Cythonizing cypari2/closure.pyx

  Error compiling Cython file:
  ------------------------------------------------------------
  ...
  #                  http://www.gnu.org/licenses/
  #*****************************************************************************

  from __future__ import absolute_import, division, print_function

  from cysignals.signals cimport sig_on, sig_off, sig_block, sig_unblock, sig_error
  ^
  ------------------------------------------------------------

  cypari2/closure.pyx:36:0: 'cysignals/signals.pxd' not found

comment:282 Changed 22 months ago by Matthias Köppe

The specific issue in cypari2 is probably https://github.com/sagemath/cypari2/issues/93 (tracked in #30922).

But overall I think it is unrealistic for these instructions to work. Let's use more restrictive instructions, in which pipenv is used from within sage --sh. Then one can at least assume that all libraries are available.

comment:283 Changed 22 months ago by Matthias Köppe

Status: needs_reviewneeds_work

comment:284 Changed 22 months ago by Matthias Köppe

(Something like ../sage -sh -c 'PIP_FIND_LINKS=file://$(cd .. && pwd)/local/var/lib/sage/wheels pipenv install -v -v' similar to what I am testing in #30913.)

comment:285 in reply to:  276 ; Changed 22 months ago by Tobias Diez

Thanks for the review! Will address most of your points later.

Replying to mkoeppe:

  • For linux: Could you elaborate on "# Set SAGE_LOCAL (required for gap)"

Without this line one gets the following error:

Traceback (most recent call last):
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/interfaces/expect.py", line 1468, in __init__
        self._name = parent._create(value, name=name)
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/interfaces/interface.py", line 500, in _create
        self.set(name, value)
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/interfaces/gap.py", line 1421, in set
        self._eval_line(cmd, allow_use_file=True)
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/interfaces/gap.py", line 790, in _eval_line
        self._start()
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/interfaces/gap.py", line 1254, in _start
        gap_reset_workspace(verbose=False)
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/interfaces/gap.py", line 1587, in gap_reset_workspace
        g.eval('SetUserPreference("HistoryMaxLines", 30)')
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/interfaces/gap.py", line 587, in eval
        result = Expect.eval(self, input_line, **kwds)
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/interfaces/expect.py", line 1381, in eval
        return '\n'.join([self._eval_line(L, allow_use_file=allow_use_file, **kwds)
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/interfaces/expect.py", line 1381, in <listcomp>
        return '\n'.join([self._eval_line(L, allow_use_file=allow_use_file, **kwds)
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/interfaces/gap.py", line 790, in _eval_line
        self._start()
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/interfaces/gap.py", line 1259, in _start
        Expect._start(self, "Failed to start GAP.")
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/interfaces/expect.py", line 520, in _start
        raise RuntimeError("unable to start %s: %s" % (self.name(), msg))
    RuntimeError: unable to start gap: End Of File (EOF). Exception style platform.
    Gap finished running /__w/sagetrac-mirror/sagetrac-mirror/.tox/local-sudo-standard/local/bin/gap -r -b -p -T -E -o 401m -s 401m -m 64m /__w/sagetrac-mirror/sagetrac-mirror/src/sage/ext_data/gap/sage.g
    command: /__w/sagetrac-mirror/sagetrac-mirror/.tox/local-sudo-standard/local/bin/gap
    args: ['/__w/sagetrac-mirror/sagetrac-mirror/.tox/local-sudo-standard/local/bin/gap', '-r', '-b', '-p', '-T', '-E', '-o', '401m', '-s', '401m', '-m', '64m', '/__w/sagetrac-mirror/sagetrac-mirror/src/sage/ext_data/gap/sage.g']
    buffer (last 100 chars): b''
    before (last 100 chars): b'Set the environment variable SAGE_LOCAL.\r\n'
    after: <class 'pexpect.exceptions.EOF'>
    match: None
    match_index: None
    exitstatus: 1
    flag_eof: True
    pid: 3124
    child_fd: 26
    closed: False
    timeout: None
    delimiter: <class 'pexpect.exceptions.EOF'>
    logfile: None
    logfile_read: None
    logfile_send: None
    maxread: 4194304
    ignorecase: False
    searchwindowsize: None
    delaybeforesend: None
    delayafterclose: 0.1
    delayafterterminate: 0.1
    searcher: searcher_re:
        0: re.compile(b'gap> ')

I guess the current github workflows don't see this as everything is invoked in one tox command and not in multiple steps (and enviroment variables are only active in each step).

comment:286 Changed 22 months ago by Tobias Diez

Concerning the cypari error, I have no clue what is needed for it to compile on mac. It's also strange that it works on the github action (for mac as well as linux) but not on your system. cysignals is actually declared in the pipenv file, so it should be installed. The simplest solution would be if cypari would publish wheels on pypi.

But overall I think it is unrealistic for these instructions to work.

It's not unrealistic, it is working very well for Linux.

Last edited 22 months ago by Tobias Diez (previous) (diff)

comment:287 in reply to:  276 ; Changed 22 months ago by Tobias Diez

Replying to mkoeppe:

  • For linux: Why make maxima conway_polynomials jmol conway_polynomials elliptic_curves combinatorial_designs polytopes_db graphs threejs (it is not done in the mac workflow). Should we create a make target that corresponds to what is needed?

That's the only way I found to install these packages into SAGE_SHARE. Will add it also to the mac workflow. I cannot judge if it would make sense to add a special make target for it, especially since you are working on making these packages normal python packages.

comment:288 in reply to:  285 Changed 22 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

Replying to mkoeppe:

  • For linux: Could you elaborate on "# Set SAGE_LOCAL (required for gap)"

Without this line one gets the following error:

...
        Expect._start(self, "Failed to start GAP.")
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/interfaces/expect.py", line 520, in _start
...

I guess the current github workflows don't see this as everything is invoked in one tox command and not in multiple steps (and enviroment variables are only active in each step).

OK, I see.

There are two ways to fix this:

  • as described in #30818 ("Set environment for subprocesses invoked by Sage"). Currently subprocesses (such as the gap process invoked by the pexpect-based gap interface) rely on sage-env being sourced
  • using pipenv run sage -t instead of pipenv run sage-runtests, which uses sage-env.

comment:289 Changed 22 months ago by Tobias Diez

I've added it to #29644 (which is part of #30818).

comment:290 in reply to:  287 ; Changed 22 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

Replying to mkoeppe:

  • For linux: Why make maxima conway_polynomials jmol conway_polynomials elliptic_curves combinatorial_designs polytopes_db graphs threejs (it is not done in the mac workflow). Should we create a make target that corresponds to what is needed?

That's the only way I found to install these packages into SAGE_SHARE. Will add it also to the mac workflow. I cannot judge if it would make sense to add a special make target for it, especially since you are working on making these packages normal python packages.

I've opened #31362 for this - which will add make all-sage-local.

comment:291 Changed 22 months ago by Tobias Diez

Thanks! Concerning the above cysignals problem, you can try to run pipenv run pip install cysignals and then pipenv install again. Similarly, pipenv run pip install gmpy2==2.1.0b5 might be necessary if you have problems with the installation of pplpy. Let me know if this helps and I will add it to the documentation.

comment:292 Changed 22 months ago by git

Commit: cf068dfaf42a32f6bbd1ebd55376006e56fb6d8a398e6dd59ac7b8e3c49dc7917295e4fe4a9bcaba

Branch pushed to git repo; I updated commit sha1. New commits:

1f1dddcRevert change to tox.ini
0c3f517Move sage_build_ext
d06c84dRun workflow for Python 3.9 as well
7fee372Doc: Also install dev packages
a506c1fUse sage_build command for generating sources
398e6ddNarrow down imports to reduce install requires deps

comment:293 Changed 22 months ago by git

Commit: 398e6dd59ac7b8e3c49dc7917295e4fe4a9bcaba12c0a2ed96e5608f8f6c54aef0d81e8d76cdcff5

Branch pushed to git repo; I updated commit sha1. New commits:

12c0a2eMac: remove pari install

comment:294 Changed 22 months ago by git

Commit: 12c0a2ed96e5608f8f6c54aef0d81e8d76cdcff50157a4eac42646f49b9448afd120d07c4848750b

Branch pushed to git repo; I updated commit sha1. New commits:

1041128Rename ecl config variable
fd05a97Add fallback
2ac4cc8Merge branch 'develop' of git://github.com/sagemath/sage into public/build/ecl-config
6370c34src/sage/env.py: Do not use capture_output, which requires python 3.7
31f1bbbAlso use universal_newlines instead of text, for python 3.6 compatibility
11ad40aMerge branch 'public/build/ecl-config' of git://trac.sagemath.org/sage into public/build/ecl-config
1a8e2e4Merge branch 'public/build/ecl-config' of git://trac.sagemath.org/sage into public/build/inplace
0157a4eReuse sage_include_directories

comment:295 Changed 22 months ago by git

Commit: 0157a4eac42646f49b9448afd120d07c4848750ba3806c58709c3b5c828668f823b37619bf1d1551

Branch pushed to git repo; I updated commit sha1. New commits:

18d8557Pretty print job names
a3806c5Update Macos workflow

comment:296 Changed 22 months ago by git

Commit: a3806c58709c3b5c828668f823b37619bf1d1551f5551673591c836942a4b7d50f971ad7c910763e

Branch pushed to git repo; I updated commit sha1. New commits:

f555167Fix workflow syntax

comment:297 Changed 22 months ago by git

Commit: f5551673591c836942a4b7d50f971ad7c910763ebb90f76abc82987e7ff1cbfd2653dfa908fefc7b

Branch pushed to git repo; I updated commit sha1. New commits:

bb90f76Well...maybe now

comment:298 Changed 22 months ago by git

Commit: bb90f76abc82987e7ff1cbfd2653dfa908fefc7bd1c2a0a00f8c675ad218b502516dca12d161933d

Branch pushed to git repo; I updated commit sha1. New commits:

d1c2a0aTry with latest version of pip

comment:299 Changed 22 months ago by git

Commit: d1c2a0a00f8c675ad218b502516dca12d161933df8599611360fcf04a54d28830c266183cd99ff99

Branch pushed to git repo; I updated commit sha1. New commits:

f859961Run everything versus the correct python

comment:300 Changed 22 months ago by git

Commit: f8599611360fcf04a54d28830c266183cd99ff99606966b86e35ae679edc08dd1c5fa156cff667e8

Branch pushed to git repo; I updated commit sha1. New commits:

606966bTry to trigger macos workflow

comment:301 Changed 22 months ago by git

Commit: 606966b86e35ae679edc08dd1c5fa156cff667e8541f7143df6f00116a3048fd34242e90a50ac40e

Branch pushed to git repo; I updated commit sha1. New commits:

541f714Remove empty line?

comment:302 Changed 22 months ago by git

Commit: 541f7143df6f00116a3048fd34242e90a50ac40e42d25a14c00a68e58bcb904783483a5514eecb5f

Branch pushed to git repo; I updated commit sha1. New commits:

42d25a1Still not?

comment:303 Changed 22 months ago by git

Commit: 42d25a14c00a68e58bcb904783483a5514eecb5ff8599611360fcf04a54d28830c266183cd99ff99

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

comment:304 Changed 22 months ago by git

Commit: f8599611360fcf04a54d28830c266183cd99ff99f312a35f92ef30fa4a38ab54560686e2ef1bd5d6

Branch pushed to git repo; I updated commit sha1. New commits:

f312a35Try to remove python3-pip again

comment:305 Changed 22 months ago by git

Commit: f312a35f92ef30fa4a38ab54560686e2ef1bd5d6b01c4c98a6c590ec7526d2f87dba7a839ebee9df

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

b01c4c9Install tox via pip

comment:306 Changed 22 months ago by git

Commit: b01c4c98a6c590ec7526d2f87dba7a839ebee9dfa1d989c3efd08d5f64612865f428102b328744da

Branch pushed to git repo; I updated commit sha1. New commits:

a1d989cInstall pip as wel of course

comment:307 Changed 22 months ago by git

Commit: a1d989c3efd08d5f64612865f428102b328744da442596137f517ff3cb3e365e1b8f79b41744aec2

Branch pushed to git repo; I updated commit sha1. New commits:

4425961Strange...

comment:308 Changed 22 months ago by git

Commit: 442596137f517ff3cb3e365e1b8f79b41744aec2f2e436c459bad755685c6ca42a3d401b1fd29ed9

Branch pushed to git repo; I updated commit sha1. New commits:

f2e436cFix path of tox venv under mac

comment:309 in reply to:  290 Changed 22 months ago by Matthias Köppe

Replying to mkoeppe:

Replying to gh-tobiasdiez:

Replying to mkoeppe:

  • For linux: Why make maxima conway_polynomials jmol conway_polynomials elliptic_curves combinatorial_designs polytopes_db graphs threejs (it is not done in the mac workflow). Should we create a make target that corresponds to what is needed?

That's the only way I found to install these packages into SAGE_SHARE. Will add it also to the mac workflow. I cannot judge if it would make sense to add a special make target for it, especially since you are working on making these packages normal python packages.

I've opened #31362 for this - which will add make all-sage-local.

OK, the top-level target is now make build-local.

comment:310 in reply to:  289 Changed 22 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

I've added it to #29644 (which is part of #30818).

Thanks. Do you plan to work on it? Otherwise a quick fix is to use pipenv run sage -t instead of pipenv run sage-runtests as I wrote above.

Last edited 22 months ago by Matthias Köppe (previous) (diff)

comment:311 in reply to:  286 Changed 22 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

Concerning the cypari error, I have no clue what is needed for it to compile on mac.

Well it certainly needs the PARI library! When running pipenv outside of the environment set using sage --sh, it depends on everything already installed in the system.

That's why I am saying the instructions to the users should run pipenv within that shell.

comment:312 Changed 22 months ago by git

Commit: f2e436c459bad755685c6ca42a3d401b1fd29ed9a6e57808dcabbb60d9c7fdefb88ed7ca4be7caff

Branch pushed to git repo; I updated commit sha1. New commits:

b86b740Don't install dev packages
a6e5780Install fplll on mac

comment:313 Changed 22 months ago by git

Commit: a6e57808dcabbb60d9c7fdefb88ed7ca4be7caff85de00c2d4103d0edbcd2a41a2efcee52ea5c26e

Branch pushed to git repo; I updated commit sha1. New commits:

85de00cAlso install libffi8ubuntu1

comment:314 Changed 22 months ago by git

Commit: 85de00c2d4103d0edbcd2a41a2efcee52ea5c26e1760cf741ff4d392e7662075637350fe8e7894bb

Branch pushed to git repo; I updated commit sha1. New commits:

1760cf7Or try to install libffi7

comment:315 Changed 22 months ago by git

Commit: 1760cf741ff4d392e7662075637350fe8e7894bbefea01f09fb253083afc5c04ffa5824bdd4d731c

Branch pushed to git repo; I updated commit sha1. New commits:

efea01fInstall wget as well

comment:316 Changed 22 months ago by git

Commit: efea01f09fb253083afc5c04ffa5824bdd4d731c5a2cc305cf50e561d400a42d3e568cee7ea082a1

Branch pushed to git repo; I updated commit sha1. New commits:

ec5629cClarify that we need all system packgaes
5a2cc30Try with libffi6

comment:317 Changed 22 months ago by git

Commit: 5a2cc305cf50e561d400a42d3e568cee7ea082a12ae43976cfacb331ae82426f25528213987225dd

Branch pushed to git repo; I updated commit sha1. New commits:

2ae4397Mac: install pari as well

comment:318 Changed 22 months ago by Tobias Diez

Is it actually desired that tox local-homebrew installs a fresh homebrew in the tox folder (if I'm not misreading the logs). Is there a way to use the system's homebrew?

comment:319 Changed 22 months ago by git

Commit: 2ae43976cfacb331ae82426f25528213987225dd88dabfa498cc3e12116d6f5f26e945ce4abd1429

Branch pushed to git repo; I updated commit sha1. New commits:

b9b1ef1Use homebrew-build-env
88dabfaCleanup setup.py

comment:320 Changed 22 months ago by git

Commit: 88dabfa498cc3e12116d6f5f26e945ce4abd14298d5e37b6ab241cb2f98462de9b536937724ed977

Branch pushed to git repo; I updated commit sha1. New commits:

b1fd3e9Run macos without local prefix
7d660e0Tox: Fix homebrew env variable
c3457e5Use local-homebrew-macos-usrlocal
d3d72cbRevert changes to tox.ini
8d5e37bFix compile errors

comment:321 Changed 22 months ago by git

Commit: 8d5e37b6ab241cb2f98462de9b536937724ed977d9ac8df40f4eb4702795e544649a9e1830f25874

Branch pushed to git repo; I updated commit sha1. New commits:

d9ac8dfRevert changes to sage.env import

comment:322 in reply to:  318 Changed 22 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

Is it actually desired that tox local-homebrew installs a fresh homebrew in the tox folder

Yes, that's by design

Is there a way to use the system's homebrew?

From tox.ini comments:

    # The variant "local-homebrew-macos-usrlocal" uses the global installation in /usr/local
    # instead.  It may install packages or update packages.  It will not remove packages.
    # Use at your own risk.

comment:323 Changed 22 months ago by Tobias Diez

Thanks! That is working nicely.

I think I've now addressed all your comments and suggestions.

The only thing is that since merging the latest develop branch the sage packages (installed via tox) are no longer found when running pipenv install. At the commit https://github.com/sagemath/sagetrac-mirror/commit/4bdfef1b21da9f376cfd3c096cb9932fe5334e4e this was working, but stopped after I've merged the develop branch. Now packages such as linbox, singular and gap are still compiled by sage's make, but pkgconfig (in sage.env) cannot find them any longer and thus the c++ compilation stopped working. I don't have these problems locally, where I don't use tox. Do you have an idea which recent changes are the origin of this issue?

I've also tried your suggestion with sage -sh pipenv but run into several issues. I guess the problem is that pipenv is trying to reuse sages virtual environment.

comment:324 Changed 22 months ago by git

Commit: d9ac8df40f4eb4702795e544649a9e1830f25874245d40fe531570a2bd3cc71e13d851a25e1ff858

Branch pushed to git repo; I updated commit sha1. New commits:

245d40fUse correct version of fpylll for mac

comment:325 Changed 22 months ago by git

Commit: 245d40fe531570a2bd3cc71e13d851a25e1ff8584b3c76f9fbe11c202c8de6aa7fe661bcfc7aeca7

Branch pushed to git repo; I updated commit sha1. New commits:

4b3c76fTry to fix PKG_CONFIG_PATH and LD_LIBRARY_PATH

comment:326 Changed 22 months ago by Matthias Köppe

...never touch LD_LIBRARY_PATH... it's always wrong

comment:327 Changed 22 months ago by Matthias Köppe

Could you please take a look at #31362 - it provides make build-local, which I think is what you need

comment:328 Changed 22 months ago by Matthias Köppe

Thanks for making all the changes.

Right now it's hard for me to test anything from this ticket because the homebrew changes in #31335 conflict.

I would propose to do the following:

  • Rebasing/squashing this branch on top of #31362, current #30770, and #30912.
  • Merging the rebase back into the current branch -- by this trick you won't get merge conflicts in your other branches that depend on this ticket (if any)

comment:329 in reply to:  323 Changed 22 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

I've also tried your suggestion with sage -sh pipenv but run into several issues. I guess the problem is that pipenv is trying to reuse sages virtual environment.

See if you can work around this by having pipenv use the base python that is shown in $SAGE_LOCAL/pyvenv.cfg.

comment:330 Changed 22 months ago by git

Commit: 4b3c76f9fbe11c202c8de6aa7fe661bcfc7aeca77c89641c4f2eed5e7cac24776e5079c346d575de

Branch pushed to git repo; I updated commit sha1. New commits:

7c89641Update lock file

comment:331 in reply to:  326 ; Changed 22 months ago by Tobias Diez

Replying to mkoeppe:

...never touch LD_LIBRARY_PATH... it's always wrong

Ok, but what's then the correct way to do this?

comment:332 Changed 22 months ago by Tobias Diez

Dependencies: #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029#30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029, #30912

comment:333 in reply to:  331 Changed 22 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

Replying to mkoeppe:

...never touch LD_LIBRARY_PATH... it's always wrong

Ok, but what's then the correct way to do this?

Building all libraries with rpath - see LDFLAGS setting in src/bin/sage-env

comment:334 Changed 22 months ago by git

Commit: 7c89641c4f2eed5e7cac24776e5079c346d575deb68ce04b04b2d2104c6643700fa0f8cbe81f9f74

Branch pushed to git repo; I updated commit sha1. New commits:

2c40179build/pkgs/sagelib/src: Update metadata
2a91862build/pkgs/sagelib/src/README.rst: Add some text
6432727Merge tag '9.3.beta2' into t/30912/sagelib__update_metadata_for_pypi_deployment
4a693f2Move build/pkgs/sagelib/src/setup.cfg to SAGE_ROOT/src, replace by symlink
ea182d7Copy changes from build/pkgs/sagelib/src to src
a1a10b9src/VERSION.txt: New
5697335src/setup.cfg: Add license_file=LICENSE.txt
deb9eb3Merge tag '9.3.beta3' into t/30912/sagelib__update_metadata_for_pypi_deployment
7ad4c0eMerge tag '9.3.beta4' into t/30912/sagelib__update_metadata_for_pypi_deployment
b68ce04Merge branch 'u/mkoeppe/sagelib__update_metadata_for_pypi_deployment' of git://trac.sagemath.org/sage into public/build/inplace

comment:335 in reply to:  328 ; Changed 22 months ago by Tobias Diez

Replying to mkoeppe:

I've merged the current versions of #30770 and #30912. Not sure what's the purpose of #31362. That would make the line make maxima conway_polynomials jmol conway_polynomials elliptic_curves combinatorial_designs polytopes_db graphs threejs obsolete right? If it's just for this somewhat cosmetic improvement, I would suggest to turn around the tables and let #31362 depend on this ticket (and make the changes in the workflow there as well). Reason: I had to play a lot around with the different targets until everything works, so I'm a bit hesitant to change things again (and since unmerging a branch is not so easy, one cannot simply revert the experiment).

comment:336 in reply to:  335 ; Changed 22 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

I've merged the current versions of #30770 and #30912.

Thanks.

Not sure what's the purpose of #31362. That would make the line make maxima conway_polynomials jmol conway_polynomials elliptic_curves combinatorial_designs polytopes_db graphs threejs obsolete right? If it's just for this somewhat cosmetic improvement

Yes, it's for making all standard non-Python packages available in SAGE_LOCAL.

Reason: I had to play a lot around with the different targets until everything works, so I'm a bit hesitant to change things again

This does not sound good. If installing additional packages from the distribution in SAGE_LOCAL breaks the proposed setup, then this ticket is definitely not ready.

comment:337 Changed 22 months ago by Matthias Köppe

Note also that #31362 does not change anything other than add two new make targets. So even if you find that these targets do not do what you need, there will be no need for unmerging it.

comment:338 in reply to:  336 Changed 22 months ago by Tobias Diez

Replying to mkoeppe:

Reason: I had to play a lot around with the different targets until everything works, so I'm a bit hesitant to change things again

This does not sound good. If installing additional packages from the distribution in SAGE_LOCAL breaks the proposed setup, then this ticket is definitely not ready.

More in the sense, that it was a bit of try-and-error to find the minimal set of make targets that make the build pass. You could have course build every package in the sage distribution, but the point is that with pipenv only a couple of them are actually needed.

So build-local = sagelib-build-deps + maxima conway_polynomials jmol conway_polynomials elliptic_curves combinatorial_designs polytopes_db graphs threejs ?

comment:339 Changed 22 months ago by Matthias Köppe

... minus the Python packages of sagelib-build-deps

comment:340 Changed 22 months ago by Matthias Köppe

For example, it would not build numpy.

comment:341 Changed 22 months ago by Matthias Köppe

Of course, there may be a number of non-Python libraries that are not needed IF you are able to pull in a wheel of the Python packages that somehow ship a bundled version of the non-Python library. For example, the numpy wheels may come with their own BLAS -- but it won't help because some of our own packages also need BLAS.

comment:342 Changed 22 months ago by git

Commit: b68ce04b04b2d2104c6643700fa0f8cbe81f9f742154a5b9f8f56ccc83bb45aa9bd10d659b015649

Branch pushed to git repo; I updated commit sha1. New commits:

bdb2237Merge branch 't/30912/sagelib__update_metadata_for_pypi_deployment' into t/31357/fixup_src_version_txt_added_in__30912
c16651csrc/VERSION.txt: Use same format as build/pkgs/sagelib/src/VERSION.txt
2154a5bMerge branch 'u/mkoeppe/fixup_src_version_txt_added_in__30912' of git://trac.sagemath.org/sage into public/build/inplace

comment:343 Changed 22 months ago by Tobias Diez

Dependencies: #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029, #30912#30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029, #30912, #31357

comment:344 Changed 22 months ago by git

Commit: 2154a5b9f8f56ccc83bb45aa9bd10d659b0156492773e6fd9c1868346d194227287dffd6c3642a5e

Branch pushed to git repo; I updated commit sha1. New commits:

55bc2aam4/sage_spkg_collect.m4: Generate install tree information for build/make/Makefile
d7dc9a2build/make/Makefile.in: New targets all-sage-local, all-sage-venv
27b589bMerge branch 't/30912/sagelib__update_metadata_for_pypi_deployment' into t/31362/make_all_sage_local__make_all_sage_venv
8796008Put pynac into SAGE_LOCAL, sagelib into SAGE_VENV
d6831b1build/make/Makefile.in: Add all-build-local, all-build-venv, which include dependency on toolchain
61f6ba6Makefile: Add top-level targets build-local, build-venv
e354672Merge branch 'u/mkoeppe/make_all_sage_local__make_all_sage_venv' of git://trac.sagemath.org/sage into public/build/inplace
2773e6fUse build-local target

comment:345 Changed 22 months ago by Tobias Diez

Dependencies: #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029, #30912, #31357#30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029, #30912, #31357, #31362

comment:346 Changed 22 months ago by Tobias Diez

ok, let's see if that works

comment:347 Changed 22 months ago by Matthias Köppe

In any case thanks to merging these tickets, I am able to merge the tickets that fix homebrew - so I'll be able to test later today.

comment:348 Changed 22 months ago by Matthias Köppe

../sage -sh -c 'ARCHFLAGS= PIP_NO_BUILD_ISOLATION=1 pipenv install -v -v' got me a bit farther. (Setting ARCHFLAGS to the empty string helps on macOS; PIP_NO_BUILD_ISOLATION=1 helps with some of the Python packages that declare their build dependencies incorrectly (see #30922). Now it fails with this:

    raise MetadataInconsistent(self._ireq, "name", dist.project_name)
pip._internal.exceptions.MetadataInconsistent: Requested fpylll from https://files.pythonhosted.org/packages/ad/c5/f30f354dcce03b928702414fdbdd916d60915c6b144360c88707784385bb/fpylll-0.5.4.tar.gz#egg=fpylll-recent (from -r /var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pipenv-7zg7bw1s-requirements/pipenv-b8jmkho7-requirement.txt (line 1)) has different name in metadata: 'fpylll'
Removed build tracker: '/private/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pip-req-tracker-z5q5pu09'
...
    raise exceptions.InstallError(c.dep.name, extra=err_lines)
pipenv.exceptions.InstallError: ERROR: Couldn't install package: fpylll-recent
 Package installation failed...

comment:349 Changed 22 months ago by git

Commit: 2773e6fd9c1868346d194227287dffd6c3642a5ee70f22ba61f2d4c1e0de5a254a15844129df7cab

Branch pushed to git repo; I updated commit sha1. New commits:

627e79dFix cpath as well
e70f22bTry to set name of fpylll-recent

comment:350 Changed 22 months ago by Tobias Diez

This error also occurred on the github workflow. I tried to fix it by setting the name in the pipfile. If this doesn't work can you please remove the fpylll-recent in the pipfile and invoke pipenv install https://files.pythonhosted.org/packages/ad/c5/f30f354dcce03b928702414fdbdd916d60915c6b144360c88707784385bb/fpylll-0.5.4.tar.gz This should work. Could you then please post what pipenv writes to the pipfile, thanks!

build-local seems to work, except that it installs some python related tools/packages (pip, setuptools_scm, wheel) and doesn't install maxima.

comment:351 Changed 22 months ago by git

Commit: e70f22ba61f2d4c1e0de5a254a15844129df7cab8c22f9ec046cb8253531056e0a7aff7118ca778e

Branch pushed to git repo; I updated commit sha1. New commits:

8c22f9eSet LIBRARY_PATH

comment:352 Changed 22 months ago by git

Commit: 8c22f9ec046cb8253531056e0a7aff7118ca778ee17019c5528c867367988d7fa99bcef76f320949

Branch pushed to git repo; I updated commit sha1. New commits:

e17019cInstall old readline as well

comment:353 in reply to:  350 Changed 22 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

build-local seems to work, except that it installs some python related tools/packages (pip, setuptools_scm, wheel) and doesn't install maxima.

wheel (which pulls in the python toolchain) is fixed in #31321.

maxima - could you double check?

comment:354 Changed 22 months ago by Tobias Diez

Ok good. And false alarm, maxima is installed (the output is just condensed to one line in contrast to the pages of text for make maxima which has thrown me off).

The latest commit should hopefully fix the fpylll error.

comment:355 Changed 22 months ago by git

Commit: e17019c5528c867367988d7fa99bcef76f320949e2ed97f2ba3af909838802c95f4151282090c7d7

Branch pushed to git repo; I updated commit sha1. New commits:

e2ed97fFix fpylll

comment:356 Changed 22 months ago by Matthias Köppe

I will put a rebased/squashed version of the setup.py/sage_setup changes on #31377, because this part seems ready and I hope we can get it into 9.3 to get developers used to the in-place install idea.

comment:357 Changed 22 months ago by Matthias Köppe

(It looks to me like the venv aspects of this ticket will take more time; in particular since they seem to involve getting an uncooperative random set of wheels from PyPI to cooperate with the system.)

comment:358 Changed 22 months ago by Tobias Diez

It's quite the opposite: since some of the python packages (fyplll, cypari2) don't provide wheels, system packages need to be installed which is quite hard to coordinate across different systems.

comment:359 Changed 22 months ago by Matthias Köppe

But the Sage distribution has already solved this problem. Just build within a sage-sh.

comment:360 Changed 22 months ago by Matthias Köppe

Please take a look if #31377 looks agreeable to you.

comment:361 in reply to:  280 Changed 22 months ago by Matthias Köppe

Replying to mkoeppe:

Also create_extension duplicates some of the code in sage_build_cython.create_extension.

Also as I just wrote in #31377, some of the code setting flags is missing, which is causing errors. Could you look into unifying this please when you have a chance?

comment:362 in reply to:  357 Changed 22 months ago by Vincent Delecroix

Replying to mkoeppe:

(It looks to me like the venv aspects of this ticket will take more time; in particular since they seem to involve getting an uncooperative random set of wheels from PyPI to cooperate with the system.)

On this topic, see also the discussion at https://github.com/sagemath/cypari2/issues/105.

I think it is a desirable feature to be able to edit the source code in an inplace install of sage (ie what pip -e provides). This would greatly benefit all Sage developers. However, I have hard time to understand the benefit of the venv. On the other hand, I do not understand how wheels could solve the libraries/softwares dependencies. More precisely, here is a scenario (which happens to be happening in Sage)

  • SageMath depends on pylib1 and pylib2 that uses the same libfoo C library. Both of these libraries provide wheels that bundle libfoo to make some of their users happy.
  • SageMath also depends on pylib3 that depends on both pylib1 and pylib2 and uses custom Cython code that uses functions from libfoo. pylib3 does not provide wheel yet. How should it be implemented?

comment:363 Changed 22 months ago by Matthias Köppe

Hi Vincent, thanks for joining the discussion.

For a discussion of possible distribution strategies using wheels, let's use #31251 (Meta-ticket: Distribution as wheels), which also has a link to a recent post by Gommers on packaging. I agree with you that it is a hard problem.

For Sage 9.3, I have a much more modest goal regarding wheels that is already fully implemented and is only waiting for review (see ticket list in #29705 for tickets by milestone). Since #29500 (already in Sage 9.2), we generate wheels in $SAGE_LOCAL/var/lib/sage/wheels. These wheels are not distributable as they do not bundle their shared library dependencies but rather rpath into $SAGE_LOCAL. Users can make their own venvs for development purposes where they install the prebuilt wheels from this directory. #31045 and #31279 add some missing wheels. #30913 generates dependency metadata that makes it easier to set up the venv; and adds tests using tox.

For editable install of sagelib as part of the normal build system (no extra venv) using Tobias' new setup.py from this ticket, see #31377.

Last edited 22 months ago by Matthias Köppe (previous) (diff)

comment:364 Changed 22 months ago by Tobias Diez

Dependencies: #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029, #30912, #31357, #31362#30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029, #30912, #31357, #31362, #31365

comment:365 Changed 22 months ago by git

Commit: e2ed97f2ba3af909838802c95f4151282090c7d73e60c00b3cb16ae500584baf484017b922ac3988

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

804ebd7sage_setup.dpcbuild.AllBuilder: Restrict workaround to macOS
b4ceee5sage_setup.docbuild: In the workaround, do not go through build_many to build serially
7369a3eMerge branch 't/31344/homebrew__docbuild_crashes__libtcl_atforkprepare' into t/31365/add_ntl_to_cython_aliases
ce03e6bsrc/sage/schemes/hyperelliptic_curves/hypellfrob.pyx: Merge distutils directives
32576b4sage.misc.cython: Add NTL aliases, cache result of cython_aliases
6e30e3aUse variables NTL_INCDIR, NTL_LIBDIR in sage_conf, separate from NTL_PREFIX used in sage-build-env-config; set -std=c++11 in NTL_CFLAGS
7b1e27bMerge distutils directives for Cython files using ntl
4d2828dSwitch Cython files that use NTL to language = c++
993c35cbuild/pkgs/ntl/spkg-configure.m4: Fix up
3e60c00Merge branch 'u/mkoeppe/add_ntl_to_cython_aliases' of git://trac.sagemath.org/sage into public/build/inplace

comment:366 Changed 22 months ago by Tobias Diez

Status: needs_workneeds_review

So, I think, I've found a solution to the fpylll problems. In fact, the github workflow should now also work for macos (at least the last run installed everything fine except sage, which failed due to an NTL error that is fixed with #31365). Since macos runs on the sage repo run currently really slow, you can find the latest run at https://github.com/tobiasdiez/sage/actions.

comment:367 Changed 22 months ago by Matthias Köppe

I've split out another ticket from here: #31384

comment:368 Changed 22 months ago by Matthias Köppe

Priority: criticalmajor
Work issues: rebase on dependencies + #31384, #31377

comment:369 Changed 22 months ago by Tobias Diez

Dependencies: #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029, #30912, #31357, #31362, #31365#30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029, #30912, #31357, #31362, #31365, #31377
Work issues: rebase on dependencies + #31384, #31377

comment:370 Changed 22 months ago by git

Commit: 3e60c00b3cb16ae500584baf484017b922ac398813d90ee00e330ff759d0f8fc0b4e3868a477248c

Branch pushed to git repo; I updated commit sha1. New commits:

217c675Try to fix gmp issue on mac
25df925For mac not linux...
90a5104Cleanup workflow
0c4ba62Try to only specify gmp includes
a978458Fix a few more errors
c1ce878Also for the manual install
13d90eeRemove a few unnecessary changes

comment:371 Changed 22 months ago by git

Commit: 13d90ee00e330ff759d0f8fc0b4e3868a477248c08fef7dd62544dfb13327d4ad38fca5ab6691190

Branch pushed to git repo; I updated commit sha1. New commits:

f8b5ef3Mark a few cython files as c++
08fef7dOnly catch import errors

comment:372 Changed 22 months ago by git

Commit: 08fef7dd62544dfb13327d4ad38fca5ab66911908f8b61533287f67be77a4920f7ffceb5ebc39b0d

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

18df2dcMerge branch 't/31365/add_ntl_to_cython_aliases' into t/31377/__configure___enable_editable
19ca17asrc/sage/graphs/graph_decompositions/tdlib/sage_tdlib.cpp: Move one directory level higher
f02963bMerge branch 't/31389/remove_clash_of_tdlib__with_tdlib_pyx_in__src_sage_graphs_graph_decompositions' into t/31377/__configure___enable_editable
2b293f8src/setup.py: Install the jupyter kernel using sage_install
b05e4babuild/make/Makefile.in (sagelib-clean): Remove generated .c and .cpp files
271d427src/setup.py: Fix sage.misc.sageinspect by setting Cython.Compiler.Options.embed_pos_in_docstring
f6b2fc4src/sage/repl/interpreter.py: Fix a doctest for editable installs of sagelib
9a0731asage.interacs.debugger: Remove deprecated module
e5d47ceMerge branch 't/31390/remove_deprecated_sage_interacts_debugger' into t/31377/__configure___enable_editable
8f8b615Merge branch 't/31377/__configure___enable_editable' into t/30371/public/build/inplace

comment:373 Changed 22 months ago by Matthias Köppe

Merged #31377, resolving edit conflicts.

comment:374 Changed 22 months ago by Matthias Köppe

I don't know which problem you are trying to solve with the big try/except. I don't think reducing ImportError is the correct solution.

If you are trying to make setup.py sdist work when libs are missing, consider using the code from build/pkgs/sagelib/src/setup.py line 61

comment:375 in reply to:  373 ; Changed 22 months ago by Tobias Diez

Replying to mkoeppe:

Merged #31377, resolving edit conflicts.

Thanks, but this did revert some of my changes (in particular to setup.py)....

comment:376 in reply to:  374 ; Changed 22 months ago by Tobias Diez

Replying to mkoeppe:

I don't know which problem you are trying to solve with the big try/except. I don't think reducing ImportError is the correct solution.

If you are trying to make setup.py sdist work when libs are missing, consider using the code from build/pkgs/sagelib/src/setup.py line 61

Yes, that was my goal. Creating the virtual environment otherwise fails as Cython is not yet installed into it. The file you refer to doesn't seem to exist.

comment:377 Changed 22 months ago by git

Commit: 8f8b61533287f67be77a4920f7ffceb5ebc39b0dfec55234a00e46d023ad583f8c6284b970141ab0

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

fec5523Merge branch 't/31377/__configure___enable_editable' into t/30371/public/build/inplace

comment:378 in reply to:  375 Changed 22 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

Replying to mkoeppe:

Merged #31377, resolving edit conflicts.

Thanks, but this did revert some of my changes (in particular to setup.py)....

Sorry, my mistake. I've redone the merge now. Please take a look.

comment:379 in reply to:  376 Changed 22 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

Replying to mkoeppe:

I don't know which problem you are trying to solve with the big try/except. I don't think reducing ImportError is the correct solution.

If you are trying to make setup.py sdist work when libs are missing, consider using the code from build/pkgs/sagelib/src/setup.py line 61

Yes, that was my goal. Creating the virtual environment otherwise fails as Cython is not yet installed into it. The file you refer to doesn't seem to exist.

This file: https://git.sagemath.org/sage.git/tree/build/pkgs/sagelib/src/setup.py?id=fec55234a00e46d023ad583f8c6284b970141ab0

comment:380 Changed 22 months ago by Matthias Köppe

comment 374?

comment:381 Changed 21 months ago by Matthias Köppe

Milestone: sage-9.3sage-9.4

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

comment:382 Changed 17 months ago by Matthias Köppe

Milestone: sage-9.4sage-9.5

Setting a new milestone for this ticket based on a cursory review.

comment:383 Changed 15 months ago by Matthias Köppe

Milestone: sage-9.5sage-wishlist
Status: needs_reviewneeds_work

comment:384 Changed 14 months ago by git

Commit: fec55234a00e46d023ad583f8c6284b970141ab084b2dd396eb3cad818c3fb7893d38fcf8eec5031

Branch pushed to git repo; I updated commit sha1. New commits:

84b2dd3Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace

comment:385 Changed 14 months ago by git

Commit: 84b2dd396eb3cad818c3fb7893d38fcf8eec5031531811aebb493bce59dcfc09d250e317c2bf42bf

Branch pushed to git repo; I updated commit sha1. New commits:

8f99a3bCode cleanup
531811aDetect sdist and egg_info, and don't require cython in this case

comment:386 Changed 14 months ago by git

Commit: 531811aebb493bce59dcfc09d250e317c2bf42bfc56ab1698fe158dc741b6ec4cef7cc883836d835

Branch pushed to git repo; I updated commit sha1. New commits:

c56ab16Install Cython manually

comment:387 Changed 14 months ago by Tobias Diez

Description: modified (diff)

comment:388 Changed 14 months ago by Tobias Diez

Description: modified (diff)

comment:389 Changed 14 months ago by git

Commit: c56ab1698fe158dc741b6ec4cef7cc883836d83529f5780478afe6612de6e50ff8371cc29403f94d

Branch pushed to git repo; I updated commit sha1. New commits:

6c5ceaeAlso install cython for mac
29f5780Update lock file

comment:390 Changed 14 months ago by git

Commit: 29f5780478afe6612de6e50ff8371cc29403f94d536fa27c5b54ecaa9884be03079dda75027a645f

Branch pushed to git repo; I updated commit sha1. New commits:

e07292bImprove handling of fpylll
536fa27Install dependencies from lock file

comment:391 Changed 14 months ago by Tobias Diez

Description: modified (diff)

comment:392 Changed 14 months ago by Tobias Diez

Description: modified (diff)

comment:393 Changed 14 months ago by git

Commit: 536fa27c5b54ecaa9884be03079dda75027a645fb88cba4dd65e2a7d398b0822adf2c074049f9910

Branch pushed to git repo; I updated commit sha1. New commits:

ec89909Use automatically generated pipfile
b88cba4git-ignore generated files

comment:394 Changed 14 months ago by git

Commit: b88cba4dd65e2a7d398b0822adf2c074049f99103e796c31b13425f16a71216ce8c299b8eda4ddad

Branch pushed to git repo; I updated commit sha1. New commits:

3e796c3Require new memory-allocator package

comment:395 Changed 14 months ago by git

Commit: 3e796c31b13425f16a71216ce8c299b8eda4ddade9ac3ee9a49c71f6cf03cd6e8823d9c79ca51d89

Branch pushed to git repo; I updated commit sha1. New commits:

38593f4Clarify wording in github action workflow
e9ac3eeRemove unnecessary special treatment of fpylll and cython

comment:396 Changed 14 months ago by Tobias Diez

Description: modified (diff)
Milestone: sage-wishlistsage-9.5
Status: needs_workneeds_review

I've finally found the time to merge the most recent version. Since most changes (in the build pipline and adding the pipfile etc) have been done already in previous tickets, this ticket now essentially amounts to adding a github action verifying that the setup works indeed. So ready for review again.


New commits:

38593f4Clarify wording in github action workflow
e9ac3eeRemove unnecessary special treatment of fpylll and cython

comment:397 Changed 14 months ago by git

Commit: e9ac3ee9a49c71f6cf03cd6e8823d9c79ca51d894b7eaa4adc898e8f9b2bfd727e31232f89e99783

Branch pushed to git repo; I updated commit sha1. New commits:

4b7eaa4Improve workflow step name

comment:398 Changed 14 months ago by Matthias Köppe

The memory_allocator dep has been added in #32608

comment:399 Changed 14 months ago by Matthias Köppe

Separate tickets please for:

  • the language=c++ changes
  • the setup.py change

comment:400 in reply to:  399 Changed 14 months ago by Matthias Köppe

Replying to mkoeppe:

Separate tickets please for:

  • the language=c++ changes

(... if these changes are actually needed)

comment:401 in reply to:  399 Changed 14 months ago by Matthias Köppe

Replying to mkoeppe:

Separate tickets please for:

  • the setup.py change

could go into #32672

comment:402 Changed 14 months ago by Tobias Diez

Description: modified (diff)

comment:403 Changed 14 months ago by Tobias Diez

Description: modified (diff)

comment:404 Changed 14 months ago by Tobias Diez

Description: modified (diff)

comment:405 Changed 14 months ago by git

Commit: 4b7eaa4adc898e8f9b2bfd727e31232f89e99783e183209eebd9d141b9a3a91af467e06f54711754

Branch pushed to git repo; I updated commit sha1. New commits:

cb06460Try to fix python version on mac
2d39970Remove memory print as this is fixed now
380eec8Improve documentation
5afbd62Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace
e183209Remove memory_allocator from pipfile (no longer needed)

comment:406 Changed 14 months ago by git

Commit: e183209eebd9d141b9a3a91af467e06f54711754c6fde5eef8edc893643513d5f99a29a2b720556c

Branch pushed to git repo; I updated commit sha1. New commits:

fb5a7cfRemove lang=c++ (which might no longer be needed)
c6fde5eAdd docs on how to update the editable install after changes to cython files.

comment:407 in reply to:  399 Changed 14 months ago by Tobias Diez

Replying to mkoeppe:

Separate tickets please for:

  • the language=c++ changes
  • the setup.py change

Okay, I've removed the c++ changes as they seem to be no longer needed and opened #32673 for the changes to setup.py.

comment:408 Changed 14 months ago by Matthias Köppe

+        # Fix path to R, since otherwise sage's r is used
+        R_HOME: /usr/local/Cellar/r/4.0.3_2/lib/R

Is this needed, and if so, do we need this something like for a normal build too?

comment:409 Changed 14 months ago by Matthias Köppe

+        export SAGE_SHARE=$GITHUB_WORKSPACE/.tox/local-homebrew-macos-usrlocal/local/share

This line (in both workflow scripts) shouldn't be needed.

I also don't understand the other environment exports in the workflow script - why not invoke the sage-env script that sets up the correct environment?

comment:410 Changed 14 months ago by Matthias Köppe

Also the documentation on Pipenv added in src/doc/en/developer/coding_basics.rst is not sufficiently specific about the steps needed before pipenv is installed and used.

comment:411 Changed 14 months ago by Matthias Köppe

And finally, checking in the Pipfile.lock really does not fit into the Sage development workflow -- it will always be out of date and will lead to merge conflicts. We really should make do with just a Pipfile (generated from Pipfile.m4) from the versions declared in build/pkgs.

comment:412 in reply to:  408 Changed 14 months ago by Tobias Diez

Replying to mkoeppe:

+        # Fix path to R, since otherwise sage's r is used
+        R_HOME: /usr/local/Cellar/r/4.0.3_2/lib/R

Is this needed, and if so, do we need this something like for a normal build too?

Otherwise the pip-install of rpy2 fails with

Collecting rpy2<3.4,>=3.3
  Downloading rpy2-3.3.6.tar.gz (179 kB)
    Running command python setup.py egg_info
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-install-m6wdy35r/rpy2_0c800ed9b8e34449aeee849f526a409b/setup.py", line 115, in <module>
        c_extension_status = get_r_c_extension_status()
      File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-install-m6wdy35r/rpy2_0c800ed9b8e34449aeee849f526a409b/setup.py", line 99, in get_r_c_extension_status
        *situation.get_r_flags(r_home, '--ldflags')
      File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-install-m6wdy35r/rpy2_0c800ed9b8e34449aeee849f526a409b/rpy2/situation.py", line 209, in get_r_flags
        _get_r_cmd_config(r_home, flags,
      File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-install-m6wdy35r/rpy2_0c800ed9b8e34449aeee849f526a409b/rpy2/situation.py", line 177, in _get_r_cmd_config
        output = subprocess.check_output(
      File "/Users/runner/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/subprocess.py", line 415, in check_output
        return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
      File "/Users/runner/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/subprocess.py", line 493, in run
        with Popen(*popenargs, **kwargs) as process:
      File "/Users/runner/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/subprocess.py", line 858, in __init__
        self._execute_child(args, executable, preexec_fn, close_fds,
      File "/Users/runner/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/subprocess.py", line 1704, in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: '/Users/runner/work/sagetrac-mirror/sagetrac-mirror/.tox/local-homebrew-macos-usrlocal/local/var/tmp/sage/build/r-3.6.3/src/bin/R'

comment:413 in reply to:  409 ; Changed 14 months ago by Tobias Diez

Replying to mkoeppe:

I also don't understand the other environment exports in the workflow script - why not invoke the sage-env script that sets up the correct environment?

I was not aware of the sage-env script. How does one uses it to set the environment for the following steps and replace the manual exports I've added? I couldn't find any documentation about that script.

comment:414 in reply to:  410 ; Changed 14 months ago by Tobias Diez

Replying to mkoeppe:

Also the documentation on Pipenv added in src/doc/en/developer/coding_basics.rst is not sufficiently specific about the steps needed before pipenv is installed and used.

What do you mean concretely? Please also note that the documentation about venv should be improved in general, see #31342.

comment:415 in reply to:  411 ; Changed 14 months ago by Tobias Diez

Replying to mkoeppe:

And finally, checking in the Pipfile.lock really does not fit into the Sage development workflow -- it will always be out of date and will lead to merge conflicts. We really should make do with just a Pipfile (generated from Pipfile.m4) from the versions declared in build/pkgs.

I agree that it's somewhat suboptimal to have the version information of some packages specified in the sage distribution and in the pipenv lock file. However, the idea behind using pipenv is to provide an alternative way to quickly create a python environment for the development (and maybe at some point distribution) of sage. Thus such a duplication is needed to some degree for now. The point of the lock file is also to ensure that the python environment is the same for everyone (which makes it easier to reproduce issues and test failures etc) and more importantly that the builds in the github action are deterministic. Finally, in can be easily updated by running pipenv lock so it's not a lot of overhead. Please also note that it doesn't need to be updated every time a sage dependency is updated since pipenv uses the install_requires information to determine the admissible versions. I would suggest to run pipenv lock after every release for now and see how this goes.

Moreover, the lock file is required for now as some of the packages (pplpy for example) need cython and other packages installed already on sdist and egg_info. This makes setting up the pipenv from scratch rather hard, since you first need to create a new virtual environment, then install those dependencies and only then run pipenv install. These problems don't occur if you use the lock file (and subsequently pipenv lock).

Anyway, because there are pros and cons for each approach I would suggest to see how it goes with the lock file committed (as strongly recommended by pipenv). If it turns out that the disadvantages overweight we can always remove it later again.

comment:416 Changed 14 months ago by git

Commit: c6fde5eef8edc893643513d5f99a29a2b720556cf40976c9a3d18bd2601366ae48cd42ebf257e750

Branch pushed to git repo; I updated commit sha1. New commits:

6601db6Fix path to R
f40976cDon't need to generate pipfile since it's static anyway

comment:417 Changed 14 months ago by Tobias Diez

Description: modified (diff)
Summary: In-place (editable) installs of sagelib in a venvUse pipenv to create venv for development

Rename because the ticket is now more about pipenv than editable installs.

comment:418 in reply to:  414 ; Changed 14 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

Replying to mkoeppe:

Also the documentation on Pipenv added in src/doc/en/developer/coding_basics.rst is not sufficiently specific about the steps needed before pipenv is installed and used.

What do you mean concretely?

I'd say start by integrating the ticket description (which has some step by step instructions) into this section of the documentation. (Perhaps best to shorten the ticket description, referring to the added documentation.)

Please also note that the documentation about venv should be improved in general, see #31342.

Yes, that's true

comment:419 in reply to:  415 Changed 14 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

... can be easily updated by running pipenv lock so it's not a lot of overhead. Please also note that it doesn't need to be updated every time a sage dependency is updated since pipenv uses the install_requires information to determine the admissible versions. I would suggest to run pipenv lock after every release for now and see how this goes.

Could you add the precise process to the documentation please? This would be a task for the release manager, who would need to be briefed about this in detail

comment:420 in reply to:  413 Changed 14 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

Replying to mkoeppe:

I also don't understand the other environment exports in the workflow script - why not invoke the sage-env script that sets up the correct environment?

I was not aware of the sage-env script. How does one uses it to set the environment for the following steps and replace the manual exports I've added? I couldn't find any documentation about that script.

The top of the file (src/bin/sage-env) explains how to use it

comment:421 in reply to:  418 Changed 14 months ago by Matthias Köppe

Replying to mkoeppe:

I'd say start by integrating the ticket description (which has some step by step instructions) into this section of the documentation. (Perhaps best to shorten the ticket description, referring to the added documentation.)

(However, the tox -e local-sudo-standard build-local shouldn't go into the documentation because we do not want to advertise this as a build/provisioning tool; it is strictly meant for portability testing.)

comment:422 Changed 14 months ago by git

Commit: f40976c9a3d18bd2601366ae48cd42ebf257e750cf832ca3f0132f2b87bf4c4d1c49ccd17a3dc80d

Branch pushed to git repo; I updated commit sha1. New commits:

cf832caTry to find out why r doesn't work on mac

comment:423 Changed 14 months ago by git

Commit: cf832ca3f0132f2b87bf4c4d1c49ccd17a3dc80d605311ed1b3603b204b71c96cdd5cf8f81b54904

Branch pushed to git repo; I updated commit sha1. New commits:

605311eTry to fix r again

comment:424 Changed 14 months ago by git

Commit: 605311ed1b3603b204b71c96cdd5cf8f81b54904ae26a5f3549413bdc877ac325b52f4faee31b12f

Branch pushed to git repo; I updated commit sha1. New commits:

4ca719cTry to fix some doctests
ae26a5fRemove r testing code

comment:425 Changed 14 months ago by git

Commit: ae26a5f3549413bdc877ac325b52f4faee31b12f4b7ca86998ec6cf2122101e9050861865ee7da26

Branch pushed to git repo; I updated commit sha1. New commits:

4b7ca86Remove sage share export

comment:426 Changed 14 months ago by git

Commit: 4b7ca86998ec6cf2122101e9050861865ee7da26853ee59214497b4173de2b13050026676bf85279

Branch pushed to git repo; I updated commit sha1. New commits:

6c2c2c9Add sphinx-autobuild as dev package
853ee59Expand documentation

comment:427 Changed 14 months ago by Tobias Diez

Thanks for your comments. I think I've addressed all of them now.

Only . bin/sage-env doesn't work for me. Afterwards, the pipenv command throws the error that the pipenv module cannot be found. I guess the reason is that pipenv is not installed in the sage python environment. Since this would also only replace the export xyz commands in the github workflow, I propose we do this in a follow-up ticket.

comment:428 Changed 14 months ago by git

Commit: 853ee59214497b4173de2b13050026676bf85279f2ce6a0fcd9cdb38d2202263dc22e0924549596e

Branch pushed to git repo; I updated commit sha1. New commits:

f2ce6a0Lets ask R about its home

comment:429 Changed 14 months ago by Tobias Diez

Description: modified (diff)

comment:430 in reply to:  409 Changed 14 months ago by Tobias Diez

Replying to mkoeppe:

+        export SAGE_SHARE=$GITHUB_WORKSPACE/.tox/local-homebrew-macos-usrlocal/local/share

This line (in both workflow scripts) shouldn't be needed.

Without it a lot of tests fail, for example because Cremona cannot be found. See https://github.com/sagemath/sagetrac-mirror/runs/3893523414?check_suite_focus=true. I've thus readded the sage share export.

comment:431 Changed 14 months ago by git

Commit: f2ce6a0fcd9cdb38d2202263dc22e0924549596eae3eae7f5764227f1ffffb0105839cb8d562ac63

Branch pushed to git repo; I updated commit sha1. New commits:

ae3eae7Readd sage_share export

comment:432 Changed 14 months ago by Matthias Köppe

Doesn't your sage-conf provide the Cremona install paths?

comment:433 Changed 14 months ago by Tobias Diez

Might this be that in the github action the installed sage-conf doesn't refer to the tox environment but to the one in /local ?

comment:434 Changed 14 months ago by Matthias Köppe

I don't know, but the idea of sage-conf is that you install a version that provides the correct setting of SAGE_LOCAL

comment:435 Changed 14 months ago by Tobias Diez

Problem is that the path of sage-conf needs to be specified in the pipfile and thus has to be known before. Maybe in the future one can add a symlink (defaulting to pkgs/sage-conf) that can be changed to point to a tox environment? But I guess this would be more or less equivalent to changing SAGE_SHARE.

comment:436 Changed 14 months ago by Matthias Köppe

The path in the source tree is always pkgs/sage-conf/

comment:437 Changed 14 months ago by Matthias Köppe

(The local .tox builds do not make a copy of SAGE_ROOT. They only use a separate SAGE_LOCAL.)

comment:438 Changed 14 months ago by git

Commit: ae3eae7f5764227f1ffffb0105839cb8d562ac63e41d022821baf31c99bfdb3aaff60a89eafe6c07

Branch pushed to git repo; I updated commit sha1. New commits:

e41d022Found R's home, so lets go there tonight

comment:439 in reply to:  436 ; Changed 14 months ago by Tobias Diez

Replying to mkoeppe:

The path in the source tree is always pkgs/sage-conf/

Okay, then I don't know what the problem is. I would propose to keep the explicit SAGE_SHARE for now and then afterwards create a new ticket in which it is removed. The github workflow right now is anyway a big compromise of "how it should work" vs "what is actually needed to make it work". There are quite a few special tweaks and workarounds that ideally shouldn't be needed.

comment:440 Changed 14 months ago by git

Commit: e41d022821baf31c99bfdb3aaff60a89eafe6c0780aa6bb1ad4d71f63ad6bc33644e0eb5c2d2e25b

Branch pushed to git repo; I updated commit sha1. New commits:

b03c120Fix mapmath also on mac
80aa6bbSligtly increase parallelism for make

comment:441 Changed 14 months ago by git

Commit: 80aa6bb1ad4d71f63ad6bc33644e0eb5c2d2e25b0fcb85665cee14abd3492ba02c4b53b246d40b2c

Branch pushed to git repo; I updated commit sha1. New commits:

0fcb856Try to fix path to gmp (and boost) on macos

comment:442 in reply to:  439 Changed 14 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

Replying to mkoeppe:

The path in the source tree is always pkgs/sage-conf/

Okay, then I don't know what the problem is.

To debug this, in the activated venv you could ask sage-config to print the configuration

comment:443 in reply to:  427 ; Changed 14 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

Only . bin/sage-env doesn't work for me.

Take a look at what bin/sage does. It first sources config files on which sage-env relies.

comment:444 in reply to:  415 ; Changed 14 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

some of the packages (pplpy for example) need cython and other packages installed already on sdist and egg_info.

This should be fixed. Could you open issues please with these packages? We also have meta-ticket #30922 for keeping track of these issues.

comment:445 in reply to:  415 ; Changed 14 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

The point of the lock file is also to ensure that the python environment is the same for everyone (which makes it easier to reproduce issues and test failures etc)

I understand that for general Python projects it has the benefit that you describe (although the benefit over plain requirements.txt is still not clear to me, but let's not elaborate on this now).

The problem is that in our case, adding the lock file locks versions that (1) are *not the same* as the ones in the Sage distribution, (2) it will be out of date next week because we have no established process to update it. So it reduces clarity for users and creates a maintenance burden for Sage development.

I think it would be a good idea to work on a process (= script/documentation) based on pipenv that actually helps Sage developers to make Python package upgrades. Currently this is a manual process based on guesswork and testing. So this could be a key added value that Pipenv provides.

comment:446 in reply to:  415 Changed 14 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

Moreover, the lock file is required for now as some of the packages (pplpy for example) need cython and other packages installed already on sdist and egg_info.

Actually pplpy declares these build dependencies correctly in https://gitlab.com/videlec/pplpy/-/blob/master/pyproject.toml

comment:447 Changed 14 months ago by git

Commit: 0fcb85665cee14abd3492ba02c4b53b246d40b2c5e60fc2ef9c0ab3db0e3431519dd551dc49b6321

Branch pushed to git repo; I updated commit sha1. New commits:

5e60fc2Lets try sage-env

comment:448 Changed 14 months ago by git

Commit: 5e60fc2ef9c0ab3db0e3431519dd551dc49b6321fe3940955e017f7445a7aae866d750841dca5db7

Branch pushed to git repo; I updated commit sha1. New commits:

fe39409sage-env thinks its time for 127, we don't agree

comment:449 Changed 14 months ago by Matthias Köppe

comment:443 applies

comment:450 Changed 14 months ago by git

Commit: fe3940955e017f7445a7aae866d750841dca5db791310ce568a590edb59b20251107e67e21981b3a

Branch pushed to git repo; I updated commit sha1. New commits:

91310ceForce success exit code...strange

comment:451 Changed 14 months ago by git

Commit: 91310ce568a590edb59b20251107e67e21981b3a28faabd70ec932d4b7f2b92681b2f202cb2e9f29

Branch pushed to git repo; I updated commit sha1. New commits:

28faabdLets try code from sage script

comment:452 in reply to:  443 Changed 14 months ago by Tobias Diez

Replying to mkoeppe:

Replying to gh-tobiasdiez:

Only . bin/sage-env doesn't work for me.

Take a look at what bin/sage does. It first sources config files on which sage-env relies.

I was not able to make it work locally nor on github (where it fails with exit code 127 without giving any details). That's mostly because of my missing experience with bash and linux things. Also sage-env does a lot of things that are not needed at this point (e.g. overwriting the python location, which breaks pipenv). Anyway, lets make this a follow-up ticket.

comment:453 Changed 14 months ago by Tobias Diez

Description: modified (diff)

comment:454 Changed 14 months ago by git

Commit: 28faabd70ec932d4b7f2b92681b2f202cb2e9f29d4d8236d91cf864744d7a7dee4710732b10a2b72

Branch pushed to git repo; I updated commit sha1. New commits:

d4d8236I give up on sage-env...

comment:455 in reply to:  444 Changed 14 months ago by Tobias Diez

Replying to mkoeppe:

Replying to gh-tobiasdiez:

some of the packages (pplpy for example) need cython and other packages installed already on sdist and egg_info.

This should be fixed. Could you open issues please with these packages? We also have meta-ticket #30922 for keeping track of these issues.

Done: https://gitlab.com/videlec/pplpy/-/issues/27

comment:456 in reply to:  445 ; Changed 14 months ago by Tobias Diez

Replying to mkoeppe:

The problem is that in our case, adding the lock file locks versions that (1) are *not the same* as the ones in the Sage distribution,

Maybe a shift in viewpoint might be helpful to see how pipenv helps. Sagelib (things in src) declares its dependencies in form of install_requires. Currently, sage-the-distrubition is one way to provide a certain set of packages that satisfy the version constraints. The newly added pipfile.lock specifies another set of packages that also satisfy the constraints. If sagelib is indeed configured correctly, then it should work with both sets. So you can see the added github workflow as a test that the install_requires constraints are correct.

(2) it will be out of date next week because we have no established process to update it. So it reduces clarity for users and creates a maintenance burden for Sage development.

dependabot or rennovate are the modern approaches to not worry about manually updating dependencies. But they don't support trac... But updating the pipfile lock manually is easy, automated and only needs to be done rarely. Its definitively more convenient then updating the versions in sage-the-distro.

I think it would be a good idea to work on a process (= script/documentation) based on pipenv that actually helps Sage developers to make Python package upgrades. Currently this is a manual process based on guesswork and testing. So this could be a key added value that Pipenv provides.

I agree. First of all, my idea with pipenv was to completely replace sages python packages and use it also for distribution. But also in the short term, pipenv can be helpful. For example, the following are the versions determined by pipenv:

alabaster                     0.7.12
argon2-cffi                   21.1.0
attrs                         21.2.0
Babel                         2.9.1
backcall                      0.2.0
backports.zoneinfo            0.2.1
bleach                        4.1.0
certifi                       2021.10.8
cffi                          1.15.0
charset-normalizer            2.0.7
cycler                        0.10.0
cypari2                       2.1.2
cysignals                     1.10.3
Cython                        0.29.24
debugpy                       1.5.0
decorator                     5.1.0
defusedxml                    0.7.1
docutils                      0.17.1
entrypoints                   0.3
fpylll                        0.5.6
gmpy2                         2.1.0b5
idna                          3.3
imagesize                     1.2.0
iniconfig                     1.1.1
ipykernel                     6.4.1
ipython                       7.28.0
ipython-genutils              0.2.0
ipywidgets                    7.6.5
jedi                          0.18.0
Jinja2                        3.0.2
jsonschema                    4.1.0
jupyter-client                7.0.6
jupyter-core                  4.8.1
jupyterlab-pygments           0.1.2
jupyterlab-widgets            1.0.2
kiwisolver                    1.3.2
MarkupSafe                    2.0.1
matplotlib                    3.4.3
matplotlib-inline             0.1.3
memory-allocator              0.1.1
mistune                       0.8.4
mpmath                        1.2.1
nbclient                      0.5.4
nbconvert                     6.2.0
nbformat                      5.1.3
nest-asyncio                  1.5.1
networkx                      2.6.3
notebook                      6.4.4
numpy                         1.21.2
packaging                     21.0
pandocfilters                 1.5.0
parso                         0.8.2
pexpect                       4.8.0
pickleshare                   0.7.5
Pillow                        8.3.2
pip                           21.2.4
pkgconfig                     1.5.5
pluggy                        1.0.0
pplpy                         0.8.7
prometheus-client             0.11.0
prompt-toolkit                3.0.20
psutil                        5.8.0
ptyprocess                    0.7.0
py                            1.10.0
pycparser                     2.20
Pygments                      2.10.0
pyparsing                     2.4.7
pyrsistent                    0.18.0
pytest                        6.2.5
python-dateutil               2.8.2
pytz                          2021.3
pyzmq                         22.3.0
requests                      2.26.0
rpy2                          3.3.6
sage-conf                     9.5b3     /__w/sagetrac-mirror/sagetrac-mirror/pkgs/sage-conf
sagemath-standard             9.5b3     /__w/sagetrac-mirror/sagetrac-mirror/src
scipy                         1.7.1
Send2Trash                    1.8.0
setuptools                    58.1.0
six                           1.16.0
snowballstemmer               2.1.0
Sphinx                        4.1.2
sphinxcontrib-applehelp       1.0.2
sphinxcontrib-devhelp         1.0.2
sphinxcontrib-htmlhelp        2.0.0
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.3
sphinxcontrib-serializinghtml 1.1.5
sympy                         1.9
terminado                     0.12.1
testpath                      0.5.0
toml                          0.10.2
tornado                       6.1
traitlets                     5.1.0
tzlocal                       3.0
urllib3                       1.26.7
wcwidth                       0.2.5
webencodings                  0.5.1
wheel                         0.37.0
widgetsnbextension            3.5.1

Now you can use those versions as the package version of sage-the-distro. Lets discuss this further in #31346.

comment:457 in reply to:  454 ; Changed 14 months ago by Matthias Köppe

Replying to git:

Branch pushed to git repo; I updated commit sha1. New commits:

d4d8236I give up on sage-env...

You forgot to use sage-env-config etc. as per comment:443

comment:458 in reply to:  456 ; Changed 14 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

Replying to mkoeppe:

The problem is that in our case, adding the lock file locks versions that (1) are *not the same* as the ones in the Sage distribution,

Maybe a shift in viewpoint might be helpful to see how pipenv helps. Sagelib (things in src) declares its dependencies in form of install_requires. Currently, sage-the-distrubition is one way to provide a certain set of packages that satisfy the version constraints. The newly added pipfile.lock specifies another set of packages that also satisfy the constraints. If sagelib is indeed configured correctly, then it should work with both sets. So you can see the added github workflow as a test that the install_requires constraints are correct.

I don't find this convincing. A checked-in Pipfile.lock is just the result of resolving the declared version constraints (as of some point in the past) against what was available on the index at some some point in the past.

When the workflow fails, it will mean just one thing: Constraints were updated and Pipfile.lock was not updated (-> maintenance burden).

It's more valuable to test on GH Actions whether the declared version constraints resolve *at the current time* to a set of versions that work correctly. If this fails, it is clear that the constraints declared in install_requires are not correct.

comment:459 Changed 14 months ago by Matthias Köppe

By the way, see SAGE_ROOT/pkgs/sagemath-standard/tox.ini for test environments for several ways to provision a venv. Looking at it, I am also reminded that we actually have two sets of Pipenv

comment:460 in reply to:  457 Changed 14 months ago by Tobias Diez

Replying to mkoeppe:

Replying to git:

Branch pushed to git repo; I updated commit sha1. New commits:

d4d8236I give up on sage-env...

You forgot to use sage-env-config etc. as per comment:443

I tried it locally and couldn't determine which other scripts are needed to run first. To be honest, I'm tired of the try and error play (especially since it only replace a couple of very easy to understand export statements).

comment:461 Changed 14 months ago by Matthias Köppe

You can also just use sage -sh to run something in the environment, like it's explained in the instructions of SAGE_ROOT/pkgs/sagemath-standard/tox.ini

comment:462 in reply to:  458 Changed 14 months ago by Tobias Diez

Replying to mkoeppe:

When the workflow fails, it will mean just one thing: Constraints were updated and Pipfile.lock was not updated (-> maintenance burden). It's more valuable to test on GH Actions whether the declared version constraints resolve *at the current time* to a set of versions that work correctly. If this fails, it is clear that the constraints declared in install_requires are not correct.

I guess it depends on what you want to test (and when you want to run the workflow):

  • Test that the changes in the actual code (src) don't break the doctests (i.e run on push). In this case, you want to keep the environment fixed (since otherwise the workflow may fail due to newly released versions on pypi).
  • Test that sage works with the newest versions of packages compatible with the install_require constraints. In this case, you probably want to run the workflow manually, before release or in certain interval. Of course, in this case one should ignore the committed pipfile.lock.

My initial motivation was the first use case (as an alternative/replacement for the patchbots), but you are right the second case is important as well. For this reason, I've now added the possibility to run the workflow in manually in "update" mode in which case the pipfile.lock is ignored. Note that the latter still needs a committed pipfile lock at the moment due to the problems described above (https://trac.sagemath.org/ticket/30922).

comment:463 Changed 14 months ago by git

Commit: d4d8236d91cf864744d7a7dee4710732b10a2b72c5b81335989d09a0c814bc58760b9a15174f23df

Branch pushed to git repo; I updated commit sha1. New commits:

c5b8133Add update mode

comment:464 in reply to:  461 Changed 14 months ago by Tobias Diez

Replying to mkoeppe:

You can also just use sage -sh to run something in the environment, like it's explained in the instructions of SAGE_ROOT/pkgs/sagemath-standard/tox.ini

I tried this as well, but then other things get messed up (e.g. PATH, which Python is used etc). What I would like to have and what is needed here is a sage-build-env in which only the build-related environment variables are changed.

comment:465 Changed 14 months ago by Tobias Diez

By the way, the "update" mode is only available after this ticket is merged due to a security constraint by github ("To trigger the workflow_dispatch event, your workflow must be in the default branch.").

comment:466 Changed 14 months ago by Matthias Köppe

I think all the troubles that you are having with the environment are the result of using tox to build the SAGE_LOCAL but then not activating the tox environment. A way out of this is to go through #31535

comment:467 Changed 14 months ago by git

Commit: c5b81335989d09a0c814bc58760b9a15174f23df8fe104855e2f23825b22b43027eeae33a6c9e593

Branch pushed to git repo; I updated commit sha1. New commits:

8fe1048Run in update mode (to check if its works)

comment:468 Changed 14 months ago by git

Commit: 8fe104855e2f23825b22b43027eeae33a6c9e593821919d2abd6309441dd4b86b752c205baa13971

Branch pushed to git repo; I updated commit sha1. New commits:

821919dUse same env variables for update as for initial install

comment:469 Changed 14 months ago by git

Commit: 821919d2abd6309441dd4b86b752c205baa13971198c6d8f88bb9e5c0fce83746b4d858b9e48bd51

Branch pushed to git repo; I updated commit sha1. New commits:

198c6d8One last try with sage-env

comment:470 Changed 14 months ago by git

Commit: 198c6d8f88bb9e5c0fce83746b4d858b9e48bd5114023b10709ce26c17ecfda0361677e63b249ff6

Branch pushed to git repo; I updated commit sha1. New commits:

3c04ea4Fix LD_LIBRARY_PATH
14023b1Improve update step

comment:471 Changed 13 months ago by git

Commit: 14023b10709ce26c17ecfda0361677e63b249ff64fbb8a9848f2d2f133522077663e8661c4c2eaab

Branch pushed to git repo; I updated commit sha1. New commits:

4fbb8a9Install git as well

comment:472 Changed 13 months ago by git

Commit: 4fbb8a9848f2d2f133522077663e8661c4c2eaabbd4684abf0709c5f19054bae3d50ef7d3cf40f29

Branch pushed to git repo; I updated commit sha1. New commits:

9a3628bTry to fix strange git error
dd009e7Small cleanup
bd4684aTry with ubuntu hirsute (should speed up things a bit)

comment:473 Changed 13 months ago by git

Commit: bd4684abf0709c5f19054bae3d50ef7d3cf40f299b74bd621de62b289bca94cdd65bb17e48bde858

Branch pushed to git repo; I updated commit sha1. New commits:

9b74bd6Use deadsnakes action to setup python on linux

comment:474 Changed 13 months ago by git

Commit: 9b74bd621de62b289bca94cdd65bb17e48bde858dd009e77906bd831f3cc73666afcbe15d9bc990d

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

comment:475 Changed 13 months ago by git

Commit: dd009e77906bd831f3cc73666afcbe15d9bc990d4adc4afa6d19187699414b364748a047775f00d8

Branch pushed to git repo; I updated commit sha1. New commits:

4adc4afDisable update mode again

comment:476 in reply to:  466 Changed 13 months ago by Tobias Diez

Replying to mkoeppe:

I think all the troubles that you are having with the environment are the result of using tox to build the SAGE_LOCAL but then not activating the tox environment. A way out of this is to go through #31535

It was actually something strange going on on my system. After make distclean I could run sage-env locally and then also in the github action. Sorry for the confusion and thanks for your help.

The update mode was a bit more work than I'd liked but its running now. See https://github.com/sagemath/sagetrac-mirror/runs/3926515272?check_suite_focus=true for an example output. It can be activated by running the workflow manually from the github interface (after it is merged into the default branch).

comment:477 Changed 13 months ago by git

Commit: 4adc4afa6d19187699414b364748a047775f00d850bd3ce3eadc344b8c8067a2bf3e7e659c39d1ef

Branch pushed to git repo; I updated commit sha1. New commits:

50bd3ceMerge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace

comment:478 Changed 12 months ago by Tobias Diez

Anything left to do here or is it good to go?

comment:479 Changed 12 months ago by John Palmieri

I tried following the instructions in the ticket description, and in particular the block of commands

./bootstrap
./configure --enable-editable
make build-local
cd src
mkdir .venv
pipenv install --dev --ignore-pipfile
pipenv run sage-runtests --all

The last command fails with

Error: the command sage-runtests could not be found within PATH or Pipfile's [scripts].
Last edited 12 months ago by John Palmieri (previous) (diff)

comment:480 Changed 12 months ago by git

Commit: 50bd3ce3eadc344b8c8067a2bf3e7e659c39d1ef148cc729c93cd164b7155a4404ec09c60d5154a6

Branch pushed to git repo; I updated commit sha1. New commits:

148cc72Merge branch 'develop' of https://github.com/sagemath/sage into public/build/inplace

comment:481 Changed 12 months ago by Tobias Diez

Description: modified (diff)

New commits:

148cc72Merge branch 'develop' of https://github.com/sagemath/sage into public/build/inplace

comment:482 Changed 12 months ago by Tobias Diez

Sorry for not testing it after my last merge of the develop branch.

It turned out there is a problem (only reported by pipenv using the --verbose flag) during the installation of sage in the virtual environment. In fact, it turns out that this is not related at all to pipenv but a general issue with the installation using build isolution, so I've opened #32904 for this.

For pipenv this can be fixed by running

pipenv run pip install -e . --upgrade --exists-action=i --no-build-isolation

after the initial build of the venv. I've changed the ticket description accordingly.

comment:483 Changed 12 months ago by git

Commit: 148cc729c93cd164b7155a4404ec09c60d5154a60b0437fc7a2df0bb34873e6d0d881a775e1ebb09

Branch pushed to git repo; I updated commit sha1. New commits:

0b0437fAdd sage-setup as editable install, update lock

comment:484 Changed 12 months ago by John Palmieri

I don't have any opinion about the code, so I am not the right person to review this, but I did get a lot of doctest failures when following the directions in the ticket:

sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/env.py  # 4 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/misc/superseded.py  # 6 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/misc/cython.py  # 21 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/misc/inline_fortran.py  # 3 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/misc/cachefunc.pyx  # 54 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/misc/session.pyx  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/misc/randstate.pyx  # 25 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/misc/inherit_comparison.pyx  # 5 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/misc/sagedoc.py  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/misc/nested_class.pyx  # 6 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/misc/lazy_attribute.pyx  # 3 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/misc/sageinspect.py  # 35 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/misc/sage_eval.py  # 7 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/misc/persist.pyx  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/schemes/elliptic_curves/ell_rational_field.py  # 21 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/schemes/elliptic_curves/lseries_ell.py  # 4 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/schemes/generic/algebraic_scheme.py  # 13 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/schemes/generic/homset.py  # 3 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/schemes/toric/fano_variety.py  # 137 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/schemes/toric/divisor_class.pyx  # 50 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/schemes/toric/points.py  # 108 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/schemes/toric/library.py  # 74 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/schemes/toric/toric_subscheme.py  # 104 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/schemes/toric/variety.py  # 229 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/schemes/toric/divisor.py  # 234 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/schemes/toric/weierstrass.py  # 6 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/schemes/toric/weierstrass_covering.py  # 9 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/schemes/toric/chow_group.py  # 159 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/schemes/toric/homset.py  # 72 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/schemes/toric/morphism.py  # 194 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/schemes/toric/sheaf/klyachko.py  # 101 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/schemes/toric/sheaf/constructor.py  # 29 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/crypto/classical.py  # 16 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/parallel/decorate.py  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/plot/graphics.py  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/plot/contour_plot.py  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/plot/line.py  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/plot/arrow.py  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/plot/plot3d/tachyon.py  # 24 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/plot/plot3d/index_face_set.pyx  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/plot/plot3d/platonic.py  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/plot/plot3d/base.pyx  # 6 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/plot/plot3d/implicit_surface.pyx  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/plot/plot3d/implicit_plot3d.py  # 6 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/plot/plot3d/parametric_plot3d.py  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/plot/plot3d/parametric_surface.pyx  # 3 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/manifolds/differentiable/vectorfield.py  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/manifolds/differentiable/tangent_vector.py  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/combinat/constellation.py  # 6 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/combinat/tutorial.py  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/combinat/quickref.py  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/combinat/permutation.py  # 3 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/combinat/designs/ext_rep.py  # Killed due to abort
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/combinat/posets/posets.py  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/combinat/root_system/weyl_group.py  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/combinat/matrices/latin.py  # 45 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/lfunctions/zero_sums.pyx  # 3 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/lfunctions/lcalc.py  # 7 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/lfunctions/sympow.py  # 10 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/ext/memory_allocator.pxd  # 3 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/ext/memory_allocator.pyx  # 7 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/doctest/control.py  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/doctest/forker.py  # 4 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/features/__init__.py  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/tests/benchmark.py  # 3 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/tests/cmdline.py  # 8 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/tests/gap_packages.py  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/tests/books/judson-abstract-algebra/groups-sage.py  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/tests/books/judson-abstract-algebra/cyclic-sage.py  # 5 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/tests/books/judson-abstract-algebra/homomorph-sage.py  # 11 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/tests/books/judson-abstract-algebra/sylow-sage.py  # 5 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/tests/books/computational-mathematics-with-sagemath/combinat_doctest.py  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/algebras/orlik_terao.py  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/algebras/orlik_solomon.py  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/repl/interpreter.py  # 6 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/repl/interface_magic.py  # 7 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/repl/ipython_extension.py  # 6 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/repl/load.py  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/groups/galois_group.py  # 3 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/groups/class_function.py  # 137 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/groups/generic.py  # 5 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/groups/perm_gps/symgp_conjugacy_class.py  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/groups/perm_gps/permgroup_morphism.py  # 13 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/groups/perm_gps/permgroup.py  # 44 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/groups/perm_gps/cubegroup.py  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/groups/perm_gps/constructor.py  # 4 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/groups/perm_gps/permgroup_element.pyx  # 10 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/groups/perm_gps/partn_ref/data_structures.pyx  # 9 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/groups/abelian_gps/dual_abelian_group_element.py  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/groups/abelian_gps/abelian_group_morphism.py  # 15 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/groups/abelian_gps/abelian_group.py  # 64 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/docs/instancedoc.pyx  # 4 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/arith/misc.py  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/arith/long.pxd  # 14 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/libs/gap/element.pyx  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/libs/gap/libgap.pyx  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/cpython/dict_del_by_value.pyx  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/cpython/wrapperdescr.pxd  # 6 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/cpython/string.pyx  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/cpython/cython_metaclass.pyx  # 4 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/cpython/getattr.pyx  # 4 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/modular/abvar/abvar.py  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/modular/arithgroup/congroup_gamma0.py  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/modular/arithgroup/congroup_gammaH.py  # 5 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/modular/arithgroup/arithgroup_perm.py  # 5 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/modular/modform/constructor.py  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/modular/hecke/submodule.py  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/functions/other.py  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/functions/exp_integral.py  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/functions/generalized.py  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/functions/piecewise.py  # 13 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/coding/binary_code.pyx  # 3 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/coding/databases.py  # 4 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/coding/linear_code.py  # 22 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/coding/golay_code.py  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/coding/kasami_codes.pyx  # 5 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/coding/two_weight_db.py  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/coding/code_constructions.py  # 4 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/coding/linear_code_no_metric.py  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/coding/goppa_code.py  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/rings/integer_ring.pyx  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/rings/integer.pyx  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/rings/infinity.py  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/rings/integer_fake.pxd  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/rings/tate_algebra_ideal.pyx  # 6 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/rings/rational_field.py  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/rings/number_field/galois_group.py  # 3 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/rings/number_field/number_field.py  # 26 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/rings/number_field/number_field_element.pyx  # 6 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/rings/number_field/totallyreal_rel.py  # 7 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/rings/polynomial/polynomial_ring.py  # 5 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/rings/polynomial/polynomial_element.pyx  # 8 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/rings/polynomial/ore_polynomial_element.pyx  # 6 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/rings/polynomial/multi_polynomial.pyx  # 8 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/rings/polynomial/multi_polynomial_ring_base.pyx  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/rings/polynomial/multi_polynomial_ideal.py  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/rings/polynomial/groebner_fan.py  # 118 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/rings/finite_rings/galois_group.py  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/rings/finite_rings/element_pari_ffelt.pyx  # 11 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/rings/finite_rings/element_givaro.pyx  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/rings/finite_rings/integer_mod_ring.py  # 6 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/rings/finite_rings/integer_mod.pyx  # 4 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/rings/finite_rings/finite_field_givaro.py  # 7 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/graphs/connectivity.pyx  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/graphs/generic_graph.py  # 5 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/graphs/graph_decompositions/fast_digraph.pyx  # 5 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/symbolic/expression.pyx  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/symbolic/constants.py  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/symbolic/pynac.pxi  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/symbolic/integration/integral.py  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/symbolic/integration/external.py  # 6 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/geometry/cone_catalog.py  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/geometry/fan_morphism.py  # 92 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/geometry/fan.py  # 85 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/geometry/toric_plotter.py  # 12 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/geometry/lattice_polytope.py  # 196 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/geometry/cone.py  # 17 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/geometry/fan_isomorphism.py  # 13 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/geometry/polyhedron/ppl_lattice_polytope.py  # 3 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/geometry/polyhedron/base_ZZ.py  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/geometry/polyhedron/backend_normaliz.py  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/geometry/polyhedron/palp_database.py  # 23 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/geometry/polyhedron/base.py  # 7 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/geometry/triangulation/element.py  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/calculus/calculus.py  # 13 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/modules/free_module_element.pyx  # 3 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/modules/filtered_vector_space.py  # 5 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/modules/with_basis/invariant.py  # 74 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/modules/with_basis/representation.py  # 5 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/structure/parent_old.pyx  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/structure/element.pyx  # 37 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/structure/factory.pyx  # 8 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/structure/element.pxd  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/structure/parent.pyx  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/categories/primer.py  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/categories/finite_dimensional_modules_with_basis.py  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/categories/pushout.py  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/categories/modules_with_basis.py  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/matrix/matrix1.pyx  # 13 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/interfaces/tachyon.py  # 3 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/interfaces/expect.py  # 49 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/interfaces/mwrank.py  # 9 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/interfaces/ecm.py  # 12 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/interfaces/qsieve.py  # 7 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/interfaces/interface.py  # 31 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/interfaces/gap3.py  # 3 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/interfaces/gap.py  # 143 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/interfaces/tests.py  # 3 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage/interfaces/giac.py  # 140 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage_setup/find.py  # 3 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 sage_docbuild/__init__.py  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 doc/ja/tutorial/tour_groups.rst  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 doc/ja/tutorial/tour_advanced.rst  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 doc/ja/tutorial/interfaces.rst  # 5 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 doc/ru/tutorial/tour_groups.rst  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 doc/ru/tutorial/tour_advanced.rst  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 doc/ru/tutorial/interfaces.rst  # 5 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 doc/pt/tutorial/tour_groups.rst  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 doc/pt/tutorial/tour_advanced.rst  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 doc/pt/tutorial/interfaces.rst  # 5 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 doc/de/tutorial/tour_groups.rst  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 doc/de/tutorial/tour_advanced.rst  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 doc/de/tutorial/interfaces.rst  # 5 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 doc/fr/tutorial/tour_groups.rst  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 doc/fr/tutorial/tour_advanced.rst  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 doc/fr/tutorial/interfaces.rst  # 5 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 doc/es/tutorial/tour_groups.rst  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 doc/en/developer/coding_in_other.rst  # 3 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 doc/en/tutorial/tour_groups.rst  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 doc/en/tutorial/tour_advanced.rst  # 2 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 doc/en/tutorial/interfaces.rst  # 5 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 doc/en/constructions/rep_theory.rst  # 18 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 doc/en/constructions/polynomials.rst  # 13 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 doc/en/constructions/rings.rst  # 6 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 doc/en/constructions/linear_algebra.rst  # 3 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 doc/en/constructions/groups.rst  # 18 doctests failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 doc/en/thematic_tutorials/coding_theory.rst  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 doc/en/thematic_tutorials/explicit_methods_in_number_theory/elliptic_curves.rst  # 1 doctest failed
sage -t --warn-long 64.4 --random-seed=60140155690716510157027757823734981182 doc/en/thematic_tutorials/lie/weyl_groups.rst  # 1 doctest failed

Some come from

pkgconfig.pkgconfig.PackageNotFoundError: fflas-ffpack not found

and I also see some instances of errors like

File "doc/es/tutorial/tour_groups.rst", line 29, in doc.es.tutorial.tour_groups
Failed example:
    G.random_element()           # random output (resultado aleatorio)
Exception raised:
    Traceback (most recent call last):
      File "/Users/jpalmier/Desktop/Sage/sage_builds/TESTING/sage-9.5.beta6/src/sage/interfaces/gap.py", line 1155, in _start
        raise OSError("GAP workspace too old")
    OSError: GAP workspace too old

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/Users/jpalmier/Desktop/Sage/sage_builds/TESTING/sage-9.5.beta6/src/sage/interfaces/expect.py", line 492, in _start
        self._expect = SageSpawn(cmd,
      File "sage/interfaces/sagespawn.pyx", line 65, in sage.interfaces.sagespawn.SageSpawn.__init__
        with ContainChildren(silent=True):
      File "sage/interfaces/sagespawn.pyx", line 66, in sage.interfaces.sagespawn.SageSpawn.__init__
        spawn.__init__(self, *args, **kwds)
      File "/Users/jpalmier/Desktop/Sage/sage_builds/TESTING/sage-9.5.beta6/src/.venv/lib/python3.9/site-packages/pexpect/pty_spawn.py", line 205, in __init__
        self._spawn(command, args, preexec_fn, dimensions)
      File "/Users/jpalmier/Desktop/Sage/sage_builds/TESTING/sage-9.5.beta6/src/.venv/lib/python3.9/site-packages/pexpect/pty_spawn.py", line 276, in _spawn
        raise ExceptionPexpect('The command was not found or was not ' +
    pexpect.exceptions.ExceptionPexpect: The command was not found or was not executable: gap.

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/Users/jpalmier/Desktop/Sage/sage_builds/TESTING/sage-9.5.beta6/src/sage/interfaces/gap.py", line 654, in _eval_line
        self._start()
      File "/Users/jpalmier/Desktop/Sage/sage_builds/TESTING/sage-9.5.beta6/src/sage/interfaces/gap.py", line 1166, in _start
        Expect._start(self, "Failed to start GAP.")
      File "/Users/jpalmier/Desktop/Sage/sage_builds/TESTING/sage-9.5.beta6/src/sage/interfaces/expect.py", line 503, in _start
        raise RuntimeError("unable to start %s because the command %r failed: %s\n%s" %
    RuntimeError: unable to start gap because the command 'gap -r -b -p -T -E -m 64m  /Users/jpalmier/Desktop/Sage/sage_builds/TESTING/sage-9.5.beta6/src/sage/ext_data/gap/sage.g' failed: The command was not found or was not executable: gap.

I haven't looked at all of the failures to see what else is there.

comment:485 Changed 12 months ago by Tobias Diez

Thanks for testing!

Some failing tests are to be expected since pipenv is rather different to the existing way to use sage (as part of sage-the-distribution). But I think for the modularization effort this ticket gives a helpful testcase how sage performs in a "pip-installable" environment.

I also had problems with gap (for the github action) and fixed it by manually setting the SAEG_LOCAL env variable. I guess there is potential for improvement as follow-up tickets but I'm not the right person to work on this as I'm not familiar with these parts of sage.

comment:486 Changed 11 months ago by Matthias Köppe

Milestone: sage-9.5sage-9.6

Stalled in needs_review or needs_info; likely won't make it into Sage 9.5.

comment:487 Changed 11 months ago by git

Commit: 0b0437fc7a2df0bb34873e6d0d881a775e1ebb09c736e0cbccc8ddb9cc8359a48094aa3181b881fa

Branch pushed to git repo; I updated commit sha1. New commits:

c736e0cMerge branch 'develop' of https://github.com/sagemath/sage into public/build/inplace

comment:488 Changed 8 months ago by Matthias Köppe

Milestone: sage-9.6sage-9.7

comment:489 Changed 2 months ago by Matthias Köppe

Milestone: sage-9.7sage-9.8
Note: See TracTickets for help on using tickets.