Opened 7 months ago

Last modified 2 weeks ago

#30371 needs_review enhancement

In-place (editable) installs of sagelib in a venv

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-9.3
Component: build Keywords: sd111
Cc: gh-tobiasdiez, fbissey, jhpalmieri, isuruf Merged in:
Authors: Tobias Diez Reviewers:
Report Upstream: N/A Work issues:
Branch: public/build/inplace (Commits, GitHub, GitLab) Commit: fec55234a00e46d023ad583f8c6284b970141ab0
Dependencies: #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029, #30912, #31357, #31362, #31365, #31377 Stopgaps:

Status badges

Description (last modified by mkoeppe)

We modify src/setup.py to support editable installs via pipenv (pipenv install -e ., which invokes pip install --editable). Since distribution continues using build/pkgs/sagelib/src/setup.py, the changes to src/setup.py made in this ticket do not affect the existing build process of the Sage library.

With this ticket, the following code works:

cd src
mkdir .venv
pipenv install
pipenv run sage-runtests --all

It generates a virtual env in src/.venv that contains all necessary libraries, and an editable install of sage. Note: This procedure might not work out of the box, and a few manual steps are necessary to install the required libraries (see below for instructions for Ubuntu 20.04 in WSL, and the github action for Ubuntu 20.10).

The output of the added GH workflow is available here: https://github.com/sagemath/sagetrac-mirror/actions?query=workflow%3A%22Build+%26+Test+using+pipenv%22+branch%3Apublic%2Fbuild%2Finplace Most tests are working, but for some reason after more or less exactly 2.5h of execution time the doctest run out of memory. This still needs investigation. #31341

TODO (as followup tickets):

  • Add documentation on how to set up the virtual environment, using either plain python -m venv, virtualenv (20.x), or pipenv. This is #31342.
  • Consider moving package dependencies from pipfile to install_requires in src/setup.py. Let's discuss this as part of #30913.
  • Consider not commit pipfile.lock but instead generate it as build artifact on ci (so that build errors can be easily reproduced by downloading the lock file, and re-creating the exact venv locally): https://github.com/pypa/pipenv/issues/1911#issuecomment-378989597. However, normally you want to put the lock file under version control for the reasons described in https://trac.sagemath.org/timeline?from=2021-02-03T13%3A15%3A22Z. The official statement is "always include in version control", see https://github.com/pypa/pipenv/issues/598.
  • Investigate what needs to be installed on macOS to have a successful build, and add these steps to the github action. (I don't have a mac, so I'll not work on this.) Then fix MacOS github action.

Previous tickets on in-place builds, editable installs:

Change History (380)

comment:1 Changed 7 months ago by gh-tobiasdiez

  • Branch set to public/build/inplace
  • Commit set to bc6e2726cb87273de0c92297156dccf4b5107f06

I've worked a bit on it. The current commit is a semi-working prototype, in the sense that all cython files are successfully compiled to C files if invoked via python src/setup.py build_ext --inplace. The second compilation of the c files does not yet work.

I had problems that none of the source cython files were correctly found on my system (wsl) and that a few dependencies in the current build files were not met ('linbox', 'Singular', 'm4ri', 'zlib', 'cblas'). For this reason, I started from scratch and only included what was needed so far. Any pointers to how to install these dependencies (I think they are needed for the second compilation).


New commits:

bc6e272Semi-working prototype for inplace compilation

comment:2 Changed 7 months ago by mkoeppe

I don't really have time to look into this before the 9.2 release. But if you are looking for where include directories are determined, take a look at sage_setup.command.sage_build_cython

comment:3 Changed 7 months ago by git

  • Commit changed from bc6e2726cb87273de0c92297156dccf4b5107f06 to 58452ce79c2d8863f139e72d75ec7b0c5904a989

Branch pushed to git repo; I updated commit sha1. New commits:

58452ceMake in-place compilation work

comment:4 Changed 7 months ago by gh-tobiasdiez

The command ./sage src/setup.py build_ext --inplace runs now fine for me. I had to change quite a few things to make it work, and found it easier to completely start over using cythonize instead of using the existing build_cython code. The resulting code is a minimal working version. Please let me know which features of the existing build_cython file you would like to keep (to be honest, I don't really understand what some of the code there is doing, and why its needed).

The code definitely needs cleanup, but otherwise is ready for a first round of review.

comment:5 Changed 7 months ago by gh-tobiasdiez

  • Status changed from new to needs_review

comment:6 Changed 7 months ago by git

  • Commit changed from 58452ce79c2d8863f139e72d75ec7b0c5904a989 to 1a99228dafea2b7b676e1754b3d4b43f794ec7a7

Branch pushed to git repo; I updated commit sha1. New commits:

1a99228Update gitignore

comment:7 Changed 7 months ago by mkoeppe

--- a/.gitignore
+++ b/.gitignore
@@ -99,7 +99,18 @@ gitlab-build-docker.log
 /src/build
 /src/Makefile
 /src/bin/sage-env-config
-/build/bin/sage-build-env-config
+/build

this is not good. Our /build/ contains important files of the sage distribution

comment:8 Changed 7 months ago by mkoeppe

And the changes removing old python2 code really need to go on a separate ticket

comment:9 Changed 7 months ago by git

  • Commit changed from 1a99228dafea2b7b676e1754b3d4b43f794ec7a7 to 22b27a051b7f1a429b7f91f965b33154ee776aea

Branch pushed to git repo; I updated commit sha1. New commits:

22b27a0Remove global gitignore of build folder

comment:10 follow-up: Changed 7 months ago by gh-tobiasdiez

Ok, I was under the assumption that the build folder is for build artifacts. If this would be the case, then I suppose it would be better to whitelist those files in the build folder that needs to be checked-in. I've now replaced the global build ignore, by a more fine-grained statement which only ignores the "temporary" files.

Concerning the changes about old Python2 code, I've created #30397.

comment:11 in reply to: ↑ 10 Changed 7 months ago by mkoeppe

Replying to gh-tobiasdiez:

I was under the assumption that the build folder is for build artifacts.

No, not at all.

comment:12 Changed 7 months ago by mkoeppe

The changes I see in setup.py don't seem to fit so well into the plan I have regarding modularization of sagelib. See #29705 and #28925 (which needs help).

comment:13 follow-up: Changed 7 months ago by gh-tobiasdiez

I wasn't aware of your changes in #28925. I guess what I'm proposing is a different strategy for the source discovery. The approach in #28925 is to change the source files themselves and enrich them by some form of metadata that decides in which distribution / package they land. Here I was proposing to specify the distribution of the files in the setup.py file. I guess both strategies have pros and cons. The main reason for me was that it seems the responsibility of the setup.py file to construct the distribution; why should the source file need to care about where they are used/included? I.e classical example of separation of concerns. I think, it's also the more flexible approach since it's easy to include a source file into multiple distributions etc. Moreover, you also gain a bit of performance at build time since you don't need to parse the files for some magic comments. What do you think?

comment:14 in reply to: ↑ 13 ; follow-up: Changed 7 months ago by mkoeppe

Replying to gh-tobiasdiez:

[...] Here I was proposing to specify the distribution of the files in the setup.py file. I guess both strategies have pros and cons. The main reason for me was that it seems the responsibility of the setup.py file to construct the distribution; why should the source file need to care about where they are used/included? I.e classical example of separation of concerns. [...]

One more thing to be aware of is that (1) setup.py by itself actually is not able to control what goes into the source distribution. In #29865 (where I prepare some experimental subset distributions), I additionally use MANIFEST.in to control that.

Separation of concerns is, of course, desirable. But so is locality: (2) My plan for splitting up the distribution is driven to a large extent by the dependencies on different sets of C/C++ libraries. And these are already encoded in Cython distutils directives.

Also I have explored a little bit the packaging options beyond using setuptools (in part motivated because of the shortcoming (1)). See #29854 (flit), #29810 (poetry). Unfortunately currently there does not seem to be a proper mainstream solution for the source layout that we want to keep (a monolithic source tree in src/sage). For example, flit (and even pip out-of-directory installs) do not handle symbolic links well. In the long term, I was hoping to transform sage_setup to implement the PEP 517 API directly (see #29845). That's another reason why I would be reluctant to hard-code distribution information in setup.py.

In any case, I am hoping to have a broader discussion about all of this on sage-devel at the beginning of the 9.3 cycle.

comment:15 in reply to: ↑ 14 ; follow-ups: Changed 7 months ago by gh-tobiasdiez

Separation of concerns is, of course, desirable. But so is locality: (2) My plan for splitting up the distribution is driven to a large extent by the dependencies on different sets of C/C++ libraries. And these are already encoded in Cython distutils directives.

One could also take this as an argument to remove the distutil dependency directives and put them into the setup.py, giving you locality and separation of concerns.

Anyway, my goal with this PR is not to redesign or modularize the build. I just wanted to have a working inplace build. For this moving the distribution declaration to setup.py was the easiest option, and I quite liked this also from a conceptual point of view. But if the current way is preferred, I change it back of course. Anything else that needs to be changed?

comment:16 in reply to: ↑ 15 Changed 7 months ago by mkoeppe

Replying to gh-tobiasdiez:

One could also take this as an argument to remove the distutil dependency directives and put them into the setup.py, giving you locality and separation of concerns.

Ha, no, we just moved them there. Getting rid of the messy file modules_list.py was an achievement in the present cycle. See #29706

comment:17 in reply to: ↑ 15 Changed 7 months ago by mkoeppe

Replying to gh-tobiasdiez:

Anything else that needs to be changed?

I haven't really had a chance to look at it in more detail yet, sorry. Soon...

comment:18 Changed 6 months ago by git

  • Commit changed from 22b27a051b7f1a429b7f91f965b33154ee776aea to 55d00d7f055628848bbdeef9075b8574c44316b2

Branch pushed to git repo; I updated commit sha1. New commits:

55d00d7Merge branch 'develop' of git://trac.sagemath.org/sage into public/build/inplace

comment:19 Changed 6 months ago by gh-tobiasdiez

  • Dependencies set to #30397

comment:20 Changed 6 months ago by git

  • Commit changed from 55d00d7f055628848bbdeef9075b8574c44316b2 to 40f35033ce241f8a6bf7fe2307a8c8b946c981e1

Branch pushed to git repo; I updated commit sha1. New commits:

8404a5dRemove old python 2 code from cython files
e9c06e3Readd constants in build script
811b78cRemove unused import
32daec5Merge branch 'public/build/removeOldCode' of git://trac.sagemath.org/sage into public/build/inplace
40f3503Fix merge problem

comment:21 Changed 6 months ago by git

  • Commit changed from 40f35033ce241f8a6bf7fe2307a8c8b946c981e1 to 851c557ccb33c61c78b005a20b908721deaf31a0

Branch pushed to git repo; I updated commit sha1. New commits:

851c557Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace

comment:22 Changed 6 months ago by git

  • Commit changed from 851c557ccb33c61c78b005a20b908721deaf31a0 to 1899dda1def19c3d7d518a5b4ccce67738415611

Branch pushed to git repo; I updated commit sha1. New commits:

1899ddaRemove unnessary changes

comment:23 Changed 6 months ago by git

  • Commit changed from 1899dda1def19c3d7d518a5b4ccce67738415611 to 74c9f7d2a328df4a2f436d3523daa12dddf655a6

Branch pushed to git repo; I updated commit sha1. New commits:

74c9f7dCleanup code

comment:24 Changed 6 months ago by gh-tobiasdiez

I've now changed the code to use the % sage_setup: distribution = xyz way. Moreover, I've cleanup up the code a bit.

Before I continue removing the code that is no longer needed, I would like to ask you which features of the current cythonize build script are missing from my new minimalistic implementation and should be reimplemented. (As I've said above, I don't really understand some of the code in build_cython.

comment:25 Changed 6 months ago by mkoeppe

These changes:

-# distutils: sources = sage/libs/ppl_shim.cc
+# distutils: sources = src/sage/libs/ppl_shim.cc

are not good. The root of the sagelib source tree is src, and paths are relative to this - this is where setup.py lives, after all.

As a way forward, I would suggest to actually work on #29864 instead of the present ticket. #29864 is a step in the modularization task (#29705), creating two new distributions - sage-core and sage-tdlib. sage-core is like sagelib minus sage-tdlib and without the fancy incremental installation tricks.

comment:26 Changed 6 months ago by mkoeppe

  • Status changed from needs_review to needs_work

comment:27 follow-up: Changed 6 months ago by gh-tobiasdiez

Thanks for the feedback. I'll try to find a way to specify the paths relative to src. I think my problem there was that setup.py is invoked from the central sage script, which sits in the root and not in the src folder.

Can you please explain the connection to #29864 in detail? For my the main point of this ticket is that I can compile the cython files to obtain compiled so files in the same folder as the original cython file (e.g. src/sage/rings). This is needed so that VS code can pick up these dependencies, resolve the imports, provide intellisense for it etc. My longterm goal would be to have a normal python installation (central or virtual env), point VS towards this runtime, and be able to use the sage code (in /src) without the need to use make at any point. How are #29864 and #29705 helping for this?

Last edited 6 months ago by gh-tobiasdiez (previous) (diff)

comment:28 in reply to: ↑ 27 Changed 6 months ago by mkoeppe

Replying to gh-tobiasdiez:

Can you please explain the connection to #29864 in detail?

Package sage-core created in #29864 will be a version of sagelib with a plain setuptools build system, intended for (modular) deployment of the Sage library on PyPI. Since setuptools supports setup.py develop, this has synergy with your goal.

And we will be able to work on the build system of sage-core, across several tickets, without having to worry about breaking sagelib builds - or what current users/developers are expecting of it.

comment:29 follow-up: Changed 6 months ago by gh-tobiasdiez

That make sense, thanks for the explanation.

The only thing that I still don't understand is how the modularization will help with the compilation of cython files. If if everything is modularized, I guess the setup.py will still be used for this. And the point of this ticket is to improve the cyhonization...

comment:30 in reply to: ↑ 29 Changed 6 months ago by mkoeppe

Replying to gh-tobiasdiez:

The only thing that I still don't understand is how the modularization will help with the compilation of cython files. If if everything is modularized, I guess the setup.py will still be used for this.

Simply by having a separate setup.py in build/pkgs/sage-core/src.

comment:31 Changed 6 months ago by mkoeppe

  • Dependencies changed from #30397 to #29500, #30527
  • Description modified (diff)
  • Summary changed from In-place installs with "./sage -pip install --editable src/" to In-place (editable) installs of sagelib in a venv

comment:32 Changed 6 months ago by mkoeppe

Slight change of plans, forget about sage-core.

comment:33 Changed 6 months ago by git

  • Commit changed from 74c9f7d2a328df4a2f436d3523daa12dddf655a6 to 5bda3321b88289cbd3f0dfe97308b07dd9dd1459

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

0a64674build/pkgs/gambit/spkg-install.in: Install via bdist_wheel
ca58693build/pkgs/pillow/spkg-install.in: Install via bdist_wheel
207d80fbuild/pkgs/pip2pi: New
2555e19build/make/install: At the end, update the repository index
5a747c4build/bin/sage-pip-{install,uninstall}: Remove pip2 support
e66243fMerge branch 't/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels' into t/30527/pep-503-simple-repository-for-wheels
9937126Merge branch 't/30527/pep-503-simple-repository-for-wheels' into t/30371/public/build/inplace
1a39442build/pkgs/sagelib/src/setup.py: Replace symlink to SAGE_SRC/setup.py by a copy
54fe0c7src/tox.ini, src/MANIFEST.in: Move here from build/pkgs/sagelib/src, leave symlinks behind
5bda332In-place (editable) installs of sagelib (#30371 squashed)

comment:34 follow-up: Changed 6 months ago by mkoeppe

TODO:

  • The changes to src/sage/cpython/debugimpl.c need to go on a separate ticket
  • The changes to src/sage/misc/persist.pyx need to go on a separate ticket
  • Don't change find_python_sources because build/pkgs/sagelib/src/setup.py continues to use it

comment:35 Changed 6 months ago by mkoeppe

  • Description modified (diff)

comment:36 Changed 6 months ago by mkoeppe

  • Description modified (diff)

comment:37 Changed 6 months ago by mkoeppe

  • Dependencies changed from #29500, #30527 to #29500, #30527, #22731

comment:38 Changed 6 months ago by mkoeppe

  • Description modified (diff)

comment:39 Changed 6 months ago by git

  • Commit changed from 5bda3321b88289cbd3f0dfe97308b07dd9dd1459 to 274538190984ea38145ffdb4d7d1defc88784124

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

67c56a6src/requirements.txt, src/MANIFEST.in: Move here from build/pkgs/sagelib/src, leave symlinks behind
2745381In-place (editable) installs of sagelib (#30371 squashed)

comment:40 Changed 6 months ago by mkoeppe

  • Dependencies changed from #29500, #30527, #22731 to #29500, #30527, #22731, #30578
  • Description modified (diff)

comment:41 in reply to: ↑ 34 Changed 6 months ago by mkoeppe

Replying to mkoeppe:

TODO:

  • The changes to src/sage/cpython/debugimpl.c need to go on a separate ticket

This is now #30590

comment:42 Changed 5 months ago by git

  • Commit changed from 274538190984ea38145ffdb4d7d1defc88784124 to a493e6e2a972ab5344fdd3a953376278e807cf3b

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

851c557Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace
1899ddaRemove unnessary changes
74c9f7dCleanup code
81fdcf3Make distutils relative to src again
faf6777Fail gracefully for missing packages
8af6e23Add minimal pipfile
456589cIgnore build files
59e8361Make it compile by ignoring a lot of errors
054fedcAdd psutil and pexpect
a493e6eMerge branch 'public/build/inplace' of git://trac.sagemath.org/sage into public/build/inplace

comment:43 Changed 5 months ago by git

  • Commit changed from a493e6e2a972ab5344fdd3a953376278e807cf3b to c5f3873cc7ddb158646051861fc10c72c6be22f5

Branch pushed to git repo; I updated commit sha1. New commits:

c5f3873Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace

comment:44 follow-ups: Changed 5 months ago by gh-tobiasdiez

I have now added a minimal pipfile including sage as an editable install. So running pipenv install results in a semi-workable virtual environment. "Semi" because not all pyx files could be compiled successfully since they rely on external dependencies (e.g. singular, giac, ecl, ratpoints, ...).

For now, I've excluded the relevant pyx files from the compilation. Any ideas how to handle these dependencies are welcome! One approach would be to treat them similar to the the optional packages by putting a "sage_setup distribution" tag to the pyx file.

I also tried to run pipenv install directly under Windows, but that's not possible at the moment as some of the dependencies cannot be installed on Windows currently: https://github.com/sagemath/cypari2/issues/19 https://github.com/sagemath/cysignals/pull/104

Last edited 5 months ago by gh-tobiasdiez (previous) (diff)

comment:45 Changed 5 months ago by git

  • Commit changed from c5f3873cc7ddb158646051861fc10c72c6be22f5 to b7fb5a9d46d8acb5068cde9a94009adb800d4783

Branch pushed to git repo; I updated commit sha1. New commits:

8a7ec88Revert format changes in debug.pyx
ba58ffaMake distutil imports relative to src (again?)
b7fb5a9Incorporate upstream changes

comment:46 in reply to: ↑ 44 Changed 5 months ago by mkoeppe

Replying to gh-tobiasdiez:

[...] not all pyx files could be compiled successfully since they rely on external dependencies (e.g. singular, giac, ecl, ratpoints, ...). One approach would be to treat them similar to the the optional packages by putting a "sage_setup distribution" tag to the pyx file.

Yes, exactly this approach is part of the "modularization" that I have in mind. See #29705 under the items titled "Deploy mildly modularized distributions", "Further modularization" for a sketch of such distributions. Basically, one distribution for each major C/C++ library (such as sage-ntl) - but some libraries come in packs that would not make much sense to separate (such as sage-flint-arb-e_antic).

comment:47 in reply to: ↑ 44 Changed 5 months ago by mkoeppe

Replying to gh-tobiasdiez:

I also tried to run pipenv install directly under Windows, but that's not possible at the moment as some of the dependencies cannot be installed on Windows currently: https://github.com/sagemath/cypari2/issues/19 https://github.com/sagemath/cysignals/pull/104

Right, I hope that when modularization is done, a native Windows port of a tiny Sage subset would be an actionable project.

Last edited 5 months ago by mkoeppe (previous) (diff)

comment:48 Changed 5 months ago by mkoeppe

As in #30578 (where I am trying the same with requirements.txt and plain venv), I would like to get the following to work:

  1. Build the sage distribution (actually all prerequisites of sagelib, using make sagelib-build-deps)
  1. Then within sage -sh (where all the libraries installed in $SAGE_LOCAL are available) do the in-place install into a new venv created by pipenv - using the wheels built in $SAGE_LOCAL/var/lib/sage/wheels.
Last edited 5 months ago by mkoeppe (previous) (diff)

comment:49 follow-ups: Changed 5 months ago by gh-tobiasdiez

Thanks for the quick feedback.

I agree the modularization is a good way to approach these problems. Sadly, I feel like this requires too much in-depth knowledge of sage for me to contribute directly. But after skimming the tickets, I suggest you have a look at my changes here to the way distributions are excluded. This seems to be somewhat opposite to say #29864 where this is handled by adding a new setup-tdlib/setup.py. For the in-place installation it is important that the src/setup.py results in a working installation.

What do you suggest as the way forward to align this ticket with the modularization effort?

As for #30578, my motivation here is a bit different: I wanted to have a prototype for sagelib with pipenv that works without ever invoking sage/make and directly uses the versions of the dependencies found on pypi instead of (re)building them locally. This seems to work in principle, and might lead to a workable solution for most purposes of development and ci testing, where the advantages of building the dependencies (like optimization) are not that important. That being said, I'm not sure if we really need to have a requirements.txt and a pipfile. I would opt for the more modern version and only keep the pipfile.

After installing singular via apt-get install libsingular4-dev libpynac-dev, a few more pyx files could be compiled. Should I add this in build\pkgs\singular\distros\debian ?

One more thing: Currently sage/rings/number_field/number_field_element_quadratic.pyx relies on the arb lib, but is also (indirectly) imported in sage.all. What's the best way to handle these imports that depend on external C/C++ libs? For now I've uncommented some of these all-imports.

comment:50 Changed 5 months ago by git

  • Commit changed from b7fb5a9d46d8acb5068cde9a94009adb800d4783 to 12e6d6711ca76bee221336093fa03e162962a896

Branch pushed to git repo; I updated commit sha1. New commits:

cc2d6ffUncomment non-working all imports
9d9bf05Reenable pyx compilation depending on singular
12e6d67Fix imports

comment:51 Changed 5 months ago by gh-tobiasdiez

Yet another question: when/how is src/sage/ext/interpreters generated?

comment:52 Changed 5 months ago by git

  • Commit changed from 12e6d6711ca76bee221336093fa03e162962a896 to 7a7e5764d619f0207c9d57d1c24b6198f944b799

Branch pushed to git repo; I updated commit sha1. New commits:

7a7e576Make import sage.manifolds.all work

comment:53 in reply to: ↑ 49 Changed 5 months ago by mkoeppe

Replying to gh-tobiasdiez:

As for #30578, my motivation here is a bit different: I wanted to have a prototype for sagelib with pipenv that works without ever invoking sage/make and directly uses the versions of the dependencies found on pypi instead of (re)building them locally.

Yes, I agree with this goal. But I would strongly suggest to focus on a smaller scope for this ticket: Make the in-place install work when all the dependencies are available. At the same time, make sure that the normal Sage build process does not get broken (e.g. take care of comment 34 "Don't change find_python_sources because build/pkgs/sagelib/src/setup.py continues to use it"). This ensures that the ticket can be merged in Sage, thus, a way forward. This is better than preparing one ticket with huge changes.

comment:54 in reply to: ↑ 49 ; follow-up: Changed 5 months ago by mkoeppe

Replying to gh-tobiasdiez:

I suggest you have a look at my changes here to the way distributions are excluded. This seems to be somewhat opposite to say #29864 where this is handled by adding a new setup-tdlib/setup.py.

As discussed, adding these sage_setup: distribution tags to Cython source files is a way forward that aligns with the modularization effort.

comment:55 in reply to: ↑ 49 ; follow-up: Changed 5 months ago by mkoeppe

Replying to gh-tobiasdiez:

Currently sage/rings/number_field/number_field_element_quadratic.pyx relies on the arb lib, but is also (indirectly) imported in sage.all. What's the best way to handle these imports that depend on external C/C++ libs?

Try lazy imports with PythonModule features, from #30616/#30647.

comment:56 in reply to: ↑ 49 Changed 5 months ago by mkoeppe

Replying to gh-tobiasdiez:

I suggest you have a look at my changes here to the way distributions are excluded.

Well, it's the same as before in sagelib's setup.py, filtering by distributions, plus some hardcoded ones. Except that you have changed the interface of the function that does the filtering (see above - this change is not mergeable because build/pkgs/sagelib/src/setup.py needs to keep working!)

comment:57 in reply to: ↑ 54 Changed 5 months ago by mkoeppe

Replying to mkoeppe:

As discussed, adding these sage_setup: distribution tags to Cython source files is a way forward that aligns with the modularization effort.

I've opened #30666 for this. (The new distribution names have to be added to setup.py, of course, so that the files continue to be built in the Sage distribution.)

comment:58 Changed 5 months ago by mkoeppe

  • Dependencies changed from #29500, #30527, #22731, #30578 to #29500, #30527, #22731, #30578, #30666

comment:59 in reply to: ↑ 55 Changed 5 months ago by mkoeppe

Replying to mkoeppe:

Replying to gh-tobiasdiez:

Currently sage/rings/number_field/number_field_element_quadratic.pyx relies on the arb lib, but is also (indirectly) imported in sage.all. What's the best way to handle these imports that depend on external C/C++ libs?

Try lazy imports with PythonModule features, from #30616/#30647.

For another approach regarding the ".all" imports, see #29865. Probably we will need a combination of this and lazy imports.

comment:60 Changed 5 months ago by git

  • Commit changed from 7a7e5764d619f0207c9d57d1c24b6198f944b799 to 4afd411dd86c37e90ca32c8f77de89a91c57a667

Branch pushed to git repo; I updated commit sha1. New commits:

4afd411Readd old find_python_sources

comment:61 Changed 5 months ago by gh-tobiasdiez

I've now restored the old find_python_sources method. Is there something else that needs to be done, except waiting for the other tickets to be finished?

comment:62 Changed 5 months ago by gh-tobiasdiez

  • Description modified (diff)

comment:63 Changed 5 months ago by mkoeppe

Nothing immediate. This is going in a good direction

comment:64 follow-up: Changed 5 months ago by git

  • Commit changed from 4afd411dd86c37e90ca32c8f77de89a91c57a667 to c82bbd56e5ab828b73dfe09ad1c32c9b7d96f866

Branch pushed to git repo; I updated commit sha1. New commits:

849350cFix missing ccobject
ee8df98Update gitignore and add vs code config
c82bbd5Add pycodestyle dep

comment:65 in reply to: ↑ 64 Changed 5 months ago by mkoeppe

Replying to git:

Branch pushed to git repo; I updated commit sha1. New commits:

849350cFix missing ccobject

Created #30672 (Remove sage/ext from sage_include_directories) for this

comment:66 Changed 5 months ago by mkoeppe

  • Dependencies changed from #29500, #30527, #22731, #30578, #30666 to #29500, #30527, #22731, #30578, #30666, #30672

comment:67 Changed 5 months ago by gh-tobiasdiez

  • Dependencies changed from #29500, #30527, #22731, #30578, #30666, #30672 to #29500, #30527, #22731, #30578, #30666, #30672, #30673

comment:68 Changed 5 months ago by gh-tobiasdiez

  • Description modified (diff)

comment:69 Changed 5 months ago by git

  • Commit changed from c82bbd56e5ab828b73dfe09ad1c32c9b7d96f866 to 62930ee7ada36664e0fc2264e5395f8b8337b650

Branch pushed to git repo; I updated commit sha1. New commits:

62930eeAdd ipykernel and tox as dev packages

comment:70 Changed 5 months ago by gh-tobiasdiez

  • Description modified (diff)

New commits:

62930eeAdd ipykernel and tox as dev packages

comment:71 Changed 5 months ago by gh-tobiasdiez

  • Dependencies changed from #29500, #30527, #22731, #30578, #30666, #30672, #30673 to #29500, #30527, #22731, #30578, #30672, #30673

comment:72 Changed 5 months ago by git

  • Commit changed from 62930ee7ada36664e0fc2264e5395f8b8337b650 to cd196a0773e36e7e6c1fbe21962ef5b1b1485cf5

Branch pushed to git repo; I updated commit sha1. New commits:

3cbb457Reenable compilation of most files relying on external libs
8174e1aReenable imports
724223fDisable non-working imports
cd196a0Disable things in ecl integration (probably because of too old ecl lib)

comment:73 Changed 5 months ago by gh-tobiasdiez

  • Description modified (diff)

comment:74 Changed 5 months ago by mkoeppe

This ticket is way too big. I really strong recommend to get a working in-place installation of sagelib in the situation when its dependencies have been built using the Sage distribution.

comment:75 follow-up: Changed 5 months ago by gh-tobiasdiez

I've now followed your advice, and tried to have a prototype that works if all library dependencies are fulfilled (instead of adding exceptions / distribution annotations everywhere). And I was successful indeed, import sage.all works without error and I can do basic calculations with the manifold package. This required tinkering with a lot of system-installed packages (instructions added in the ticket).

The following is not yet working, and I would like to get input on how to resolve this:

  • sage/ext/interpreters is empty. Where is the code generated? Can this easily be extracted to a script that can be invoked separately?
  • The code relying on the arb library still doesn't compile although I've libflint-arb-dev installed. The compilation fails with ld larb was not found. Why is this, and what's the best way to solve it?
  • The sage.doctest package conflicted with the sympy (I believe), i.e. sympy tried to load classes from sage.doctest for some reason, and couldn't locate them. I've fixed this by renaming src/sage/doctest to sage-doctest. Is the doctest package really required under the sage folder, or could it be e.g moved up to src/doctest?

I agree the ticket is too big at the moment. Once I found a solution to the above points, I'll clean the code. I sadly cannot work with the dependencies created by the sage distribution, as the build process fails due to compilation errors in the pip library...

comment:76 Changed 5 months ago by git

  • Commit changed from cd196a0773e36e7e6c1fbe21962ef5b1b1485cf5 to 1422cf461178919c2ccf1a30b6a82e1ac5dee396

Branch pushed to git repo; I updated commit sha1. New commits:

1422cf4Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace

comment:77 in reply to: ↑ 75 Changed 5 months ago by mkoeppe

Replying to gh-tobiasdiez:

I've now followed your advice, and tried to have a prototype that works if all library dependencies are fulfilled (instead of adding exceptions / distribution annotations everywhere). And I was successful indeed, import sage.all works without error and I can do basic calculations with the manifold package. This required tinkering with a lot of system-installed packages (instructions added in the ticket).

Great.

The following is not yet working, and I would like to get input on how to resolve this:

  • sage/ext/interpreters is empty. Where is the code generated? Can this easily be extracted to a script that can be invoked separately?

sage_setup.command.sage_build

  • The code relying on the arb library still doesn't compile although I've libflint-arb-dev installed. The compilation fails with ld larb was not found. Why is this, and what's the best way to solve it?

Use sage.env.cython_aliases. In particular ARB_LIBRARY gives you the name. (The Debian installation uses a different name from upstream.)

  • The sage.doctest package conflicted with the sympy (I believe), i.e. sympy tried to load classes from sage.doctest for some reason, and couldn't locate them. I've fixed this by renaming src/sage/doctest to sage-doctest. Is the doctest package really required under the sage folder, or could it be e.g moved up to src/doctest?

Sounds like a bad PYTHONPATH. src/sage should not be in it.

I agree the ticket is too big at the moment. Once I found a solution to the above points, I'll clean the code. I sadly cannot work with the dependencies created by the sage distribution, as the build process fails due to compilation errors in the pip library...

Hm, that's still happening? Sorry I lost track of this since various other things seemed to break on WSL. Have you tried with downgrading/upgrading the pip package?

comment:78 Changed 5 months ago by git

  • Commit changed from 1422cf461178919c2ccf1a30b6a82e1ac5dee396 to 05713b87fff7948a68562358a5d32a29701bfa2f

Branch pushed to git repo; I updated commit sha1. New commits:

05713b8Remove dependency on #30527

comment:79 Changed 5 months ago by mkoeppe

By the way, for dev purposes until the WSL issues are resolved, you could try running Sage in Docker instead (bind-mount the source tree into it)

comment:80 Changed 5 months ago by gh-tobiasdiez

  • Dependencies changed from #29500, #30527, #22731, #30578, #30672, #30673 to #29500, #22731, #30672, #30673

New commits:

05713b8Remove dependency on #30527

New commits:

05713b8Remove dependency on #30527

comment:81 Changed 5 months ago by gh-tobiasdiez

  • Dependencies changed from #29500, #22731, #30672, #30673 to #29500, #30672, #30673

comment:82 Changed 5 months ago by git

  • Commit changed from 05713b87fff7948a68562358a5d32a29701bfa2f to ac07cc0d005de5e067e29c22f154521537f5906d

Branch pushed to git repo; I updated commit sha1. New commits:

ac07cc0Move requirements file back

comment:83 in reply to: ↑ description Changed 5 months ago by mkoeppe

  • Fix installation of ecl:

After apt-get install ecl ecl is broken under WSL. To fix this, first run apt-get install libtool autoconf, then follow https://github.com/rdp/ffmpeg-windows-build-helpers/issues/452#issuecomment-638639182.

Could you try the related #30629 on WSL please?

comment:84 Changed 5 months ago by gh-tobiasdiez

  • Dependencies changed from #29500, #30672, #30673 to #29500, #30672, #30673, #30590
  • Description modified (diff)

comment:85 Changed 5 months ago by git

  • Commit changed from ac07cc0d005de5e067e29c22f154521537f5906d to 3023260e2c36e80113daf0af69e95d6091de461e

Branch pushed to git repo; I updated commit sha1. New commits:

9141ae9Reenable doctests
cb50197Fix compilation of interpreters
3023260Renable all imports

comment:86 Changed 5 months ago by git

  • Commit changed from 3023260e2c36e80113daf0af69e95d6091de461e to 5b12a924507cf85a2eca749993835927b9398668

Branch pushed to git repo; I updated commit sha1. New commits:

e620323Remove unnessary files
6502495Reenable ppl
5b12a92Cleanup setup script

comment:87 Changed 5 months ago by gh-tobiasdiez

  • Description modified (diff)
  • Status changed from needs_work to needs_review

comment:88 Changed 5 months ago by gh-tobiasdiez

Perfect, thanks a lot Matthias! I got now a working installation of all libraries, and could import sage.all without the need to uncomment any imports. Yeaaaah!

I've also cleaned the code and extended the ticket description, so this ticket is now ready for review.

comment:89 Changed 5 months ago by mkoeppe

Could you put the changes to sage_setup/autogen/interpreters/ on a separate ticket please?

comment:90 Changed 5 months ago by mkoeppe

The changes regarding imports (mostly around sage.rings) should also be on a separate ticket.

comment:91 Changed 5 months ago by mkoeppe

Also for the changes to src/sage/libs/eclsig.h -- which will certainly need discussion

comment:92 follow-up: Changed 5 months ago by mkoeppe

+compile_time_env = dict(
+    PY_PLATFORM=sys.platform
+)
+cython_directives = dict(
+    language_level="3str",
+    cdivision=True,
+)

This duplicates settings defined in sage_setup.command.sage_build_cython. Could you refactor this so that these directives are defined in a reusable way? Perhaps in a new module sage_setup.cython_options

Last edited 5 months ago by mkoeppe (previous) (diff)

comment:93 Changed 5 months ago by mkoeppe

  • Authors set to Tobias Diez

comment:94 in reply to: ↑ 92 Changed 5 months ago by mkoeppe

Replying to mkoeppe:

Could you refactor this so that these directives are defined in a reusable way? Perhaps in a new module sage_setup.cython_options

... in a separate ticket of course.

comment:95 follow-up: Changed 5 months ago by gh-tobiasdiez

Will do so. For the cython options, they are actually only a small subset of the ones in sage_build_cython. Should I still refactor? (Feels a bit strange as there are more differences than similarities.) But I'm also not sure if the other cython options are needed...

comment:96 in reply to: ↑ 95 Changed 5 months ago by mkoeppe

Replying to gh-tobiasdiez:

Will do so. For the cython options, they are actually only a small subset of the ones in sage_build_cython. Should I still refactor? (Feels a bit strange as there are more differences than similarities.) But I'm also not sure if the other cython options are needed...

Well, the idea would be to see whether you can just build with the options that Sage uses instead of using your own set of options.

comment:97 Changed 5 months ago by mkoeppe

Whether other Cython build options can/should be used - should be orthogonal to whether an in-place or regular build is done. So it should not be done on this ticket.

comment:98 in reply to: ↑ description Changed 5 months ago by mkoeppe

Some comments on items in the ticket description:

TODO (as followup tickets):

  • Automatically generate the apt-get install command with the necessary libraries (add them as distros/debian.txt?)

./configure takes care of all of this. Do you plan to do system interface configuration and system package advice without using ./configure (or, more generally, Unix shell scripts)?

A way forward is to reimplement (cleaner version of) some of the simple shell code for system package advice as Python code in sage_bootstrap. See #29146.

  • Remove ARB_LIBRARY env variable, and move check from arb/config.m4 to env.py

My suggestion: Instead of discussing to remove this variable (or other configuration variables that are set by ./configure), rather discuss different ways of generating the sage_conf module / or providing different implementations of the interface defined by the sage_conf module for different use cases (such as supporting a platform that does not have a Unix shell).

  • Move package dependencies from pipfile to install_requires in src/setup.py.

Best to put them in src/setup.cfg instead of src/setup.py (see #30578)

comment:99 Changed 5 months ago by git

  • Commit changed from 5b12a924507cf85a2eca749993835927b9398668 to 8abab5dfb588a3b6e6fe0c1d32525656ddc13cf6

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

82e8b7dNarrow down to essenital changes
8abab5dAdd copy of setup.py

comment:100 Changed 5 months ago by mkoeppe

Thanks for preparing this cleaned up branch.

Trying to test it, I seem to be running into something that looks like https://discuss.python.org/t/pip-19-1-and-installing-in-editable-mode-with-pyproject-toml/1553 : Something is generating a pyproject.toml in src/ and then it dies with

Writing supplied requirement line to temporary file: '-e .'
Installing 'e1839a8'
$ ['/Users/mkoeppe/.local/share/virtualenvs/src-BXov0SKG/bin/pip', 'install', '--verbose', '--upgrade', '--no-use-pep517', '--no-deps', '--exists-action=i', '-r', '/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pipenv-_yopr9l3-requirements/pipenv-7uv8tb0m-requirement.txt', '-i', 'https://pypi.org/simple']
Using source directory: '/Users/mkoeppe/.local/share/virtualenvs/src-BXov0SKG/src'
Using pip 20.2.1 from /Users/mkoeppe/.local/share/virtualenvs/src-BXov0SKG/lib/python3.8/site-packages/pip (python 3.8)
Non-user install by explicit request
...
  File "/Users/mkoeppe/.local/share/virtualenvs/src-BXov0SKG/lib/python3.8/site-packages/pip/_internal/distributions/sdist.py", line 33, in prepare_distribution_metadata
    self.req.load_pyproject_toml()
  File "/Users/mkoeppe/.local/share/virtualenvs/src-BXov0SKG/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 503, in load_pyproject_toml
    pyproject_toml_data = load_pyproject_toml(
  File "/Users/mkoeppe/.local/share/virtualenvs/src-BXov0SKG/lib/python3.8/site-packages/pip/_internal/pyproject.py", line 95, in load_pyproject_toml
    raise InstallationError(
pip._internal.exceptions.InstallationError: Disabling PEP 517 processing is invalid: project specifies a build backend of setuptools.build_meta in pyproject.toml
Removed file:///Users/mkoeppe/s/sage/sage-rebasing/src (from -r /var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pipenv-_yopr9l3-requirements/pipenv-7uv8tb0m-requirement.txt (line 1)) from build tracker '/private/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pip-req-tracker-lxrv7bcd'
Removed build tracker: '/private/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pip-req-tracker-lxrv7bcd'

comment:101 Changed 5 months ago by gh-tobiasdiez

Strange. For some reason, for you pip is invoked with --no-use-pep517. For me it is not:

$ "/mnt/d/Programming/Projects/sage/src/.venv/bin/pip" install   --verbose  --no-deps  -e "." -i https://pypi.org/simple --exists-action w...

What version of pipenv do you use? For me: pipenv 11.9.0 (and the pip in the virtual env also differs: pip 20.0.2)

comment:102 Changed 5 months ago by mkoeppe

I'm using pipenv from homebrew, which is version 2020.8.13

comment:103 Changed 5 months ago by gh-tobiasdiez

I will try to update my pipenv later. For now you can have a look if setting the environment variable PIP_USE_PEP517 to true resolves the issue.

comment:104 Changed 5 months ago by mkoeppe

I'll try, but I think editable installs are not compatible with PEP517

comment:105 Changed 5 months ago by mkoeppe

No change after setting this environment variable

comment:106 Changed 5 months ago by gh-tobiasdiez

  • Dependencies changed from #29500, #30672, #30673, #30590 to #29500, #30672, #30673, #30590, #30709

comment:107 Changed 5 months ago by gh-tobiasdiez

I had a closer look at this, and it turns out that the first time pip is correctly called without any --no-use-pep517 argument, only on the retry this argument is used. Thus, the error actually shadows another error that occurs during the initial pip install.

Can you thus run the following pipenv run pip install -e . --no-build-isolation after pipenv install. This should point to the real issue why the editable install fails.

I've created a github workflow because I don't have a mac. https://github.com/tobiasdiez/sage/blob/public/build/inplace/.github/workflows/ci-pipenv.yml It shows the same problem, the pipenv install fails apparently due to a pep517 issue, but the editable install actually fails due to missing interpreters and other dependencies (which requires manual installations as in the ticket description).

comment:108 Changed 5 months ago by mkoeppe

Hm...

(sage-sh) mkoeppe@egret:src$ pipenv run pip install -e . --no-build-isolation
Obtaining file:///Users/mkoeppe/s/sage/sage-rebasing/src
    Preparing wheel metadata ... error
    ERROR: Command errored out with exit status 1:
     command: /Users/mkoeppe/.local/share/virtualenvs/src-BXov0SKG/bin/python /Users/mkoeppe/.local/share/virtualenvs/src-BXov0SKG/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/tmpvt150u93
         cwd: /Users/mkoeppe/s/sage/sage-rebasing/src
    Complete output (14 lines):
    Traceback (most recent call last):
      File "/Users/mkoeppe/.local/share/virtualenvs/src-BXov0SKG/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
        main()
      File "/Users/mkoeppe/.local/share/virtualenvs/src-BXov0SKG/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/Users/mkoeppe/.local/share/virtualenvs/src-BXov0SKG/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 133, in prepare_metadata_for_build_wheel
        return hook(metadata_directory, config_settings)
      File "/Users/mkoeppe/.local/share/virtualenvs/src-BXov0SKG/lib/python3.8/site-packages/setuptools/build_meta.py", line 161, in prepare_metadata_for_build_wheel
        self.run_setup()
      File "/Users/mkoeppe/.local/share/virtualenvs/src-BXov0SKG/lib/python3.8/site-packages/setuptools/build_meta.py", line 145, in run_setup
        exec(compile(code, __file__, 'exec'), locals())
      File "setup.py", line 23, in <module>
        import sage.env
    ModuleNotFoundError: No module named 'sage'
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/mkoeppe/.local/share/virtualenvs/src-BXov0SKG/bin/python /Users/mkoeppe/.local/share/virtualenvs/src-BXov0SKG/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/tmpvt150u93 Check the logs for full command output.

comment:109 Changed 5 months ago by gh-tobiasdiez

No idea....for the github action this is not a problem: https://github.com/tobiasdiez/sage/runs/1205452529?check_suite_focus=true. Maybe pipenv run pip install -e . --verbose --upgrade --exists-action=i --no-build-isolation works, or you need to clean the venv first (pipenv -rm).

comment:110 Changed 5 months ago by mkoeppe

Sure, I'll play with it more in the next days.

comment:111 Changed 5 months ago by gh-tobiasdiez

  • Description modified (diff)

comment:112 Changed 5 months ago by git

  • Commit changed from 8abab5dfb588a3b6e6fe0c1d32525656ddc13cf6 to edde293052d7e4660df11ce13513f08edf44c5e5

Branch pushed to git repo; I updated commit sha1. New commits:

edde293Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace

comment:113 Changed 5 months ago by git

  • Commit changed from edde293052d7e4660df11ce13513f08edf44c5e5 to 74ecddbe6a71f133458cdb018f45e36ba0aaf818

Branch pushed to git repo; I updated commit sha1. New commits:

74ecddbUpdate dependencies

comment:114 Changed 5 months ago by git

  • Commit changed from 74ecddbe6a71f133458cdb018f45e36ba0aaf818 to 7110841f2c816001c14205c1426088fb873499b3

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

4f56517Duplicate src/setup.py
18a091bNarrow down to essenital changes
7110841Update dependencies

comment:115 Changed 5 months ago by mkoeppe

  • Dependencies changed from #29500, #30672, #30673, #30590, #30709 to #30779, #30672, #30673, #30709

comment:116 Changed 5 months ago by git

  • Commit changed from 7110841f2c816001c14205c1426088fb873499b3 to 0b5403699f9472f102b4781e3fd9ebae80e6bb8a

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

52e2107sage_include_directories: Remove sage/ext
6ccce6bFix missing ccobject
1402129Merge branch 'u/mkoeppe/remove__sage_ext__from__sage_include_directories_' of git://trac.sagemath.org/sage into t/30371/public/build/inplace
6eb3845Extend gitignore and add VS code config
bbe73eeRemove VS code settings, and add exceptions for already checked-in files
22b7cacMerge branch 'public/build/enhancegitignore' of git://trac.sagemath.org/sage into t/30371/public/build/inplace
1aef376Extract cython options to new module
68ab47bMerge branch 'public/build/cythonConfig' of git://trac.sagemath.org/sage into t/30371/public/build/inplace
d1b1288Narrow down to essenital changes
0b54036Update dependencies

comment:117 Changed 5 months ago by mkoeppe

Rebased on top of the dependencies

comment:118 Changed 5 months ago by mkoeppe

  • Status changed from needs_review to needs_work
  • Work issues set to Use the cython options

comment:119 Changed 5 months ago by git

  • Commit changed from 0b5403699f9472f102b4781e3fd9ebae80e6bb8a to 55930254f941314d9996bfc6a9a4d5f9767c2e6d

Branch pushed to git repo; I updated commit sha1. New commits:

82e8b7dNarrow down to essenital changes
8abab5dAdd copy of setup.py
edde293Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace
74ecddbUpdate dependencies
1286fcbMerge branch 'public/build/inplace' of git://trac.sagemath.org/sage into public/build/inplace
946737eUse new cython option modules
5593025Update dependencies

comment:120 Changed 5 months ago by gh-tobiasdiez

  • Status changed from needs_work to needs_review
  • Work issues Use the cython options deleted

Thanks for the rebase. Next time please try to not force-push to a branch that others have checked out because it messes up their git history.

I've updated the code to use the new cython options module.

comment:121 Changed 5 months ago by mkoeppe

Sorry if the push made it harder for you to continue on this branch. But rebasing it on top of the branches of the dependency tickets was a necessary cleanup

comment:122 Changed 5 months ago by gh-tobiasdiez

  • Description modified (diff)

comment:123 Changed 5 months ago by mkoeppe

What's the status of this branch? Does it work for you? Can you run doctests in the in-place installation and does it pass them?

comment:124 follow-up: Changed 5 months ago by gh-tobiasdiez

It's working for me. pipenv install -dev gives a fully functional virtual environment (so the cython compilation works). Moreover, import sage.all works as well, and one can then use it to run sage-code. So far, I've tested it with some code from the manifolds package. Not yet tried doctests, as I've no idea how to run them from the virtual environment.

comment:125 in reply to: ↑ 124 Changed 5 months ago by mkoeppe

Replying to gh-tobiasdiez:

Not yet tried doctests, as I've no idea how to run them from the virtual environment.

Well, the virtual environment has a bin/sage script, so sage -t "should" work. If it does not, this needs to be fixed. Likely some work in the direction of #22731 is necessary.

comment:126 follow-up: Changed 5 months ago by gh-tobiasdiez

Yeah, sage -t still uses the python in sage/local. So this does not work, and a mechanism is needed to use the python of the activated virtual env (without changing env variables which is too inflexible in my opinion).

Directly running bin/sage-runtests --all also leads to errors:

no stored timings available
Running doctests with ID 2020-10-20-21-18-29-b0a05b3d.
Using --optional=dochtml,memlimit,sage,sage_spkg
Doctesting entire Sage library.
Traceback (most recent call last):
  File "bin/sage-runtests", line 182, in <module>
    err = DC.run()
  File "/mnt/d/Programming/Projects/sage/src/sage/doctest/control.py", line 1234, in run
    self.expand_files_into_sources()
  File "/mnt/d/Programming/Projects/sage/src/sage/doctest/control.py", line 817, in expand_files_into_sources
    self.sources = [FileDocTestSource(path, self.options) for path in expand()]
  File "/mnt/d/Programming/Projects/sage/src/sage/doctest/control.py", line 817, in <listcomp>
    self.sources = [FileDocTestSource(path, self.options) for path in expand()]
  File "/mnt/d/Programming/Projects/sage/src/sage/doctest/sources.py", line 528, in __init__
    raise ValueError("unknown file extension %r"%ext)
ValueError: unknown file extension ''

(I guess some files are not found.)

Anyway, that's probably a topic for another ticket.

comment:127 in reply to: ↑ 126 Changed 5 months ago by mkoeppe

Replying to gh-tobiasdiez:

Yeah, sage -t still uses the python in sage/local. So this does not work, and a mechanism is needed to use the python of the activated virtual env (without changing env variables which is too inflexible in my opinion).

Yes, this is precisely #22731

comment:128 Changed 5 months ago by git

  • Commit changed from 55930254f941314d9996bfc6a9a4d5f9767c2e6d to 9dc8a2b0bcd6ff1cada426c60184b322312618e4

Branch pushed to git repo; I updated commit sha1. New commits:

9dc8a2bGenerate code automatically

comment:129 Changed 5 months ago by git

  • Commit changed from 9dc8a2b0bcd6ff1cada426c60184b322312618e4 to 19482048d4fffe52e94fcd94b81fa8ef59000558

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

fcef2b6Use pip to install pipenv
4160c9fUpdate ci-pipenv.yml
d0e01f2Update ci-pipenv.yml
711a2a6Update ci-pipenv.yml
341be72Update ci-pipenv.yml
918e1a5Update ci-pipenv.yml
573ec51Try with only one python
961abc3Update ci-pipenv.yml
5a9b803Update ci-pipenv.yml
1948204Merge branch 'public/build/inplace' of https://github.com/tobiasdiez/sage into public/build/inplace

comment:130 Changed 5 months ago by git

  • Commit changed from 19482048d4fffe52e94fcd94b81fa8ef59000558 to fffea7a7a50dd8c79827a07d6ef4eb3568fc1817

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

0016538Merge branch 'public/build/inplace' of https://github.com/tobiasdiez/sage into public/build/inplace
d65d85eUpdate ci-pipenv.yml
c1b8aefMerge branch 'public/build/inplace' of https://github.com/tobiasdiez/sage into public/build/inplace
23d17adRename cblas to blas
3cfe1c9Merge branch 'develop' of git://github.com/sagemath/sage into public/build/cblas
ea4a44cSupport different blas libraries
75966fcPut cblas first
273bb93Make cblas pc module search configurable via sage_conf
e6ff9baMerge branch 'public/build/cblas' of git://trac.sagemath.org/sage into public/build/inplace
fffea7aInstall more dependencies

comment:131 Changed 5 months ago by gh-tobiasdiez

  • Dependencies changed from #30779, #30672, #30673, #30709 to #30779, #30672, #30673, #30709, #30706

comment:132 Changed 5 months ago by git

  • Commit changed from fffea7a7a50dd8c79827a07d6ef4eb3568fc1817 to 062746cad45c3c3edc9f0d905de8312df1c020ac

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

ed74c46Fix powershell script
88f4529Make powershell run
4171062Cleanup
bfdc1c0Fix problem with libhomfly
3bda21eFix arb lib
3d476d5Install more deps for macos (as recommended by configure)
4302ac9Also install ecl
e34a6afAlso install pynac
8679770Disable copying of sage-env-config
062746cTry with hard-coded version

comment:133 follow-up: Changed 5 months ago by gh-tobiasdiez

I've now added github actions to have a reproducible test in a clean environment. The compilation is working now (under linux), but the singular library cannot be found during runtime. https://github.com/tobiasdiez/sage/runs/1287237342?check_suite_focus=true

The problem appears to be that sage is looking for singular in sysconfig.get_config_var('LIBDIR') (e.g. /usr/local/lib) but it is installed as /usr/lib/x86_64-linux-gnu/libsingular-Singular.so. Is there a reason why LIBDIR is used in env.py instead of the approach using pkgconfig as during compilation time?

comment:134 Changed 5 months ago by mkoeppe

_get_shared_lib_filename actually also looks in MULTILIB and is supposed to find this Debian-specific install location via this mechanism. But it is possible that this is broken because we currently actually do not use system singular (nor gap) at all (see #29644, #29024, meta-ticket #27330). This needs more investigation.

comment:135 in reply to: ↑ 133 Changed 5 months ago by mkoeppe

Replying to gh-tobiasdiez:

Is there a reason why LIBDIR is used in env.py instead of the approach using pkgconfig as during compilation time?

pkgconfig does not really reveal the libname directly, so one would need to parse pkgconfig --libs.

Another reason is that the .pc files are usually only available in "dev" packages. So the runtime of Sage should not depend on it.

#29024 intends to make the location configurable (at build time!) via sage_conf.

comment:136 Changed 4 months ago by git

  • Commit changed from 062746cad45c3c3edc9f0d905de8312df1c020ac to 7523baa68b46732cc381387e58ea1a9559c11e96

Branch pushed to git repo; I updated commit sha1. New commits:

38acc87Fix _get_shared_lib_filename to find singular
94787d1Try to run without build-isolation (manual)
8afec5aTry to set PIP_NO_BUILD_ISOLATION:
a321431Strange convention for PIP_NO variables
7523baaCleanup

comment:137 follow-up: Changed 4 months ago by gh-tobiasdiez

Thanks for the input. I had a closer look at _get_shared_lib_filename and the problem was that libsingular-Singular.so is in the subfolder /usr/lib/x86_64-linux-gnu/ while _get_shared_lib_filename only searched the top folder /usr/lib. I've now changed it to also look in subfolders. That should be reasonable fix until #29024 is implemented.

With these changes, the github action workflow passes successful on linux: https://github.com/tobiasdiez/sage/runs/1303324520?check_suite_focus=true It's nice to see that it only takes about 1 hour to have a fully workable sage environment.

This is now really ready for review.

TODOs (for further tickets):

  • Investigate what needs to be installed on macOS to have a successful build, and add these steps to the github action. (I don't have a mac, so I'll not work on this.)
  • Run doctests in the github action as soon as they no longer rely on SAGE_LOCAL (#22731)

comment:138 Changed 4 months ago by gh-tobiasdiez

  • Description modified (diff)

comment:139 in reply to: ↑ 137 Changed 4 months ago by mkoeppe

Replying to gh-tobiasdiez:

Thanks for the input. I had a closer look at _get_shared_lib_filename and the problem was that libsingular-Singular.so is in the subfolder /usr/lib/x86_64-linux-gnu/ while _get_shared_lib_filename only searched the top folder /usr/lib. I've now changed it to also look in subfolders.

This should have been taking care of by MULTILIB. Is this not configured in your system?

Last edited 4 months ago by mkoeppe (previous) (diff)

comment:140 Changed 4 months ago by gh-tobiasdiez

MULTILIB is empty on my system (WSL Ubuntu 20.04), nor was it working for the github actions (Ubuntu 20.10).

comment:141 Changed 4 months ago by mkoeppe

Can you show the full output of python3 -m sysconfig on this system?

comment:142 Changed 4 months ago by gh-tobiasdiez

Platform: "linux-x86_64"
Python version: "3.8"
Current installation scheme: "posix_prefix"

Paths: 
        data = "/mnt/d/Programming/Projects/sage/src/.venv"
        include = "/usr/include/python3.8"
        platinclude = "/usr/include/python3.8"
        platlib = "/mnt/d/Programming/Projects/sage/src/.venv/lib/python3.8/site-packages"
        platstdlib = "/mnt/d/Programming/Projects/sage/src/.venv/lib/python3.8"
        purelib = "/mnt/d/Programming/Projects/sage/src/.venv/lib/python3.8/site-packages"
        scripts = "/mnt/d/Programming/Projects/sage/src/.venv/bin"
        stdlib = "/usr/lib/python3.8"

Variables: 
        ABIFLAGS = ""
        AC_APPLE_UNIVERSAL_BUILD = "0"
        AIX_GENUINE_CPLUSPLUS = "0"
        ALT_SOABI = "0"
        ANDROID_API_LEVEL = "0"
        AR = "x86_64-linux-gnu-gcc-ar"
        ARFLAGS = "rcs"
        BASECFLAGS = "-Wno-unused-result -Wsign-compare"
        BASECPPFLAGS = "-IObjects -IInclude -IPython"
        BASEMODLIBS = ""
        BINDIR = "/usr/bin"
        BINLIBDEST = "/usr/lib/python3.8"
        BLDLIBRARY = "-lpython3.8"
        BLDSHARED = "x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions  -Wl,-z,relro -g -fwrapv -O2   "
        BUILDEXE = ""
        BUILDPYTHON = "python"
        BUILD_GNU_TYPE = "x86_64-pc-linux-gnu"
        BYTESTR_DEPS = "\"
        CC = "x86_64-linux-gnu-gcc -pthread"
        CCSHARED = "-fPIC"
        CFLAGS = "-Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g   -fstack-protector-strong -Wformat -Werror=format-security  -g -fwrapv -O2   "
        CFLAGSFORSHARED = "-fPIC"
        CFLAGS_ALIASING = ""
        CFLAGS_NODIST = ""
        CONFIGFILES = "configure configure.ac acconfig.h pyconfig.h.in Makefile.pre.in"
        CONFIGURE_CFLAGS = "-g   -fstack-protector-strong -Wformat -Werror=format-security"
        CONFIGURE_CFLAGS_NODIST = "-std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration"
        CONFIGURE_CPPFLAGS = "-Wdate-time -D_FORTIFY_SOURCE=2"
        CONFIGURE_LDFLAGS = "-Wl,-Bsymbolic-functions  -Wl,-z,relro -g -fwrapv -O2   "
        CONFIGURE_LDFLAGS_NODIST = ""
        CONFIG_ARGS = "'--enable-shared' '--prefix=/usr' '--enable-ipv6' '--enable-loadable-sqlite-extensions' '--with-dbmliborder=bdb:gdbm' '--with-computed-gotos' '--without-ensurepip' '--with-system-expat' '--with-system-libmpdec' '--with-dtrace' '--with-system-ffi' 'CC=x86_64-linux-gnu-gcc' 'CFLAGS=-g   -fstack-protector-strong -Wformat -Werror=format-security ' 'LDFLAGS=-Wl,-Bsymbolic-functions  -Wl,-z,relro -g -fwrapv -O2   ' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2'"
        CONFINCLUDEDIR = "/usr/include"
        CONFINCLUDEPY = "/usr/include/python3.8"
        COREPYTHONPATH = ""
        COVERAGE_INFO = "/build/python3.8-6QL2k7/python3.8-3.8.2/build-shared/coverage.info"
        COVERAGE_REPORT = "/build/python3.8-6QL2k7/python3.8-3.8.2/build-shared/lcov-report"
        COVERAGE_REPORT_OPTIONS = "--no-branch-coverage --title "CPython lcov report""
        CPPFLAGS = "-IObjects -IInclude -IPython -I. -I../Include -Wdate-time -D_FORTIFY_SOURCE=2"
        CXX = "x86_64-linux-gnu-g++ -pthread"
        DESTDIRS = "/usr /usr/lib /usr/lib/python3.8 /usr/lib/python3.8/lib-dynload"
        DESTLIB = "/usr/lib/python3.8"
        DESTPATH = ""
        DESTSHARED = "/usr/lib/python3.8/lib-dynload"
        DFLAGS = ""
        DIRMODE = "755"
        DIST = "README.rst ChangeLog configure configure.ac acconfig.h pyconfig.h.in Makefile.pre.in Include Lib Misc Ext-dummy"
        DISTDIRS = "Include Lib Misc Ext-dummy"
        DISTFILES = "README.rst ChangeLog configure configure.ac acconfig.h pyconfig.h.in Makefile.pre.in"
        DLINCLDIR = "."
        DLLLIBRARY = ""
        DOUBLE_IS_ARM_MIXED_ENDIAN_IEEE754 = "0"
        DOUBLE_IS_BIG_ENDIAN_IEEE754 = "0"
        DOUBLE_IS_LITTLE_ENDIAN_IEEE754 = "1"
        DTRACE = "/usr/bin/dtrace"
        DTRACE_DEPS = "\"
        DTRACE_HEADERS = "Include/pydtrace_probes.h"
        DTRACE_OBJS = "Python/pydtrace.o"
        DYNLOADFILE = "dynload_shlib.o"
        ENABLE_IPV6 = "1"
        ENSUREPIP = "no"
        EXE = ""
        EXEMODE = "755"
        EXTRATESTOPTS = ""
        EXT_SUFFIX = ".cpython-38-x86_64-linux-gnu.so"
        FILEMODE = "644"
        FLOAT_WORDS_BIGENDIAN = "0"
        FLOCK_NEEDS_LIBBSD = "0"
        GETPGRP_HAVE_ARG = "0"
        GETTIMEOFDAY_NO_TZ = "0"
        GITBRANCH = ""
        GITTAG = ""
        GITVERSION = ""
        GNULD = "yes"
        HAVE_ACCEPT4 = "1"
        HAVE_ACOSH = "1"
        HAVE_ADDRINFO = "1"
        HAVE_ALARM = "1"
        HAVE_ALIGNED_REQUIRED = "0"
        HAVE_ALLOCA_H = "1"
        HAVE_ALTZONE = "0"
        HAVE_ASINH = "1"
        HAVE_ASM_TYPES_H = "1"
        HAVE_ATANH = "1"
        HAVE_BIND_TEXTDOMAIN_CODESET = "1"
        HAVE_BLUETOOTH_BLUETOOTH_H = "1"
        HAVE_BLUETOOTH_H = "0"
        HAVE_BROKEN_MBSTOWCS = "0"
        HAVE_BROKEN_NICE = "0"
        HAVE_BROKEN_PIPE_BUF = "0"
        HAVE_BROKEN_POLL = "0"
        HAVE_BROKEN_POSIX_SEMAPHORES = "0"
        HAVE_BROKEN_PTHREAD_SIGMASK = "0"
        HAVE_BROKEN_SEM_GETVALUE = "0"
        HAVE_BROKEN_UNSETENV = "0"
        HAVE_BUILTIN_ATOMIC = "1"
        HAVE_CHFLAGS = "0"
        HAVE_CHOWN = "1"
        HAVE_CHROOT = "1"
        HAVE_CLOCK = "1"
        HAVE_CLOCK_GETRES = "1"
        HAVE_CLOCK_GETTIME = "1"
        HAVE_CLOCK_SETTIME = "1"
        HAVE_COMPUTED_GOTOS = "1"
        HAVE_CONFSTR = "1"
        HAVE_CONIO_H = "0"
        HAVE_COPYSIGN = "1"
        HAVE_COPY_FILE_RANGE = "1"
        HAVE_CRYPT_H = "1"
        HAVE_CRYPT_R = "1"
        HAVE_CTERMID = "1"
        HAVE_CTERMID_R = "0"
        HAVE_CURSES_FILTER = "1"
        HAVE_CURSES_H = "1"
        HAVE_CURSES_HAS_KEY = "1"
        HAVE_CURSES_IMMEDOK = "1"
        HAVE_CURSES_IS_PAD = "1"
        HAVE_CURSES_IS_TERM_RESIZED = "1"
        HAVE_CURSES_RESIZETERM = "1"
        HAVE_CURSES_RESIZE_TERM = "1"
        HAVE_CURSES_SYNCOK = "1"
        HAVE_CURSES_TYPEAHEAD = "1"
        HAVE_CURSES_USE_ENV = "1"
        HAVE_CURSES_WCHGAT = "1"
        HAVE_DECL_ISFINITE = "1"
        HAVE_DECL_ISINF = "1"
        HAVE_DECL_ISNAN = "1"
        HAVE_DECL_RTLD_DEEPBIND = "1"
        HAVE_DECL_RTLD_GLOBAL = "1"
        HAVE_DECL_RTLD_LAZY = "1"
        HAVE_DECL_RTLD_LOCAL = "1"
        HAVE_DECL_RTLD_MEMBER = "0"
        HAVE_DECL_RTLD_NODELETE = "1"
        HAVE_DECL_RTLD_NOLOAD = "1"
        HAVE_DECL_RTLD_NOW = "1"
        HAVE_DECL_TZNAME = "0"
        HAVE_DEVICE_MACROS = "1"
        HAVE_DEV_PTC = "0"
        HAVE_DEV_PTMX = "1"
        HAVE_DIRECT_H = "0"
        HAVE_DIRENT_D_TYPE = "1"
        HAVE_DIRENT_H = "1"
        HAVE_DIRFD = "1"
        HAVE_DLFCN_H = "1"
        HAVE_DLOPEN = "1"
        HAVE_DUP2 = "1"
        HAVE_DUP3 = "1"
        HAVE_DYNAMIC_LOADING = "1"
        HAVE_ENDIAN_H = "1"
        HAVE_EPOLL = "1"
        HAVE_EPOLL_CREATE1 = "1"
        HAVE_ERF = "1"
        HAVE_ERFC = "1"
        HAVE_ERRNO_H = "1"
        HAVE_EXECV = "1"
        HAVE_EXPLICIT_BZERO = "1"
        HAVE_EXPLICIT_MEMSET = "0"
        HAVE_EXPM1 = "1"
        HAVE_FACCESSAT = "1"
        HAVE_FCHDIR = "1"
        HAVE_FCHMOD = "1"
        HAVE_FCHMODAT = "1"
        HAVE_FCHOWN = "1"
        HAVE_FCHOWNAT = "1"
        HAVE_FCNTL_H = "1"
        HAVE_FDATASYNC = "1"
        HAVE_FDOPENDIR = "1"
        HAVE_FDWALK = "0"
        HAVE_FEXECVE = "1"
        HAVE_FINITE = "1"
        HAVE_FLOCK = "1"
        HAVE_FORK = "1"
        HAVE_FORKPTY = "1"
        HAVE_FPATHCONF = "1"
        HAVE_FSEEK64 = "0"
        HAVE_FSEEKO = "1"
        HAVE_FSTATAT = "1"
        HAVE_FSTATVFS = "1"
        HAVE_FSYNC = "1"
        HAVE_FTELL64 = "0"
        HAVE_FTELLO = "1"
        HAVE_FTIME = "1"
        HAVE_FTRUNCATE = "1"
        HAVE_FUTIMENS = "1"
        HAVE_FUTIMES = "1"
        HAVE_FUTIMESAT = "1"
        HAVE_GAI_STRERROR = "1"
        HAVE_GAMMA = "1"
        HAVE_GCC_ASM_FOR_MC68881 = "0"
        HAVE_GCC_ASM_FOR_X64 = "1"
        HAVE_GCC_ASM_FOR_X87 = "1"
        HAVE_GCC_UINT128_T = "1"
        HAVE_GETADDRINFO = "1"
        HAVE_GETC_UNLOCKED = "1"
        HAVE_GETENTROPY = "1"
        HAVE_GETGRGID_R = "1"
        HAVE_GETGRNAM_R = "1"
        HAVE_GETGROUPLIST = "1"
        HAVE_GETGROUPS = "1"
        HAVE_GETHOSTBYNAME = "0"
        HAVE_GETHOSTBYNAME_R = "1"
        HAVE_GETHOSTBYNAME_R_3_ARG = "0"
        HAVE_GETHOSTBYNAME_R_5_ARG = "0"
        HAVE_GETHOSTBYNAME_R_6_ARG = "1"
        HAVE_GETITIMER = "1"
        HAVE_GETLOADAVG = "1"
        HAVE_GETLOGIN = "1"
        HAVE_GETNAMEINFO = "1"
        HAVE_GETPAGESIZE = "1"
        HAVE_GETPEERNAME = "1"
        HAVE_GETPGID = "1"
        HAVE_GETPGRP = "1"
        HAVE_GETPID = "1"
        HAVE_GETPRIORITY = "1"
        HAVE_GETPWENT = "1"
        HAVE_GETPWNAM_R = "1"
        HAVE_GETPWUID_R = "1"
        HAVE_GETRANDOM = "1"
        HAVE_GETRANDOM_SYSCALL = "1"
        HAVE_GETRESGID = "1"
        HAVE_GETRESUID = "1"
        HAVE_GETSID = "1"
        HAVE_GETSPENT = "1"
        HAVE_GETSPNAM = "1"
        HAVE_GETTIMEOFDAY = "1"
        HAVE_GETWD = "1"
        HAVE_GLIBC_MEMMOVE_BUG = "0"
        HAVE_GRP_H = "1"
        HAVE_HSTRERROR = "1"
        HAVE_HTOLE64 = "1"
        HAVE_HYPOT = "1"
        HAVE_IEEEFP_H = "0"
        HAVE_IF_NAMEINDEX = "1"
        HAVE_INET_ATON = "1"
        HAVE_INET_PTON = "1"
        HAVE_INITGROUPS = "1"
        HAVE_INTTYPES_H = "1"
        HAVE_IO_H = "0"
        HAVE_IPA_PURE_CONST_BUG = "0"
        HAVE_KILL = "1"
        HAVE_KILLPG = "1"
        HAVE_KQUEUE = "0"
        HAVE_LANGINFO_H = "1"
        HAVE_LARGEFILE_SUPPORT = "0"
        HAVE_LCHFLAGS = "0"
        HAVE_LCHMOD = "0"
        HAVE_LCHOWN = "1"
        HAVE_LGAMMA = "1"
        HAVE_LIBDL = "1"
        HAVE_LIBDLD = "0"
        HAVE_LIBIEEE = "0"
        HAVE_LIBINTL_H = "1"
        HAVE_LIBREADLINE = "1"
        HAVE_LIBRESOLV = "0"
        HAVE_LIBSENDFILE = "0"
        HAVE_LIBUTIL_H = "0"
        HAVE_LINK = "1"
        HAVE_LINKAT = "1"
        HAVE_LINUX_CAN_BCM_H = "1"
        HAVE_LINUX_CAN_H = "1"
        HAVE_LINUX_CAN_RAW_FD_FRAMES = "1"
        HAVE_LINUX_CAN_RAW_H = "1"
        HAVE_LINUX_MEMFD_H = "1"
        HAVE_LINUX_NETLINK_H = "1"
        HAVE_LINUX_QRTR_H = "1"
        HAVE_LINUX_RANDOM_H = "1"
        HAVE_LINUX_TIPC_H = "1"
        HAVE_LINUX_VM_SOCKETS_H = "1"
        HAVE_LOCKF = "1"
        HAVE_LOG1P = "1"
        HAVE_LOG2 = "1"
        HAVE_LONG_DOUBLE = "1"
        HAVE_LSTAT = "1"
        HAVE_LUTIMES = "1"
        HAVE_MADVISE = "1"
        HAVE_MAKEDEV = "1"
        HAVE_MBRTOWC = "1"
        HAVE_MEMFD_CREATE = "1"
        HAVE_MEMORY_H = "1"
        HAVE_MEMRCHR = "1"
        HAVE_MKDIRAT = "1"
        HAVE_MKFIFO = "1"
        HAVE_MKFIFOAT = "1"
        HAVE_MKNOD = "1"
        HAVE_MKNODAT = "1"
        HAVE_MKTIME = "1"
        HAVE_MMAP = "1"
        HAVE_MREMAP = "1"
        HAVE_NCURSES_H = "1"
        HAVE_NDIR_H = "0"
        HAVE_NETPACKET_PACKET_H = "1"
        HAVE_NET_IF_H = "1"
        HAVE_NICE = "1"
        HAVE_OPENAT = "1"
        HAVE_OPENPTY = "1"
        HAVE_PATHCONF = "1"
        HAVE_PAUSE = "1"
        HAVE_PIPE2 = "1"
        HAVE_PLOCK = "0"
        HAVE_POLL = "1"
        HAVE_POLL_H = "1"
        HAVE_POSIX_FADVISE = "1"
        HAVE_POSIX_FALLOCATE = "1"
        HAVE_POSIX_SPAWN = "1"
        HAVE_POSIX_SPAWNP = "1"
        HAVE_PREAD = "1"
        HAVE_PREADV = "1"
        HAVE_PREADV2 = "1"
        HAVE_PRLIMIT = "1"
        HAVE_PROCESS_H = "0"
        HAVE_PROTOTYPES = "1"
        HAVE_PTHREAD_CONDATTR_SETCLOCK = "1"
        HAVE_PTHREAD_DESTRUCTOR = "0"
        HAVE_PTHREAD_GETCPUCLOCKID = "1"
        HAVE_PTHREAD_H = "1"
        HAVE_PTHREAD_INIT = "0"
        HAVE_PTHREAD_KILL = "1"
        HAVE_PTHREAD_SIGMASK = "1"
        HAVE_PTY_H = "1"
        HAVE_PUTENV = "1"
        HAVE_PWRITE = "1"
        HAVE_PWRITEV = "1"
        HAVE_PWRITEV2 = "1"
        HAVE_READLINK = "1"
        HAVE_READLINKAT = "1"
        HAVE_READV = "1"
        HAVE_REALPATH = "1"
        HAVE_RENAMEAT = "1"
        HAVE_RL_APPEND_HISTORY = "1"
        HAVE_RL_CATCH_SIGNAL = "1"
        HAVE_RL_COMPLETION_APPEND_CHARACTER = "1"
        HAVE_RL_COMPLETION_DISPLAY_MATCHES_HOOK = "1"
        HAVE_RL_COMPLETION_MATCHES = "1"
        HAVE_RL_COMPLETION_SUPPRESS_APPEND = "1"
        HAVE_RL_PRE_INPUT_HOOK = "1"
        HAVE_RL_RESIZE_TERMINAL = "1"
        HAVE_ROUND = "1"
        HAVE_RTPSPAWN = "0"
        HAVE_SCHED_GET_PRIORITY_MAX = "1"
        HAVE_SCHED_H = "1"
        HAVE_SCHED_RR_GET_INTERVAL = "1"
        HAVE_SCHED_SETAFFINITY = "1"
        HAVE_SCHED_SETPARAM = "1"
        HAVE_SCHED_SETSCHEDULER = "1"
        HAVE_SEM_GETVALUE = "1"
        HAVE_SEM_OPEN = "1"
        HAVE_SEM_TIMEDWAIT = "1"
        HAVE_SEM_UNLINK = "1"
        HAVE_SENDFILE = "1"
        HAVE_SETEGID = "1"
        HAVE_SETEUID = "1"
        HAVE_SETGID = "1"
        HAVE_SETGROUPS = "1"
        HAVE_SETHOSTNAME = "1"
        HAVE_SETITIMER = "1"
        HAVE_SETLOCALE = "1"
        HAVE_SETPGID = "1"
        HAVE_SETPGRP = "1"
        HAVE_SETPRIORITY = "1"
        HAVE_SETREGID = "1"
        HAVE_SETRESGID = "1"
        HAVE_SETRESUID = "1"
        HAVE_SETREUID = "1"
        HAVE_SETSID = "1"
        HAVE_SETUID = "1"
        HAVE_SETVBUF = "1"
        HAVE_SHADOW_H = "1"
        HAVE_SHM_OPEN = "1"
        HAVE_SHM_UNLINK = "1"
        HAVE_SIGACTION = "1"
        HAVE_SIGALTSTACK = "1"
        HAVE_SIGFILLSET = "1"
        HAVE_SIGINFO_T_SI_BAND = "1"
        HAVE_SIGINTERRUPT = "1"
        HAVE_SIGNAL_H = "1"
        HAVE_SIGPENDING = "1"
        HAVE_SIGRELSE = "1"
        HAVE_SIGTIMEDWAIT = "1"
        HAVE_SIGWAIT = "1"
        HAVE_SIGWAITINFO = "1"
        HAVE_SNPRINTF = "1"
        HAVE_SOCKADDR_ALG = "1"
        HAVE_SOCKADDR_SA_LEN = "0"
        HAVE_SOCKADDR_STORAGE = "1"
        HAVE_SOCKETPAIR = "1"
        HAVE_SPAWN_H = "1"
        HAVE_SSIZE_T = "1"
        HAVE_STATVFS = "1"
        HAVE_STAT_TV_NSEC = "1"
        HAVE_STAT_TV_NSEC2 = "0"
        HAVE_STDARG_PROTOTYPES = "1"
        HAVE_STDINT_H = "1"
        HAVE_STDLIB_H = "1"
        HAVE_STD_ATOMIC = "1"
        HAVE_STRDUP = "1"
        HAVE_STRFTIME = "1"
        HAVE_STRINGS_H = "1"
        HAVE_STRING_H = "1"
        HAVE_STRLCPY = "0"
        HAVE_STROPTS_H = "0"
        HAVE_STRSIGNAL = "1"
        HAVE_STRUCT_PASSWD_PW_GECOS = "1"
        HAVE_STRUCT_PASSWD_PW_PASSWD = "1"
        HAVE_STRUCT_STAT_ST_BIRTHTIME = "0"
        HAVE_STRUCT_STAT_ST_BLKSIZE = "1"
        HAVE_STRUCT_STAT_ST_BLOCKS = "1"
        HAVE_STRUCT_STAT_ST_FLAGS = "0"
        HAVE_STRUCT_STAT_ST_GEN = "0"
        HAVE_STRUCT_STAT_ST_RDEV = "1"
        HAVE_STRUCT_TM_TM_ZONE = "1"
        HAVE_SYMLINK = "1"
        HAVE_SYMLINKAT = "1"
        HAVE_SYNC = "1"
        HAVE_SYSCONF = "1"
        HAVE_SYSEXITS_H = "1"
        HAVE_SYS_AUDIOIO_H = "0"
        HAVE_SYS_BSDTTY_H = "0"
        HAVE_SYS_DEVPOLL_H = "0"
        HAVE_SYS_DIR_H = "0"
        HAVE_SYS_ENDIAN_H = "0"
        HAVE_SYS_EPOLL_H = "1"
        HAVE_SYS_EVENT_H = "0"
        HAVE_SYS_FILE_H = "1"
        HAVE_SYS_IOCTL_H = "1"
        HAVE_SYS_KERN_CONTROL_H = "0"
        HAVE_SYS_LOADAVG_H = "0"
        HAVE_SYS_LOCK_H = "0"
        HAVE_SYS_MEMFD_H = "0"
        HAVE_SYS_MKDEV_H = "0"
        HAVE_SYS_MMAN_H = "1"
        HAVE_SYS_MODEM_H = "0"
        HAVE_SYS_NDIR_H = "0"
        HAVE_SYS_PARAM_H = "1"
        HAVE_SYS_POLL_H = "1"
        HAVE_SYS_RANDOM_H = "1"
        HAVE_SYS_RESOURCE_H = "1"
        HAVE_SYS_SELECT_H = "1"
        HAVE_SYS_SENDFILE_H = "1"
        HAVE_SYS_SOCKET_H = "1"
        HAVE_SYS_STATVFS_H = "1"
        HAVE_SYS_STAT_H = "1"
        HAVE_SYS_SYSCALL_H = "1"
        HAVE_SYS_SYSMACROS_H = "1"
        HAVE_SYS_SYS_DOMAIN_H = "0"
        HAVE_SYS_TERMIO_H = "0"
        HAVE_SYS_TIMES_H = "1"
        HAVE_SYS_TIME_H = "1"
        HAVE_SYS_TYPES_H = "1"
        HAVE_SYS_UIO_H = "1"
        HAVE_SYS_UN_H = "1"
        HAVE_SYS_UTSNAME_H = "1"
        HAVE_SYS_WAIT_H = "1"
        HAVE_SYS_XATTR_H = "1"
        HAVE_TCGETPGRP = "1"
        HAVE_TCSETPGRP = "1"
        HAVE_TEMPNAM = "1"
        HAVE_TERMIOS_H = "1"
        HAVE_TERM_H = "1"
        HAVE_TGAMMA = "1"
        HAVE_TIMEGM = "1"
        HAVE_TIMES = "1"
        HAVE_TMPFILE = "1"
        HAVE_TMPNAM = "1"
        HAVE_TMPNAM_R = "1"
        HAVE_TM_ZONE = "1"
        HAVE_TRUNCATE = "1"
        HAVE_TZNAME = "0"
        HAVE_UCS4_TCL = "0"
        HAVE_UNAME = "1"
        HAVE_UNISTD_H = "1"
        HAVE_UNLINKAT = "1"
        HAVE_UNSETENV = "1"
        HAVE_USABLE_WCHAR_T = "0"
        HAVE_UTIL_H = "0"
        HAVE_UTIMENSAT = "1"
        HAVE_UTIMES = "1"
        HAVE_UTIME_H = "1"
        HAVE_UUID_CREATE = "0"
        HAVE_UUID_ENC_BE = "0"
        HAVE_UUID_GENERATE_TIME_SAFE = "1"
        HAVE_UUID_H = "0"
        HAVE_UUID_UUID_H = "1"
        HAVE_WAIT3 = "1"
        HAVE_WAIT4 = "1"
        HAVE_WAITID = "1"
        HAVE_WAITPID = "1"
        HAVE_WCHAR_H = "1"
        HAVE_WCSCOLL = "1"
        HAVE_WCSFTIME = "1"
        HAVE_WCSXFRM = "1"
        HAVE_WMEMCMP = "1"
        HAVE_WORKING_TZSET = "1"
        HAVE_WRITEV = "1"
        HAVE_X509_VERIFY_PARAM_SET1_HOST = "1"
        HAVE_ZLIB_COPY = "1"
        HAVE__GETPTY = "0"
        HOST_GNU_TYPE = "x86_64-pc-linux-gnu"
        INCLDIRSTOMAKE = "/usr/include /usr/include /usr/include/python3.8 /usr/include/python3.8"
        INCLUDEDIR = "/usr/include"
        INCLUDEPY = "/usr/include/python3.8"
        INSTALL = "/usr/bin/install -c"
        INSTALL_DATA = "/usr/bin/install -c -m 644"
        INSTALL_PROGRAM = "/usr/bin/install -c"
        INSTALL_SCRIPT = "/usr/bin/install -c"
        INSTALL_SHARED = "/usr/bin/install -c -m 755"
        INSTSONAME = "libpython3.8.so.1.0"
        IO_H = "Modules/_io/_iomodule.h"
        IO_OBJS = "\"
        LDCXXSHARED = "x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions"
        LDFLAGS = "-Wl,-Bsymbolic-functions  -Wl,-z,relro -g -fwrapv -O2   "
        LDFLAGS_NODIST = ""
        LDLIBRARY = "libpython3.8.so"
        LDLIBRARYDIR = ""
        LDSHARED = "x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions  -Wl,-z,relro -g -fwrapv -O2   "
        LDVERSION = "3.8"
        LIBC = ""
        LIBDEST = "/usr/lib/python3.8"
        LIBDIR = "/usr/lib"
        LIBFFI_INCLUDEDIR = ""
        LIBM = "-lm"
        LIBOBJDIR = "Python/"
        LIBOBJS = ""
        LIBPC = "/usr/lib/x86_64-linux-gnu/pkgconfig"
        LIBPL = "/usr/lib/python3.8/config-3.8-x86_64-linux-gnu"
        LIBPYTHON = ""
        LIBRARY = "libpython3.8.a"
        LIBRARY_OBJS = "\"
        LIBRARY_OBJS_OMIT_FROZEN = "\"
        LIBS = "-lcrypt -lpthread -ldl  -lutil -lm"
        LIBSUBDIRS = "tkinter tkinter/test tkinter/test/test_tkinter \"
        LINKCC = "x86_64-linux-gnu-gcc -pthread"
        LINKFORSHARED = "-Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions"
        LIPO_32BIT_FLAGS = ""
        LLVM_PROF_ERR = "no"
        LLVM_PROF_FILE = ""
        LLVM_PROF_MERGER = "true"
        LN = "ln"
        LOCALMODLIBS = "-lexpat                       -L/usr/lib -lz                       -lexpat"
        MACHDEP = "linux"
        MACHDEP_OBJS = ""
        MACHDESTLIB = "/usr/lib/python3.8"
        MACOSX_DEPLOYMENT_TARGET = ""
        MAINCC = "x86_64-linux-gnu-gcc -pthread"
        MAJOR_IN_MKDEV = "0"
        MAJOR_IN_SYSMACROS = "1"
        MAKESETUP = "../Modules/makesetup"
        MANDIR = "/usr/share/man"
        MKDIR_P = "/bin/mkdir -p"
        MODBUILT_NAMES = "array  cmath  math  _struct  _random  _elementtree  _pickle  _datetime  _bisect  _heapq  _statistics  unicodedata  fcntl  spwd  grp  select  _csv  _socket  _posixsubprocess  _md5  _sha1  _sha256  _sha512  _sha3  _blake2  syslog  binascii  zlib  posix  errno  pwd  _sre  _codecs  _weakref  _functools  _operator  _collections  _abc  itertools  atexit  _signal  _stat  time  _thread  _locale  _io  faulthandler  _tracemalloc  _symtable  pyexpat  xxsubtype"
        MODDISABLED_NAMES = ""
        MODLIBS = "-lexpat                       -L/usr/lib -lz                       -lexpat"
        MODOBJS = "$(sort   Modules/arraymodule.o  Modules/cmathmodule.o Modules/_math.o  Modules/mathmodule.o Modules/_math.o  Modules/_struct.o  Modules/_randommodule.o  Modules/_elementtree.o  Modules/_pickle.o  Modules/_datetimemodule.o  Modules/_bisectmodule.o  Modules/_heapqmodule.o  Modules/_statisticsmodule.o  Modules/unicodedata.o  Modules/fcntlmodule.o  Modules/spwdmodule.o  Modules/grpmodule.o  Modules/selectmodule.o  Modules/_csv.o  Modules/socketmodule.o  Modules/_posixsubprocess.o  Modules/md5module.o  Modules/sha1module.o  Modules/sha256module.o  Modules/sha512module.o  Modules/sha3module.o  Modules/blake2module.o Modules/blake2b_impl.o Modules/blake2s_impl.o  Modules/syslogmodule.o  Modules/binascii.o  Modules/zlibmodule.o  Modules/posixmodule.o  Modules/errnomodule.o  Modules/pwdmodule.o  Modules/_sre.o  Modules/_codecsmodule.o  Modules/_weakref.o  Modules/_functoolsmodule.o  Modules/_operator.o  Modules/_collectionsmodule.o  Modules/_abc.o  Modules/itertoolsmodule.o  Modules/atexitmodule.o  Modules/signalmodule.o  Modules/_stat.o  Modules/timemodule.o  Modules/_threadmodule.o  Modules/_localemodule.o  Modules/_iomodule.o Modules/iobase.o Modules/fileio.o Modules/bytesio.o Modules/bufferedio.o Modules/textio.o Modules/stringio.o  Modules/faulthandler.o  Modules/_tracemalloc.o Modules/hashtable.o  Modules/symtablemodule.o  Modules/pyexpat.o  Modules/xxsubtype.o)"
        MODULE_OBJS = "\"
        MULTIARCH = "x86_64-linux-gnu"
        MULTIARCH_CPPFLAGS = "-DMULTIARCH=\"x86_64-linux-gnu\""
        MVWDELCH_IS_EXPRESSION = "1"
        NO_AS_NEEDED = "-Wl,--no-as-needed"
        OBJECT_OBJS = "\"
        OPENSSL_INCLUDES = ""
        OPENSSL_LDFLAGS = ""
        OPENSSL_LIBS = "-lssl -lcrypto"
        OPT = "-DNDEBUG -g -fwrapv -O2 -Wall"
        OTHER_LIBTOOL_OPT = ""
        PACKAGE_BUGREPORT = "0"
        PACKAGE_NAME = "0"
        PACKAGE_STRING = "0"
        PACKAGE_TARNAME = "0"
        PACKAGE_URL = "0"
        PACKAGE_VERSION = "0"
        PARSER_HEADERS = "\"
        PARSER_OBJS = "\ Parser/myreadline.o Parser/parsetok.o Parser/tokenizer.o"
        PGO_PROF_GEN_FLAG = "-fprofile-generate"
        PGO_PROF_USE_FLAG = ""
        POBJS = "\"
        POSIX_SEMAPHORES_NOT_ENABLED = "0"
        PROFILE_TASK = "-m test --pgo"
        PTHREAD_KEY_T_IS_COMPATIBLE_WITH_INT = "1"
        PTHREAD_SYSTEM_SCHED_SUPPORTED = "1"
        PURIFY = ""
        PY3LIBRARY = "libpython3.so"
        PYLONG_BITS_IN_DIGIT = "0"
        PYTHON = "python"
        PYTHONFRAMEWORK = ""
        PYTHONFRAMEWORKDIR = "no-framework"
        PYTHONFRAMEWORKINSTALLDIR = ""
        PYTHONFRAMEWORKPREFIX = ""
        PYTHONPATH = ""
        PYTHON_FOR_BUILD = "./python -E"
        PYTHON_FOR_REGEN = "python3.8"
        PYTHON_HEADERS = "\"
        PYTHON_OBJS = "\"
        PY_BUILTIN_MODULE_CFLAGS = "-Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g   -fstack-protector-strong -Wformat -Werror=format-security  -g -fwrapv -O2    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration  -I../Include/internal -IObjects -IInclude -IPython -I. -I../Include -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DPy_BUILD_CORE_BUILTIN"
        PY_CFLAGS = "-Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g   -fstack-protector-strong -Wformat -Werror=format-security  -g -fwrapv -O2   "
        PY_CFLAGS_NODIST = "-std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration  -I../Include/internal"
        PY_COERCE_C_LOCALE = "1"
        PY_CORE_CFLAGS = "-Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g   -fstack-protector-strong -Wformat -Werror=format-security  -g -fwrapv -O2    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration  -I../Include/internal -IObjects -IInclude -IPython -I. -I../Include -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DPy_BUILD_CORE"
        PY_CORE_LDFLAGS = "-Wl,-Bsymbolic-functions  -Wl,-z,relro -g -fwrapv -O2   "
        PY_CPPFLAGS = "-IObjects -IInclude -IPython -I. -I../Include -Wdate-time -D_FORTIFY_SOURCE=2"
        PY_FORMAT_SIZE_T = ""z""
        PY_LDFLAGS = "-Wl,-Bsymbolic-functions  -Wl,-z,relro -g -fwrapv -O2   "
        PY_LDFLAGS_NODIST = ""
        PY_SSL_DEFAULT_CIPHERS = "1"
        PY_SSL_DEFAULT_CIPHER_STRING = "0"
        PY_STDMODULE_CFLAGS = "-Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g   -fstack-protector-strong -Wformat -Werror=format-security  -g -fwrapv -O2    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration  -I../Include/internal -IObjects -IInclude -IPython -I. -I../Include -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC"
        Py_DEBUG = "0"
        Py_ENABLE_SHARED = "1"
        Py_HASH_ALGORITHM = "0"
        Py_TRACE_REFS = "0"
        QUICKTESTOPTS = "-x test_subprocess test_io test_lib2to3 \"
        READELF = "readelf"
        RESSRCDIR = "Mac/Resources/framework"
        RETSIGTYPE = "void"
        RUNSHARED = "LD_LIBRARY_PATH=/build/python3.8-6QL2k7/python3.8-3.8.2/build-shared"
        SCRIPTDIR = "/usr/lib"
        SETPGRP_HAVE_ARG = "0"
        SGI_ABI = "@SGI_ABI@"
        SHELL = "/bin/sh"
        SHLIBS = "-lcrypt -lpthread -ldl  -lutil -lm"
        SHLIB_SUFFIX = ".so"
        SHM_NEEDS_LIBRT = "1"
        SIGNED_RIGHT_SHIFT_ZERO_FILLS = "0"
        SITEPATH = ""
        SIZEOF_DOUBLE = "8"
        SIZEOF_FLOAT = "4"
        SIZEOF_FPOS_T = "16"
        SIZEOF_INT = "4"
        SIZEOF_LONG = "8"
        SIZEOF_LONG_DOUBLE = "16"
        SIZEOF_LONG_LONG = "8"
        SIZEOF_OFF_T = "8"
        SIZEOF_PID_T = "4"
        SIZEOF_PTHREAD_KEY_T = "4"
        SIZEOF_PTHREAD_T = "8"
        SIZEOF_SHORT = "2"
        SIZEOF_SIZE_T = "8"
        SIZEOF_TIME_T = "8"
        SIZEOF_UINTPTR_T = "8"
        SIZEOF_VOID_P = "8"
        SIZEOF_WCHAR_T = "4"
        SIZEOF__BOOL = "1"
        SO = ".cpython-38-x86_64-linux-gnu.so"
        SOABI = "cpython-38-x86_64-linux-gnu"
        SRCDIRS = "Parser Objects Python Modules Modules/_io Programs"
        SRC_GDB_HOOKS = "../Tools/gdb/libpython.py"
        STDC_HEADERS = "1"
        STRICT_SYSV_CURSES = "/* Don't use ncurses extensions */"
        STRIPFLAG = "-s"
        SUBDIRS = ""
        SUBDIRSTOO = "Include Lib Misc"
        SYSLIBS = "-lm"
        SYS_SELECT_WITH_SYS_TIME = "1"
        TCLTK_INCLUDES = ""
        TCLTK_LIBS = ""
        TESTOPTS = ""
        TESTPATH = ""
        TESTPYTHON = "LD_LIBRARY_PATH=/build/python3.8-6QL2k7/python3.8-3.8.2/build-shared ./python"
        TESTPYTHONOPTS = ""
        TESTRUNNER = "LD_LIBRARY_PATH=/build/python3.8-6QL2k7/python3.8-3.8.2/build-shared ./python ../Tools/scripts/run_tests.py"
        TESTTIMEOUT = "1200"
        TIMEMODULE_LIB = "0"
        TIME_WITH_SYS_TIME = "1"
        TM_IN_SYS_TIME = "0"
        UNICODE_DEPS = "\"
        UNIVERSALSDK = ""
        UPDATE_FILE = "python3.8 ../Tools/scripts/update_file.py"
        USE_COMPUTED_GOTOS = "1"
        VERSION = "3.8"
        VPATH = ".."
        WINDOW_HAS_FLAGS = "1"
        WITH_DOC_STRINGS = "1"
        WITH_DTRACE = "1"
        WITH_DYLD = "0"
        WITH_LIBINTL = "0"
        WITH_NEXT_FRAMEWORK = "0"
        WITH_PYMALLOC = "1"
        WITH_VALGRIND = "0"
        X87_DOUBLE_ROUNDING = "0"
        XMLLIBSUBDIRS = "xml xml/dom xml/etree xml/parsers xml/sax"
        abiflags = ""
        abs_builddir = "/build/python3.8-6QL2k7/python3.8-3.8.2/build-shared"
        abs_srcdir = "/build/python3.8-6QL2k7/python3.8-3.8.2/build-shared/.."
        base = "/mnt/d/Programming/Projects/sage/src/.venv"
        datarootdir = "/usr/share"
        exec_prefix = "/usr"
        installed_base = "/usr"
        installed_platbase = "/usr"
        multiarchsubdir = "/x86_64-linux-gnu"
        platbase = "/mnt/d/Programming/Projects/sage/src/.venv"
        prefix = "/usr"
        projectbase = "/usr/bin"
        py_version = "3.8.2"
        py_version_nodot = "38"
        py_version_short = "3.8"
        srcdir = "/usr/lib/python3.8/config-3.8-x86_64-linux-gnu"
        userbase = "/home/tobias/.local"

comment:143 Changed 4 months ago by mkoeppe

I see MULTIARCH = "x86_64-linux-gnu" and multiarchsubdir = "/x86_64-linux-gnu".

comment:144 Changed 4 months ago by mkoeppe

One of these variables should be used in _get_shared_lib_filename. Why it is instead looking for MULTILIB, which indeed does not exist, should be investigated...

comment:145 Changed 4 months ago by mkoeppe

Let's make this a separate ticket please

comment:146 Changed 4 months ago by mkoeppe

This is now #30833

comment:147 Changed 4 months ago by gh-tobiasdiez

  • Dependencies changed from #30779, #30672, #30673, #30709, #30706 to #30779, #30672, #30673, #30709, #30706, #30833

comment:148 Changed 4 months ago by git

  • Commit changed from 7523baa68b46732cc381387e58ea1a9559c11e96 to 443424cf878ac7fa9c57a1bad313f1b4d84ed031

Branch pushed to git repo; I updated commit sha1. New commits:

289c3efImprove _get_shared_lib_filename
443424cCleanup code

comment:149 Changed 4 months ago by gh-tobiasdiez

  • Dependencies changed from #30779, #30672, #30673, #30709, #30706, #30833 to #30779, #30672, #30673, #30709, #30706, #30833, #30748

comment:150 Changed 4 months ago by git

  • Commit changed from 443424cf878ac7fa9c57a1bad313f1b4d84ed031 to b3045684c751317f252ff61f59ab53782df76177

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

2351ae3Update, extend and reorder dependencies
99ae00cReplace startup exception by warning
11882e5Use context manager
d858e46Merge branch 'public/build/startupWarning' of git://trac.sagemath.org/sage into public/build/inplace
46ac365Update numpy
3fcd71fMerge tag '9.3.beta0' into t/30709/public/build/cythonConfig
a646a24src/sage_setup/cython_options.py: Restore auto_pickle=False
8eae4bcMerge branch 'public/build/cythonConfig' of git://trac.sagemath.org/sage into public/build/inplace
5ac3a29Add matlibplot library
b304568Run doctests in github action

comment:151 Changed 4 months ago by git

  • Commit changed from b3045684c751317f252ff61f59ab53782df76177 to 9a94a29bc1c6179282dbd94a7afd19247884e525

Branch pushed to git repo; I updated commit sha1. New commits:

9a94a29Don't install texlive-generic-extra

comment:152 Changed 4 months ago by git

  • Commit changed from 9a94a29bc1c6179282dbd94a7afd19247884e525 to 16674ba8da9daa53709c9ad4bfc6fca6bdba13ee

Branch pushed to git repo; I updated commit sha1. New commits:

16674baRemove lazy import finish startup

comment:153 Changed 4 months ago by git

  • Commit changed from 16674ba8da9daa53709c9ad4bfc6fca6bdba13ee to 511775ec71b3d2efef8f6e9ca2ea95f234c566ad

Branch pushed to git repo; I updated commit sha1. New commits:

f078aa0Speedup a bit
3eac0afRefactor
dba18d0Add none check for IPython
511775eAdd caching

comment:154 Changed 4 months ago by gh-tobiasdiez

I've now added the doctest run to the github workflow. It works in principle although there are quite a lot of failing tests (mainly because some of the system-installed libs are not discovered). In order to keep the focus of this ticket narrow, fixing this should better be done in follow-up tickets.

comment:155 Changed 4 months ago by gh-tobiasdiez

  • Description modified (diff)

comment:156 follow-up: Changed 4 months ago by mkoeppe

It has been reported that the current src/setup.py (in Sage 9.2) is able to install a working sagelib on top of conda (see #28745).

I'll create a test for this in #30845, and then we can test the present ticket to make sure it does not break anything.

comment:157 Changed 4 months ago by git

  • Commit changed from 511775ec71b3d2efef8f6e9ca2ea95f234c566ad to 373d1c12e77647541fa7fc8b315aae98ee0298cd

Branch pushed to git repo; I updated commit sha1. New commits:

373d1c1Add rope as dev dependency

comment:158 in reply to: ↑ description Changed 4 months ago by mkoeppe

Replying to mkoeppe:

  • A few libraries are not contained in distros/debian.txt files. Should they be added? In particular, this concerns libsingular4-dev, liblinbox-dev, libratpoints-dev, libgap-dev, ecl, libpynac-dev.

Yes, let's add them in #30859.

Some of the information that you put in the ticket description above could go as comments into the debian.txt files.

comment:159 Changed 4 months ago by gh-tobiasdiez

Thanks for opening a new ticket for this!

Concerning the conda integration, this ticket here might actually help with this. You can use pipenv on top of conda, see https://pipenv.pypa.io/en/latest/advanced/#pipenv-and-other-python-distributions.

comment:160 Changed 4 months ago by gh-tobiasdiez

  • Description modified (diff)

comment:161 Changed 4 months ago by git

  • Commit changed from 373d1c12e77647541fa7fc8b315aae98ee0298cd to 096df15a1253e6cb0fa75c92e29458f29d5ad77d

Branch pushed to git repo; I updated commit sha1. New commits:

096df15Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace

comment:162 Changed 4 months ago by git

  • Commit changed from 096df15a1253e6cb0fa75c92e29458f29d5ad77d to 8eb2f8ec11943c8f7175ce4d8a293673ab18b774

Branch pushed to git repo; I updated commit sha1. New commits:

8eb2f8eRemove homfly workaround

comment:163 Changed 4 months ago by git

  • Commit changed from 8eb2f8ec11943c8f7175ce4d8a293673ab18b774 to 17f367eebf0e329275773e0c048f3009b2de19ed

Branch pushed to git repo; I updated commit sha1. New commits:

17f367eRefactor shared library discovery

comment:164 Changed 4 months ago by gh-tobiasdiez

  • Dependencies changed from #30779, #30672, #30673, #30709, #30706, #30833, #30748 to #30779, #30672, #30673, #30709, #30706, #30901, #30748

comment:165 Changed 4 months ago by git

  • Commit changed from 17f367eebf0e329275773e0c048f3009b2de19ed to 680f85d72cf88ae96e0bc4bf7b422e694f353437

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

c47c4bfCorrect indent
23b6336Merge branch 'public/build/multiarchsimple' of git://trac.sagemath.org/sage into public/build/inplace
6a52fbfRemove lazy import finish startup
f96025eMerge branch 'develop' of git://github.com/sagemath/sage into public/build/startupWarning
b952bb5Fix doctests
c6857ddHopefully fix doctest
23da14bWrap each sage.all import in startup guard
2f4055eSpecify that exit was succesful
d4d94a8Indeed replace exception by print statement
680f85dMerge branch 'public/build/startupWarning' of git://trac.sagemath.org/sage into public/build/inplace

comment:166 Changed 4 months ago by git

  • Commit changed from 680f85d72cf88ae96e0bc4bf7b422e694f353437 to ddcf2e3ba09cb3731042c1372b93e4db84f93b13

Branch pushed to git repo; I updated commit sha1. New commits:

ddcf2e3Cleanup code

comment:167 Changed 4 months ago by git

  • Commit changed from ddcf2e3ba09cb3731042c1372b93e4db84f93b13 to 772baf208259ff75edebdbf6f340c700be0681ab

Branch pushed to git repo; I updated commit sha1. New commits:

2088ffdUpgrade to dotnet 5.0 rc2
1b1750dUpgrade to dotnet 5.0
638ecbeInstall wget before using it
772baf2Install tzdata earlier

comment:168 Changed 4 months ago by gh-tobiasdiez

I've now merged the latest version of the dependencies, and cleaned the code.

Matthias (and others, of course), it would be nice if you could review the code (and its dependencies). I have a hard time working on other tickets as long as this ticket is not merged. So I would be grateful if that happens as soon as possible. Thanks!

comment:169 follow-up: Changed 4 months ago by mkoeppe

See if you can get doctesting to work on this branch - by merging #30578.

comment:170 follow-up: Changed 4 months ago by mkoeppe

Also removing sage-env-config from the list of installed scripts is not good. Instead please review and merge #29850 - which reassigns this script to the sage_conf package.

comment:171 in reply to: ↑ 169 ; follow-up: Changed 4 months ago by mkoeppe

Replying to mkoeppe:

See if you can get doctesting to work on this branch - by merging #30578.

In #30578 I had to add a few more packages to requirements.txt that you don't have in Pipfile yet, so that more doctests pass.

comment:172 in reply to: ↑ 170 Changed 4 months ago by gh-tobiasdiez

Replying to mkoeppe:

Also removing sage-env-config from the list of installed scripts is not good. Instead please review and merge #29850 - which reassigns this script to the sage_conf package.

I agree #29850 is a better solution. I had a look at that ticket, but don't really feel comfortable enough with the code to review it. Sorry! However, I've now added it as a dependency of this branch, and will merge as soon as it is reviewed.

comment:173 Changed 4 months ago by gh-tobiasdiez

  • Dependencies changed from #30779, #30672, #30673, #30709, #30706, #30901, #30748 to #30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850

comment:174 in reply to: ↑ 171 ; follow-up: Changed 4 months ago by gh-tobiasdiez

Replying to mkoeppe:

Replying to mkoeppe:

See if you can get doctesting to work on this branch - by merging #30578.

In #30578 I had to add a few more packages to requirements.txt that you don't have in Pipfile yet, so that more doctests pass.

Thanks for the pointer! I've now added these packages also to the pipfile. However, it is way beyond the scope of this ticket to make the doctest pass in the GH pipenv workflow. There are just so many things that need to be changed in order for the doctests to find the correct dependencies. For example, here are a few relevant error messages:

FileNotFoundError: [Errno 2] No such file or directory: '/mnt/d/Programming/Projects/sage/src/.venv/bin/gap'

RuntimeError: unable to start singular because the command 'Singular -t --ticks-per-sec 1000 --cntrlc=a' failed: The command was not found or was not executable: Singular.

RuntimeError: In order to initialize the database, /__w/sage/sage/src/.venv/share/conway_polynomials/conway_polynomials.p must exist.

I know that some of these errors would disappear if one use your recent progress towards installation of sage-build wheels.

Anyway, the aim of this ticket is to create a mostly-working editable install of sagelib. Making the doctests pass relative to this editable install in a venv should be subject of a follow-up ticket.

comment:175 Changed 4 months ago by git

  • Commit changed from 772baf208259ff75edebdbf6f340c700be0681ab to ad0f33cd4c67d2eb31cbf864e8112caeb8958990

Branch pushed to git repo; I updated commit sha1. New commits:

d3174ffFix merge problems
7e926c7Run compilation in parallel
488e6dcUpdate and extend dependencies
68c98d3Fix merge error for gap
f1b1f01Install singular package as well
ad0f33cRemove parallelization again (doesn't work)

comment:176 Changed 4 months ago by git

  • Commit changed from ad0f33cd4c67d2eb31cbf864e8112caeb8958990 to 77e1af16bcfce183bfe072cfb7324e82ee95a478

Branch pushed to git repo; I updated commit sha1. New commits:

20885b6MacOS: Find openblas
7f11466Correctly specify arb dependency not flint
223c2e4Mac: add flint as dependency
77e1af1Mac: Add arb

comment:177 in reply to: ↑ 174 Changed 4 months ago by mkoeppe

Replying to gh-tobiasdiez:

Anyway, the aim of this ticket is to create a mostly-working editable install of sagelib.

Most of the changes that you are doing in this ticket have really nothing to do with this goal. As I have pointed out previously, you are trying to solve the orthogonal problem of making sagelib work outside of the Sage distribution, i.e., without SAGE_LOCAL.

comment:178 follow-up: Changed 4 months ago by gh-tobiasdiez

I agree. The thing is that it is currently my only way to work with sage, and thus test the editable install. But it's good that you are pushing from the other side and work on the editable install based on a working sage distribution.

So, is there something that I can still do here? I see that verifying and testing that the editable install works is important. It's just not very realistic to have a fully functional doctest without SAGE_LOCAL at the moment. However, I can also not very actively contribute to the editable install with SAGE_LOCAL.

comment:179 in reply to: ↑ 178 ; follow-up: Changed 4 months ago by mkoeppe

Replying to gh-tobiasdiez:

I agree. The thing is that it is currently my only way to work with sage, and thus test the editable install.

Is the WSL build of the Sage distribution still failing for you? Have you tried a different Linux distribution? So far the WSL runs on GH Actions have not been very conclusive - they have failed for unrelated reasons - you could work on #30910 to improve it

comment:180 follow-up: Changed 4 months ago by mkoeppe

Also, I am interested in a GH Actions workflow for the various new installations (venv based on requirements on top of SAGE_LOCAL, venv based on Pipfile on top of SAGE_LOCAL, conda without SAGE_LOCAL, etc.) This ticket adds something like this, but I don't like that it does Ubuntu-specific things again. We have a better way of doing such things already so that everything can be tested on all supported distributions, not just some Ubuntu. I would hope that we can add such workflows in #30845. #30859 will help with this. The powershell bits to work with system packages could also be valuable - see #30865.

comment:181 in reply to: ↑ 179 Changed 4 months ago by gh-tobiasdiez

Replying to mkoeppe:

Is the WSL build of the Sage distribution still failing for you? Have you tried a different Linux distribution?

Yes, it's still not working. In a few weeks I'll get a new PC and then will try again afresh using WSL 2.0.

comment:182 in reply to: ↑ 180 Changed 4 months ago by gh-tobiasdiez

Replying to mkoeppe:

This ticket adds something like this, but I don't like that it does Ubuntu-specific things again. We have a better way of doing such things already so that everything can be tested on all supported distributions, not just some Ubuntu.

Yes and no. Since the whole workflow only takes about 2 hours, my hope was that one can use it in the longterm to run it on each push to each branch, replacing/complementing the patchbot. But of course it could also serve as the blueprint for a complete matrix integration test suite. Anyway, for the moment it serves as a record of the things I did locally on my system, and proofs that the editable install without SAGE_LOCAL works to a large degree.

comment:183 Changed 4 months ago by gh-tobiasdiez

  • Dependencies changed from #30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850 to #30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850, #29314

After playing around with it further I was able to make most of the tests pass. There are still a few hundreds that fail (see below for details), most of which are related to issues with maxima (TypeError?: unable to start maxima: End Of File (EOF). Exception style platform.) or fpylll (blocked due to https://github.com/fplll/fpylll/issues/185). Since most test pass and there are many subpackages for which all tests pass, I'd say that's enough "proof of concept" for this ticket and would leave the rest for follow-up tickets.

2020-11-17T19:31:00.7800025Z sage -t --random-seed=0 /__w/sage/sage/src/sage/interacts/debugger.py  # 2 doctests failed
2020-11-17T19:31:00.7801032Z sage -t --random-seed=0 /__w/sage/sage/src/sage/interfaces/singular.py  # 5 doctests failed
2020-11-17T19:31:00.7802028Z sage -t --random-seed=0 /__w/sage/sage/src/sage/interfaces/maxima_lib.py  # 1 doctest failed
2020-11-17T19:31:00.7803005Z sage -t --random-seed=0 /__w/sage/sage/src/sage/interfaces/quit.py  # 6 doctests failed
2020-11-17T19:31:00.7803974Z sage -t --random-seed=0 /__w/sage/sage/src/sage/interfaces/maxima.py  # 144 doctests failed
2020-11-17T19:31:00.7804998Z sage -t --random-seed=0 /__w/sage/sage/src/sage/interfaces/maxima_abstract.py  # 168 doctests failed
2020-11-17T19:31:00.7806039Z sage -t --random-seed=0 /__w/sage/sage/src/sage/interfaces/interface.py  # 90 doctests failed
2020-11-17T19:31:00.7807034Z sage -t --random-seed=0 /__w/sage/sage/src/sage/interfaces/expect.py  # 6 doctests failed
2020-11-17T19:31:00.7808001Z sage -t --random-seed=0 /__w/sage/sage/src/sage/interfaces/tests.py  # 3 doctests failed
2020-11-17T19:31:00.7809010Z sage -t --random-seed=0 /__w/sage/sage/src/sage/interfaces/gap_workspace.py  # 2 doctests failed
2020-11-17T19:31:00.7810009Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/rational.pyx  # 2 doctests failed
2020-11-17T19:31:00.7810921Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/ring.pyx  # 1 doctest failed
2020-11-17T19:31:00.7811830Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/infinity.py  # 1 doctest failed
2020-11-17T19:31:00.7812754Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/real_mpfr.pyx  # 9 doctests failed
2020-11-17T19:31:00.7813669Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/real_mpfi.pyx  # 3 doctests failed
2020-11-17T19:31:00.7814600Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/complex_mpfr.pyx  # 2 doctests failed
2020-11-17T19:31:00.7815569Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/localization.py  # 1 doctest failed
2020-11-17T19:31:00.7816785Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/tests.py  # 2 doctests failed
2020-11-17T19:31:00.7817714Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/integer_ring.pyx  # 5 doctests failed
2020-11-17T19:31:00.7818771Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/polynomial/polynomial_zmod_flint.pyx  # 1 doctest failed
2020-11-17T19:31:00.7819934Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/polynomial/polynomial_quotient_ring.py  # 1 doctest failed
2020-11-17T19:31:00.7821093Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/polynomial/polynomial_modn_dense_ntl.pyx  # 6 doctests failed
2020-11-17T19:31:00.7822245Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/polynomial/polynomial_element.pyx  # 13 doctests failed
2020-11-17T19:31:00.7823414Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/polynomial/polynomial_number_field.pyx  # 19 doctests failed
2020-11-17T19:31:00.7824560Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/polynomial/multi_polynomial_ring.py  # 2 doctests failed
2020-11-17T19:31:00.7825670Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/finite_rings/residue_field.pyx  # 14 doctests failed
2020-11-17T19:31:00.7826727Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/number_field/S_unit_solver.py  # 18 doctests failed
2020-11-17T19:31:00.7827787Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/number_field/number_field_base.pyx  # 15 doctests failed
2020-11-17T19:31:00.7828969Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/number_field/class_group.py  # 2 doctests failed
2020-11-17T19:31:00.7830020Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/number_field/number_field_ideal.py  # 1 doctest failed
2020-11-17T19:31:00.7831068Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/number_field/galois_group.py  # 58 doctests failed
2020-11-17T19:31:00.7832134Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/number_field/splitting_field.py  # 28 doctests failed
2020-11-17T19:31:00.7833172Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/number_field/order.py  # 6 doctests failed
2020-11-17T19:31:00.7834186Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/number_field/number_field.py  # 33 doctests failed
2020-11-17T19:31:00.7835200Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/number_field/unit_group.py  # 1 doctest failed
2020-11-17T19:31:00.7836234Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/number_field/number_field_rel.py  # 11 doctests failed
2020-11-17T19:31:00.7837488Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/number_field/number_field_element.pyx  # 27 doctests failed
2020-11-17T19:31:00.7838576Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/number_field/bdd_height.py  # 2 doctests failed
2020-11-17T19:31:00.7839663Z sage -t --random-seed=0 /__w/sage/sage/src/sage/rings/valuation/inductive_valuation.py  # 4 doctests failed
2020-11-17T19:31:00.7840725Z sage -t --random-seed=0 /__w/sage/sage/src/sage/cpython/dict_del_by_value.pyx  # 2 doctests failed
2020-11-17T19:31:00.7841710Z sage -t --random-seed=0 /__w/sage/sage/src/sage/databases/cremona.py  # 2 doctests failed
2020-11-17T19:31:00.7842673Z sage -t --random-seed=0 /__w/sage/sage/src/sage/groups/cubic_braid.py  # 6 doctests failed
2020-11-17T19:31:00.7843635Z sage -t --random-seed=0 /__w/sage/sage/src/sage/sandpiles/sandpile.py  # 5 doctests failed
2020-11-17T19:31:00.7844616Z sage -t --random-seed=0 /__w/sage/sage/src/sage/coding/gabidulin_code.py  # 1 doctest failed
2020-11-17T19:31:00.7845566Z sage -t --random-seed=0 /__w/sage/sage/src/sage/tests/cmdline.py  # 59 doctests failed
2020-11-17T19:31:00.7846502Z sage -t --random-seed=0 /__w/sage/sage/src/sage/tests/benchmark.py  # 4 doctests failed
2020-11-17T19:31:00.7847448Z sage -t --random-seed=0 /__w/sage/sage/src/sage/tests/gap_packages.py  # 1 doctest failed
2020-11-17T19:31:00.7848644Z sage -t --random-seed=0 /__w/sage/sage/src/sage/tests/books/judson-abstract-algebra/galois-sage.py  # 9 doctests failed
2020-11-17T19:31:00.7850157Z sage -t --random-seed=0 /__w/sage/sage/src/sage/tests/books/judson-abstract-algebra/fields-sage.py  # 17 doctests failed
2020-11-17T19:31:00.7851577Z sage -t --random-seed=0 /__w/sage/sage/src/sage/tests/books/judson-abstract-algebra/domains-sage.py  # 5 doctests failed
2020-11-17T19:31:00.7853321Z sage -t --random-seed=0 /__w/sage/sage/src/sage/tests/books/computational-mathematics-with-sagemath/polynomes_doctest.py  # 1 doctest failed
2020-11-17T19:31:00.7855341Z sage -t --random-seed=0 /__w/sage/sage/src/sage/tests/books/computational-mathematics-with-sagemath/domaines_doctest.py  # 3 doctests failed
2020-11-17T19:31:00.7857340Z sage -t --random-seed=0 /__w/sage/sage/src/sage/tests/books/computational-mathematics-with-sagemath/combinat_doctest.py  # 1 doctest failed
2020-11-17T19:31:00.7859360Z sage -t --random-seed=0 /__w/sage/sage/src/sage/tests/books/computational-mathematics-with-sagemath/integration_doctest.py  # 1 doctest failed
2020-11-17T19:31:00.7861360Z sage -t --random-seed=0 /__w/sage/sage/src/sage/tests/books/computational-mathematics-with-sagemath/mpoly_doctest.py  # 2 doctests failed
2020-11-17T19:31:00.7863355Z sage -t --random-seed=0 /__w/sage/sage/src/sage/tests/books/computational-mathematics-with-sagemath/calculus_doctest.py  # 3 doctests failed
2020-11-17T19:31:00.7865352Z sage -t --random-seed=0 /__w/sage/sage/src/sage/tests/books/computational-mathematics-with-sagemath/sol/mpoly_doctest.py  # 2 doctests failed
2020-11-17T19:31:00.7867492Z sage -t --random-seed=0 /__w/sage/sage/src/sage/tests/books/computational-mathematics-with-sagemath/sol/graphique_doctest.py  # 8 doctests failed
2020-11-17T19:31:00.7868993Z sage -t --random-seed=0 /__w/sage/sage/src/sage/repl/configuration.py  # 1 doctest failed
2020-11-17T19:31:00.7869942Z sage -t --random-seed=0 /__w/sage/sage/src/sage/repl/image.py  # 4 doctests failed
2020-11-17T19:31:00.7870875Z sage -t --random-seed=0 /__w/sage/sage/src/sage/repl/interpreter.py  # 7 doctests failed
2020-11-17T19:31:00.7871818Z sage -t --random-seed=0 /__w/sage/sage/src/sage/repl/preparse.py  # 1 doctest failed
2020-11-17T19:31:00.7872750Z sage -t --random-seed=0 /__w/sage/sage/src/sage/repl/ipython_tests.py  # 1 doctest failed
2020-11-17T19:31:00.7873726Z sage -t --random-seed=0 /__w/sage/sage/src/sage/repl/rich_output/pretty_print.py  # 3 doctests failed
2020-11-17T19:31:00.7874739Z sage -t --random-seed=0 /__w/sage/sage/src/sage/repl/ipython_kernel/install.py  # 1 doctest failed
2020-11-17T19:31:00.7875703Z sage -t --random-seed=0 /__w/sage/sage/src/sage/doctest/forker.py  # 1 doctest failed
2020-11-17T19:31:00.7876633Z sage -t --random-seed=0 /__w/sage/sage/src/sage/doctest/control.py  # 2 doctests failed
2020-11-17T19:31:00.7883133Z sage -t --random-seed=0 /__w/sage/sage/src/sage/doctest/test.py  # 3 doctests failed
2020-11-17T19:31:00.7884402Z sage -t --random-seed=0 /__w/sage/sage/src/sage/libs/singular/function.pyx  # 1 doctest failed
2020-11-17T19:31:00.7885406Z sage -t --random-seed=0 /__w/sage/sage/src/sage/libs/pari/convert_sage.pyx  # 10 doctests failed
2020-11-17T19:31:00.7886362Z sage -t --random-seed=0 /__w/sage/sage/src/sage/libs/pari/__init__.py  # 12 doctests failed
2020-11-17T19:31:00.7887265Z sage -t --random-seed=0 /__w/sage/sage/src/sage/libs/pari/tests.py  # 9 doctests failed
2020-11-17T19:31:00.7888179Z sage -t --random-seed=0 /__w/sage/sage/src/sage/libs/pynac/pynac.pyx  # 1 doctest failed
2020-11-17T19:31:00.7889240Z sage -t --random-seed=0 /__w/sage/sage/src/sage/dynamics/arithmetic_dynamics/generic_ds.py  # 10 doctests failed
2020-11-17T19:31:00.7890445Z sage -t --random-seed=0 /__w/sage/sage/src/sage/dynamics/arithmetic_dynamics/projective_ds.py  # 25 doctests failed
2020-11-17T19:31:00.7891469Z sage -t --random-seed=0 /__w/sage/sage/src/sage/arith/misc.py  # 20 doctests failed
2020-11-17T19:31:00.7892598Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/hyperelliptic_curves/hyperelliptic_generic.py  # 3 doctests failed
2020-11-17T19:31:00.7893789Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/product_projective/space.py  # 1 doctest failed
2020-11-17T19:31:00.7895096Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/product_projective/rational_point.py  # 1 doctest failed
2020-11-17T19:31:00.7896170Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/affine/affine_space.py  # 1 doctest failed
2020-11-17T19:31:00.7897209Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/generic/algebraic_scheme.py  # 1 doctest failed
2020-11-17T19:31:00.7898340Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/riemann_surfaces/riemann_surface.py  # 31 doctests failed
2020-11-17T19:31:00.7899522Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/projective/projective_rational_point.py  # 4 doctests failed
2020-11-17T19:31:00.7900680Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/projective/projective_space.py  # 1 doctest failed
2020-11-17T19:31:00.7901771Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/toric/toric_subscheme.py  # 11 doctests failed
2020-11-17T19:31:00.7902793Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/toric/divisor.py  # 2 doctests failed
2020-11-17T19:31:00.7903768Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/toric/points.py  # 1 doctest failed
2020-11-17T19:31:00.7904739Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/toric/variety.py  # 4 doctests failed
2020-11-17T19:31:00.7905707Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/toric/ideal.py  # 43 doctests failed
2020-11-17T19:31:00.7906825Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/heegner.py  # 33 doctests failed
2020-11-17T19:31:00.7907884Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/kraus.py  # 4 doctests failed
2020-11-17T19:31:00.7908978Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/constructor.py  # 4 doctests failed
2020-11-17T19:31:00.7910128Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/gal_reps_number_field.py  # 2 doctests failed
2020-11-17T19:31:00.7911278Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/ell_rational_field.py  # 14 doctests failed
2020-11-17T19:31:00.7912397Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/isogeny_class.py  # 2 doctests failed
2020-11-17T19:31:00.7913509Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/ell_number_field.py  # 26 doctests failed
2020-11-17T19:31:00.7914601Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/gal_reps.py  # 2 doctests failed
2020-11-17T19:31:00.7915708Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/isogeny_small_degree.py  # 1 doctest failed
2020-11-17T19:31:00.7916803Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/ell_egros.py  # 4 doctests failed
2020-11-17T19:31:00.7918998Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/ell_curve_isogeny.py  # 1 doctest failed
2020-11-17T19:31:00.7920093Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/ell_field.py  # 2 doctests failed
2020-11-17T19:31:00.7921146Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/plane_conics/con_field.py  # 7 doctests failed
2020-11-17T19:31:00.7922203Z sage -t --random-seed=0 /__w/sage/sage/src/sage/schemes/plane_conics/con_number_field.py  # 3 doctests failed
2020-11-17T19:31:00.7923188Z sage -t --random-seed=0 /__w/sage/sage/src/sage/features/__init__.py  # 1 doctest failed
2020-11-17T19:31:00.7924137Z sage -t --random-seed=0 /__w/sage/sage/src/sage/calculus/desolvers.py  # 8 doctests failed
2020-11-17T19:31:00.7925128Z sage -t --random-seed=0 /__w/sage/sage/src/sage/calculus/calculus.py  # 25 doctests failed
2020-11-17T19:31:00.7926071Z sage -t --random-seed=0 /__w/sage/sage/src/sage/crypto/lattice.py  # 1 doctest failed
2020-11-17T19:31:00.7927011Z sage -t --random-seed=0 /__w/sage/sage/src/sage/combinat/tutorial.py  # 1 doctest failed
2020-11-17T19:31:00.7927975Z sage -t --random-seed=0 /__w/sage/sage/src/sage/combinat/combinat.py  # 5 doctests failed
2020-11-17T19:31:00.7929092Z sage -t --random-seed=0 /__w/sage/sage/src/sage/combinat/degree_sequences.pyx  # 1 doctest failed
2020-11-17T19:31:00.7930148Z sage -t --random-seed=0 /__w/sage/sage/src/sage/categories/algebra_functor.py  # 5 doctests failed
2020-11-17T19:31:00.7931175Z sage -t --random-seed=0 /__w/sage/sage/src/sage/categories/number_fields.py  # 1 doctest failed
2020-11-17T19:31:00.7932168Z sage -t --random-seed=0 /__w/sage/sage/src/sage/categories/rings.py  # 12 doctests failed
2020-11-17T19:31:00.7933122Z sage -t --random-seed=0 /__w/sage/sage/src/sage/categories/primer.py  # 1 doctest failed
2020-11-17T19:31:00.7934130Z sage -t --random-seed=0 /__w/sage/sage/src/sage/categories/quotient_fields.py  # 7 doctests failed
2020-11-17T19:31:00.7935102Z sage -t --random-seed=0 /__w/sage/sage/src/sage/modular/cusps_nf.py  # 1 doctest failed
2020-11-17T19:31:00.7936063Z sage -t --random-seed=0 /__w/sage/sage/src/sage/modular/etaproducts.py  # 10 doctests failed
2020-11-17T19:31:00.7937057Z sage -t --random-seed=0 /__w/sage/sage/src/sage/modular/modsym/p1list_nf.py  # 1 doctest failed
2020-11-17T19:31:00.7938011Z sage -t --random-seed=0 /__w/sage/sage/src/sage/structure/formal_sum.py  # 1 doctest failed
2020-11-17T19:31:00.7938985Z sage -t --random-seed=0 /__w/sage/sage/src/sage/structure/element.pyx  # 5 doctests failed
2020-11-17T19:31:00.7940062Z sage -t --random-seed=0 /__w/sage/sage/src/sage/structure/dynamic_class.py  # 1 doctest failed
2020-11-17T19:31:00.7941022Z sage -t --random-seed=0 /__w/sage/sage/src/sage/misc/nested_class.pyx  # 1 doctest failed
2020-11-17T19:31:00.7941915Z sage -t --random-seed=0 /__w/sage/sage/src/sage/misc/trace.py  # 3 doctests failed
2020-11-17T19:31:00.7942828Z sage -t --random-seed=0 /__w/sage/sage/src/sage/misc/sage_ostools.pyx  # 1 doctest failed
2020-11-17T19:31:00.7943764Z sage -t --random-seed=0 /__w/sage/sage/src/sage/misc/functional.py  # 1 doctest failed
2020-11-17T19:31:00.7944692Z sage -t --random-seed=0 /__w/sage/sage/src/sage/misc/edit_module.py  # 1 doctest failed
2020-11-17T19:31:00.7945625Z sage -t --random-seed=0 /__w/sage/sage/src/sage/misc/temporary_file.py  # 1 doctest failed
2020-11-17T19:31:00.7946578Z sage -t --random-seed=0 /__w/sage/sage/src/sage/misc/lazy_attribute.pyx  # 1 doctest failed
2020-11-17T19:31:00.7947551Z sage -t --random-seed=0 /__w/sage/sage/src/sage/misc/sageinspect.py  # 13 doctests failed
2020-11-17T19:31:00.7948497Z sage -t --random-seed=0 /__w/sage/sage/src/sage/misc/sagedoc.py  # 3 doctests failed
2020-11-17T19:31:00.7949390Z sage -t --random-seed=0 /__w/sage/sage/src/sage/misc/compat.py  # 1 doctest failed
2020-11-17T19:31:00.7950304Z sage -t --random-seed=0 /__w/sage/sage/src/sage/misc/sphinxify.py  # 4 doctests failed
2020-11-17T19:31:00.7951326Z sage -t --random-seed=0 /__w/sage/sage/src/sage/functions/hypergeometric.py  # 7 doctests failed
2020-11-17T19:31:00.7952357Z sage -t --random-seed=0 /__w/sage/sage/src/sage/functions/bessel.py  # 7 doctests failed
2020-11-17T19:31:00.7953305Z sage -t --random-seed=0 /__w/sage/sage/src/sage/functions/gamma.py  # 2 doctests failed
2020-11-17T19:31:00.7954306Z sage -t --random-seed=0 /__w/sage/sage/src/sage/functions/special.py  # 1 doctest failed
2020-11-17T19:31:00.7955238Z sage -t --random-seed=0 /__w/sage/sage/src/sage/functions/trig.py  # 1 doctest failed
2020-11-17T19:31:00.7956168Z sage -t --random-seed=0 /__w/sage/sage/src/sage/functions/error.py  # 7 doctests failed
2020-11-17T19:31:00.7957101Z sage -t --random-seed=0 /__w/sage/sage/src/sage/functions/log.py  # 6 doctests failed
2020-11-17T19:31:00.7958207Z sage -t --random-seed=0 /__w/sage/sage/src/sage/functions/other.py  # 15 doctests failed
2020-11-17T19:31:00.7959216Z sage -t --random-seed=0 /__w/sage/sage/src/sage/functions/orthogonal_polys.py  # 11 doctests failed
2020-11-17T19:31:00.7960272Z sage -t --random-seed=0 /__w/sage/sage/src/sage/geometry/polyhedron/parent.py  # 2 doctests failed
2020-11-17T19:31:00.7961352Z sage -t --random-seed=0 /__w/sage/sage/src/sage/geometry/polyhedron/palp_database.py  # 1 doctest failed
2020-11-17T19:31:00.7962568Z sage -t --random-seed=0 /__w/sage/sage/src/sage/algebras/quaternion_algebra_element.py  # 4 doctests failed
2020-11-17T19:31:00.7963697Z sage -t --random-seed=0 /__w/sage/sage/src/sage/algebras/quatalg/quaternion_algebra.py  # 6 doctests failed
2020-11-17T19:31:00.7964873Z sage -t --random-seed=0 /__w/sage/sage/src/sage/algebras/quatalg/quaternion_algebra_element.pyx  # 38 doctests failed
2020-11-17T19:31:00.7965949Z sage -t --random-seed=0 /__w/sage/sage/src/sage/modules/free_module.py  # 5 doctests failed
2020-11-17T19:31:00.7966956Z sage -t --random-seed=0 /__w/sage/sage/src/sage/modules/free_module_morphism.py  # 3 doctests failed
2020-11-17T19:31:00.7967988Z sage -t --random-seed=0 /__w/sage/sage/src/sage/modules/free_module_integer.py  # 57 doctests failed
2020-11-17T19:31:00.7969027Z sage -t --random-seed=0 /__w/sage/sage/src/sage/modules/vector_space_morphism.py  # 3 doctests failed
2020-11-17T19:31:00.7970041Z sage -t --random-seed=0 /__w/sage/sage/src/sage/modules/fg_pid/fgp_module.py  # 3 doctests failed
2020-11-17T19:31:00.7971051Z sage -t --random-seed=0 /__w/sage/sage/src/sage/symbolic/expression.pyx  # 17 doctests failed
2020-11-17T19:31:00.7972071Z sage -t --random-seed=0 /__w/sage/sage/src/sage/symbolic/assumptions.py  # 3 doctests failed
2020-11-17T19:31:00.7973064Z sage -t --random-seed=0 /__w/sage/sage/src/sage/symbolic/constants_c.pyx  # 1 doctest failed
2020-11-17T19:31:00.7974140Z sage -t --random-seed=0 /__w/sage/sage/src/sage/symbolic/maxima_wrapper.py  # 2 doctests failed
2020-11-17T19:31:00.7975123Z sage -t --random-seed=0 /__w/sage/sage/src/sage/symbolic/relation.py  # 4 doctests failed
2020-11-17T19:31:00.7976102Z sage -t --random-seed=0 /__w/sage/sage/src/sage/symbolic/constants.py  # 17 doctests failed
2020-11-17T19:31:00.7977173Z sage -t --random-seed=0 /__w/sage/sage/src/sage/symbolic/expression_conversions.py  # 10 doctests failed
2020-11-17T19:31:00.7978261Z sage -t --random-seed=0 /__w/sage/sage/src/sage/graphs/comparability.pyx  # 1 doctest failed
2020-11-17T19:31:00.7979313Z sage -t --random-seed=0 /__w/sage/sage/src/sage/graphs/digraph_generators.py  # 8 doctests failed
2020-11-17T19:31:00.7980236Z sage -t --random-seed=0 /__w/sage/sage/src/sage/graphs/graph.py  # Timed out
2020-11-17T19:31:00.7981184Z sage -t --random-seed=0 /__w/sage/sage/src/sage/graphs/connectivity.pyx  # 2 doctests failed
2020-11-17T19:31:00.7982192Z sage -t --random-seed=0 /__w/sage/sage/src/sage/graphs/graph_generators.py  # Timed out
2020-11-17T19:31:00.7983217Z sage -t --random-seed=0 /__w/sage/sage/src/sage/graphs/hypergraph_generators.py  # 9 doctests failed
2020-11-17T19:31:00.7984259Z sage -t --random-seed=0 /__w/sage/sage/src/sage/graphs/graph_coloring.pyx  # 2 doctests failed
2020-11-17T19:31:00.7985206Z sage -t --random-seed=0 /__w/sage/sage/src/sage/graphs/genus.pyx  # 1 doctest failed
2020-11-17T19:31:00.7986147Z sage -t --random-seed=0 /__w/sage/sage/src/sage/graphs/generic_graph_pyx.pyx  # 2 doctests failed
2020-11-17T19:31:00.7987202Z sage -t --random-seed=0 /__w/sage/sage/src/sage/sets/set_from_iterator.py  # Timed out (and interrupt failed)
2020-11-17T19:31:00.7988200Z sage -t --random-seed=0 /__w/sage/sage/src/sage/matrix/matrix1.pyx  # 3 doctests failed
2020-11-17T19:31:00.7989131Z sage -t --random-seed=0 /__w/sage/sage/src/sage/matrix/matrix2.pyx  # 3 doctests failed
2020-11-17T19:31:00.7990123Z sage -t --random-seed=0 /__w/sage/sage/src/sage/matrix/matrix_rational_dense.pyx  # 1 doctest failed
2020-11-17T19:31:00.7991189Z sage -t --random-seed=0 /__w/sage/sage/src/sage/matrix/matrix_integer_dense.pyx  # 14 doctests failed
2020-11-17T19:31:00.7992217Z sage -t --random-seed=0 /__w/sage/sage/src/sage/matrix/matrix_symbolic_dense.pyx  # 1 doctest failed

https://github.com/tobiasdiez/sage/runs/1413136300?check_suite_focus=true

comment:184 Changed 4 months ago by git

  • Commit changed from 77e1af16bcfce183bfe072cfb7324e82ee95a478 to d1341b0ab3b8bbc47a25beea2a8392632c981bfb

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

9a5a145version bump
d4c4453remove old workaround from #22626
ebe9780this file should exist - leave just comments there
a507fb9trac 29314: fix version numbers in patches for gap_packages
9836f79move GAP's PackageManager to gap from gap_packages
435d82fMerge branch 'u/dimpase/packages/gap411' of git://trac.sagemath.org/sage into public/build/inplace
2fc8bf5Hangs with maxima, so remove again
f6c1d87Try to install maxima again
13e96adFix workflow syntax
d1341b0Can we now activate cysignals again?

comment:185 Changed 4 months ago by gh-tobiasdiez

  • Dependencies changed from #30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850, #29314 to #30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850, #29314, #30937, #30936, #30935, #30934

comment:186 Changed 3 months ago by gh-tobiasdiez

  • Dependencies changed from #30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850, #29314, #30937, #30936, #30935, #30934 to #30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850, #29314, #30937, #30936, #30935, #30934, #30859, #30923, #30910

comment:187 Changed 3 months ago by gh-tobiasdiez

  • Description modified (diff)

comment:188 Changed 3 months ago by gh-tobiasdiez

  • Dependencies changed from #30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850, #29314, #30937, #30936, #30935, #30934, #30859, #30923, #30910 to #30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850, #29314, #30937, #30936, #30935, #30934, #30859, #30923, #30910, #30944

comment:189 Changed 3 months ago by gh-tobiasdiez

  • Dependencies changed from #30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850, #29314, #30937, #30936, #30935, #30934, #30859, #30923, #30910, #30944 to #30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850, #29314, #30937, #30936, #30935, #30731 , #30859, #30923, #30910, #30944

comment:190 Changed 3 months ago by mkoeppe

  • Keywords sd111 added

Hoping we can make progress on this ticket this week - https://wiki.sagemath.org/days111

comment:191 Changed 3 months ago by mkoeppe

  • Status changed from needs_review to needs_work

comment:192 Changed 3 months ago by git

  • Commit changed from d1341b0ab3b8bbc47a25beea2a8392632c981bfb to 17e06daef346efc2f83f7b5304fc598c181079de

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

4393801Build all of sage deps
2027ebbRun make in parallel
405e8d2Install dependencies on mac with tox as well
894b9d5Fix indent
4cefc89Always run manually to see what's going on
ee6aaebDon't require spkg-configure.m4 file
8cc37e7Cleanup mac workflow
c571a07Fix maxima install
8ee33f2Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace
17e06daUpgrade packages

comment:193 Changed 3 months ago by gh-tobiasdiez

From my side, this is actually ready for review. I've been using it now constantly the last few weeks and it is working. Or do you have something that I should be improving/working on?

comment:194 Changed 3 months ago by dimpase

  • Description modified (diff)

comment:195 Changed 3 months ago by git

  • Commit changed from 17e06daef346efc2f83f7b5304fc598c181079de to 5c719fe14a0fa6a0aa966031786cf6f3c437b239

Branch pushed to git repo; I updated commit sha1. New commits:

3fcaf5fMerge branch 'develop' of git://github.com/sagemath/sage into public/build/multiarchsimple
090e6f1Simplify code
fa4556aRemove _get_sage_local
7cc2005Merge branch 'public/build/multiarchsimple' of git://trac.sagemath.org/sage into public/build/inplace
a590924Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace
afa0518Make autogen more stable
5c719feFix functional.py:56: DeprecationWarning: invalid escape sequence \d

comment:196 Changed 3 months ago by gh-tobiasdiez

  • Status changed from needs_work to needs_review

Merged the latest develop branch.

comment:197 Changed 3 months ago by gh-tobiasdiez

  • Description modified (diff)

comment:198 Changed 3 months ago by git

  • Commit changed from 5c719fe14a0fa6a0aa966031786cf6f3c437b239 to 7f0335de97d12b1bd5e6cc56879ab47f7423665a

Branch pushed to git repo; I updated commit sha1. New commits:

5e4f607Change import exception to print message
a5eb21dFix doctest
7f0335dMerge branch 'public/build/startupWarningMinimal' of git://trac.sagemath.org/sage into public/build/inplace

comment:199 Changed 3 months ago by gh-tobiasdiez

  • Dependencies changed from #30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850, #29314, #30937, #30936, #30935, #30731 , #30859, #30923, #30910, #30944 to #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30936, #30935, #30731 , #30859, #30923, #30910, #30944

New commits:

5e4f607Change import exception to print message
a5eb21dFix doctest
7f0335dMerge branch 'public/build/startupWarningMinimal' of git://trac.sagemath.org/sage into public/build/inplace

comment:200 in reply to: ↑ 156 Changed 2 months ago by mkoeppe

Replying to mkoeppe:

It has been reported that the current src/setup.py (in Sage 9.2) is able to install a working sagelib on top of conda (see #28745).

I'll create a test for this in #30845, and then we can test the present ticket to make sure it does not break anything.

Ensuring that the feature of installing sagelib on top of conda is not broken is a prerequisite for merging this ticket.

To move this ticket forward, could you help me with this GH Actions test? (This should be on top of #30944, which still needs review)

I think there should be one new workflow for all tests that do not build the Sage distribution (no build/pkgs).

comment:201 follow-ups: Changed 2 months ago by gh-tobiasdiez

Sure, I will have a look at this. Could you by extending #30845 to include in more detail how the manual workflow with conda looks like (I've no experience with conda) and why #30944 is needed.

Maybe it's a good idea to review how the GH workflows are structured and revise this if necessary. For example, focus more on the purpose vs how it done, e.g. "test that building packages in build/pgks" works, "test that sagelib installed on top of various setups works". For the latter, my hope with pipenv was actually that one can easily run it on every push on every branch as a gateway in addition to the patchbot (since the time it takes is very minimal, time of setup of < 1h, + time for doctests).

comment:202 in reply to: ↑ 201 Changed 2 months ago by mkoeppe

Replying to gh-tobiasdiez:

Sure, I will have a look at this. Could you by extending #30845 to include in more detail how the manual workflow with conda looks like (I've no experience with conda) and

Done.

why #30944 is needed.

Well, it will be testing something that currently happens to work but is not automatically tested.

Maybe it's a good idea to review how the GH workflows are structured and revise this if necessary.

Sure, I'll be happy about more eyes on the workflows.

For example, focus more on the purpose vs how it done

All workflows test from-scratch builds of the Sage distribution. Two workflows separately test optional / experimental packages. That's really all.

comment:203 in reply to: ↑ 201 Changed 2 months ago by mkoeppe

Replying to gh-tobiasdiez:

"test that sagelib installed on top of various setups works". For the latter, my hope with pipenv was actually that one can easily run it on every push on every branch as a gateway in addition to the patchbot (since the time it takes is very minimal, time of setup of < 1h, + time for doctests).

An incremental build setup based on Docker has already been developed, which drives the current GitLab workflow. I'd recommend to start from there (it needs attention) instead of developing another thing from scratch... Take a look at #29536

comment:204 Changed 2 months ago by git

  • Commit changed from 7f0335de97d12b1bd5e6cc56879ab47f7423665a to f96b923fd5c342c5401f4e8f993546b523fe2cc0

Branch pushed to git repo; I updated commit sha1. New commits:

f96b923Use sage_conf

comment:205 Changed 2 months ago by gh-tobiasdiez

  • Dependencies changed from #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30936, #30935, #30731 , #30859, #30923, #30910, #30944 to #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944

comment:206 Changed 2 months ago by git

  • Commit changed from f96b923fd5c342c5401f4e8f993546b523fe2cc0 to 8c97e7c9290c067af0e0bd974f60f4b3966a516c

Branch pushed to git repo; I updated commit sha1. New commits:

8c97e7cCleanup code

comment:207 Changed 2 months ago by gh-tobiasdiez

Since now most dependencies are merged in the develop branch, this ticket is now definitely ready for review.

comment:208 Changed 8 weeks ago by git

  • Commit changed from 8c97e7c9290c067af0e0bd974f60f4b3966a516c to f871fc77be3b6ec719dc3f64a033ca78e6deea67

Branch pushed to git repo; I updated commit sha1. New commits:

71b4c06Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace
f871fc7Add rstcheck

comment:209 follow-ups: Changed 8 weeks ago by gh-tobiasdiez

Matthias, I have a question: the current code works very well for system-installed packages. However, there are a few packages that are required but for which system-installed versions are not currently not supported (e.g. ecl). How could I use the sage-built versions of these packages for the built process of sagelib, i.e. for pip install -e .?

Take for example the import cdef extern from "ecl/ecl.h" in libs/ecl.pxd. I guess one needs to pass the path of local/include/ecl as argument to the C++ compiler. How is currently done?

comment:210 in reply to: ↑ 209 ; follow-ups: Changed 8 weeks ago by mkoeppe

Replying to gh-tobiasdiez:

Take for example the import cdef extern from "ecl/ecl.h" in libs/ecl.pxd. I guess one needs to pass the path of local/include/ecl as argument to the C++ compiler. How is currently done?

We install all packages that are not accepted from the system into $SAGE_LOCAL, and scripts such as sage-env and sage-build-env set up environment variables such as LIBRARY_PATH and CPATH so that compilers find stuff. Configuration for Cython is somewhat separate - you are already familiar with the cython_aliases function.

For ECL, there's ticket #30770 to make this configuration more flexible.

comment:211 Changed 8 weeks ago by mkoeppe

Also, if you are able to use system packages for which we do not have system package detection yet (spkg-configure.m4), you can join the effort in #27330.

comment:212 in reply to: ↑ 209 ; follow-up: Changed 8 weeks ago by mkoeppe

Replying to gh-tobiasdiez:

How could I use the sage-built versions of these packages for the built process of sagelib, i.e. for pip install -e .?

One part of the answer to this involves the sage_conf package: Installing it into your virtual environment provides sage-env-config and the sage_conf module. So you may want to add it to Pipfile. (See also #30913).

comment:213 in reply to: ↑ 210 Changed 8 weeks ago by gh-tobiasdiez

Replying to mkoeppe:

For ECL, there's ticket #30770 to make this configuration more flexible.

Thanks! I'll see if I can make progress towards this.

comment:214 in reply to: ↑ 212 Changed 8 weeks ago by gh-tobiasdiez

Replying to mkoeppe:

One part of the answer to this involves the sage_conf package: Installing it into your virtual environment provides sage-env-config and the sage_conf module. So you may want to add it to Pipfile. (See also #30913).

That's already done:

+[dev-packages.8c46424]
+path = "./../build/pkgs/sage_conf/src"
+editable = true

comment:215 Changed 8 weeks ago by gh-tobiasdiez

  • Dependencies changed from #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944 to #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770

comment:216 Changed 8 weeks ago by git

  • Commit changed from f871fc77be3b6ec719dc3f64a033ca78e6deea67 to deeeb75c316c0e06c40e36ec65f472d4d3b90972

Branch pushed to git repo; I updated commit sha1. New commits:

8920ad5Update dependencies
3049e53Use ecl-config to determine compiler/linker flags for ecl
deeeb75Merge branch 'public/build/ecl-config' of git://trac.sagemath.org/sage into public/build/inplace

comment:217 Changed 8 weeks ago by gh-tobiasdiez

  • Description modified (diff)

comment:218 Changed 8 weeks ago by git

  • Commit changed from deeeb75c316c0e06c40e36ec65f472d4d3b90972 to 2ea4935dbae81bf1f2c9f9dc4e29f2feeb1a733f

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

9988c5fci-cygwin*.yml: Adjust to new script packages _bootstrap, _prereq
ab19133Merge branch 't/29124/script-packages-prereq-toolchain-bootstrap' into t/30944/tox__improve_local_sudo_ubuntu_standard
c6ad06eMerge branch 't/30944/tox__improve_local_sudo_ubuntu_standard' into t/31216/tox_ini__local___add_environment_variables_to_skip_system_package_installs__mechanism_for_a_local_interactive_shell
13f58fctox.ini (local): Run an interactive shell if posargs are bash
ace4ef6tox.ini (local): Add option SKIP_SYSTEM_PKG_INSTALL=1
ddc4f0btox.ini (local): Handle environment variables SKIP_BOOTSTRAP, SKIP_CONFIGURE
bb6aec4src/doc/en/developer/portability_testing.rst: Updates
6253bd8src/doc/en/developer/portability_testing.rst: New section on SKIP_... environment variables
a52e899Merge branch 'u/mkoeppe/tox_ini__local___add_environment_variables_to_skip_system_package_installs__mechanism_for_a_local_interactive_shell' of git://trac.sagemath.org/sage into public/build/inplace
2ea4935Split workflow in install and build

comment:219 Changed 8 weeks ago by gh-tobiasdiez

  • Dependencies changed from #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770 to #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216

comment:220 Changed 8 weeks ago by git

  • Commit changed from 2ea4935dbae81bf1f2c9f9dc4e29f2feeb1a733f to 6bcc5c34991dc4b81362b2427ff8d38b50f32c7e

Branch pushed to git repo; I updated commit sha1. New commits:

6bcc5c3Add esac

comment:221 Changed 8 weeks ago by git

  • Commit changed from 6bcc5c34991dc4b81362b2427ff8d38b50f32c7e to d32b22b7091051522180ea58ea861b022e16e4d7

Branch pushed to git repo; I updated commit sha1. New commits:

d32b22bTry with config

comment:222 Changed 8 weeks ago by git

  • Commit changed from d32b22b7091051522180ea58ea861b022e16e4d7 to 73c99f3b7f797a7ab61507392c40268bb50e0a00

Branch pushed to git repo; I updated commit sha1. New commits:

73c99f3Ignore errors for now

comment:223 Changed 8 weeks ago by git

  • Commit changed from 73c99f3b7f797a7ab61507392c40268bb50e0a00 to 5cb58cf579b41780791b9bf0cb46563713efd1e5

Branch pushed to git repo; I updated commit sha1. New commits:

5cb58cfEnable build as root

comment:224 Changed 8 weeks ago by git

  • Commit changed from 5cb58cf579b41780791b9bf0cb46563713efd1e5 to 02542220efcfc3989a682ed082092c7f6e9a1c30

Branch pushed to git repo; I updated commit sha1. New commits:

0254222Cleanup github workflow

comment:225 Changed 8 weeks ago by git

  • Commit changed from 02542220efcfc3989a682ed082092c7f6e9a1c30 to 4dcd083615e7eb7218b595c61fd9b9a4754222cb

Branch pushed to git repo; I updated commit sha1. New commits:

4dcd083Use --all for the tests

comment:226 Changed 8 weeks ago by git

  • Commit changed from 4dcd083615e7eb7218b595c61fd9b9a4754222cb to d30fda0a1bd9682d624feed871c3c94fa53921df

Branch pushed to git repo; I updated commit sha1. New commits:

d30fda0Also install maxima

comment:227 Changed 8 weeks ago by git

  • Commit changed from d30fda0a1bd9682d624feed871c3c94fa53921df to b17e6faef915267735b95135e326ffc74fd558b6

Branch pushed to git repo; I updated commit sha1. New commits:

b17e6faUse configure.ac as make target

comment:228 in reply to: ↑ 210 ; follow-up: Changed 8 weeks ago by gh-tobiasdiez

Replying to mkoeppe:

For ECL, there's ticket #30770 to make this configuration more flexible.

This seems to work now. Thanks for you help!

Two follow-up questions:

  1. Is there a make target similar to sagelib-build-deps that doesn't install pythonic packages such as cython.
  2. Why doesn't sagelib-build-deps install maxima?

comment:229 Changed 8 weeks ago by git

  • Commit changed from b17e6faef915267735b95135e326ffc74fd558b6 to 5d64c5a44b44e36c260eca75c60e179093f0b67f

Branch pushed to git repo; I updated commit sha1. New commits:

5d64c5aInstall/fix other packages

comment:230 Changed 8 weeks ago by git

  • Commit changed from 5d64c5a44b44e36c260eca75c60e179093f0b67f to dbf2df2f53f7b1b7e7760f0fd578f86a44c9df4b

Branch pushed to git repo; I updated commit sha1. New commits:

dbf2df2Install more dependencies manually

comment:231 in reply to: ↑ 228 Changed 8 weeks ago by mkoeppe

Replying to gh-tobiasdiez:

  1. Is there a make target similar to sagelib-build-deps that doesn't install pythonic packages such as cython.

Not yet. #30896 will go into this direction

  1. Why doesn't sagelib-build-deps install maxima?

maxima is only needed at runtime of Sage, not while installing sagelib.

comment:232 Changed 8 weeks ago by git

  • Commit changed from dbf2df2f53f7b1b7e7760f0fd578f86a44c9df4b to b66df86359a2ee8be8ab481d0aea20b8960d0ef9

Branch pushed to git repo; I updated commit sha1. New commits:

b66df86Run tests in parallel

comment:233 Changed 7 weeks ago by git

  • Commit changed from b66df86359a2ee8be8ab481d0aea20b8960d0ef9 to 1fdfe680752f2878c76234733dc34780e3107715

Branch pushed to git repo; I updated commit sha1. New commits:

5458776Disable timeout in tests
1fdfe68Install fpylll

comment:234 Changed 7 weeks ago by git

  • Commit changed from 1fdfe680752f2878c76234733dc34780e3107715 to f0bd45cf7ff1ac240bfd10af8844df187c1f97f4

Branch pushed to git repo; I updated commit sha1. New commits:

f0bd45cInstall singular-data to find gftables?

comment:235 Changed 7 weeks ago by git

  • Commit changed from f0bd45cf7ff1ac240bfd10af8844df187c1f97f4 to 8a401efbb30b1049b05af3b4e756087d64968505

Branch pushed to git repo; I updated commit sha1. New commits:

0b18fd031029: upgrade cypari2 to 2.1.2
b9aadfd31029: adapt sage library code to cypari2 version 2.1.2
8a401efMerge branch 'u/vdelecroix/31029' of git://trac.sagemath.org/sage into public/build/inplace

comment:236 Changed 7 weeks ago by git

  • Commit changed from 8a401efbb30b1049b05af3b4e756087d64968505 to 22e8fe7c5ab241996824216396b9025997c939be

Branch pushed to git repo; I updated commit sha1. New commits:

f6c3285Workflow cleanup
22e8fe7Set high timeout

comment:237 Changed 7 weeks ago by gh-tobiasdiez

  • Dependencies changed from #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216 to #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029

Thanks for the help! I've now improved the github workflow and the doctest pass now (except a handful). So I would appreciate if this ticket could be review!

https://github.com/sagemath/sagetrac-mirror/actions?query=workflow%3A%22Build+%26+Test+using+pipenv%22

Last edited 7 weeks ago by gh-tobiasdiez (previous) (diff)

comment:238 Changed 7 weeks ago by git

  • Commit changed from 22e8fe7c5ab241996824216396b9025997c939be to 3d071819a98f2d8f8e50935223d38042e04923e1

Branch pushed to git repo; I updated commit sha1. New commits:

3d07181Run tests in serial again

comment:239 Changed 7 weeks ago by git

  • Commit changed from 3d071819a98f2d8f8e50935223d38042e04923e1 to cd38e05d1b88d4879bd65aa32eb1614356569802

Branch pushed to git repo; I updated commit sha1. New commits:

cd38e05Don't set SAGE_LOCAL in workflow

comment:240 Changed 7 weeks ago by git

  • Commit changed from cd38e05d1b88d4879bd65aa32eb1614356569802 to e42e1249042e31381f40889cef835e5947add799

Branch pushed to git repo; I updated commit sha1. New commits:

e42e124Update deps

comment:241 Changed 7 weeks ago by git

  • Commit changed from e42e1249042e31381f40889cef835e5947add799 to d1079de570eb56f0031153286ed3ff79e273986c

Branch pushed to git repo; I updated commit sha1. New commits:

2d84b5atox.ini: Add missing esac
3fd16f4Merge branch 'u/mkoeppe/tox_ini__local___add_environment_variables_to_skip_system_package_installs__mechanism_for_a_local_interactive_shell' of git://trac.sagemath.org/sage into public/build/inplace
d1079deMerge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace

comment:242 follow-up: Changed 6 weeks ago by mkoeppe

Instead of introducing a new environment variable SAGE_NUM_BUILD_JOBS, probably best to use SAGE_NUM_THREADS, which is used already in the same source tree.

comment:243 in reply to: ↑ 242 Changed 6 weeks ago by gh-tobiasdiez

Replying to mkoeppe:

Instead of introducing a new environment variable SAGE_NUM_BUILD_JOBS, probably best to use SAGE_NUM_THREADS, which is used already in the same source tree.

Ok, done!

comment:244 Changed 6 weeks ago by git

  • Commit changed from d1079de570eb56f0031153286ed3ff79e273986c to d798925a1bb48a96087c3db03950e90059c9c37b

Branch pushed to git repo; I updated commit sha1. New commits:

54ec549Reuse SAGE_NUM_THREADS
d798925Update dependencies

comment:245 Changed 6 weeks ago by git

  • Commit changed from d798925a1bb48a96087c3db03950e90059c9c37b to cd8a8d19ea80b99357c465b4e1ff3f285a8b26b5

Branch pushed to git repo; I updated commit sha1. New commits:

cd8a8d1Move pgkconfig to normal package

comment:246 Changed 6 weeks ago by git

  • Commit changed from cd8a8d19ea80b99357c465b4e1ff3f285a8b26b5 to 63c7997e078bb7b3a1ab39ef1a2c29d210376f71

Branch pushed to git repo; I updated commit sha1. New commits:

63c7997Make cython and sage_conf also normal packages

comment:247 Changed 6 weeks ago by git

  • Commit changed from 63c7997e078bb7b3a1ab39ef1a2c29d210376f71 to a18b3c59eff2940159fdb930eea4af04b444e828

Branch pushed to git repo; I updated commit sha1. New commits:

87e2992Move ipywidgets to normal package
a18b3c5Install dev packages as well on ci

comment:248 Changed 6 weeks ago by git

  • Commit changed from a18b3c59eff2940159fdb930eea4af04b444e828 to f88cfc6a6382cf5cf35a602f83b7bedc06fc67eb

Branch pushed to git repo; I updated commit sha1. New commits:

f88cfc6Set SAGE_LOCAL in workflow for gap

comment:249 Changed 6 weeks ago by git

  • Commit changed from f88cfc6a6382cf5cf35a602f83b7bedc06fc67eb to d42fbfc8fe52e06a7c6a54cf6c0a7ea94c6e3a29

Branch pushed to git repo; I updated commit sha1. New commits:

d42fbfcSee if error comes from memory problems

comment:250 Changed 6 weeks ago by git

  • Commit changed from d42fbfc8fe52e06a7c6a54cf6c0a7ea94c6e3a29 to 8580789c62f04a49ea893c53bf28d20a80a1aa73

Branch pushed to git repo; I updated commit sha1. New commits:

8580789Fix indent

comment:251 Changed 6 weeks ago by git

  • Commit changed from 8580789c62f04a49ea893c53bf28d20a80a1aa73 to 1145360fdc51ee246c27715ce298c6005fd4b281

Branch pushed to git repo; I updated commit sha1. New commits:

1145360Fix paths to memory usage

comment:252 Changed 6 weeks ago by git

  • Commit changed from 1145360fdc51ee246c27715ce298c6005fd4b281 to 4bdfef1b21da9f376cfd3c096cb9932fe5334e4e

Branch pushed to git repo; I updated commit sha1. New commits:

4bdfef1Try with sudo

comment:253 Changed 6 weeks ago by gh-tobiasdiez

  • Description modified (diff)

comment:254 Changed 5 weeks ago by mkoeppe

I think it should be done on top of #30913

comment:255 Changed 5 weeks ago by gh-tobiasdiez

Since the pipfile in this ticket is more uptodate and is welltested (manually and through the added github action), I would actually argue that #30913 should be on top of this ticket.

Moreover, #30770 is the only open dependency of this ticket, so I guess its also more realistic to get this in first.

comment:256 follow-up: Changed 5 weeks ago by mkoeppe

This approach would also work. But I am skeptical about checking in the Pipfile.lock. Stuff for nightmare merge conflicts for everyone. I think it would be better if this was an ignored file.

Last edited 5 weeks ago by mkoeppe (previous) (diff)

comment:257 Changed 5 weeks ago by mkoeppe

  • Cc isuruf added

comment:258 Changed 5 weeks ago by mkoeppe

When the next beta comes out (hopefully soon), let's rebase it on top of it (and #30770). The commit history is a bit too wild

comment:259 Changed 5 weeks ago by mkoeppe

  • Priority changed from major to critical

Also, could you please:

  • clean up the ticket description
  • move documentation items from there to the manuals (installation and/or developer's manual)
  • move the remaining issues to new or existing tickets

comment:260 in reply to: ↑ 256 ; follow-up: Changed 4 weeks ago by gh-tobiasdiez

Replying to mkoeppe:

But I am skeptical about checking in the Pipfile.lock. Stuff for nightmare merge conflicts for everyone. I think it would be better if this was an ignored file.

Without the lock file you wouldn't get a reproducible virtual environment, since pipenv would always use the latest versions (that are compatible with each other). Moreover, the lock file is normally not changed, except if you call pipenv update. The workflow to update Python packages would then look as follows:

  • Run pipenv update
  • Test
  • If everything is fine, create ticket updating pipfile.lock and create a ticket for each updated dependency that is also listed in sage's build/pgks folder.

Will have a look at the other todo's you mention.

comment:261 Changed 4 weeks ago by gh-tobiasdiez

  • Description modified (diff)

comment:262 Changed 4 weeks ago by gh-tobiasdiez

Cleaned up the ticket description and added a bit of documentation. Since this is still somewhat experimental, I've only put it in the coding basics of the development doc. I guess with the recent changes and support of virtual environments, it might soon be time to update the recommended way of setting up a development platform.

comment:263 Changed 4 weeks ago by git

  • Commit changed from 4bdfef1b21da9f376cfd3c096cb9932fe5334e4e to 543b6d4e55879dee44f0e3d482b1ca1ebeda1d04

Branch pushed to git repo; I updated commit sha1. New commits:

543b6d4Add basic documentation

comment:264 in reply to: ↑ 260 Changed 4 weeks ago by mkoeppe

Replying to gh-tobiasdiez:

The workflow to update Python packages would then look as follows [...]

I would say it's not a good idea to try to change the developer workflow regarding package upgrades as part of this ticket.

comment:265 Changed 4 weeks ago by gh-tobiasdiez

I agree, that was more meant to illustrate possible applications of the new pipenv environment (and the lockfile for that purpose).

comment:266 Changed 4 weeks ago by mkoeppe

OK. I have created #31346 as a place for this.

comment:267 Changed 4 weeks ago by mkoeppe

Regarding the documentation: "After following the normal procedure of setting up your local development environment" -- I think we should be explicit about requiring the full Sage distribution to be built first. We cannot support experimental workflows as you (still) describe them in the ticket description.

comment:268 Changed 4 weeks ago by git

  • Commit changed from 543b6d4e55879dee44f0e3d482b1ca1ebeda1d04 to 510a3bb5344b544e70b12cd53dd7c48e19701963

Branch pushed to git repo; I updated commit sha1. New commits:

510a3bbClarify point in the docs

comment:269 Changed 4 weeks ago by gh-tobiasdiez

Sure!

comment:270 Changed 4 weeks ago by git

  • Commit changed from 510a3bb5344b544e70b12cd53dd7c48e19701963 to c21cc76386b75dbf07f01ae822c120ef9f29c9e9

Branch pushed to git repo; I updated commit sha1. New commits:

c21cc76Update deps

comment:271 Changed 4 weeks ago by mkoeppe

  • Description modified (diff)

I've moved the part of the ticket description describing the experimental workflow on bare ubuntu to #31347.

comment:272 Changed 4 weeks ago by mkoeppe

9.3.beta7 is out, time for rebasing

comment:273 Changed 4 weeks ago by git

  • Commit changed from c21cc76386b75dbf07f01ae822c120ef9f29c9e9 to cf068dfaf42a32f6bbd1ebd55376006e56fb6d8a

Branch pushed to git repo; I updated commit sha1. New commits:

cf068dfMerge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace

comment:274 Changed 4 weeks ago by gh-tobiasdiez

Nice, thanks for letting me know. I've merged the latest develop branch. The history of this branch still needs to be cleanup up, but let's do this after positive review.

comment:275 Changed 4 weeks ago by mkoeppe

Could you:

  • remove the change to tox.ini please?
  • Rename sage_build_ext and move it to a new module of sage_setup.command (and document what is different to the existing sage_build_ext)

comment:276 follow-ups: Changed 4 weeks ago by mkoeppe

In the new CI workflows:

  • For mac: I do not understand the line: brew install pari
  • For mac: Instead of setting PKG_CONFIG_PATH manually, perhaps use .homebrew-build-env because this is what configure suggests users to use
  • For linux: Why make maxima conway_polynomials jmol conway_polynomials elliptic_curves combinatorial_designs polytopes_db graphs threejs (it is not done in the mac workflow). Should we create a make target that corresponds to what is needed?
  • For linux: Could you elaborate on "# Set SAGE_LOCAL (required for gap)"

Could you review and add comments to the file please?

comment:277 Changed 4 weeks ago by mkoeppe

Also, as #30589 has been merged, probably best to switch from python 3.8 to 3.9 here too

comment:278 Changed 4 weeks ago by mkoeppe

Also #30912 will bring merge conflicts that are best to resolve now

comment:279 Changed 4 weeks ago by mkoeppe

Another conflict from the current version of #30770.

comment:280 follow-up: Changed 4 weeks ago by mkoeppe

Also create_extension duplicates some of the code in sage_build_cython.create_extension. In particular, it should not use numpy.get_include() directly but rather use sage.env.sage_include_directories (note #31333 for a related change).

comment:281 Changed 4 weeks ago by mkoeppe

I have tested this ticket (on top of #31335, which fixes macOS-specific problems, and resolving the mentioned merge conflicts); following the instructions from the added documentation, pipenv install leads to ERROR: Couldn't install package: cypari2.

The first error leading to this is:

  [1/7] Cythonizing cypari2/closure.pyx

  Error compiling Cython file:
  ------------------------------------------------------------
  ...
  #                  http://www.gnu.org/licenses/
  #*****************************************************************************

  from __future__ import absolute_import, division, print_function

  from cysignals.signals cimport sig_on, sig_off, sig_block, sig_unblock, sig_error
  ^
  ------------------------------------------------------------

  cypari2/closure.pyx:36:0: 'cysignals/signals.pxd' not found

comment:282 Changed 4 weeks ago by mkoeppe

The specific issue in cypari2 is probably https://github.com/sagemath/cypari2/issues/93 (tracked in #30922).

But overall I think it is unrealistic for these instructions to work. Let's use more restrictive instructions, in which pipenv is used from within sage --sh. Then one can at least assume that all libraries are available.

comment:283 Changed 4 weeks ago by mkoeppe

  • Status changed from needs_review to needs_work

comment:284 Changed 4 weeks ago by mkoeppe

(Something like ../sage -sh -c 'PIP_FIND_LINKS=file://$(cd .. && pwd)/local/var/lib/sage/wheels pipenv install -v -v' similar to what I am testing in #30913.)

comment:285 in reply to: ↑ 276 ; follow-up: Changed 4 weeks ago by gh-tobiasdiez

Thanks for the review! Will address most of your points later.

Replying to mkoeppe:

  • For linux: Could you elaborate on "# Set SAGE_LOCAL (required for gap)"

Without this line one gets the following error:

Traceback (most recent call last):
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/interfaces/expect.py", line 1468, in __init__
        self._name = parent._create(value, name=name)
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/interfaces/interface.py", line 500, in _create
        self.set(name, value)
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/interfaces/gap.py", line 1421, in set
        self._eval_line(cmd, allow_use_file=True)
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/interfaces/gap.py", line 790, in _eval_line
        self._start()
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/interfaces/gap.py", line 1254, in _start
        gap_reset_workspace(verbose=False)
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/interfaces/gap.py", line 1587, in gap_reset_workspace
        g.eval('SetUserPreference("HistoryMaxLines", 30)')
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/interfaces/gap.py", line 587, in eval
        result = Expect.eval(self, input_line, **kwds)
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/interfaces/expect.py", line 1381, in eval
        return '\n'.join([self._eval_line(L, allow_use_file=allow_use_file, **kwds)
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/interfaces/expect.py", line 1381, in <listcomp>
        return '\n'.join([self._eval_line(L, allow_use_file=allow_use_file, **kwds)
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/interfaces/gap.py", line 790, in _eval_line
        self._start()
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/interfaces/gap.py", line 1259, in _start
        Expect._start(self, "Failed to start GAP.")
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/interfaces/expect.py", line 520, in _start
        raise RuntimeError("unable to start %s: %s" % (self.name(), msg))
    RuntimeError: unable to start gap: End Of File (EOF). Exception style platform.
    Gap finished running /__w/sagetrac-mirror/sagetrac-mirror/.tox/local-sudo-standard/local/bin/gap -r -b -p -T -E -o 401m -s 401m -m 64m /__w/sagetrac-mirror/sagetrac-mirror/src/sage/ext_data/gap/sage.g
    command: /__w/sagetrac-mirror/sagetrac-mirror/.tox/local-sudo-standard/local/bin/gap
    args: ['/__w/sagetrac-mirror/sagetrac-mirror/.tox/local-sudo-standard/local/bin/gap', '-r', '-b', '-p', '-T', '-E', '-o', '401m', '-s', '401m', '-m', '64m', '/__w/sagetrac-mirror/sagetrac-mirror/src/sage/ext_data/gap/sage.g']
    buffer (last 100 chars): b''
    before (last 100 chars): b'Set the environment variable SAGE_LOCAL.\r\n'
    after: <class 'pexpect.exceptions.EOF'>
    match: None
    match_index: None
    exitstatus: 1
    flag_eof: True
    pid: 3124
    child_fd: 26
    closed: False
    timeout: None
    delimiter: <class 'pexpect.exceptions.EOF'>
    logfile: None
    logfile_read: None
    logfile_send: None
    maxread: 4194304
    ignorecase: False
    searchwindowsize: None
    delaybeforesend: None
    delayafterclose: 0.1
    delayafterterminate: 0.1
    searcher: searcher_re:
        0: re.compile(b'gap> ')

I guess the current github workflows don't see this as everything is invoked in one tox command and not in multiple steps (and enviroment variables are only active in each step).

comment:286 follow-up: Changed 4 weeks ago by gh-tobiasdiez

Concerning the cypari error, I have no clue what is needed for it to compile on mac. It's also strange that it works on the github action (for mac as well as linux) but not on your system. cysignals is actually declared in the pipenv file, so it should be installed. The simplest solution would be if cypari would publish wheels on pypi.

But overall I think it is unrealistic for these instructions to work.

It's not unrealistic, it is working very well for Linux.

Last edited 4 weeks ago by gh-tobiasdiez (previous) (diff)

comment:287 in reply to: ↑ 276 ; follow-up: Changed 4 weeks ago by gh-tobiasdiez

Replying to mkoeppe:

  • For linux: Why make maxima conway_polynomials jmol conway_polynomials elliptic_curves combinatorial_designs polytopes_db graphs threejs (it is not done in the mac workflow). Should we create a make target that corresponds to what is needed?

That's the only way I found to install these packages into SAGE_SHARE. Will add it also to the mac workflow. I cannot judge if it would make sense to add a special make target for it, especially since you are working on making these packages normal python packages.

comment:288 in reply to: ↑ 285 Changed 4 weeks ago by mkoeppe

Replying to gh-tobiasdiez:

Replying to mkoeppe:

  • For linux: Could you elaborate on "# Set SAGE_LOCAL (required for gap)"

Without this line one gets the following error:

...
        Expect._start(self, "Failed to start GAP.")
      File "/__w/sagetrac-mirror/sagetrac-mirror/src/sage/interfaces/expect.py", line 520, in _start
...

I guess the current github workflows don't see this as everything is invoked in one tox command and not in multiple steps (and enviroment variables are only active in each step).

OK, I see.

There are two ways to fix this:

  • as described in #30818 ("Set environment for subprocesses invoked by Sage"). Currently subprocesses (such as the gap process invoked by the pexpect-based gap interface) rely on sage-env being sourced
  • using pipenv run sage -t instead of pipenv run sage-runtests, which uses sage-env.

comment:289 follow-up: Changed 4 weeks ago by gh-tobiasdiez

I've added it to #29644 (which is part of #30818).

comment:290 in reply to: ↑ 287 ; follow-up: Changed 4 weeks ago by mkoeppe

Replying to gh-tobiasdiez:

Replying to mkoeppe:

  • For linux: Why make maxima conway_polynomials jmol conway_polynomials elliptic_curves combinatorial_designs polytopes_db graphs threejs (it is not done in the mac workflow). Should we create a make target that corresponds to what is needed?

That's the only way I found to install these packages into SAGE_SHARE. Will add it also to the mac workflow. I cannot judge if it would make sense to add a special make target for it, especially since you are working on making these packages normal python packages.

I've opened #31362 for this - which will add make all-sage-local.

comment:291 Changed 4 weeks ago by gh-tobiasdiez

Thanks! Concerning the above cysignals problem, you can try to run pipenv run pip install cysignals and then pipenv install again. Similarly, pipenv run pip install gmpy2==2.1.0b5 might be necessary if you have problems with the installation of pplpy. Let me know if this helps and I will add it to the documentation.

comment:292 Changed 4 weeks ago by git

  • Commit changed from cf068dfaf42a32f6bbd1ebd55376006e56fb6d8a to 398e6dd59ac7b8e3c49dc7917295e4fe4a9bcaba

Branch pushed to git repo; I updated commit sha1. New commits:

1f1dddcRevert change to tox.ini
0c3f517Move sage_build_ext
d06c84dRun workflow for Python 3.9 as well
7fee372Doc: Also install dev packages
a506c1fUse sage_build command for generating sources
398e6ddNarrow down imports to reduce install requires deps

comment:293 Changed 4 weeks ago by git

  • Commit changed from 398e6dd59ac7b8e3c49dc7917295e4fe4a9bcaba to 12c0a2ed96e5608f8f6c54aef0d81e8d76cdcff5

Branch pushed to git repo; I updated commit sha1. New commits:

12c0a2eMac: remove pari install

comment:294 Changed 4 weeks ago by git

  • Commit changed from 12c0a2ed96e5608f8f6c54aef0d81e8d76cdcff5 to 0157a4eac42646f49b9448afd120d07c4848750b

Branch pushed to git repo; I updated commit sha1. New commits:

1041128Rename ecl config variable
fd05a97Add fallback
2ac4cc8Merge branch 'develop' of git://github.com/sagemath/sage into public/build/ecl-config
6370c34src/sage/env.py: Do not use capture_output, which requires python 3.7
31f1bbbAlso use universal_newlines instead of text, for python 3.6 compatibility
11ad40aMerge branch 'public/build/ecl-config' of git://trac.sagemath.org/sage into public/build/ecl-config
1a8e2e4Merge branch 'public/build/ecl-config' of git://trac.sagemath.org/sage into public/build/inplace
0157a4eReuse sage_include_directories

comment:295 Changed 4 weeks ago by git

  • Commit changed from 0157a4eac42646f49b9448afd120d07c4848750b to a3806c58709c3b5c828668f823b37619bf1d1551

Branch pushed to git repo; I updated commit sha1. New commits:

18d8557Pretty print job names
a3806c5Update Macos workflow

comment:296 Changed 4 weeks ago by git

  • Commit changed from a3806c58709c3b5c828668f823b37619bf1d1551 to f5551673591c836942a4b7d50f971ad7c910763e

Branch pushed to git repo; I updated commit sha1. New commits:

f555167Fix workflow syntax

comment:297 Changed 4 weeks ago by git

  • Commit changed from f5551673591c836942a4b7d50f971ad7c910763e to bb90f76abc82987e7ff1cbfd2653dfa908fefc7b

Branch pushed to git repo; I updated commit sha1. New commits:

bb90f76Well...maybe now

comment:298 Changed 4 weeks ago by git

  • Commit changed from bb90f76abc82987e7ff1cbfd2653dfa908fefc7b to d1c2a0a00f8c675ad218b502516dca12d161933d

Branch pushed to git repo; I updated commit sha1. New commits:

d1c2a0aTry with latest version of pip

comment:299 Changed 4 weeks ago by git

  • Commit changed from d1c2a0a00f8c675ad218b502516dca12d161933d to f8599611360fcf04a54d28830c266183cd99ff99

Branch pushed to git repo; I updated commit sha1. New commits:

f859961Run everything versus the correct python

comment:300 Changed 4 weeks ago by git

  • Commit changed from f8599611360fcf04a54d28830c266183cd99ff99 to 606966b86e35ae679edc08dd1c5fa156cff667e8

Branch pushed to git repo; I updated commit sha1. New commits:

606966bTry to trigger macos workflow

comment:301 Changed 4 weeks ago by git

  • Commit changed from 606966b86e35ae679edc08dd1c5fa156cff667e8 to 541f7143df6f00116a3048fd34242e90a50ac40e

Branch pushed to git repo; I updated commit sha1. New commits:

541f714Remove empty line?

comment:302 Changed 4 weeks ago by git

  • Commit changed from 541f7143df6f00116a3048fd34242e90a50ac40e to 42d25a14c00a68e58bcb904783483a5514eecb5f

Branch pushed to git repo; I updated commit sha1. New commits:

42d25a1Still not?

comment:303 Changed 4 weeks ago by git

  • Commit changed from 42d25a14c00a68e58bcb904783483a5514eecb5f to f8599611360fcf04a54d28830c266183cd99ff99

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

comment:304 Changed 4 weeks ago by git

  • Commit changed from f8599611360fcf04a54d28830c266183cd99ff99 to f312a35f92ef30fa4a38ab54560686e2ef1bd5d6

Branch pushed to git repo; I updated commit sha1. New commits:

f312a35Try to remove python3-pip again

comment:305 Changed 4 weeks ago by git

  • Commit changed from f312a35f92ef30fa4a38ab54560686e2ef1bd5d6 to b01c4c98a6c590ec7526d2f87dba7a839ebee9df

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

b01c4c9Install tox via pip

comment:306 Changed 4 weeks ago by git

  • Commit changed from b01c4c98a6c590ec7526d2f87dba7a839ebee9df to a1d989c3efd08d5f64612865f428102b328744da

Branch pushed to git repo; I updated commit sha1. New commits:

a1d989cInstall pip as wel of course

comment:307 Changed 4 weeks ago by git

  • Commit changed from a1d989c3efd08d5f64612865f428102b328744da to 442596137f517ff3cb3e365e1b8f79b41744aec2

Branch pushed to git repo; I updated commit sha1. New commits:

4425961Strange...

comment:308 Changed 4 weeks ago by git

  • Commit changed from 442596137f517ff3cb3e365e1b8f79b41744aec2 to f2e436c459bad755685c6ca42a3d401b1fd29ed9

Branch pushed to git repo; I updated commit sha1. New commits:

f2e436cFix path of tox venv under mac

comment:309 in reply to: ↑ 290 Changed 4 weeks ago by mkoeppe

Replying to mkoeppe:

Replying to gh-tobiasdiez:

Replying to mkoeppe:

  • For linux: Why make maxima conway_polynomials jmol conway_polynomials elliptic_curves combinatorial_designs polytopes_db graphs threejs (it is not done in the mac workflow). Should we create a make target that corresponds to what is needed?

That's the only way I found to install these packages into SAGE_SHARE. Will add it also to the mac workflow. I cannot judge if it would make sense to add a special make target for it, especially since you are working on making these packages normal python packages.

I've opened #31362 for this - which will add make all-sage-local.

OK, the top-level target is now make build-local.

comment:310 in reply to: ↑ 289 Changed 4 weeks ago by mkoeppe

Replying to gh-tobiasdiez:

I've added it to #29644 (which is part of #30818).

Thanks. Do you plan to work on it? Otherwise a quick fix is to use pipenv run sage -t instead of pipenv run sage-runtests as I wrote above.

Last edited 4 weeks ago by mkoeppe (previous) (diff)

comment:311 in reply to: ↑ 286 Changed 4 weeks ago by mkoeppe

Replying to gh-tobiasdiez:

Concerning the cypari error, I have no clue what is needed for it to compile on mac.

Well it certainly needs the PARI library! When running pipenv outside of the environment set using sage --sh, it depends on everything already installed in the system.

That's why I am saying the instructions to the users should run pipenv within that shell.

comment:312 Changed 4 weeks ago by git

  • Commit changed from f2e436c459bad755685c6ca42a3d401b1fd29ed9 to a6e57808dcabbb60d9c7fdefb88ed7ca4be7caff

Branch pushed to git repo; I updated commit sha1. New commits:

b86b740Don't install dev packages
a6e5780Install fplll on mac

comment:313 Changed 4 weeks ago by git

  • Commit changed from a6e57808dcabbb60d9c7fdefb88ed7ca4be7caff to 85de00c2d4103d0edbcd2a41a2efcee52ea5c26e

Branch pushed to git repo; I updated commit sha1. New commits:

85de00cAlso install libffi8ubuntu1

comment:314 Changed 4 weeks ago by git

  • Commit changed from 85de00c2d4103d0edbcd2a41a2efcee52ea5c26e to 1760cf741ff4d392e7662075637350fe8e7894bb

Branch pushed to git repo; I updated commit sha1. New commits:

1760cf7Or try to install libffi7

comment:315 Changed 4 weeks ago by git

  • Commit changed from 1760cf741ff4d392e7662075637350fe8e7894bb to efea01f09fb253083afc5c04ffa5824bdd4d731c

Branch pushed to git repo; I updated commit sha1. New commits:

efea01fInstall wget as well

comment:316 Changed 4 weeks ago by git

  • Commit changed from efea01f09fb253083afc5c04ffa5824bdd4d731c to 5a2cc305cf50e561d400a42d3e568cee7ea082a1

Branch pushed to git repo; I updated commit sha1. New commits:

ec5629cClarify that we need all system packgaes
5a2cc30Try with libffi6

comment:317 Changed 4 weeks ago by git

  • Commit changed from 5a2cc305cf50e561d400a42d3e568cee7ea082a1 to 2ae43976cfacb331ae82426f25528213987225dd

Branch pushed to git repo; I updated commit sha1. New commits:

2ae4397Mac: install pari as well

comment:318 follow-up: Changed 4 weeks ago by gh-tobiasdiez

Is it actually desired that tox local-homebrew installs a fresh homebrew in the tox folder (if I'm not misreading the logs). Is there a way to use the system's homebrew?

comment:319 Changed 4 weeks ago by git

  • Commit changed from 2ae43976cfacb331ae82426f25528213987225dd to 88dabfa498cc3e12116d6f5f26e945ce4abd1429

Branch pushed to git repo; I updated commit sha1. New commits:

b9b1ef1Use homebrew-build-env
88dabfaCleanup setup.py

comment:320 Changed 4 weeks ago by git

  • Commit changed from 88dabfa498cc3e12116d6f5f26e945ce4abd1429 to 8d5e37b6ab241cb2f98462de9b536937724ed977

Branch pushed to git repo; I updated commit sha1. New commits:

b1fd3e9Run macos without local prefix
7d660e0Tox: Fix homebrew env variable
c3457e5Use local-homebrew-macos-usrlocal
d3d72cbRevert changes to tox.ini
8d5e37bFix compile errors

comment:321 Changed 4 weeks ago by git

  • Commit changed from 8d5e37b6ab241cb2f98462de9b536937724ed977 to d9ac8df40f4eb4702795e544649a9e1830f25874

Branch pushed to git repo; I updated commit sha1. New commits:

d9ac8dfRevert changes to sage.env import

comment:322 in reply to: ↑ 318 Changed 4 weeks ago by mkoeppe

Replying to gh-tobiasdiez:

Is it actually desired that tox local-homebrew installs a fresh homebrew in the tox folder

Yes, that's by design

Is there a way to use the system's homebrew?

From tox.ini comments:

    # The variant "local-homebrew-macos-usrlocal" uses the global installation in /usr/local
    # instead.  It may install packages or update packages.  It will not remove packages.
    # Use at your own risk.

comment:323 follow-up: Changed 4 weeks ago by gh-tobiasdiez

Thanks! That is working nicely.

I think I've now addressed all your comments and suggestions.

The only thing is that since merging the latest develop branch the sage packages (installed via tox) are no longer found when running pipenv install. At the commit https://github.com/sagemath/sagetrac-mirror/commit/4bdfef1b21da9f376cfd3c096cb9932fe5334e4e this was working, but stopped after I've merged the develop branch. Now packages such as linbox, singular and gap are still compiled by sage's make, but pkgconfig (in sage.env) cannot find them any longer and thus the c++ compilation stopped working. I don't have these problems locally, where I don't use tox. Do you have an idea which recent changes are the origin of this issue?

I've also tried your suggestion with sage -sh pipenv but run into several issues. I guess the problem is that pipenv is trying to reuse sages virtual environment.

comment:324 Changed 4 weeks ago by git

  • Commit changed from d9ac8df40f4eb4702795e544649a9e1830f25874 to 245d40fe531570a2bd3cc71e13d851a25e1ff858

Branch pushed to git repo; I updated commit sha1. New commits:

245d40fUse correct version of fpylll for mac

comment:325 Changed 4 weeks ago by git

  • Commit changed from 245d40fe531570a2bd3cc71e13d851a25e1ff858 to 4b3c76f9fbe11c202c8de6aa7fe661bcfc7aeca7

Branch pushed to git repo; I updated commit sha1. New commits:

4b3c76fTry to fix PKG_CONFIG_PATH and LD_LIBRARY_PATH

comment:326 follow-up: Changed 4 weeks ago by mkoeppe

...never touch LD_LIBRARY_PATH... it's always wrong

comment:327 Changed 4 weeks ago by mkoeppe

Could you please take a look at #31362 - it provides make build-local, which I think is what you need

comment:328 follow-up: Changed 4 weeks ago by mkoeppe

Thanks for making all the changes.

Right now it's hard for me to test anything from this ticket because the homebrew changes in #31335 conflict.

I would propose to do the following:

  • Rebasing/squashing this branch on top of #31362, current #30770, and #30912.
  • Merging the rebase back into the current branch -- by this trick you won't get merge conflicts in your other branches that depend on this ticket (if any)

comment:329 in reply to: ↑ 323 Changed 4 weeks ago by mkoeppe

Replying to gh-tobiasdiez:

I've also tried your suggestion with sage -sh pipenv but run into several issues. I guess the problem is that pipenv is trying to reuse sages virtual environment.

See if you can work around this by having pipenv use the base python that is shown in $SAGE_LOCAL/pyvenv.cfg.

comment:330 Changed 4 weeks ago by git

  • Commit changed from 4b3c76f9fbe11c202c8de6aa7fe661bcfc7aeca7 to 7c89641c4f2eed5e7cac24776e5079c346d575de

Branch pushed to git repo; I updated commit sha1. New commits:

7c89641Update lock file

comment:331 in reply to: ↑ 326 ; follow-up: Changed 4 weeks ago by gh-tobiasdiez

Replying to mkoeppe:

...never touch LD_LIBRARY_PATH... it's always wrong

Ok, but what's then the correct way to do this?

comment:332 Changed 4 weeks ago by gh-tobiasdiez

  • Dependencies changed from #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029 to #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029, #30912

comment:333 in reply to: ↑ 331 Changed 4 weeks ago by mkoeppe

Replying to gh-tobiasdiez:

Replying to mkoeppe:

...never touch LD_LIBRARY_PATH... it's always wrong

Ok, but what's then the correct way to do this?

Building all libraries with rpath - see LDFLAGS setting in src/bin/sage-env

comment:334 Changed 4 weeks ago by git

  • Commit changed from 7c89641c4f2eed5e7cac24776e5079c346d575de to b68ce04b04b2d2104c6643700fa0f8cbe81f9f74

Branch pushed to git repo; I updated commit sha1. New commits:

2c40179build/pkgs/sagelib/src: Update metadata
2a91862build/pkgs/sagelib/src/README.rst: Add some text
6432727Merge tag '9.3.beta2' into t/30912/sagelib__update_metadata_for_pypi_deployment
4a693f2Move build/pkgs/sagelib/src/setup.cfg to SAGE_ROOT/src, replace by symlink
ea182d7Copy changes from build/pkgs/sagelib/src to src
a1a10b9src/VERSION.txt: New
5697335src/setup.cfg: Add license_file=LICENSE.txt
deb9eb3Merge tag '9.3.beta3' into t/30912/sagelib__update_metadata_for_pypi_deployment
7ad4c0eMerge tag '9.3.beta4' into t/30912/sagelib__update_metadata_for_pypi_deployment
b68ce04Merge branch 'u/mkoeppe/sagelib__update_metadata_for_pypi_deployment' of git://trac.sagemath.org/sage into public/build/inplace

comment:335 in reply to: ↑ 328 ; follow-up: Changed 4 weeks ago by gh-tobiasdiez

Replying to mkoeppe:

I've merged the current versions of #30770 and #30912. Not sure what's the purpose of #31362. That would make the line make maxima conway_polynomials jmol conway_polynomials elliptic_curves combinatorial_designs polytopes_db graphs threejs obsolete right? If it's just for this somewhat cosmetic improvement, I would suggest to turn around the tables and let #31362 depend on this ticket (and make the changes in the workflow there as well). Reason: I had to play a lot around with the different targets until everything works, so I'm a bit hesitant to change things again (and since unmerging a branch is not so easy, one cannot simply revert the experiment).

comment:336 in reply to: ↑ 335 ; follow-up: Changed 4 weeks ago by mkoeppe

Replying to gh-tobiasdiez:

I've merged the current versions of #30770 and #30912.

Thanks.

Not sure what's the purpose of #31362. That would make the line make maxima conway_polynomials jmol conway_polynomials elliptic_curves combinatorial_designs polytopes_db graphs threejs obsolete right? If it's just for this somewhat cosmetic improvement

Yes, it's for making all standard non-Python packages available in SAGE_LOCAL.

Reason: I had to play a lot around with the different targets until everything works, so I'm a bit hesitant to change things again

This does not sound good. If installing additional packages from the distribution in SAGE_LOCAL breaks the proposed setup, then this ticket is definitely not ready.

comment:337 Changed 4 weeks ago by mkoeppe

Note also that #31362 does not change anything other than add two new make targets. So even if you find that these targets do not do what you need, there will be no need for unmerging it.

comment:338 in reply to: ↑ 336 Changed 4 weeks ago by gh-tobiasdiez

Replying to mkoeppe:

Reason: I had to play a lot around with the different targets until everything works, so I'm a bit hesitant to change things again

This does not sound good. If installing additional packages from the distribution in SAGE_LOCAL breaks the proposed setup, then this ticket is definitely not ready.

More in the sense, that it was a bit of try-and-error to find the minimal set of make targets that make the build pass. You could have course build every package in the sage distribution, but the point is that with pipenv only a couple of them are actually needed.

So build-local = sagelib-build-deps + maxima conway_polynomials jmol conway_polynomials elliptic_curves combinatorial_designs polytopes_db graphs threejs ?

comment:339 Changed 4 weeks ago by mkoeppe

... minus the Python packages of sagelib-build-deps

comment:340 Changed 4 weeks ago by mkoeppe

For example, it would not build numpy.

comment:341 Changed 4 weeks ago by mkoeppe

Of course, there may be a number of non-Python libraries that are not needed IF you are able to pull in a wheel of the Python packages that somehow ship a bundled version of the non-Python library. For example, the numpy wheels may come with their own BLAS -- but it won't help because some of our own packages also need BLAS.

comment:342 Changed 4 weeks ago by git

  • Commit changed from b68ce04b04b2d2104c6643700fa0f8cbe81f9f74 to 2154a5b9f8f56ccc83bb45aa9bd10d659b015649

Branch pushed to git repo; I updated commit sha1. New commits:

bdb2237Merge branch 't/30912/sagelib__update_metadata_for_pypi_deployment' into t/31357/fixup_src_version_txt_added_in__30912
c16651csrc/VERSION.txt: Use same format as build/pkgs/sagelib/src/VERSION.txt
2154a5bMerge branch 'u/mkoeppe/fixup_src_version_txt_added_in__30912' of git://trac.sagemath.org/sage into public/build/inplace

comment:343 Changed 4 weeks ago by gh-tobiasdiez

  • Dependencies changed from #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029, #30912 to #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029, #30912, #31357

comment:344 Changed 4 weeks ago by git

  • Commit changed from 2154a5b9f8f56ccc83bb45aa9bd10d659b015649 to 2773e6fd9c1868346d194227287dffd6c3642a5e

Branch pushed to git repo; I updated commit sha1. New commits:

55bc2aam4/sage_spkg_collect.m4: Generate install tree information for build/make/Makefile
d7dc9a2build/make/Makefile.in: New targets all-sage-local, all-sage-venv
27b589bMerge branch 't/30912/sagelib__update_metadata_for_pypi_deployment' into t/31362/make_all_sage_local__make_all_sage_venv
8796008Put pynac into SAGE_LOCAL, sagelib into SAGE_VENV
d6831b1build/make/Makefile.in: Add all-build-local, all-build-venv, which include dependency on toolchain
61f6ba6Makefile: Add top-level targets build-local, build-venv
e354672Merge branch 'u/mkoeppe/make_all_sage_local__make_all_sage_venv' of git://trac.sagemath.org/sage into public/build/inplace
2773e6fUse build-local target

comment:345 Changed 4 weeks ago by gh-tobiasdiez

  • Dependencies changed from #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029, #30912, #31357 to #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029, #30912, #31357, #31362

comment:346 Changed 4 weeks ago by gh-tobiasdiez

ok, let's see if that works

comment:347 Changed 4 weeks ago by mkoeppe

In any case thanks to merging these tickets, I am able to merge the tickets that fix homebrew - so I'll be able to test later today.

comment:348 Changed 3 weeks ago by mkoeppe

../sage -sh -c 'ARCHFLAGS= PIP_NO_BUILD_ISOLATION=1 pipenv install -v -v' got me a bit farther. (Setting ARCHFLAGS to the empty string helps on macOS; PIP_NO_BUILD_ISOLATION=1 helps with some of the Python packages that declare their build dependencies incorrectly (see #30922). Now it fails with this:

    raise MetadataInconsistent(self._ireq, "name", dist.project_name)
pip._internal.exceptions.MetadataInconsistent: Requested fpylll from https://files.pythonhosted.org/packages/ad/c5/f30f354dcce03b928702414fdbdd916d60915c6b144360c88707784385bb/fpylll-0.5.4.tar.gz#egg=fpylll-recent (from -r /var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pipenv-7zg7bw1s-requirements/pipenv-b8jmkho7-requirement.txt (line 1)) has different name in metadata: 'fpylll'
Removed build tracker: '/private/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pip-req-tracker-z5q5pu09'
...
    raise exceptions.InstallError(c.dep.name, extra=err_lines)
pipenv.exceptions.InstallError: ERROR: Couldn't install package: fpylll-recent
 Package installation failed...

comment:349 Changed 3 weeks ago by git

  • Commit changed from 2773e6fd9c1868346d194227287dffd6c3642a5e to e70f22ba61f2d4c1e0de5a254a15844129df7cab

Branch pushed to git repo; I updated commit sha1. New commits:

627e79dFix cpath as well
e70f22bTry to set name of fpylll-recent

comment:350 follow-up: Changed 3 weeks ago by gh-tobiasdiez

This error also occurred on the github workflow. I tried to fix it by setting the name in the pipfile. If this doesn't work can you please remove the fpylll-recent in the pipfile and invoke pipenv install https://files.pythonhosted.org/packages/ad/c5/f30f354dcce03b928702414fdbdd916d60915c6b144360c88707784385bb/fpylll-0.5.4.tar.gz This should work. Could you then please post what pipenv writes to the pipfile, thanks!

build-local seems to work, except that it installs some python related tools/packages (pip, setuptools_scm, wheel) and doesn't install maxima.

comment:351 Changed 3 weeks ago by git

  • Commit changed from e70f22ba61f2d4c1e0de5a254a15844129df7cab to 8c22f9ec046cb8253531056e0a7aff7118ca778e

Branch pushed to git repo; I updated commit sha1. New commits:

8c22f9eSet LIBRARY_PATH

comment:352 Changed 3 weeks ago by git

  • Commit changed from 8c22f9ec046cb8253531056e0a7aff7118ca778e to e17019c5528c867367988d7fa99bcef76f320949

Branch pushed to git repo; I updated commit sha1. New commits:

e17019cInstall old readline as well

comment:353 in reply to: ↑ 350 Changed 3 weeks ago by mkoeppe

Replying to gh-tobiasdiez:

build-local seems to work, except that it installs some python related tools/packages (pip, setuptools_scm, wheel) and doesn't install maxima.

wheel (which pulls in the python toolchain) is fixed in #31321.

maxima - could you double check?

comment:354 Changed 3 weeks ago by gh-tobiasdiez

Ok good. And false alarm, maxima is installed (the output is just condensed to one line in contrast to the pages of text for make maxima which has thrown me off).

The latest commit should hopefully fix the fpylll error.

comment:355 Changed 3 weeks ago by git

  • Commit changed from e17019c5528c867367988d7fa99bcef76f320949 to e2ed97f2ba3af909838802c95f4151282090c7d7

Branch pushed to git repo; I updated commit sha1. New commits:

e2ed97fFix fpylll

comment:356 Changed 3 weeks ago by mkoeppe

I will put a rebased/squashed version of the setup.py/sage_setup changes on #31377, because this part seems ready and I hope we can get it into 9.3 to get developers used to the in-place install idea.

comment:357 follow-up: Changed 3 weeks ago by mkoeppe

(It looks to me like the venv aspects of this ticket will take more time; in particular since they seem to involve getting an uncooperative random set of wheels from PyPI to cooperate with the system.)

comment:358 Changed 3 weeks ago by gh-tobiasdiez

It's quite the opposite: since some of the python packages (fyplll, cypari2) don't provide wheels, system packages need to be installed which is quite hard to coordinate across different systems.

comment:359 Changed 3 weeks ago by mkoeppe

But the Sage distribution has already solved this problem. Just build within a sage-sh.

comment:360 Changed 3 weeks ago by mkoeppe

Please take a look if #31377 looks agreeable to you.

comment:361 in reply to: ↑ 280 Changed 3 weeks ago by mkoeppe

Replying to mkoeppe:

Also create_extension duplicates some of the code in sage_build_cython.create_extension.

Also as I just wrote in #31377, some of the code setting flags is missing, which is causing errors. Could you look into unifying this please when you have a chance?

comment:362 in reply to: ↑ 357 Changed 3 weeks ago by vdelecroix

Replying to mkoeppe:

(It looks to me like the venv aspects of this ticket will take more time; in particular since they seem to involve getting an uncooperative random set of wheels from PyPI to cooperate with the system.)

On this topic, see also the discussion at https://github.com/sagemath/cypari2/issues/105.

I think it is a desirable feature to be able to edit the source code in an inplace install of sage (ie what pip -e provides). This would greatly benefit all Sage developers. However, I have hard time to understand the benefit of the venv. On the other hand, I do not understand how wheels could solve the libraries/softwares dependencies. More precisely, here is a scenario (which happens to be happening in Sage)

  • SageMath depends on pylib1 and pylib2 that uses the same libfoo C library. Both of these libraries provide wheels that bundle libfoo to make some of their users happy.
  • SageMath also depends on pylib3 that depends on both pylib1 and pylib2 and uses custom Cython code that uses functions from libfoo. pylib3 does not provide wheel yet. How should it be implemented?

comment:363 Changed 3 weeks ago by mkoeppe

Hi Vincent, thanks for joining the discussion.

For a discussion of possible distribution strategies using wheels, let's use #31251 (Meta-ticket: Distribution as wheels), which also has a link to a recent post by Gommers on packaging. I agree with you that it is a hard problem.

For Sage 9.3, I have a much more modest goal regarding wheels that is already fully implemented and is only waiting for review (see ticket list in #29705 for tickets by milestone). Since #29500 (already in Sage 9.2), we generate wheels in $SAGE_LOCAL/var/lib/sage/wheels. These wheels are not distributable as they do not bundle their shared library dependencies but rather rpath into $SAGE_LOCAL. Users can make their own venvs for development purposes where they install the prebuilt wheels from this directory. #31045 and #31279 add some missing wheels. #30913 generates dependency metadata that makes it easier to set up the venv; and adds tests using tox.

For editable install of sagelib as part of the normal build system (no extra venv) using Tobias' new setup.py from this ticket, see #31377.

Last edited 3 weeks ago by mkoeppe (previous) (diff)

comment:364 Changed 3 weeks ago by gh-tobiasdiez

  • Dependencies changed from #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029, #30912, #31357, #31362 to #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029, #30912, #31357, #31362, #31365

comment:365 Changed 3 weeks ago by git

  • Commit changed from e2ed97f2ba3af909838802c95f4151282090c7d7 to 3e60c00b3cb16ae500584baf484017b922ac3988

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

804ebd7sage_setup.dpcbuild.AllBuilder: Restrict workaround to macOS
b4ceee5sage_setup.docbuild: In the workaround, do not go through build_many to build serially
7369a3eMerge branch 't/31344/homebrew__docbuild_crashes__libtcl_atforkprepare' into t/31365/add_ntl_to_cython_aliases
ce03e6bsrc/sage/schemes/hyperelliptic_curves/hypellfrob.pyx: Merge distutils directives
32576b4sage.misc.cython: Add NTL aliases, cache result of cython_aliases
6e30e3aUse variables NTL_INCDIR, NTL_LIBDIR in sage_conf, separate from NTL_PREFIX used in sage-build-env-config; set -std=c++11 in NTL_CFLAGS
7b1e27bMerge distutils directives for Cython files using ntl
4d2828dSwitch Cython files that use NTL to language = c++
993c35cbuild/pkgs/ntl/spkg-configure.m4: Fix up
3e60c00Merge branch 'u/mkoeppe/add_ntl_to_cython_aliases' of git://trac.sagemath.org/sage into public/build/inplace

comment:366 Changed 3 weeks ago by gh-tobiasdiez

  • Status changed from needs_work to needs_review

So, I think, I've found a solution to the fpylll problems. In fact, the github workflow should now also work for macos (at least the last run installed everything fine except sage, which failed due to an NTL error that is fixed with #31365). Since macos runs on the sage repo run currently really slow, you can find the latest run at https://github.com/tobiasdiez/sage/actions.

comment:367 Changed 3 weeks ago by mkoeppe

I've split out another ticket from here: #31384

comment:368 Changed 3 weeks ago by mkoeppe

  • Priority changed from critical to major
  • Work issues set to rebase on dependencies + #31384, #31377

comment:369 Changed 3 weeks ago by gh-tobiasdiez

  • Dependencies changed from #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029, #30912, #31357, #31362, #31365 to #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029, #30912, #31357, #31362, #31365, #31377
  • Work issues rebase on dependencies + #31384, #31377 deleted

comment:370 Changed 3 weeks ago by git

  • Commit changed from 3e60c00b3cb16ae500584baf484017b922ac3988 to 13d90ee00e330ff759d0f8fc0b4e3868a477248c

Branch pushed to git repo; I updated commit sha1. New commits:

217c675Try to fix gmp issue on mac
25df925For mac not linux...
90a5104Cleanup workflow
0c4ba62Try to only specify gmp includes
a978458Fix a few more errors
c1ce878Also for the manual install
13d90eeRemove a few unnecessary changes

comment:371 Changed 3 weeks ago by git

  • Commit changed from 13d90ee00e330ff759d0f8fc0b4e3868a477248c to 08fef7dd62544dfb13327d4ad38fca5ab6691190

Branch pushed to git repo; I updated commit sha1. New commits:

f8b5ef3Mark a few cython files as c++
08fef7dOnly catch import errors

comment:372 Changed 3 weeks ago by git

  • Commit changed from 08fef7dd62544dfb13327d4ad38fca5ab6691190 to 8f8b61533287f67be77a4920f7ffceb5ebc39b0d

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

18df2dcMerge branch 't/31365/add_ntl_to_cython_aliases' into t/31377/__configure___enable_editable
19ca17asrc/sage/graphs/graph_decompositions/tdlib/sage_tdlib.cpp: Move one directory level higher
f02963bMerge branch 't/31389/remove_clash_of_tdlib__with_tdlib_pyx_in__src_sage_graphs_graph_decompositions' into t/31377/__configure___enable_editable
2b293f8src/setup.py: Install the jupyter kernel using sage_install
b05e4babuild/make/Makefile.in (sagelib-clean): Remove generated .c and .cpp files
271d427src/setup.py: Fix sage.misc.sageinspect by setting Cython.Compiler.Options.embed_pos_in_docstring
f6b2fc4src/sage/repl/interpreter.py: Fix a doctest for editable installs of sagelib
9a0731asage.interacs.debugger: Remove deprecated module
e5d47ceMerge branch 't/31390/remove_deprecated_sage_interacts_debugger' into t/31377/__configure___enable_editable
8f8b615Merge branch 't/31377/__configure___enable_editable' into t/30371/public/build/inplace

comment:373 follow-up: Changed 3 weeks ago by mkoeppe

Merged #31377, resolving edit conflicts.

comment:374 follow-up: Changed 3 weeks ago by mkoeppe

I don't know which problem you are trying to solve with the big try/except. I don't think reducing ImportError is the correct solution.

If you are trying to make setup.py sdist work when libs are missing, consider using the code from build/pkgs/sagelib/src/setup.py line 61

comment:375 in reply to: ↑ 373 ; follow-up: Changed 3 weeks ago by gh-tobiasdiez

Replying to mkoeppe:

Merged #31377, resolving edit conflicts.

Thanks, but this did revert some of my changes (in particular to setup.py)....

comment:376 in reply to: ↑ 374 ; follow-up: Changed 3 weeks ago by gh-tobiasdiez

Replying to mkoeppe:

I don't know which problem you are trying to solve with the big try/except. I don't think reducing ImportError is the correct solution.

If you are trying to make setup.py sdist work when libs are missing, consider using the code from build/pkgs/sagelib/src/setup.py line 61

Yes, that was my goal. Creating the virtual environment otherwise fails as Cython is not yet installed into it. The file you refer to doesn't seem to exist.

comment:377 Changed 3 weeks ago by git

  • Commit changed from 8f8b61533287f67be77a4920f7ffceb5ebc39b0d to fec55234a00e46d023ad583f8c6284b970141ab0

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

fec5523Merge branch 't/31377/__configure___enable_editable' into t/30371/public/build/inplace

comment:378 in reply to: ↑ 375 Changed 3 weeks ago by mkoeppe

Replying to gh-tobiasdiez:

Replying to mkoeppe:

Merged #31377, resolving edit conflicts.

Thanks, but this did revert some of my changes (in particular to setup.py)....

Sorry, my mistake. I've redone the merge now. Please take a look.

comment:379 in reply to: ↑ 376 Changed 3 weeks ago by mkoeppe

Replying to gh-tobiasdiez:

Replying to mkoeppe:

I don't know which problem you are trying to solve with the big try/except. I don't think reducing ImportError is the correct solution.

If you are trying to make setup.py sdist work when libs are missing, consider using the code from build/pkgs/sagelib/src/setup.py line 61

Yes, that was my goal. Creating the virtual environment otherwise fails as Cython is not yet installed into it. The file you refer to doesn't seem to exist.

This file: https://git.sagemath.org/sage.git/tree/build/pkgs/sagelib/src/setup.py?id=fec55234a00e46d023ad583f8c6284b970141ab0

comment:380 Changed 2 weeks ago by mkoeppe

comment 374?

Note: See TracTickets for help on using tickets.