Opened 2 years ago
Last modified 2 months ago
#30371 needs_review enhancement
Use pipenv to create venv for development
Reported by:  Matthias Köppe  Owned by:  

Priority:  major  Milestone:  sage9.8 
Component:  build  Keywords:  sd111 
Cc:  Tobias Diez, François Bissey, John Palmieri, Isuru Fernando  Merged in:  
Authors:  Tobias Diez  Reviewers:  
Report Upstream:  N/A  Work issues:  
Branch:  public/build/inplace (Commits, GitHub, GitLab)  Commit:  c736e0cbccc8ddb9cc8359a48094aa3181b881fa 
Dependencies:  #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029, #30912, #31357, #31362, #31365, #31377  Stopgaps: 
Description (last modified by )
We modify src/setup.py
to support editable installs via pipenv (pipenv install e .
, which invokes pip install editable
). Since distribution continues using build/pkgs/sagelib/src/setup.py, the changes to src/setup.py made in this ticket do not affect the existing build process of the Sage library.
With this ticket, the following code works:
./bootstrap ./configure enableeditable make buildlocal cd src mkdir .venv pipenv install dev ignorepipfile pipenv run pip install e . upgrade existsaction=i nobuildisolation pipenv run sageruntests all
Instead of the first three commands, one can of course also use tox e localsudostandard buildlocal
.
It generates a virtual env in src/.venv
that contains all necessary Python packages, and an editable install of sage.
Moreover, the installation of some packages might fail (depending on your system configuration). In this case one can manually build it using sage and then install the sagebuilt wheel in the pipenv. For example,
make fpylll cd src && pipenv run pip install <sage>/local/var/lib/sage/venvpython3.8/var/lib/sage/wheels/fpylll0.5.6cp38cp38linux_x86_64.whl
The output of the added GH workflow is available here: https://github.com/sagemath/sagetracmirror/actions?query=workflow%3A%22Build+%26+Test+using+pipenv%22+branch%3Apublic%2Fbuild%2Finplace Most tests are working, but a few are failing. Only a few tests are failing:
sage t randomseed=0 sage/misc/sagedoc.py # 4 doctests failed sage t randomseed=0 sage/docs/conf.py # 1 doctest failed sage t randomseed=0 sage/matrix/matrix_integer_dense.pyx # 1 doctest failed sage t randomseed=0 sage/cpython/dict_del_by_value.pyx # 2 doctests failed sage t randomseed=0 sage/tests/cmdline.py # 8 doctests failed
TODO (as followup tickets):
 Expand on the added documentation on how to set up the virtual environment, using either plain
python m venv
,virtualenv
(20.x), orpipenv
. This is #31342.  Consider not commit pipfile.lock but instead generate it as build artifact on ci (so that build errors can be easily reproduced by downloading the lock file, and recreating the exact venv locally): https://github.com/pypa/pipenv/issues/1911#issuecomment378989597. However, normally you want to put the lock file under version control for the reasons described in https://trac.sagemath.org/timeline?from=20210203T13%3A15%3A22Z. The official statement is "always include in version control", see https://github.com/pypa/pipenv/issues/598.
 Replace manual definition of build variables (PKG_CONFIG_PATH, LIBRARY_PATH etc) by
sageenv
.  Fix failing doctests.
Previous tickets on inplace builds, editable installs:
Change History (489)
comment:1 Changed 2 years ago by
Branch:  → public/build/inplace 

Commit:  → bc6e2726cb87273de0c92297156dccf4b5107f06 
comment:2 Changed 2 years ago by
I don't really have time to look into this before the 9.2 release. But if you are looking for where include directories are determined, take a look at sage_setup.command.sage_build_cython
comment:3 Changed 2 years ago by
Commit:  bc6e2726cb87273de0c92297156dccf4b5107f06 → 58452ce79c2d8863f139e72d75ec7b0c5904a989 

Branch pushed to git repo; I updated commit sha1. New commits:
58452ce  Make inplace compilation work

comment:4 Changed 2 years ago by
The command ./sage src/setup.py build_ext inplace
runs now fine for me. I had to change quite a few things to make it work, and found it easier to completely start over using cythonize
instead of using the existing build_cython
code. The resulting code is a minimal working version. Please let me know which features of the existing build_cython
file you would like to keep (to be honest, I don't really understand what some of the code there is doing, and why its needed).
The code definitely needs cleanup, but otherwise is ready for a first round of review.
comment:5 Changed 2 years ago by
Status:  new → needs_review 

comment:6 Changed 2 years ago by
Commit:  58452ce79c2d8863f139e72d75ec7b0c5904a989 → 1a99228dafea2b7b676e1754b3d4b43f794ec7a7 

Branch pushed to git repo; I updated commit sha1. New commits:
1a99228  Update gitignore

comment:7 Changed 2 years ago by
 a/.gitignore +++ b/.gitignore @@ 99,7 +99,18 @@ gitlabbuilddocker.log /src/build /src/Makefile /src/bin/sageenvconfig /build/bin/sagebuildenvconfig +/build
this is not good. Our /build/
contains important files of the sage distribution
comment:8 Changed 2 years ago by
And the changes removing old python2 code really need to go on a separate ticket
comment:9 Changed 2 years ago by
Commit:  1a99228dafea2b7b676e1754b3d4b43f794ec7a7 → 22b27a051b7f1a429b7f91f965b33154ee776aea 

Branch pushed to git repo; I updated commit sha1. New commits:
22b27a0  Remove global gitignore of build folder

comment:10 followup: 11 Changed 2 years ago by
Ok, I was under the assumption that the build folder is for build artifacts. If this would be the case, then I suppose it would be better to whitelist those files in the build folder that needs to be checkedin. I've now replaced the global build ignore, by a more finegrained statement which only ignores the "temporary" files.
Concerning the changes about old Python2 code, I've created #30397.
comment:11 Changed 2 years ago by
Replying to ghtobiasdiez:
I was under the assumption that the build folder is for build artifacts.
No, not at all.
comment:12 Changed 2 years ago by
comment:13 followup: 14 Changed 2 years ago by
I wasn't aware of your changes in #28925. I guess what I'm proposing is a different strategy for the source discovery. The approach in #28925 is to change the source files themselves and enrich them by some form of metadata that decides in which distribution / package they land. Here I was proposing to specify the distribution of the files in the setup.py file. I guess both strategies have pros and cons. The main reason for me was that it seems the responsibility of the setup.py file to construct the distribution; why should the source file need to care about where they are used/included? I.e classical example of separation of concerns. I think, it's also the more flexible approach since it's easy to include a source file into multiple distributions etc. Moreover, you also gain a bit of performance at build time since you don't need to parse the files for some magic comments. What do you think?
comment:14 followup: 15 Changed 2 years ago by
Replying to ghtobiasdiez:
[...] Here I was proposing to specify the distribution of the files in the setup.py file. I guess both strategies have pros and cons. The main reason for me was that it seems the responsibility of the setup.py file to construct the distribution; why should the source file need to care about where they are used/included? I.e classical example of separation of concerns. [...]
One more thing to be aware of is that (1) setup.py
by itself actually is not able to control what goes into the source distribution. In #29865 (where I prepare some experimental subset distributions), I additionally use MANIFEST.in
to control that.
Separation of concerns is, of course, desirable. But so is locality: (2) My plan for splitting up the distribution is driven to a large extent by the dependencies on different sets of C/C++ libraries. And these are already encoded in Cython distutils directives.
Also I have explored a little bit the packaging options beyond using setuptools (in part motivated because of the shortcoming (1)). See #29854 (flit), #29810 (poetry). Unfortunately currently there does not seem to be a proper mainstream solution for the source layout that we want to keep (a monolithic source tree in src/sage
). For example, flit
(and even pip
outofdirectory installs) do not handle symbolic links well. In the long term, I was hoping to transform sage_setup
to implement the PEP 517 API directly (see #29845). That's another reason why I would be reluctant to hardcode distribution information in setup.py
.
In any case, I am hoping to have a broader discussion about all of this on sagedevel at the beginning of the 9.3 cycle.
comment:15 followups: 16 17 Changed 2 years ago by
Separation of concerns is, of course, desirable. But so is locality: (2) My plan for splitting up the distribution is driven to a large extent by the dependencies on different sets of C/C++ libraries. And these are already encoded in Cython distutils directives.
One could also take this as an argument to remove the distutil dependency directives and put them into the setup.py, giving you locality and separation of concerns.
Anyway, my goal with this PR is not to redesign or modularize the build. I just wanted to have a working inplace build. For this moving the distribution declaration to setup.py was the easiest option, and I quite liked this also from a conceptual point of view. But if the current way is preferred, I change it back of course. Anything else that needs to be changed?
comment:16 Changed 2 years ago by
Replying to ghtobiasdiez:
One could also take this as an argument to remove the distutil dependency directives and put them into the setup.py, giving you locality and separation of concerns.
Ha, no, we just moved them there. Getting rid of the messy file modules_list.py
was an achievement in the present cycle. See #29706
comment:17 Changed 2 years ago by
Replying to ghtobiasdiez:
Anything else that needs to be changed?
I haven't really had a chance to look at it in more detail yet, sorry. Soon...
comment:18 Changed 2 years ago by
Commit:  22b27a051b7f1a429b7f91f965b33154ee776aea → 55d00d7f055628848bbdeef9075b8574c44316b2 

Branch pushed to git repo; I updated commit sha1. New commits:
55d00d7  Merge branch 'develop' of git://trac.sagemath.org/sage into public/build/inplace

comment:19 Changed 2 years ago by
Dependencies:  → #30397 

comment:20 Changed 2 years ago by
Commit:  55d00d7f055628848bbdeef9075b8574c44316b2 → 40f35033ce241f8a6bf7fe2307a8c8b946c981e1 

Branch pushed to git repo; I updated commit sha1. New commits:
8404a5d  Remove old python 2 code from cython files

e9c06e3  Readd constants in build script

811b78c  Remove unused import

32daec5  Merge branch 'public/build/removeOldCode' of git://trac.sagemath.org/sage into public/build/inplace

40f3503  Fix merge problem

comment:21 Changed 2 years ago by
Commit:  40f35033ce241f8a6bf7fe2307a8c8b946c981e1 → 851c557ccb33c61c78b005a20b908721deaf31a0 

Branch pushed to git repo; I updated commit sha1. New commits:
851c557  Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace

comment:22 Changed 2 years ago by
Commit:  851c557ccb33c61c78b005a20b908721deaf31a0 → 1899dda1def19c3d7d518a5b4ccce67738415611 

Branch pushed to git repo; I updated commit sha1. New commits:
1899dda  Remove unnessary changes

comment:23 Changed 2 years ago by
Commit:  1899dda1def19c3d7d518a5b4ccce67738415611 → 74c9f7d2a328df4a2f436d3523daa12dddf655a6 

Branch pushed to git repo; I updated commit sha1. New commits:
74c9f7d  Cleanup code

comment:24 Changed 2 years ago by
I've now changed the code to use the % sage_setup: distribution = xyz
way. Moreover, I've cleanup up the code a bit.
Before I continue removing the code that is no longer needed, I would like to ask you which features of the current cythonize build script are missing from my new minimalistic implementation and should be reimplemented. (As I've said above, I don't really understand some of the code in build_cython
.
comment:25 Changed 2 years ago by
These changes:
# distutils: sources = sage/libs/ppl_shim.cc +# distutils: sources = src/sage/libs/ppl_shim.cc
are not good. The root of the sagelib source tree is src
, and paths are relative to this  this is where setup.py
lives, after all.
As a way forward, I would suggest to actually work on #29864 instead of the present ticket. #29864 is a step in the modularization task (#29705), creating two new distributions  sagecore
and sagetdlib
. sagecore
is like sagelib
minus sagetdlib
and without the fancy incremental installation tricks.
comment:26 Changed 2 years ago by
Status:  needs_review → needs_work 

comment:27 followup: 28 Changed 2 years ago by
Thanks for the feedback. I'll try to find a way to specify the paths relative to src
. I think my problem there was that setup.py
is invoked from the central sage
script, which sits in the root and not in the src folder.
Can you please explain the connection to #29864 in detail? For my the main point of this ticket is that I can compile the cython files to obtain compiled so
files in the same folder as the original cython file (e.g. src/sage/rings
). This is needed so that VS code can pick up these dependencies, resolve the imports, provide intellisense for it etc. My longterm goal would be to have a normal python installation (central or virtual env), point VS towards this runtime, and be able to use the sage code (in /src) without the need to use make
at any point. How are #29864 and #29705 helping for this?
comment:28 Changed 2 years ago by
Replying to ghtobiasdiez:
Can you please explain the connection to #29864 in detail?
Package sagecore
created in #29864 will be a version of sagelib with a plain setuptools build system, intended for (modular) deployment of the Sage library on PyPI. Since setuptools supports setup.py develop
, this has synergy with your goal.
And we will be able to work on the build system of sagecore
, across several tickets, without having to worry about breaking sagelib
builds  or what current users/developers are expecting of it.
comment:29 followup: 30 Changed 2 years ago by
That make sense, thanks for the explanation.
The only thing that I still don't understand is how the modularization will help with the compilation of cython files. If if everything is modularized, I guess the setup.py will still be used for this. And the point of this ticket is to improve the cyhonization...
comment:30 Changed 2 years ago by
Replying to ghtobiasdiez:
The only thing that I still don't understand is how the modularization will help with the compilation of cython files. If if everything is modularized, I guess the setup.py will still be used for this.
Simply by having a separate setup.py
in build/pkgs/sagecore/src
.
comment:31 Changed 2 years ago by
Dependencies:  #30397 → #29500, #30527 

Description:  modified (diff) 
Summary:  Inplace installs with "./sage pip install editable src/" → Inplace (editable) installs of sagelib in a venv 
comment:33 Changed 2 years ago by
Commit:  74c9f7d2a328df4a2f436d3523daa12dddf655a6 → 5bda3321b88289cbd3f0dfe97308b07dd9dd1459 

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
0a64674  build/pkgs/gambit/spkginstall.in: Install via bdist_wheel

ca58693  build/pkgs/pillow/spkginstall.in: Install via bdist_wheel

207d80f  build/pkgs/pip2pi: New

2555e19  build/make/install: At the end, update the repository index

5a747c4  build/bin/sagepip{install,uninstall}: Remove pip2 support

e66243f  Merge branch 't/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels' into t/30527/pep503simplerepositoryforwheels

9937126  Merge branch 't/30527/pep503simplerepositoryforwheels' into t/30371/public/build/inplace

1a39442  build/pkgs/sagelib/src/setup.py: Replace symlink to SAGE_SRC/setup.py by a copy

54fe0c7  src/tox.ini, src/MANIFEST.in: Move here from build/pkgs/sagelib/src, leave symlinks behind

5bda332  Inplace (editable) installs of sagelib (#30371 squashed)

comment:34 followup: 41 Changed 2 years ago by
TODO:
 The changes to
src/sage/cpython/debugimpl.c
need to go on a separate ticket  The changes to
src/sage/misc/persist.pyx
need to go on a separate ticket  Don't change
find_python_sources
becausebuild/pkgs/sagelib/src/setup.py
continues to use it
comment:35 Changed 2 years ago by
Description:  modified (diff) 

comment:36 Changed 2 years ago by
Description:  modified (diff) 

comment:37 Changed 2 years ago by
Dependencies:  #29500, #30527 → #29500, #30527, #22731 

comment:38 Changed 2 years ago by
Description:  modified (diff) 

comment:39 Changed 2 years ago by
Commit:  5bda3321b88289cbd3f0dfe97308b07dd9dd1459 → 274538190984ea38145ffdb4d7d1defc88784124 

comment:40 Changed 2 years ago by
Dependencies:  #29500, #30527, #22731 → #29500, #30527, #22731, #30578 

Description:  modified (diff) 
comment:41 Changed 2 years ago by
comment:42 Changed 2 years ago by
Commit:  274538190984ea38145ffdb4d7d1defc88784124 → a493e6e2a972ab5344fdd3a953376278e807cf3b 

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
851c557  Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace

1899dda  Remove unnessary changes

74c9f7d  Cleanup code

81fdcf3  Make distutils relative to src again

faf6777  Fail gracefully for missing packages

8af6e23  Add minimal pipfile

456589c  Ignore build files

59e8361  Make it compile by ignoring a lot of errors

054fedc  Add psutil and pexpect

a493e6e  Merge branch 'public/build/inplace' of git://trac.sagemath.org/sage into public/build/inplace

comment:43 Changed 2 years ago by
Commit:  a493e6e2a972ab5344fdd3a953376278e807cf3b → c5f3873cc7ddb158646051861fc10c72c6be22f5 

Branch pushed to git repo; I updated commit sha1. New commits:
c5f3873  Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace

comment:44 followups: 46 47 Changed 2 years ago by
I have now added a minimal pipfile including sage as an editable install. So running pipenv install
results in a semiworkable virtual environment. "Semi" because not all pyx files could be compiled successfully since they rely on external dependencies (e.g. singular, giac, ecl, ratpoints, ...).
For now, I've excluded the relevant pyx files from the compilation. Any ideas how to handle these dependencies are welcome! One approach would be to treat them similar to the the optional packages by putting a "sage_setup distribution" tag to the pyx file.
I also tried to run pipenv install
directly under Windows, but that's not possible at the moment as some of the dependencies cannot be installed on Windows currently:
https://github.com/sagemath/cypari2/issues/19
https://github.com/sagemath/cysignals/pull/104
comment:45 Changed 2 years ago by
Commit:  c5f3873cc7ddb158646051861fc10c72c6be22f5 → b7fb5a9d46d8acb5068cde9a94009adb800d4783 

comment:46 Changed 2 years ago by
Replying to ghtobiasdiez:
[...] not all pyx files could be compiled successfully since they rely on external dependencies (e.g. singular, giac, ecl, ratpoints, ...). One approach would be to treat them similar to the the optional packages by putting a "sage_setup distribution" tag to the pyx file.
Yes, exactly this approach is part of the "modularization" that I have in mind. See #29705 under the items titled "Deploy mildly modularized distributions", "Further modularization" for a sketch of such distributions. Basically, one distribution for each major C/C++ library (such as sagentl
)  but some libraries come in packs that would not make much sense to separate (such as sageflintarbe_antic
).
comment:47 Changed 2 years ago by
Replying to ghtobiasdiez:
I also tried to run
pipenv install
directly under Windows, but that's not possible at the moment as some of the dependencies cannot be installed on Windows currently: https://github.com/sagemath/cypari2/issues/19 https://github.com/sagemath/cysignals/pull/104
Right, I hope that when modularization is done, a native Windows port of a tiny Sage subset would be an actionable project.
comment:48 Changed 2 years ago by
As in #30578 (where I am trying the same with requirements.txt
and plain venv
), I would like to get the following to work:
 Build the sage distribution (actually all prerequisites of sagelib, using
make sagelibbuilddeps
)
 Then within
sage sh
(where all the libraries installed in$SAGE_LOCAL
are available) do the inplace install into a new venv created bypipenv
 using the wheels built in$SAGE_LOCAL/var/lib/sage/wheels
.
comment:49 followups: 53 54 55 56 Changed 2 years ago by
Thanks for the quick feedback.
I agree the modularization is a good way to approach these problems. Sadly, I feel like this requires too much indepth knowledge of sage for me to contribute directly. But after skimming the tickets, I suggest you have a look at my changes here to the way distributions are excluded. This seems to be somewhat opposite to say #29864 where this is handled by adding a new setuptdlib/setup.py. For the inplace installation it is important that the src/setup.py results in a working installation.
What do you suggest as the way forward to align this ticket with the modularization effort?
As for #30578, my motivation here is a bit different: I wanted to have a prototype for sagelib with pipenv that works without ever invoking sage/make
and directly uses the versions of the dependencies found on pypi instead of (re)building them locally. This seems to work in principle, and might lead to a workable solution for most purposes of development and ci testing, where the advantages of building the dependencies (like optimization) are not that important. That being said, I'm not sure if we really need to have a requirements.txt and a pipfile. I would opt for the more modern version and only keep the pipfile.
After installing singular via aptget install libsingular4dev libpynacdev
, a few more pyx files could be compiled. Should I add this in build\pkgs\singular\distros\debian ?
One more thing: Currently sage/rings/number_field/number_field_element_quadratic.pyx relies on the arb lib, but is also (indirectly) imported in sage.all. What's the best way to handle these imports that depend on external C/C++ libs? For now I've uncommented some of these allimports.
comment:50 Changed 2 years ago by
Commit:  b7fb5a9d46d8acb5068cde9a94009adb800d4783 → 12e6d6711ca76bee221336093fa03e162962a896 

comment:51 Changed 2 years ago by
Yet another question: when/how is src/sage/ext/interpreters generated?
comment:52 Changed 2 years ago by
Commit:  12e6d6711ca76bee221336093fa03e162962a896 → 7a7e5764d619f0207c9d57d1c24b6198f944b799 

Branch pushed to git repo; I updated commit sha1. New commits:
7a7e576  Make import sage.manifolds.all work

comment:53 Changed 2 years ago by
Replying to ghtobiasdiez:
As for #30578, my motivation here is a bit different: I wanted to have a prototype for sagelib with pipenv that works without ever invoking
sage/make
and directly uses the versions of the dependencies found on pypi instead of (re)building them locally.
Yes, I agree with this goal. But I would strongly suggest to focus on a smaller scope for this ticket: Make the inplace install work when all the dependencies are available. At the same time, make sure that the normal Sage build process does not get broken (e.g. take care of comment 34 "Don't change find_python_sources because build/pkgs/sagelib/src/setup.py continues to use it"). This ensures that the ticket can be merged in Sage, thus, a way forward. This is better than preparing one ticket with huge changes.
comment:54 followup: 57 Changed 2 years ago by
Replying to ghtobiasdiez:
I suggest you have a look at my changes here to the way distributions are excluded. This seems to be somewhat opposite to say #29864 where this is handled by adding a new setuptdlib/setup.py.
As discussed, adding these sage_setup: distribution
tags to Cython source files is a way forward that aligns with the modularization effort.
comment:55 followup: 59 Changed 2 years ago by
Replying to ghtobiasdiez:
Currently sage/rings/number_field/number_field_element_quadratic.pyx relies on the arb lib, but is also (indirectly) imported in sage.all. What's the best way to handle these imports that depend on external C/C++ libs?
Try lazy imports with PythonModule
features, from #30616/#30647.
comment:56 Changed 2 years ago by
Replying to ghtobiasdiez:
I suggest you have a look at my changes here to the way distributions are excluded.
Well, it's the same as before in sagelib's setup.py, filtering by distributions, plus some hardcoded ones. Except that you have changed the interface of the function that does the filtering (see above  this change is not mergeable because build/pkgs/sagelib/src/setup.py
needs to keep working!)
comment:57 Changed 2 years ago by
Replying to mkoeppe:
As discussed, adding these
sage_setup: distribution
tags to Cython source files is a way forward that aligns with the modularization effort.
I've opened #30666 for this. (The new distribution names have to be added to setup.py
, of course, so that the files continue to be built in the Sage distribution.)
comment:58 Changed 2 years ago by
Dependencies:  #29500, #30527, #22731, #30578 → #29500, #30527, #22731, #30578, #30666 

comment:59 Changed 2 years ago by
Replying to mkoeppe:
Replying to ghtobiasdiez:
Currently sage/rings/number_field/number_field_element_quadratic.pyx relies on the arb lib, but is also (indirectly) imported in sage.all. What's the best way to handle these imports that depend on external C/C++ libs?
Try lazy imports with
PythonModule
features, from #30616/#30647.
For another approach regarding the ".all" imports, see #29865. Probably we will need a combination of this and lazy imports.
comment:60 Changed 2 years ago by
Commit:  7a7e5764d619f0207c9d57d1c24b6198f944b799 → 4afd411dd86c37e90ca32c8f77de89a91c57a667 

Branch pushed to git repo; I updated commit sha1. New commits:
4afd411  Readd old find_python_sources

comment:61 Changed 2 years ago by
I've now restored the old find_python_sources method. Is there something else that needs to be done, except waiting for the other tickets to be finished?
comment:62 Changed 2 years ago by
Description:  modified (diff) 

comment:64 followup: 65 Changed 2 years ago by
Commit:  4afd411dd86c37e90ca32c8f77de89a91c57a667 → c82bbd56e5ab828b73dfe09ad1c32c9b7d96f866 

comment:65 Changed 2 years ago by
comment:66 Changed 2 years ago by
Dependencies:  #29500, #30527, #22731, #30578, #30666 → #29500, #30527, #22731, #30578, #30666, #30672 

comment:67 Changed 2 years ago by
Dependencies:  #29500, #30527, #22731, #30578, #30666, #30672 → #29500, #30527, #22731, #30578, #30666, #30672, #30673 

comment:68 Changed 2 years ago by
Description:  modified (diff) 

comment:69 Changed 2 years ago by
Commit:  c82bbd56e5ab828b73dfe09ad1c32c9b7d96f866 → 62930ee7ada36664e0fc2264e5395f8b8337b650 

Branch pushed to git repo; I updated commit sha1. New commits:
62930ee  Add ipykernel and tox as dev packages

comment:70 Changed 2 years ago by
Description:  modified (diff) 

New commits:
62930ee  Add ipykernel and tox as dev packages

comment:71 Changed 2 years ago by
Dependencies:  #29500, #30527, #22731, #30578, #30666, #30672, #30673 → #29500, #30527, #22731, #30578, #30672, #30673 

comment:72 Changed 2 years ago by
Commit:  62930ee7ada36664e0fc2264e5395f8b8337b650 → cd196a0773e36e7e6c1fbe21962ef5b1b1485cf5 

comment:73 Changed 2 years ago by
Description:  modified (diff) 

comment:74 Changed 2 years ago by
This ticket is way too big. I really strong recommend to get a working inplace installation of sagelib in the situation when its dependencies have been built using the Sage distribution.
comment:75 followup: 77 Changed 2 years ago by
I've now followed your advice, and tried to have a prototype that works if all library dependencies are fulfilled (instead of adding exceptions / distribution annotations everywhere). And I was successful indeed, import sage.all
works without error and I can do basic calculations with the manifold package. This required tinkering with a lot of systeminstalled packages (instructions added in the ticket).
The following is not yet working, and I would like to get input on how to resolve this:
sage/ext/interpreters
is empty. Where is the code generated? Can this easily be extracted to a script that can be invoked separately? The code relying on the arb library still doesn't compile although I've
libflintarbdev
installed. The compilation fails withld larb
was not found. Why is this, and what's the best way to solve it?  The
sage.doctest
package conflicted with thesympy
(I believe), i.e. sympy tried to load classes fromsage.doctest
for some reason, and couldn't locate them. I've fixed this by renamingsrc/sage/doctest
tosagedoctest
. Is thedoctest
package really required under thesage
folder, or could it be e.g moved up tosrc/doctest
?
I agree the ticket is too big at the moment. Once I found a solution to the above points, I'll clean the code. I sadly cannot work with the dependencies created by the sage distribution, as the build process fails due to compilation errors in the pip library...
comment:76 Changed 2 years ago by
Commit:  cd196a0773e36e7e6c1fbe21962ef5b1b1485cf5 → 1422cf461178919c2ccf1a30b6a82e1ac5dee396 

Branch pushed to git repo; I updated commit sha1. New commits:
1422cf4  Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace

comment:77 Changed 2 years ago by
Replying to ghtobiasdiez:
I've now followed your advice, and tried to have a prototype that works if all library dependencies are fulfilled (instead of adding exceptions / distribution annotations everywhere). And I was successful indeed,
import sage.all
works without error and I can do basic calculations with the manifold package. This required tinkering with a lot of systeminstalled packages (instructions added in the ticket).
Great.
The following is not yet working, and I would like to get input on how to resolve this:
sage/ext/interpreters
is empty. Where is the code generated? Can this easily be extracted to a script that can be invoked separately?
sage_setup.command.sage_build
 The code relying on the arb library still doesn't compile although I've
libflintarbdev
installed. The compilation fails withld larb
was not found. Why is this, and what's the best way to solve it?
Use sage.env.cython_aliases
. In particular ARB_LIBRARY
gives you the name. (The Debian installation uses a different name from upstream.)
 The
sage.doctest
package conflicted with thesympy
(I believe), i.e. sympy tried to load classes fromsage.doctest
for some reason, and couldn't locate them. I've fixed this by renamingsrc/sage/doctest
tosagedoctest
. Is thedoctest
package really required under thesage
folder, or could it be e.g moved up tosrc/doctest
?
Sounds like a bad PYTHONPATH. src/sage
should not be in it.
I agree the ticket is too big at the moment. Once I found a solution to the above points, I'll clean the code. I sadly cannot work with the dependencies created by the sage distribution, as the build process fails due to compilation errors in the pip library...
Hm, that's still happening? Sorry I lost track of this since various other things seemed to break on WSL. Have you tried with downgrading/upgrading the pip package?
comment:78 Changed 2 years ago by
Commit:  1422cf461178919c2ccf1a30b6a82e1ac5dee396 → 05713b87fff7948a68562358a5d32a29701bfa2f 

Branch pushed to git repo; I updated commit sha1. New commits:
05713b8  Remove dependency on #30527

comment:79 Changed 2 years ago by
By the way, for dev purposes until the WSL issues are resolved, you could try running Sage in Docker instead (bindmount the source tree into it)
comment:80 Changed 2 years ago by
Dependencies:  #29500, #30527, #22731, #30578, #30672, #30673 → #29500, #22731, #30672, #30673 

comment:81 Changed 2 years ago by
Dependencies:  #29500, #22731, #30672, #30673 → #29500, #30672, #30673 

comment:82 Changed 2 years ago by
Commit:  05713b87fff7948a68562358a5d32a29701bfa2f → ac07cc0d005de5e067e29c22f154521537f5906d 

Branch pushed to git repo; I updated commit sha1. New commits:
ac07cc0  Move requirements file back

comment:83 Changed 2 years ago by
 Fix installation of
ecl
:After
aptget install ecl
ecl is broken under WSL. To fix this, first runaptget install libtool autoconf
, then follow https://github.com/rdp/ffmpegwindowsbuildhelpers/issues/452#issuecomment638639182.
Could you try the related #30629 on WSL please?
comment:84 Changed 2 years ago by
Dependencies:  #29500, #30672, #30673 → #29500, #30672, #30673, #30590 

Description:  modified (diff) 
comment:85 Changed 2 years ago by
Commit:  ac07cc0d005de5e067e29c22f154521537f5906d → 3023260e2c36e80113daf0af69e95d6091de461e 

comment:86 Changed 2 years ago by
Commit:  3023260e2c36e80113daf0af69e95d6091de461e → 5b12a924507cf85a2eca749993835927b9398668 

comment:87 Changed 2 years ago by
Description:  modified (diff) 

Status:  needs_work → needs_review 
comment:88 Changed 2 years ago by
Perfect, thanks a lot Matthias! I got now a working installation of all libraries, and could import sage.all
without the need to uncomment any imports. Yeaaaah!
I've also cleaned the code and extended the ticket description, so this ticket is now ready for review.
comment:89 Changed 2 years ago by
Could you put the changes to sage_setup/autogen/interpreters/
on a separate ticket please?
comment:90 Changed 2 years ago by
The changes regarding imports (mostly around sage.rings
) should also be on a separate ticket.
comment:91 Changed 2 years ago by
Also for the changes to src/sage/libs/eclsig.h
 which will certainly need discussion
comment:92 followup: 94 Changed 2 years ago by
+compile_time_env = dict( + PY_PLATFORM=sys.platform +) +cython_directives = dict( + language_level="3str", + cdivision=True, +)
This duplicates settings defined in sage_setup.command.sage_build_cython
.
Could you refactor this so that these directives are defined in a reusable way?
Perhaps in a new module sage_setup.cython_options
comment:93 Changed 2 years ago by
Authors:  → Tobias Diez 

comment:94 Changed 2 years ago by
Replying to mkoeppe:
Could you refactor this so that these directives are defined in a reusable way? Perhaps in a new module
sage_setup.cython_options
... in a separate ticket of course.
comment:95 followup: 96 Changed 2 years ago by
Will do so. For the cython options, they are actually only a small subset of the ones in sage_build_cython
. Should I still refactor? (Feels a bit strange as there are more differences than similarities.) But I'm also not sure if the other cython options are needed...
comment:96 Changed 2 years ago by
Replying to ghtobiasdiez:
Will do so. For the cython options, they are actually only a small subset of the ones in
sage_build_cython
. Should I still refactor? (Feels a bit strange as there are more differences than similarities.) But I'm also not sure if the other cython options are needed...
Well, the idea would be to see whether you can just build with the options that Sage uses instead of using your own set of options.
comment:97 Changed 2 years ago by
Whether other Cython build options can/should be used  should be orthogonal to whether an inplace or regular build is done. So it should not be done on this ticket.
comment:98 Changed 2 years ago by
Some comments on items in the ticket description:
TODO (as followup tickets):
 Automatically generate the
aptget install
command with the necessary libraries (add them asdistros/debian.txt
?)
./configure
takes care of all of this. Do you plan to do system interface configuration and system package advice without using ./configure
(or, more generally, Unix shell scripts)?
A way forward is to reimplement (cleaner version of) some of the simple shell code for system package advice as Python code in sage_bootstrap
. See #29146.
 Remove
ARB_LIBRARY
env variable, and move check fromarb/config.m4
toenv.py
My suggestion: Instead of discussing to remove this variable (or other configuration variables that are set by ./configure
), rather discuss different ways of generating the sage_conf
module / or providing different implementations of the interface defined by the sage_conf
module for different use cases (such as supporting a platform that does not have a Unix shell).
 Move package dependencies from pipfile to install_requires in
src/setup.py
.
Best to put them in src/setup.cfg
instead of src/setup.py
(see #30578)
comment:99 Changed 2 years ago by
Commit:  5b12a924507cf85a2eca749993835927b9398668 → 8abab5dfb588a3b6e6fe0c1d32525656ddc13cf6 

comment:100 Changed 2 years ago by
Thanks for preparing this cleaned up branch.
Trying to test it, I seem to be running into something that looks like https://discuss.python.org/t/pip191andinstallingineditablemodewithpyprojecttoml/1553 :
Something is generating a pyproject.toml
in src/
and then it dies with
Writing supplied requirement line to temporary file: 'e .' Installing 'e1839a8' $ ['/Users/mkoeppe/.local/share/virtualenvs/srcBXov0SKG/bin/pip', 'install', 'verbose', 'upgrade', 'nousepep517', 'nodeps', 'existsaction=i', 'r', '/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pipenv_yopr9l3requirements/pipenv7uv8tb0mrequirement.txt', 'i', 'https://pypi.org/simple'] Using source directory: '/Users/mkoeppe/.local/share/virtualenvs/srcBXov0SKG/src' Using pip 20.2.1 from /Users/mkoeppe/.local/share/virtualenvs/srcBXov0SKG/lib/python3.8/sitepackages/pip (python 3.8) Nonuser install by explicit request ... File "/Users/mkoeppe/.local/share/virtualenvs/srcBXov0SKG/lib/python3.8/sitepackages/pip/_internal/distributions/sdist.py", line 33, in prepare_distribution_metadata self.req.load_pyproject_toml() File "/Users/mkoeppe/.local/share/virtualenvs/srcBXov0SKG/lib/python3.8/sitepackages/pip/_internal/req/req_install.py", line 503, in load_pyproject_toml pyproject_toml_data = load_pyproject_toml( File "/Users/mkoeppe/.local/share/virtualenvs/srcBXov0SKG/lib/python3.8/sitepackages/pip/_internal/pyproject.py", line 95, in load_pyproject_toml raise InstallationError( pip._internal.exceptions.InstallationError: Disabling PEP 517 processing is invalid: project specifies a build backend of setuptools.build_meta in pyproject.toml Removed file:///Users/mkoeppe/s/sage/sagerebasing/src (from r /var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pipenv_yopr9l3requirements/pipenv7uv8tb0mrequirement.txt (line 1)) from build tracker '/private/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pipreqtrackerlxrv7bcd' Removed build tracker: '/private/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pipreqtrackerlxrv7bcd'
comment:101 Changed 2 years ago by
Strange. For some reason, for you pip is invoked with nousepep517
. For me it is not:
$ "/mnt/d/Programming/Projects/sage/src/.venv/bin/pip" install verbose nodeps e "." i https://pypi.org/simple existsaction w...
What version of pipenv do you use? For me: pipenv 11.9.0 (and the pip in the virtual env also differs: pip 20.0.2)
comment:103 Changed 2 years ago by
I will try to update my pipenv later. For now you can have a look if setting the environment variable PIP_USE_PEP517 to true resolves the issue.
comment:104 Changed 2 years ago by
I'll try, but I think editable installs are not compatible with PEP517
comment:106 Changed 2 years ago by
Dependencies:  #29500, #30672, #30673, #30590 → #29500, #30672, #30673, #30590, #30709 

comment:107 Changed 2 years ago by
I had a closer look at this, and it turns out that the first time pip is correctly called without any nousepep517
argument, only on the retry this argument is used. Thus, the error actually shadows another error that occurs during the initial pip install
.
Can you thus run the following pipenv run pip install e . nobuildisolation
after pipenv install
. This should point to the real issue why the editable install fails.
I've created a github workflow because I don't have a mac. https://github.com/tobiasdiez/sage/blob/public/build/inplace/.github/workflows/cipipenv.yml It shows the same problem, the pipenv install
fails apparently due to a pep517 issue, but the editable install actually fails due to missing interpreters and other dependencies (which requires manual installations as in the ticket description).
comment:108 Changed 2 years ago by
Hm...
(sagesh) mkoeppe@egret:src$ pipenv run pip install e . nobuildisolation Obtaining file:///Users/mkoeppe/s/sage/sagerebasing/src Preparing wheel metadata ... error ERROR: Command errored out with exit status 1: command: /Users/mkoeppe/.local/share/virtualenvs/srcBXov0SKG/bin/python /Users/mkoeppe/.local/share/virtualenvs/srcBXov0SKG/lib/python3.8/sitepackages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/tmpvt150u93 cwd: /Users/mkoeppe/s/sage/sagerebasing/src Complete output (14 lines): Traceback (most recent call last): File "/Users/mkoeppe/.local/share/virtualenvs/srcBXov0SKG/lib/python3.8/sitepackages/pip/_vendor/pep517/_in_process.py", line 280, in <module> main() File "/Users/mkoeppe/.local/share/virtualenvs/srcBXov0SKG/lib/python3.8/sitepackages/pip/_vendor/pep517/_in_process.py", line 263, in main json_out['return_val'] = hook(**hook_input['kwargs']) File "/Users/mkoeppe/.local/share/virtualenvs/srcBXov0SKG/lib/python3.8/sitepackages/pip/_vendor/pep517/_in_process.py", line 133, in prepare_metadata_for_build_wheel return hook(metadata_directory, config_settings) File "/Users/mkoeppe/.local/share/virtualenvs/srcBXov0SKG/lib/python3.8/sitepackages/setuptools/build_meta.py", line 161, in prepare_metadata_for_build_wheel self.run_setup() File "/Users/mkoeppe/.local/share/virtualenvs/srcBXov0SKG/lib/python3.8/sitepackages/setuptools/build_meta.py", line 145, in run_setup exec(compile(code, __file__, 'exec'), locals()) File "setup.py", line 23, in <module> import sage.env ModuleNotFoundError: No module named 'sage'  ERROR: Command errored out with exit status 1: /Users/mkoeppe/.local/share/virtualenvs/srcBXov0SKG/bin/python /Users/mkoeppe/.local/share/virtualenvs/srcBXov0SKG/lib/python3.8/sitepackages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/tmpvt150u93 Check the logs for full command output.
comment:109 Changed 2 years ago by
No idea....for the github action this is not a problem: https://github.com/tobiasdiez/sage/runs/1205452529?check_suite_focus=true. Maybe pipenv run pip install e . verbose upgrade existsaction=i nobuildisolation
works, or you need to clean the venv first (pipenv rm).
comment:111 Changed 2 years ago by
Description:  modified (diff) 

comment:112 Changed 2 years ago by
Commit:  8abab5dfb588a3b6e6fe0c1d32525656ddc13cf6 → edde293052d7e4660df11ce13513f08edf44c5e5 

Branch pushed to git repo; I updated commit sha1. New commits:
edde293  Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace

comment:113 Changed 2 years ago by
Commit:  edde293052d7e4660df11ce13513f08edf44c5e5 → 74ecddbe6a71f133458cdb018f45e36ba0aaf818 

Branch pushed to git repo; I updated commit sha1. New commits:
74ecddb  Update dependencies

comment:114 Changed 2 years ago by
Commit:  74ecddbe6a71f133458cdb018f45e36ba0aaf818 → 7110841f2c816001c14205c1426088fb873499b3 

comment:115 Changed 2 years ago by
Dependencies:  #29500, #30672, #30673, #30590, #30709 → #30779, #30672, #30673, #30709 

comment:116 Changed 2 years ago by
Commit:  7110841f2c816001c14205c1426088fb873499b3 → 0b5403699f9472f102b4781e3fd9ebae80e6bb8a 

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
52e2107  sage_include_directories: Remove sage/ext

6ccce6b  Fix missing ccobject

1402129  Merge branch 'u/mkoeppe/remove__sage_ext__from__sage_include_directories_' of git://trac.sagemath.org/sage into t/30371/public/build/inplace

6eb3845  Extend gitignore and add VS code config

bbe73ee  Remove VS code settings, and add exceptions for already checkedin files

22b7cac  Merge branch 'public/build/enhancegitignore' of git://trac.sagemath.org/sage into t/30371/public/build/inplace

1aef376  Extract cython options to new module

68ab47b  Merge branch 'public/build/cythonConfig' of git://trac.sagemath.org/sage into t/30371/public/build/inplace

d1b1288  Narrow down to essenital changes

0b54036  Update dependencies

comment:118 Changed 2 years ago by
Status:  needs_review → needs_work 

Work issues:  → Use the cython options 
comment:119 Changed 2 years ago by
Commit:  0b5403699f9472f102b4781e3fd9ebae80e6bb8a → 55930254f941314d9996bfc6a9a4d5f9767c2e6d 

Branch pushed to git repo; I updated commit sha1. New commits:
82e8b7d  Narrow down to essenital changes

8abab5d  Add copy of setup.py

edde293  Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace

74ecddb  Update dependencies

1286fcb  Merge branch 'public/build/inplace' of git://trac.sagemath.org/sage into public/build/inplace

946737e  Use new cython option modules

5593025  Update dependencies

comment:120 Changed 2 years ago by
Status:  needs_work → needs_review 

Work issues:  Use the cython options 
Thanks for the rebase. Next time please try to not forcepush to a branch that others have checked out because it messes up their git history.
I've updated the code to use the new cython options module.
comment:121 Changed 2 years ago by
Sorry if the push made it harder for you to continue on this branch. But rebasing it on top of the branches of the dependency tickets was a necessary cleanup
comment:122 Changed 2 years ago by
Description:  modified (diff) 

comment:123 Changed 2 years ago by
What's the status of this branch? Does it work for you? Can you run doctests in the inplace installation and does it pass them?
comment:124 followup: 125 Changed 2 years ago by
It's working for me. pipenv install dev
gives a fully functional virtual environment (so the cython compilation works). Moreover, import sage.all
works as well, and one can then use it to run sagecode. So far, I've tested it with some code from the manifolds package. Not yet tried doctests, as I've no idea how to run them from the virtual environment.
comment:125 Changed 2 years ago by
Replying to ghtobiasdiez:
Not yet tried doctests, as I've no idea how to run them from the virtual environment.
Well, the virtual environment has a bin/sage
script, so sage t
"should" work. If it does not, this needs to be fixed. Likely some work in the direction of #22731 is necessary.
comment:126 followup: 127 Changed 2 years ago by
Yeah, sage t
still uses the python in sage/local
. So this does not work, and a mechanism is needed to use the python of the activated virtual env (without changing env variables which is too inflexible in my opinion).
Directly running bin/sageruntests all
also leads to errors:
no stored timings available Running doctests with ID 20201020211829b0a05b3d. Using optional=dochtml,memlimit,sage,sage_spkg Doctesting entire Sage library. Traceback (most recent call last): File "bin/sageruntests", line 182, in <module> err = DC.run() File "/mnt/d/Programming/Projects/sage/src/sage/doctest/control.py", line 1234, in run self.expand_files_into_sources() File "/mnt/d/Programming/Projects/sage/src/sage/doctest/control.py", line 817, in expand_files_into_sources self.sources = [FileDocTestSource(path, self.options) for path in expand()] File "/mnt/d/Programming/Projects/sage/src/sage/doctest/control.py", line 817, in <listcomp> self.sources = [FileDocTestSource(path, self.options) for path in expand()] File "/mnt/d/Programming/Projects/sage/src/sage/doctest/sources.py", line 528, in __init__ raise ValueError("unknown file extension %r"%ext) ValueError: unknown file extension ''
(I guess some files are not found.)
Anyway, that's probably a topic for another ticket.
comment:127 Changed 2 years ago by
Replying to ghtobiasdiez:
Yeah,
sage t
still uses the python insage/local
. So this does not work, and a mechanism is needed to use the python of the activated virtual env (without changing env variables which is too inflexible in my opinion).
Yes, this is precisely #22731
comment:128 Changed 2 years ago by
Commit:  55930254f941314d9996bfc6a9a4d5f9767c2e6d → 9dc8a2b0bcd6ff1cada426c60184b322312618e4 

Branch pushed to git repo; I updated commit sha1. New commits:
9dc8a2b  Generate code automatically

comment:129 Changed 2 years ago by
Commit:  9dc8a2b0bcd6ff1cada426c60184b322312618e4 → 19482048d4fffe52e94fcd94b81fa8ef59000558 

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
fcef2b6  Use pip to install pipenv

4160c9f  Update cipipenv.yml

d0e01f2  Update cipipenv.yml

711a2a6  Update cipipenv.yml

341be72  Update cipipenv.yml

918e1a5  Update cipipenv.yml

573ec51  Try with only one python

961abc3  Update cipipenv.yml

5a9b803  Update cipipenv.yml

1948204  Merge branch 'public/build/inplace' of https://github.com/tobiasdiez/sage into public/build/inplace

comment:130 Changed 2 years ago by
Commit:  19482048d4fffe52e94fcd94b81fa8ef59000558 → fffea7a7a50dd8c79827a07d6ef4eb3568fc1817 

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
0016538  Merge branch 'public/build/inplace' of https://github.com/tobiasdiez/sage into public/build/inplace

d65d85e  Update cipipenv.yml

c1b8aef  Merge branch 'public/build/inplace' of https://github.com/tobiasdiez/sage into public/build/inplace

23d17ad  Rename cblas to blas

3cfe1c9  Merge branch 'develop' of git://github.com/sagemath/sage into public/build/cblas

ea4a44c  Support different blas libraries

75966fc  Put cblas first

273bb93  Make cblas pc module search configurable via sage_conf

e6ff9ba  Merge branch 'public/build/cblas' of git://trac.sagemath.org/sage into public/build/inplace

fffea7a  Install more dependencies

comment:131 Changed 2 years ago by
Dependencies:  #30779, #30672, #30673, #30709 → #30779, #30672, #30673, #30709, #30706 

comment:132 Changed 2 years ago by
Commit:  fffea7a7a50dd8c79827a07d6ef4eb3568fc1817 → 062746cad45c3c3edc9f0d905de8312df1c020ac 

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
ed74c46  Fix powershell script

88f4529  Make powershell run

4171062  Cleanup

bfdc1c0  Fix problem with libhomfly

3bda21e  Fix arb lib

3d476d5  Install more deps for macos (as recommended by configure)

4302ac9  Also install ecl

e34a6af  Also install pynac

8679770  Disable copying of sageenvconfig

062746c  Try with hardcoded version

comment:133 followup: 135 Changed 2 years ago by
I've now added github actions to have a reproducible test in a clean environment. The compilation is working now (under linux), but the singular library cannot be found during runtime. https://github.com/tobiasdiez/sage/runs/1287237342?check_suite_focus=true
The problem appears to be that sage is looking for singular in sysconfig.get_config_var('LIBDIR')
(e.g. /usr/local/lib
) but it is installed as /usr/lib/x86_64linuxgnu/libsingularSingular.so
. Is there a reason why LIBDIR
is used in env.py
instead of the approach using pkgconfig
as during compilation time?
comment:134 Changed 2 years ago by
_get_shared_lib_filename
actually also looks in MULTILIB
and is supposed to find this Debianspecific install location via this mechanism. But it is possible that this is broken because we currently actually do not use system singular
(nor gap
) at all (see #29644, #29024, metaticket #27330). This needs more investigation.
comment:135 Changed 2 years ago by
Replying to ghtobiasdiez:
Is there a reason why
LIBDIR
is used inenv.py
instead of the approach usingpkgconfig
as during compilation time?
pkgconfig does not really reveal the libname directly, so one would need to parse pkgconfig libs
.
Another reason is that the .pc files are usually only available in "dev" packages. So the runtime of Sage should not depend on it.
#29024 intends to make the location configurable (at build time!) via sage_conf
.
comment:136 Changed 2 years ago by
Commit:  062746cad45c3c3edc9f0d905de8312df1c020ac → 7523baa68b46732cc381387e58ea1a9559c11e96 

comment:137 followup: 139 Changed 2 years ago by
Thanks for the input. I had a closer look at _get_shared_lib_filename
and the problem was that libsingularSingular.so
is in the subfolder /usr/lib/x86_64linuxgnu/
while _get_shared_lib_filename
only searched the top folder /usr/lib
. I've now changed it to also look in subfolders. That should be reasonable fix until #29024 is implemented.
With these changes, the github action workflow passes successful on linux: https://github.com/tobiasdiez/sage/runs/1303324520?check_suite_focus=true It's nice to see that it only takes about 1 hour to have a fully workable sage environment.
This is now really ready for review.
TODOs (for further tickets):
 Investigate what needs to be installed on macOS to have a successful build, and add these steps to the github action. (I don't have a mac, so I'll not work on this.)
 Run doctests in the github action as soon as they no longer rely on
SAGE_LOCAL
(#22731)
comment:138 Changed 2 years ago by
Description:  modified (diff) 

comment:139 Changed 2 years ago by
Replying to ghtobiasdiez:
Thanks for the input. I had a closer look at
_get_shared_lib_filename
and the problem was thatlibsingularSingular.so
is in the subfolder/usr/lib/x86_64linuxgnu/
while_get_shared_lib_filename
only searched the top folder/usr/lib
. I've now changed it to also look in subfolders.
This should have been taking care of by MULTILIB
. Is this not configured in your system?
comment:140 Changed 2 years ago by
MULTILIB
is empty on my system (WSL Ubuntu 20.04), nor was it working for the github actions (Ubuntu 20.10).
comment:141 Changed 2 years ago by
Can you show the full output of python3 m sysconfig
on this system?
comment:142 Changed 2 years ago by
Platform: "linuxx86_64" Python version: "3.8" Current installation scheme: "posix_prefix" Paths: data = "/mnt/d/Programming/Projects/sage/src/.venv" include = "/usr/include/python3.8" platinclude = "/usr/include/python3.8" platlib = "/mnt/d/Programming/Projects/sage/src/.venv/lib/python3.8/sitepackages" platstdlib = "/mnt/d/Programming/Projects/sage/src/.venv/lib/python3.8" purelib = "/mnt/d/Programming/Projects/sage/src/.venv/lib/python3.8/sitepackages" scripts = "/mnt/d/Programming/Projects/sage/src/.venv/bin" stdlib = "/usr/lib/python3.8" Variables: ABIFLAGS = "" AC_APPLE_UNIVERSAL_BUILD = "0" AIX_GENUINE_CPLUSPLUS = "0" ALT_SOABI = "0" ANDROID_API_LEVEL = "0" AR = "x86_64linuxgnugccar" ARFLAGS = "rcs" BASECFLAGS = "Wnounusedresult Wsigncompare" BASECPPFLAGS = "IObjects IInclude IPython" BASEMODLIBS = "" BINDIR = "/usr/bin" BINLIBDEST = "/usr/lib/python3.8" BLDLIBRARY = "lpython3.8" BLDSHARED = "x86_64linuxgnugcc pthread shared Wl,O1 Wl,Bsymbolicfunctions Wl,Bsymbolicfunctions Wl,z,relro g fwrapv O2 " BUILDEXE = "" BUILDPYTHON = "python" BUILD_GNU_TYPE = "x86_64pclinuxgnu" BYTESTR_DEPS = "\" CC = "x86_64linuxgnugcc pthread" CCSHARED = "fPIC" CFLAGS = "Wnounusedresult Wsigncompare DNDEBUG g fwrapv O2 Wall g fstackprotectorstrong Wformat Werror=formatsecurity g fwrapv O2 " CFLAGSFORSHARED = "fPIC" CFLAGS_ALIASING = "" CFLAGS_NODIST = "" CONFIGFILES = "configure configure.ac acconfig.h pyconfig.h.in Makefile.pre.in" CONFIGURE_CFLAGS = "g fstackprotectorstrong Wformat Werror=formatsecurity" CONFIGURE_CFLAGS_NODIST = "std=c99 Wextra Wnounusedresult Wnounusedparameter Wnomissingfieldinitializers Werror=implicitfunctiondeclaration" CONFIGURE_CPPFLAGS = "Wdatetime D_FORTIFY_SOURCE=2" CONFIGURE_LDFLAGS = "Wl,Bsymbolicfunctions Wl,z,relro g fwrapv O2 " CONFIGURE_LDFLAGS_NODIST = "" CONFIG_ARGS = "'enableshared' 'prefix=/usr' 'enableipv6' 'enableloadablesqliteextensions' 'withdbmliborder=bdb:gdbm' 'withcomputedgotos' 'withoutensurepip' 'withsystemexpat' 'withsystemlibmpdec' 'withdtrace' 'withsystemffi' 'CC=x86_64linuxgnugcc' 'CFLAGS=g fstackprotectorstrong Wformat Werror=formatsecurity ' 'LDFLAGS=Wl,Bsymbolicfunctions Wl,z,relro g fwrapv O2 ' 'CPPFLAGS=Wdatetime D_FORTIFY_SOURCE=2'" CONFINCLUDEDIR = "/usr/include" CONFINCLUDEPY = "/usr/include/python3.8" COREPYTHONPATH = "" COVERAGE_INFO = "/build/python3.86QL2k7/python3.83.8.2/buildshared/coverage.info" COVERAGE_REPORT = "/build/python3.86QL2k7/python3.83.8.2/buildshared/lcovreport" COVERAGE_REPORT_OPTIONS = "nobranchcoverage title "CPython lcov report"" CPPFLAGS = "IObjects IInclude IPython I. I../Include Wdatetime D_FORTIFY_SOURCE=2" CXX = "x86_64linuxgnug++ pthread" DESTDIRS = "/usr /usr/lib /usr/lib/python3.8 /usr/lib/python3.8/libdynload" DESTLIB = "/usr/lib/python3.8" DESTPATH = "" DESTSHARED = "/usr/lib/python3.8/libdynload" DFLAGS = "" DIRMODE = "755" DIST = "README.rst ChangeLog configure configure.ac acconfig.h pyconfig.h.in Makefile.pre.in Include Lib Misc Extdummy" DISTDIRS = "Include Lib Misc Extdummy" DISTFILES = "README.rst ChangeLog configure configure.ac acconfig.h pyconfig.h.in Makefile.pre.in" DLINCLDIR = "." DLLLIBRARY = "" DOUBLE_IS_ARM_MIXED_ENDIAN_IEEE754 = "0" DOUBLE_IS_BIG_ENDIAN_IEEE754 = "0" DOUBLE_IS_LITTLE_ENDIAN_IEEE754 = "1" DTRACE = "/usr/bin/dtrace" DTRACE_DEPS = "\" DTRACE_HEADERS = "Include/pydtrace_probes.h" DTRACE_OBJS = "Python/pydtrace.o" DYNLOADFILE = "dynload_shlib.o" ENABLE_IPV6 = "1" ENSUREPIP = "no" EXE = "" EXEMODE = "755" EXTRATESTOPTS = "" EXT_SUFFIX = ".cpython38x86_64linuxgnu.so" FILEMODE = "644" FLOAT_WORDS_BIGENDIAN = "0" FLOCK_NEEDS_LIBBSD = "0" GETPGRP_HAVE_ARG = "0" GETTIMEOFDAY_NO_TZ = "0" GITBRANCH = "" GITTAG = "" GITVERSION = "" GNULD = "yes" HAVE_ACCEPT4 = "1" HAVE_ACOSH = "1" HAVE_ADDRINFO = "1" HAVE_ALARM = "1" HAVE_ALIGNED_REQUIRED = "0" HAVE_ALLOCA_H = "1" HAVE_ALTZONE = "0" HAVE_ASINH = "1" HAVE_ASM_TYPES_H = "1" HAVE_ATANH = "1" HAVE_BIND_TEXTDOMAIN_CODESET = "1" HAVE_BLUETOOTH_BLUETOOTH_H = "1" HAVE_BLUETOOTH_H = "0" HAVE_BROKEN_MBSTOWCS = "0" HAVE_BROKEN_NICE = "0" HAVE_BROKEN_PIPE_BUF = "0" HAVE_BROKEN_POLL = "0" HAVE_BROKEN_POSIX_SEMAPHORES = "0" HAVE_BROKEN_PTHREAD_SIGMASK = "0" HAVE_BROKEN_SEM_GETVALUE = "0" HAVE_BROKEN_UNSETENV = "0" HAVE_BUILTIN_ATOMIC = "1" HAVE_CHFLAGS = "0" HAVE_CHOWN = "1" HAVE_CHROOT = "1" HAVE_CLOCK = "1" HAVE_CLOCK_GETRES = "1" HAVE_CLOCK_GETTIME = "1" HAVE_CLOCK_SETTIME = "1" HAVE_COMPUTED_GOTOS = "1" HAVE_CONFSTR = "1" HAVE_CONIO_H = "0" HAVE_COPYSIGN = "1" HAVE_COPY_FILE_RANGE = "1" HAVE_CRYPT_H = "1" HAVE_CRYPT_R = "1" HAVE_CTERMID = "1" HAVE_CTERMID_R = "0" HAVE_CURSES_FILTER = "1" HAVE_CURSES_H = "1" HAVE_CURSES_HAS_KEY = "1" HAVE_CURSES_IMMEDOK = "1" HAVE_CURSES_IS_PAD = "1" HAVE_CURSES_IS_TERM_RESIZED = "1" HAVE_CURSES_RESIZETERM = "1" HAVE_CURSES_RESIZE_TERM = "1" HAVE_CURSES_SYNCOK = "1" HAVE_CURSES_TYPEAHEAD = "1" HAVE_CURSES_USE_ENV = "1" HAVE_CURSES_WCHGAT = "1" HAVE_DECL_ISFINITE = "1" HAVE_DECL_ISINF = "1" HAVE_DECL_ISNAN = "1" HAVE_DECL_RTLD_DEEPBIND = "1" HAVE_DECL_RTLD_GLOBAL = "1" HAVE_DECL_RTLD_LAZY = "1" HAVE_DECL_RTLD_LOCAL = "1" HAVE_DECL_RTLD_MEMBER = "0" HAVE_DECL_RTLD_NODELETE = "1" HAVE_DECL_RTLD_NOLOAD = "1" HAVE_DECL_RTLD_NOW = "1" HAVE_DECL_TZNAME = "0" HAVE_DEVICE_MACROS = "1" HAVE_DEV_PTC = "0" HAVE_DEV_PTMX = "1" HAVE_DIRECT_H = "0" HAVE_DIRENT_D_TYPE = "1" HAVE_DIRENT_H = "1" HAVE_DIRFD = "1" HAVE_DLFCN_H = "1" HAVE_DLOPEN = "1" HAVE_DUP2 = "1" HAVE_DUP3 = "1" HAVE_DYNAMIC_LOADING = "1" HAVE_ENDIAN_H = "1" HAVE_EPOLL = "1" HAVE_EPOLL_CREATE1 = "1" HAVE_ERF = "1" HAVE_ERFC = "1" HAVE_ERRNO_H = "1" HAVE_EXECV = "1" HAVE_EXPLICIT_BZERO = "1" HAVE_EXPLICIT_MEMSET = "0" HAVE_EXPM1 = "1" HAVE_FACCESSAT = "1" HAVE_FCHDIR = "1" HAVE_FCHMOD = "1" HAVE_FCHMODAT = "1" HAVE_FCHOWN = "1" HAVE_FCHOWNAT = "1" HAVE_FCNTL_H = "1" HAVE_FDATASYNC = "1" HAVE_FDOPENDIR = "1" HAVE_FDWALK = "0" HAVE_FEXECVE = "1" HAVE_FINITE = "1" HAVE_FLOCK = "1" HAVE_FORK = "1" HAVE_FORKPTY = "1" HAVE_FPATHCONF = "1" HAVE_FSEEK64 = "0" HAVE_FSEEKO = "1" HAVE_FSTATAT = "1" HAVE_FSTATVFS = "1" HAVE_FSYNC = "1" HAVE_FTELL64 = "0" HAVE_FTELLO = "1" HAVE_FTIME = "1" HAVE_FTRUNCATE = "1" HAVE_FUTIMENS = "1" HAVE_FUTIMES = "1" HAVE_FUTIMESAT = "1" HAVE_GAI_STRERROR = "1" HAVE_GAMMA = "1" HAVE_GCC_ASM_FOR_MC68881 = "0" HAVE_GCC_ASM_FOR_X64 = "1" HAVE_GCC_ASM_FOR_X87 = "1" HAVE_GCC_UINT128_T = "1" HAVE_GETADDRINFO = "1" HAVE_GETC_UNLOCKED = "1" HAVE_GETENTROPY = "1" HAVE_GETGRGID_R = "1" HAVE_GETGRNAM_R = "1" HAVE_GETGROUPLIST = "1" HAVE_GETGROUPS = "1" HAVE_GETHOSTBYNAME = "0" HAVE_GETHOSTBYNAME_R = "1" HAVE_GETHOSTBYNAME_R_3_ARG = "0" HAVE_GETHOSTBYNAME_R_5_ARG = "0" HAVE_GETHOSTBYNAME_R_6_ARG = "1" HAVE_GETITIMER = "1" HAVE_GETLOADAVG = "1" HAVE_GETLOGIN = "1" HAVE_GETNAMEINFO = "1" HAVE_GETPAGESIZE = "1" HAVE_GETPEERNAME = "1" HAVE_GETPGID = "1" HAVE_GETPGRP = "1" HAVE_GETPID = "1" HAVE_GETPRIORITY = "1" HAVE_GETPWENT = "1" HAVE_GETPWNAM_R = "1" HAVE_GETPWUID_R = "1" HAVE_GETRANDOM = "1" HAVE_GETRANDOM_SYSCALL = "1" HAVE_GETRESGID = "1" HAVE_GETRESUID = "1" HAVE_GETSID = "1" HAVE_GETSPENT = "1" HAVE_GETSPNAM = "1" HAVE_GETTIMEOFDAY = "1" HAVE_GETWD = "1" HAVE_GLIBC_MEMMOVE_BUG = "0" HAVE_GRP_H = "1" HAVE_HSTRERROR = "1" HAVE_HTOLE64 = "1" HAVE_HYPOT = "1" HAVE_IEEEFP_H = "0" HAVE_IF_NAMEINDEX = "1" HAVE_INET_ATON = "1" HAVE_INET_PTON = "1" HAVE_INITGROUPS = "1" HAVE_INTTYPES_H = "1" HAVE_IO_H = "0" HAVE_IPA_PURE_CONST_BUG = "0" HAVE_KILL = "1" HAVE_KILLPG = "1" HAVE_KQUEUE = "0" HAVE_LANGINFO_H = "1" HAVE_LARGEFILE_SUPPORT = "0" HAVE_LCHFLAGS = "0" HAVE_LCHMOD = "0" HAVE_LCHOWN = "1" HAVE_LGAMMA = "1" HAVE_LIBDL = "1" HAVE_LIBDLD = "0" HAVE_LIBIEEE = "0" HAVE_LIBINTL_H = "1" HAVE_LIBREADLINE = "1" HAVE_LIBRESOLV = "0" HAVE_LIBSENDFILE = "0" HAVE_LIBUTIL_H = "0" HAVE_LINK = "1" HAVE_LINKAT = "1" HAVE_LINUX_CAN_BCM_H = "1" HAVE_LINUX_CAN_H = "1" HAVE_LINUX_CAN_RAW_FD_FRAMES = "1" HAVE_LINUX_CAN_RAW_H = "1" HAVE_LINUX_MEMFD_H = "1" HAVE_LINUX_NETLINK_H = "1" HAVE_LINUX_QRTR_H = "1" HAVE_LINUX_RANDOM_H = "1" HAVE_LINUX_TIPC_H = "1" HAVE_LINUX_VM_SOCKETS_H = "1" HAVE_LOCKF = "1" HAVE_LOG1P = "1" HAVE_LOG2 = "1" HAVE_LONG_DOUBLE = "1" HAVE_LSTAT = "1" HAVE_LUTIMES = "1" HAVE_MADVISE = "1" HAVE_MAKEDEV = "1" HAVE_MBRTOWC = "1" HAVE_MEMFD_CREATE = "1" HAVE_MEMORY_H = "1" HAVE_MEMRCHR = "1" HAVE_MKDIRAT = "1" HAVE_MKFIFO = "1" HAVE_MKFIFOAT = "1" HAVE_MKNOD = "1" HAVE_MKNODAT = "1" HAVE_MKTIME = "1" HAVE_MMAP = "1" HAVE_MREMAP = "1" HAVE_NCURSES_H = "1" HAVE_NDIR_H = "0" HAVE_NETPACKET_PACKET_H = "1" HAVE_NET_IF_H = "1" HAVE_NICE = "1" HAVE_OPENAT = "1" HAVE_OPENPTY = "1" HAVE_PATHCONF = "1" HAVE_PAUSE = "1" HAVE_PIPE2 = "1" HAVE_PLOCK = "0" HAVE_POLL = "1" HAVE_POLL_H = "1" HAVE_POSIX_FADVISE = "1" HAVE_POSIX_FALLOCATE = "1" HAVE_POSIX_SPAWN = "1" HAVE_POSIX_SPAWNP = "1" HAVE_PREAD = "1" HAVE_PREADV = "1" HAVE_PREADV2 = "1" HAVE_PRLIMIT = "1" HAVE_PROCESS_H = "0" HAVE_PROTOTYPES = "1" HAVE_PTHREAD_CONDATTR_SETCLOCK = "1" HAVE_PTHREAD_DESTRUCTOR = "0" HAVE_PTHREAD_GETCPUCLOCKID = "1" HAVE_PTHREAD_H = "1" HAVE_PTHREAD_INIT = "0" HAVE_PTHREAD_KILL = "1" HAVE_PTHREAD_SIGMASK = "1" HAVE_PTY_H = "1" HAVE_PUTENV = "1" HAVE_PWRITE = "1" HAVE_PWRITEV = "1" HAVE_PWRITEV2 = "1" HAVE_READLINK = "1" HAVE_READLINKAT = "1" HAVE_READV = "1" HAVE_REALPATH = "1" HAVE_RENAMEAT = "1" HAVE_RL_APPEND_HISTORY = "1" HAVE_RL_CATCH_SIGNAL = "1" HAVE_RL_COMPLETION_APPEND_CHARACTER = "1" HAVE_RL_COMPLETION_DISPLAY_MATCHES_HOOK = "1" HAVE_RL_COMPLETION_MATCHES = "1" HAVE_RL_COMPLETION_SUPPRESS_APPEND = "1" HAVE_RL_PRE_INPUT_HOOK = "1" HAVE_RL_RESIZE_TERMINAL = "1" HAVE_ROUND = "1" HAVE_RTPSPAWN = "0" HAVE_SCHED_GET_PRIORITY_MAX = "1" HAVE_SCHED_H = "1" HAVE_SCHED_RR_GET_INTERVAL = "1" HAVE_SCHED_SETAFFINITY = "1" HAVE_SCHED_SETPARAM = "1" HAVE_SCHED_SETSCHEDULER = "1" HAVE_SEM_GETVALUE = "1" HAVE_SEM_OPEN = "1" HAVE_SEM_TIMEDWAIT = "1" HAVE_SEM_UNLINK = "1" HAVE_SENDFILE = "1" HAVE_SETEGID = "1" HAVE_SETEUID = "1" HAVE_SETGID = "1" HAVE_SETGROUPS = "1" HAVE_SETHOSTNAME = "1" HAVE_SETITIMER = "1" HAVE_SETLOCALE = "1" HAVE_SETPGID = "1" HAVE_SETPGRP = "1" HAVE_SETPRIORITY = "1" HAVE_SETREGID = "1" HAVE_SETRESGID = "1" HAVE_SETRESUID = "1" HAVE_SETREUID = "1" HAVE_SETSID = "1" HAVE_SETUID = "1" HAVE_SETVBUF = "1" HAVE_SHADOW_H = "1" HAVE_SHM_OPEN = "1" HAVE_SHM_UNLINK = "1" HAVE_SIGACTION = "1" HAVE_SIGALTSTACK = "1" HAVE_SIGFILLSET = "1" HAVE_SIGINFO_T_SI_BAND = "1" HAVE_SIGINTERRUPT = "1" HAVE_SIGNAL_H = "1" HAVE_SIGPENDING = "1" HAVE_SIGRELSE = "1" HAVE_SIGTIMEDWAIT = "1" HAVE_SIGWAIT = "1" HAVE_SIGWAITINFO = "1" HAVE_SNPRINTF = "1" HAVE_SOCKADDR_ALG = "1" HAVE_SOCKADDR_SA_LEN = "0" HAVE_SOCKADDR_STORAGE = "1" HAVE_SOCKETPAIR = "1" HAVE_SPAWN_H = "1" HAVE_SSIZE_T = "1" HAVE_STATVFS = "1" HAVE_STAT_TV_NSEC = "1" HAVE_STAT_TV_NSEC2 = "0" HAVE_STDARG_PROTOTYPES = "1" HAVE_STDINT_H = "1" HAVE_STDLIB_H = "1" HAVE_STD_ATOMIC = "1" HAVE_STRDUP = "1" HAVE_STRFTIME = "1" HAVE_STRINGS_H = "1" HAVE_STRING_H = "1" HAVE_STRLCPY = "0" HAVE_STROPTS_H = "0" HAVE_STRSIGNAL = "1" HAVE_STRUCT_PASSWD_PW_GECOS = "1" HAVE_STRUCT_PASSWD_PW_PASSWD = "1" HAVE_STRUCT_STAT_ST_BIRTHTIME = "0" HAVE_STRUCT_STAT_ST_BLKSIZE = "1" HAVE_STRUCT_STAT_ST_BLOCKS = "1" HAVE_STRUCT_STAT_ST_FLAGS = "0" HAVE_STRUCT_STAT_ST_GEN = "0" HAVE_STRUCT_STAT_ST_RDEV = "1" HAVE_STRUCT_TM_TM_ZONE = "1" HAVE_SYMLINK = "1" HAVE_SYMLINKAT = "1" HAVE_SYNC = "1" HAVE_SYSCONF = "1" HAVE_SYSEXITS_H = "1" HAVE_SYS_AUDIOIO_H = "0" HAVE_SYS_BSDTTY_H = "0" HAVE_SYS_DEVPOLL_H = "0" HAVE_SYS_DIR_H = "0" HAVE_SYS_ENDIAN_H = "0" HAVE_SYS_EPOLL_H = "1" HAVE_SYS_EVENT_H = "0" HAVE_SYS_FILE_H = "1" HAVE_SYS_IOCTL_H = "1" HAVE_SYS_KERN_CONTROL_H = "0" HAVE_SYS_LOADAVG_H = "0" HAVE_SYS_LOCK_H = "0" HAVE_SYS_MEMFD_H = "0" HAVE_SYS_MKDEV_H = "0" HAVE_SYS_MMAN_H = "1" HAVE_SYS_MODEM_H = "0" HAVE_SYS_NDIR_H = "0" HAVE_SYS_PARAM_H = "1" HAVE_SYS_POLL_H = "1" HAVE_SYS_RANDOM_H = "1" HAVE_SYS_RESOURCE_H = "1" HAVE_SYS_SELECT_H = "1" HAVE_SYS_SENDFILE_H = "1" HAVE_SYS_SOCKET_H = "1" HAVE_SYS_STATVFS_H = "1" HAVE_SYS_STAT_H = "1" HAVE_SYS_SYSCALL_H = "1" HAVE_SYS_SYSMACROS_H = "1" HAVE_SYS_SYS_DOMAIN_H = "0" HAVE_SYS_TERMIO_H = "0" HAVE_SYS_TIMES_H = "1" HAVE_SYS_TIME_H = "1" HAVE_SYS_TYPES_H = "1" HAVE_SYS_UIO_H = "1" HAVE_SYS_UN_H = "1" HAVE_SYS_UTSNAME_H = "1" HAVE_SYS_WAIT_H = "1" HAVE_SYS_XATTR_H = "1" HAVE_TCGETPGRP = "1" HAVE_TCSETPGRP = "1" HAVE_TEMPNAM = "1" HAVE_TERMIOS_H = "1" HAVE_TERM_H = "1" HAVE_TGAMMA = "1" HAVE_TIMEGM = "1" HAVE_TIMES = "1" HAVE_TMPFILE = "1" HAVE_TMPNAM = "1" HAVE_TMPNAM_R = "1" HAVE_TM_ZONE = "1" HAVE_TRUNCATE = "1" HAVE_TZNAME = "0" HAVE_UCS4_TCL = "0" HAVE_UNAME = "1" HAVE_UNISTD_H = "1" HAVE_UNLINKAT = "1" HAVE_UNSETENV = "1" HAVE_USABLE_WCHAR_T = "0" HAVE_UTIL_H = "0" HAVE_UTIMENSAT = "1" HAVE_UTIMES = "1" HAVE_UTIME_H = "1" HAVE_UUID_CREATE = "0" HAVE_UUID_ENC_BE = "0" HAVE_UUID_GENERATE_TIME_SAFE = "1" HAVE_UUID_H = "0" HAVE_UUID_UUID_H = "1" HAVE_WAIT3 = "1" HAVE_WAIT4 = "1" HAVE_WAITID = "1" HAVE_WAITPID = "1" HAVE_WCHAR_H = "1" HAVE_WCSCOLL = "1" HAVE_WCSFTIME = "1" HAVE_WCSXFRM = "1" HAVE_WMEMCMP = "1" HAVE_WORKING_TZSET = "1" HAVE_WRITEV = "1" HAVE_X509_VERIFY_PARAM_SET1_HOST = "1" HAVE_ZLIB_COPY = "1" HAVE__GETPTY = "0" HOST_GNU_TYPE = "x86_64pclinuxgnu" INCLDIRSTOMAKE = "/usr/include /usr/include /usr/include/python3.8 /usr/include/python3.8" INCLUDEDIR = "/usr/include" INCLUDEPY = "/usr/include/python3.8" INSTALL = "/usr/bin/install c" INSTALL_DATA = "/usr/bin/install c m 644" INSTALL_PROGRAM = "/usr/bin/install c" INSTALL_SCRIPT = "/usr/bin/install c" INSTALL_SHARED = "/usr/bin/install c m 755" INSTSONAME = "libpython3.8.so.1.0" IO_H = "Modules/_io/_iomodule.h" IO_OBJS = "\" LDCXXSHARED = "x86_64linuxgnug++ pthread shared Wl,O1 Wl,Bsymbolicfunctions" LDFLAGS = "Wl,Bsymbolicfunctions Wl,z,relro g fwrapv O2 " LDFLAGS_NODIST = "" LDLIBRARY = "libpython3.8.so" LDLIBRARYDIR = "" LDSHARED = "x86_64linuxgnugcc pthread shared Wl,O1 Wl,Bsymbolicfunctions Wl,Bsymbolicfunctions Wl,z,relro g fwrapv O2 " LDVERSION = "3.8" LIBC = "" LIBDEST = "/usr/lib/python3.8" LIBDIR = "/usr/lib" LIBFFI_INCLUDEDIR = "" LIBM = "lm" LIBOBJDIR = "Python/" LIBOBJS = "" LIBPC = "/usr/lib/x86_64linuxgnu/pkgconfig" LIBPL = "/usr/lib/python3.8/config3.8x86_64linuxgnu" LIBPYTHON = "" LIBRARY = "libpython3.8.a" LIBRARY_OBJS = "\" LIBRARY_OBJS_OMIT_FROZEN = "\" LIBS = "lcrypt lpthread ldl lutil lm" LIBSUBDIRS = "tkinter tkinter/test tkinter/test/test_tkinter \" LINKCC = "x86_64linuxgnugcc pthread" LINKFORSHARED = "Xlinker exportdynamic Wl,O1 Wl,Bsymbolicfunctions" LIPO_32BIT_FLAGS = "" LLVM_PROF_ERR = "no" LLVM_PROF_FILE = "" LLVM_PROF_MERGER = "true" LN = "ln" LOCALMODLIBS = "lexpat L/usr/lib lz lexpat" MACHDEP = "linux" MACHDEP_OBJS = "" MACHDESTLIB = "/usr/lib/python3.8" MACOSX_DEPLOYMENT_TARGET = "" MAINCC = "x86_64linuxgnugcc pthread" MAJOR_IN_MKDEV = "0" MAJOR_IN_SYSMACROS = "1" MAKESETUP = "../Modules/makesetup" MANDIR = "/usr/share/man" MKDIR_P = "/bin/mkdir p" MODBUILT_NAMES = "array cmath math _struct _random _elementtree _pickle _datetime _bisect _heapq _statistics unicodedata fcntl spwd grp select _csv _socket _posixsubprocess _md5 _sha1 _sha256 _sha512 _sha3 _blake2 syslog binascii zlib posix errno pwd _sre _codecs _weakref _functools _operator _collections _abc itertools atexit _signal _stat time _thread _locale _io faulthandler _tracemalloc _symtable pyexpat xxsubtype" MODDISABLED_NAMES = "" MODLIBS = "lexpat L/usr/lib lz lexpat" MODOBJS = "$(sort Modules/arraymodule.o Modules/cmathmodule.o Modules/_math.o Modules/mathmodule.o Modules/_math.o Modules/_struct.o Modules/_randommodule.o Modules/_elementtree.o Modules/_pickle.o Modules/_datetimemodule.o Modules/_bisectmodule.o Modules/_heapqmodule.o Modules/_statisticsmodule.o Modules/unicodedata.o Modules/fcntlmodule.o Modules/spwdmodule.o Modules/grpmodule.o Modules/selectmodule.o Modules/_csv.o Modules/socketmodule.o Modules/_posixsubprocess.o Modules/md5module.o Modules/sha1module.o Modules/sha256module.o Modules/sha512module.o Modules/sha3module.o Modules/blake2module.o Modules/blake2b_impl.o Modules/blake2s_impl.o Modules/syslogmodule.o Modules/binascii.o Modules/zlibmodule.o Modules/posixmodule.o Modules/errnomodule.o Modules/pwdmodule.o Modules/_sre.o Modules/_codecsmodule.o Modules/_weakref.o Modules/_functoolsmodule.o Modules/_operator.o Modules/_collectionsmodule.o Modules/_abc.o Modules/itertoolsmodule.o Modules/atexitmodule.o Modules/signalmodule.o Modules/_stat.o Modules/timemodule.o Modules/_threadmodule.o Modules/_localemodule.o Modules/_iomodule.o Modules/iobase.o Modules/fileio.o Modules/bytesio.o Modules/bufferedio.o Modules/textio.o Modules/stringio.o Modules/faulthandler.o Modules/_tracemalloc.o Modules/hashtable.o Modules/symtablemodule.o Modules/pyexpat.o Modules/xxsubtype.o)" MODULE_OBJS = "\" MULTIARCH = "x86_64linuxgnu" MULTIARCH_CPPFLAGS = "DMULTIARCH=\"x86_64linuxgnu\"" MVWDELCH_IS_EXPRESSION = "1" NO_AS_NEEDED = "Wl,noasneeded" OBJECT_OBJS = "\" OPENSSL_INCLUDES = "" OPENSSL_LDFLAGS = "" OPENSSL_LIBS = "lssl lcrypto" OPT = "DNDEBUG g fwrapv O2 Wall" OTHER_LIBTOOL_OPT = "" PACKAGE_BUGREPORT = "0" PACKAGE_NAME = "0" PACKAGE_STRING = "0" PACKAGE_TARNAME = "0" PACKAGE_URL = "0" PACKAGE_VERSION = "0" PARSER_HEADERS = "\" PARSER_OBJS = "\ Parser/myreadline.o Parser/parsetok.o Parser/tokenizer.o" PGO_PROF_GEN_FLAG = "fprofilegenerate" PGO_PROF_USE_FLAG = "" POBJS = "\" POSIX_SEMAPHORES_NOT_ENABLED = "0" PROFILE_TASK = "m test pgo" PTHREAD_KEY_T_IS_COMPATIBLE_WITH_INT = "1" PTHREAD_SYSTEM_SCHED_SUPPORTED = "1" PURIFY = "" PY3LIBRARY = "libpython3.so" PYLONG_BITS_IN_DIGIT = "0" PYTHON = "python" PYTHONFRAMEWORK = "" PYTHONFRAMEWORKDIR = "noframework" PYTHONFRAMEWORKINSTALLDIR = "" PYTHONFRAMEWORKPREFIX = "" PYTHONPATH = "" PYTHON_FOR_BUILD = "./python E" PYTHON_FOR_REGEN = "python3.8" PYTHON_HEADERS = "\" PYTHON_OBJS = "\" PY_BUILTIN_MODULE_CFLAGS = "Wnounusedresult Wsigncompare DNDEBUG g fwrapv O2 Wall g fstackprotectorstrong Wformat Werror=formatsecurity g fwrapv O2 std=c99 Wextra Wnounusedresult Wnounusedparameter Wnomissingfieldinitializers Werror=implicitfunctiondeclaration I../Include/internal IObjects IInclude IPython I. I../Include Wdatetime D_FORTIFY_SOURCE=2 fPIC DPy_BUILD_CORE_BUILTIN" PY_CFLAGS = "Wnounusedresult Wsigncompare DNDEBUG g fwrapv O2 Wall g fstackprotectorstrong Wformat Werror=formatsecurity g fwrapv O2 " PY_CFLAGS_NODIST = "std=c99 Wextra Wnounusedresult Wnounusedparameter Wnomissingfieldinitializers Werror=implicitfunctiondeclaration I../Include/internal" PY_COERCE_C_LOCALE = "1" PY_CORE_CFLAGS = "Wnounusedresult Wsigncompare DNDEBUG g fwrapv O2 Wall g fstackprotectorstrong Wformat Werror=formatsecurity g fwrapv O2 std=c99 Wextra Wnounusedresult Wnounusedparameter Wnomissingfieldinitializers Werror=implicitfunctiondeclaration I../Include/internal IObjects IInclude IPython I. I../Include Wdatetime D_FORTIFY_SOURCE=2 fPIC DPy_BUILD_CORE" PY_CORE_LDFLAGS = "Wl,Bsymbolicfunctions Wl,z,relro g fwrapv O2 " PY_CPPFLAGS = "IObjects IInclude IPython I. I../Include Wdatetime D_FORTIFY_SOURCE=2" PY_FORMAT_SIZE_T = ""z"" PY_LDFLAGS = "Wl,Bsymbolicfunctions Wl,z,relro g fwrapv O2 " PY_LDFLAGS_NODIST = "" PY_SSL_DEFAULT_CIPHERS = "1" PY_SSL_DEFAULT_CIPHER_STRING = "0" PY_STDMODULE_CFLAGS = "Wnounusedresult Wsigncompare DNDEBUG g fwrapv O2 Wall g fstackprotectorstrong Wformat Werror=formatsecurity g fwrapv O2 std=c99 Wextra Wnounusedresult Wnounusedparameter Wnomissingfieldinitializers Werror=implicitfunctiondeclaration I../Include/internal IObjects IInclude IPython I. I../Include Wdatetime D_FORTIFY_SOURCE=2 fPIC" Py_DEBUG = "0" Py_ENABLE_SHARED = "1" Py_HASH_ALGORITHM = "0" Py_TRACE_REFS = "0" QUICKTESTOPTS = "x test_subprocess test_io test_lib2to3 \" READELF = "readelf" RESSRCDIR = "Mac/Resources/framework" RETSIGTYPE = "void" RUNSHARED = "LD_LIBRARY_PATH=/build/python3.86QL2k7/python3.83.8.2/buildshared" SCRIPTDIR = "/usr/lib" SETPGRP_HAVE_ARG = "0" SGI_ABI = "@SGI_ABI@" SHELL = "/bin/sh" SHLIBS = "lcrypt lpthread ldl lutil lm" SHLIB_SUFFIX = ".so" SHM_NEEDS_LIBRT = "1" SIGNED_RIGHT_SHIFT_ZERO_FILLS = "0" SITEPATH = "" SIZEOF_DOUBLE = "8" SIZEOF_FLOAT = "4" SIZEOF_FPOS_T = "16" SIZEOF_INT = "4" SIZEOF_LONG = "8" SIZEOF_LONG_DOUBLE = "16" SIZEOF_LONG_LONG = "8" SIZEOF_OFF_T = "8" SIZEOF_PID_T = "4" SIZEOF_PTHREAD_KEY_T = "4" SIZEOF_PTHREAD_T = "8" SIZEOF_SHORT = "2" SIZEOF_SIZE_T = "8" SIZEOF_TIME_T = "8" SIZEOF_UINTPTR_T = "8" SIZEOF_VOID_P = "8" SIZEOF_WCHAR_T = "4" SIZEOF__BOOL = "1" SO = ".cpython38x86_64linuxgnu.so" SOABI = "cpython38x86_64linuxgnu" SRCDIRS = "Parser Objects Python Modules Modules/_io Programs" SRC_GDB_HOOKS = "../Tools/gdb/libpython.py" STDC_HEADERS = "1" STRICT_SYSV_CURSES = "/* Don't use ncurses extensions */" STRIPFLAG = "s" SUBDIRS = "" SUBDIRSTOO = "Include Lib Misc" SYSLIBS = "lm" SYS_SELECT_WITH_SYS_TIME = "1" TCLTK_INCLUDES = "" TCLTK_LIBS = "" TESTOPTS = "" TESTPATH = "" TESTPYTHON = "LD_LIBRARY_PATH=/build/python3.86QL2k7/python3.83.8.2/buildshared ./python" TESTPYTHONOPTS = "" TESTRUNNER = "LD_LIBRARY_PATH=/build/python3.86QL2k7/python3.83.8.2/buildshared ./python ../Tools/scripts/run_tests.py" TESTTIMEOUT = "1200" TIMEMODULE_LIB = "0" TIME_WITH_SYS_TIME = "1" TM_IN_SYS_TIME = "0" UNICODE_DEPS = "\" UNIVERSALSDK = "" UPDATE_FILE = "python3.8 ../Tools/scripts/update_file.py" USE_COMPUTED_GOTOS = "1" VERSION = "3.8" VPATH = ".." WINDOW_HAS_FLAGS = "1" WITH_DOC_STRINGS = "1" WITH_DTRACE = "1" WITH_DYLD = "0" WITH_LIBINTL = "0" WITH_NEXT_FRAMEWORK = "0" WITH_PYMALLOC = "1" WITH_VALGRIND = "0" X87_DOUBLE_ROUNDING = "0" XMLLIBSUBDIRS = "xml xml/dom xml/etree xml/parsers xml/sax" abiflags = "" abs_builddir = "/build/python3.86QL2k7/python3.83.8.2/buildshared" abs_srcdir = "/build/python3.86QL2k7/python3.83.8.2/buildshared/.." base = "/mnt/d/Programming/Projects/sage/src/.venv" datarootdir = "/usr/share" exec_prefix = "/usr" installed_base = "/usr" installed_platbase = "/usr" multiarchsubdir = "/x86_64linuxgnu" platbase = "/mnt/d/Programming/Projects/sage/src/.venv" prefix = "/usr" projectbase = "/usr/bin" py_version = "3.8.2" py_version_nodot = "38" py_version_short = "3.8" srcdir = "/usr/lib/python3.8/config3.8x86_64linuxgnu" userbase = "/home/tobias/.local"
comment:143 Changed 2 years ago by
I see MULTIARCH = "x86_64linuxgnu"
and multiarchsubdir = "/x86_64linuxgnu"
.
comment:144 Changed 2 years ago by
One of these variables should be used in _get_shared_lib_filename
. Why it is instead looking for MULTILIB
, which indeed does not exist, should be investigated...
comment:147 Changed 2 years ago by
Dependencies:  #30779, #30672, #30673, #30709, #30706 → #30779, #30672, #30673, #30709, #30706, #30833 

comment:148 Changed 2 years ago by
Commit:  7523baa68b46732cc381387e58ea1a9559c11e96 → 443424cf878ac7fa9c57a1bad313f1b4d84ed031 

comment:149 Changed 2 years ago by
Dependencies:  #30779, #30672, #30673, #30709, #30706, #30833 → #30779, #30672, #30673, #30709, #30706, #30833, #30748 

comment:150 Changed 2 years ago by
Commit:  443424cf878ac7fa9c57a1bad313f1b4d84ed031 → b3045684c751317f252ff61f59ab53782df76177 

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
2351ae3  Update, extend and reorder dependencies

99ae00c  Replace startup exception by warning

11882e5  Use context manager

d858e46  Merge branch 'public/build/startupWarning' of git://trac.sagemath.org/sage into public/build/inplace

46ac365  Update numpy

3fcd71f  Merge tag '9.3.beta0' into t/30709/public/build/cythonConfig

a646a24  src/sage_setup/cython_options.py: Restore auto_pickle=False

8eae4bc  Merge branch 'public/build/cythonConfig' of git://trac.sagemath.org/sage into public/build/inplace

5ac3a29  Add matlibplot library

b304568  Run doctests in github action

comment:151 Changed 2 years ago by
Commit:  b3045684c751317f252ff61f59ab53782df76177 → 9a94a29bc1c6179282dbd94a7afd19247884e525 

Branch pushed to git repo; I updated commit sha1. New commits:
9a94a29  Don't install texlivegenericextra

comment:152 Changed 2 years ago by
Commit:  9a94a29bc1c6179282dbd94a7afd19247884e525 → 16674ba8da9daa53709c9ad4bfc6fca6bdba13ee 

Branch pushed to git repo; I updated commit sha1. New commits:
16674ba  Remove lazy import finish startup

comment:153 Changed 2 years ago by
Commit:  16674ba8da9daa53709c9ad4bfc6fca6bdba13ee → 511775ec71b3d2efef8f6e9ca2ea95f234c566ad 

comment:154 Changed 2 years ago by
I've now added the doctest run to the github workflow. It works in principle although there are quite a lot of failing tests (mainly because some of the systeminstalled libs are not discovered). In order to keep the focus of this ticket narrow, fixing this should better be done in followup tickets.
comment:155 Changed 2 years ago by
Description:  modified (diff) 

comment:156 followup: 200 Changed 2 years ago by
comment:157 Changed 2 years ago by
Commit:  511775ec71b3d2efef8f6e9ca2ea95f234c566ad → 373d1c12e77647541fa7fc8b315aae98ee0298cd 

Branch pushed to git repo; I updated commit sha1. New commits:
373d1c1  Add rope as dev dependency

comment:158 Changed 2 years ago by
Replying to mkoeppe:
 A few libraries are not contained in
distros/debian.txt
files. Should they be added? In particular, this concerns libsingular4dev, liblinboxdev, libratpointsdev, libgapdev, ecl, libpynacdev.
Yes, let's add them in #30859.
Some of the information that you put in the ticket description above could go as comments into the debian.txt files.
comment:159 Changed 2 years ago by
Thanks for opening a new ticket for this!
Concerning the conda integration, this ticket here might actually help with this. You can use pipenv on top of conda, see https://pipenv.pypa.io/en/latest/advanced/#pipenvandotherpythondistributions.
comment:160 Changed 2 years ago by
Description:  modified (diff) 

comment:161 Changed 2 years ago by
Commit:  373d1c12e77647541fa7fc8b315aae98ee0298cd → 096df15a1253e6cb0fa75c92e29458f29d5ad77d 

Branch pushed to git repo; I updated commit sha1. New commits:
096df15  Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace

comment:162 Changed 2 years ago by
Commit:  096df15a1253e6cb0fa75c92e29458f29d5ad77d → 8eb2f8ec11943c8f7175ce4d8a293673ab18b774 

Branch pushed to git repo; I updated commit sha1. New commits:
8eb2f8e  Remove homfly workaround

comment:163 Changed 2 years ago by
Commit:  8eb2f8ec11943c8f7175ce4d8a293673ab18b774 → 17f367eebf0e329275773e0c048f3009b2de19ed 

Branch pushed to git repo; I updated commit sha1. New commits:
17f367e  Refactor shared library discovery

comment:164 Changed 2 years ago by
Dependencies:  #30779, #30672, #30673, #30709, #30706, #30833, #30748 → #30779, #30672, #30673, #30709, #30706, #30901, #30748 

comment:165 Changed 2 years ago by
Commit:  17f367eebf0e329275773e0c048f3009b2de19ed → 680f85d72cf88ae96e0bc4bf7b422e694f353437 

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
c47c4bf  Correct indent

23b6336  Merge branch 'public/build/multiarchsimple' of git://trac.sagemath.org/sage into public/build/inplace

6a52fbf  Remove lazy import finish startup

f96025e  Merge branch 'develop' of git://github.com/sagemath/sage into public/build/startupWarning

b952bb5  Fix doctests

c6857dd  Hopefully fix doctest

23da14b  Wrap each sage.all import in startup guard

2f4055e  Specify that exit was succesful

d4d94a8  Indeed replace exception by print statement

680f85d  Merge branch 'public/build/startupWarning' of git://trac.sagemath.org/sage into public/build/inplace

comment:166 Changed 2 years ago by
Commit:  680f85d72cf88ae96e0bc4bf7b422e694f353437 → ddcf2e3ba09cb3731042c1372b93e4db84f93b13 

Branch pushed to git repo; I updated commit sha1. New commits:
ddcf2e3  Cleanup code

comment:167 Changed 2 years ago by
Commit:  ddcf2e3ba09cb3731042c1372b93e4db84f93b13 → 772baf208259ff75edebdbf6f340c700be0681ab 

comment:168 Changed 2 years ago by
I've now merged the latest version of the dependencies, and cleaned the code.
Matthias (and others, of course), it would be nice if you could review the code (and its dependencies). I have a hard time working on other tickets as long as this ticket is not merged. So I would be grateful if that happens as soon as possible. Thanks!
comment:169 followup: 171 Changed 2 years ago by
See if you can get doctesting to work on this branch  by merging #30578.
comment:170 followup: 172 Changed 2 years ago by
Also removing sageenvconfig
from the list of installed scripts is not good. Instead please review and merge #29850  which reassigns this script to the sage_conf
package.
comment:171 followup: 174 Changed 2 years ago by
comment:172 Changed 2 years ago by
Replying to mkoeppe:
Also removing
sageenvconfig
from the list of installed scripts is not good. Instead please review and merge #29850  which reassigns this script to thesage_conf
package.
I agree #29850 is a better solution. I had a look at that ticket, but don't really feel comfortable enough with the code to review it. Sorry! However, I've now added it as a dependency of this branch, and will merge as soon as it is reviewed.
comment:173 Changed 2 years ago by
Dependencies:  #30779, #30672, #30673, #30709, #30706, #30901, #30748 → #30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850 

comment:174 followup: 177 Changed 2 years ago by
Replying to mkoeppe:
Replying to mkoeppe:
See if you can get doctesting to work on this branch  by merging #30578.
In #30578 I had to add a few more packages to
requirements.txt
that you don't have inPipfile
yet, so that more doctests pass.
Thanks for the pointer! I've now added these packages also to the pipfile. However, it is way beyond the scope of this ticket to make the doctest pass in the GH pipenv workflow. There are just so many things that need to be changed in order for the doctests to find the correct dependencies. For example, here are a few relevant error messages:
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/d/Programming/Projects/sage/src/.venv/bin/gap' RuntimeError: unable to start singular because the command 'Singular t tickspersec 1000 cntrlc=a' failed: The command was not found or was not executable: Singular. RuntimeError: In order to initialize the database, /__w/sage/sage/src/.venv/share/conway_polynomials/conway_polynomials.p must exist.
I know that some of these errors would disappear if one use your recent progress towards installation of sagebuild wheels.
Anyway, the aim of this ticket is to create a mostlyworking editable install of sagelib. Making the doctests pass relative to this editable install in a venv should be subject of a followup ticket.
comment:175 Changed 2 years ago by
Commit:  772baf208259ff75edebdbf6f340c700be0681ab → ad0f33cd4c67d2eb31cbf864e8112caeb8958990 

comment:176 Changed 2 years ago by
Commit:  ad0f33cd4c67d2eb31cbf864e8112caeb8958990 → 77e1af16bcfce183bfe072cfb7324e82ee95a478 

comment:177 Changed 2 years ago by
Replying to ghtobiasdiez:
Anyway, the aim of this ticket is to create a mostlyworking editable install of sagelib.
Most of the changes that you are doing in this ticket have really nothing to do with this goal. As I have pointed out previously, you are trying to solve the orthogonal problem of making sagelib work outside of the Sage distribution, i.e., without SAGE_LOCAL.
comment:178 followup: 179 Changed 2 years ago by
I agree. The thing is that it is currently my only way to work with sage, and thus test the editable install. But it's good that you are pushing from the other side and work on the editable install based on a working sage distribution.
So, is there something that I can still do here? I see that verifying and testing that the editable install works is important. It's just not very realistic to have a fully functional doctest without SAGE_LOCAL at the moment. However, I can also not very actively contribute to the editable install with SAGE_LOCAL.
comment:179 followup: 181 Changed 2 years ago by
Replying to ghtobiasdiez:
I agree. The thing is that it is currently my only way to work with sage, and thus test the editable install.
Is the WSL build of the Sage distribution still failing for you? Have you tried a different Linux distribution? So far the WSL runs on GH Actions have not been very conclusive  they have failed for unrelated reasons  you could work on #30910 to improve it
comment:180 followup: 182 Changed 2 years ago by
Also, I am interested in a GH Actions workflow for the various new installations (venv based on requirements on top of SAGE_LOCAL, venv based on Pipfile on top of SAGE_LOCAL, conda without SAGE_LOCAL, etc.) This ticket adds something like this, but I don't like that it does Ubuntuspecific things again. We have a better way of doing such things already so that everything can be tested on all supported distributions, not just some Ubuntu. I would hope that we can add such workflows in #30845. #30859 will help with this. The powershell bits to work with system packages could also be valuable  see #30865.
comment:181 Changed 2 years ago by
Replying to mkoeppe:
Is the WSL build of the Sage distribution still failing for you? Have you tried a different Linux distribution?
Yes, it's still not working. In a few weeks I'll get a new PC and then will try again afresh using WSL 2.0.
comment:182 Changed 2 years ago by
Replying to mkoeppe:
This ticket adds something like this, but I don't like that it does Ubuntuspecific things again. We have a better way of doing such things already so that everything can be tested on all supported distributions, not just some Ubuntu.
Yes and no. Since the whole workflow only takes about 2 hours, my hope was that one can use it in the longterm to run it on each push to each branch, replacing/complementing the patchbot. But of course it could also serve as the blueprint for a complete matrix integration test suite. Anyway, for the moment it serves as a record of the things I did locally on my system, and proofs that the editable install without SAGE_LOCAL works to a large degree.
comment:183 Changed 2 years ago by
Dependencies:  #30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850 → #30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850, #29314 

After playing around with it further I was able to make most of the tests pass. There are still a few hundreds that fail (see below for details), most of which are related to issues with maxima (TypeError?: unable to start maxima: End Of File (EOF). Exception style platform.) or fpylll (blocked due to https://github.com/fplll/fpylll/issues/185). Since most test pass and there are many subpackages for which all tests pass, I'd say that's enough "proof of concept" for this ticket and would leave the rest for followup tickets.
20201117T19:31:00.7800025Z sage t randomseed=0 /__w/sage/sage/src/sage/interacts/debugger.py # 2 doctests failed 20201117T19:31:00.7801032Z sage t randomseed=0 /__w/sage/sage/src/sage/interfaces/singular.py # 5 doctests failed 20201117T19:31:00.7802028Z sage t randomseed=0 /__w/sage/sage/src/sage/interfaces/maxima_lib.py # 1 doctest failed 20201117T19:31:00.7803005Z sage t randomseed=0 /__w/sage/sage/src/sage/interfaces/quit.py # 6 doctests failed 20201117T19:31:00.7803974Z sage t randomseed=0 /__w/sage/sage/src/sage/interfaces/maxima.py # 144 doctests failed 20201117T19:31:00.7804998Z sage t randomseed=0 /__w/sage/sage/src/sage/interfaces/maxima_abstract.py # 168 doctests failed 20201117T19:31:00.7806039Z sage t randomseed=0 /__w/sage/sage/src/sage/interfaces/interface.py # 90 doctests failed 20201117T19:31:00.7807034Z sage t randomseed=0 /__w/sage/sage/src/sage/interfaces/expect.py # 6 doctests failed 20201117T19:31:00.7808001Z sage t randomseed=0 /__w/sage/sage/src/sage/interfaces/tests.py # 3 doctests failed 20201117T19:31:00.7809010Z sage t randomseed=0 /__w/sage/sage/src/sage/interfaces/gap_workspace.py # 2 doctests failed 20201117T19:31:00.7810009Z sage t randomseed=0 /__w/sage/sage/src/sage/rings/rational.pyx # 2 doctests failed 20201117T19:31:00.7810921Z sage t randomseed=0 /__w/sage/sage/src/sage/rings/ring.pyx # 1 doctest failed 20201117T19:31:00.7811830Z sage t randomseed=0 /__w/sage/sage/src/sage/rings/infinity.py # 1 doctest failed 20201117T19:31:00.7812754Z sage t randomseed=0 /__w/sage/sage/src/sage/rings/real_mpfr.pyx # 9 doctests failed 20201117T19:31:00.7813669Z sage t randomseed=0 /__w/sage/sage/src/sage/rings/real_mpfi.pyx # 3 doctests failed 20201117T19:31:00.7814600Z sage t randomseed=0 /__w/sage/sage/src/sage/rings/complex_mpfr.pyx # 2 doctests failed 20201117T19:31:00.7815569Z sage t randomseed=0 /__w/sage/sage/src/sage/rings/localization.py # 1 doctest failed 20201117T19:31:00.7816785Z sage t randomseed=0 /__w/sage/sage/src/sage/rings/tests.py # 2 doctests failed 20201117T19:31:00.7817714Z sage t randomseed=0 /__w/sage/sage/src/sage/rings/integer_ring.pyx # 5 doctests failed 20201117T19:31:00.7818771Z sage t randomseed=0 /__w/sage/sage/src/sage/rings/polynomial/polynomial_zmod_flint.pyx # 1 doctest failed 20201117T19:31:00.7819934Z sage t randomseed=0 /__w/sage/sage/src/sage/rings/polynomial/polynomial_quotient_ring.py # 1 doctest failed 20201117T19:31:00.7821093Z sage t randomseed=0 /__w/sage/sage/src/sage/rings/polynomial/polynomial_modn_dense_ntl.pyx # 6 doctests failed 20201117T19:31:00.7822245Z sage t randomseed=0 /__w/sage/sage/src/sage/rings/polynomial/polynomial_element.pyx # 13 doctests failed 20201117T19:31:00.7823414Z sage t randomseed=0 /__w/sage/sage/src/sage/rings/polynomial/polynomial_number_field.pyx # 19 doctests failed 20201117T19:31:00.7824560Z sage t randomseed=0 /__w/sage/sage/src/sage/rings/polynomial/multi_polynomial_ring.py # 2 doctests failed 20201117T19:31:00.7825670Z sage t randomseed=0 /__w/sage/sage/src/sage/rings/finite_rings/residue_field.pyx # 14 doctests failed 20201117T19:31:00.7826727Z sage t randomseed=0 /__w/sage/sage/src/sage/rings/number_field/S_unit_solver.py # 18 doctests failed 20201117T19:31:00.7827787Z sage t randomseed=0 /__w/sage/sage/src/sage/rings/number_field/number_field_base.pyx # 15 doctests failed 20201117T19:31:00.7828969Z sage t randomseed=0 /__w/sage/sage/src/sage/rings/number_field/class_group.py # 2 doctests failed 20201117T19:31:00.7830020Z sage t randomseed=0 /__w/sage/sage/src/sage/rings/number_field/number_field_ideal.py # 1 doctest failed 20201117T19:31:00.7831068Z sage t randomseed=0 /__w/sage/sage/src/sage/rings/number_field/galois_group.py # 58 doctests failed 20201117T19:31:00.7832134Z sage t randomseed=0 /__w/sage/sage/src/sage/rings/number_field/splitting_field.py # 28 doctests failed 20201117T19:31:00.7833172Z sage t randomseed=0 /__w/sage/sage/src/sage/rings/number_field/order.py # 6 doctests failed 20201117T19:31:00.7834186Z sage t randomseed=0 /__w/sage/sage/src/sage/rings/number_field/number_field.py # 33 doctests failed 20201117T19:31:00.7835200Z sage t randomseed=0 /__w/sage/sage/src/sage/rings/number_field/unit_group.py # 1 doctest failed 20201117T19:31:00.7836234Z sage t randomseed=0 /__w/sage/sage/src/sage/rings/number_field/number_field_rel.py # 11 doctests failed 20201117T19:31:00.7837488Z sage t randomseed=0 /__w/sage/sage/src/sage/rings/number_field/number_field_element.pyx # 27 doctests failed 20201117T19:31:00.7838576Z sage t randomseed=0 /__w/sage/sage/src/sage/rings/number_field/bdd_height.py # 2 doctests failed 20201117T19:31:00.7839663Z sage t randomseed=0 /__w/sage/sage/src/sage/rings/valuation/inductive_valuation.py # 4 doctests failed 20201117T19:31:00.7840725Z sage t randomseed=0 /__w/sage/sage/src/sage/cpython/dict_del_by_value.pyx # 2 doctests failed 20201117T19:31:00.7841710Z sage t randomseed=0 /__w/sage/sage/src/sage/databases/cremona.py # 2 doctests failed 20201117T19:31:00.7842673Z sage t randomseed=0 /__w/sage/sage/src/sage/groups/cubic_braid.py # 6 doctests failed 20201117T19:31:00.7843635Z sage t randomseed=0 /__w/sage/sage/src/sage/sandpiles/sandpile.py # 5 doctests failed 20201117T19:31:00.7844616Z sage t randomseed=0 /__w/sage/sage/src/sage/coding/gabidulin_code.py # 1 doctest failed 20201117T19:31:00.7845566Z sage t randomseed=0 /__w/sage/sage/src/sage/tests/cmdline.py # 59 doctests failed 20201117T19:31:00.7846502Z sage t randomseed=0 /__w/sage/sage/src/sage/tests/benchmark.py # 4 doctests failed 20201117T19:31:00.7847448Z sage t randomseed=0 /__w/sage/sage/src/sage/tests/gap_packages.py # 1 doctest failed 20201117T19:31:00.7848644Z sage t randomseed=0 /__w/sage/sage/src/sage/tests/books/judsonabstractalgebra/galoissage.py # 9 doctests failed 20201117T19:31:00.7850157Z sage t randomseed=0 /__w/sage/sage/src/sage/tests/books/judsonabstractalgebra/fieldssage.py # 17 doctests failed 20201117T19:31:00.7851577Z sage t randomseed=0 /__w/sage/sage/src/sage/tests/books/judsonabstractalgebra/domainssage.py # 5 doctests failed 20201117T19:31:00.7853321Z sage t randomseed=0 /__w/sage/sage/src/sage/tests/books/computationalmathematicswithsagemath/polynomes_doctest.py # 1 doctest failed 20201117T19:31:00.7855341Z sage t randomseed=0 /__w/sage/sage/src/sage/tests/books/computationalmathematicswithsagemath/domaines_doctest.py # 3 doctests failed 20201117T19:31:00.7857340Z sage t randomseed=0 /__w/sage/sage/src/sage/tests/books/computationalmathematicswithsagemath/combinat_doctest.py # 1 doctest failed 20201117T19:31:00.7859360Z sage t randomseed=0 /__w/sage/sage/src/sage/tests/books/computationalmathematicswithsagemath/integration_doctest.py # 1 doctest failed 20201117T19:31:00.7861360Z sage t randomseed=0 /__w/sage/sage/src/sage/tests/books/computationalmathematicswithsagemath/mpoly_doctest.py # 2 doctests failed 20201117T19:31:00.7863355Z sage t randomseed=0 /__w/sage/sage/src/sage/tests/books/computationalmathematicswithsagemath/calculus_doctest.py # 3 doctests failed 20201117T19:31:00.7865352Z sage t randomseed=0 /__w/sage/sage/src/sage/tests/books/computationalmathematicswithsagemath/sol/mpoly_doctest.py # 2 doctests failed 20201117T19:31:00.7867492Z sage t randomseed=0 /__w/sage/sage/src/sage/tests/books/computationalmathematicswithsagemath/sol/graphique_doctest.py # 8 doctests failed 20201117T19:31:00.7868993Z sage t randomseed=0 /__w/sage/sage/src/sage/repl/configuration.py # 1 doctest failed 20201117T19:31:00.7869942Z sage t randomseed=0 /__w/sage/sage/src/sage/repl/image.py # 4 doctests failed 20201117T19:31:00.7870875Z sage t randomseed=0 /__w/sage/sage/src/sage/repl/interpreter.py # 7 doctests failed 20201117T19:31:00.7871818Z sage t randomseed=0 /__w/sage/sage/src/sage/repl/preparse.py # 1 doctest failed 20201117T19:31:00.7872750Z sage t randomseed=0 /__w/sage/sage/src/sage/repl/ipython_tests.py # 1 doctest failed 20201117T19:31:00.7873726Z sage t randomseed=0 /__w/sage/sage/src/sage/repl/rich_output/pretty_print.py # 3 doctests failed 20201117T19:31:00.7874739Z sage t randomseed=0 /__w/sage/sage/src/sage/repl/ipython_kernel/install.py # 1 doctest failed 20201117T19:31:00.7875703Z sage t randomseed=0 /__w/sage/sage/src/sage/doctest/forker.py # 1 doctest failed 20201117T19:31:00.7876633Z sage t randomseed=0 /__w/sage/sage/src/sage/doctest/control.py # 2 doctests failed 20201117T19:31:00.7883133Z sage t randomseed=0 /__w/sage/sage/src/sage/doctest/test.py # 3 doctests failed 20201117T19:31:00.7884402Z sage t randomseed=0 /__w/sage/sage/src/sage/libs/singular/function.pyx # 1 doctest failed 20201117T19:31:00.7885406Z sage t randomseed=0 /__w/sage/sage/src/sage/libs/pari/convert_sage.pyx # 10 doctests failed 20201117T19:31:00.7886362Z sage t randomseed=0 /__w/sage/sage/src/sage/libs/pari/__init__.py # 12 doctests failed 20201117T19:31:00.7887265Z sage t randomseed=0 /__w/sage/sage/src/sage/libs/pari/tests.py # 9 doctests failed 20201117T19:31:00.7888179Z sage t randomseed=0 /__w/sage/sage/src/sage/libs/pynac/pynac.pyx # 1 doctest failed 20201117T19:31:00.7889240Z sage t randomseed=0 /__w/sage/sage/src/sage/dynamics/arithmetic_dynamics/generic_ds.py # 10 doctests failed 20201117T19:31:00.7890445Z sage t randomseed=0 /__w/sage/sage/src/sage/dynamics/arithmetic_dynamics/projective_ds.py # 25 doctests failed 20201117T19:31:00.7891469Z sage t randomseed=0 /__w/sage/sage/src/sage/arith/misc.py # 20 doctests failed 20201117T19:31:00.7892598Z sage t randomseed=0 /__w/sage/sage/src/sage/schemes/hyperelliptic_curves/hyperelliptic_generic.py # 3 doctests failed 20201117T19:31:00.7893789Z sage t randomseed=0 /__w/sage/sage/src/sage/schemes/product_projective/space.py # 1 doctest failed 20201117T19:31:00.7895096Z sage t randomseed=0 /__w/sage/sage/src/sage/schemes/product_projective/rational_point.py # 1 doctest failed 20201117T19:31:00.7896170Z sage t randomseed=0 /__w/sage/sage/src/sage/schemes/affine/affine_space.py # 1 doctest failed 20201117T19:31:00.7897209Z sage t randomseed=0 /__w/sage/sage/src/sage/schemes/generic/algebraic_scheme.py # 1 doctest failed 20201117T19:31:00.7898340Z sage t randomseed=0 /__w/sage/sage/src/sage/schemes/riemann_surfaces/riemann_surface.py # 31 doctests failed 20201117T19:31:00.7899522Z sage t randomseed=0 /__w/sage/sage/src/sage/schemes/projective/projective_rational_point.py # 4 doctests failed 20201117T19:31:00.7900680Z sage t randomseed=0 /__w/sage/sage/src/sage/schemes/projective/projective_space.py # 1 doctest failed 20201117T19:31:00.7901771Z sage t randomseed=0 /__w/sage/sage/src/sage/schemes/toric/toric_subscheme.py # 11 doctests failed 20201117T19:31:00.7902793Z sage t randomseed=0 /__w/sage/sage/src/sage/schemes/toric/divisor.py # 2 doctests failed 20201117T19:31:00.7903768Z sage t randomseed=0 /__w/sage/sage/src/sage/schemes/toric/points.py # 1 doctest failed 20201117T19:31:00.7904739Z sage t randomseed=0 /__w/sage/sage/src/sage/schemes/toric/variety.py # 4 doctests failed 20201117T19:31:00.7905707Z sage t randomseed=0 /__w/sage/sage/src/sage/schemes/toric/ideal.py # 43 doctests failed 20201117T19:31:00.7906825Z sage t randomseed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/heegner.py # 33 doctests failed 20201117T19:31:00.7907884Z sage t randomseed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/kraus.py # 4 doctests failed 20201117T19:31:00.7908978Z sage t randomseed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/constructor.py # 4 doctests failed 20201117T19:31:00.7910128Z sage t randomseed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/gal_reps_number_field.py # 2 doctests failed 20201117T19:31:00.7911278Z sage t randomseed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/ell_rational_field.py # 14 doctests failed 20201117T19:31:00.7912397Z sage t randomseed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/isogeny_class.py # 2 doctests failed 20201117T19:31:00.7913509Z sage t randomseed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/ell_number_field.py # 26 doctests failed 20201117T19:31:00.7914601Z sage t randomseed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/gal_reps.py # 2 doctests failed 20201117T19:31:00.7915708Z sage t randomseed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/isogeny_small_degree.py # 1 doctest failed 20201117T19:31:00.7916803Z sage t randomseed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/ell_egros.py # 4 doctests failed 20201117T19:31:00.7918998Z sage t randomseed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/ell_curve_isogeny.py # 1 doctest failed 20201117T19:31:00.7920093Z sage t randomseed=0 /__w/sage/sage/src/sage/schemes/elliptic_curves/ell_field.py # 2 doctests failed 20201117T19:31:00.7921146Z sage t randomseed=0 /__w/sage/sage/src/sage/schemes/plane_conics/con_field.py # 7 doctests failed 20201117T19:31:00.7922203Z sage t randomseed=0 /__w/sage/sage/src/sage/schemes/plane_conics/con_number_field.py # 3 doctests failed 20201117T19:31:00.7923188Z sage t randomseed=0 /__w/sage/sage/src/sage/features/__init__.py # 1 doctest failed 20201117T19:31:00.7924137Z sage t randomseed=0 /__w/sage/sage/src/sage/calculus/desolvers.py # 8 doctests failed 20201117T19:31:00.7925128Z sage t randomseed=0 /__w/sage/sage/src/sage/calculus/calculus.py # 25 doctests failed 20201117T19:31:00.7926071Z sage t randomseed=0 /__w/sage/sage/src/sage/crypto/lattice.py # 1 doctest failed 20201117T19:31:00.7927011Z sage t randomseed=0 /__w/sage/sage/src/sage/combinat/tutorial.py # 1 doctest failed 20201117T19:31:00.7927975Z sage t randomseed=0 /__w/sage/sage/src/sage/combinat/combinat.py # 5 doctests failed 20201117T19:31:00.7929092Z sage t randomseed=0 /__w/sage/sage/src/sage/combinat/degree_sequences.pyx # 1 doctest failed 20201117T19:31:00.7930148Z sage t randomseed=0 /__w/sage/sage/src/sage/categories/algebra_functor.py # 5 doctests failed 20201117T19:31:00.7931175Z sage t randomseed=0 /__w/sage/sage/src/sage/categories/number_fields.py # 1 doctest failed 20201117T19:31:00.7932168Z sage t randomseed=0 /__w/sage/sage/src/sage/categories/rings.py # 12 doctests failed 20201117T19:31:00.7933122Z sage t randomseed=0 /__w/sage/sage/src/sage/categories/primer.py # 1 doctest failed 20201117T19:31:00.7934130Z sage t randomseed=0 /__w/sage/sage/src/sage/categories/quotient_fields.py # 7 doctests failed 20201117T19:31:00.7935102Z sage t randomseed=0 /__w/sage/sage/src/sage/modular/cusps_nf.py # 1 doctest failed 20201117T19:31:00.7936063Z sage t randomseed=0 /__w/sage/sage/src/sage/modular/etaproducts.py # 10 doctests failed 20201117T19:31:00.7937057Z sage t randomseed=0 /__w/sage/sage/src/sage/modular/modsym/p1list_nf.py # 1 doctest failed 20201117T19:31:00.7938011Z sage t randomseed=0 /__w/sage/sage/src/sage/structure/formal_sum.py # 1 doctest failed 20201117T19:31:00.7938985Z sage t randomseed=0 /__w/sage/sage/src/sage/structure/element.pyx # 5 doctests failed 20201117T19:31:00.7940062Z sage t randomseed=0 /__w/sage/sage/src/sage/structure/dynamic_class.py # 1 doctest failed 20201117T19:31:00.7941022Z sage t randomseed=0 /__w/sage/sage/src/sage/misc/nested_class.pyx # 1 doctest failed 20201117T19:31:00.7941915Z sage t randomseed=0 /__w/sage/sage/src/sage/misc/trace.py # 3 doctests failed 20201117T19:31:00.7942828Z sage t randomseed=0 /__w/sage/sage/src/sage/misc/sage_ostools.pyx # 1 doctest failed 20201117T19:31:00.7943764Z sage t randomseed=0 /__w/sage/sage/src/sage/misc/functional.py # 1 doctest failed 20201117T19:31:00.7944692Z sage t randomseed=0 /__w/sage/sage/src/sage/misc/edit_module.py # 1 doctest failed 20201117T19:31:00.7945625Z sage t randomseed=0 /__w/sage/sage/src/sage/misc/temporary_file.py # 1 doctest failed 20201117T19:31:00.7946578Z sage t randomseed=0 /__w/sage/sage/src/sage/misc/lazy_attribute.pyx # 1 doctest failed 20201117T19:31:00.7947551Z sage t randomseed=0 /__w/sage/sage/src/sage/misc/sageinspect.py # 13 doctests failed 20201117T19:31:00.7948497Z sage t randomseed=0 /__w/sage/sage/src/sage/misc/sagedoc.py # 3 doctests failed 20201117T19:31:00.7949390Z sage t randomseed=0 /__w/sage/sage/src/sage/misc/compat.py # 1 doctest failed 20201117T19:31:00.7950304Z sage t randomseed=0 /__w/sage/sage/src/sage/misc/sphinxify.py # 4 doctests failed 20201117T19:31:00.7951326Z sage t randomseed=0 /__w/sage/sage/src/sage/functions/hypergeometric.py # 7 doctests failed 20201117T19:31:00.7952357Z sage t randomseed=0 /__w/sage/sage/src/sage/functions/bessel.py # 7 doctests failed 20201117T19:31:00.7953305Z sage t randomseed=0 /__w/sage/sage/src/sage/functions/gamma.py # 2 doctests failed 20201117T19:31:00.7954306Z sage t randomseed=0 /__w/sage/sage/src/sage/functions/special.py # 1 doctest failed 20201117T19:31:00.7955238Z sage t randomseed=0 /__w/sage/sage/src/sage/functions/trig.py # 1 doctest failed 20201117T19:31:00.7956168Z sage t randomseed=0 /__w/sage/sage/src/sage/functions/error.py # 7 doctests failed 20201117T19:31:00.7957101Z sage t randomseed=0 /__w/sage/sage/src/sage/functions/log.py # 6 doctests failed 20201117T19:31:00.7958207Z sage t randomseed=0 /__w/sage/sage/src/sage/functions/other.py # 15 doctests failed 20201117T19:31:00.7959216Z sage t randomseed=0 /__w/sage/sage/src/sage/functions/orthogonal_polys.py # 11 doctests failed 20201117T19:31:00.7960272Z sage t randomseed=0 /__w/sage/sage/src/sage/geometry/polyhedron/parent.py # 2 doctests failed 20201117T19:31:00.7961352Z sage t randomseed=0 /__w/sage/sage/src/sage/geometry/polyhedron/palp_database.py # 1 doctest failed 20201117T19:31:00.7962568Z sage t randomseed=0 /__w/sage/sage/src/sage/algebras/quaternion_algebra_element.py # 4 doctests failed 20201117T19:31:00.7963697Z sage t randomseed=0 /__w/sage/sage/src/sage/algebras/quatalg/quaternion_algebra.py # 6 doctests failed 20201117T19:31:00.7964873Z sage t randomseed=0 /__w/sage/sage/src/sage/algebras/quatalg/quaternion_algebra_element.pyx # 38 doctests failed 20201117T19:31:00.7965949Z sage t randomseed=0 /__w/sage/sage/src/sage/modules/free_module.py # 5 doctests failed 20201117T19:31:00.7966956Z sage t randomseed=0 /__w/sage/sage/src/sage/modules/free_module_morphism.py # 3 doctests failed 20201117T19:31:00.7967988Z sage t randomseed=0 /__w/sage/sage/src/sage/modules/free_module_integer.py # 57 doctests failed 20201117T19:31:00.7969027Z sage t randomseed=0 /__w/sage/sage/src/sage/modules/vector_space_morphism.py # 3 doctests failed 20201117T19:31:00.7970041Z sage t randomseed=0 /__w/sage/sage/src/sage/modules/fg_pid/fgp_module.py # 3 doctests failed 20201117T19:31:00.7971051Z sage t randomseed=0 /__w/sage/sage/src/sage/symbolic/expression.pyx # 17 doctests failed 20201117T19:31:00.7972071Z sage t randomseed=0 /__w/sage/sage/src/sage/symbolic/assumptions.py # 3 doctests failed 20201117T19:31:00.7973064Z sage t randomseed=0 /__w/sage/sage/src/sage/symbolic/constants_c.pyx # 1 doctest failed 20201117T19:31:00.7974140Z sage t randomseed=0 /__w/sage/sage/src/sage/symbolic/maxima_wrapper.py # 2 doctests failed 20201117T19:31:00.7975123Z sage t randomseed=0 /__w/sage/sage/src/sage/symbolic/relation.py # 4 doctests failed 20201117T19:31:00.7976102Z sage t randomseed=0 /__w/sage/sage/src/sage/symbolic/constants.py # 17 doctests failed 20201117T19:31:00.7977173Z sage t randomseed=0 /__w/sage/sage/src/sage/symbolic/expression_conversions.py # 10 doctests failed 20201117T19:31:00.7978261Z sage t randomseed=0 /__w/sage/sage/src/sage/graphs/comparability.pyx # 1 doctest failed 20201117T19:31:00.7979313Z sage t randomseed=0 /__w/sage/sage/src/sage/graphs/digraph_generators.py # 8 doctests failed 20201117T19:31:00.7980236Z sage t randomseed=0 /__w/sage/sage/src/sage/graphs/graph.py # Timed out 20201117T19:31:00.7981184Z sage t randomseed=0 /__w/sage/sage/src/sage/graphs/connectivity.pyx # 2 doctests failed 20201117T19:31:00.7982192Z sage t randomseed=0 /__w/sage/sage/src/sage/graphs/graph_generators.py # Timed out 20201117T19:31:00.7983217Z sage t randomseed=0 /__w/sage/sage/src/sage/graphs/hypergraph_generators.py # 9 doctests failed 20201117T19:31:00.7984259Z sage t randomseed=0 /__w/sage/sage/src/sage/graphs/graph_coloring.pyx # 2 doctests failed 20201117T19:31:00.7985206Z sage t randomseed=0 /__w/sage/sage/src/sage/graphs/genus.pyx # 1 doctest failed 20201117T19:31:00.7986147Z sage t randomseed=0 /__w/sage/sage/src/sage/graphs/generic_graph_pyx.pyx # 2 doctests failed 20201117T19:31:00.7987202Z sage t randomseed=0 /__w/sage/sage/src/sage/sets/set_from_iterator.py # Timed out (and interrupt failed) 20201117T19:31:00.7988200Z sage t randomseed=0 /__w/sage/sage/src/sage/matrix/matrix1.pyx # 3 doctests failed 20201117T19:31:00.7989131Z sage t randomseed=0 /__w/sage/sage/src/sage/matrix/matrix2.pyx # 3 doctests failed 20201117T19:31:00.7990123Z sage t randomseed=0 /__w/sage/sage/src/sage/matrix/matrix_rational_dense.pyx # 1 doctest failed 20201117T19:31:00.7991189Z sage t randomseed=0 /__w/sage/sage/src/sage/matrix/matrix_integer_dense.pyx # 14 doctests failed 20201117T19:31:00.7992217Z sage t randomseed=0 /__w/sage/sage/src/sage/matrix/matrix_symbolic_dense.pyx # 1 doctest failed
https://github.com/tobiasdiez/sage/runs/1413136300?check_suite_focus=true
comment:184 Changed 2 years ago by
Commit:  77e1af16bcfce183bfe072cfb7324e82ee95a478 → d1341b0ab3b8bbc47a25beea2a8392632c981bfb 

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
9a5a145  version bump

d4c4453  remove old workaround from #22626

ebe9780  this file should exist  leave just comments there

a507fb9  trac 29314: fix version numbers in patches for gap_packages

9836f79  move GAP's PackageManager to gap from gap_packages

435d82f  Merge branch 'u/dimpase/packages/gap411' of git://trac.sagemath.org/sage into public/build/inplace

2fc8bf5  Hangs with maxima, so remove again

f6c1d87  Try to install maxima again

13e96ad  Fix workflow syntax

d1341b0  Can we now activate cysignals again?

comment:185 Changed 2 years ago by
Dependencies:  #30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850, #29314 → #30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850, #29314, #30937, #30936, #30935, #30934 

comment:186 Changed 2 years ago by
Dependencies:  #30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850, #29314, #30937, #30936, #30935, #30934 → #30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850, #29314, #30937, #30936, #30935, #30934, #30859, #30923, #30910 

comment:187 Changed 2 years ago by
Description:  modified (diff) 

comment:188 Changed 2 years ago by
Dependencies:  #30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850, #29314, #30937, #30936, #30935, #30934, #30859, #30923, #30910 → #30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850, #29314, #30937, #30936, #30935, #30934, #30859, #30923, #30910, #30944 

comment:189 Changed 2 years ago by
Dependencies:  #30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850, #29314, #30937, #30936, #30935, #30934, #30859, #30923, #30910, #30944 → #30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850, #29314, #30937, #30936, #30935, #30731 , #30859, #30923, #30910, #30944 

comment:190 Changed 2 years ago by
Keywords:  sd111 added 

Hoping we can make progress on this ticket this week  https://wiki.sagemath.org/days111
comment:191 Changed 2 years ago by
Status:  needs_review → needs_work 

comment:192 Changed 2 years ago by
Commit:  d1341b0ab3b8bbc47a25beea2a8392632c981bfb → 17e06daef346efc2f83f7b5304fc598c181079de 

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
4393801  Build all of sage deps

2027ebb  Run make in parallel

405e8d2  Install dependencies on mac with tox as well

894b9d5  Fix indent

4cefc89  Always run manually to see what's going on

ee6aaeb  Don't require spkgconfigure.m4 file

8cc37e7  Cleanup mac workflow

c571a07  Fix maxima install

8ee33f2  Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace

17e06da  Upgrade packages

comment:193 Changed 2 years ago by
From my side, this is actually ready for review. I've been using it now constantly the last few weeks and it is working. Or do you have something that I should be improving/working on?
comment:194 Changed 2 years ago by
Description:  modified (diff) 

comment:195 Changed 2 years ago by
Commit:  17e06daef346efc2f83f7b5304fc598c181079de → 5c719fe14a0fa6a0aa966031786cf6f3c437b239 

Branch pushed to git repo; I updated commit sha1. New commits:
3fcaf5f  Merge branch 'develop' of git://github.com/sagemath/sage into public/build/multiarchsimple

090e6f1  Simplify code

fa4556a  Remove _get_sage_local

7cc2005  Merge branch 'public/build/multiarchsimple' of git://trac.sagemath.org/sage into public/build/inplace

a590924  Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace

afa0518  Make autogen more stable

5c719fe  Fix functional.py:56: DeprecationWarning: invalid escape sequence \d

comment:196 Changed 2 years ago by
Status:  needs_work → needs_review 

Merged the latest develop branch.
comment:197 Changed 2 years ago by
Description:  modified (diff) 

comment:198 Changed 2 years ago by
Commit:  5c719fe14a0fa6a0aa966031786cf6f3c437b239 → 7f0335de97d12b1bd5e6cc56879ab47f7423665a 

comment:199 Changed 2 years ago by
Dependencies:  #30779, #30672, #30673, #30709, #30706, #30901, #30748, #29850, #29314, #30937, #30936, #30935, #30731 , #30859, #30923, #30910, #30944 → #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30936, #30935, #30731 , #30859, #30923, #30910, #30944 

comment:200 Changed 2 years ago by
Replying to mkoeppe:
It has been reported that the current
src/setup.py
(in Sage 9.2) is able to install a working sagelib on top of conda (see #28745).I'll create a test for this in #30845, and then we can test the present ticket to make sure it does not break anything.
Ensuring that the feature of installing sagelib on top of conda is not broken is a prerequisite for merging this ticket.
To move this ticket forward, could you help me with this GH Actions test? (This should be on top of #30944, which still needs review)
I think there should be one new workflow for all tests that do not build the Sage distribution (no build/pkgs
).
comment:201 followups: 202 203 Changed 2 years ago by
Sure, I will have a look at this. Could you by extending #30845 to include in more detail how the manual workflow with conda looks like (I've no experience with conda) and why #30944 is needed.
Maybe it's a good idea to review how the GH workflows are structured and revise this if necessary. For example, focus more on the purpose vs how it done, e.g. "test that building packages in build/pgks" works, "test that sagelib installed on top of various setups works". For the latter, my hope with pipenv was actually that one can easily run it on every push on every branch as a gateway in addition to the patchbot (since the time it takes is very minimal, time of setup of < 1h, + time for doctests).
comment:202 Changed 2 years ago by
Replying to ghtobiasdiez:
Sure, I will have a look at this. Could you by extending #30845 to include in more detail how the manual workflow with conda looks like (I've no experience with conda) and
Done.
why #30944 is needed.
Well, it will be testing something that currently happens to work but is not automatically tested.
Maybe it's a good idea to review how the GH workflows are structured and revise this if necessary.
Sure, I'll be happy about more eyes on the workflows.
For example, focus more on the purpose vs how it done
All workflows test fromscratch builds of the Sage distribution. Two workflows separately test optional / experimental packages. That's really all.
comment:203 Changed 2 years ago by
Replying to ghtobiasdiez:
"test that sagelib installed on top of various setups works". For the latter, my hope with pipenv was actually that one can easily run it on every push on every branch as a gateway in addition to the patchbot (since the time it takes is very minimal, time of setup of < 1h, + time for doctests).
An incremental build setup based on Docker has already been developed, which drives the current GitLab workflow. I'd recommend to start from there (it needs attention) instead of developing another thing from scratch... Take a look at #29536
comment:204 Changed 2 years ago by
Commit:  7f0335de97d12b1bd5e6cc56879ab47f7423665a → f96b923fd5c342c5401f4e8f993546b523fe2cc0 

Branch pushed to git repo; I updated commit sha1. New commits:
f96b923  Use sage_conf

comment:205 Changed 2 years ago by
Dependencies:  #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30936, #30935, #30731 , #30859, #30923, #30910, #30944 → #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944 

comment:206 Changed 2 years ago by
Commit:  f96b923fd5c342c5401f4e8f993546b523fe2cc0 → 8c97e7c9290c067af0e0bd974f60f4b3966a516c 

Branch pushed to git repo; I updated commit sha1. New commits:
8c97e7c  Cleanup code

comment:207 Changed 23 months ago by
Since now most dependencies are merged in the develop branch, this ticket is now definitely ready for review.
comment:208 Changed 23 months ago by
Commit:  8c97e7c9290c067af0e0bd974f60f4b3966a516c → f871fc77be3b6ec719dc3f64a033ca78e6deea67 

comment:209 followups: 210 212 Changed 23 months ago by
Matthias, I have a question: the current code works very well for systeminstalled packages. However, there are a few packages that are required but for which systeminstalled versions are not currently not supported (e.g. ecl). How could I use the sagebuilt versions of these packages for the built process of sagelib, i.e. for pip install e .
?
Take for example the import cdef extern from "ecl/ecl.h"
in libs/ecl.pxd
.
I guess one needs to pass the path of local/include/ecl
as argument to the C++ compiler. How is currently done?
comment:210 followups: 213 228 Changed 23 months ago by
Replying to ghtobiasdiez:
Take for example the import
cdef extern from "ecl/ecl.h"
inlibs/ecl.pxd
. I guess one needs to pass the path oflocal/include/ecl
as argument to the C++ compiler. How is currently done?
We install all packages that are not accepted from the system into $SAGE_LOCAL, and scripts such as sageenv
and sagebuildenv
set up environment variables such as LIBRARY_PATH
and CPATH
so that compilers find stuff. Configuration for Cython is somewhat separate  you are already familiar with the cython_aliases
function.
For ECL, there's ticket #30770 to make this configuration more flexible.
comment:211 Changed 23 months ago by
Also, if you are able to use system packages for which we do not have system package detection yet (spkgconfigure.m4
), you can join the effort in #27330.
comment:212 followup: 214 Changed 23 months ago by
Replying to ghtobiasdiez:
How could I use the sagebuilt versions of these packages for the built process of sagelib, i.e. for
pip install e .
?
One part of the answer to this involves the sage_conf
package: Installing it into your virtual environment provides sageenvconfig
and the sage_conf
module. So you may want to add it to Pipfile
. (See also #30913).
comment:213 Changed 23 months ago by
comment:214 Changed 23 months ago by
Replying to mkoeppe:
One part of the answer to this involves the
sage_conf
package: Installing it into your virtual environment providessageenvconfig
and thesage_conf
module. So you may want to add it toPipfile
. (See also #30913).
That's already done:
+[devpackages.8c46424] +path = "./../build/pkgs/sage_conf/src" +editable = true
comment:215 Changed 23 months ago by
Dependencies:  #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944 → #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770 

comment:216 Changed 23 months ago by
Commit:  f871fc77be3b6ec719dc3f64a033ca78e6deea67 → deeeb75c316c0e06c40e36ec65f472d4d3b90972 

comment:217 Changed 23 months ago by
Description:  modified (diff) 

comment:218 Changed 23 months ago by
Commit:  deeeb75c316c0e06c40e36ec65f472d4d3b90972 → 2ea4935dbae81bf1f2c9f9dc4e29f2feeb1a733f 

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
9988c5f  cicygwin*.yml: Adjust to new script packages _bootstrap, _prereq

ab19133  Merge branch 't/29124/scriptpackagesprereqtoolchainbootstrap' into t/30944/tox__improve_local_sudo_ubuntu_standard

c6ad06e  Merge branch 't/30944/tox__improve_local_sudo_ubuntu_standard' into t/31216/tox_ini__local___add_environment_variables_to_skip_system_package_installs__mechanism_for_a_local_interactive_shell

13f58fc  tox.ini (local): Run an interactive shell if posargs are bash

ace4ef6  tox.ini (local): Add option SKIP_SYSTEM_PKG_INSTALL=1

ddc4f0b  tox.ini (local): Handle environment variables SKIP_BOOTSTRAP, SKIP_CONFIGURE

bb6aec4  src/doc/en/developer/portability_testing.rst: Updates

6253bd8  src/doc/en/developer/portability_testing.rst: New section on SKIP_... environment variables

a52e899  Merge branch 'u/mkoeppe/tox_ini__local___add_environment_variables_to_skip_system_package_installs__mechanism_for_a_local_interactive_shell' of git://trac.sagemath.org/sage into public/build/inplace

2ea4935  Split workflow in install and build

comment:219 Changed 23 months ago by
Dependencies:  #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770 → #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216 

comment:220 Changed 23 months ago by
Commit:  2ea4935dbae81bf1f2c9f9dc4e29f2feeb1a733f → 6bcc5c34991dc4b81362b2427ff8d38b50f32c7e 

Branch pushed to git repo; I updated commit sha1. New commits:
6bcc5c3  Add esac

comment:221 Changed 23 months ago by
Commit:  6bcc5c34991dc4b81362b2427ff8d38b50f32c7e → d32b22b7091051522180ea58ea861b022e16e4d7 

Branch pushed to git repo; I updated commit sha1. New commits:
d32b22b  Try with config

comment:222 Changed 23 months ago by
Commit:  d32b22b7091051522180ea58ea861b022e16e4d7 → 73c99f3b7f797a7ab61507392c40268bb50e0a00 

Branch pushed to git repo; I updated commit sha1. New commits:
73c99f3  Ignore errors for now

comment:223 Changed 23 months ago by
Commit:  73c99f3b7f797a7ab61507392c40268bb50e0a00 → 5cb58cf579b41780791b9bf0cb46563713efd1e5 

Branch pushed to git repo; I updated commit sha1. New commits:
5cb58cf  Enable build as root

comment:224 Changed 23 months ago by
Commit:  5cb58cf579b41780791b9bf0cb46563713efd1e5 → 02542220efcfc3989a682ed082092c7f6e9a1c30 

Branch pushed to git repo; I updated commit sha1. New commits:
0254222  Cleanup github workflow

comment:225 Changed 23 months ago by
Commit:  02542220efcfc3989a682ed082092c7f6e9a1c30 → 4dcd083615e7eb7218b595c61fd9b9a4754222cb 

Branch pushed to git repo; I updated commit sha1. New commits:
4dcd083  Use all for the tests

comment:226 Changed 23 months ago by
Commit:  4dcd083615e7eb7218b595c61fd9b9a4754222cb → d30fda0a1bd9682d624feed871c3c94fa53921df 

Branch pushed to git repo; I updated commit sha1. New commits:
d30fda0  Also install maxima

comment:227 Changed 23 months ago by
Commit:  d30fda0a1bd9682d624feed871c3c94fa53921df → b17e6faef915267735b95135e326ffc74fd558b6 

Branch pushed to git repo; I updated commit sha1. New commits:
b17e6fa  Use configure.ac as make target

comment:228 followup: 231 Changed 23 months ago by
Replying to mkoeppe:
For ECL, there's ticket #30770 to make this configuration more flexible.
This seems to work now. Thanks for you help!
Two followup questions:
 Is there a make target similar to sagelibbuilddeps that doesn't install pythonic packages such as cython.
 Why doesn't sagelibbuilddeps install maxima?
comment:229 Changed 23 months ago by
Commit:  b17e6faef915267735b95135e326ffc74fd558b6 → 5d64c5a44b44e36c260eca75c60e179093f0b67f 

Branch pushed to git repo; I updated commit sha1. New commits:
5d64c5a  Install/fix other packages

comment:230 Changed 23 months ago by
Commit:  5d64c5a44b44e36c260eca75c60e179093f0b67f → dbf2df2f53f7b1b7e7760f0fd578f86a44c9df4b 

Branch pushed to git repo; I updated commit sha1. New commits:
dbf2df2  Install more dependencies manually

comment:231 Changed 23 months ago by
Replying to ghtobiasdiez:
 Is there a make target similar to sagelibbuilddeps that doesn't install pythonic packages such as cython.
Not yet. #30896 will go into this direction
 Why doesn't sagelibbuilddeps install maxima?
maxima is only needed at runtime of Sage, not while installing sagelib.
comment:232 Changed 23 months ago by
Commit:  dbf2df2f53f7b1b7e7760f0fd578f86a44c9df4b → b66df86359a2ee8be8ab481d0aea20b8960d0ef9 

Branch pushed to git repo; I updated commit sha1. New commits:
b66df86  Run tests in parallel

comment:233 Changed 23 months ago by
Commit:  b66df86359a2ee8be8ab481d0aea20b8960d0ef9 → 1fdfe680752f2878c76234733dc34780e3107715 

comment:234 Changed 23 months ago by
Commit:  1fdfe680752f2878c76234733dc34780e3107715 → f0bd45cf7ff1ac240bfd10af8844df187c1f97f4 

Branch pushed to git repo; I updated commit sha1. New commits:
f0bd45c  Install singulardata to find gftables?

comment:235 Changed 23 months ago by
Commit:  f0bd45cf7ff1ac240bfd10af8844df187c1f97f4 → 8a401efbb30b1049b05af3b4e756087d64968505 

comment:236 Changed 23 months ago by
Commit:  8a401efbb30b1049b05af3b4e756087d64968505 → 22e8fe7c5ab241996824216396b9025997c939be 

comment:237 Changed 23 months ago by
Dependencies:  #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216 → #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029 

Thanks for the help! I've now improved the github workflow and the doctest pass now (except a handful). So I would appreciate if this ticket could be review!
comment:238 Changed 23 months ago by
Commit:  22e8fe7c5ab241996824216396b9025997c939be → 3d071819a98f2d8f8e50935223d38042e04923e1 

Branch pushed to git repo; I updated commit sha1. New commits:
3d07181  Run tests in serial again

comment:239 Changed 23 months ago by
Commit:  3d071819a98f2d8f8e50935223d38042e04923e1 → cd38e05d1b88d4879bd65aa32eb1614356569802 

Branch pushed to git repo; I updated commit sha1. New commits:
cd38e05  Don't set SAGE_LOCAL in workflow

comment:240 Changed 23 months ago by
Commit:  cd38e05d1b88d4879bd65aa32eb1614356569802 → e42e1249042e31381f40889cef835e5947add799 

Branch pushed to git repo; I updated commit sha1. New commits:
e42e124  Update deps

comment:241 Changed 23 months ago by
Commit:  e42e1249042e31381f40889cef835e5947add799 → d1079de570eb56f0031153286ed3ff79e273986c 

Branch pushed to git repo; I updated commit sha1. New commits:
2d84b5a  tox.ini: Add missing esac

3fd16f4  Merge branch 'u/mkoeppe/tox_ini__local___add_environment_variables_to_skip_system_package_installs__mechanism_for_a_local_interactive_shell' of git://trac.sagemath.org/sage into public/build/inplace

d1079de  Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace

comment:242 followup: 243 Changed 23 months ago by
Instead of introducing a new environment variable SAGE_NUM_BUILD_JOBS
, probably best to use SAGE_NUM_THREADS
, which is used already in the same source tree.
comment:243 Changed 22 months ago by
Replying to mkoeppe:
Instead of introducing a new environment variable
SAGE_NUM_BUILD_JOBS
, probably best to useSAGE_NUM_THREADS
, which is used already in the same source tree.
Ok, done!
comment:244 Changed 22 months ago by
Commit:  d1079de570eb56f0031153286ed3ff79e273986c → d798925a1bb48a96087c3db03950e90059c9c37b 

comment:245 Changed 22 months ago by
Commit:  d798925a1bb48a96087c3db03950e90059c9c37b → cd8a8d19ea80b99357c465b4e1ff3f285a8b26b5 

Branch pushed to git repo; I updated commit sha1. New commits:
cd8a8d1  Move pgkconfig to normal package

comment:246 Changed 22 months ago by
Commit:  cd8a8d19ea80b99357c465b4e1ff3f285a8b26b5 → 63c7997e078bb7b3a1ab39ef1a2c29d210376f71 

Branch pushed to git repo; I updated commit sha1. New commits:
63c7997  Make cython and sage_conf also normal packages

comment:247 Changed 22 months ago by
Commit:  63c7997e078bb7b3a1ab39ef1a2c29d210376f71 → a18b3c59eff2940159fdb930eea4af04b444e828 

comment:248 Changed 22 months ago by
Commit:  a18b3c59eff2940159fdb930eea4af04b444e828 → f88cfc6a6382cf5cf35a602f83b7bedc06fc67eb 

Branch pushed to git repo; I updated commit sha1. New commits:
f88cfc6  Set SAGE_LOCAL in workflow for gap

comment:249 Changed 22 months ago by
Commit:  f88cfc6a6382cf5cf35a602f83b7bedc06fc67eb → d42fbfc8fe52e06a7c6a54cf6c0a7ea94c6e3a29 

Branch pushed to git repo; I updated commit sha1. New commits:
d42fbfc  See if error comes from memory problems

comment:250 Changed 22 months ago by
Commit:  d42fbfc8fe52e06a7c6a54cf6c0a7ea94c6e3a29 → 8580789c62f04a49ea893c53bf28d20a80a1aa73 

Branch pushed to git repo; I updated commit sha1. New commits:
8580789  Fix indent

comment:251 Changed 22 months ago by
Commit:  8580789c62f04a49ea893c53bf28d20a80a1aa73 → 1145360fdc51ee246c27715ce298c6005fd4b281 

Branch pushed to git repo; I updated commit sha1. New commits:
1145360  Fix paths to memory usage

comment:252 Changed 22 months ago by
Commit:  1145360fdc51ee246c27715ce298c6005fd4b281 → 4bdfef1b21da9f376cfd3c096cb9932fe5334e4e 

Branch pushed to git repo; I updated commit sha1. New commits:
4bdfef1  Try with sudo

comment:253 Changed 22 months ago by
Description:  modified (diff) 

comment:255 Changed 22 months ago by
Since the pipfile in this ticket is more uptodate and is welltested (manually and through the added github action), I would actually argue that #30913 should be on top of this ticket.
Moreover, #30770 is the only open dependency of this ticket, so I guess its also more realistic to get this in first.
comment:256 followup: 260 Changed 22 months ago by
This approach would also work. But I am skeptical about checking in the Pipfile.lock
. Stuff for nightmare merge conflicts for everyone. I think it would be better if this was an ignored file.
comment:257 Changed 22 months ago by
Cc:  Isuru Fernando added 

comment:258 Changed 22 months ago by
When the next beta comes out (hopefully soon), let's rebase it on top of it (and #30770). The commit history is a bit too wild
comment:259 Changed 22 months ago by
Priority:  major → critical 

Also, could you please:
 clean up the ticket description
 move documentation items from there to the manuals (installation and/or developer's manual)
 move the remaining issues to new or existing tickets
comment:260 followup: 264 Changed 22 months ago by
Replying to mkoeppe:
But I am skeptical about checking in the
Pipfile.lock
. Stuff for nightmare merge conflicts for everyone. I think it would be better if this was an ignored file.
Without the lock file you wouldn't get a reproducible virtual environment, since pipenv would always use the latest versions (that are compatible with each other). Moreover, the lock file is normally not changed, except if you call pipenv update
.
The workflow to update Python packages would then look as follows:
 Run
pipenv update
 Test
 If everything is fine, create ticket updating pipfile.lock and create a ticket for each updated dependency that is also listed in sage's build/pgks folder.
Will have a look at the other todo's you mention.
comment:261 Changed 22 months ago by
Description:  modified (diff) 

comment:262 Changed 22 months ago by
Cleaned up the ticket description and added a bit of documentation. Since this is still somewhat experimental, I've only put it in the coding basics of the development doc. I guess with the recent changes and support of virtual environments, it might soon be time to update the recommended way of setting up a development platform.
comment:263 Changed 22 months ago by
Commit:  4bdfef1b21da9f376cfd3c096cb9932fe5334e4e → 543b6d4e55879dee44f0e3d482b1ca1ebeda1d04 

Branch pushed to git repo; I updated commit sha1. New commits:
543b6d4  Add basic documentation

comment:264 Changed 22 months ago by
Replying to ghtobiasdiez:
The workflow to update Python packages would then look as follows [...]
I would say it's not a good idea to try to change the developer workflow regarding package upgrades as part of this ticket.
comment:265 Changed 22 months ago by
I agree, that was more meant to illustrate possible applications of the new pipenv environment (and the lockfile for that purpose).
comment:267 Changed 22 months ago by
Regarding the documentation: "After following the normal procedure of setting up your local development environment"  I think we should be explicit about requiring the full Sage distribution to be built first. We cannot support experimental workflows as you (still) describe them in the ticket description.
comment:268 Changed 22 months ago by
Commit:  543b6d4e55879dee44f0e3d482b1ca1ebeda1d04 → 510a3bb5344b544e70b12cd53dd7c48e19701963 

Branch pushed to git repo; I updated commit sha1. New commits:
510a3bb  Clarify point in the docs

comment:270 Changed 22 months ago by
Commit:  510a3bb5344b544e70b12cd53dd7c48e19701963 → c21cc76386b75dbf07f01ae822c120ef9f29c9e9 

Branch pushed to git repo; I updated commit sha1. New commits:
c21cc76  Update deps

comment:271 Changed 22 months ago by
Description:  modified (diff) 

I've moved the part of the ticket description describing the experimental workflow on bare ubuntu to #31347.
comment:273 Changed 22 months ago by
Commit:  c21cc76386b75dbf07f01ae822c120ef9f29c9e9 → cf068dfaf42a32f6bbd1ebd55376006e56fb6d8a 

Branch pushed to git repo; I updated commit sha1. New commits:
cf068df  Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace

comment:274 Changed 22 months ago by
Nice, thanks for letting me know. I've merged the latest develop branch. The history of this branch still needs to be cleanup up, but let's do this after positive review.
comment:275 Changed 22 months ago by
Could you:
 remove the change to
tox.ini
please?  Rename
sage_build_ext
and move it to a new module ofsage_setup.command
(and document what is different to the existingsage_build_ext
)
comment:276 followups: 285 287 Changed 22 months ago by
In the new CI workflows:
 For mac: I do not understand the line:
brew install pari
 For mac: Instead of setting
PKG_CONFIG_PATH
manually, perhaps use.homebrewbuildenv
because this is whatconfigure
suggests users to use  For linux: Why
make maxima conway_polynomials jmol conway_polynomials elliptic_curves combinatorial_designs polytopes_db graphs threejs
(it is not done in the mac workflow). Should we create amake
target that corresponds to what is needed?  For linux: Could you elaborate on "# Set SAGE_LOCAL (required for gap)"
Could you review and add comments to the file please?
comment:277 Changed 22 months ago by
Also, as #30589 has been merged, probably best to switch from python 3.8 to 3.9 here too
comment:278 Changed 22 months ago by
Also #30912 will bring merge conflicts that are best to resolve now
comment:280 followup: 361 Changed 22 months ago by
Also create_extension
duplicates some of the code in sage_build_cython.create_extension
. In particular, it should not use numpy.get_include()
directly but rather use sage.env.sage_include_directories
(note #31333 for a related change).
comment:281 Changed 22 months ago by
I have tested this ticket (on top of #31335, which fixes macOSspecific problems, and resolving the mentioned merge conflicts); following the instructions from the added documentation, pipenv install
leads to ERROR: Couldn't install package: cypari2
.
The first error leading to this is:
[1/7] Cythonizing cypari2/closure.pyx Error compiling Cython file:  ... # http://www.gnu.org/licenses/ #***************************************************************************** from __future__ import absolute_import, division, print_function from cysignals.signals cimport sig_on, sig_off, sig_block, sig_unblock, sig_error ^  cypari2/closure.pyx:36:0: 'cysignals/signals.pxd' not found
comment:282 Changed 22 months ago by
The specific issue in cypari2
is probably https://github.com/sagemath/cypari2/issues/93 (tracked in #30922).
But overall I think it is unrealistic for these instructions to work.
Let's use more restrictive instructions, in which pipenv
is used from within sage sh
. Then one can at least assume that all libraries are available.
comment:283 Changed 22 months ago by
Status:  needs_review → needs_work 

comment:284 Changed 22 months ago by
(Something like ../sage sh c 'PIP_FIND_LINKS=file://$(cd .. && pwd)/local/var/lib/sage/wheels pipenv install v v'
similar to what I am testing in #30913.)
comment:285 followup: 288 Changed 22 months ago by
Thanks for the review! Will address most of your points later.
Replying to mkoeppe:
 For linux: Could you elaborate on "# Set SAGE_LOCAL (required for gap)"
Without this line one gets the following error:
Traceback (most recent call last): File "/__w/sagetracmirror/sagetracmirror/src/sage/interfaces/expect.py", line 1468, in __init__ self._name = parent._create(value, name=name) File "/__w/sagetracmirror/sagetracmirror/src/sage/interfaces/interface.py", line 500, in _create self.set(name, value) File "/__w/sagetracmirror/sagetracmirror/src/sage/interfaces/gap.py", line 1421, in set self._eval_line(cmd, allow_use_file=True) File "/__w/sagetracmirror/sagetracmirror/src/sage/interfaces/gap.py", line 790, in _eval_line self._start() File "/__w/sagetracmirror/sagetracmirror/src/sage/interfaces/gap.py", line 1254, in _start gap_reset_workspace(verbose=False) File "/__w/sagetracmirror/sagetracmirror/src/sage/interfaces/gap.py", line 1587, in gap_reset_workspace g.eval('SetUserPreference("HistoryMaxLines", 30)') File "/__w/sagetracmirror/sagetracmirror/src/sage/interfaces/gap.py", line 587, in eval result = Expect.eval(self, input_line, **kwds) File "/__w/sagetracmirror/sagetracmirror/src/sage/interfaces/expect.py", line 1381, in eval return '\n'.join([self._eval_line(L, allow_use_file=allow_use_file, **kwds) File "/__w/sagetracmirror/sagetracmirror/src/sage/interfaces/expect.py", line 1381, in <listcomp> return '\n'.join([self._eval_line(L, allow_use_file=allow_use_file, **kwds) File "/__w/sagetracmirror/sagetracmirror/src/sage/interfaces/gap.py", line 790, in _eval_line self._start() File "/__w/sagetracmirror/sagetracmirror/src/sage/interfaces/gap.py", line 1259, in _start Expect._start(self, "Failed to start GAP.") File "/__w/sagetracmirror/sagetracmirror/src/sage/interfaces/expect.py", line 520, in _start raise RuntimeError("unable to start %s: %s" % (self.name(), msg)) RuntimeError: unable to start gap: End Of File (EOF). Exception style platform. Gap finished running /__w/sagetracmirror/sagetracmirror/.tox/localsudostandard/local/bin/gap r b p T E o 401m s 401m m 64m /__w/sagetracmirror/sagetracmirror/src/sage/ext_data/gap/sage.g command: /__w/sagetracmirror/sagetracmirror/.tox/localsudostandard/local/bin/gap args: ['/__w/sagetracmirror/sagetracmirror/.tox/localsudostandard/local/bin/gap', 'r', 'b', 'p', 'T', 'E', 'o', '401m', 's', '401m', 'm', '64m', '/__w/sagetracmirror/sagetracmirror/src/sage/ext_data/gap/sage.g'] buffer (last 100 chars): b'' before (last 100 chars): b'Set the environment variable SAGE_LOCAL.\r\n' after: <class 'pexpect.exceptions.EOF'> match: None match_index: None exitstatus: 1 flag_eof: True pid: 3124 child_fd: 26 closed: False timeout: None delimiter: <class 'pexpect.exceptions.EOF'> logfile: None logfile_read: None logfile_send: None maxread: 4194304 ignorecase: False searchwindowsize: None delaybeforesend: None delayafterclose: 0.1 delayafterterminate: 0.1 searcher: searcher_re: 0: re.compile(b'gap> ')
I guess the current github workflows don't see this as everything is invoked in one tox command and not in multiple steps (and enviroment variables are only active in each step).
comment:286 followup: 311 Changed 22 months ago by
Concerning the cypari error, I have no clue what is needed for it to compile on mac. It's also strange that it works on the github action (for mac as well as linux) but not on your system. cysignals is actually declared in the pipenv file, so it should be installed. The simplest solution would be if cypari would publish wheels on pypi.
But overall I think it is unrealistic for these instructions to work.
It's not unrealistic, it is working very well for Linux.
comment:287 followup: 290 Changed 22 months ago by
Replying to mkoeppe:
 For linux: Why
make maxima conway_polynomials jmol conway_polynomials elliptic_curves combinatorial_designs polytopes_db graphs threejs
(it is not done in the mac workflow). Should we create amake
target that corresponds to what is needed?
That's the only way I found to install these packages into SAGE_SHARE
. Will add it also to the mac workflow. I cannot judge if it would make sense to add a special make target for it, especially since you are working on making these packages normal python packages.
comment:288 Changed 22 months ago by
Replying to ghtobiasdiez:
Replying to mkoeppe:
 For linux: Could you elaborate on "# Set SAGE_LOCAL (required for gap)"
Without this line one gets the following error:
... Expect._start(self, "Failed to start GAP.") File "/__w/sagetracmirror/sagetracmirror/src/sage/interfaces/expect.py", line 520, in _start ...I guess the current github workflows don't see this as everything is invoked in one tox command and not in multiple steps (and enviroment variables are only active in each step).
OK, I see.
There are two ways to fix this:
 as described in #30818 ("Set environment for subprocesses invoked by Sage"). Currently subprocesses (such as the gap process invoked by the pexpectbased gap interface) rely on
sageenv
being sourced  using
pipenv run sage t
instead ofpipenv run sageruntests
, which usessageenv
.
comment:289 followup: 310 Changed 22 months ago by
comment:290 followup: 309 Changed 22 months ago by
Replying to ghtobiasdiez:
Replying to mkoeppe:
 For linux: Why
make maxima conway_polynomials jmol conway_polynomials elliptic_curves combinatorial_designs polytopes_db graphs threejs
(it is not done in the mac workflow). Should we create amake
target that corresponds to what is needed?That's the only way I found to install these packages into
SAGE_SHARE
. Will add it also to the mac workflow. I cannot judge if it would make sense to add a special make target for it, especially since you are working on making these packages normal python packages.
I've opened #31362 for this  which will add make allsagelocal
.
comment:291 Changed 22 months ago by
Thanks!
Concerning the above cysignals problem, you can try to run pipenv run pip install cysignals
and then pipenv install
again. Similarly, pipenv run pip install gmpy2==2.1.0b5
might be necessary if you have problems with the installation of pplpy. Let me know if this helps and I will add it to the documentation.
comment:292 Changed 22 months ago by
Commit:  cf068dfaf42a32f6bbd1ebd55376006e56fb6d8a → 398e6dd59ac7b8e3c49dc7917295e4fe4a9bcaba 

Branch pushed to git repo; I updated commit sha1. New commits:
1f1dddc  Revert change to tox.ini

0c3f517  Move sage_build_ext

d06c84d  Run workflow for Python 3.9 as well

7fee372  Doc: Also install dev packages

a506c1f  Use sage_build command for generating sources

398e6dd  Narrow down imports to reduce install requires deps

comment:293 Changed 22 months ago by
Commit:  398e6dd59ac7b8e3c49dc7917295e4fe4a9bcaba → 12c0a2ed96e5608f8f6c54aef0d81e8d76cdcff5 

Branch pushed to git repo; I updated commit sha1. New commits:
12c0a2e  Mac: remove pari install

comment:294 Changed 22 months ago by
Commit:  12c0a2ed96e5608f8f6c54aef0d81e8d76cdcff5 → 0157a4eac42646f49b9448afd120d07c4848750b 

Branch pushed to git repo; I updated commit sha1. New commits:
1041128  Rename ecl config variable

fd05a97  Add fallback

2ac4cc8  Merge branch 'develop' of git://github.com/sagemath/sage into public/build/eclconfig

6370c34  src/sage/env.py: Do not use capture_output, which requires python 3.7

31f1bbb  Also use universal_newlines instead of text, for python 3.6 compatibility

11ad40a  Merge branch 'public/build/eclconfig' of git://trac.sagemath.org/sage into public/build/eclconfig

1a8e2e4  Merge branch 'public/build/eclconfig' of git://trac.sagemath.org/sage into public/build/inplace

0157a4e  Reuse sage_include_directories

comment:295 Changed 22 months ago by
Commit:  0157a4eac42646f49b9448afd120d07c4848750b → a3806c58709c3b5c828668f823b37619bf1d1551 

comment:296 Changed 22 months ago by
Commit:  a3806c58709c3b5c828668f823b37619bf1d1551 → f5551673591c836942a4b7d50f971ad7c910763e 

Branch pushed to git repo; I updated commit sha1. New commits:
f555167  Fix workflow syntax

comment:297 Changed 22 months ago by
Commit:  f5551673591c836942a4b7d50f971ad7c910763e → bb90f76abc82987e7ff1cbfd2653dfa908fefc7b 

Branch pushed to git repo; I updated commit sha1. New commits:
bb90f76  Well...maybe now

comment:298 Changed 22 months ago by
Commit:  bb90f76abc82987e7ff1cbfd2653dfa908fefc7b → d1c2a0a00f8c675ad218b502516dca12d161933d 

Branch pushed to git repo; I updated commit sha1. New commits:
d1c2a0a  Try with latest version of pip

comment:299 Changed 22 months ago by
Commit:  d1c2a0a00f8c675ad218b502516dca12d161933d → f8599611360fcf04a54d28830c266183cd99ff99 

Branch pushed to git repo; I updated commit sha1. New commits:
f859961  Run everything versus the correct python

comment:300 Changed 22 months ago by
Commit:  f8599611360fcf04a54d28830c266183cd99ff99 → 606966b86e35ae679edc08dd1c5fa156cff667e8 

Branch pushed to git repo; I updated commit sha1. New commits:
606966b  Try to trigger macos workflow

comment:301 Changed 22 months ago by
Commit:  606966b86e35ae679edc08dd1c5fa156cff667e8 → 541f7143df6f00116a3048fd34242e90a50ac40e 

Branch pushed to git repo; I updated commit sha1. New commits:
541f714  Remove empty line?

comment:302 Changed 22 months ago by
Commit:  541f7143df6f00116a3048fd34242e90a50ac40e → 42d25a14c00a68e58bcb904783483a5514eecb5f 

Branch pushed to git repo; I updated commit sha1. New commits:
42d25a1  Still not?

comment:303 Changed 22 months ago by
Commit:  42d25a14c00a68e58bcb904783483a5514eecb5f → f8599611360fcf04a54d28830c266183cd99ff99 

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
comment:304 Changed 22 months ago by
Commit:  f8599611360fcf04a54d28830c266183cd99ff99 → f312a35f92ef30fa4a38ab54560686e2ef1bd5d6 

Branch pushed to git repo; I updated commit sha1. New commits:
f312a35  Try to remove python3pip again

comment:305 Changed 22 months ago by
Commit:  f312a35f92ef30fa4a38ab54560686e2ef1bd5d6 → b01c4c98a6c590ec7526d2f87dba7a839ebee9df 

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
b01c4c9  Install tox via pip

comment:306 Changed 22 months ago by
Commit:  b01c4c98a6c590ec7526d2f87dba7a839ebee9df → a1d989c3efd08d5f64612865f428102b328744da 

Branch pushed to git repo; I updated commit sha1. New commits:
a1d989c  Install pip as wel of course

comment:307 Changed 22 months ago by
Commit:  a1d989c3efd08d5f64612865f428102b328744da → 442596137f517ff3cb3e365e1b8f79b41744aec2 

Branch pushed to git repo; I updated commit sha1. New commits:
4425961  Strange...

comment:308 Changed 22 months ago by
Commit:  442596137f517ff3cb3e365e1b8f79b41744aec2 → f2e436c459bad755685c6ca42a3d401b1fd29ed9 

Branch pushed to git repo; I updated commit sha1. New commits:
f2e436c  Fix path of tox venv under mac

comment:309 Changed 22 months ago by
Replying to mkoeppe:
Replying to ghtobiasdiez:
Replying to mkoeppe:
 For linux: Why
make maxima conway_polynomials jmol conway_polynomials elliptic_curves combinatorial_designs polytopes_db graphs threejs
(it is not done in the mac workflow). Should we create amake
target that corresponds to what is needed?That's the only way I found to install these packages into
SAGE_SHARE
. Will add it also to the mac workflow. I cannot judge if it would make sense to add a special make target for it, especially since you are working on making these packages normal python packages.I've opened #31362 for this  which will add
make allsagelocal
.
OK, the toplevel target is now make buildlocal
.
comment:310 Changed 22 months ago by
Replying to ghtobiasdiez:
Thanks. Do you plan to work on it? Otherwise a quick fix is to use pipenv run sage t
instead of pipenv run sageruntests
as I wrote above.
comment:311 Changed 22 months ago by
Replying to ghtobiasdiez:
Concerning the cypari error, I have no clue what is needed for it to compile on mac.
Well it certainly needs the PARI library! When running pipenv
outside of the environment set using sage sh
, it depends on everything already installed in the system.
That's why I am saying the instructions to the users should run pipenv
within that shell.
comment:312 Changed 22 months ago by
Commit:  f2e436c459bad755685c6ca42a3d401b1fd29ed9 → a6e57808dcabbb60d9c7fdefb88ed7ca4be7caff 

comment:313 Changed 22 months ago by
Commit:  a6e57808dcabbb60d9c7fdefb88ed7ca4be7caff → 85de00c2d4103d0edbcd2a41a2efcee52ea5c26e 

Branch pushed to git repo; I updated commit sha1. New commits:
85de00c  Also install libffi8ubuntu1

comment:314 Changed 22 months ago by
Commit:  85de00c2d4103d0edbcd2a41a2efcee52ea5c26e → 1760cf741ff4d392e7662075637350fe8e7894bb 

Branch pushed to git repo; I updated commit sha1. New commits:
1760cf7  Or try to install libffi7

comment:315 Changed 22 months ago by
Commit:  1760cf741ff4d392e7662075637350fe8e7894bb → efea01f09fb253083afc5c04ffa5824bdd4d731c 

Branch pushed to git repo; I updated commit sha1. New commits:
efea01f  Install wget as well

comment:316 Changed 22 months ago by
Commit:  efea01f09fb253083afc5c04ffa5824bdd4d731c → 5a2cc305cf50e561d400a42d3e568cee7ea082a1 

comment:317 Changed 22 months ago by
Commit:  5a2cc305cf50e561d400a42d3e568cee7ea082a1 → 2ae43976cfacb331ae82426f25528213987225dd 

Branch pushed to git repo; I updated commit sha1. New commits:
2ae4397  Mac: install pari as well

comment:318 followup: 322 Changed 22 months ago by
Is it actually desired that tox localhomebrew installs a fresh homebrew in the tox folder (if I'm not misreading the logs). Is there a way to use the system's homebrew?
comment:319 Changed 22 months ago by
Commit:  2ae43976cfacb331ae82426f25528213987225dd → 88dabfa498cc3e12116d6f5f26e945ce4abd1429 

comment:320 Changed 22 months ago by
Commit:  88dabfa498cc3e12116d6f5f26e945ce4abd1429 → 8d5e37b6ab241cb2f98462de9b536937724ed977 

comment:321 Changed 22 months ago by
Commit:  8d5e37b6ab241cb2f98462de9b536937724ed977 → d9ac8df40f4eb4702795e544649a9e1830f25874 

Branch pushed to git repo; I updated commit sha1. New commits:
d9ac8df  Revert changes to sage.env import

comment:322 Changed 22 months ago by
Replying to ghtobiasdiez:
Is it actually desired that tox localhomebrew installs a fresh homebrew in the tox folder
Yes, that's by design
Is there a way to use the system's homebrew?
From tox.ini comments:
# The variant "localhomebrewmacosusrlocal" uses the global installation in /usr/local # instead. It may install packages or update packages. It will not remove packages. # Use at your own risk.
comment:323 followup: 329 Changed 22 months ago by
Thanks! That is working nicely.
I think I've now addressed all your comments and suggestions.
The only thing is that since merging the latest develop branch the sage packages (installed via tox) are no longer found when running pipenv install
. At the commit https://github.com/sagemath/sagetracmirror/commit/4bdfef1b21da9f376cfd3c096cb9932fe5334e4e this was working, but stopped after I've merged the develop branch. Now packages such as linbox, singular and gap are still compiled by sage's make, but pkgconfig (in sage.env) cannot find them any longer and thus the c++ compilation stopped working. I don't have these problems locally, where I don't use tox. Do you have an idea which recent changes are the origin of this issue?
I've also tried your suggestion with sage sh pipenv
but run into several issues. I guess the problem is that pipenv is trying to reuse sages virtual environment.
comment:324 Changed 22 months ago by
Commit:  d9ac8df40f4eb4702795e544649a9e1830f25874 → 245d40fe531570a2bd3cc71e13d851a25e1ff858 

Branch pushed to git repo; I updated commit sha1. New commits:
245d40f  Use correct version of fpylll for mac

comment:325 Changed 22 months ago by
Commit:  245d40fe531570a2bd3cc71e13d851a25e1ff858 → 4b3c76f9fbe11c202c8de6aa7fe661bcfc7aeca7 

Branch pushed to git repo; I updated commit sha1. New commits:
4b3c76f  Try to fix PKG_CONFIG_PATH and LD_LIBRARY_PATH

comment:326 followup: 331 Changed 22 months ago by
...never touch LD_LIBRARY_PATH... it's always wrong
comment:327 Changed 22 months ago by
Could you please take a look at #31362  it provides make buildlocal
, which I think is what you need
comment:328 followup: 335 Changed 22 months ago by
Thanks for making all the changes.
Right now it's hard for me to test anything from this ticket because the homebrew changes in #31335 conflict.
I would propose to do the following:
comment:329 Changed 22 months ago by
Replying to ghtobiasdiez:
I've also tried your suggestion with
sage sh pipenv
but run into several issues. I guess the problem is that pipenv is trying to reuse sages virtual environment.
See if you can work around this by having pipenv
use the base python that is shown in $SAGE_LOCAL/pyvenv.cfg
.
comment:330 Changed 22 months ago by
Commit:  4b3c76f9fbe11c202c8de6aa7fe661bcfc7aeca7 → 7c89641c4f2eed5e7cac24776e5079c346d575de 

Branch pushed to git repo; I updated commit sha1. New commits:
7c89641  Update lock file

comment:331 followup: 333 Changed 22 months ago by
Replying to mkoeppe:
...never touch LD_LIBRARY_PATH... it's always wrong
Ok, but what's then the correct way to do this?
comment:332 Changed 22 months ago by
Dependencies:  #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029 → #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029, #30912 

comment:333 Changed 22 months ago by
Replying to ghtobiasdiez:
Replying to mkoeppe:
...never touch LD_LIBRARY_PATH... it's always wrong
Ok, but what's then the correct way to do this?
Building all libraries with rpath  see LDFLAGS setting in src/bin/sageenv
comment:334 Changed 22 months ago by
Commit:  7c89641c4f2eed5e7cac24776e5079c346d575de → b68ce04b04b2d2104c6643700fa0f8cbe81f9f74 

Branch pushed to git repo; I updated commit sha1. New commits:
2c40179  build/pkgs/sagelib/src: Update metadata

2a91862  build/pkgs/sagelib/src/README.rst: Add some text

6432727  Merge tag '9.3.beta2' into t/30912/sagelib__update_metadata_for_pypi_deployment

4a693f2  Move build/pkgs/sagelib/src/setup.cfg to SAGE_ROOT/src, replace by symlink

ea182d7  Copy changes from build/pkgs/sagelib/src to src

a1a10b9  src/VERSION.txt: New

5697335  src/setup.cfg: Add license_file=LICENSE.txt

deb9eb3  Merge tag '9.3.beta3' into t/30912/sagelib__update_metadata_for_pypi_deployment

7ad4c0e  Merge tag '9.3.beta4' into t/30912/sagelib__update_metadata_for_pypi_deployment

b68ce04  Merge branch 'u/mkoeppe/sagelib__update_metadata_for_pypi_deployment' of git://trac.sagemath.org/sage into public/build/inplace

comment:335 followup: 336 Changed 22 months ago by
Replying to mkoeppe:
I've merged the current versions of #30770 and #30912. Not sure what's the purpose of #31362. That would make the line make maxima conway_polynomials jmol conway_polynomials elliptic_curves combinatorial_designs polytopes_db graphs threejs
obsolete right? If it's just for this somewhat cosmetic improvement, I would suggest to turn around the tables and let #31362 depend on this ticket (and make the changes in the workflow there as well). Reason: I had to play a lot around with the different targets until everything works, so I'm a bit hesitant to change things again (and since unmerging a branch is not so easy, one cannot simply revert the experiment).
comment:336 followup: 338 Changed 22 months ago by
Replying to ghtobiasdiez:
Thanks.
Not sure what's the purpose of #31362. That would make the line
make maxima conway_polynomials jmol conway_polynomials elliptic_curves combinatorial_designs polytopes_db graphs threejs
obsolete right? If it's just for this somewhat cosmetic improvement
Yes, it's for making all standard nonPython packages available in SAGE_LOCAL
.
Reason: I had to play a lot around with the different targets until everything works, so I'm a bit hesitant to change things again
This does not sound good. If installing additional packages from the distribution in SAGE_LOCAL
breaks the proposed setup, then this ticket is definitely not ready.
comment:337 Changed 22 months ago by
Note also that #31362 does not change anything other than add two new make targets. So even if you find that these targets do not do what you need, there will be no need for unmerging it.
comment:338 Changed 22 months ago by
Replying to mkoeppe:
Reason: I had to play a lot around with the different targets until everything works, so I'm a bit hesitant to change things again
This does not sound good. If installing additional packages from the distribution in
SAGE_LOCAL
breaks the proposed setup, then this ticket is definitely not ready.
More in the sense, that it was a bit of tryanderror to find the minimal set of make targets that make the build pass. You could have course build every package in the sage distribution, but the point is that with pipenv only a couple of them are actually needed.
So buildlocal = sagelibbuilddeps + maxima conway_polynomials jmol conway_polynomials elliptic_curves combinatorial_designs polytopes_db graphs threejs
?
comment:341 Changed 22 months ago by
Of course, there may be a number of nonPython libraries that are not needed IF you are able to pull in a wheel of the Python packages that somehow ship a bundled version of the nonPython library. For example, the numpy
wheels may come with their own BLAS  but it won't help because some of our own packages also need BLAS.
comment:342 Changed 22 months ago by
Commit:  b68ce04b04b2d2104c6643700fa0f8cbe81f9f74 → 2154a5b9f8f56ccc83bb45aa9bd10d659b015649 

Branch pushed to git repo; I updated commit sha1. New commits:
bdb2237  Merge branch 't/30912/sagelib__update_metadata_for_pypi_deployment' into t/31357/fixup_src_version_txt_added_in__30912

c16651c  src/VERSION.txt: Use same format as build/pkgs/sagelib/src/VERSION.txt

2154a5b  Merge branch 'u/mkoeppe/fixup_src_version_txt_added_in__30912' of git://trac.sagemath.org/sage into public/build/inplace

comment:343 Changed 22 months ago by
Dependencies:  #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029, #30912 → #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029, #30912, #31357 

comment:344 Changed 22 months ago by
Commit:  2154a5b9f8f56ccc83bb45aa9bd10d659b015649 → 2773e6fd9c1868346d194227287dffd6c3642a5e 

Branch pushed to git repo; I updated commit sha1. New commits:
55bc2aa  m4/sage_spkg_collect.m4: Generate install tree information for build/make/Makefile

d7dc9a2  build/make/Makefile.in: New targets allsagelocal, allsagevenv

27b589b  Merge branch 't/30912/sagelib__update_metadata_for_pypi_deployment' into t/31362/make_all_sage_local__make_all_sage_venv

8796008  Put pynac into SAGE_LOCAL, sagelib into SAGE_VENV

d6831b1  build/make/Makefile.in: Add allbuildlocal, allbuildvenv, which include dependency on toolchain

61f6ba6  Makefile: Add toplevel targets buildlocal, buildvenv

e354672  Merge branch 'u/mkoeppe/make_all_sage_local__make_all_sage_venv' of git://trac.sagemath.org/sage into public/build/inplace

2773e6f  Use buildlocal target

comment:345 Changed 22 months ago by
Dependencies:  #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029, #30912, #31357 → #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029, #30912, #31357, #31362 

comment:347 Changed 22 months ago by
In any case thanks to merging these tickets, I am able to merge the tickets that fix homebrew  so I'll be able to test later today.
comment:348 Changed 22 months ago by
../sage sh c 'ARCHFLAGS= PIP_NO_BUILD_ISOLATION=1 pipenv install v v'
got me a bit farther. (Setting ARCHFLAGS
to the empty string helps on macOS; PIP_NO_BUILD_ISOLATION=1
helps with some of the Python packages that declare their build dependencies incorrectly (see #30922). Now it fails with this:
raise MetadataInconsistent(self._ireq, "name", dist.project_name) pip._internal.exceptions.MetadataInconsistent: Requested fpylll from https://files.pythonhosted.org/packages/ad/c5/f30f354dcce03b928702414fdbdd916d60915c6b144360c88707784385bb/fpylll0.5.4.tar.gz#egg=fpylllrecent (from r /var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pipenv7zg7bw1srequirements/pipenvb8jmkho7requirement.txt (line 1)) has different name in metadata: 'fpylll' Removed build tracker: '/private/var/folders/38/wnh4gf1552g_crsjnv2vmmww0000gp/T/pipreqtrackerz5q5pu09' ... raise exceptions.InstallError(c.dep.name, extra=err_lines) pipenv.exceptions.InstallError: ERROR: Couldn't install package: fpylllrecent Package installation failed...
comment:349 Changed 22 months ago by
Commit:  2773e6fd9c1868346d194227287dffd6c3642a5e → e70f22ba61f2d4c1e0de5a254a15844129df7cab 

comment:350 followup: 353 Changed 22 months ago by
This error also occurred on the github workflow. I tried to fix it by setting the name in the pipfile. If this doesn't work can you please remove the fpylllrecent in the pipfile and invoke pipenv install https://files.pythonhosted.org/packages/ad/c5/f30f354dcce03b928702414fdbdd916d60915c6b144360c88707784385bb/fpylll0.5.4.tar.gz
This should work. Could you then please post what pipenv writes to the pipfile, thanks!
buildlocal
seems to work, except that it installs some python related tools/packages (pip, setuptools_scm, wheel) and doesn't install maxima.
comment:351 Changed 22 months ago by
Commit:  e70f22ba61f2d4c1e0de5a254a15844129df7cab → 8c22f9ec046cb8253531056e0a7aff7118ca778e 

Branch pushed to git repo; I updated commit sha1. New commits:
8c22f9e  Set LIBRARY_PATH

comment:352 Changed 22 months ago by
Commit:  8c22f9ec046cb8253531056e0a7aff7118ca778e → e17019c5528c867367988d7fa99bcef76f320949 

Branch pushed to git repo; I updated commit sha1. New commits:
e17019c  Install old readline as well

comment:353 Changed 22 months ago by
Replying to ghtobiasdiez:
buildlocal
seems to work, except that it installs some python related tools/packages (pip, setuptools_scm, wheel) and doesn't install maxima.
wheel (which pulls in the python toolchain) is fixed in #31321.
maxima  could you double check?
comment:354 Changed 22 months ago by
Ok good. And false alarm, maxima is installed (the output is just condensed to one line in contrast to the pages of text for make maxima
which has thrown me off).
The latest commit should hopefully fix the fpylll error.
comment:355 Changed 22 months ago by
Commit:  e17019c5528c867367988d7fa99bcef76f320949 → e2ed97f2ba3af909838802c95f4151282090c7d7 

Branch pushed to git repo; I updated commit sha1. New commits:
e2ed97f  Fix fpylll

comment:356 Changed 22 months ago by
I will put a rebased/squashed version of the setup.py
/sage_setup
changes on #31377, because this part seems ready and I hope we can get it into 9.3 to get developers used to the inplace install idea.
comment:357 followup: 362 Changed 22 months ago by
(It looks to me like the venv aspects of this ticket will take more time; in particular since they seem to involve getting an uncooperative random set of wheels from PyPI to cooperate with the system.)
comment:358 Changed 22 months ago by
It's quite the opposite: since some of the python packages (fyplll, cypari2) don't provide wheels, system packages need to be installed which is quite hard to coordinate across different systems.
comment:359 Changed 22 months ago by
But the Sage distribution has already solved this problem. Just build within a sagesh.
comment:361 Changed 22 months ago by
comment:362 Changed 22 months ago by
Replying to mkoeppe:
(It looks to me like the venv aspects of this ticket will take more time; in particular since they seem to involve getting an uncooperative random set of wheels from PyPI to cooperate with the system.)
On this topic, see also the discussion at https://github.com/sagemath/cypari2/issues/105.
I think it is a desirable feature to be able to edit the source code in an inplace install of sage (ie what pip e
provides). This would greatly benefit all Sage developers. However, I have hard time to understand the benefit of the venv. On the other hand, I do not understand how wheels could solve the libraries/softwares dependencies. More precisely, here is a scenario (which happens to be happening in Sage)
 SageMath depends on
pylib1
andpylib2
that uses the samelibfoo
C library. Both of these libraries provide wheels that bundlelibfoo
to make some of their users happy.  SageMath also depends on
pylib3
that depends on bothpylib1
andpylib2
and uses custom Cython code that uses functions fromlibfoo
.pylib3
does not provide wheel yet. How should it be implemented?
comment:363 Changed 22 months ago by
Hi Vincent, thanks for joining the discussion.
For a discussion of possible distribution strategies using wheels, let's use #31251 (Metaticket: Distribution as wheels), which also has a link to a recent post by Gommers on packaging. I agree with you that it is a hard problem.
For Sage 9.3, I have a much more modest goal regarding wheels that is already fully implemented and is only waiting for review (see ticket list in #29705 for tickets by milestone). Since #29500 (already in Sage 9.2), we generate wheels in $SAGE_LOCAL/var/lib/sage/wheels
. These wheels are not distributable as they do not bundle their shared library dependencies but rather rpath into $SAGE_LOCAL
. Users can make their own venvs for development purposes where they install the prebuilt wheels from this directory. #31045 and #31279 add some missing wheels. #30913 generates dependency metadata that makes it easier to set up the venv; and adds tests using tox
.
For editable install of sagelib as part of the normal build system (no extra venv) using Tobias' new setup.py
from this ticket, see #31377.
comment:364 Changed 22 months ago by
Dependencies:  #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029, #30912, #31357, #31362 → #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029, #30912, #31357, #31362, #31365 

comment:365 Changed 22 months ago by
Commit:  e2ed97f2ba3af909838802c95f4151282090c7d7 → 3e60c00b3cb16ae500584baf484017b922ac3988 

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
804ebd7  sage_setup.dpcbuild.AllBuilder: Restrict workaround to macOS

b4ceee5  sage_setup.docbuild: In the workaround, do not go through build_many to build serially

7369a3e  Merge branch 't/31344/homebrew__docbuild_crashes__libtcl_atforkprepare' into t/31365/add_ntl_to_cython_aliases

ce03e6b  src/sage/schemes/hyperelliptic_curves/hypellfrob.pyx: Merge distutils directives

32576b4  sage.misc.cython: Add NTL aliases, cache result of cython_aliases

6e30e3a  Use variables NTL_INCDIR, NTL_LIBDIR in sage_conf, separate from NTL_PREFIX used in sagebuildenvconfig; set std=c++11 in NTL_CFLAGS

7b1e27b  Merge distutils directives for Cython files using ntl

4d2828d  Switch Cython files that use NTL to language = c++

993c35c  build/pkgs/ntl/spkgconfigure.m4: Fix up

3e60c00  Merge branch 'u/mkoeppe/add_ntl_to_cython_aliases' of git://trac.sagemath.org/sage into public/build/inplace

comment:366 Changed 22 months ago by
Status:  needs_work → needs_review 

So, I think, I've found a solution to the fpylll problems. In fact, the github workflow should now also work for macos (at least the last run installed everything fine except sage, which failed due to an NTL error that is fixed with #31365). Since macos runs on the sage repo run currently really slow, you can find the latest run at https://github.com/tobiasdiez/sage/actions.
comment:368 Changed 22 months ago by
Priority:  critical → major 

Work issues:  → rebase on dependencies + #31384, #31377 
comment:369 Changed 22 months ago by
Dependencies:  #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029, #30912, #31357, #31362, #31365 → #30779, #30672, #30673, #30709, #30706, #30901, #31080, #29850, #29314, #30937, #30935, #30731 , #30859, #30923, #30910, #30944, #30770, #31216, #31029, #30912, #31357, #31362, #31365, #31377 

Work issues:  rebase on dependencies + #31384, #31377 
comment:370 Changed 22 months ago by
Commit:  3e60c00b3cb16ae500584baf484017b922ac3988 → 13d90ee00e330ff759d0f8fc0b4e3868a477248c 

Branch pushed to git repo; I updated commit sha1. New commits:
217c675  Try to fix gmp issue on mac

25df925  For mac not linux...

90a5104  Cleanup workflow

0c4ba62  Try to only specify gmp includes

a978458  Fix a few more errors

c1ce878  Also for the manual install

13d90ee  Remove a few unnecessary changes

comment:371 Changed 22 months ago by
Commit:  13d90ee00e330ff759d0f8fc0b4e3868a477248c → 08fef7dd62544dfb13327d4ad38fca5ab6691190 

comment:372 Changed 22 months ago by
Commit:  08fef7dd62544dfb13327d4ad38fca5ab6691190 → 8f8b61533287f67be77a4920f7ffceb5ebc39b0d 

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
18df2dc  Merge branch 't/31365/add_ntl_to_cython_aliases' into t/31377/__configure___enable_editable

19ca17a  src/sage/graphs/graph_decompositions/tdlib/sage_tdlib.cpp: Move one directory level higher

f02963b  Merge branch 't/31389/remove_clash_of_tdlib__with_tdlib_pyx_in__src_sage_graphs_graph_decompositions' into t/31377/__configure___enable_editable

2b293f8  src/setup.py: Install the jupyter kernel using sage_install

b05e4ba  build/make/Makefile.in (sagelibclean): Remove generated .c and .cpp files

271d427  src/setup.py: Fix sage.misc.sageinspect by setting Cython.Compiler.Options.embed_pos_in_docstring

f6b2fc4  src/sage/repl/interpreter.py: Fix a doctest for editable installs of sagelib

9a0731a  sage.interacs.debugger: Remove deprecated module

e5d47ce  Merge branch 't/31390/remove_deprecated_sage_interacts_debugger' into t/31377/__configure___enable_editable

8f8b615  Merge branch 't/31377/__configure___enable_editable' into t/30371/public/build/inplace

comment:374 followup: 376 Changed 22 months ago by
I don't know which problem you are trying to solve with the big try/except. I don't think reducing ImportError
is the correct solution.
If you are trying to make setup.py sdist
work when libs are missing, consider using the code from build/pkgs/sagelib/src/setup.py
line 61
comment:375 followup: 378 Changed 22 months ago by
comment:376 followup: 379 Changed 22 months ago by
Replying to mkoeppe:
I don't know which problem you are trying to solve with the big try/except. I don't think reducing
ImportError
is the correct solution.If you are trying to make
setup.py sdist
work when libs are missing, consider using the code frombuild/pkgs/sagelib/src/setup.py
line 61
Yes, that was my goal. Creating the virtual environment otherwise fails as Cython is not yet installed into it. The file you refer to doesn't seem to exist.
comment:377 Changed 22 months ago by
Commit:  8f8b61533287f67be77a4920f7ffceb5ebc39b0d → fec55234a00e46d023ad583f8c6284b970141ab0 

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
fec5523  Merge branch 't/31377/__configure___enable_editable' into t/30371/public/build/inplace

comment:378 Changed 22 months ago by
Replying to ghtobiasdiez:
Replying to mkoeppe:
Merged #31377, resolving edit conflicts.
Thanks, but this did revert some of my changes (in particular to setup.py)....
Sorry, my mistake. I've redone the merge now. Please take a look.
comment:379 Changed 22 months ago by
Replying to ghtobiasdiez:
Replying to mkoeppe:
I don't know which problem you are trying to solve with the big try/except. I don't think reducing
ImportError
is the correct solution.If you are trying to make
setup.py sdist
work when libs are missing, consider using the code frombuild/pkgs/sagelib/src/setup.py
line 61Yes, that was my goal. Creating the virtual environment otherwise fails as Cython is not yet installed into it. The file you refer to doesn't seem to exist.
comment:381 Changed 21 months ago by
Milestone:  sage9.3 → sage9.4 

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.
comment:382 Changed 17 months ago by
Milestone:  sage9.4 → sage9.5 

Setting a new milestone for this ticket based on a cursory review.
comment:383 Changed 15 months ago by
Milestone:  sage9.5 → sagewishlist 

Status:  needs_review → needs_work 
comment:384 Changed 14 months ago by
Commit:  fec55234a00e46d023ad583f8c6284b970141ab0 → 84b2dd396eb3cad818c3fb7893d38fcf8eec5031 

Branch pushed to git repo; I updated commit sha1. New commits:
84b2dd3  Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace

comment:385 Changed 14 months ago by
Commit:  84b2dd396eb3cad818c3fb7893d38fcf8eec5031 → 531811aebb493bce59dcfc09d250e317c2bf42bf 

comment:386 Changed 14 months ago by
Commit:  531811aebb493bce59dcfc09d250e317c2bf42bf → c56ab1698fe158dc741b6ec4cef7cc883836d835 

Branch pushed to git repo; I updated commit sha1. New commits:
c56ab16  Install Cython manually

comment:387 Changed 14 months ago by
Description:  modified (diff) 

comment:388 Changed 14 months ago by
Description:  modified (diff) 

comment:389 Changed 14 months ago by
Commit:  c56ab1698fe158dc741b6ec4cef7cc883836d835 → 29f5780478afe6612de6e50ff8371cc29403f94d 

comment:390 Changed 14 months ago by
Commit:  29f5780478afe6612de6e50ff8371cc29403f94d → 536fa27c5b54ecaa9884be03079dda75027a645f 

comment:391 Changed 14 months ago by
Description:  modified (diff) 

comment:392 Changed 14 months ago by
Description:  modified (diff) 

comment:393 Changed 14 months ago by
Commit:  536fa27c5b54ecaa9884be03079dda75027a645f → b88cba4dd65e2a7d398b0822adf2c074049f9910 

comment:394 Changed 14 months ago by
Commit:  b88cba4dd65e2a7d398b0822adf2c074049f9910 → 3e796c31b13425f16a71216ce8c299b8eda4ddad 

Branch pushed to git repo; I updated commit sha1. New commits:
3e796c3  Require new memoryallocator package

comment:395 Changed 14 months ago by
Commit:  3e796c31b13425f16a71216ce8c299b8eda4ddad → e9ac3ee9a49c71f6cf03cd6e8823d9c79ca51d89 

comment:396 Changed 14 months ago by
Description:  modified (diff) 

Milestone:  sagewishlist → sage9.5 
Status:  needs_work → needs_review 
I've finally found the time to merge the most recent version. Since most changes (in the build pipline and adding the pipfile etc) have been done already in previous tickets, this ticket now essentially amounts to adding a github action verifying that the setup works indeed. So ready for review again.
New commits:
38593f4  Clarify wording in github action workflow

e9ac3ee  Remove unnecessary special treatment of fpylll and cython

comment:397 Changed 14 months ago by
Commit:  e9ac3ee9a49c71f6cf03cd6e8823d9c79ca51d89 → 4b7eaa4adc898e8f9b2bfd727e31232f89e99783 

Branch pushed to git repo; I updated commit sha1. New commits:
4b7eaa4  Improve workflow step name

comment:399 followups: 400 401 407 Changed 14 months ago by
Separate tickets please for:
 the language=c++ changes
 the setup.py change
comment:400 Changed 14 months ago by
Replying to mkoeppe:
Separate tickets please for:
 the language=c++ changes
(... if these changes are actually needed)
comment:401 Changed 14 months ago by
comment:402 Changed 14 months ago by
Description:  modified (diff) 

comment:403 Changed 14 months ago by
Description:  modified (diff) 

comment:404 Changed 14 months ago by
Description:  modified (diff) 

comment:405 Changed 14 months ago by
Commit:  4b7eaa4adc898e8f9b2bfd727e31232f89e99783 → e183209eebd9d141b9a3a91af467e06f54711754 

Branch pushed to git repo; I updated commit sha1. New commits:
cb06460  Try to fix python version on mac

2d39970  Remove memory print as this is fixed now

380eec8  Improve documentation

5afbd62  Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace

e183209  Remove memory_allocator from pipfile (no longer needed)

comment:406 Changed 14 months ago by
Commit:  e183209eebd9d141b9a3a91af467e06f54711754 → c6fde5eef8edc893643513d5f99a29a2b720556c 

comment:407 Changed 14 months ago by
comment:408 followup: 412 Changed 14 months ago by
+ # Fix path to R, since otherwise sage's r is used + R_HOME: /usr/local/Cellar/r/4.0.3_2/lib/R
Is this needed, and if so, do we need this something like for a normal build too?
comment:409 followups: 413 430 Changed 14 months ago by
+ export SAGE_SHARE=$GITHUB_WORKSPACE/.tox/localhomebrewmacosusrlocal/local/share
This line (in both workflow scripts) shouldn't be needed.
I also don't understand the other environment exports in the workflow script  why not invoke the sageenv
script that sets up the correct environment?
comment:410 followup: 414 Changed 14 months ago by
Also the documentation on Pipenv added in src/doc/en/developer/coding_basics.rst
is not sufficiently specific about the steps needed before pipenv is installed and used.
comment:411 followup: 415 Changed 14 months ago by
And finally, checking in the Pipfile.lock
really does not fit into the Sage development workflow  it will always be out of date and will lead to merge conflicts. We really should make do with just a Pipfile
(generated from Pipfile.m4
) from the versions declared in build/pkgs
.
comment:412 Changed 14 months ago by
Replying to mkoeppe:
+ # Fix path to R, since otherwise sage's r is used + R_HOME: /usr/local/Cellar/r/4.0.3_2/lib/RIs this needed, and if so, do we need this something like for a normal build too?
Otherwise the pipinstall of rpy2 fails with
Collecting rpy2<3.4,>=3.3 Downloading rpy23.3.6.tar.gz (179 kB) Running command python setup.py egg_info Traceback (most recent call last): File "<string>", line 1, in <module> File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pipinstallm6wdy35r/rpy2_0c800ed9b8e34449aeee849f526a409b/setup.py", line 115, in <module> c_extension_status = get_r_c_extension_status() File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pipinstallm6wdy35r/rpy2_0c800ed9b8e34449aeee849f526a409b/setup.py", line 99, in get_r_c_extension_status *situation.get_r_flags(r_home, 'ldflags') File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pipinstallm6wdy35r/rpy2_0c800ed9b8e34449aeee849f526a409b/rpy2/situation.py", line 209, in get_r_flags _get_r_cmd_config(r_home, flags, File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pipinstallm6wdy35r/rpy2_0c800ed9b8e34449aeee849f526a409b/rpy2/situation.py", line 177, in _get_r_cmd_config output = subprocess.check_output( File "/Users/runner/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/subprocess.py", line 415, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, File "/Users/runner/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/subprocess.py", line 493, in run with Popen(*popenargs, **kwargs) as process: File "/Users/runner/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/subprocess.py", line 858, in __init__ self._execute_child(args, executable, preexec_fn, close_fds, File "/Users/runner/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/subprocess.py", line 1704, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) FileNotFoundError: [Errno 2] No such file or directory: '/Users/runner/work/sagetracmirror/sagetracmirror/.tox/localhomebrewmacosusrlocal/local/var/tmp/sage/build/r3.6.3/src/bin/R'
comment:413 followup: 420 Changed 14 months ago by
Replying to mkoeppe:
I also don't understand the other environment exports in the workflow script  why not invoke the
sageenv
script that sets up the correct environment?
I was not aware of the sageenv script. How does one uses it to set the environment for the following steps and replace the manual exports I've added? I couldn't find any documentation about that script.
comment:414 followup: 418 Changed 14 months ago by
Replying to mkoeppe:
Also the documentation on Pipenv added in
src/doc/en/developer/coding_basics.rst
is not sufficiently specific about the steps needed before pipenv is installed and used.
What do you mean concretely? Please also note that the documentation about venv should be improved in general, see #31342.
comment:415 followups: 419 444 445 446 Changed 14 months ago by
Replying to mkoeppe:
And finally, checking in the
Pipfile.lock
really does not fit into the Sage development workflow  it will always be out of date and will lead to merge conflicts. We really should make do with just aPipfile
(generated fromPipfile.m4
) from the versions declared inbuild/pkgs
.
I agree that it's somewhat suboptimal to have the version information of some packages specified in the sage distribution and in the pipenv lock file. However, the idea behind using pipenv is to provide an alternative way to quickly create a python environment for the development (and maybe at some point distribution) of sage. Thus such a duplication is needed to some degree for now. The point of the lock file is also to ensure that the python environment is the same for everyone (which makes it easier to reproduce issues and test failures etc) and more importantly that the builds in the github action are deterministic. Finally, in can be easily updated by running pipenv lock
so it's not a lot of overhead. Please also note that it doesn't need to be updated every time a sage dependency is updated since pipenv uses the install_requires
information to determine the admissible versions. I would suggest to run pipenv lock
after every release for now and see how this goes.
Moreover, the lock file is required for now as some of the packages (pplpy for example) need cython and other packages installed already on sdist and egg_info. This makes setting up the pipenv from scratch rather hard, since you first need to create a new virtual environment, then install those dependencies and only then run pipenv install. These problems don't occur if you use the lock file (and subsequently pipenv lock
).
Anyway, because there are pros and cons for each approach I would suggest to see how it goes with the lock file committed (as strongly recommended by pipenv). If it turns out that the disadvantages overweight we can always remove it later again.
comment:416 Changed 14 months ago by
Commit:  c6fde5eef8edc893643513d5f99a29a2b720556c → f40976c9a3d18bd2601366ae48cd42ebf257e750 

comment:417 Changed 14 months ago by
Description:  modified (diff) 

Summary:  Inplace (editable) installs of sagelib in a venv → Use pipenv to create venv for development 
Rename because the ticket is now more about pipenv than editable installs.
comment:418 followup: 421 Changed 14 months ago by
Replying to ghtobiasdiez:
Replying to mkoeppe:
Also the documentation on Pipenv added in
src/doc/en/developer/coding_basics.rst
is not sufficiently specific about the steps needed before pipenv is installed and used.What do you mean concretely?
I'd say start by integrating the ticket description (which has some step by step instructions) into this section of the documentation. (Perhaps best to shorten the ticket description, referring to the added documentation.)
Please also note that the documentation about venv should be improved in general, see #31342.
Yes, that's true
comment:419 Changed 14 months ago by
Replying to ghtobiasdiez:
... can be easily updated by running
pipenv lock
so it's not a lot of overhead. Please also note that it doesn't need to be updated every time a sage dependency is updated since pipenv uses theinstall_requires
information to determine the admissible versions. I would suggest to runpipenv lock
after every release for now and see how this goes.
Could you add the precise process to the documentation please? This would be a task for the release manager, who would need to be briefed about this in detail
comment:420 Changed 14 months ago by
Replying to ghtobiasdiez:
Replying to mkoeppe:
I also don't understand the other environment exports in the workflow script  why not invoke the
sageenv
script that sets up the correct environment?I was not aware of the sageenv script. How does one uses it to set the environment for the following steps and replace the manual exports I've added? I couldn't find any documentation about that script.
The top of the file (src/bin/sageenv
) explains how to use it
comment:421 Changed 14 months ago by
Replying to mkoeppe:
I'd say start by integrating the ticket description (which has some step by step instructions) into this section of the documentation. (Perhaps best to shorten the ticket description, referring to the added documentation.)
(However, the tox e localsudostandard buildlocal
shouldn't go into the documentation because we do not want to advertise this as a build/provisioning tool; it is strictly meant for portability testing.)
comment:422 Changed 14 months ago by
Commit:  f40976c9a3d18bd2601366ae48cd42ebf257e750 → cf832ca3f0132f2b87bf4c4d1c49ccd17a3dc80d 

Branch pushed to git repo; I updated commit sha1. New commits:
cf832ca  Try to find out why r doesn't work on mac

comment:423 Changed 14 months ago by
Commit:  cf832ca3f0132f2b87bf4c4d1c49ccd17a3dc80d → 605311ed1b3603b204b71c96cdd5cf8f81b54904 

Branch pushed to git repo; I updated commit sha1. New commits:
605311e  Try to fix r again

comment:424 Changed 14 months ago by
Commit:  605311ed1b3603b204b71c96cdd5cf8f81b54904 → ae26a5f3549413bdc877ac325b52f4faee31b12f 

comment:425 Changed 14 months ago by
Commit:  ae26a5f3549413bdc877ac325b52f4faee31b12f → 4b7ca86998ec6cf2122101e9050861865ee7da26 

Branch pushed to git repo; I updated commit sha1. New commits:
4b7ca86  Remove sage share export

comment:426 Changed 14 months ago by
Commit:  4b7ca86998ec6cf2122101e9050861865ee7da26 → 853ee59214497b4173de2b13050026676bf85279 

comment:427 followup: 443 Changed 14 months ago by
Thanks for your comments. I think I've addressed all of them now.
Only . bin/sageenv
doesn't work for me. Afterwards, the pipenv
command throws the error that the pipenv module cannot be found. I guess the reason is that pipenv is not installed in the sage python environment. Since this would also only replace the export xyz
commands in the github workflow, I propose we do this in a followup ticket.
comment:428 Changed 14 months ago by
Commit:  853ee59214497b4173de2b13050026676bf85279 → f2ce6a0fcd9cdb38d2202263dc22e0924549596e 

Branch pushed to git repo; I updated commit sha1. New commits:
f2ce6a0  Lets ask R about its home

comment:429 Changed 14 months ago by
Description:  modified (diff) 

comment:430 Changed 14 months ago by
Replying to mkoeppe:
+ export SAGE_SHARE=$GITHUB_WORKSPACE/.tox/localhomebrewmacosusrlocal/local/shareThis line (in both workflow scripts) shouldn't be needed.
Without it a lot of tests fail, for example because Cremona cannot be found. See https://github.com/sagemath/sagetracmirror/runs/3893523414?check_suite_focus=true. I've thus readded the sage share export.
comment:431 Changed 14 months ago by
Commit:  f2ce6a0fcd9cdb38d2202263dc22e0924549596e → ae3eae7f5764227f1ffffb0105839cb8d562ac63 

Branch pushed to git repo; I updated commit sha1. New commits:
ae3eae7  Readd sage_share export

comment:433 Changed 14 months ago by
Might this be that in the github action the installed sageconf doesn't refer to the tox environment but to the one in /local
?
comment:434 Changed 14 months ago by
I don't know, but the idea of sageconf
is that you install a version that provides the correct setting of SAGE_LOCAL
comment:435 Changed 14 months ago by
Problem is that the path of sageconf needs to be specified in the pipfile and thus has to be known before. Maybe in the future one can add a symlink (defaulting to pkgs/sageconf) that can be changed to point to a tox environment? But I guess this would be more or less equivalent to changing SAGE_SHARE.
comment:436 followup: 439 Changed 14 months ago by
The path in the source tree is always pkgs/sageconf/
comment:437 Changed 14 months ago by
(The local
.tox builds do not make a copy of SAGE_ROOT. They only use a separate SAGE_LOCAL.)
comment:438 Changed 14 months ago by
Commit:  ae3eae7f5764227f1ffffb0105839cb8d562ac63 → e41d022821baf31c99bfdb3aaff60a89eafe6c07 

Branch pushed to git repo; I updated commit sha1. New commits:
e41d022  Found R's home, so lets go there tonight

comment:439 followup: 442 Changed 14 months ago by
Replying to mkoeppe:
The path in the source tree is always
pkgs/sageconf/
Okay, then I don't know what the problem is. I would propose to keep the explicit SAGE_SHARE for now and then afterwards create a new ticket in which it is removed. The github workflow right now is anyway a big compromise of "how it should work" vs "what is actually needed to make it work". There are quite a few special tweaks and workarounds that ideally shouldn't be needed.
comment:440 Changed 14 months ago by
Commit:  e41d022821baf31c99bfdb3aaff60a89eafe6c07 → 80aa6bb1ad4d71f63ad6bc33644e0eb5c2d2e25b 

comment:441 Changed 14 months ago by
Commit:  80aa6bb1ad4d71f63ad6bc33644e0eb5c2d2e25b → 0fcb85665cee14abd3492ba02c4b53b246d40b2c 

Branch pushed to git repo; I updated commit sha1. New commits:
0fcb856  Try to fix path to gmp (and boost) on macos

comment:442 Changed 14 months ago by
Replying to ghtobiasdiez:
Replying to mkoeppe:
The path in the source tree is always
pkgs/sageconf/
Okay, then I don't know what the problem is.
To debug this, in the activated venv you could ask sageconfig
to print the configuration
comment:443 followup: 452 Changed 14 months ago by
Replying to ghtobiasdiez:
Only
. bin/sageenv
doesn't work for me.
Take a look at what bin/sage
does. It first sources config files on which sageenv
relies.
comment:444 followup: 455 Changed 14 months ago by
Replying to ghtobiasdiez:
some of the packages (pplpy for example) need cython and other packages installed already on sdist and egg_info.
This should be fixed. Could you open issues please with these packages? We also have metaticket #30922 for keeping track of these issues.
comment:445 followup: 456 Changed 14 months ago by
Replying to ghtobiasdiez:
The point of the lock file is also to ensure that the python environment is the same for everyone (which makes it easier to reproduce issues and test failures etc)
I understand that for general Python projects it has the benefit that you describe (although the benefit over plain requirements.txt
is still not clear to me, but let's not elaborate on this now).
The problem is that in our case, adding the lock file locks versions that (1) are *not the same* as the ones in the Sage distribution, (2) it will be out of date next week because we have no established process to update it. So it reduces clarity for users and creates a maintenance burden for Sage development.
I think it would be a good idea to work on a process (= script/documentation) based on pipenv that actually helps Sage developers to make Python package upgrades. Currently this is a manual process based on guesswork and testing. So this could be a key added value that Pipenv provides.
comment:446 Changed 14 months ago by
Replying to ghtobiasdiez:
Moreover, the lock file is required for now as some of the packages (pplpy for example) need cython and other packages installed already on sdist and egg_info.
Actually pplpy declares these build dependencies correctly in https://gitlab.com/videlec/pplpy//blob/master/pyproject.toml
comment:447 Changed 14 months ago by
Commit:  0fcb85665cee14abd3492ba02c4b53b246d40b2c → 5e60fc2ef9c0ab3db0e3431519dd551dc49b6321 

Branch pushed to git repo; I updated commit sha1. New commits:
5e60fc2  Lets try sageenv

comment:448 Changed 14 months ago by
Commit:  5e60fc2ef9c0ab3db0e3431519dd551dc49b6321 → fe3940955e017f7445a7aae866d750841dca5db7 

Branch pushed to git repo; I updated commit sha1. New commits:
fe39409  sageenv thinks its time for 127, we don't agree

comment:450 Changed 14 months ago by
Commit:  fe3940955e017f7445a7aae866d750841dca5db7 → 91310ce568a590edb59b20251107e67e21981b3a 

Branch pushed to git repo; I updated commit sha1. New commits:
91310ce  Force success exit code...strange

comment:451 Changed 14 months ago by
Commit:  91310ce568a590edb59b20251107e67e21981b3a → 28faabd70ec932d4b7f2b92681b2f202cb2e9f29 

Branch pushed to git repo; I updated commit sha1. New commits:
28faabd  Lets try code from sage script

comment:452 Changed 14 months ago by
Replying to mkoeppe:
Replying to ghtobiasdiez:
Only
. bin/sageenv
doesn't work for me.Take a look at what
bin/sage
does. It first sources config files on whichsageenv
relies.
I was not able to make it work locally nor on github (where it fails with exit code 127 without giving any details). That's mostly because of my missing experience with bash and linux things. Also sageenv does a lot of things that are not needed at this point (e.g. overwriting the python location, which breaks pipenv). Anyway, lets make this a followup ticket.
comment:453 Changed 14 months ago by
Description:  modified (diff) 

comment:454 followup: 457 Changed 14 months ago by
Commit:  28faabd70ec932d4b7f2b92681b2f202cb2e9f29 → d4d8236d91cf864744d7a7dee4710732b10a2b72 

Branch pushed to git repo; I updated commit sha1. New commits:
d4d8236  I give up on sageenv...

comment:455 Changed 14 months ago by
Replying to mkoeppe:
Replying to ghtobiasdiez:
some of the packages (pplpy for example) need cython and other packages installed already on sdist and egg_info.
This should be fixed. Could you open issues please with these packages? We also have metaticket #30922 for keeping track of these issues.
comment:456 followup: 458 Changed 14 months ago by
Replying to mkoeppe:
The problem is that in our case, adding the lock file locks versions that (1) are *not the same* as the ones in the Sage distribution,
Maybe a shift in viewpoint might be helpful to see how pipenv helps. Sagelib (things in src
) declares its dependencies in form of install_requires
. Currently, sagethedistrubition is one way to provide a certain set of packages that satisfy the version constraints. The newly added pipfile.lock specifies another set of packages that also satisfy the constraints. If sagelib is indeed configured correctly, then it should work with both sets. So you can see the added github workflow as a test that the install_requires constraints are correct.
(2) it will be out of date next week because we have no established process to update it. So it reduces clarity for users and creates a maintenance burden for Sage development.
dependabot or rennovate are the modern approaches to not worry about manually updating dependencies. But they don't support trac... But updating the pipfile lock manually is easy, automated and only needs to be done rarely. Its definitively more convenient then updating the versions in sagethedistro.
I think it would be a good idea to work on a process (= script/documentation) based on pipenv that actually helps Sage developers to make Python package upgrades. Currently this is a manual process based on guesswork and testing. So this could be a key added value that Pipenv provides.
I agree. First of all, my idea with pipenv was to completely replace sages python packages and use it also for distribution. But also in the short term, pipenv can be helpful. For example, the following are the versions determined by pipenv:
alabaster 0.7.12 argon2cffi 21.1.0 attrs 21.2.0 Babel 2.9.1 backcall 0.2.0 backports.zoneinfo 0.2.1 bleach 4.1.0 certifi 2021.10.8 cffi 1.15.0 charsetnormalizer 2.0.7 cycler 0.10.0 cypari2 2.1.2 cysignals 1.10.3 Cython 0.29.24 debugpy 1.5.0 decorator 5.1.0 defusedxml 0.7.1 docutils 0.17.1 entrypoints 0.3 fpylll 0.5.6 gmpy2 2.1.0b5 idna 3.3 imagesize 1.2.0 iniconfig 1.1.1 ipykernel 6.4.1 ipython 7.28.0 ipythongenutils 0.2.0 ipywidgets 7.6.5 jedi 0.18.0 Jinja2 3.0.2 jsonschema 4.1.0 jupyterclient 7.0.6 jupytercore 4.8.1 jupyterlabpygments 0.1.2 jupyterlabwidgets 1.0.2 kiwisolver 1.3.2 MarkupSafe 2.0.1 matplotlib 3.4.3 matplotlibinline 0.1.3 memoryallocator 0.1.1 mistune 0.8.4 mpmath 1.2.1 nbclient 0.5.4 nbconvert 6.2.0 nbformat 5.1.3 nestasyncio 1.5.1 networkx 2.6.3 notebook 6.4.4 numpy 1.21.2 packaging 21.0 pandocfilters 1.5.0 parso 0.8.2 pexpect 4.8.0 pickleshare 0.7.5 Pillow 8.3.2 pip 21.2.4 pkgconfig 1.5.5 pluggy 1.0.0 pplpy 0.8.7 prometheusclient 0.11.0 prompttoolkit 3.0.20 psutil 5.8.0 ptyprocess 0.7.0 py 1.10.0 pycparser 2.20 Pygments 2.10.0 pyparsing 2.4.7 pyrsistent 0.18.0 pytest 6.2.5 pythondateutil 2.8.2 pytz 2021.3 pyzmq 22.3.0 requests 2.26.0 rpy2 3.3.6 sageconf 9.5b3 /__w/sagetracmirror/sagetracmirror/pkgs/sageconf sagemathstandard 9.5b3 /__w/sagetracmirror/sagetracmirror/src scipy 1.7.1 Send2Trash 1.8.0 setuptools 58.1.0 six 1.16.0 snowballstemmer 2.1.0 Sphinx 4.1.2 sphinxcontribapplehelp 1.0.2 sphinxcontribdevhelp 1.0.2 sphinxcontribhtmlhelp 2.0.0 sphinxcontribjsmath 1.0.1 sphinxcontribqthelp 1.0.3 sphinxcontribserializinghtml 1.1.5 sympy 1.9 terminado 0.12.1 testpath 0.5.0 toml 0.10.2 tornado 6.1 traitlets 5.1.0 tzlocal 3.0 urllib3 1.26.7 wcwidth 0.2.5 webencodings 0.5.1 wheel 0.37.0 widgetsnbextension 3.5.1
Now you can use those versions as the package version of sagethedistro. Lets discuss this further in #31346.
comment:457 followup: 460 Changed 14 months ago by
Replying to git:
Branch pushed to git repo; I updated commit sha1. New commits:
d4d8236 I give up on sageenv...
You forgot to use sageenvconfig etc. as per comment:443
comment:458 followup: 462 Changed 14 months ago by
Replying to ghtobiasdiez:
Replying to mkoeppe:
The problem is that in our case, adding the lock file locks versions that (1) are *not the same* as the ones in the Sage distribution,
Maybe a shift in viewpoint might be helpful to see how pipenv helps. Sagelib (things in
src
) declares its dependencies in form ofinstall_requires
. Currently, sagethedistrubition is one way to provide a certain set of packages that satisfy the version constraints. The newly added pipfile.lock specifies another set of packages that also satisfy the constraints. If sagelib is indeed configured correctly, then it should work with both sets. So you can see the added github workflow as a test that the install_requires constraints are correct.
I don't find this convincing. A checkedin Pipfile.lock
is just the result of resolving the declared version constraints (as of some point in the past) against what was available on the index at some some point in the past.
When the workflow fails, it will mean just one thing: Constraints were updated and Pipfile.lock was not updated (> maintenance burden).
It's more valuable to test on GH Actions whether the declared version constraints resolve *at the current time* to a set of versions that work correctly. If this fails, it is clear that the constraints declared in install_requires
are not correct.
comment:459 Changed 14 months ago by
By the way, see SAGE_ROOT/pkgs/sagemathstandard/tox.ini
for test environments for several ways to provision a venv. Looking at it, I am also reminded that we actually have two sets of Pipenv
comment:460 Changed 14 months ago by
Replying to mkoeppe:
Replying to git:
Branch pushed to git repo; I updated commit sha1. New commits:
d4d8236 I give up on sageenv...
You forgot to use sageenvconfig etc. as per comment:443
I tried it locally and couldn't determine which other scripts are needed to run first. To be honest, I'm tired of the try and error play (especially since it only replace a couple of very easy to understand export statements).
comment:461 followup: 464 Changed 14 months ago by
You can also just use sage sh
to run something in the environment, like it's explained in the instructions of SAGE_ROOT/pkgs/sagemathstandard/tox.ini
comment:462 Changed 14 months ago by
Replying to mkoeppe:
When the workflow fails, it will mean just one thing: Constraints were updated and Pipfile.lock was not updated (> maintenance burden). It's more valuable to test on GH Actions whether the declared version constraints resolve *at the current time* to a set of versions that work correctly. If this fails, it is clear that the constraints declared in
install_requires
are not correct.
I guess it depends on what you want to test (and when you want to run the workflow):
 Test that the changes in the actual code (
src
) don't break the doctests (i.e run on push). In this case, you want to keep the environment fixed (since otherwise the workflow may fail due to newly released versions on pypi).  Test that sage works with the newest versions of packages compatible with the install_require constraints. In this case, you probably want to run the workflow manually, before release or in certain interval. Of course, in this case one should ignore the committed pipfile.lock.
My initial motivation was the first use case (as an alternative/replacement for the patchbots), but you are right the second case is important as well. For this reason, I've now added the possibility to run the workflow in manually in "update" mode in which case the pipfile.lock is ignored. Note that the latter still needs a committed pipfile lock at the moment due to the problems described above (https://trac.sagemath.org/ticket/30922).
comment:463 Changed 14 months ago by
Commit:  d4d8236d91cf864744d7a7dee4710732b10a2b72 → c5b81335989d09a0c814bc58760b9a15174f23df 

Branch pushed to git repo; I updated commit sha1. New commits:
c5b8133  Add update mode

comment:464 Changed 14 months ago by
Replying to mkoeppe:
You can also just use
sage sh
to run something in the environment, like it's explained in the instructions ofSAGE_ROOT/pkgs/sagemathstandard/tox.ini
I tried this as well, but then other things get messed up (e.g. PATH, which Python is used etc). What I would like to have and what is needed here is a sagebuildenv
in which only the buildrelated environment variables are changed.
comment:465 Changed 14 months ago by
By the way, the "update" mode is only available after this ticket is merged due to a security constraint by github ("To trigger the workflow_dispatch event, your workflow must be in the default branch.").
comment:466 followup: 476 Changed 14 months ago by
I think all the troubles that you are having with the environment are the result of using tox to build the SAGE_LOCAL but then not activating the tox environment. A way out of this is to go through #31535
comment:467 Changed 14 months ago by
Commit:  c5b81335989d09a0c814bc58760b9a15174f23df → 8fe104855e2f23825b22b43027eeae33a6c9e593 

Branch pushed to git repo; I updated commit sha1. New commits:
8fe1048  Run in update mode (to check if its works)

comment:468 Changed 14 months ago by
Commit:  8fe104855e2f23825b22b43027eeae33a6c9e593 → 821919d2abd6309441dd4b86b752c205baa13971 

Branch pushed to git repo; I updated commit sha1. New commits:
821919d  Use same env variables for update as for initial install

comment:469 Changed 14 months ago by
Commit:  821919d2abd6309441dd4b86b752c205baa13971 → 198c6d8f88bb9e5c0fce83746b4d858b9e48bd51 

Branch pushed to git repo; I updated commit sha1. New commits:
198c6d8  One last try with sageenv

comment:470 Changed 14 months ago by
Commit:  198c6d8f88bb9e5c0fce83746b4d858b9e48bd51 → 14023b10709ce26c17ecfda0361677e63b249ff6 

comment:471 Changed 13 months ago by
Commit:  14023b10709ce26c17ecfda0361677e63b249ff6 → 4fbb8a9848f2d2f133522077663e8661c4c2eaab 

Branch pushed to git repo; I updated commit sha1. New commits:
4fbb8a9  Install git as well

comment:472 Changed 13 months ago by
Commit:  4fbb8a9848f2d2f133522077663e8661c4c2eaab → bd4684abf0709c5f19054bae3d50ef7d3cf40f29 

comment:473 Changed 13 months ago by
Commit:  bd4684abf0709c5f19054bae3d50ef7d3cf40f29 → 9b74bd621de62b289bca94cdd65bb17e48bde858 

Branch pushed to git repo; I updated commit sha1. New commits:
9b74bd6  Use deadsnakes action to setup python on linux

comment:474 Changed 13 months ago by
Commit:  9b74bd621de62b289bca94cdd65bb17e48bde858 → dd009e77906bd831f3cc73666afcbe15d9bc990d 

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
comment:475 Changed 13 months ago by
Commit:  dd009e77906bd831f3cc73666afcbe15d9bc990d → 4adc4afa6d19187699414b364748a047775f00d8 

Branch pushed to git repo; I updated commit sha1. New commits:
4adc4af  Disable update mode again

comment:476 Changed 13 months ago by
Replying to mkoeppe:
I think all the troubles that you are having with the environment are the result of using tox to build the SAGE_LOCAL but then not activating the tox environment. A way out of this is to go through #31535
It was actually something strange going on on my system. After make distclean I could run sageenv locally and then also in the github action. Sorry for the confusion and thanks for your help.
The update mode was a bit more work than I'd liked but its running now. See https://github.com/sagemath/sagetracmirror/runs/3926515272?check_suite_focus=true for an example output. It can be activated by running the workflow manually from the github interface (after it is merged into the default branch).
comment:477 Changed 13 months ago by
Commit:  4adc4afa6d19187699414b364748a047775f00d8 → 50bd3ce3eadc344b8c8067a2bf3e7e659c39d1ef 

Branch pushed to git repo; I updated commit sha1. New commits:
50bd3ce  Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace

comment:479 Changed 12 months ago by
I tried following the instructions in the ticket description, and in particular the block of commands
./bootstrap ./configure enableeditable make buildlocal cd src mkdir .venv pipenv install dev ignorepipfile pipenv run sageruntests all
The last command fails with
Error: the command sageruntests could not be found within PATH or Pipfile's [scripts].
comment:480 Changed 12 months ago by
Commit:  50bd3ce3eadc344b8c8067a2bf3e7e659c39d1ef → 148cc729c93cd164b7155a4404ec09c60d5154a6 

Branch pushed to git repo; I updated commit sha1. New commits:
148cc72  Merge branch 'develop' of https://github.com/sagemath/sage into public/build/inplace

comment:481 Changed 12 months ago by
Description:  modified (diff) 

New commits:
148cc72  Merge branch 'develop' of https://github.com/sagemath/sage into public/build/inplace

comment:482 Changed 12 months ago by
Sorry for not testing it after my last merge of the develop branch.
It turned out there is a problem (only reported by pipenv using the verbose
flag) during the installation of sage in the virtual environment. In fact, it turns out that this is not related at all to pipenv but a general issue with the installation using build isolution, so I've opened #32904 for this.
For pipenv this can be fixed by running
pipenv run pip install e . upgrade existsaction=i nobuildisolation
after the initial build of the venv. I've changed the ticket description accordingly.
comment:483 Changed 12 months ago by
Commit:  148cc729c93cd164b7155a4404ec09c60d5154a6 → 0b0437fc7a2df0bb34873e6d0d881a775e1ebb09 

Branch pushed to git repo; I updated commit sha1. New commits:
0b0437f  Add sagesetup as editable install, update lock

comment:484 Changed 12 months ago by
I don't have any opinion about the code, so I am not the right person to review this, but I did get a lot of doctest failures when following the directions in the ticket:
sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/env.py # 4 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/misc/superseded.py # 6 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/misc/cython.py # 21 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/misc/inline_fortran.py # 3 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/misc/cachefunc.pyx # 54 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/misc/session.pyx # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/misc/randstate.pyx # 25 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/misc/inherit_comparison.pyx # 5 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/misc/sagedoc.py # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/misc/nested_class.pyx # 6 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/misc/lazy_attribute.pyx # 3 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/misc/sageinspect.py # 35 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/misc/sage_eval.py # 7 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/misc/persist.pyx # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/schemes/elliptic_curves/ell_rational_field.py # 21 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/schemes/elliptic_curves/lseries_ell.py # 4 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/schemes/generic/algebraic_scheme.py # 13 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/schemes/generic/homset.py # 3 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/schemes/toric/fano_variety.py # 137 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/schemes/toric/divisor_class.pyx # 50 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/schemes/toric/points.py # 108 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/schemes/toric/library.py # 74 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/schemes/toric/toric_subscheme.py # 104 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/schemes/toric/variety.py # 229 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/schemes/toric/divisor.py # 234 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/schemes/toric/weierstrass.py # 6 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/schemes/toric/weierstrass_covering.py # 9 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/schemes/toric/chow_group.py # 159 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/schemes/toric/homset.py # 72 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/schemes/toric/morphism.py # 194 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/schemes/toric/sheaf/klyachko.py # 101 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/schemes/toric/sheaf/constructor.py # 29 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/crypto/classical.py # 16 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/parallel/decorate.py # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/plot/graphics.py # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/plot/contour_plot.py # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/plot/line.py # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/plot/arrow.py # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/plot/plot3d/tachyon.py # 24 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/plot/plot3d/index_face_set.pyx # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/plot/plot3d/platonic.py # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/plot/plot3d/base.pyx # 6 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/plot/plot3d/implicit_surface.pyx # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/plot/plot3d/implicit_plot3d.py # 6 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/plot/plot3d/parametric_plot3d.py # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/plot/plot3d/parametric_surface.pyx # 3 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/manifolds/differentiable/vectorfield.py # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/manifolds/differentiable/tangent_vector.py # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/combinat/constellation.py # 6 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/combinat/tutorial.py # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/combinat/quickref.py # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/combinat/permutation.py # 3 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/combinat/designs/ext_rep.py # Killed due to abort sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/combinat/posets/posets.py # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/combinat/root_system/weyl_group.py # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/combinat/matrices/latin.py # 45 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/lfunctions/zero_sums.pyx # 3 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/lfunctions/lcalc.py # 7 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/lfunctions/sympow.py # 10 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/ext/memory_allocator.pxd # 3 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/ext/memory_allocator.pyx # 7 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/doctest/control.py # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/doctest/forker.py # 4 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/features/__init__.py # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/tests/benchmark.py # 3 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/tests/cmdline.py # 8 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/tests/gap_packages.py # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/tests/books/judsonabstractalgebra/groupssage.py # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/tests/books/judsonabstractalgebra/cyclicsage.py # 5 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/tests/books/judsonabstractalgebra/homomorphsage.py # 11 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/tests/books/judsonabstractalgebra/sylowsage.py # 5 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/tests/books/computationalmathematicswithsagemath/combinat_doctest.py # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/algebras/orlik_terao.py # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/algebras/orlik_solomon.py # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/repl/interpreter.py # 6 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/repl/interface_magic.py # 7 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/repl/ipython_extension.py # 6 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/repl/load.py # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/groups/galois_group.py # 3 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/groups/class_function.py # 137 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/groups/generic.py # 5 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/groups/perm_gps/symgp_conjugacy_class.py # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/groups/perm_gps/permgroup_morphism.py # 13 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/groups/perm_gps/permgroup.py # 44 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/groups/perm_gps/cubegroup.py # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/groups/perm_gps/constructor.py # 4 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/groups/perm_gps/permgroup_element.pyx # 10 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/groups/perm_gps/partn_ref/data_structures.pyx # 9 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/groups/abelian_gps/dual_abelian_group_element.py # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/groups/abelian_gps/abelian_group_morphism.py # 15 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/groups/abelian_gps/abelian_group.py # 64 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/docs/instancedoc.pyx # 4 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/arith/misc.py # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/arith/long.pxd # 14 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/libs/gap/element.pyx # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/libs/gap/libgap.pyx # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/cpython/dict_del_by_value.pyx # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/cpython/wrapperdescr.pxd # 6 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/cpython/string.pyx # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/cpython/cython_metaclass.pyx # 4 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/cpython/getattr.pyx # 4 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/modular/abvar/abvar.py # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/modular/arithgroup/congroup_gamma0.py # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/modular/arithgroup/congroup_gammaH.py # 5 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/modular/arithgroup/arithgroup_perm.py # 5 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/modular/modform/constructor.py # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/modular/hecke/submodule.py # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/functions/other.py # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/functions/exp_integral.py # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/functions/generalized.py # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/functions/piecewise.py # 13 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/coding/binary_code.pyx # 3 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/coding/databases.py # 4 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/coding/linear_code.py # 22 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/coding/golay_code.py # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/coding/kasami_codes.pyx # 5 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/coding/two_weight_db.py # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/coding/code_constructions.py # 4 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/coding/linear_code_no_metric.py # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/coding/goppa_code.py # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/rings/integer_ring.pyx # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/rings/integer.pyx # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/rings/infinity.py # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/rings/integer_fake.pxd # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/rings/tate_algebra_ideal.pyx # 6 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/rings/rational_field.py # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/rings/number_field/galois_group.py # 3 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/rings/number_field/number_field.py # 26 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/rings/number_field/number_field_element.pyx # 6 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/rings/number_field/totallyreal_rel.py # 7 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/rings/polynomial/polynomial_ring.py # 5 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/rings/polynomial/polynomial_element.pyx # 8 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/rings/polynomial/ore_polynomial_element.pyx # 6 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/rings/polynomial/multi_polynomial.pyx # 8 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/rings/polynomial/multi_polynomial_ring_base.pyx # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/rings/polynomial/multi_polynomial_ideal.py # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/rings/polynomial/groebner_fan.py # 118 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/rings/finite_rings/galois_group.py # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/rings/finite_rings/element_pari_ffelt.pyx # 11 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/rings/finite_rings/element_givaro.pyx # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/rings/finite_rings/integer_mod_ring.py # 6 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/rings/finite_rings/integer_mod.pyx # 4 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/rings/finite_rings/finite_field_givaro.py # 7 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/graphs/connectivity.pyx # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/graphs/generic_graph.py # 5 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/graphs/graph_decompositions/fast_digraph.pyx # 5 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/symbolic/expression.pyx # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/symbolic/constants.py # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/symbolic/pynac.pxi # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/symbolic/integration/integral.py # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/symbolic/integration/external.py # 6 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/geometry/cone_catalog.py # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/geometry/fan_morphism.py # 92 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/geometry/fan.py # 85 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/geometry/toric_plotter.py # 12 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/geometry/lattice_polytope.py # 196 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/geometry/cone.py # 17 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/geometry/fan_isomorphism.py # 13 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/geometry/polyhedron/ppl_lattice_polytope.py # 3 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/geometry/polyhedron/base_ZZ.py # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/geometry/polyhedron/backend_normaliz.py # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/geometry/polyhedron/palp_database.py # 23 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/geometry/polyhedron/base.py # 7 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/geometry/triangulation/element.py # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/calculus/calculus.py # 13 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/modules/free_module_element.pyx # 3 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/modules/filtered_vector_space.py # 5 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/modules/with_basis/invariant.py # 74 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/modules/with_basis/representation.py # 5 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/structure/parent_old.pyx # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/structure/element.pyx # 37 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/structure/factory.pyx # 8 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/structure/element.pxd # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/structure/parent.pyx # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/categories/primer.py # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/categories/finite_dimensional_modules_with_basis.py # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/categories/pushout.py # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/categories/modules_with_basis.py # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/matrix/matrix1.pyx # 13 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/interfaces/tachyon.py # 3 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/interfaces/expect.py # 49 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/interfaces/mwrank.py # 9 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/interfaces/ecm.py # 12 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/interfaces/qsieve.py # 7 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/interfaces/interface.py # 31 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/interfaces/gap3.py # 3 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/interfaces/gap.py # 143 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/interfaces/tests.py # 3 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage/interfaces/giac.py # 140 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage_setup/find.py # 3 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 sage_docbuild/__init__.py # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 doc/ja/tutorial/tour_groups.rst # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 doc/ja/tutorial/tour_advanced.rst # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 doc/ja/tutorial/interfaces.rst # 5 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 doc/ru/tutorial/tour_groups.rst # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 doc/ru/tutorial/tour_advanced.rst # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 doc/ru/tutorial/interfaces.rst # 5 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 doc/pt/tutorial/tour_groups.rst # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 doc/pt/tutorial/tour_advanced.rst # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 doc/pt/tutorial/interfaces.rst # 5 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 doc/de/tutorial/tour_groups.rst # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 doc/de/tutorial/tour_advanced.rst # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 doc/de/tutorial/interfaces.rst # 5 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 doc/fr/tutorial/tour_groups.rst # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 doc/fr/tutorial/tour_advanced.rst # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 doc/fr/tutorial/interfaces.rst # 5 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 doc/es/tutorial/tour_groups.rst # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 doc/en/developer/coding_in_other.rst # 3 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 doc/en/tutorial/tour_groups.rst # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 doc/en/tutorial/tour_advanced.rst # 2 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 doc/en/tutorial/interfaces.rst # 5 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 doc/en/constructions/rep_theory.rst # 18 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 doc/en/constructions/polynomials.rst # 13 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 doc/en/constructions/rings.rst # 6 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 doc/en/constructions/linear_algebra.rst # 3 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 doc/en/constructions/groups.rst # 18 doctests failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 doc/en/thematic_tutorials/coding_theory.rst # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 doc/en/thematic_tutorials/explicit_methods_in_number_theory/elliptic_curves.rst # 1 doctest failed sage t warnlong 64.4 randomseed=60140155690716510157027757823734981182 doc/en/thematic_tutorials/lie/weyl_groups.rst # 1 doctest failed
Some come from
pkgconfig.pkgconfig.PackageNotFoundError: fflasffpack not found
and I also see some instances of errors like
File "doc/es/tutorial/tour_groups.rst", line 29, in doc.es.tutorial.tour_groups Failed example: G.random_element() # random output (resultado aleatorio) Exception raised: Traceback (most recent call last): File "/Users/jpalmier/Desktop/Sage/sage_builds/TESTING/sage9.5.beta6/src/sage/interfaces/gap.py", line 1155, in _start raise OSError("GAP workspace too old") OSError: GAP workspace too old During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/jpalmier/Desktop/Sage/sage_builds/TESTING/sage9.5.beta6/src/sage/interfaces/expect.py", line 492, in _start self._expect = SageSpawn(cmd, File "sage/interfaces/sagespawn.pyx", line 65, in sage.interfaces.sagespawn.SageSpawn.__init__ with ContainChildren(silent=True): File "sage/interfaces/sagespawn.pyx", line 66, in sage.interfaces.sagespawn.SageSpawn.__init__ spawn.__init__(self, *args, **kwds) File "/Users/jpalmier/Desktop/Sage/sage_builds/TESTING/sage9.5.beta6/src/.venv/lib/python3.9/sitepackages/pexpect/pty_spawn.py", line 205, in __init__ self._spawn(command, args, preexec_fn, dimensions) File "/Users/jpalmier/Desktop/Sage/sage_builds/TESTING/sage9.5.beta6/src/.venv/lib/python3.9/sitepackages/pexpect/pty_spawn.py", line 276, in _spawn raise ExceptionPexpect('The command was not found or was not ' + pexpect.exceptions.ExceptionPexpect: The command was not found or was not executable: gap. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/jpalmier/Desktop/Sage/sage_builds/TESTING/sage9.5.beta6/src/sage/interfaces/gap.py", line 654, in _eval_line self._start() File "/Users/jpalmier/Desktop/Sage/sage_builds/TESTING/sage9.5.beta6/src/sage/interfaces/gap.py", line 1166, in _start Expect._start(self, "Failed to start GAP.") File "/Users/jpalmier/Desktop/Sage/sage_builds/TESTING/sage9.5.beta6/src/sage/interfaces/expect.py", line 503, in _start raise RuntimeError("unable to start %s because the command %r failed: %s\n%s" % RuntimeError: unable to start gap because the command 'gap r b p T E m 64m /Users/jpalmier/Desktop/Sage/sage_builds/TESTING/sage9.5.beta6/src/sage/ext_data/gap/sage.g' failed: The command was not found or was not executable: gap.
I haven't looked at all of the failures to see what else is there.
comment:485 Changed 12 months ago by
Thanks for testing!
Some failing tests are to be expected since pipenv is rather different to the existing way to use sage (as part of sagethedistribution). But I think for the modularization effort this ticket gives a helpful testcase how sage performs in a "pipinstallable" environment.
I also had problems with gap (for the github action) and fixed it by manually setting the SAEG_LOCAL env variable. I guess there is potential for improvement as followup tickets but I'm not the right person to work on this as I'm not familiar with these parts of sage.
comment:486 Changed 11 months ago by
Milestone:  sage9.5 → sage9.6 

Stalled in needs_review
or needs_info
; likely won't make it into Sage 9.5.
comment:487 Changed 11 months ago by
Commit:  0b0437fc7a2df0bb34873e6d0d881a775e1ebb09 → c736e0cbccc8ddb9cc8359a48094aa3181b881fa 

Branch pushed to git repo; I updated commit sha1. New commits:
c736e0c  Merge branch 'develop' of https://github.com/sagemath/sage into public/build/inplace

comment:488 Changed 8 months ago by
Milestone:  sage9.6 → sage9.7 

comment:489 Changed 2 months ago by
Milestone:  sage9.7 → sage9.8 

I've worked a bit on it. The current commit is a semiworking prototype, in the sense that all cython files are successfully compiled to C files if invoked via
python src/setup.py build_ext inplace
. The second compilation of the c files does not yet work.I had problems that none of the source cython files were correctly found on my system (wsl) and that a few dependencies in the current build files were not met (
'linbox', 'Singular', 'm4ri', 'zlib', 'cblas'
). For this reason, I started from scratch and only included what was needed so far. Any pointers to how to install these dependencies (I think they are needed for the second compilation).New commits:
Semiworking prototype for inplace compilation