Opened 3 years ago

Closed 2 years ago

Last modified 14 months ago

#29105 closed enhancement (fixed)

tox.ini, build/bin/write-dockerfile.sh: Add gentoo linux, add more gentoo packages

Reported by: Matthias Köppe Owned by:
Priority: major Milestone: sage-9.2
Component: porting Keywords:
Cc: Michael Orlitzky, François Bissey, Dima Pasechnik, Andrew Merged in:
Authors: Andrey Belgorodski Reviewers: Matthias Koeppe
Report Upstream: N/A Work issues:
Branch: 9959ee9 (Commits, GitHub, GitLab) Commit:
Dependencies: #29273 Stopgaps:

Status badges

Description (last modified by Matthias Köppe)

As a followup to #29053 and #29273, we

To test: tox -e docker-gentoo-standard

This is a part of:

  • #29060: Meta-ticket: Add Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages

Resources:

Attachments (6)

gentoo-log-2.txt (2.1 MB) - added by Matthias Köppe 3 years ago.
Dockerfile-sage_gentoo (3.0 KB) - added by Matthias Köppe 3 years ago.
docker-gentoo-installed.txt (9.5 KB) - added by Andrew 2 years ago.
docker-gentoo-installed.txt
tox.ini.patch (718 bytes) - added by Andrew 2 years ago.
write-dockerfile.sh.patch (759 bytes) - added by Andrew 2 years ago.
Dockerfile_gentoo-20.06-gcc9-py37 (2.0 KB) - added by Andrew 2 years ago.

Change History (114)

comment:1 Changed 3 years ago by Matthias Köppe

Description: modified (diff)

Creating a dockerfile for setting up sage prerequisites on a Gentoo "prefix" was previously discussed in #28905.

An attempt (incomplete) is at https://trac.sagemath.org/attachment/ticket/28905/Dockerfile-sage_gentoo and would need help from Gentoo experts.

comment:2 Changed 3 years ago by François Bissey

compiler can't create executables in a configure phase of gcc typically means the compiler can't find libgmp and al. Do you have a log for me to review? Prefix on ubuntu is definitely supported. The only thing I can think of is that the bootstrap usually object being run as root.

comment:3 Changed 3 years ago by Matthias Köppe

Thanks, yes, turns out this was not specific to the gentoo prefix installer. I had to add some more packages. Let me see how far I get with this.

Changed 3 years ago by Matthias Köppe

Attachment: gentoo-log-2.txt added

Changed 3 years ago by Matthias Köppe

Attachment: Dockerfile-sage_gentoo added

comment:4 Changed 3 years ago by Matthias Köppe

OK... this is where I got now. The ./bootstrap-prefix.sh script was claiming to bootstrap some packages - such as xz and patch - and succeeded with that; but then it was missing "xz" to unpack the Python sources. I think something is wrong there, but I added xz&patch in ubuntu. This is attached Dockerfile-sage_gentoo and log from 'docker build' with that.

As you can see in the log, now it's complaining about a module missing in the Python that it just built:

  File "/gentoo/var/tmp/python-3.6.10/Python-3.6.10/Lib/xml/parsers/expat.py", line 4, in <module>
    from pyexpat import *
ModuleNotFoundError: No module named 'pyexpat'
make: *** [install] Error 1
Makefile:1102: recipe for target 'install' failed
??? Python failed to install *sigh* continuing anyway
* Python bootstrapped
http://distfiles.gentoo.org/snapshots

and then dies with:

* Compiling prefix-portage
./configure --host=x86_64-pc-linux-gnu --prefix=/gentoo/tmp/usr --mandir=/gentoo/tmp/usr/share/man --infodir=/gentoo/tmp/usr/share/info --datadir=/gentoo/tmp/usr/share --sysconfdir=/gentoo/tmp/etc --localstatedir=/gentoo/tmp/var/lib --build=x86_64-pc-linux-gnu --with-offset-prefix=/gentoo/tmp --with-portage-user=root --with-portage-group=root --with-extra-path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking target system type... x86_64-pc-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... no
checking for mawk... mawk
....
checking whether /usr/bin/id is good enough... yes
checking for python... no
configure: error: no python found in your path

comment:5 Changed 3 years ago by François Bissey

xz is rearing up its head again. Yes it should be an early one. Because many more sources are in .xz I am not shocked that it should be a pre-requisite along with patch. The pyexpat stuff is weird. I'll do some experimenting later. I haven't really touched prefix since ppc64 was retired (because I didn't want or could maintain it long term).

comment:6 Changed 3 years ago by Matthias Köppe

Description: modified (diff)

comment:7 Changed 3 years ago by Matthias Köppe

Cc: Dima Pasechnik added

comment:8 Changed 3 years ago by Matthias Köppe

Dependencies: #29273
Description: modified (diff)
Summary: Add gentoo package information, extend dockerfile generator to gentooExtend dockerfile generator to gentoo

comment:9 Changed 3 years ago by Matthias Köppe

This tickets needs help from someone who actually uses Gentoo. I found the instructions for installing stuff inadequate.

comment:10 Changed 3 years ago by Dima Pasechnik

I use Gentoo, but neither prefix, not docker.

comment:11 Changed 3 years ago by François Bissey

I need to make time for this. I cannot give you a time frame.

comment:12 in reply to:  10 ; Changed 3 years ago by Matthias Köppe

Replying to dimpase:

I use Gentoo, but [...] not docker.

After installing a docker client, it only takes

docker run -it gentoo/stage3-amd64

This gives you root prompt. I would need instructions on what to do next to install relevant gentoo packages here.

comment:13 Changed 3 years ago by François Bissey

If I understand well you have a minimal gentoo image running now. What do you want to install? Do you have a list somewhere?

comment:14 in reply to:  13 Changed 3 years ago by Matthias Köppe

Replying to fbissey:

If I understand well you have a minimal gentoo image running now. What do you want to install? Do you have a list somewhere?

The equivalent of the lists in build/pkgs/debian-bootstrap.txt and build/pkgs/debian.txt (minimal requirements for bootstrapping and building) -- which would be added in #29273 as well since gentoo packages are being added there.

Last edited 3 years ago by Matthias Köppe (previous) (diff)

comment:15 in reply to:  12 Changed 3 years ago by Dima Pasechnik

Replying to mkoeppe:

Replying to dimpase:

I use Gentoo, but [...] not docker.

After installing a docker client, it only takes

docker run -it gentoo/stage3-amd64

This gives you root prompt. I would need instructions on what to do next to install relevant gentoo packages here.

I can try either this or Gentoo in lxc container, if such a thing exists somewhere.

comment:16 Changed 3 years ago by Matthias Köppe

Docker please because that's what my tox.ini infrastructure uses. If we want to do the same with lxc instead of docker, why not, but perhaps one step at a time?

comment:17 Changed 3 years ago by Matthias Köppe

I have created ticket #29283 for the goal of lxc containers for testing, in case someone wants to work on that.

comment:18 Changed 3 years ago by Matthias Köppe

Summary: Extend dockerfile generator to gentooExtend dockerfile generator to gentoo, add more gentoo packages

comment:19 Changed 3 years ago by Matthias Köppe

Milestone: sage-9.1sage-9.2

comment:20 Changed 2 years ago by Matthias Köppe

Cc: Andrew added

comment:22 Changed 2 years ago by Andrew

When I run tox -e docker-gentoo I get half-baked sage/.tox/docker-gentoo/Dockerfile, I need a way to shove the hand-crafted Dockerfile instead (for debugging purposes). What is the best way to do it?

comment:23 Changed 2 years ago by Matthias Köppe

Does your handcrafted Dockerfile work with docker build already?

comment:24 Changed 2 years ago by Matthias Köppe

You could insert a line before ` # From https://hub.docker.com/r/multiarch/ubuntu-core/ ` that says

    docker-gentoo:  bash -c 'whatever command to create a handcrafted {envdir}/Dockerfile'    

comment:25 Changed 2 years ago by Andrew

So right after

docker:        bash -c 'build/bin/write-dockerfile.sh  ... > {envdir}/Dockerfile'

Thank you!

Changed 2 years ago by Andrew

Attachment: docker-gentoo-installed.txt added

docker-gentoo-installed.txt

comment:26 Changed 2 years ago by Andrew

The work is still in progress. I managed to install 555 packages.

comment:27 Changed 2 years ago by Matthias Köppe

Nice

comment:28 Changed 2 years ago by Andrew

real    14m14.888s
user    98m59.353s
sys     3m4.286s
SAGE_CHECK=warn, so scanning the log files. This may take a few seconds.
Sage build/upgrade complete!
make[1]: Leaving directory '/sage'
Removing intermediate container e0e5bbfb22ea
 ---> 14316b24894b
[Warning] One or more build-args [TARGETS_OPTIONAL] were not consumed
Successfully built 14316b24894b
Successfully tagged sage-docker-gentoo-standard-with-targets:9.2.beta1-dirty
_________________________ summary __________________________
  docker-gentoo-standard: commands succeeded
  congratulations :)

1) run docker build -t fbissey/sage-on-gentoo-stage4:20.6 -t fbissey/sage-on-gentoo-stage4:latest -f Dockerfile_gentoo-20.06-gcc9-py37 .

2) patch tox.ini and write-dockerfile.sh

Changed 2 years ago by Andrew

Attachment: tox.ini.patch added

Changed 2 years ago by Andrew

Attachment: write-dockerfile.sh.patch added

Changed 2 years ago by Andrew

comment:29 Changed 2 years ago by Matthias Köppe

comment:30 in reply to:  29 Changed 2 years ago by Andrew

Replying to mkoeppe:

https://hub.docker.com/r/fbissey/sage-on-gentoo-stage/tags ... this does not seem to exist yet?

It does not exist. Someone have to decide where to store an image. I have no idea, my docker-fu is weak.

comment:31 Changed 2 years ago by Matthias Köppe

Why not create an account on docker hub? Next step: https://docs.docker.com/docker-hub/builds/

comment:32 Changed 2 years ago by Andrew

I am working on Dockerfile_gentoo-20.06-gcc9-py37​ to make image size much much smaller :(

comment:33 Changed 2 years ago by Andrew

docker pull sheerluck/sage-on-gentoo-stage4:latest

comment:34 Changed 2 years ago by Andrew

$ docker run -it sheerluck/sage-on-gentoo-stage4 /bin/bash
cf6f39e01c4a / # ipython --no-banner

In [1]: from os import walk
   ...: pth, ext = "/var/db/pkg", ".ebuild"
   ...: lex, res = len(ext), []
   ...: for _, _, u in walk(pth):
   ...:     res += [x[:-lex] for x in u if x.endswith(ext)]
   ...: for  pkg  in sorted(res): print(pkg)
Alnuth-3.1.1
App-pwhich-1.150.0
Authen-SASL-2.160.0-r1
...
zn_poly-0.9.2
zope-interface-4.7.1
zstd-1.4.4-r4
zziplib-0.13.69-r1

comment:35 Changed 2 years ago by Matthias Köppe

Awesome!

comment:36 Changed 2 years ago by Matthias Köppe

Could you put the changes to the sage scripts on a git branch please?

comment:37 Changed 2 years ago by Andrew

$ git push trac HEAD:u/gh-sheerluck/gentoo
...
To trac.sagemath.org:sage.git
 * [new branch]            HEAD -> u/gh-sheerluck/gentoo

comment:38 Changed 2 years ago by Matthias Köppe

Branch: u/gh-sheerluck/gentoo
Commit: 25d05792b7d44103de445e2e591393f3e5ef3862

New commits:

25d0579write-dockerfile.sh is extended to support gentoo

comment:39 Changed 2 years ago by Matthias Köppe

sed: ./build/pkgs/gentoo.txt: No such file or directory
sed: ./build/pkgs/gentoo-bootstrap.txt: No such file or directory

comment:40 Changed 2 years ago by Matthias Köppe

Also, at the end of configure, I see:

configure: notice: the following SPKGs did not find equivalent system packages: cbc eclib gp2c isl lcalc libsemigroups pandoc pari pari_elldata pari_galdata pari_galpol pari_nftables pari_seadata pari_seadata_small perl_cpan_polymake_prereq perl_term_readline_gnu r yasm
checking for the package system in use... gentoo
configure: hint: installing the following system packages is recommended and may avoid building some of the above SPKGs from source:
configure: # install the following packages: sci-mathematics/eclib[flint] sci-mathematics/gp2c dev-libs/isl app-text/pandoc sci-mathematics/pari sci-mathematics/pari-data XML-Writer XML-LibXML XML-LibXSLT File-Slurp dev-perl/Term-ReadLine-Gnu JSON SVG dev-perl/MongoDB dev-perl/Term-ReadLine-Gnu dev-lang/R

Could you update build/bin/sage-print-system-package-command so that an actual installation command is displayed?

comment:41 Changed 2 years ago by Andrew

I added gentoo.txt and gentoo-bootstrap.txt and updated sage-print-system-package-command and I am going to refresh sheerluck/sage-on-gentoo-stage4:latest with additional packages (except pandoc) and after testing I will push in u/gh-sheerluck/gentoo branch

comment:42 Changed 2 years ago by git

Commit: 25d05792b7d44103de445e2e591393f3e5ef3862c55cfa9bd2d0d29df3cd5cc24be5b743eee8aadf

Branch pushed to git repo; I updated commit sha1. New commits:

c55cfa9sage-print-system-package-command updated

comment:43 Changed 2 years ago by Andrew

Now configure says "configure: $ sudo emerge sci-mathematics/eclib[flint] sci-mathematics/gp2c app-text/pandoc sci-mathematics/pari sci-mathematics/pari-data XML-Writer XML-LibXML XML-LibXSLT File-Slurp dev-perl/Term-ReadLine?-Gnu JSON SVG dev-perl/MongoDB dev-lang/R" although everything is installed in sheerluck/sage-on-gentoo-stage4:latest (except pandoc)

comment:44 Changed 2 years ago by Matthias Köppe

For sci-mathematics/eclib[flint], please check whether this syntax does whatever was intended there. See previous discussion in #29273.

config.log should give information on why a particular package was rejected by configure.

comment:45 Changed 2 years ago by Andrew

Reading config.log is fascinating.

For example, "cbc >= 2.9.4" might refer to sci-libs/coinor-cbc-2.10.5 (https://github.com/coin-or/Cbc)

WIP

comment:46 in reply to:  43 Changed 2 years ago by François Bissey

Replying to gh-sheerluck:

Now configure says "configure: $ sudo emerge sci-mathematics/eclib[flint] sci-mathematics/gp2c app-text/pandoc sci-mathematics/pari sci-mathematics/pari-data XML-Writer XML-LibXML XML-LibXSLT File-Slurp dev-perl/Term-ReadLine?-Gnu JSON SVG dev-perl/MongoDB dev-lang/R" although everything is installed in sheerluck/sage-on-gentoo-stage4:latest (except pandoc)

USE=flint sudo emerge sci-mathematics/eclib? I am not completely sure USE is passed to the sudo environment that way. I am not sure how many people use sudo on their Gentoo system either. It is not quite the default. I would go

USE=flint su -c "emerge sci-mathematics/eclib"
Last edited 2 years ago by François Bissey (previous) (diff)

comment:47 Changed 2 years ago by Michael Orlitzky

You can leave off the sudo/su command. It's understood that you should have root privileges to use it, and if not, emerge itself will tell you.

The emerge a[foo] b[bar] syntax isn't right but there is no one-line command that does what we want. The closest thing is USE="foo bar" emerge a b, but the next time the user updates his system those USE flags will be lost. It also has the side effect of enabling USE=foo for b and USE=bar for a, which may not work. What should really be done is to place

a foo
b bar

in /etc/portage/package.use, or whatever is equivalent for pkgcore (the other Gentoo package manager). Pkgcore has its own, different, installation commands too.

The a[foo] syntax does make sense in users' heads, so to keep things simple, we could probably just have it say "install the following packages: a[foo] b[bar]", or something like that.

comment:48 Changed 2 years ago by Andrew

Do not worry about USE flags. Every USE flag in sheerluck/sage-on-gentoo-stage4:latest already is correct, because Dockerfile uses "sage-on-gentoo" overlay. If suddenly we need sys-devel/bc[foo] I will go to https://github.com/cschwan/sage-on-gentoo/issues, wait for @fbissey and then refresh latest image. There is no need to have two conflicting places for USE flags, overlay is enough

comment:49 Changed 2 years ago by Michael Orlitzky

I was referring only to,

Now configure says "configure: $ sudo emerge sci-mathematics/eclib[flint]

which won't work =)

comment:50 Changed 2 years ago by Andrew

I also thought it won't work but I tried and it works. Maybe there's a bug in portage =)

comment:51 Changed 2 years ago by Michael Orlitzky

Ok, you got me, I had no idea that worked!

It still won't automatically enable the USE flag if it's not set, you get something like

$ emerge -pv1 sci-mathematics/eclib[flint]
...
The following USE changes are necessary to proceed:
 (see "package.use" in the portage(5) man page for more details)
# required by sci-mathematics/eclib[flint] (argument)
>=sci-mathematics/eclib-20190909 flint

but that's fine. It's telling you to do the right thing.

comment:52 Changed 2 years ago by François Bissey

That's interesting it is basically processed like an atom dependency from an ebuild. It is probably designed so the dependency list can be feed as is to emerge.

comment:53 Changed 2 years ago by Andrew

https://github.com/cschwan/sage-on-gentoo/pull/591

Thanks to @fbissey, today I used the repoman for the first time in my life.

comment:54 Changed 2 years ago by Andrew

 will be installed as an SPKG      already installed in latest image
===============================================================================
 alabaster-0.7.12                  dev-python/alabaster-0.7.12
 appnope-0.1.0.p0                  
 babel-2.6.0                       dev-python/Babel-2.8.0-r1
 backports_abc-0.5                 
 backports_functools_lru_cache-1.5 
 backports_shutil_get_terminal_size
 backports_ssl_match_hostname-3.5.0
 bleach-3.1.0                      dev-python/bleach-3.1.4-r1
 brial-1.2.8                       sci-libs/brial-1.2.8
 certifi-2019.3.9                  dev-python/certifi-2020.4.5.1
 combinatorial_designs-20140630.p0 sci-mathematics/sage-data-combinatorial_designs-20140630
 configparser-3.7.4                
 conway_polynomials-0.5            sci-mathematics/sage-data-conway_polynomials-0.5
 cvxopt-1.2.3                      dev-python/cvxopt-1.2.5-r1
 cycler-0.10.0.p0                  dev-python/cycler-0.10.0-r1
 cypari-2.1.1                      dev-python/cypari2-2.1.1
 cysignals-1.10.2                  dev-python/cysignals-1.10.2
 cython-0.29.17                    dev-python/cython-0.29.19
 dateutil-2.8.1                    
 decorator-4.4.0                   dev-python/decorator-4.4.2
 defusedxml-0.6.0                  
 docutils-0.14                     dev-python/docutils-0.16
 ecl-16.1.2.p5                     dev-lisp/ecls-16.1.2-r1
 elliptic_curves-0.8.1             sci-mathematics/sage-data-elliptic_curves-0.8
 entrypoints-0.3                   dev-python/entrypoints-0.3-r1
 enum34-1.1.6                      
 fpylll-0.5.1dev                   dev-python/fpylll-0.5.1
 functools32-3.2.3-2.p0            
 future-0.17.1                     dev-python/future-0.18.2
 gap-4.10.2.p1                     sci-mathematics/gap-4.10.2-r1
 gc-8.0.4                          
 giac-1.5.0.63-p0                  sci-mathematics/giac-1.5.0.87
 gmpy2-2.1.0b1                     dev-python/gmpy-2.1.0_beta1-r1
 graphs-20161026.p0                sci-mathematics/sage-data-graphs-20161026
 html5lib-1.0.1                    dev-python/html5lib-1.0.1-r2
 imagesize-1.1.0                   dev-python/imagesize-1.1.0
 ipaddress-1.0.22                  
 ipykernel-4.8.2                   dev-python/ipykernel-5.1.4
 ipython-5.8.0                     dev-python/ipython-7.5.0
 ipython_genutils-0.2.0            dev-python/ipython_genutils-0.2.0
 ipywidgets-7.4.2                  dev-python/ipywidgets-7.4.2-r1
 itsdangerous-1.1.0                
 jinja2-2.10                       dev-python/jinja-2.11.1
 jmol-14.29.52                     
 jsonschema-2.6.0                  dev-python/jsonschema-3.0.1
 jupyter_client-5.2.4              dev-python/jupyter_client-5.2.3-r1
 jupyter_core-4.4.0                dev-python/jupyter_core-4.6.1-r1
 kiwisolver-1.0.1                  dev-python/kiwisolver-1.0.1
 libhomfly-1.02r4.p0               sci-libs/libhomfly-1.02.5
 linbox-1.6.3.p0                   sci-libs/linbox-1.6.3
 markupsafe-1.1.0                  dev-python/markupsafe-1.1.1
 mathjax-2.7.4.p0                  dev-libs/mathjax-2.7.7
 matplotlib-3.2.1                  dev-python/matplotlib-3.1.2  (3.2.1 available)
 maxima-5.42.2                     sci-mathematics/maxima-5.42.2
 mistune-0.8.4                     dev-python/mistune-0.8.4
 mpmath-1.1.0                      dev-python/mpmath-1.1.0
 nbconvert-5.4.0                   dev-python/nbconvert-5.2.1 (5.6.1 available)
 nbformat-4.4.0                    dev-python/nbformat-4.4.0-r1
 networkx-2.2                      dev-python/networkx-2.4-r2
 nose-1.3.7                        
 notebook-5.7.6.p0                 dev-python/notebook-5.7.8
 numpy-1.16.6                      dev-python/numpy-1.17.4-r3
 packaging-18.0                    dev-python/packaging-20.3
 palp-2.1.p2                       sci-mathematics/palp-2.11-r1
 pandocfilters-1.4.2               dev-python/pandocfilters-1.4.2-r1
 pathlib2-2.3.3                    
 pathpy-7.1.p0                     dev-python/path-py-12.0.2-r2
 pexpect-4.6.0.p0                  dev-python/pexpect-4.6.0
 pickleshare-0.7.5                 dev-python/pickleshare-0.7.4-r1
 pillow-5.3.0.p0                   dev-python/pillow-7.0.0
 pip-18.1                          
 pkgconfig-1.5.1                   dev-python/pkgconfig-1.5.1
 polytopes_db-20170220.p0          sci-mathematics/sage-data-polytopes_db-20170220
 pplpy-0.8.4                       dev-python/pplpy-0.8.4-r2
 prometheus_client-0.5.0           dev-python/prometheus_client-0.7.1-r1
 prompt_toolkit-1.0.15             dev-python/prompt_toolkit-2.0.10
 psutil-5.2.0.p2                   dev-python/psutil-5.7.0
 ptyprocess-0.5.1.p0               dev-python/ptyprocess-0.6.0
 pycygwin-0.1                      
 pygments-2.3.1.p0                 dev-python/pygments-2.6.1
 pynac-0.7.26.sage-2020-04-03      sci-libs/pynac-0.7.26
 pyparsing-2.3.0                   dev-python/pyparsing-2.4.6
 pytz-2018.7                       dev-python/pytz-2019.3
 pyzmq-19.0.0                      dev-python/pyzmq-19.0.0
 ratpoints-2.1.3.p5                sci-mathematics/ratpoints-2.1.3-r4
 requests-2.13.0                   dev-python/requests-2.23.0
 rpy2-2.8.2.p1                     dev-python/rpy-2.9.0
 rubiks-20070912.p21               
 sage_brial-1.2.8                  dev-python/sage-brial-1.2.8
 sage_conf-none                    
 sagenb_export-3.3                 sci-mathematics/ExportSageNB-3.3
 sagetex-3.4                       
 scandir-1.9.0                     
 scipy-1.2.3                       sci-libs/scipy-1.4.1
 send2trash-1.5.0                  
 setuptools-40.6.3                 dev-python/send2trash-1.5.0-r1
 setuptools_scm-3.1.0              dev-python/setuptools_scm-3.5.0
 simplegeneric-0.8.1.p0            
 singledispatch-3.4.0.3.p0         
 singular-4.1.1p2.p0               sci-mathematics/singular-4.1.1_p2-r2
 six-1.12.0                        dev-python/six-1.14.0
 snowballstemmer-1.2.1.p0          dev-python/snowballstemmer-2.0.0
 sphinx-1.8.5.p0                   dev-python/sphinx-3.0.4
 sphinxcontrib_websupport-1.1.0    
 subprocess32-3.5.3                
 sympy-1.5                         dev-python/sympy-1.5.1-r1
 terminado-0.8.1                   dev-python/terminado-0.8.3
 testpath-0.4.2                    dev-python/testpath-0.3.1
 thebe-9624e0a0.p0                 www-misc/thebe-20160525
 threejs-r110                      media-gfx/threejs-sage-extension-110
 tornado-4.5.2                     www-servers/tornado-6.0.4
 traitlets-4.3.3                   dev-python/traitlets-4.3.2
 typing-3.6.6                      
 vcversioner-2.16.0.0.p0           dev-python/vcversioner-2.16.0.0
 wcwidth-0.1.7.p0                  dev-python/wcwidth-0.1.9
 webencodings-0.5.1                dev-python/webencodings-0.5.1-r1
 werkzeug-0.14.1                   
 widgetsnbextension-3.4.2          dev-python/widgetsnbextension-3.4.2
 zn_poly-0.9.1.p0                  sci-libs/zn_poly-0.9.2
 zope_interface-4.6.0              dev-python/zope-interface-4.7.1     

comment:55 Changed 2 years ago by Andrew

$ docker run -it sheerluck/sage-on-gentoo-stage4 /bin/bash
# python -c 'import sys, pkgutil; sys.exit(0 if pkgutil.find_loader("notinstalled") else 1)'
# echo $?
1
# python -c 'import sys, pkgutil; sys.exit(0 if pkgutil.find_loader("fpylll") else 1)'
# echo $?
0
# python -c 'import sys, pkgutil; sys.exit(0 if pkgutil.find_loader("numpy") else 1)'
# echo $?
0

comment:56 Changed 2 years ago by Andrew

I read m4/sage_spkg_configure.m4 and created build/pkgs/numpy/spkg-configure.m4

SAGE_SPKG_CONFIGURE([numpy], [
    if build/bin/sage-system-python -c 'import pkgutil; exit(not pkgutil.find_loader("numpy"))'; then
      sage_spkg_install_numpy=no
    else
      sage_spkg_install_numpy=yes
    fi  
    if test x$sage_spkg_install_numpy = xyes; then
      AC_MSG_RESULT([using Sage's numpy SPKG])
    else
      AC_MSG_RESULT([using numpy library from the system])
    fi  
])

and tox -e docker-gentoo-standard shows

-----------------------------------------------------------------------------
Checking whether SageMath should install SPKG numpy...
using numpy library from the system
configure: will use system package and not install SPKG numpy
-----------------------------------------------------------------------------
...
numpy-1.16.6:                using system package; SPKG will not be installed

I can copy and paste that spkg-configure.m4 for all dev-python/* packages from the list above. What do you think? Should I proceed?

comment:57 Changed 2 years ago by Matthias Köppe

Please don't. Sage-the-distribution does not use the system Python's site packages. We have a ticket for future work on that: #29023. It will definitely NOT go through spkg-configure.m4.

comment:58 Changed 2 years ago by Matthias Köppe

What's the status of the current branch? Is it ready for testing?

comment:59 Changed 2 years ago by Andrew

It is ready

comment:60 Changed 2 years ago by Matthias Köppe

Add author name please and set to "needs_review" then

comment:61 Changed 2 years ago by Matthias Köppe

Also, could you edit .github/workflows/tox.yml and tox-optional.yml, adding gentoo to the list of tested systems?

comment:62 Changed 2 years ago by git

Commit: c55cfa9bd2d0d29df3cd5cc24be5b743eee8aadf9959ee996823a796140a70e6b3ee16d45d7b1314

Branch pushed to git repo; I updated commit sha1. New commits:

9959ee9tox_system_factor: gentoo

comment:63 Changed 2 years ago by Andrew

Authors: gh-sheerluck
Status: newneeds_review

comment:64 Changed 2 years ago by Matthias Köppe

Reviewers: Matthias Koeppe

comment:66 Changed 2 years ago by Matthias Köppe

Could you update the ticket description so that it matches what is implemented?

comment:67 Changed 2 years ago by Andrew

I have hard time describing git diff 9.2.beta1..u/gh-sheerluck/gentoo , not much was implemented, most of the work is in https://github.com/sheerluck/sage-on-gentoo-stage4

comment:68 Changed 2 years ago by Matthias Köppe

It would be enough to just update the sentence starting with "In this ticket,"

comment:69 Changed 2 years ago by Andrew

My mistake was I tried only tox -e docker-gentoo-standard

I never tried

tox -e docker-gentoo-minimal -- build doc-html ptest

tox -e docker-gentoo-standard -- build doc-html ptest

tox -e docker-gentoo-maximal -- build doc-html ptest

I'm working on it

comment:70 Changed 2 years ago by Matthias Köppe

Status: needs_reviewneeds_work

comment:71 Changed 2 years ago by Andrew

Previous error was due tachyon package, latest image is fixed: SPKG version will be used.

Both docker-gentoo-minimal and docker-gentoo-maximal commands failed same way:

sage -t src/sage/lfunctions/dokchitser.py  # 2 doctests failed
sage -t src/sage/lfunctions/pari.py  # 1 doctest failed
sage -t src/sage/libs/glpk/error.pyx  # 1 doctest failed
sage -t src/sage/numerical/backends/glpk_backend.pyx  # 1 doctest failed
sage -t src/sage/rings/number_field/number_field_element.pyx  # 1 doctest failed
sage -t src/sage/rings/number_field/number_field_ideal.py  # 2 doctests failed
sage -t src/sage/rings/number_field/number_field.py  # 7 doctests failed
sage -t src/sage/rings/number_field/unit_group.py  # 1 doctest failed
sage -t src/sage/rings/polynomial/polynomial_quotient_ring.py  # 2 doctests failed
sage -t src/sage/schemes/elliptic_curves/ell_number_field.py  # 2 doctests failed

comment:72 Changed 2 years ago by Andrew

Status: needs_workneeds_review

comment:74 Changed 2 years ago by Matthias Köppe

Description: modified (diff)

comment:75 Changed 2 years ago by Matthias Köppe

Description: modified (diff)
Summary: Extend dockerfile generator to gentoo, add more gentoo packagestox.ini, build/bin/write-dockerfile.sh: Add gentoo linux, add more gentoo packages

comment:76 Changed 2 years ago by Matthias Köppe

Status: needs_reviewpositive_review

Looking great, thanks for this contribution. Would you mind setting "Authors" to a real name?

comment:77 Changed 2 years ago by Andrew

Next week in addition to existing Dockerfile_gentoo-20.06-gcc9-py37

I plan to make

Dockerfile_gentoo-20.07-gcc9-py37 (with tags 20.07 and latest)

Dockerfile_gentoo-20.07-gcc9-py38 (with tags py38-20.07 and py38-latest)

And maybe even

Dockerfile_gentoo-20.07-clang10-py37

Dockerfile_gentoo-20.07-clang10-py38

gcc-10 is not ready for this kind of fun yet

comment:78 Changed 2 years ago by Frédéric Chapoton

author name should be Full Real Name, please !

comment:79 Changed 2 years ago by Andrew

Authors: gh-sheerluckFrançois Bissey

comment:80 Changed 2 years ago by Matthias Köppe

plot twist?

comment:81 Changed 2 years ago by François Bissey

I didn't author any of these. I cannot claim responsibility. If you don't want your name to be published, say so and we may find something.

comment:82 Changed 2 years ago by Andrew

I don't want my name to be published

comment:83 Changed 2 years ago by Frédéric Chapoton

In github, your are "Andrew" from Belgorod, Russia. What's your exact problem with our rule that contributors should give their full real name ?

comment:84 Changed 2 years ago by Matthias Köppe

Authors: François BisseyAndrey Belgorodski

comment:85 Changed 2 years ago by Volker Braun

Branch: u/gh-sheerluck/gentoo9959ee996823a796140a70e6b3ee16d45d7b1314
Resolution: fixed
Status: positive_reviewclosed

comment:86 Changed 2 years ago by Andrew

Commit: 9959ee996823a796140a70e6b3ee16d45d7b1314

What’s New in sheerluck/sage-on-gentoo-stage4:20.7

1) base image:
   FROM gentoo/portage:20200701 as portage
   FROM gentoo/stage3-amd64:20200701

2) python compiled with sqlite support
   (fixes ModuleNotFoundError: No module named 'sqlite3')

3) updates:
   pandoc             2.9.2.1 -> 2.10 
   sci-libs/amd       2.3.1   -> 2.4.6
   sci-libs/cholmod   2.1.2   -> 3.0.13
   sci-libs/colamd    2.8.0   -> 2.9.6
   sci-libs/openblas  0.3.9   -> 0.3.9-r1
   dev-lang/perl      5.30.1  -> 5.30.3
   dev-lang/R         4.0.0   -> 4.0.2
   sci-libs/umfpack   5.6.2   -> 5.7.9 

with latest image only 1 doctest failed: combinat/cluster_algebra_quiver/quiver.py

comment:87 Changed 2 years ago by Matthias Köppe

Great! I am assuming no change to tox.ini is required?

comment:88 Changed 2 years ago by Andrew

no changes

comment:89 Changed 2 years ago by Andrew

Last error from https://github.com/mkoeppe/sage/actions, namely this one:

It might be possible to solve this slot collision
by applying all of the following changes:
   - sci-libs/m4ri-20200125 (Change USE: +png)
   - sci-libs/brial-1.2.8 (Change USE: +png)

The following USE changes are necessary to proceed:
 (see "package.use" in the portage(5) man page for more details)
# required by sci-libs/brial-1.2.8::sage-on-gentoo
# required by sci-libs/brial (argument)
>=sci-libs/m4ri-20200125 -png 

is solved with png for sci-libs/brial in latest image

(checked on u/mkoeppe/fix_tox_docker_builds_broken_by__29884)

comment:90 Changed 2 years ago by Andrew

-----------------
All tests passed!
-----------------
____ summary ____
docker-gentoo-standard: commands succeeded
congratulations :)

comment:91 Changed 2 years ago by Andrew

I've created latest-gcc10 in addition to latest. For fun.

 version in            latest         latest-gcc10
====================================================
 sys-devel/gcc         9.3.0          10.1.0-r2
 sci-libs/arpack       3.1.5          3.5.0
 sci-mathematics/gp2c  0.0.11.2-r1    0.0.11.3
 app-misc/rlwrap       0.42           0.43

But tox -e docker-gentoo-gcc10-maximal -- build doc-html doc-pdf ptest says

checking gcc version... 10.1.0
checking g++ version... 10.1.0
configure: Installing GCC because g++ -std=gnu++11 is g++ version 10.1.0, which is too recent for this version of Sage 
configure: no suitable system package found for SPKG gcc

Sage is not ready for gcc10 yet...

but if anyone wants to play with latest-gcc10 it's available on hub.docker.com

comment:92 Changed 2 years ago by Dima Pasechnik

Thanks. A related latest annoyance is that on macOS Homebrew provides gcc package, which is gcc-10, and this breaks things (even though only gfortran is used, not gcc).

comment:93 Changed 2 years ago by Andrew

What’s New in sheerluck/sage-on-gentoo-stage4:20.8

1) base image:
   FROM gentoo/portage:20200801 as portage
   FROM gentoo/stage3-amd64:20200801

2) updates:
   arb                2.16.0  -> 2.18.0
   arpack             3.1.5   -> 3.5.0
   cython             0.29.19 -> 0.29.21 
   flint              2.5.2   -> 2.6.0
   fplll              5.3.2   -> 5.3.3
   gp2c               0.11.2  -> 0.11.3
   ipython            7.5.0   -> 7.16.1
   matplotlib         3.1.2   -> 3.2.2
   numpy              1.17.4  -> 1.19.1
   pandas             1.0.1   -> 1.0.5
   pandoc             2.10    -> 2.10.1
   pillow             7.0.0   -> 7.1.2
   python             3.7.7   -> 3.7.8
   scipy              1.4.1   -> 1.5.2 
   sqlite             3.31.1  -> 3.32.3

no changes to tox.ini is required

comment:94 Changed 2 years ago by Andrew

Good news, everyone! We used one latest image, now we have two images:

  • latest with python-3.8.5
  • latest-py37 with python-3.7.8
checking ... whether /usr/bin/python3 is good... no, Python 3.8.5 is too recent

no changes to tox.ini is required, unless we want to be prepared for #29023 when most of installed packages would be used, in that case we can have both "gentoo" and "gentoo-py37" in tox.ini

comment:95 Changed 2 years ago by Matthias Köppe

Great! You may want to test with #27754, which aims for Python 3.8 compatibility

Last edited 2 years ago by Matthias Köppe (previous) (diff)

comment:96 Changed 2 years ago by François Bissey

For the record sage-on-gentoo enables building with python 3.8 since 9.2.beta7. Building of the documentation with it may currently be failing though (but I managed a build between beta6 and beta7 somehow).

comment:97 Changed 2 years ago by Andrew

Sage is ready for gcc-10, so...

What’s New in sheerluck/sage-on-gentoo-stage4:20.9

1) base image:
   FROM gentoo/portage:20200901 as portage
   FROM gentoo/stage3-amd64:20200901

2) gcc-10 + python-3.8.5

3) updates:
   arb                2.18.0  -> 2.18.1
   flint              2.6.0   -> 2.6.3
   glibc              2.30-r8 -> 2.31-r6
   libhomfly          1.02.5  -> 1.02.6
   matplotlib         3.2.2   -> 3.3.1
   mpfr               4.0.2   -> 4.1.0
   sphinx             3.1.2   -> 3.2.1

no changes to tox.ini are required

comment:98 Changed 2 years ago by Andrew

What’s New in sheerluck/sage-on-gentoo-stage4:20.10

1) base image:
   FROM gentoo/portage:20201001 as portage
   FROM gentoo/stage3:20201001

2) gcc-10 + python-3.8.6

3) updates:
   arpack-3.7.0       new
   tox-3.19.0         new
   brial              1.2.8   -> 1.2.9
   gmpy-2.1.0         b1-r1   -> b5
   guile              2.2.4   -> 2.2.6
   ipython            7.16.1  -> 7.17.0
   sqlite             3.32.3  -> 3.33.0
   swig               3.0.12  -> 4.0.0

sheerluck/sage-on-gentoo-stage4:20.10-py37 is the same, only

2) gcc-9 + python-3.7.9

comment:99 Changed 2 years ago by François Bissey

Please mask brial-1.2.9 there are issues with it and it will be replaced by 1.2.10 in short order.

comment:100 Changed 2 years ago by Andrew

masked, downgraded, squashed, tested, pushed

comment:101 Changed 2 years ago by Andrew

checking for python3 >= 3.6.0, < 3.10.0 with modules ...  
checking ... whether /usr/bin/python3 is good... yes

Although overlay cschwan/sage-on-gentoo is not ready for python-3.9 yet, develop branch of sagemath/sage accepts python-3.9.

So as if two images weren't enough, I used some ugly hacks to create third image -- latest-py39

If someone finds it usefull to play with python-3.9, tox.ini may be changed this way:

--- a/tox.ini
+++ b/tox.ini
@@ -208,6 +208,7 @@ setenv =
     gentoo:      SYSTEM=gentoo
     gentoo:      BASE_IMAGE=sheerluck/sage-on-gentoo-stage4
     gentoo-python3.7: BASE_TAG=latest-py37
+    gentoo-python3.9: BASE_TAG=latest-py39
     #    
     # https://hub.docker.com/_/archlinux/
     #    

comment:102 Changed 2 years ago by Matthias Köppe

I have created #30785 for this

comment:103 Changed 21 months ago by Andrew

I was under impression that something is not right with my images until I saw https://trac.sagemath.org/ticket/31395 That bug has ruined monthful of latest https://github.com/mkoeppe/sage/actions/ :(

comment:104 in reply to:  103 ; Changed 21 months ago by Michael Orlitzky

Replying to gh-sheerluck:

I was under impression that something is not right with my images until I saw https://trac.sagemath.org/ticket/31395 That bug has ruined monthful of latest https://github.com/mkoeppe/sage/actions/ :(

I'm going to post a branch that removes the memlimit and then turn off my computer for a month while people yell at me.

comment:105 in reply to:  104 Changed 20 months ago by Dima Pasechnik

Replying to mjo:

Replying to gh-sheerluck:

I was under impression that something is not right with my images until I saw https://trac.sagemath.org/ticket/31395 That bug has ruined monthful of latest https://github.com/mkoeppe/sage/actions/ :(

I'm going to post a branch that removes the memlimit and then turn off my computer for a month while people yell at me.

see, #31395 got in :-)

comment:106 Changed 15 months ago by Andrew

latest image contains clang-12.0.1 for tox -e docker-gentoo-maximal

comment:107 Changed 14 months ago by Andrew

At https://github.com/orlitzky/sage/runs/3851208039?check_suite_focus=true#step:7:11493 I see Pytest is not installed, skip checking tests that rely on it.

Well, that is not true, latest image got /usr/bin/py.test and /usr/bin/pytest

comment:108 Changed 14 months ago by Dima Pasechnik

you need pytest in sage's venv.

e.g.

make pytest

will do the trick

Note: See TracTickets for help on using tickets.