Opened 22 months ago

Closed 17 months ago

Last modified 7 weeks ago

#29105 closed enhancement (fixed)

tox.ini, build/bin/write-dockerfile.sh: Add gentoo linux, add more gentoo packages

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-9.2
Component: porting Keywords:
Cc: mjo, fbissey, dimpase, gh-sheerluck Merged in:
Authors: Andrey Belgorodski Reviewers: Matthias Koeppe
Report Upstream: N/A Work issues:
Branch: 9959ee9 (Commits, GitHub, GitLab) Commit:
Dependencies: #29273 Stopgaps:

Status badges

Description (last modified by mkoeppe)

As a followup to #29053 and #29273, we

To test: tox -e docker-gentoo-standard

This is a part of:

  • #29060: Meta-ticket: Add Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages

Resources:

Attachments (6)

gentoo-log-2.txt (2.1 MB) - added by mkoeppe 22 months ago.
Dockerfile-sage_gentoo (3.0 KB) - added by mkoeppe 22 months ago.
docker-gentoo-installed.txt (9.5 KB) - added by gh-sheerluck 18 months ago.
docker-gentoo-installed.txt
tox.ini.patch (718 bytes) - added by gh-sheerluck 18 months ago.
write-dockerfile.sh.patch (759 bytes) - added by gh-sheerluck 18 months ago.
Dockerfile_gentoo-20.06-gcc9-py37 (2.0 KB) - added by gh-sheerluck 18 months ago.

Change History (114)

comment:1 Changed 22 months ago by mkoeppe

  • Description modified (diff)

Creating a dockerfile for setting up sage prerequisites on a Gentoo "prefix" was previously discussed in #28905.

An attempt (incomplete) is at https://trac.sagemath.org/attachment/ticket/28905/Dockerfile-sage_gentoo and would need help from Gentoo experts.

comment:2 Changed 22 months ago by fbissey

compiler can't create executables in a configure phase of gcc typically means the compiler can't find libgmp and al. Do you have a log for me to review? Prefix on ubuntu is definitely supported. The only thing I can think of is that the bootstrap usually object being run as root.

comment:3 Changed 22 months ago by mkoeppe

Thanks, yes, turns out this was not specific to the gentoo prefix installer. I had to add some more packages. Let me see how far I get with this.

Changed 22 months ago by mkoeppe

Changed 22 months ago by mkoeppe

comment:4 Changed 22 months ago by mkoeppe

OK... this is where I got now. The ./bootstrap-prefix.sh script was claiming to bootstrap some packages - such as xz and patch - and succeeded with that; but then it was missing "xz" to unpack the Python sources. I think something is wrong there, but I added xz&patch in ubuntu. This is attached Dockerfile-sage_gentoo and log from 'docker build' with that.

As you can see in the log, now it's complaining about a module missing in the Python that it just built:

  File "/gentoo/var/tmp/python-3.6.10/Python-3.6.10/Lib/xml/parsers/expat.py", line 4, in <module>
    from pyexpat import *
ModuleNotFoundError: No module named 'pyexpat'
make: *** [install] Error 1
Makefile:1102: recipe for target 'install' failed
??? Python failed to install *sigh* continuing anyway
* Python bootstrapped
http://distfiles.gentoo.org/snapshots

and then dies with:

* Compiling prefix-portage
./configure --host=x86_64-pc-linux-gnu --prefix=/gentoo/tmp/usr --mandir=/gentoo/tmp/usr/share/man --infodir=/gentoo/tmp/usr/share/info --datadir=/gentoo/tmp/usr/share --sysconfdir=/gentoo/tmp/etc --localstatedir=/gentoo/tmp/var/lib --build=x86_64-pc-linux-gnu --with-offset-prefix=/gentoo/tmp --with-portage-user=root --with-portage-group=root --with-extra-path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking target system type... x86_64-pc-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... no
checking for mawk... mawk
....
checking whether /usr/bin/id is good enough... yes
checking for python... no
configure: error: no python found in your path

comment:5 Changed 22 months ago by fbissey

xz is rearing up its head again. Yes it should be an early one. Because many more sources are in .xz I am not shocked that it should be a pre-requisite along with patch. The pyexpat stuff is weird. I'll do some experimenting later. I haven't really touched prefix since ppc64 was retired (because I didn't want or could maintain it long term).

comment:6 Changed 22 months ago by mkoeppe

  • Description modified (diff)

comment:7 Changed 21 months ago by mkoeppe

  • Cc dimpase added

comment:8 Changed 21 months ago by mkoeppe

  • Dependencies set to #29273
  • Description modified (diff)
  • Summary changed from Add gentoo package information, extend dockerfile generator to gentoo to Extend dockerfile generator to gentoo

comment:9 Changed 21 months ago by mkoeppe

This tickets needs help from someone who actually uses Gentoo. I found the instructions for installing stuff inadequate.

comment:10 follow-up: Changed 21 months ago by dimpase

I use Gentoo, but neither prefix, not docker.

comment:11 Changed 21 months ago by fbissey

I need to make time for this. I cannot give you a time frame.

comment:12 in reply to: ↑ 10 ; follow-up: Changed 21 months ago by mkoeppe

Replying to dimpase:

I use Gentoo, but [...] not docker.

After installing a docker client, it only takes

docker run -it gentoo/stage3-amd64

This gives you root prompt. I would need instructions on what to do next to install relevant gentoo packages here.

comment:13 follow-up: Changed 21 months ago by fbissey

If I understand well you have a minimal gentoo image running now. What do you want to install? Do you have a list somewhere?

comment:14 in reply to: ↑ 13 Changed 21 months ago by mkoeppe

Replying to fbissey:

If I understand well you have a minimal gentoo image running now. What do you want to install? Do you have a list somewhere?

The equivalent of the lists in build/pkgs/debian-bootstrap.txt and build/pkgs/debian.txt (minimal requirements for bootstrapping and building) -- which would be added in #29273 as well since gentoo packages are being added there.

Last edited 21 months ago by mkoeppe (previous) (diff)

comment:15 in reply to: ↑ 12 Changed 21 months ago by dimpase

Replying to mkoeppe:

Replying to dimpase:

I use Gentoo, but [...] not docker.

After installing a docker client, it only takes

docker run -it gentoo/stage3-amd64

This gives you root prompt. I would need instructions on what to do next to install relevant gentoo packages here.

I can try either this or Gentoo in lxc container, if such a thing exists somewhere.

comment:16 Changed 21 months ago by mkoeppe

Docker please because that's what my tox.ini infrastructure uses. If we want to do the same with lxc instead of docker, why not, but perhaps one step at a time?

comment:17 Changed 21 months ago by mkoeppe

I have created ticket #29283 for the goal of lxc containers for testing, in case someone wants to work on that.

comment:18 Changed 21 months ago by mkoeppe

  • Summary changed from Extend dockerfile generator to gentoo to Extend dockerfile generator to gentoo, add more gentoo packages

comment:19 Changed 20 months ago by mkoeppe

  • Milestone changed from sage-9.1 to sage-9.2

comment:20 Changed 18 months ago by mkoeppe

  • Cc gh-sheerluck added

comment:22 Changed 18 months ago by gh-sheerluck

When I run tox -e docker-gentoo I get half-baked sage/.tox/docker-gentoo/Dockerfile, I need a way to shove the hand-crafted Dockerfile instead (for debugging purposes). What is the best way to do it?

comment:23 Changed 18 months ago by mkoeppe

Does your handcrafted Dockerfile work with docker build already?

comment:24 Changed 18 months ago by mkoeppe

You could insert a line before ` # From https://hub.docker.com/r/multiarch/ubuntu-core/ ` that says

    docker-gentoo:  bash -c 'whatever command to create a handcrafted {envdir}/Dockerfile'    

comment:25 Changed 18 months ago by gh-sheerluck

So right after

docker:        bash -c 'build/bin/write-dockerfile.sh  ... > {envdir}/Dockerfile'

Thank you!

Changed 18 months ago by gh-sheerluck

docker-gentoo-installed.txt

comment:26 Changed 18 months ago by gh-sheerluck

The work is still in progress. I managed to install 555 packages.

comment:27 Changed 18 months ago by mkoeppe

Nice

comment:28 Changed 18 months ago by gh-sheerluck

real    14m14.888s
user    98m59.353s
sys     3m4.286s
SAGE_CHECK=warn, so scanning the log files. This may take a few seconds.
Sage build/upgrade complete!
make[1]: Leaving directory '/sage'
Removing intermediate container e0e5bbfb22ea
 ---> 14316b24894b
[Warning] One or more build-args [TARGETS_OPTIONAL] were not consumed
Successfully built 14316b24894b
Successfully tagged sage-docker-gentoo-standard-with-targets:9.2.beta1-dirty
_________________________ summary __________________________
  docker-gentoo-standard: commands succeeded
  congratulations :)

1) run docker build -t fbissey/sage-on-gentoo-stage4:20.6 -t fbissey/sage-on-gentoo-stage4:latest -f Dockerfile_gentoo-20.06-gcc9-py37 .

2) patch tox.ini and write-dockerfile.sh

Changed 18 months ago by gh-sheerluck

Changed 18 months ago by gh-sheerluck

Changed 18 months ago by gh-sheerluck

comment:29 follow-up: Changed 18 months ago by mkoeppe

comment:30 in reply to: ↑ 29 Changed 18 months ago by gh-sheerluck

Replying to mkoeppe:

https://hub.docker.com/r/fbissey/sage-on-gentoo-stage/tags ... this does not seem to exist yet?

It does not exist. Someone have to decide where to store an image. I have no idea, my docker-fu is weak.

comment:31 Changed 18 months ago by mkoeppe

Why not create an account on docker hub? Next step: https://docs.docker.com/docker-hub/builds/

comment:32 Changed 18 months ago by gh-sheerluck

I am working on Dockerfile_gentoo-20.06-gcc9-py37​ to make image size much much smaller :(

comment:33 Changed 18 months ago by gh-sheerluck

docker pull sheerluck/sage-on-gentoo-stage4:latest

comment:34 Changed 18 months ago by gh-sheerluck

$ docker run -it sheerluck/sage-on-gentoo-stage4 /bin/bash
cf6f39e01c4a / # ipython --no-banner

In [1]: from os import walk
   ...: pth, ext = "/var/db/pkg", ".ebuild"
   ...: lex, res = len(ext), []
   ...: for _, _, u in walk(pth):
   ...:     res += [x[:-lex] for x in u if x.endswith(ext)]
   ...: for  pkg  in sorted(res): print(pkg)
Alnuth-3.1.1
App-pwhich-1.150.0
Authen-SASL-2.160.0-r1
...
zn_poly-0.9.2
zope-interface-4.7.1
zstd-1.4.4-r4
zziplib-0.13.69-r1

comment:35 Changed 18 months ago by mkoeppe

Awesome!

comment:36 Changed 18 months ago by mkoeppe

Could you put the changes to the sage scripts on a git branch please?

comment:37 Changed 18 months ago by gh-sheerluck

$ git push trac HEAD:u/gh-sheerluck/gentoo
...
To trac.sagemath.org:sage.git
 * [new branch]            HEAD -> u/gh-sheerluck/gentoo

comment:38 Changed 18 months ago by mkoeppe

  • Branch set to u/gh-sheerluck/gentoo
  • Commit set to 25d05792b7d44103de445e2e591393f3e5ef3862

New commits:

25d0579write-dockerfile.sh is extended to support gentoo

comment:39 Changed 18 months ago by mkoeppe

sed: ./build/pkgs/gentoo.txt: No such file or directory
sed: ./build/pkgs/gentoo-bootstrap.txt: No such file or directory

comment:40 Changed 18 months ago by mkoeppe

Also, at the end of configure, I see:

configure: notice: the following SPKGs did not find equivalent system packages: cbc eclib gp2c isl lcalc libsemigroups pandoc pari pari_elldata pari_galdata pari_galpol pari_nftables pari_seadata pari_seadata_small perl_cpan_polymake_prereq perl_term_readline_gnu r yasm
checking for the package system in use... gentoo
configure: hint: installing the following system packages is recommended and may avoid building some of the above SPKGs from source:
configure: # install the following packages: sci-mathematics/eclib[flint] sci-mathematics/gp2c dev-libs/isl app-text/pandoc sci-mathematics/pari sci-mathematics/pari-data XML-Writer XML-LibXML XML-LibXSLT File-Slurp dev-perl/Term-ReadLine-Gnu JSON SVG dev-perl/MongoDB dev-perl/Term-ReadLine-Gnu dev-lang/R

Could you update build/bin/sage-print-system-package-command so that an actual installation command is displayed?

comment:41 Changed 18 months ago by gh-sheerluck

I added gentoo.txt and gentoo-bootstrap.txt and updated sage-print-system-package-command and I am going to refresh sheerluck/sage-on-gentoo-stage4:latest with additional packages (except pandoc) and after testing I will push in u/gh-sheerluck/gentoo branch

comment:42 Changed 18 months ago by git

  • Commit changed from 25d05792b7d44103de445e2e591393f3e5ef3862 to c55cfa9bd2d0d29df3cd5cc24be5b743eee8aadf

Branch pushed to git repo; I updated commit sha1. New commits:

c55cfa9sage-print-system-package-command updated

comment:43 follow-up: Changed 18 months ago by gh-sheerluck

Now configure says "configure: $ sudo emerge sci-mathematics/eclib[flint] sci-mathematics/gp2c app-text/pandoc sci-mathematics/pari sci-mathematics/pari-data XML-Writer XML-LibXML XML-LibXSLT File-Slurp dev-perl/Term-ReadLine?-Gnu JSON SVG dev-perl/MongoDB dev-lang/R" although everything is installed in sheerluck/sage-on-gentoo-stage4:latest (except pandoc)

comment:44 Changed 18 months ago by mkoeppe

For sci-mathematics/eclib[flint], please check whether this syntax does whatever was intended there. See previous discussion in #29273.

config.log should give information on why a particular package was rejected by configure.

comment:45 Changed 18 months ago by gh-sheerluck

Reading config.log is fascinating.

For example, "cbc >= 2.9.4" might refer to sci-libs/coinor-cbc-2.10.5 (https://github.com/coin-or/Cbc)

WIP

comment:46 in reply to: ↑ 43 Changed 18 months ago by fbissey

Replying to gh-sheerluck:

Now configure says "configure: $ sudo emerge sci-mathematics/eclib[flint] sci-mathematics/gp2c app-text/pandoc sci-mathematics/pari sci-mathematics/pari-data XML-Writer XML-LibXML XML-LibXSLT File-Slurp dev-perl/Term-ReadLine?-Gnu JSON SVG dev-perl/MongoDB dev-lang/R" although everything is installed in sheerluck/sage-on-gentoo-stage4:latest (except pandoc)

USE=flint sudo emerge sci-mathematics/eclib? I am not completely sure USE is passed to the sudo environment that way. I am not sure how many people use sudo on their Gentoo system either. It is not quite the default. I would go

USE=flint su -c "emerge sci-mathematics/eclib"
Last edited 18 months ago by fbissey (previous) (diff)

comment:47 Changed 18 months ago by mjo

You can leave off the sudo/su command. It's understood that you should have root privileges to use it, and if not, emerge itself will tell you.

The emerge a[foo] b[bar] syntax isn't right but there is no one-line command that does what we want. The closest thing is USE="foo bar" emerge a b, but the next time the user updates his system those USE flags will be lost. It also has the side effect of enabling USE=foo for b and USE=bar for a, which may not work. What should really be done is to place

a foo
b bar

in /etc/portage/package.use, or whatever is equivalent for pkgcore (the other Gentoo package manager). Pkgcore has its own, different, installation commands too.

The a[foo] syntax does make sense in users' heads, so to keep things simple, we could probably just have it say "install the following packages: a[foo] b[bar]", or something like that.

comment:48 Changed 18 months ago by gh-sheerluck

Do not worry about USE flags. Every USE flag in sheerluck/sage-on-gentoo-stage4:latest already is correct, because Dockerfile uses "sage-on-gentoo" overlay. If suddenly we need sys-devel/bc[foo] I will go to https://github.com/cschwan/sage-on-gentoo/issues, wait for @fbissey and then refresh latest image. There is no need to have two conflicting places for USE flags, overlay is enough

comment:49 Changed 18 months ago by mjo

I was referring only to,

Now configure says "configure: $ sudo emerge sci-mathematics/eclib[flint]

which won't work =)

comment:50 Changed 18 months ago by gh-sheerluck

I also thought it won't work but I tried and it works. Maybe there's a bug in portage =)

comment:51 Changed 18 months ago by mjo

Ok, you got me, I had no idea that worked!

It still won't automatically enable the USE flag if it's not set, you get something like

$ emerge -pv1 sci-mathematics/eclib[flint]
...
The following USE changes are necessary to proceed:
 (see "package.use" in the portage(5) man page for more details)
# required by sci-mathematics/eclib[flint] (argument)
>=sci-mathematics/eclib-20190909 flint

but that's fine. It's telling you to do the right thing.

comment:52 Changed 18 months ago by fbissey

That's interesting it is basically processed like an atom dependency from an ebuild. It is probably designed so the dependency list can be feed as is to emerge.

comment:53 Changed 18 months ago by gh-sheerluck

https://github.com/cschwan/sage-on-gentoo/pull/591

Thanks to @fbissey, today I used the repoman for the first time in my life.

comment:54 Changed 17 months ago by gh-sheerluck

 will be installed as an SPKG      already installed in latest image
===============================================================================
 alabaster-0.7.12                  dev-python/alabaster-0.7.12
 appnope-0.1.0.p0                  
 babel-2.6.0                       dev-python/Babel-2.8.0-r1
 backports_abc-0.5                 
 backports_functools_lru_cache-1.5 
 backports_shutil_get_terminal_size
 backports_ssl_match_hostname-3.5.0
 bleach-3.1.0                      dev-python/bleach-3.1.4-r1
 brial-1.2.8                       sci-libs/brial-1.2.8
 certifi-2019.3.9                  dev-python/certifi-2020.4.5.1
 combinatorial_designs-20140630.p0 sci-mathematics/sage-data-combinatorial_designs-20140630
 configparser-3.7.4                
 conway_polynomials-0.5            sci-mathematics/sage-data-conway_polynomials-0.5
 cvxopt-1.2.3                      dev-python/cvxopt-1.2.5-r1
 cycler-0.10.0.p0                  dev-python/cycler-0.10.0-r1
 cypari-2.1.1                      dev-python/cypari2-2.1.1
 cysignals-1.10.2                  dev-python/cysignals-1.10.2
 cython-0.29.17                    dev-python/cython-0.29.19
 dateutil-2.8.1                    
 decorator-4.4.0                   dev-python/decorator-4.4.2
 defusedxml-0.6.0                  
 docutils-0.14                     dev-python/docutils-0.16
 ecl-16.1.2.p5                     dev-lisp/ecls-16.1.2-r1
 elliptic_curves-0.8.1             sci-mathematics/sage-data-elliptic_curves-0.8
 entrypoints-0.3                   dev-python/entrypoints-0.3-r1
 enum34-1.1.6                      
 fpylll-0.5.1dev                   dev-python/fpylll-0.5.1
 functools32-3.2.3-2.p0            
 future-0.17.1                     dev-python/future-0.18.2
 gap-4.10.2.p1                     sci-mathematics/gap-4.10.2-r1
 gc-8.0.4                          
 giac-1.5.0.63-p0                  sci-mathematics/giac-1.5.0.87
 gmpy2-2.1.0b1                     dev-python/gmpy-2.1.0_beta1-r1
 graphs-20161026.p0                sci-mathematics/sage-data-graphs-20161026
 html5lib-1.0.1                    dev-python/html5lib-1.0.1-r2
 imagesize-1.1.0                   dev-python/imagesize-1.1.0
 ipaddress-1.0.22                  
 ipykernel-4.8.2                   dev-python/ipykernel-5.1.4
 ipython-5.8.0                     dev-python/ipython-7.5.0
 ipython_genutils-0.2.0            dev-python/ipython_genutils-0.2.0
 ipywidgets-7.4.2                  dev-python/ipywidgets-7.4.2-r1
 itsdangerous-1.1.0                
 jinja2-2.10                       dev-python/jinja-2.11.1
 jmol-14.29.52                     
 jsonschema-2.6.0                  dev-python/jsonschema-3.0.1
 jupyter_client-5.2.4              dev-python/jupyter_client-5.2.3-r1
 jupyter_core-4.4.0                dev-python/jupyter_core-4.6.1-r1
 kiwisolver-1.0.1                  dev-python/kiwisolver-1.0.1
 libhomfly-1.02r4.p0               sci-libs/libhomfly-1.02.5
 linbox-1.6.3.p0                   sci-libs/linbox-1.6.3
 markupsafe-1.1.0                  dev-python/markupsafe-1.1.1
 mathjax-2.7.4.p0                  dev-libs/mathjax-2.7.7
 matplotlib-3.2.1                  dev-python/matplotlib-3.1.2  (3.2.1 available)
 maxima-5.42.2                     sci-mathematics/maxima-5.42.2
 mistune-0.8.4                     dev-python/mistune-0.8.4
 mpmath-1.1.0                      dev-python/mpmath-1.1.0
 nbconvert-5.4.0                   dev-python/nbconvert-5.2.1 (5.6.1 available)
 nbformat-4.4.0                    dev-python/nbformat-4.4.0-r1
 networkx-2.2                      dev-python/networkx-2.4-r2
 nose-1.3.7                        
 notebook-5.7.6.p0                 dev-python/notebook-5.7.8
 numpy-1.16.6                      dev-python/numpy-1.17.4-r3
 packaging-18.0                    dev-python/packaging-20.3
 palp-2.1.p2                       sci-mathematics/palp-2.11-r1
 pandocfilters-1.4.2               dev-python/pandocfilters-1.4.2-r1
 pathlib2-2.3.3                    
 pathpy-7.1.p0                     dev-python/path-py-12.0.2-r2
 pexpect-4.6.0.p0                  dev-python/pexpect-4.6.0
 pickleshare-0.7.5                 dev-python/pickleshare-0.7.4-r1
 pillow-5.3.0.p0                   dev-python/pillow-7.0.0
 pip-18.1                          
 pkgconfig-1.5.1                   dev-python/pkgconfig-1.5.1
 polytopes_db-20170220.p0          sci-mathematics/sage-data-polytopes_db-20170220
 pplpy-0.8.4                       dev-python/pplpy-0.8.4-r2
 prometheus_client-0.5.0           dev-python/prometheus_client-0.7.1-r1
 prompt_toolkit-1.0.15             dev-python/prompt_toolkit-2.0.10
 psutil-5.2.0.p2                   dev-python/psutil-5.7.0
 ptyprocess-0.5.1.p0               dev-python/ptyprocess-0.6.0
 pycygwin-0.1                      
 pygments-2.3.1.p0                 dev-python/pygments-2.6.1
 pynac-0.7.26.sage-2020-04-03      sci-libs/pynac-0.7.26
 pyparsing-2.3.0                   dev-python/pyparsing-2.4.6
 pytz-2018.7                       dev-python/pytz-2019.3
 pyzmq-19.0.0                      dev-python/pyzmq-19.0.0
 ratpoints-2.1.3.p5                sci-mathematics/ratpoints-2.1.3-r4
 requests-2.13.0                   dev-python/requests-2.23.0
 rpy2-2.8.2.p1                     dev-python/rpy-2.9.0
 rubiks-20070912.p21               
 sage_brial-1.2.8                  dev-python/sage-brial-1.2.8
 sage_conf-none                    
 sagenb_export-3.3                 sci-mathematics/ExportSageNB-3.3
 sagetex-3.4                       
 scandir-1.9.0                     
 scipy-1.2.3                       sci-libs/scipy-1.4.1
 send2trash-1.5.0                  
 setuptools-40.6.3                 dev-python/send2trash-1.5.0-r1
 setuptools_scm-3.1.0              dev-python/setuptools_scm-3.5.0
 simplegeneric-0.8.1.p0            
 singledispatch-3.4.0.3.p0         
 singular-4.1.1p2.p0               sci-mathematics/singular-4.1.1_p2-r2
 six-1.12.0                        dev-python/six-1.14.0
 snowballstemmer-1.2.1.p0          dev-python/snowballstemmer-2.0.0
 sphinx-1.8.5.p0                   dev-python/sphinx-3.0.4
 sphinxcontrib_websupport-1.1.0    
 subprocess32-3.5.3                
 sympy-1.5                         dev-python/sympy-1.5.1-r1
 terminado-0.8.1                   dev-python/terminado-0.8.3
 testpath-0.4.2                    dev-python/testpath-0.3.1
 thebe-9624e0a0.p0                 www-misc/thebe-20160525
 threejs-r110                      media-gfx/threejs-sage-extension-110
 tornado-4.5.2                     www-servers/tornado-6.0.4
 traitlets-4.3.3                   dev-python/traitlets-4.3.2
 typing-3.6.6                      
 vcversioner-2.16.0.0.p0           dev-python/vcversioner-2.16.0.0
 wcwidth-0.1.7.p0                  dev-python/wcwidth-0.1.9
 webencodings-0.5.1                dev-python/webencodings-0.5.1-r1
 werkzeug-0.14.1                   
 widgetsnbextension-3.4.2          dev-python/widgetsnbextension-3.4.2
 zn_poly-0.9.1.p0                  sci-libs/zn_poly-0.9.2
 zope_interface-4.6.0              dev-python/zope-interface-4.7.1     

comment:55 Changed 17 months ago by gh-sheerluck

$ docker run -it sheerluck/sage-on-gentoo-stage4 /bin/bash
# python -c 'import sys, pkgutil; sys.exit(0 if pkgutil.find_loader("notinstalled") else 1)'
# echo $?
1
# python -c 'import sys, pkgutil; sys.exit(0 if pkgutil.find_loader("fpylll") else 1)'
# echo $?
0
# python -c 'import sys, pkgutil; sys.exit(0 if pkgutil.find_loader("numpy") else 1)'
# echo $?
0

comment:56 Changed 17 months ago by gh-sheerluck

I read m4/sage_spkg_configure.m4 and created build/pkgs/numpy/spkg-configure.m4

SAGE_SPKG_CONFIGURE([numpy], [
    if build/bin/sage-system-python -c 'import pkgutil; exit(not pkgutil.find_loader("numpy"))'; then
      sage_spkg_install_numpy=no
    else
      sage_spkg_install_numpy=yes
    fi  
    if test x$sage_spkg_install_numpy = xyes; then
      AC_MSG_RESULT([using Sage's numpy SPKG])
    else
      AC_MSG_RESULT([using numpy library from the system])
    fi  
])

and tox -e docker-gentoo-standard shows

-----------------------------------------------------------------------------
Checking whether SageMath should install SPKG numpy...
using numpy library from the system
configure: will use system package and not install SPKG numpy
-----------------------------------------------------------------------------
...
numpy-1.16.6:                using system package; SPKG will not be installed

I can copy and paste that spkg-configure.m4 for all dev-python/* packages from the list above. What do you think? Should I proceed?

comment:57 Changed 17 months ago by mkoeppe

Please don't. Sage-the-distribution does not use the system Python's site packages. We have a ticket for future work on that: #29023. It will definitely NOT go through spkg-configure.m4.

comment:58 Changed 17 months ago by mkoeppe

What's the status of the current branch? Is it ready for testing?

comment:59 Changed 17 months ago by gh-sheerluck

It is ready

comment:60 Changed 17 months ago by mkoeppe

Add author name please and set to "needs_review" then

comment:61 Changed 17 months ago by mkoeppe

Also, could you edit .github/workflows/tox.yml and tox-optional.yml, adding gentoo to the list of tested systems?

comment:62 Changed 17 months ago by git

  • Commit changed from c55cfa9bd2d0d29df3cd5cc24be5b743eee8aadf to 9959ee996823a796140a70e6b3ee16d45d7b1314

Branch pushed to git repo; I updated commit sha1. New commits:

9959ee9tox_system_factor: gentoo

comment:63 Changed 17 months ago by gh-sheerluck

  • Authors set to gh-sheerluck
  • Status changed from new to needs_review

comment:64 Changed 17 months ago by mkoeppe

  • Reviewers set to Matthias Koeppe

comment:66 Changed 17 months ago by mkoeppe

Could you update the ticket description so that it matches what is implemented?

comment:67 Changed 17 months ago by gh-sheerluck

I have hard time describing git diff 9.2.beta1..u/gh-sheerluck/gentoo , not much was implemented, most of the work is in https://github.com/sheerluck/sage-on-gentoo-stage4

comment:68 Changed 17 months ago by mkoeppe

It would be enough to just update the sentence starting with "In this ticket,"

comment:69 Changed 17 months ago by gh-sheerluck

My mistake was I tried only tox -e docker-gentoo-standard

I never tried

tox -e docker-gentoo-minimal -- build doc-html ptest

tox -e docker-gentoo-standard -- build doc-html ptest

tox -e docker-gentoo-maximal -- build doc-html ptest

I'm working on it

comment:70 Changed 17 months ago by mkoeppe

  • Status changed from needs_review to needs_work

comment:71 Changed 17 months ago by gh-sheerluck

Previous error was due tachyon package, latest image is fixed: SPKG version will be used.

Both docker-gentoo-minimal and docker-gentoo-maximal commands failed same way:

sage -t src/sage/lfunctions/dokchitser.py  # 2 doctests failed
sage -t src/sage/lfunctions/pari.py  # 1 doctest failed
sage -t src/sage/libs/glpk/error.pyx  # 1 doctest failed
sage -t src/sage/numerical/backends/glpk_backend.pyx  # 1 doctest failed
sage -t src/sage/rings/number_field/number_field_element.pyx  # 1 doctest failed
sage -t src/sage/rings/number_field/number_field_ideal.py  # 2 doctests failed
sage -t src/sage/rings/number_field/number_field.py  # 7 doctests failed
sage -t src/sage/rings/number_field/unit_group.py  # 1 doctest failed
sage -t src/sage/rings/polynomial/polynomial_quotient_ring.py  # 2 doctests failed
sage -t src/sage/schemes/elliptic_curves/ell_number_field.py  # 2 doctests failed

comment:72 Changed 17 months ago by gh-sheerluck

  • Status changed from needs_work to needs_review

comment:74 Changed 17 months ago by mkoeppe

  • Description modified (diff)

comment:75 Changed 17 months ago by mkoeppe

  • Description modified (diff)
  • Summary changed from Extend dockerfile generator to gentoo, add more gentoo packages to tox.ini, build/bin/write-dockerfile.sh: Add gentoo linux, add more gentoo packages

comment:76 Changed 17 months ago by mkoeppe

  • Status changed from needs_review to positive_review

Looking great, thanks for this contribution. Would you mind setting "Authors" to a real name?

comment:77 Changed 17 months ago by gh-sheerluck

Next week in addition to existing Dockerfile_gentoo-20.06-gcc9-py37

I plan to make

Dockerfile_gentoo-20.07-gcc9-py37 (with tags 20.07 and latest)

Dockerfile_gentoo-20.07-gcc9-py38 (with tags py38-20.07 and py38-latest)

And maybe even

Dockerfile_gentoo-20.07-clang10-py37

Dockerfile_gentoo-20.07-clang10-py38

gcc-10 is not ready for this kind of fun yet

comment:78 Changed 17 months ago by chapoton

author name should be Full Real Name, please !

comment:79 Changed 17 months ago by gh-sheerluck

  • Authors changed from gh-sheerluck to François Bissey

comment:80 Changed 17 months ago by mkoeppe

plot twist?

comment:81 Changed 17 months ago by fbissey

I didn't author any of these. I cannot claim responsibility. If you don't want your name to be published, say so and we may find something.

comment:82 Changed 17 months ago by gh-sheerluck

I don't want my name to be published

comment:83 Changed 17 months ago by chapoton

In github, your are "Andrew" from Belgorod, Russia. What's your exact problem with our rule that contributors should give their full real name ?

comment:84 Changed 17 months ago by mkoeppe

  • Authors changed from François Bissey to Andrey Belgorodski

comment:85 Changed 17 months ago by vbraun

  • Branch changed from u/gh-sheerluck/gentoo to 9959ee996823a796140a70e6b3ee16d45d7b1314
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:86 Changed 17 months ago by gh-sheerluck

  • Commit 9959ee996823a796140a70e6b3ee16d45d7b1314 deleted

What’s New in sheerluck/sage-on-gentoo-stage4:20.7

1) base image:
   FROM gentoo/portage:20200701 as portage
   FROM gentoo/stage3-amd64:20200701

2) python compiled with sqlite support
   (fixes ModuleNotFoundError: No module named 'sqlite3')

3) updates:
   pandoc             2.9.2.1 -> 2.10 
   sci-libs/amd       2.3.1   -> 2.4.6
   sci-libs/cholmod   2.1.2   -> 3.0.13
   sci-libs/colamd    2.8.0   -> 2.9.6
   sci-libs/openblas  0.3.9   -> 0.3.9-r1
   dev-lang/perl      5.30.1  -> 5.30.3
   dev-lang/R         4.0.0   -> 4.0.2
   sci-libs/umfpack   5.6.2   -> 5.7.9 

with latest image only 1 doctest failed: combinat/cluster_algebra_quiver/quiver.py

comment:87 Changed 17 months ago by mkoeppe

Great! I am assuming no change to tox.ini is required?

comment:88 Changed 17 months ago by gh-sheerluck

no changes

comment:89 Changed 17 months ago by gh-sheerluck

Last error from https://github.com/mkoeppe/sage/actions, namely this one:

It might be possible to solve this slot collision
by applying all of the following changes:
   - sci-libs/m4ri-20200125 (Change USE: +png)
   - sci-libs/brial-1.2.8 (Change USE: +png)

The following USE changes are necessary to proceed:
 (see "package.use" in the portage(5) man page for more details)
# required by sci-libs/brial-1.2.8::sage-on-gentoo
# required by sci-libs/brial (argument)
>=sci-libs/m4ri-20200125 -png 

is solved with png for sci-libs/brial in latest image

(checked on u/mkoeppe/fix_tox_docker_builds_broken_by__29884)

comment:90 Changed 17 months ago by gh-sheerluck

-----------------
All tests passed!
-----------------
____ summary ____
docker-gentoo-standard: commands succeeded
congratulations :)

comment:91 Changed 17 months ago by gh-sheerluck

I've created latest-gcc10 in addition to latest. For fun.

 version in            latest         latest-gcc10
====================================================
 sys-devel/gcc         9.3.0          10.1.0-r2
 sci-libs/arpack       3.1.5          3.5.0
 sci-mathematics/gp2c  0.0.11.2-r1    0.0.11.3
 app-misc/rlwrap       0.42           0.43

But tox -e docker-gentoo-gcc10-maximal -- build doc-html doc-pdf ptest says

checking gcc version... 10.1.0
checking g++ version... 10.1.0
configure: Installing GCC because g++ -std=gnu++11 is g++ version 10.1.0, which is too recent for this version of Sage 
configure: no suitable system package found for SPKG gcc

Sage is not ready for gcc10 yet...

but if anyone wants to play with latest-gcc10 it's available on hub.docker.com

comment:92 Changed 17 months ago by dimpase

Thanks. A related latest annoyance is that on macOS Homebrew provides gcc package, which is gcc-10, and this breaks things (even though only gfortran is used, not gcc).

comment:93 Changed 16 months ago by gh-sheerluck

What’s New in sheerluck/sage-on-gentoo-stage4:20.8

1) base image:
   FROM gentoo/portage:20200801 as portage
   FROM gentoo/stage3-amd64:20200801

2) updates:
   arb                2.16.0  -> 2.18.0
   arpack             3.1.5   -> 3.5.0
   cython             0.29.19 -> 0.29.21 
   flint              2.5.2   -> 2.6.0
   fplll              5.3.2   -> 5.3.3
   gp2c               0.11.2  -> 0.11.3
   ipython            7.5.0   -> 7.16.1
   matplotlib         3.1.2   -> 3.2.2
   numpy              1.17.4  -> 1.19.1
   pandas             1.0.1   -> 1.0.5
   pandoc             2.10    -> 2.10.1
   pillow             7.0.0   -> 7.1.2
   python             3.7.7   -> 3.7.8
   scipy              1.4.1   -> 1.5.2 
   sqlite             3.31.1  -> 3.32.3

no changes to tox.ini is required

comment:94 Changed 16 months ago by gh-sheerluck

Good news, everyone! We used one latest image, now we have two images:

  • latest with python-3.8.5
  • latest-py37 with python-3.7.8
checking ... whether /usr/bin/python3 is good... no, Python 3.8.5 is too recent

no changes to tox.ini is required, unless we want to be prepared for #29023 when most of installed packages would be used, in that case we can have both "gentoo" and "gentoo-py37" in tox.ini

comment:95 Changed 16 months ago by mkoeppe

Great! You may want to test with #27754, which aims for Python 3.7 compatibility

Version 0, edited 16 months ago by mkoeppe (next)

comment:96 Changed 16 months ago by fbissey

For the record sage-on-gentoo enables building with python 3.8 since 9.2.beta7. Building of the documentation with it may currently be failing though (but I managed a build between beta6 and beta7 somehow).

comment:97 Changed 15 months ago by gh-sheerluck

Sage is ready for gcc-10, so...

What’s New in sheerluck/sage-on-gentoo-stage4:20.9

1) base image:
   FROM gentoo/portage:20200901 as portage
   FROM gentoo/stage3-amd64:20200901

2) gcc-10 + python-3.8.5

3) updates:
   arb                2.18.0  -> 2.18.1
   flint              2.6.0   -> 2.6.3
   glibc              2.30-r8 -> 2.31-r6
   libhomfly          1.02.5  -> 1.02.6
   matplotlib         3.2.2   -> 3.3.1
   mpfr               4.0.2   -> 4.1.0
   sphinx             3.1.2   -> 3.2.1

no changes to tox.ini are required

comment:98 Changed 14 months ago by gh-sheerluck

What’s New in sheerluck/sage-on-gentoo-stage4:20.10

1) base image:
   FROM gentoo/portage:20201001 as portage
   FROM gentoo/stage3:20201001

2) gcc-10 + python-3.8.6

3) updates:
   arpack-3.7.0       new
   tox-3.19.0         new
   brial              1.2.8   -> 1.2.9
   gmpy-2.1.0         b1-r1   -> b5
   guile              2.2.4   -> 2.2.6
   ipython            7.16.1  -> 7.17.0
   sqlite             3.32.3  -> 3.33.0
   swig               3.0.12  -> 4.0.0

sheerluck/sage-on-gentoo-stage4:20.10-py37 is the same, only

2) gcc-9 + python-3.7.9

comment:99 Changed 14 months ago by fbissey

Please mask brial-1.2.9 there are issues with it and it will be replaced by 1.2.10 in short order.

comment:100 Changed 14 months ago by gh-sheerluck

masked, downgraded, squashed, tested, pushed

comment:101 Changed 14 months ago by gh-sheerluck

checking for python3 >= 3.6.0, < 3.10.0 with modules ...  
checking ... whether /usr/bin/python3 is good... yes

Although overlay cschwan/sage-on-gentoo is not ready for python-3.9 yet, develop branch of sagemath/sage accepts python-3.9.

So as if two images weren't enough, I used some ugly hacks to create third image -- latest-py39

If someone finds it usefull to play with python-3.9, tox.ini may be changed this way:

--- a/tox.ini
+++ b/tox.ini
@@ -208,6 +208,7 @@ setenv =
     gentoo:      SYSTEM=gentoo
     gentoo:      BASE_IMAGE=sheerluck/sage-on-gentoo-stage4
     gentoo-python3.7: BASE_TAG=latest-py37
+    gentoo-python3.9: BASE_TAG=latest-py39
     #    
     # https://hub.docker.com/_/archlinux/
     #    

comment:102 Changed 14 months ago by mkoeppe

I have created #30785 for this

comment:103 follow-up: Changed 9 months ago by gh-sheerluck

I was under impression that something is not right with my images until I saw https://trac.sagemath.org/ticket/31395 That bug has ruined monthful of latest https://github.com/mkoeppe/sage/actions/ :(

comment:104 in reply to: ↑ 103 ; follow-up: Changed 9 months ago by mjo

Replying to gh-sheerluck:

I was under impression that something is not right with my images until I saw https://trac.sagemath.org/ticket/31395 That bug has ruined monthful of latest https://github.com/mkoeppe/sage/actions/ :(

I'm going to post a branch that removes the memlimit and then turn off my computer for a month while people yell at me.

comment:105 in reply to: ↑ 104 Changed 8 months ago by dimpase

Replying to mjo:

Replying to gh-sheerluck:

I was under impression that something is not right with my images until I saw https://trac.sagemath.org/ticket/31395 That bug has ruined monthful of latest https://github.com/mkoeppe/sage/actions/ :(

I'm going to post a branch that removes the memlimit and then turn off my computer for a month while people yell at me.

see, #31395 got in :-)

comment:106 Changed 3 months ago by gh-sheerluck

latest image contains clang-12.0.1 for tox -e docker-gentoo-maximal

comment:107 Changed 7 weeks ago by gh-sheerluck

At https://github.com/orlitzky/sage/runs/3851208039?check_suite_focus=true#step:7:11493 I see Pytest is not installed, skip checking tests that rely on it.

Well, that is not true, latest image got /usr/bin/py.test and /usr/bin/pytest

comment:108 Changed 7 weeks ago by dimpase

you need pytest in sage's venv.

e.g.

make pytest

will do the trick

Note: See TracTickets for help on using tickets.