Opened 11 years ago
Last modified 10 years ago
#12016 closed enhancement
parallelism in Sage: just use value of 'MAKE' — at Version 11
Reported by: | jhpalmieri | Owned by: | GeorgSWeber |
---|---|---|---|
Priority: | critical | Milestone: | sage-4.8 |
Component: | build | Keywords: | |
Cc: | jdemeyer, leif | Merged in: | |
Authors: | John Palmieri, Jeroen Demeyer | Reviewers: | |
Report Upstream: | N/A | Work issues: | |
Branch: | Commit: | ||
Dependencies: | #11969 | Stopgaps: |
Description (last modified by )
With the attached patches, along with the changes from #11959, the various parallel aspects of Sage should be controlled by setting the -j
flag in MAKE
. That is, if MAKE='make -j16'
, then
- running
make
will build spkg's in parallel, using 16 processes (this was done in #11959). This is standardmake
behaviour, no patches are needed.
- running
make ptestlong
orsage -tp 0 <files>
will doctest in parallel using 16 threads. If the-j
flag inMAKE
is not set, then determine the number of threads as before:min(8, cpu_count())
.
- running
./sage -b
will build the Sage library using 16 threads. If the-j
flag inMAKE
is not set, then use only 1 thread.
In #6495, we should implement the same behavior for doc building.
Apply:
- trac_12016-root.v2.patch to the
SAGE_ROOT
repository. - trac_12016-scripts.v2.patch to the
SCRIPTS
repository. - trac_12016-sage.v2.patch to the Sage library.
Change History (14)
comment:1 Changed 11 years ago by
comment:2 Changed 11 years ago by
- Description modified (diff)
comment:3 Changed 11 years ago by
- Description modified (diff)
comment:4 Changed 11 years ago by
- Description modified (diff)
comment:5 Changed 11 years ago by
- Description modified (diff)
comment:6 follow-up: ↓ 7 Changed 11 years ago by
- Description modified (diff)
John, with your solution there is a lot of code duplication (determining the number of threads is done in 3 places, potentially in 3 different ways). How about having code in sage-sage
or sage-env
to determine the number of threads and saving it in an environment variable SAGE_NUM_PROCESSES
(which the user could set by hand; if not set, the value comes from MAKE
or MAKEFLAGS
; if no -j
option is given, set to 1).
comment:7 in reply to: ↑ 6 ; follow-up: ↓ 9 Changed 11 years ago by
Replying to jdemeyer:
John, with your solution there is a lot of code duplication (determining the number of threads is done in 3 places, potentially in 3 different ways). How about having code in
sage-sage
orsage-env
to determine the number of threads and saving it in an environment variableSAGE_NUM_PROCESSES
Sounds okay.
(which the user could set by hand; if not set, the value comes from
MAKE
orMAKEFLAGS
; if no-j
option is given, set to 1).
If you run "sage -tp <files>", should you use 1 process or more than 1? The "-tp" option means "parallel", so perhaps the default should be more than 1 in this case. In other cases (like docbuilding, for example), the default should be 1.
comment:8 follow-up: ↓ 10 Changed 11 years ago by
For something like make -j16 ptestlong
, how do we recover the number 16? If I execute this command (with MAKE
unset), I see
MAKEFLAGS= --jobserver-fds=3,4 -j MFLAGS=- --jobserver-fds=3,4 -j
but I don't see "16" anywhere in the listing of the environment variables.
comment:9 in reply to: ↑ 7 Changed 11 years ago by
Replying to jhpalmieri:
Replying to jdemeyer:
John, with your solution there is a lot of code duplication (determining the number of threads is done in 3 places, potentially in 3 different ways). How about having code in
sage-sage
orsage-env
to determine the number of threads and saving it in an environment variableSAGE_NUM_PROCESSES
Sounds okay.
(which the user could set by hand; if not set, the value comes from
MAKE
orMAKEFLAGS
; if no-j
option is given, set to 1).If you run "sage -tp <files>", should you use 1 process or more than 1? The "-tp" option means "parallel", so perhaps the default should be more than 1 in this case. In other cases (like docbuilding, for example), the default should be 1.
Sure, that is what I meant. We should compute the value once, but in sage -tp
we can still decide to use the number of processes.
comment:10 in reply to: ↑ 8 Changed 11 years ago by
Replying to jhpalmieri:
For something like
make -j16 ptestlong
, how do we recover the number 16? If I execute this command (withMAKE
unset), I seeMAKEFLAGS= --jobserver-fds=3,4 -j MFLAGS=- --jobserver-fds=3,4 -jbut I don't see "16" anywhere in the listing of the environment variables.
You are right. I had not tried this before. So let's scrap that idea.
Changed 11 years ago by
Changed 11 years ago by
comment:11 Changed 11 years ago by
- Description modified (diff)
- Status changed from new to needs_review
Here are new patches. These use SAGE_NUM_THREADS
if it is set, and otherwise try to extract a number from MAKE
. (My method for doing this is probably not ideal, but the options This is done in sage-env. Running sage -b
should use this setting now, also.
I don't know how to get the number of threads from
make -j16 ptestlong
so I removed that from the "to do" list in the ticket description.
In the file sage-ptest, I removed the "FIXME" comment in
try: # FIXME: Nice, but <NUMTHREADS> should immediately follow '-tp' etc., # i.e., be the next argument. We might have file or directory # names that properly convert to an int... numthreads = int(argv[1]) infiles = argv[2:] except ValueError: # can't convert first arg to an integer: arg was probably omitted numthreads = 1
The script sage-ptest doesn't get a "tp" argument; it is instead called by sage-sage, and the way it is called, the first argument to sage-ptest is precisely what ever came after "-tp". So I don't think anything needs fixing. If we ever rewrite sage-sage (#21) to properly parse arguments, we can make sure that "-tp" has a default numerical argument of zero.
We should remove
NUM_THREADS
from the top-levelMakefile
.