Opened 11 years ago

Last modified 10 years ago

#12016 closed enhancement

parallelism in Sage: just use value of 'MAKE' — at Version 11

Reported by: jhpalmieri Owned by: GeorgSWeber
Priority: critical Milestone: sage-4.8
Component: build Keywords:
Cc: jdemeyer, leif Merged in:
Authors: John Palmieri, Jeroen Demeyer Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: #11969 Stopgaps:

Status badges

Description (last modified by jhpalmieri)

With the attached patches, along with the changes from #11959, the various parallel aspects of Sage should be controlled by setting the -j flag in MAKE. That is, if MAKE='make -j16', then

  • running make will build spkg's in parallel, using 16 processes (this was done in #11959). This is standard make behaviour, no patches are needed.
  • running make ptestlong or sage -tp 0 <files> will doctest in parallel using 16 threads. If the -j flag in MAKE is not set, then determine the number of threads as before: min(8, cpu_count()).
  • running ./sage -b will build the Sage library using 16 threads. If the -j flag in MAKE is not set, then use only 1 thread.

In #6495, we should implement the same behavior for doc building.

Apply:

  1. trac_12016-root.v2.patch to the SAGE_ROOT repository.
  2. trac_12016-scripts.v2.patch to the SCRIPTS repository.
  3. trac_12016-sage.v2.patch to the Sage library.

Change History (14)

comment:1 Changed 11 years ago by jdemeyer

We should remove NUM_THREADS from the top-level Makefile.

comment:2 Changed 11 years ago by jdemeyer

  • Authors changed from John Palmieri to John Palmieri, Jeroen Demeyer
  • Description modified (diff)

comment:3 Changed 11 years ago by jdemeyer

  • Description modified (diff)

comment:4 Changed 11 years ago by jdemeyer

  • Description modified (diff)

comment:5 Changed 11 years ago by jdemeyer

  • Description modified (diff)

comment:6 follow-up: Changed 11 years ago by jdemeyer

  • Description modified (diff)

John, with your solution there is a lot of code duplication (determining the number of threads is done in 3 places, potentially in 3 different ways). How about having code in sage-sage or sage-env to determine the number of threads and saving it in an environment variable SAGE_NUM_PROCESSES (which the user could set by hand; if not set, the value comes from MAKE or MAKEFLAGS; if no -j option is given, set to 1).

comment:7 in reply to: ↑ 6 ; follow-up: Changed 11 years ago by jhpalmieri

Replying to jdemeyer:

John, with your solution there is a lot of code duplication (determining the number of threads is done in 3 places, potentially in 3 different ways). How about having code in sage-sage or sage-env to determine the number of threads and saving it in an environment variable SAGE_NUM_PROCESSES

Sounds okay.

(which the user could set by hand; if not set, the value comes from MAKE or MAKEFLAGS; if no -j option is given, set to 1).

If you run "sage -tp <files>", should you use 1 process or more than 1? The "-tp" option means "parallel", so perhaps the default should be more than 1 in this case. In other cases (like docbuilding, for example), the default should be 1.

comment:8 follow-up: Changed 11 years ago by jhpalmieri

For something like make -j16 ptestlong, how do we recover the number 16? If I execute this command (with MAKE unset), I see

MAKEFLAGS= --jobserver-fds=3,4 -j
MFLAGS=- --jobserver-fds=3,4 -j

but I don't see "16" anywhere in the listing of the environment variables.

comment:9 in reply to: ↑ 7 Changed 11 years ago by jdemeyer

Replying to jhpalmieri:

Replying to jdemeyer:

John, with your solution there is a lot of code duplication (determining the number of threads is done in 3 places, potentially in 3 different ways). How about having code in sage-sage or sage-env to determine the number of threads and saving it in an environment variable SAGE_NUM_PROCESSES

Sounds okay.

(which the user could set by hand; if not set, the value comes from MAKE or MAKEFLAGS; if no -j option is given, set to 1).

If you run "sage -tp <files>", should you use 1 process or more than 1? The "-tp" option means "parallel", so perhaps the default should be more than 1 in this case. In other cases (like docbuilding, for example), the default should be 1.

Sure, that is what I meant. We should compute the value once, but in sage -tp we can still decide to use the number of processes.

comment:10 in reply to: ↑ 8 Changed 11 years ago by jdemeyer

Replying to jhpalmieri:

For something like make -j16 ptestlong, how do we recover the number 16? If I execute this command (with MAKE unset), I see

MAKEFLAGS= --jobserver-fds=3,4 -j
MFLAGS=- --jobserver-fds=3,4 -j

but I don't see "16" anywhere in the listing of the environment variables.

You are right. I had not tried this before. So let's scrap that idea.

Changed 11 years ago by jhpalmieri

Changed 11 years ago by jhpalmieri

comment:11 Changed 11 years ago by jhpalmieri

  • Description modified (diff)
  • Status changed from new to needs_review

Here are new patches. These use SAGE_NUM_THREADS if it is set, and otherwise try to extract a number from MAKE. (My method for doing this is probably not ideal, but the options This is done in sage-env. Running sage -b should use this setting now, also.

I don't know how to get the number of threads from

make -j16 ptestlong

so I removed that from the "to do" list in the ticket description.

In the file sage-ptest, I removed the "FIXME" comment in

    try:
        # FIXME: Nice, but <NUMTHREADS> should immediately follow '-tp' etc.,
        #        i.e., be the next argument. We might have file or directory
        #        names that properly convert to an int...
        numthreads = int(argv[1])
        infiles = argv[2:]
    except ValueError: # can't convert first arg to an integer: arg was probably omitted
        numthreads = 1

The script sage-ptest doesn't get a "tp" argument; it is instead called by sage-sage, and the way it is called, the first argument to sage-ptest is precisely what ever came after "-tp". So I don't think anything needs fixing. If we ever rewrite sage-sage (#21) to properly parse arguments, we can make sure that "-tp" has a default numerical argument of zero.

Changed 11 years ago by jhpalmieri

Note: See TracTickets for help on using tickets.