Opened 11 years ago

Last modified 10 years ago

#12016 closed enhancement

parallelism in Sage: just use value of 'MAKE' — at Version 6

Reported by: jhpalmieri Owned by: GeorgSWeber
Priority: critical Milestone: sage-4.8
Component: build Keywords:
Cc: jdemeyer, leif Merged in:
Authors: John Palmieri, Jeroen Demeyer Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: #11969 Stopgaps:

Status badges

Description (last modified by jdemeyer)

With the attached patches, along with the changes from #11959, the various parallel aspects of Sage should be controlled by setting the -j flag in MAKE. That is, if MAKE='make -j16', then

  • running make will build spkg's in parallel, using 16 processes (this was done in #11959). This is standard make behaviour, no patches are needed.
  • running make ptestlong or sage -tp 0 <files> will doctest in parallel using 16 threads. If the -j flag in MAKE is not set, then determine the number of threads as before: min(8, cpu_count()).
  • TODO: running ./sage -b will build the Sage library using 16 threads. If the -j flag in MAKE is not set, then use only 1 thread.

In #6495, we should implement the same behavior for doc building.

TODO: We should also support:

  • make -j16 ptestlong (as opposed to MAKE="make -j16" make ptestlong). Use MAKEFLAGS.
  • make -j (unlimited number of jobs, say set to 9999 if we really need a number).
  • make -j37 -j1 (last option takes precedence).

Apply:

  1. 12016-root.v2.patch to the SAGE_ROOT repository.
  2. 12016-scripts.v2.patch to the SCRIPTS repository.
  3. 12016-sage.v2.patch to the Sage library.

Change History (6)

comment:1 Changed 11 years ago by jdemeyer

We should remove NUM_THREADS from the top-level Makefile.

comment:2 Changed 11 years ago by jdemeyer

  • Authors changed from John Palmieri to John Palmieri, Jeroen Demeyer
  • Description modified (diff)

comment:3 Changed 11 years ago by jdemeyer

  • Description modified (diff)

comment:4 Changed 11 years ago by jdemeyer

  • Description modified (diff)

comment:5 Changed 11 years ago by jdemeyer

  • Description modified (diff)

comment:6 Changed 11 years ago by jdemeyer

  • Description modified (diff)

John, with your solution there is a lot of code duplication (determining the number of threads is done in 3 places, potentially in 3 different ways). How about having code in sage-sage or sage-env to determine the number of threads and saving it in an environment variable SAGE_NUM_PROCESSES (which the user could set by hand; if not set, the value comes from MAKE or MAKEFLAGS; if no -j option is given, set to 1).

Note: See TracTickets for help on using tickets.