Opened 12 years ago
Closed 8 years ago
#6255 closed defect (duplicate)
update doc system to jsmath and improve build system (parallel doc builds)
Reported by: | ncalexan | Owned by: | mhansen |
---|---|---|---|
Priority: | major | Milestone: | sage-duplicate/invalid/wontfix |
Component: | documentation | Keywords: | documentation build sphinx parallel |
Cc: | mhansen, craigcitro | Merged in: | |
Authors: | Reviewers: | Florent Hivert | |
Report Upstream: | N/A | Work issues: | |
Branch: | Commit: | ||
Dependencies: | Stopgaps: |
Description
This is a reminder ticket for mhansen.
Attachments (2)
Change History (12)
comment:1 Changed 12 years ago by
- Cc craigcitro added
comment:2 Changed 12 years ago by
- Owner changed from tba to mhansen
- Status changed from new to assigned
- Summary changed from update doc system to latex sphinx and improve build system to update doc system to jsmath and improve build system (parallel doc builds)
comment:3 Changed 11 years ago by
- Report Upstream set to N/A
comment:4 Changed 9 years ago by
I made an attempt to have parallel doc build. It seems that I have it for the write part of the doc generation process. Here is the diff
-
builders/__init__.py
diff --git a/builders/__init__.py b/builders/__init__.py
a b class Builder(object): 286 286 # write target files 287 287 warnings = [] 288 288 self.env.set_warnfunc(lambda *args: warnings.append(args)) 289 #for docname in self.status_iterator( 290 # sorted(docnames), 'writing output... ', darkgreen, len(docnames)): 291 # doctree = self.env.get_and_resolve_doctree(docname, self) 292 # self.write_doc(docname, doctree) 293 from sage.parallel.decorate import parallel 294 import itertools 295 worker = parallel('fork')(self.write_doc_parallel_worker) 296 pariter = itertools.imap(lambda x:x[1], worker(sorted(docnames))) 289 297 for docname in self.status_iterator( 290 sorted(docnames), 'writing output... ', darkgreen, len(docnames)):291 doctree = self.env.get_and_resolve_doctree(docname, self)292 self.write_doc(docname, doctree)298 pariter, 'writing output... ', darkgreen, len(docnames)): 299 # done in the iterator !!! 300 pass 293 301 for warning in warnings: 294 302 self.warn(*warning) 295 303 self.env.set_warnfunc(self.warn) 296 304 305 def write_doc_parallel_worker(self, docname): 306 doctree = self.env.get_and_resolve_doctree(docname, self) 307 self.write_doc(docname, doctree) 308 return docname 309 297 310 def prepare_writing(self, docnames): 298 311 raise NotImplementedError
The read part could be more tricky but it doesn't seems unfeasible. Note that I may be dreaming here.
comment:5 Changed 9 years ago by
Hi there,
I just attached two patches. They need to be applied in
$SAGE_ROOT/local/lib/python2.7/site-packages/Sphinx-1.1.2-py2.7.egg/sphinx/
(I didn't regenerate a spkg yet). Those two packages are very experimental and
they certainly break a lot of things. The goal of timing.patch
is to
improve Sphinx timning and progress report. The second one uses
@parallel
to parallelize the writing part of the doc generation. This is
very raw and could certainly be optimized using Pool, Queue and the
like. Still the improvement is allready here:
On a intel i7 8 multithreaded core:
- serie:
reading sources... Elapsed time = 385.334967136 writing output... Elapsed time = 1903.10733795
- parallel:
reading sources... Elapsed time = 418.675282001 writing output... Elapsed time = 253.907614946
On a 24 core server:
- serie:
reading sources... Elapsed time = 243.982397079 writing output... Elapsed time = 1366.98643208
- parallel:
reading sources... Elapsed time = 243.729380131 writing output... Elapsed time = 176.76424408
Florent
comment:6 Changed 9 years ago by
With a little tunning I managed to have
serie writing output... Elapsed time = 1366.98643208 parallel writing output... Elapsed time = 106.421586037
Looks efficient !
comment:7 Changed 8 years ago by
- Status changed from new to needs_review
This should be closed as a duplicate for #6495.
comment:8 Changed 8 years ago by
- Reviewers set to Florent Hivert
- Status changed from needs_review to positive_review
comment:9 Changed 8 years ago by
- Milestone changed from sage-5.11 to sage-duplicate/invalid/wontfix
comment:10 Changed 8 years ago by
- Resolution set to duplicate
- Status changed from positive_review to closed
Please see #6495 for a tentative approach to parallel doc builds