Opened 10 years ago

Closed 6 years ago

#6255 closed defect (duplicate)

update doc system to jsmath and improve build system (parallel doc builds)

Reported by: ncalexan Owned by: mhansen
Priority: major Milestone: sage-duplicate/invalid/wontfix
Component: documentation Keywords: documentation build sphinx parallel
Cc: mhansen, craigcitro Merged in:
Authors: Reviewers: Florent Hivert
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description

This is a reminder ticket for mhansen.

Attachments (2)

timing.patch (1.2 KB) - added by hivert 7 years ago.
Experimental timing patch
paral.patch (2.0 KB) - added by hivert 7 years ago.
Experimental parallel doc output patch

Download all attachments as: .zip

Change History (12)

comment:1 Changed 10 years ago by craigcitro

  • Cc craigcitro added

comment:2 Changed 10 years ago by mhansen

  • Owner changed from tba to mhansen
  • Status changed from new to assigned
  • Summary changed from update doc system to latex sphinx and improve build system to update doc system to jsmath and improve build system (parallel doc builds)

comment:3 Changed 10 years ago by mpatel

  • Report Upstream set to N/A

Please see #6495 for a tentative approach to parallel doc builds

comment:4 Changed 7 years ago by hivert

I made an attempt to have parallel doc build. It seems that I have it for the write part of the doc generation process. Here is the diff

  • builders/__init__.py

    diff --git a/builders/__init__.py b/builders/__init__.py
    a b class Builder(object): 
    286286        # write target files
    287287        warnings = []
    288288        self.env.set_warnfunc(lambda *args: warnings.append(args))
     289        #for docname in self.status_iterator(
     290        #    sorted(docnames), 'writing output... ', darkgreen, len(docnames)):
     291        #    doctree = self.env.get_and_resolve_doctree(docname, self)
     292        #    self.write_doc(docname, doctree)
     293        from sage.parallel.decorate import parallel
     294        import itertools
     295        worker = parallel('fork')(self.write_doc_parallel_worker)
     296        pariter = itertools.imap(lambda x:x[1], worker(sorted(docnames)))
    289297        for docname in self.status_iterator(
    290             sorted(docnames), 'writing output... ', darkgreen, len(docnames)):
    291             doctree = self.env.get_and_resolve_doctree(docname, self)
    292             self.write_doc(docname, doctree)
     298            pariter, 'writing output... ', darkgreen, len(docnames)):
     299            # done in the iterator !!!
     300            pass
    293301        for warning in warnings:
    294302            self.warn(*warning)
    295303        self.env.set_warnfunc(self.warn)
    296304
     305    def write_doc_parallel_worker(self, docname):
     306        doctree = self.env.get_and_resolve_doctree(docname, self)
     307        self.write_doc(docname, doctree)
     308        return docname
     309
    297310    def prepare_writing(self, docnames):
    298311        raise NotImplementedError

The read part could be more tricky but it doesn't seems unfeasible. Note that I may be dreaming here.

Changed 7 years ago by hivert

Experimental timing patch

Changed 7 years ago by hivert

Experimental parallel doc output patch

comment:5 Changed 7 years ago by hivert

Hi there,

I just attached two patches. They need to be applied in

$SAGE_ROOT/local/lib/python2.7/site-packages/Sphinx-1.1.2-py2.7.egg/sphinx/

(I didn't regenerate a spkg yet). Those two packages are very experimental and they certainly break a lot of things. The goal of timing.patch is to improve Sphinx timning and progress report. The second one uses @parallel to parallelize the writing part of the doc generation. This is very raw and could certainly be optimized using Pool, Queue and the like. Still the improvement is allready here: On a intel i7 8 multithreaded core:

  • serie:
    reading sources...  Elapsed time = 385.334967136
    writing output...  Elapsed time = 1903.10733795
    
  • parallel:
    reading sources...  Elapsed time = 418.675282001
    writing output...  Elapsed time = 253.907614946
    

On a 24 core server:

  • serie:
    reading sources...  Elapsed time = 243.982397079
    writing output...  Elapsed time = 1366.98643208
    
  • parallel:
    reading sources...  Elapsed time = 243.729380131
    writing output...  Elapsed time = 176.76424408
    

Florent

comment:6 Changed 7 years ago by hivert

With a little tunning I managed to have

serie    writing output...  Elapsed time = 1366.98643208
parallel writing output...  Elapsed time = 106.421586037

Looks efficient !

comment:7 Changed 6 years ago by hivert

  • Status changed from new to needs_review

This should be closed as a duplicate for #6495.

comment:8 Changed 6 years ago by hivert

  • Reviewers set to Florent Hivert
  • Status changed from needs_review to positive_review

comment:9 Changed 6 years ago by jdemeyer

  • Milestone changed from sage-5.11 to sage-duplicate/invalid/wontfix

comment:10 Changed 6 years ago by jdemeyer

  • Resolution set to duplicate
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.