Opened 2 years ago

Closed 22 months ago

#21389 closed enhancement (fixed)

docbuild: Don't use multiprocessing when SAGE_NUM_THREADS=1

Reported by: embray Owned by:
Priority: minor Milestone: sage-7.5
Component: documentation Keywords: docbuild
Cc: tscrim Merged in:
Authors: Erik Bray Reviewers: Jeroen Demeyer
Report Upstream: N/A Work issues:
Branch: 4165010 (Commits) Commit: 4165010efd074f431c5ddaab5eb23bcf9e1e59f6
Dependencies: #21044 Stopgaps:

Description

This is a slight tweak to the docbuild code to avoid the overhead of using multiprocessing.Pool for a single-process doc build.

On Cygwin particularly the docbuild crashes when using multiprocessing. This seems to be a Cygwin fork problem. Nevertheless this simple change makes sense in general (even if single-process doc builds are not common on any other platform).

Change History (16)

comment:1 Changed 2 years ago by embray

  • Keywords docbuild added
  • Status changed from new to needs_review

comment:2 Changed 2 years ago by tscrim

  • Cc tscrim added

comment:3 Changed 2 years ago by jdemeyer

  • Reviewers set to Jeroen Demeyer

comment:4 Changed 2 years ago by jdemeyer

  • Status changed from needs_review to needs_work

This actually breaks a single-process build, it seems that the forking is really required:

[dochtml] [categorie] /usr/local/src/sage-config/local/lib/python2.7/site-packages/sage/categories/finite_groups.py:docstring of sage.categories.finite_groups.FiniteGroups.ParentMethods.cardinality:4: WARNING: more than one target found for cross-reference u'order': sage.combinat.diagram_algebras.DiagramAlgebra.order, sage.categories.groups.Groups.CartesianProducts.ParentMethods.order, sage.algebras.quatalg.quaternion_algebra.QuaternionAlgebra_abstract.order, sage.algebras.steenrod.steenrod_algebra.SteenrodAlgebra_generic.order
[dochtml] [categorie] /usr/local/src/sage-config/local/lib/python2.7/site-packages/sage/categories/finite_groups.py:docstring of sage.categories.finite_groups.FiniteGroups.ParentMethods.cardinality:4: WARNING: more than one target found for cross-reference u'order': sage.combinat.diagram_algebras.DiagramAlgebra.order, sage.categories.groups.Groups.CartesianProducts.ParentMethods.order, sage.algebras.quatalg.quaternion_algebra.QuaternionAlgebra_abstract.order, sage.algebras.steenrod.steenrod_algebra.SteenrodAlgebra_generic.order
[dochtml] [categorie] /usr/local/src/sage-config/local/lib/python2.7/site-packages/sage/categories/groups.py:docstring of sage.categories.groups.Groups.ParentMethods.group_generators:3: WARNING: more than one target found for cross-reference u'gens': sage.algebras.shuffle_algebra.DualPBWBasis.gens, sage.algebras.group_algebra.GroupAlgebra.gens, sage.algebras.free_algebra.FreeAlgebra_generic.gens, sage.categories.pushout.PermutationGroupFunctor.gens, sage.rings.asymptotic.growth_group.ExponentialGrowthGroup.gens, sage.algebras.free_zinbiel_algebra.FreeZinbielAlgebra.gens, sage.rings.asymptotic.asymptotic_ring.AsymptoticRing.gens, sage.algebras.quatalg.quaternion_algebra.QuaternionOrder.gens, sage.algebras.jordan_algebra.SpecialJordanAlgebra.gens, sage.combinat.free_prelie_algebra.FreePreLieAlgebra.gens, sage.algebras.jordan_algebra.JordanAlgebraSymmetricBilinear.gens, sage.algebras.steenrod.steenrod_algebra.SteenrodAlgebra_generic.gens, sage.rings.asymptotic.growth_group.GenericGrowthGroup.gens, sage.algebras.yokonuma_hecke_algebra.YokonumaHeckeAlgebra.gens, sage.algebras.free_algebra.PBWBasisOfFreeAlgebra.gens, sage.algebras.quatalg.quaternion_algebra.QuaternionFractionalIdeal_rational.gens, sage.algebras.clifford_algebra.CliffordAlgebra.gens, sage.algebras.shuffle_algebra.ShuffleAlgebra.gens

comment:5 Changed 2 years ago by embray

  • Dependencies set to #21044
  • Status changed from needs_work to needs_review

It's not so much that this breaks a single-process build, as single process builds are broken for other reasons outside the scope of this ticket. For example: #21044

I think maybe this is not a problem without forking if it results in each document being built in a separate process though. Try this with #21044 and see if that fixes it.

comment:6 Changed 2 years ago by jdemeyer

The doc seems to build now. I haven't looked at the patch yet.

comment:7 Changed 2 years ago by jdemeyer

  • Milestone changed from sage-7.4 to sage-7.5
  • Status changed from needs_review to needs_work

Shouldn't this be in the single-threaded builder also:

        try:
            ...
        except Exception:
            if ABORT_ON_ERROR:
                raise

I must admit that I don't know its full purpose, but it should be there for consistency.

Other than that, this looks good.

comment:8 Changed 23 months ago by embray

I was't sure what it was for either at first--but I believe you're right that there should be a similar try/except.

comment:9 Changed 23 months ago by git

  • Commit changed from 17c5cb4cc02139b465b596aba2209bbc45285c1f to 3d36ba524a077cbef73fe6a9a6b66393cb9be0d9

Branch pushed to git repo; I updated commit sha1. New commits:

3d36ba5If an exception occurs while building one of the docs ignore the exception and continue building other docs unless ABORT_ON_ERROR, as in the multiprocessing case.

comment:10 Changed 23 months ago by embray

  • Status changed from needs_work to needs_review

comment:11 Changed 23 months ago by jdemeyer

  • Status changed from needs_review to needs_work

I get

[dochtml] 
[dochtml] Building reference manual, first pass.
[dochtml] 
[dochtml] Error building the documentation.
[dochtml] Traceback (most recent call last):
[dochtml]   File "/usr/local/src/sage-config/local/lib/python/runpy.py", line 162, in _run_module_as_main
[dochtml]     "__main__", fname, loader, pkg_name)
[dochtml]   File "/usr/local/src/sage-config/local/lib/python/runpy.py", line 72, in _run_code
[dochtml]     exec code in run_globals
[dochtml]   File "/usr/local/src/sage-config/local/lib/python2.7/site-packages/sage_setup/docbuild/__main__.py", line 2, in <module>
[dochtml]     main()
[dochtml]   File "/usr/local/src/sage-config/local/lib/python2.7/site-packages/sage_setup/docbuild/__init__.py", line 1638, in main
[dochtml]     builder()
[dochtml]   File "/usr/local/src/sage-config/local/lib/python2.7/site-packages/sage_setup/docbuild/__init__.py", line 315, in _wrapper
[dochtml]     getattr(get_builder(document), 'inventory')(*args, **kwds)
[dochtml]   File "/usr/local/src/sage-config/local/lib/python2.7/site-packages/sage_setup/docbuild/__init__.py", line 509, in _wrapper
[dochtml]     build_many(build_ref_doc, L)
[dochtml]   File "/usr/local/src/sage-config/local/lib/python2.7/site-packages/sage_setup/docbuild/__init__.py", line 265, in build_many
[dochtml]     results.append(target(*arg))
[dochtml] TypeError: build_ref_doc() takes exactly 1 argument (4 given)

comment:12 Changed 23 months ago by embray

Ah, of course. There shouldn't be a * there.

comment:13 Changed 23 months ago by git

  • Commit changed from 3d36ba524a077cbef73fe6a9a6b66393cb9be0d9 to 4165010efd074f431c5ddaab5eb23bcf9e1e59f6

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

4165010If an exception occurs while building one of the docs ignore the exception and continue building other docs unless ABORT_ON_ERROR, as in the multiprocessing case.

comment:14 Changed 23 months ago by embray

  • Status changed from needs_work to needs_review

New commits:

4165010If an exception occurs while building one of the docs ignore the exception and continue building other docs unless ABORT_ON_ERROR, as in the multiprocessing case.
Last edited 23 months ago by embray (previous) (diff)

comment:15 Changed 23 months ago by jdemeyer

  • Status changed from needs_review to positive_review

comment:16 Changed 22 months ago by vbraun

  • Branch changed from u/embray/docbuild-single-proc to 4165010efd074f431c5ddaab5eb23bcf9e1e59f6
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.