src/doc/bootstrap: Generate list of packages (with info and links) for ref manual from build/pkgs

[mkoeppe]

This ticket converts all SPKG.txt files (from an ad-hoc version of mediawiki markup) to SPKG.rst.

It adds to src/doc/bootstrap a script that generate a full list of packages that Sage uses (with links to documentation - see #27495) for the reference manual from information in build/pkgs.

[mkoeppe]

For example, I suppose we could transform the ad-hoc markup in SPKG.txt to rst and then include these files in the manual...

[mkoeppe]

Hmm. is the format of SPKG.txt mediawiki? Then we can just convert it using pandoc

---

New commits:

​e1da294

build/bin/sage-spkg-info: Factor out from build/bin/sage-spkg

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​fde28d6

Accept SPKG.rst in place of SPKG.txt

[mkoeppe]

Replying to mkoeppe:

Hmm. is the format of SPKG.txt mediawiki? Then we can just convert it using pandoc

for f in build/pkgs/*/SPKG.txt; do pandoc --from=mediawiki --to=rst $f --output=$(dirname $f)/SPKG.rst; git rm -f $f; git add $(dirname $f)/SPKG.rst; done'

works reasonably well

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​aa9ada9	Delete changelog sections from all SPKG.txt files
​079224e	Remove remaining SPKG.txt changelog sections manually
​38be0f0	build/bin/sage-spkg-info: Factor out from build/bin/sage-spkg
​eff0d77	Accept SPKG.rst in place of SPKG.txt

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d57d9c5

Convert SPKG.txt to SPKG.rst

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​ac5ffd5	Remove rst labels
​d74cbcb	build/pkgs/*/SPKG.rst: Some manual format fixes
​6257875	Generate External Packages chapter for the reference manual

[dimpase]

All hail Pandoc!

[dimpase]

OK, this works, and looks nice in the reference manual, too.

[mkoeppe]

Thanks!

[jhpalmieri]

Fixes so PDF docs build.

---

New commits:

​8cc4f28

trac 29633: fixes for PDF docbuilding

[mkoeppe]

Thank you!

[mkoeppe]

Merged in the new rc to resolve a merge conflict.

---

New commits:

​32bb2fe

Merge tag '9.1.rc3' into t/29633/src_doc_bootstrap__generate_list_of_packages__with_links_to_documentation__for_ref_manual_from_build_pkgs

[mkoeppe]

Updated the ticket description to what is actually in the branch.

Follow-up = #29655.

[vbraun]

PDF docs fail

[vbraun]

Also this

File "src/sage/tests/cmdline.py", line 217, in sage.tests.cmdline.test_executable
Failed example:
    print(out)  # optional - build
Expected:
    Found local metadata for sqlite-...
    = SQLite =
    ...
    SQLite is a software library that implements a self-contained,
    serverless, zero-configuration, transactional SQL database engine.
    ...
Got:
    Found local metadata for sqlite-3290000
    SQLite
    ======
    <BLANKLINE>
    Description
    -----------
    <BLANKLINE>
    SQLite is a software library that implements a self-contained,
    serverless, zero-configuration, transactional SQL database engine.
    <BLANKLINE>
    License

[vbraun]

Also unmerging this ticket breaks builds, "make clean" apparently doesn't delete the new spkg docs directory

[jhpalmieri]

Replying to vbraun:

PDF docs fail

I can't reproduce this. Can you provide more details?

[fbissey]

pdf failure in spkg

[fbissey]

I can reproduce the failure in sage-on-gentoo. Find the log in attachment. I also have the full generated spkg.tex if necessary, but this is where it fails

\section{Description}
\label{\detokenize{database_jones_numfield:description}}
This is a table of number fields with bounded ramification and degree
≤6.

it shokes on the symbol before the 6. Probably misinterpreted unicode.

[dimpase]

I guess it is ≤ somewhere in the text.

1005	Chapter 45.
1006	! Missing $ inserted.
1007	<inserted text> 
1008	                $
1009	l.1955 ≤
1010	          6.
1011	?
1012	! Emergency stop.
1013	<inserted text> 
1014	                $
1015	l.1955 ≤
1016	          6.

This one?

$ grep -R "≤" src/doc/
src/doc/en/reference/spkg/database_jones_numfield.rst:≤6.

So one would need to modify this package, don't know how.

[fbissey]

Yes, that one. Just replace the freaking symbol with a bit of text.

lower or equal to 6.

there is no imperative for a symbolic short hand there.

[dimpase]

well, yes, but how does one modify that jones.sobj which lives in the package's file, database_jones_numfield-4.tar.gz?

[fbissey]

Ah! I see, you don't understand how this ticket works. You have to edit build/pkg/database_jones_numfield/SPKG.rst which contains the original offending character.

[dimpase]

Replying to fbissey:

Ah! I see, you don't understand how this ticket works. You have to edit build/pkg/database_jones_numfield/SPKG.rst which contains the original offending character.

oops, I see. This is easy to fix indeed. Matthias, can you do this on your branch?

[jhpalmieri]

Replying to jhpalmieri:

Fixes so PDF docs build.

---

New commits:

​8cc4f28

trac 29633: fixes for PDF docbuilding

What happened to this commit? Please use it to fix the PDF docbuilding.

[mkoeppe]

Apparently was lost in some late night merging activity

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​4672d55

trac 29633: fixes for PDF docbuilding

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2ee9bef

Merge tag '9.1' into t/29633/src_doc_bootstrap__generate_list_of_packages__with_links_to_documentation__for_ref_manual_from_build_pkgs

[jhpalmieri]

Fixing the doctest in comment:25.

---

New commits:

​9c7ea5f

trac 29633: fix SQLite doctests in cmdline.py

[mkoeppe]

Ah! Thank you

[jhpalmieri]

See #29745 for a followup. The conversion SPKG.txt -> SPKG.rst introduced some spurious blank lines and other small issues, and I've tried to fix those.