Opened 2 years ago

Last modified 7 days ago

#27495 needs_work enhancement

Use sphinx.ext.extlinks to add Sphinx roles for links to external package docs

Reported by: dimpase Owned by:
Priority: major Milestone: sage-9.4
Component: documentation Keywords: sd111
Cc: thansen, kcrisman, fbissey, embray, schilly, mkoeppe, arojas, gh-mwageringel, vdelecroix Merged in:
Authors: Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by mkoeppe)

We wish to link to external packages' documentation:

  • if the package documentation is installed locally in the Sage distribution, using the file link
  • in downstream packaging, using a file link appropriate for the doc installation scheme of the system (without having to patch the documentation)
  • if it is not available locally, using an http link to an online version of the package's documentation.

To this end, we use the sphinx.ext.extlinks facility, for which we already have infrastructure in sage.misc.sagedoc, assigning a Sphinx role to every external document.

We use a systematic naming scheme for these roles: :spkg_SPKG_DOC: where DOC is usually just doc but if a package has several manuals, it can be used to distinguish them.

In this ticket, we do this for the documents covered by a previous iteration of this ticket:

The location of some of the locally installed documentation files can be determined at runtime, see https://trac.sagemath.org/ticket/27495#comment:58 for maxima. This should probably go through sage.feature.

Related:

Change History (90)

comment:1 Changed 2 years ago by dimpase

  • Authors set to Dima Pasechnik
  • Branch set to u/dimpase/doc/docrefs
  • Commit set to f82a7c2d22d69c7cf575fa987afb0a1d059338d8
  • Description modified (diff)
  • Milestone changed from sage-8.8 to sage-8.7

comment:2 Changed 2 years ago by git

  • Commit changed from f82a7c2d22d69c7cf575fa987afb0a1d059338d8 to 13c2fd55fad6df1f31126d0fd0fa06d64ee84e04

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

13c2fd5trac 27495: links to external docs in reference index.rst

comment:3 Changed 2 years ago by dimpase

  • Cc fbissey embray added
  • Status changed from new to needs_review

I don't know why it has taken so long to get started with this, to link together all these loose spkg docs in SAGE_LOCAL - something one might not even notice otherwise.

At least now SageNB docs are nicely linked - and available as HTML, in particular this will be there on the website, I suppose.

Let's get it in quickly, and add more on a followup ticket.

comment:4 follow-up: Changed 2 years ago by fbissey

Nice intentions I guess. From the point of view of sage in isolation it looks ok.

I don't use configure in sage-on-gentoo, so I would rather want something that applies from setup.py. Link to other packages documentation, all distros will probably want to patch. Note how it is all over the place? There is no way we let things be that messy. Documentation lives under some form of /usrshare/doc/, /usr/share/$PKG/doc and variations are frowned upon. But that's not easily fixable without you guys deciding on a standard and applying it across the board. We can wait a few more years before getting there.

comment:5 Changed 2 years ago by git

  • Commit changed from 13c2fd55fad6df1f31126d0fd0fa06d64ee84e04 to 85299a30eaae4d781a9e0994507da634c31b02a5

Branch pushed to git repo; I updated commit sha1. New commits:

85299a3linking pplpy docs

comment:6 Changed 2 years ago by dimpase

It seems to be a tall order to harmonise naming schemes and locations for docs of all the Sage packages.

I'd rather take what each package provides and in spkg-configure.m4 record this location into a variable (perhaps if the package is not built, but a system version used, then no doc link needs to be generated at all).

Another important reason for having this is that authors of external packages sometimes are annoyed by Sage incorporating their work silently. This showing/linking of docs will make it less silent...

comment:7 follow-up: Changed 2 years ago by jhpalmieri

As fbissey said, what about doing this with Python instead of configure? Perhaps in src/sage_setup/docbuild/__init__.py we could modify index.rst, or take index.rst.in as input and write index.rst as output.

Or if it's just reference/index.rst that's relevant, it could be done in reference/conf.py.

Last edited 2 years ago by jhpalmieri (previous) (diff)

comment:8 in reply to: ↑ 7 ; follow-ups: Changed 2 years ago by dimpase

Replying to jhpalmieri:

As fbissey said, what about doing this with Python instead of configure?

Why? What's wrong with doing it at configuration time? Should we redo the whole Sage building process with Python only? :-)

Mind you, Python does not have tools for this, autoconf does, as you see.

Perhaps in src/sage_setup/docbuild/__init__.py we could modify index.rst, or take index.rst.in as input and write index.rst as output.

Or if it's just reference/index.rst that's relevant, it could be done in reference/conf.py.

comment:9 in reply to: ↑ 8 Changed 2 years ago by fbissey

Replying to dimpase:

Replying to jhpalmieri:

As fbissey said, what about doing this with Python instead of configure?

Why? What's wrong with doing it at configuration time? Should we redo the whole Sage building process with Python only? :-)

That, you see, is the result of conflating "sage the distribution" with "sage the software". A confusion made easier by the fact that sage the software is not distributed as a proper package like the rest of its dependencies. The top configure is really mostly for sage the distribution and mingling the top level configure to stuff in sage the software means that you are not wanting sage the software to be pip installable ever.

Mind you, Python does not have tools for this, autoconf does, as you see.

Perhaps in src/sage_setup/docbuild/__init__.py we could modify index.rst, or take index.rst.in as input and write index.rst as output.

Or if it's just reference/index.rst that's relevant, it could be done in reference/conf.py.

Yes I am aware that what you are thinking of is easy with autotools but not trivial with python. But sometimes doing the easy stuff is just painting yourself in a corner in the long run.

comment:10 Changed 2 years ago by dimpase

We don't have a special manual for Sage the software (sagelib), only for Sage the distro. This ticket makes no attempt to change this situation, it merely improves the existing reference manual, in particular corrects the fact that SageNB docs are no longer available on http://doc.sagemath.org.

One would surely welcome an effort to partition the docs between sagelib and Sage the distro, but arguably a large part of the top index.rst, including everything on this ticket, would not belong to sagelib docs.

comment:11 Changed 2 years ago by dimpase

  • Cc schilly added
  • Description modified (diff)

comment:12 in reply to: ↑ 4 ; follow-up: Changed 2 years ago by gh-timokau

Replying to fbissey:

Link to other packages documentation, all distros will probably want to patch. Note how it is all over the place? There is no way we let things be that messy. Documentation lives under some form of /usrshare/doc/, /usr/share/$PKG/doc and variations are frowned upon.

+1, could we make the individual doc locations configurable?

comment:13 in reply to: ↑ 12 ; follow-up: Changed 2 years ago by dimpase

Replying to gh-timokau:

Replying to fbissey:

Link to other packages documentation, all distros will probably want to patch. Note how it is all over the place? There is no way we let things be that messy. Documentation lives under some form of /usrshare/doc/, /usr/share/$PKG/doc and variations are frowned upon.

+1, could we make the individual doc locations configurable?

individual packages have their own ideas where the docs should go. Mostly it is either /usr/share/$PKG/doc or /usr/share/doc/$PKG, but not always. E.g. cmake puts docs in $prefix/doc. They also don't always have a topmost index.html, or quite often only provide pdf docs. Some probably have it configurable, some (most?) probably not.

Some packages have only little English documentation, e.g. giac.

This ticket is not meant to sort this huge mess out. It rather fixes most glaring omissions and provides a heads-up on docs that otherwise merely collect digital dust in $SAGE_LOCAL.

comment:14 in reply to: ↑ 13 ; follow-up: Changed 2 years ago by gh-timokau

Replying to dimpase:

individual packages have their own ideas where the docs should go

I'm not talking about configuring where they are installed (which is a sage-the-distribution concern), but where sage looks for them when they are linked. Distributions don't install the docs of dependencies through sage's build system.

comment:15 in reply to: ↑ 14 Changed 2 years ago by dimpase

Replying to gh-timokau:

Replying to dimpase:

individual packages have their own ideas where the docs should go

I'm not talking about configuring where they are installed (which is a sage-the-distribution concern), but where sage looks for them when they are linked. Distributions don't install the docs of dependencies through sage's build system.

I can give templates used in index.rst.in more generic names. Please suggest what would be suitable.

comment:16 follow-up: Changed 2 years ago by gh-timokau

Being able to override them with SAGENB_DOC, SINGULAR_DOC etc. would be helpful. I also currently don't use the ./configure script, but I don't see any reason I couldn't add that.

I agree that it is a step backwards from the "pip-installability" goal though.

comment:17 in reply to: ↑ 16 Changed 2 years ago by dimpase

Replying to gh-timokau:

Being able to override them with SAGENB_DOC, SINGULAR_DOC etc. would be helpful. I also currently don't use the ./configure script, but I don't see any reason I couldn't add that.

OK, will do.

I agree that it is a step backwards from the "pip-installability" goal though.

It is not really so, as the top level reference manual already has things that are not about sagelib, e.g. some of it are to be in the development manual.

I agree though that it's desirable to add one more level to Sage distro docs, something akin to http://doc.sagemath.org/

comment:18 in reply to: ↑ 8 Changed 2 years ago by jhpalmieri

Replying to dimpase:

Replying to jhpalmieri:

As fbissey said, what about doing this with Python instead of configure?

Why? What's wrong with doing it at configuration time? Should we redo the whole Sage building process with Python only? :-)

Mind you, Python does not have tools for this, autoconf does, as you see.

It's a search-and-replace; I think Python can handle it. Google led me to this, for example, for a generic search-and-replace:

import fileinput

for line in fileinput.FileInput(filename, inplace=True):
    print(line.replace(text_to_search, replacement_text))

comment:19 follow-up: Changed 2 years ago by dimpase

We are talking about package-wise configuration settings, and it's easy to move things from configure.ac into spkg-configure.m4 files in build/pkgs/ files once we are more settled on this.

Doing this in Python probably would mean instead doing these template changes in spkg-install scripts instead, and this feels to me like we are mixing up configuration and installation. This is the stuff we struggle on #27373 (which sort of convinced me that sage -i is evil, but that's anther story).

comment:20 in reply to: ↑ 19 ; follow-up: Changed 2 years ago by jhpalmieri

Replying to dimpase:

We are talking about package-wise configuration settings, and it's easy to move things from configure.ac into spkg-configure.m4 files in build/pkgs/ files once we are more settled on this.

Sorry, I don't understand what you mean.

Are you expecting to apply the substitutions in files other than reference/index.rst? As it stands, the Python file reference/conf.py already imports SAGE_DOC (for example) from sage.env, so it can surely do the same with SAGE_SHARE. What makes changing "@SAGE_SHARE@" to the value of the appropriate variable configuration rather than installation? It seems to me that the configuration part is putting "@SAGE_SHARE@" in the file to begin with. At that point, people can make sure that the appropriate variables are set how they want before they build and install the docs.

comment:21 in reply to: ↑ 20 Changed 2 years ago by dimpase

Replying to jhpalmieri:

Replying to dimpase:

We are talking about package-wise configuration settings, and it's easy to move things from configure.ac into spkg-configure.m4 files in build/pkgs/ files once we are more settled on this.

Sorry, I don't understand what you mean.

Are you expecting to apply the substitutions in files other than reference/index.rst?

Sorry, I should have added more context. I expect to generate contents of the substitution patterns per package, in spkg-configure.m4. It might be that the package isn't even installed, then the contents would contain a pointer to an external URL.

As it stands, the Python file reference/conf.py already imports SAGE_DOC (for example) from sage.env, so it can surely do the same with SAGE_SHARE. What makes changing "@SAGE_SHARE@" to the value of the appropriate variable configuration rather than installation? It seems to me that the configuration part is putting "@SAGE_SHARE@" in the file to begin with. At that point, people can make sure that the appropriate variables are set how they want before they build and install the docs.

As I say above, for some packages it might boil down to writing an external URL somewhere into index.rst, with the package itself not even installed. Surely, doing this substitution at installation rather than at configuration would make the whole thing even more complicated than it already is.

comment:22 Changed 2 years ago by jhpalmieri

Can you *also* do this in conf.py, as a backup plan (for the distro people who don't run ./configure)? Something like the following, which is untested:

  • src/doc/en/reference/conf.py

    diff --git a/src/doc/en/reference/conf.py b/src/doc/en/reference/conf.py
    index 9e55717545..98aaeecb13 100644
    a b  
    1111# All configuration values have a default; values that are commented out
    1212# serve to show the default.
    1313
    14 import sys, os
    15 from sage.env import SAGE_DOC_SRC, SAGE_DOC
     14import sys
     15import os
     16import fileinput
     17from sage.env import SAGE_DOC_SRC, SAGE_DOC, SAGE_SHARE
    1618sys.path.append(SAGE_DOC_SRC)
    1719from common.conf import *
    1820
    multidocs_subdoc_list = sorted([x for x in os.listdir(ref_src) 
    7072exclude_patterns += multidocs_subdoc_list + [
    7173    'sage', 'sagenb', 'options'
    7274    ]
     75
     76# Convert index.rst.in to index.rst, if this hasn't already been done
     77# by the top-level configure script.
     78if not os.path.exists('index.rst'):
     79    with open('index.rst', 'w') as output:
     80        for line in fileinput.FileInput('index.rst.in'):
     81            l = line.replace('@SAGE_DOC@', SAGE_DOC).replace('@SAGE_SHARE@', SAGE_SHARE)
     82            output.write(l)

comment:23 follow-up: Changed 2 years ago by jhpalmieri

I've tested this a little and it seems to work. I don't know if

	if not os.path.exists('index.rst'):

is the right thing, or if instead we should compare modification times for index.rst and index.rst.in.

comment:24 follow-up: Changed 2 years ago by jhpalmieri

In your line

* The legacy notebook interface `SageNB <@SAGE_DOC@/sagenb/index.html>`_ (no longer actively maintained; `source repository <https://github.com/sagemath/sagenb>`_).

the link to SageNB should use SAGE_SHARE, not SAGE_DOC.

comment:25 in reply to: ↑ 24 Changed 2 years ago by jhpalmieri

Replying to jhpalmieri:

In your line

* The legacy notebook interface `SageNB <@SAGE_DOC@/sagenb/index.html>`_ (no longer actively maintained; `source repository <https://github.com/sagemath/sagenb>`_).

the link to SageNB should use SAGE_SHARE, not SAGE_DOC.

Sorry, I'm wrong about this, but I was confused because Sage sets the environment variable SAGE_DOC to be SAGE_SHARE/doc/sage, and that's not how you define it: you don't include the sage at the end. So it would be best to not use SAGE_DOC as you are. Please either use @SAGE_SHARE@/doc (my preference) or invent a new variable (we have enough already, I think).

comment:26 in reply to: ↑ 23 Changed 2 years ago by dimpase

Replying to jhpalmieri:

I've tested this a little and it seems to work. I don't know if

	if not os.path.exists('index.rst'):

is the right thing, or if instead we should compare modification times for index.rst and index.rst.in.

No, this is not, one indeed should compare modification times (as an update will update index.rst.in, and might not remove stale index.rst).

comment:27 Changed 2 years ago by dimpase

So you can insert an extra condition

if os.path.getmtime('index.rst') <= os.path.getmtime('index.rst.in'):
   ... # update index.rst

and this even is supposed to work for py2/3 and for all OS's of interest.

comment:28 Changed 2 years ago by jhpalmieri

Okay, this works for me:

  • src/doc/en/reference/conf.py

    diff --git a/src/doc/en/reference/conf.py b/src/doc/en/reference/conf.py
    index 9e55717545..f8ffa03aa7 100644
    a b  
    1111# All configuration values have a default; values that are commented out
    1212# serve to show the default.
    1313
    14 import sys, os
    15 from sage.env import SAGE_DOC_SRC, SAGE_DOC
     14import sys
     15import os
     16import fileinput
     17from sage.env import SAGE_DOC_SRC, SAGE_DOC, SAGE_SHARE
    1618sys.path.append(SAGE_DOC_SRC)
    1719from common.conf import *
    1820
    multidocs_subdoc_list = sorted([x for x in os.listdir(ref_src) 
    7072exclude_patterns += multidocs_subdoc_list + [
    7173    'sage', 'sagenb', 'options'
    7274    ]
     75
     76# Convert index.rst.in to index.rst, in case this hasn't already been done
     77# by the top-level configure script.
     78if (not os.path.exists('index.rst') or
     79    (os.path.getmtime('index.rst') <= os.path.getmtime('index.rst.in'))):
     80    with open('index.rst', 'w') as output:
     81        for line in fileinput.FileInput('index.rst.in'):
     82            l = line.replace('@SAGE_DOC@', SAGE_DOC).replace('@SAGE_SHARE@', SAGE_SHARE)
     83            output.write(l)

comment:29 Changed 2 years ago by jhpalmieri

  • Branch changed from u/dimpase/doc/docrefs to u/jhpalmieri/doc/docrefs

comment:30 Changed 2 years ago by jhpalmieri

  • Commit changed from 85299a30eaae4d781a9e0994507da634c31b02a5 to 122a4c4427ec37aa41770d3583ef223348ce3ee9

Here's a branch making my suggested changes.


New commits:

122a4c4trac 27495: in reference/conf.py, convert index.rst.in to index.rst

comment:31 Changed 2 years ago by jhpalmieri

I would also suggest this change, although with my changes, @SAGE_DOC@ isn't present in index.rst.in anymore:

  • configure.ac

    diff --git a/configure.ac b/configure.ac
    index 0ac744e0d7..3c9417e06d 100644
    a b AC_CONFIG_COMMANDS(mkdirs, 
    515515    ])
    516516
    517517dnl used in index.rst of Reference Manual
    518 AC_SUBST([SAGE_DOC],[`pwd -P`/$SAGE_LOCAL/share/doc])
     518AC_SUBST([SAGE_DOC],[`pwd -P`/$SAGE_LOCAL/share/doc/sage])
    519519AC_SUBST([SAGE_SHARE],[`pwd -P`/$SAGE_LOCAL/share])
    520520
    521521AC_OUTPUT()

comment:32 Changed 2 years ago by git

  • Commit changed from 122a4c4427ec37aa41770d3583ef223348ce3ee9 to a23a345a932b7b741028a44107134ef49d88ea01

Branch pushed to git repo; I updated commit sha1. New commits:

a23a345trac 27495: @SAGE_DOC@ should point to the same place as SAGE_DOC

comment:33 Changed 2 years ago by embray

  • Milestone changed from sage-8.7 to sage-8.8

Ticket retargeted after milestone closed (if you don't believe this ticket is appropriate for the Sage 8.8 release please retarget manually)

comment:34 Changed 21 months ago by kcrisman

Apparently merge fails?

I think this is a fine solution for now, and I don't see why John and Dima can't review each others' contributions to this ticket.

comment:35 Changed 21 months ago by jhpalmieri

With 8.8.rc1, both git merge develop and git rebase develop work fine, so I'm not sure what trac is complaining about.

comment:36 Changed 21 months ago by dimpase

it might be that different git settings might play a role. Just push a rebased version.

comment:37 Changed 21 months ago by git

  • Commit changed from a23a345a932b7b741028a44107134ef49d88ea01 to 7c29a5a77460e313603c93f3c0c64aafcf40fffe

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

b2e60dctrac 27495: links to external docs in reference index.rst
f2f3594linking pplpy docs
72f1cc2trac 27495: in reference/conf.py, convert index.rst.in to index.rst
7c29a5atrac 27495: @SAGE_DOC@ should point to the same place as SAGE_DOC

comment:38 Changed 21 months ago by jhpalmieri

Done. I don't know if we should not add

AC_SUBST([SAGE_DOC],[`pwd -P`/$SAGE_LOCAL/share/doc/sage])

to configure.ac since we don't use it anymore. It's still there in case future changes use it.

comment:39 Changed 20 months ago by embray

  • Milestone changed from sage-8.8 to sage-8.9

Moving tickets from the Sage 8.8 milestone that have been actively worked on in the last six months to the next release milestone (optimistically).

comment:40 Changed 15 months ago by mkoeppe

  • Milestone changed from sage-8.9 to sage-9.0

Needs rebase.

comment:41 Changed 14 months ago by jhpalmieri

A question: the branch here currently adds a line

+* `Introduction to Maxima <@SAGE_SHARE@/maxima/5.41.0/doc/html/intromax.html>`_

to the main reference manual index. This is no longer valid because we are using version 5.42.2 of Maxima. I am inclined to omit this rather than try some clever way of determining the Maxima version in Sage. Opinions?

comment:42 Changed 14 months ago by git

  • Commit changed from 7c29a5a77460e313603c93f3c0c64aafcf40fffe to 31f6bf070afdd2638d12a09fc51b7d76bd48cb97

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

55285d3trac 27495: links to external docs in reference index.rst
c7ec752linking pplpy docs
32dbc74trac 27495: in reference/conf.py, convert index.rst.in to index.rst
d75a650trac 27495: @SAGE_DOC@ should point to the same place as SAGE_DOC
31f6bf0trac 27495: rebased for Sage 9.0

comment:43 Changed 14 months ago by embray

  • Milestone changed from sage-9.0 to sage-9.1

Ticket retargeted after milestone closed

comment:44 Changed 14 months ago by mkoeppe

  • Cc mkoeppe added

comment:45 Changed 14 months ago by dimpase

  • Authors changed from Dima Pasechnik to Dima Pasechnik, John Palmieri
  • Reviewers set to Dima Pasechnik
  • Status changed from needs_review to positive_review

comment:46 Changed 14 months ago by dimpase

  • Status changed from positive_review to needs_work

@MAXIMA_VERSION@ does not get replaced, probably cause .replace cannot do more than one pattern per line.

comment:47 Changed 14 months ago by dimpase

Even this does not work:

  • src/doc/en/reference/conf.py

    a b from sage.env import SAGE_SHARE 
    1717from sage.docs.conf import *
    1818from sage.interfaces.maxima import maxima
    1919
    20 MAXIMA_VERSION = maxima.version()
    21 
    2220ref_src = os.path.join(SAGE_DOC_SRC, 'en', 'reference')
    2321ref_out = os.path.join(SAGE_DOC, 'html', 'en', 'reference')
    2422
    if (not os.path.exists('index.rst') or 
    8078    (os.path.getmtime('index.rst') <= os.path.getmtime('index.rst.in'))):
    8179    with open('index.rst', 'w') as output:
    8280        for line in fileinput.FileInput('index.rst.in'):
    83             l = line.replace('@SAGE_DOC@', SAGE_DOC).replace('@SAGE_SHARE@', SAGE_SHARE).replace('@MAXIMA_VERSION@', MAXIMA_VERSION)
     81            l0 = line.replace('@SAGE_DOC@', SAGE_DOC).replace('@SAGE_SHARE@', SAGE_SHARE)
     82            l = l0.replace('@MAXIMA_VERSION@', maxima.version())
    8483            output.write(l)

John, can you look into it? It's your branch after all. Probably it matters that I'm trying it with Python 3 now.

comment:48 Changed 14 months ago by jhpalmieri

It works for me with OS X and Python 3. I'm not sure what you mean by ".replace cannot do more than one pattern per line":

$ sage
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 9.1.beta0, Release Date: 2020-01-10               │
│ Using Python 3.7.3. Type "help()" for help.                        │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable.     ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
sage: s = 'abcdef'
sage: s.replace('a', 'x').replace('b', 'y')
'xycdef'

Maybe the timestamps of index.rst or index.rst.in are off: if index.rst is newer, it won't get overwritten. Try touch index.rst.in: does that help?

comment:49 follow-up: Changed 14 months ago by jhpalmieri

Oh, or maybe index.rst was written by ./configure, in which case the timestamps would be wrong. So do we add a substitution for the Maxima version in configure.ac? How would that work?

comment:50 in reply to: ↑ 49 Changed 14 months ago by dimpase

Replying to jhpalmieri:

Oh, or maybe index.rst was written by ./configure, in which case the timestamps would be wrong. So do we add a substitution for the Maxima version in configure.ac? How would that work?

Indeed, index.rst is written by ./configure. But is there any reason for doing this?

Isn't your branch meant to replace this with Python code in src/doc/en/reference/conf.py

comment:51 Changed 14 months ago by dimpase

At configure time at the moment the only way to get Maxima version is to parse build/pkgs/maxima/package-version.txt (the 1st X.Y.Z are the version) in build/pkgs/maxima/spkg-configure.m4.

As we go on with adding capabilites to use system packages, we'd be able to call instead maxima --version to get it.

So this is doable, but is more work than after Sage is built.

comment:52 Changed 14 months ago by jhpalmieri

I'm happy to remove the changes to configure.ac and let Python do all of the work after Sage has been built. This will go against what you said in comment:21, though. If you give the go ahead, I'll switch to the Python-only approach.

comment:53 Changed 14 months ago by mkoeppe

I'm in favor of the Python-only approach too, because it would support the goal of having sagelib as an ordinary Python package that is able to build its own documentation out of the directory src/.

comment:54 Changed 14 months ago by dimpase

One way or another, mixing autoconf with Python the way it's done here is messy. If you like to make it Python-only, please go ahead.

comment:55 Changed 14 months ago by git

  • Commit changed from 31f6bf070afdd2638d12a09fc51b7d76bd48cb97 to dab737d291ae450e680df4f0edfbaa45d2414979

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

dab737dtrac 27495: add links to external docs in reference/index.rst:

comment:56 Changed 14 months ago by jhpalmieri

Done. I also squashed everything into a single commit.

comment:57 Changed 14 months ago by jhpalmieri

  • Status changed from needs_work to needs_review

comment:58 follow-up: Changed 14 months ago by fbissey

Very vanilla sage centric. For maxima I would favor parsing the output of maxima -d to find the relevant directories

fbissey@moonloop ~/Work/Overlays/testing/sci-libs/m4ri $ maxima -d
;;; Loading #P"/usr/lib64/ecl-16.1.2/sb-bsd-sockets.fas"
;;; Loading #P"/usr/lib64/ecl-16.1.2/sockets.fas"
;;; Loading #P"/usr/lib64/ecl-16.1.2/defsystem.fas"
;;; Loading #P"/usr/lib64/ecl-16.1.2/cmp.fas"
maxima-prefix=/usr
maxima-topdir=/usr/share/maxima/5.42.2
maxima-imagesdir=/usr/lib64/maxima/5.42.2/binary-ecl
maxima-sharedir=/usr/share/maxima/5.42.2/share
maxima-srcdir=/usr/share/maxima/5.42.2/src
maxima-demodir=/usr/share/maxima/5.42.2/demo
maxima-testsdir=/usr/share/maxima/5.42.2/tests
maxima-docdir=/usr/share/maxima/5.42.2/doc
maxima-infodir=/usr/share/info
maxima-htmldir=/usr/share/maxima/5.42.2/doc/html
maxima-plotdir=/usr/libexec/maxima/5.42.2
maxima-layout-autotools=T
maxima-userdir=/home/fbissey/.maxima
maxima-tempdir=/tmp
maxima-lang-subdir=NIL
maxima-objdir=/home/fbissey/.maxima/binary/5_42_2/ecl/16_1_2

maxima-docdir is what we are after.

comment:59 Changed 14 months ago by dimpase

  • Status changed from needs_review to positive_review

OK, this works.

comment:60 in reply to: ↑ 58 ; follow-up: Changed 14 months ago by jhpalmieri

Replying to fbissey:

Very vanilla sage centric. For maxima I would favor parsing the output of maxima -d to find the relevant directories

To be honest, I don't think it's worth it. I would rather omit a link to local Maxima documentation.

comment:61 in reply to: ↑ 60 ; follow-up: Changed 14 months ago by fbissey

Replying to jhpalmieri:

Replying to fbissey:

Very vanilla sage centric. For maxima I would favor parsing the output of maxima -d to find the relevant directories

To be honest, I don't think it's worth it. I would rather omit a link to local Maxima documentation.

Me and my distro buddies think otherwise. I think we would prefer nothing than the current state of this ticket.

I approve doing it in python rather than configure - my opinion on configure for sage is well known.

I am not sure about running things from doc/en/reference/conf.py. May be one day we'll have translation of the reference manual in other languages, we may think again about it then.

It may be easier on distro if you were assembling a single string to replace. Then we could replace with our own more easily.

Last edited 14 months ago by fbissey (previous) (diff)

comment:62 in reply to: ↑ 61 ; follow-up: Changed 14 months ago by jhpalmieri

  • Status changed from positive_review to needs_info

Your reply confuses me.

Replying to fbissey:

Replying to jhpalmieri:

Replying to fbissey:

Very vanilla sage centric. For maxima I would favor parsing the output of maxima -d to find the relevant directories

To be honest, I don't think it's worth it. I would rather omit a link to local Maxima documentation.

Me and my distro buddies think otherwise.

So you think we should include a link to local Maxima documentation?

I think we would prefer nothing than the current state of this ticket.

Or don't include it?

I approve doing it in python rather than configure - my opinion on configure for sage is well known.

I am not sure about running things from doc/en/reference/conf.py. May be one day we'll have translation of the reference manual in other languages, we may think again about it then.

Since it's modifying the file doc/en/reference/index.rst, it seems like an appropriate place. If we ever have translations of the reference manual, we will have to rethink doc/en/reference/conf.py in other ways, too. So I think it's okay to do the work in this file.

It may be easier on distro if you were assembling a single string to replace. Then we could replace with our own more easily.

If I omit local Maxima documentation, the only replacement would be of the string @SAGE_SHARE@.

So what exactly are you requesting?

Last edited 14 months ago by jhpalmieri (previous) (diff)

comment:63 in reply to: ↑ 62 Changed 14 months ago by fbissey

Replying to jhpalmieri:

Your reply confuses me.

Replying to fbissey:

Replying to jhpalmieri:

Replying to fbissey:

Very vanilla sage centric. For maxima I would favor parsing the output of maxima -d to find the relevant directories

To be honest, I don't think it's worth it. I would rather omit a link to local Maxima documentation.

Me and my distro buddies think otherwise.

So you think we should include a link to local Maxima documentation?

From the distro point of view, I don't care either way. From the sage point of view, it is a nice to have.

I think we would prefer nothing than the current state of this ticket.

Or don't include it?

Not in a way that relies on a regular vanilla sage layout. It has no future if sage ever becomes a pip installable package. It has to be smarter and to not rely on preconceptions. I admit that it is harder. Connecting the data parts of two separate packages whatever happens with the packaging is very hard and need forethought.

I approve doing it in python rather than configure - my opinion on configure for sage is well known.

I am not sure about running things from doc/en/reference/conf.py. May be one day we'll have translation of the reference manual in other languages, we may think again about it then.

Since it's modifying the file doc/en/reference/index.rst, it seems like an appropriate place. If we ever have translations of the reference manual, we will have to rethink doc/en/reference/conf.py in other ways, too. So I think it's okay to do the work in this file.

It may be easier on distro if you were assembling a single string to replace. Then we could replace with our own more easily.

If I omit local Maxima documentation, the only replacement would be of the string @SAGE_SHARE@.

So what exactly are you requesting?

Apart from giving in my tantrum to use the output of maxima -d (thanks maxima devs to give me tools to know about your install, that's good forethought). I am suggesting creating a string in conf.py and replacing a single value @MAXIMA_HTLM_DOC_DIR@ in index.rst. That way I can replace/patch with something appropriate in conf.py much more easily.

I'll look to see how hard it would be to use maxima -d.

Note that when I was talking about forethought here, the last problem is that once installed, sage's documentation is static. If you change something in the maxima install the link is likely to break. There is no real solution to that, apart linking to some place on the web and pray it stays there for the life of the deployments.

comment:64 Changed 14 months ago by fbissey

Of course I was so focused on maxima that I thought the other entries were comments :(

Too complicated to do something right for everyone in 5mn. Proceed if you want, we'll come and fix the stuff in distros later.

comment:65 Changed 14 months ago by fbissey

The only thing that I will add, is that we have PPLPY_DOCS in env.py to use with intersphinx (was not happy either with that) and you may as well use it already.

Last edited 14 months ago by fbissey (previous) (diff)

comment:66 follow-up: Changed 14 months ago by mkoeppe

fbissey, suppose src/setup.py provides a command build_sphinx that builds the sage documentation. How would it be invoked to provide the paths?

comment:67 in reply to: ↑ 66 ; follow-up: Changed 14 months ago by fbissey

Replying to mkoeppe:

fbissey, suppose src/setup.py provides a command build_sphinx that builds the sage documentation. How would it be invoked to provide the paths?

To be honest, I have no idea. There are so many potential paths that it would be too cumbersome to pass them on the command line (still doable, just no one really wants to do that). It would need to be automated one way or another. Especially if you want every single one to be provided.

comment:68 follow-up: Changed 14 months ago by gh-timokau

I agree with fbissey that the current state of this ticket is a step back from the point of packaging. There is no reason that either the maxima documentation or the singular documentation should live under SAGE_SHARE, or even share a common prefix.

Detecting the correct dir automagically with maxima -d would be perfect. Otherwise there should at the very least be some unique string I could replace with sed for each package. A bit better would be an variable in env.py. Best would still be an automatic solution.

comment:69 in reply to: ↑ 68 ; follow-up: Changed 14 months ago by dimpase

Replying to gh-timokau:

I agree with fbissey that the current state of this ticket is a step back from the point of packaging. There is no reason that either the maxima documentation or the singular documentation should live under SAGE_SHARE, or even share a common prefix.

this ticket has nothing to do with maxima or singular documentation being installed by Sage, or not. It merely creates links to these docs in the top level of Sage reference manual (hey, Sage should document its components!)

A solution I proposed at the start of this ticket was to create the correct links to docs at configure time. Then already installed Sage components could be analysed and correct links created in a, hopefully, platform/distro-independent way.

I was told that many distros are allergic to ./configure, and John cooked up something supposedly better for distros, in pure Python/Sage, the current branch.

One of the main purposes of this ticket was to improve doc.sagemath.org, to provide it with these links. I feel this has gone totally in a direction I did not intend.

If this ticket is useless for distros in its present form then I don't see a reason to do the task at hand with Python/Sage, it's a wrong tool IMHO. (It is configuring things, and we have ./configure for this task.)

Detecting the correct dir automagically with maxima -d would be perfect. Otherwise there should at the very least be some unique string I could replace with sed for each package. A bit better would be an variable in env.py. Best would still be an automatic solution.

comment:70 in reply to: ↑ 69 ; follow-up: Changed 14 months ago by gh-timokau

Replying to dimpase:

Replying to gh-timokau:

I agree with fbissey that the current state of this ticket is a step back from the point of packaging. There is no reason that either the maxima documentation or the singular documentation should live under SAGE_SHARE, or even share a common prefix.

this ticket has nothing to do with maxima or singular documentation being installed by Sage, or not. It merely creates links to these docs in the top level of Sage reference manual (hey, Sage should document its components!)

It should have nothing to do with that, but in its current state it assumes that they are sage-installed or at least at the location they would be if they were sage-installed.

A solution I proposed at the start of this ticket was to create the correct links to docs at configure time. Then already installed Sage components could be analysed and correct links created in a, hopefully, platform/distro-independent way.

I was told that many distros are allergic to ./configure, and John cooked up something supposedly better for distros, in pure Python/Sage, the current branch.

Its pure Python/Sage?, but not configurable. That is not better. The ideal solution would not need any configuration (determining the correct paths programmatically). The next best solution would allow configuration in some way (through environment variables, an easily-patchable file, etc.). The least good (for distros) solution doesn't allow any configuration.

One of the main purposes of this ticket was to improve doc.sagemath.org, to provide it with these links. I feel this has gone totally in a direction I did not intend.

If this ticket is useless for distros in its present form then I don't see a reason to do the task at hand with Python/Sage, it's a wrong tool IMHO. (It is configuring things, and we have ./configure for this task.)

Detecting the correct dir automagically with maxima -d would be perfect. Otherwise there should at the very least be some unique string I could replace with sed for each package. A bit better would be an variable in env.py. Best would still be an automatic solution.

comment:71 in reply to: ↑ 70 Changed 14 months ago by mkoeppe

Replying to gh-timokau:

The ideal solution would not need any configuration (determining the correct paths programmatically). The next best solution would allow configuration in some way (through environment variables, an easily-patchable file, etc.). The least good (for distros) solution doesn't allow any configuration.

I would like to suggest to use #29022 as the configuration mechanism for the "next best solution".

comment:72 Changed 14 months ago by jhpalmieri

The easiest solution is to just add links to the web page for the appropriate piece of software. For example the page https://www.singular.uni-kl.de/index.php/singular-manual.html might be pretty stable, whereas I have no idea how to find local Singular documentation programmatically.

(The disadvantage to this is that there will often be version mismatches between what Sage is using and the most up-to-date version.)

Last edited 14 months ago by jhpalmieri (previous) (diff)

comment:73 follow-up: Changed 14 months ago by jhpalmieri

I am tempted to create a new page, linked to from the main table of contents, that lists all of Sage's packages, with web pages for each. For Maxima, since we have a well-defined way to do it, we could list the local documentation, too. As we figure out what to do with other packages, we can add their local documentation to that list. So it would look something like:

* alabaster - link to alabaster web page
* appnope - link to web page
* arb - 
...
* maxima: local documentation: .... web page: ...
...

Opinions?

comment:74 in reply to: ↑ 67 Changed 14 months ago by mkoeppe

Replying to fbissey:

Replying to mkoeppe:

fbissey, suppose src/setup.py provides a command build_sphinx that builds the sage documentation. How would it be invoked to provide the paths?

To be honest, I have no idea. There are so many potential paths that it would be too cumbersome to pass them on the command line (still doable, just no one really wants to do that). It would need to be automated one way or another. Especially if you want every single one to be provided.

I have a new solution for this at #29038 - "Python package sage_conf: Provides optional configuration information for sagelib". Input from packagers would be appreciated.

comment:75 Changed 11 months ago by mkoeppe

  • Milestone changed from sage-9.1 to sage-9.2

comment:76 in reply to: ↑ 73 Changed 10 months ago by mkoeppe

Replying to jhpalmieri:

I am tempted to create a new page, linked to from the main table of contents, that lists all of Sage's packages, with web pages for each. For Maxima, since we have a well-defined way to do it, we could list the local documentation, too. As we figure out what to do with other packages, we can add their local documentation to that list. So it would look something like:

* alabaster - link to alabaster web page
* appnope - link to web page
* arb - 
...
* maxima: local documentation: .... web page: ...
...

Opinions?

In #29633, I suggest to generate this page from information in build/pkgs.

comment:77 Changed 10 months ago by mkoeppe

With #29633, links to external packages' documentation can now be simply added to SPKG.rst

comment:78 Changed 10 months ago by mkoeppe

  • Status changed from needs_info to needs_work

comment:79 Changed 6 months ago by mkoeppe

  • Milestone changed from sage-9.2 to sage-9.3

comment:80 Changed 3 months ago by mkoeppe

  • Keywords sd111 added

Hoping we can make progress on this ticket this week - https://wiki.sagemath.org/days111

comment:81 Changed 3 months ago by mkoeppe

  • Cc arojas added

comment:82 Changed 2 months ago by mkoeppe

  • Authors Dima Pasechnik, John Palmieri deleted
  • Branch u/jhpalmieri/doc/docrefs deleted
  • Cc gh-mwageringel added
  • Commit dab737d291ae450e680df4f0edfbaa45d2414979 deleted
  • Description modified (diff)
  • Reviewers Dima Pasechnik deleted
  • Summary changed from links to external docs in main reference index.rst to Use sphinx.ext.extlinks to add Sphinx roles for links to external package docs

comment:83 Changed 2 months ago by mkoeppe

  • Description modified (diff)

comment:84 Changed 2 months ago by mkoeppe

  • Description modified (diff)

comment:85 Changed 3 weeks ago by mkoeppe

  • Milestone changed from sage-9.3 to sage-9.4

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

comment:86 Changed 2 weeks ago by mkoeppe

  • Cc vdelecroix added

comment:87 Changed 2 weeks ago by mkoeppe

  • Description modified (diff)

comment:88 Changed 2 weeks ago by mkoeppe

  • Description modified (diff)

comment:89 follow-up: Changed 8 days ago by embray

Just bikeshedding, but I don't know why the verbose names like :spkg_giac_doc: and not just something like :giac:. I understand giac_doc is supposed to refer to an SPKG specifically for the GIAC docs, but in the context of the documentation that should be a given anyways.

Also how does this relate to other Python packages that already use Sphinx, and can use intersphinx instead?

comment:90 in reply to: ↑ 89 Changed 7 days ago by mkoeppe

Replying to embray:

Just bikeshedding, but I don't know why the verbose names like :spkg_giac_doc: and not just something like :giac:.

Fine with me to simplify these names.

Also how does this relate to other Python packages that already use Sphinx, and can use intersphinx instead?

The present ticket here is easier and can set the naming convention.

Packages where we can do intersphinx should use a compatible naming convention.

Note: See TracTickets for help on using tickets.