py3: do not include the notebook documentation in sage

[chapoton]

because sagenb is not yet available for python3

[chapoton]

I have no clear idea about what should be done here. Where will the sagenb doc live, if we get rid of it ?

---

New commits:

​501b4d2

trying to remove sagenb doc

[embray]

Are we never going to get sagenb working in Python 3? I've just been assuming that for now we don't have it, but that we would eventually port it at least for now. Or are we truly abandoning sagenb for Python 3?

[chapoton]

Well, if somebody wants to do the job of converting sagenb to full python3 compatibility.. I have no idea of the difficulty of the task.

[vbraun]

I'm fine with removing sagenb when we drop Python2 support unless somebody wants to port it... The SageNB docs should just say that (and suggest to use the jupyter notebook)

[kcrisman]

If the ETA is still some time from now for Py2 drop that seems plausible. But hopefully it won't be that bad; I feel like #22431 was closer than we thought.

[chapoton]

We should try in the ticket #22431 to see if we can re-activate sagenb in py3-sage. We decided some time ago to de-activate it as it was blocking the build of py3-sage. Try to-re-insert sagenb requires first #24269 to fix py3-sage. Then rewrite the spkg-install on top of #25394.

On the side-point to get rid of sagenb imports inside sage, tt would be helpful to positive-review #24994.

[chapoton]

let us close this one as invalid, as we are on the way towards building the doc on py3, including sagenb doc

[gh-timokau]

I think this is still a good idea (for python2 and python3). Given that sagenb has been deprecated for a while, I think it would be a good idea to either just remove it or make it a optional spkg. #24994 is a nice step in that direction and this would be another. Why should the documentation of the deprecated sagenb be included in sage when the documentation of other spkgs isn't?

For context: sagenb is a pain to package. First because of ​https://github.com/sagemath/sagenb/issues/440 and now while updating flask I encountered more problems.

[kcrisman]

Deprecated in what sense? Unofficially, sure - but probably this would need to be a much bigger announcement and very obvious that it is removed, rather than the (very sensible) procedure of upgrade option we currently offer.

[vbraun]

+1 to communicating the deprecation better, e.g. display a deprecation notice when you start sagenb. I don't really mind it being standard vs. optional, I just don't want somebody new to Sage get started with sagenb in 2018 just because it comes up by default.

[gh-timokau]

Deprecated because I was told so in ​https://github.com/sagemath/sagenb/issues/440 (I was surprised there so I agree that it should be communicated better). Also the docs refer to it as the "legacy SageNB".

[kcrisman]

I don't think that anyone would be having it come up by default, the default is Jeroen's converter and pretty soon most people's default ends up the Jupyter. Maybe an intermediate thing to being an optional package (which is very bad for the kind of end users who would need sagenb!) would be to formally change the default interface to being Jupyter, with VERY clear instructions for how to continue conversion after that point sprinkled everywhere.

[jdemeyer]

Replying to kcrisman:

Jeroen's converter

Just to give proper credit: Volker started that project and did most of the work. It's true that I also worked on it in order to make it the default interface, fixing various bugs and adding the "run SageNB" option.

[gh-timokau]

Replying to kcrisman:

I don't think that anyone would be having it come up by default, the default is Jeroen's converter and pretty soon most people's default ends up the Jupyter.

I think that is even more of a reason to work towards making sagenb optional.

Maybe an intermediate thing to being an optional package (which is very bad for the kind of end users who would need sagenb!)

Are we sure there even are users that keep up with the latest version of sage (e.g. aren't using 6.0 forever or something) *and* still use sagenb? And even making sagenb optional wouldn't immediately make that impossible, it would just make it a tiny bit more inconvenient. That might be a good thing because then they might either complain here (showing us that there are such users) or realize that it might be time to look into the jupyter notebook.

would be to formally change the default interface to being Jupyter, with VERY clear instructions for how to continue conversion after that point sprinkled everywhere.

I agree.

[fbissey]

<clear throat>Well in the math department at my university (university of Canterbury, New Zealand) they have reasonably recent installs of sage. But when I talk to them they also have hundreds of sagenb notebook because they have been using sage for quite some time. What they are after is literally a mass conversion tool. And even with one, the inertia of the old sagenb here sounds quite big.

[gh-timokau]

Ah, good to know. What about that converter kcrisman was talking about?

[fbissey]

Replying to gh-timokau:

Ah, good to know. What about that converter kcrisman was talking about?

There is probably a way to script it but it's just no one really want to take that job. I may have to do it for them at some point but that will be a tough sell.

[gh-timokau]

Replying to fbissey:

Replying to gh-timokau:

Ah, good to know. What about that converter kcrisman was talking about?

There is probably a way to script it but it's just no one really want to take that job. I may have to do it for them at some point but that will be a tough sell.

Yeah I get that. We could still make the deprecation more obvious and further disentangle sagenb and sage. Maybe a big deprecation warning in 8.3 and then making it optional in 8.4 or something.

[chapoton]

Removing sagenb doc would be one step towards building the doc of sage with python3. Otherwise, one needs to fix ​https://github.com/sagemath/sagenb/issues/454.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​affbb9e	Merge branch 'public/25382' in 8.4.b0
​820100e	proposal : refer to github sagenb sources

[chapoton]

any comment on the proposed removal ?

[jhpalmieri]

Is there a way to conditionally remove the docs, depending on whether Sage was built with Python 2 or 3?

[kcrisman]

Agreed.

[jhpalmieri]

By the way, with the branch here, make doc and make doc-pdf still try to build the notebook directory in src/doc/en/reference. So for this to work with Python 3, as long as SageNB is not Python 3 compatible, we have to rewrite /src/doc/en/reference/notebook/index.rst as well. It may not matter what goes into that file when using Python 3, since it won't be referenced from anywhere.

[jhpalmieri]

My idea was something like this:

src/doc/en/reference/conf.py

@@ -15 +15 @@
 from sage.env import SAGE_DOC_SRC, SAGE_DOC
 sys.path.append(SAGE_DOC_SRC)
 from common.conf import *
+from sage_setup.docbuild.build_options import OMIT
 
 ref_src = os.path.join(SAGE_DOC_SRC, 'en', 'reference')
 ref_out = os.path.join(SAGE_DOC, 'html', 'en', 'reference')
@@ -61 +62 @@
 # for 'static' and 'templates', and to deal with upgrades: 'sage',
 # 'sagenb', 'media', and 'other'.
 bad_directories = ['static', 'templates', 'sage', 'sagenb', 'media', 'other']
+for dir in OMIT:
+    if dir.startswith('reference' + os.sep):
+        bad_directories.append(dir.split(os.sep)[1])
 multidocs_subdoc_list = sorted([x for x in os.listdir(ref_src)
                                 if os.path.isdir(os.path.join(ref_src, x))
                                 and x not in bad_directories])

src/doc/en/reference/index.rst

@@ -12 +12 @@
 User Interface
 ==============
 
-* :doc:`Command Line Interface (REPL) <repl/index>`
+.. only:: Python2
 
-* For the jupyter notebook interface, see this `Documentation <https://jupyter-notebook.readthedocs.io/en/latest/notebook.html>`_.
+    * :doc:`Command Line Interface (REPL) <repl/index>`
+    * :doc:`Web Notebook <notebook/index>`
 
-* For the legacy notebook interface, which is no longer actively maintained, see the `source repository <https://github.com/sagemath/sagenb>`_.
+.. only:: Python3
+
+    * :doc:`Command Line Interface (REPL) <repl/index>`
+    * For the jupyter notebook interface, see this `Documentation <https://jupyter-notebook.readthedocs.io/en/latest/notebook.html>`_.
+    * For the legacy notebook interface, which is no longer actively maintained, see the `source repository <https://github.com/sagemath/sagenb>`_.
 
 Graphics
 ========

src/sage_setup/docbuild/__init__.py

@@ -669 +669 @@
 
         for doc in os.listdir(refdir):
             directory = os.path.join(refdir, doc)
-            if os.path.exists(os.path.join(directory, 'index.rst')):
+            if (os.path.exists(os.path.join(directory, 'index.rst'))
+                and os.path.join("reference", doc) not in OMIT):
                 n = len(os.listdir(directory))
                 documents.append((-n, os.path.join(self.name, doc)))
 

src/sage_setup/docbuild/build_options.py

@@ -16 +16 @@
 else:
     PAPEROPTS = ""
 
+# The legacy Sage notebook is not compatible with Python 3, and its
+# documentation will not build with Python 3, so omit it in that case.
+# The command line options '-t Python3' and '-t Python2' define tags
+# Python2 and Python3 which are used by Sphinx for conditional
+# inclusion of text: text in a block marked '.. only:: Python2' will
+# be included when Sphinx is invoked with '-t Python2'. See
+# http://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#including-content-based-on-tags
+# http://www.sphinx-doc.org/en/master/man/sphinx-build.html#cmdoption-sphinx-build-t
+SAGE_PYTHON3 = os.environ.get("SAGE_PYTHON3", "no") == "yes"
+if SAGE_PYTHON3:
+    TAGS = "-t Python3 "
+    OMIT.append(os.path.join("reference", "notebook"))
+else:
+    TAGS = "-t Python2 "
+
 #Note that this needs to have the doctrees dir
-ALLSPHINXOPTS = SPHINXOPTS + " " + PAPEROPTS + " "
+ALLSPHINXOPTS = SPHINXOPTS + " " + PAPEROPTS + " " + TAGS + " "
 WEBSITESPHINXOPTS = ""
 
 # Number of threads to use for parallel-building the documentation.

but it doesn't work with Python 3: even though we are supposed to be ignoring the directory src/doc/en/reference/notebook, Sphinx still tries to use it in compiling indices, or something like that. Maybe there are more settings for ignoring files/directories that I could add.

[jhpalmieri]

So, you can define tags in reST which Sphinx will then pay attention to, for example

.. only:: my-tag

    stuff

Then if you run Sphinx with the option -t my-tag, stuff is included in the document. My first approach was to have tags Python2 and Python3 and run Sphinx with the corresponding -t Python[2|3]. Unfortunately, the text in stuff seems to be read even if the tag is not active, so any references to, for example, reference/notebook eventually lead to problems with Python3.

The only way I can think to get around this is to have two different index.rst files, one for use with each version of Python, and then create a symlink to the appropriate one when building the docs. One more point: any file ending in .rst in a documentation directory seems to be read by Sphinx, even if it is not explicitly referenced anywhere. So the two different index.rst files can't end in .rst. So here is a branch which implements all of this. Works for me with Python 3 (if I get around the __div__ problem discussed at #25840).

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​47ca8d6

trac 25382: build the reference/notebook docs with Python 2 only: skip

[jhpalmieri]

By the way, even if this is the right approach, are symlinks the right thing to use? Are they available on Cygwin, for example? It's easy enough to copy the file instead.

[gh-timokau]

Can't we just not include the sagenb documentation in both cases? Besides historical baggage, I don't see why sagenb should be treated differently from other dependencies here. It would simplify this change and would also simplify future deprecation.

[jhpalmieri]

Replying to gh-timokau:

Can't we just not include the sagenb documentation in both cases?

Hasn't been what the bulk of the discussion on this ticket has been about? It seems that not everyone agrees with this plan, so I suggested an alternative.

[gh-timokau]

The way I read it most of this discussion is actually about weather or not we want to deprecate sagenb (#25837) and/or just disable it for python3. I don't see anybody arguing against the actual purpose of the ticket, just removing the documentation. Of course we would need to provide a user with a way to still access the documentation, just as we now do it with the jupyter notebook.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​72c0cb2

trac 25382: add versions of index.rst for Python2 and Python3.

[jhpalmieri]

I think that the decision of whether to remove the docs for SageNB in Python 2 belongs on sage-devel, not here. There is also the larger question of whether SageNB should be (or is?) officially deprecated. That also belongs on sage-devel. I don't think the few participants on this ticket should be making such policy decisions on our own.

[fbissey]

I haven't been very active in this ticket. Nevertheless I'd like to voice an opinion now. Can anyone point me to another case where package A contains the documentation for package B. It would be so much easier if sageNB was shipping its own documentation. We wouldn't even have this conversation.

[jhpalmieri]

Of course the old Sage notebook used to be part of Sage, and (as you pointed out) it is still used as an interface to Sage. Anyway, can someone build its documentation and put is somewhere obvious on the sagenb github site? Then our reference manual could include a link to it (or skip it with Python 3, if we want conditional documentation depending on the Python version).

[gh-timokau]

What @fbissey said was exactly the point I was also trying to make. sagenb is very much deprecated (as in no new features, minimal support, referred to as legacy in the docs) but still supported for now. #25837 discusses the future status of that.

But independently of that, I don't see why sagenb has special status. We don't include the docs of other dependencies either. We could just explain the user where he can find the docs of sagenb, which are shipped separately (as part of sagenb).

[jdemeyer]

We should only remove the docs if the notebook has effectively been deprecated.

[dimpase]

Given that I am basically a glorified postdoc with funding (without anything in it earmarked for sagenb) running out in a year, I definitely do not want to spend time on sagenb. I am all for deprecating and removing it as soon as possible.

[dimpase]

Moving the sagenb documentation to sagenb itself should be doable.

[chapoton]

I would prefer the simpler solution to just remove the sagenb doc inconditionally..

And if someone feels able to find another place for this doc somewhere (readthedocs ?), please do !

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​debe3ea

trac 25382: remove the sagenb docs from the reference manual.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​059de77

trac 25382: remove the sagenb docs from the reference manual.

[jhpalmieri]

Replying to chapoton:

I would prefer the simpler solution to just remove the sagenb doc inconditionally..

Like this?

[chapoton]

yes, just like that

[fbissey]

I won't lie. I like it. That's all that's ever needed on sage side. If someone wants to do something sagenb side, then that's another tracker.

[gh-timokau]

I agree.

[jhpalmieri]

I asked about the idea of removing the sagenb docs on sage-devel. Please add comments there.

[chapoton]

no need for the dependency to #25852, I think

[chapoton]

One week has passed on sage-devel, with only positive opinions. I think it's time to set this to positive. Agreed ?

[fbissey]

Yes. That means the paperwork needs to be done. Authors and reviewers fields need to be filled in. And the ticket set to review and so on and so forth.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​591e91a

trac 25382 remove remaining crosslinks

[chapoton]

and we also need to check that all doctests pass..

[chapoton]

This is ready for review now. Patchbot is morally green.

[dimpase]

OK, so we should keep a copy of the ready to read sagenb docs somewhere (linked at its github project). This is now ​https://github.com/sagemath/sagenb/issues/460

[dimpase]

Replying to chapoton:

This is ready for review now. Patchbot is morally green.

Are these Python3 warnings/errors from ​https://patchbot.sagemath.org/log/25382/Ubuntu/16.04/x86_64/4.4.0-134-generic/atlas/2018-10-17%2009:49:44?plugin=python3 harmless? Just wondering - ik habla helemaal geen Espanol...

[chapoton]

I think that the poor patchbot (not smart enough) is troubled by the removal of the files. Or maybe by the accented letters. But this should just be noise.

[dimpase]

I've made a new release of sagenb, see #26499.

Should we also do downgrading sagenb to optional there?

[chapoton]

Replying to dimpase:

I've made a new release of sagenb, see #26499.

Should we also do downgrading sagenb to optional there?

Note: the tasks of ugrading sagenb and making it optional are both fully independent of the present ticket.

[dimpase]

On #26499 I've implemented installing sagenb (html) docs to $SAGELOCAL/share/docs/sagenb --- that's where they belong to by right.

[dimpase]

I think that doc/en/reference/index.rst should have a link to docs of "external" packages in SAGELOCAL/share/doc (note that Sage's docs themselves are in SAGELOCAL/share/doc/sage). Then in view of #26499 one can put the link to docs in SAGELOCAL/share/doc/sagenb

[kcrisman]

Thanks very much everyone for handling all this properly so people can still find stuff.

[dimpase]

Looks good to me. I'll modify ​https://github.com/sagemath/sagenb/blob/master/README.rst to point to SAGELOCAL/share/doc/sagenb as the location for docs.

We should also find a way to put it up to doc.sagemath.org, but this is also not for this ticket.

[dcoudert]

FYI: graph_editor.py imports sagenb.notebook.interact, but has no pointer to the documentation of sagenb.

We will certainly have to remove this tool when sagenb will be removed.

[gh-timokau]

@dcoudert also see #25837.

[kcrisman]

Unfortunately now it seems that the interact documentation is not in Sage's usual online doc. That will break a number of links out there, plus interact is in many different interfaces to Sage, not just sagenb.

[fbissey]

Well just like sagenb documentation should not be shipped with sage, sage documentation should not be in sagenb. That means some documentation has be migrated/rewritten in sage proper.

[dimpase]

sagenb docs are installed in SAGE_LOCAL/share/doc/, among with many others.

This is however not at all reflected in the reference manual. I'm not sure how to achieve this with sphinx. By relative paths, like ../../ ?

[gh-timokau]

Not sure how its done right now, but intersphinx may be worth looking at: ​http://www.sphinx-doc.org/en/master/usage/extensions/intersphinx.html

[dimpase]

I agree that intersphinx looks like the right tool for the job.

[kcrisman]

In any case, are we agreed that the interact documentation should be visible now?