Opened 6 years ago

Closed 6 years ago

#19371 closed enhancement (fixed)

Install Jupyter kernel spec and nbextensions in $SAGE_LOCAL

Reported by: jdemeyer Owned by:
Priority: major Milestone: sage-6.10
Component: interfaces Keywords:
Cc: Merged in:
Authors: Jeroen Demeyer Reviewers: Emmanuel Charpentier
Report Upstream: N/A Work issues:
Branch: 4f06d3d (Commits, GitHub, GitLab) Commit: 4f06d3df612175c6fceb7f564495aff7ce52d7bf
Dependencies: #19373 Stopgaps:

Status badges

Description (last modified by jdemeyer)

Instead of installing the Jupyter files in the user's home directory, we should install it in $SAGE_LOCAL (a "system" install). This has two main advantages:

  1. it avoids conflicts in case multiple installations of Sage exist.
  2. the Sage Jupyter notebook can be used more easily outside of Sage (e.g. using Jupyterhub)

We should not use versioned identifiers. For example, all these actually refer to the same Sage installation:

$ ls -l ~/.local/share/jupyter/kernels/
total 12
drwx------ 2 jdemeyer jdemeyer 4096 Sep 24 18:05 sage_6_9_beta7
drwx------ 2 jdemeyer jdemeyer 4096 Oct  7 22:41 sage_6_9_rc0
drwx------ 2 jdemeyer jdemeyer 4096 Oct  7 22:04 sage_6_9_rc3

I'm also replacing IPython -> Jupyter and Sage -> SageMath

Change History (35)

comment:1 Changed 6 years ago by jdemeyer

  • Description modified (diff)

comment:2 Changed 6 years ago by jdemeyer

  • Branch set to u/jdemeyer/install_jupyter_kernel_spec_in__sage_local

comment:3 Changed 6 years ago by jdemeyer

  • Commit set to 7c916ef8bab1dad676559fb52694f59d06a389ba
  • Status changed from new to needs_review

New commits:

7c916efInstall Jupyter kernel spec in $SAGE_LOCAL

comment:4 Changed 6 years ago by egourgoulhon

  • Reviewers set to Eric Gourgoulhon
  • Status changed from needs_review to positive_review

Looks good to me. But it would be nice if someone more expert on Jupyter gives a look too.

comment:5 Changed 6 years ago by jdemeyer

  • Priority changed from major to blocker

comment:6 follow-up: Changed 6 years ago by fbissey

You are concerned with multiple SageMath installs I am more concerned with global install for multiple users (SMC and sage-on-gentoo) and I am not sure about that change. It gives me problem on install at the moment because it violates sandbox rules of packaging. It's more the fault of jupyter_core for not giving me a way of over-ridding ENV_JUPYTER_PATH than yours.

But I am more worried about the fact that on a global install people won't be able to touch that directory. If you don't need to put anything else in it at "user runtime" great. If not we are in a big no situation as far as I am concerned (and William will almost certainly not be happy either).

As for running multiple SageMath install side by side without bad interactions from the content of ~/.sage you can always define DOTSAGE. A bit inconvenient (unless you version it inside of sage) but it works.

comment:7 in reply to: ↑ 6 ; follow-up: Changed 6 years ago by jdemeyer

Replying to fbissey:

It gives me problem on install at the moment because it violates sandbox rules of packaging.

I think you need to explain this more. The only possible issue I can see is that maybe the directory ENV_JUPYTER_PATH[0] is the "wrong" place in your Sage-on-Gentoo setup.

If ENV_JUPYTER_PATH[0] is some directory inside $SAGE_LOCAL (which it should be), I don't see why installing this one particular file in $SAGE_LOCAL is any different from installing the huge number of other files in $SAGE_LOCAL...

But I am more worried about the fact that on a global install people won't be able to touch that directory. If you don't need to put anything else in it at "user runtime" great.

We do not need to put anything else in that directory at runtime. Why do you think we do?

comment:8 in reply to: ↑ 7 ; follow-ups: Changed 6 years ago by fbissey

Replying to jdemeyer:

Replying to fbissey:

It gives me problem on install at the moment because it violates sandbox rules of packaging.

I think you need to explain this more. The only possible issue I can see is that maybe the directory ENV_JUPYTER_PATH[0] is the "wrong" place in your Sage-on-Gentoo setup.

ENV_JUPYTER_PATH is defined as os.path.join(sys.prefix, 'share', 'jupyter') which translates as /usr/share/jupyter (or $SAGE_LOCAL/share/jupyter) in concrete terms. The problem is the sage installation procedure is now trying to create and write files out of the sandbox.

Proper packaging in the sense of rpm, deb and most other doesn't allow you to write in the "live" system. Things are installed and written in a sandbox space and then copied (merged) to the system. Because of the way Andrew wrote env.py (and I was grateful to him for that) I can set most SAGE_* variables so sage will write things in my sandbox. If jupyter_core was allowing me to over-ride ENV_JUPYTER_PATH I could set in the appropriate place in the sandbox and it would be merged in the right place.

If ENV_JUPYTER_PATH[0] is some directory inside $SAGE_LOCAL (which it should be), I don't see why installing this one particular file in $SAGE_LOCAL is any different from installing the huge number of other files in $SAGE_LOCAL...

sys.prefix is SAGE_LOCAL, actually even in sage-on-gentoo, the difference is that you don't protect your live prefix during build.

But I am more worried about the fact that on a global install people won't be able to touch that directory. If you don't need to put anything else in it at "user runtime" great.

We do not need to put anything else in that directory at runtime. Why do you think we do?

Because I am uncertain of how it works. You are obviously creating something there as place holder but it was possibly silly of me.

comment:9 in reply to: ↑ 8 Changed 6 years ago by jdemeyer

Replying to fbissey:

sys.prefix is SAGE_LOCAL, actually even in sage-on-gentoo

So, if sys.prefix is $SAGE_LOCAL and you can set $SAGE_LOCAL to whatever you want, then I don't see the problem.

the difference is that you don't protect your live prefix during build.

I have absolutely no idea what this sentence means.

comment:10 in reply to: ↑ 8 Changed 6 years ago by jdemeyer

Replying to fbissey:

Because I am uncertain of how it works.

It's really just one configuration file. That's the whole thing I don't understand: what makes this one particular file so special? You apparently manage to install all of Sage, but not this one particular file?

comment:11 follow-ups: Changed 6 years ago by fbissey

Two things:

  • the fact that sys.prefix is SAGE_LOCAL is incidental, it doesn't have to be.
  • Once jupyter_core is in place it doesn't matter what I set SAGE_LOCAL to, it doesn't change the value of sys.prefix jupyter_core or rather python is reporting.

The install procedure for sage is now trying to create /usr/share/jupyter/kernels/sagemath and any missing directory. rpm, deb and ebuild do not allow you to do that directly. You install $SOMESANDBOX/usr/share/jupyter/kernels/sagemath and later you merge into the live system. That's what DESTDIR is for in autotools, you set DESTDIR=$SOMESANDBOX and things magically install in the sandbox.

Now my main concern was the potential for user to do something to {/usr,$SAGE_LOCAL}/share/jupyter/kernels/sagemath which would be problematic for multi-user install. It is unfounded. The rest is an upstream problem and we can stop debating packaging tech in this ticket.

comment:12 in reply to: ↑ 11 Changed 6 years ago by jdemeyer

Replying to fbissey:

You install $SOMESANDBOX/usr/share/jupyter/kernels/sagemath and later you merge into the live system. That's what DESTDIR is for in autotools, you set DESTDIR=$SOMESANDBOX and things magically install in the sandbox.

Right, and how does this work with distutils? I assume (but I might be wrong) that distutils also uses sys.prefix to determine where to install stuff.

comment:13 in reply to: ↑ 11 ; follow-up: Changed 6 years ago by jdemeyer

Replying to fbissey:

The rest is an upstream problem and we can stop debating packaging tech in this ticket.

It happens that I am debating packaging tech with upstream, so I'd like to understand your issue.

comment:14 in reply to: ↑ 13 Changed 6 years ago by fbissey

Replying to jdemeyer:

Replying to fbissey:

The rest is an upstream problem and we can stop debating packaging tech in this ticket.

It happens that I am debating packaging tech with upstream, so I'd like to understand your issue.

I have just read it ( before seeing your comment here). It is a bit messy but thinking more about it there must a bug of some kind. distutils as used from within portage should set a sandboxed prefix, therefore sys.prefix should point to a sandboxed location, so something is wrong there.

comment:15 Changed 6 years ago by fbissey

I know what the problem is. We have three phases in order: build, test (optional) and install. Only the install phase is run with a prefix, this is the only time when it should be necessary. But the installation of the jupyter kernel is attempted during the build phase. So from my point of view that bit has to be restricted to install time.

comment:16 Changed 6 years ago by jdemeyer

  • Status changed from positive_review to needs_work

comment:17 Changed 6 years ago by git

  • Commit changed from 7c916ef8bab1dad676559fb52694f59d06a389ba to 2cc3a749e812aacb2f699bf69e7b585e65526564

Branch pushed to git repo; I updated commit sha1. New commits:

2cc3a74Fix dependencies; install kernel spec with custom install class

comment:18 Changed 6 years ago by jdemeyer

  • Status changed from needs_work to needs_review

comment:19 Changed 6 years ago by jdemeyer

  • Status changed from needs_review to needs_work

I think we need to uncouple the IPython from the kernel spec installation.

comment:20 Changed 6 years ago by jdemeyer

  • Milestone changed from sage-6.9 to sage-6.10
  • Priority changed from blocker to major

Since this turns out to be a lot trickier than I initially thought, this should not be a 6.9 blocker. Which means that we need to unrebase #19374.

comment:21 Changed 6 years ago by fbissey

Thank you for taking my concern on board.

comment:22 Changed 6 years ago by jdemeyer

  • Dependencies set to #19371

comment:23 Changed 6 years ago by git

  • Commit changed from 2cc3a749e812aacb2f699bf69e7b585e65526564 to 6c7261c29f1e6c4f7698b8023fd6d3280597102f

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

52a717aUse relative links to local help
6c7261cInstall Jupyter kernel spec in $SAGE_LOCAL

comment:24 Changed 6 years ago by jdemeyer

  • Dependencies changed from #19371 to #19373

comment:25 Changed 6 years ago by jdemeyer

  • Description modified (diff)
  • Summary changed from Install Jupyter kernel spec in $SAGE_LOCAL to Install Jupyter kernel spec and nbextensions in $SAGE_LOCAL

comment:26 Changed 6 years ago by git

  • Commit changed from 6c7261c29f1e6c4f7698b8023fd6d3280597102f to 4f06d3df612175c6fceb7f564495aff7ce52d7bf

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

4f06d3dInstall Jupyter kernel and nbextensions in $SAGE_LOCAL

comment:27 Changed 6 years ago by jdemeyer

  • Description modified (diff)
  • Status changed from needs_work to needs_review

comment:28 Changed 6 years ago by egourgoulhon

  • Reviewers Eric Gourgoulhon deleted

Sorry I'm afraid I cannot review this one (don't feel competent about it).

comment:29 follow-up: Changed 6 years ago by charpent

I had a problem on an Ubuntu machine wher I could get MathJax? in sheets created in my home directory but nowhere else.

I checked the present patch and recompiled :

  1. It seems to solve my Ubuntu-specific problem
  2. It passes ptestlong.

Is that enough for a positive review ?

comment:30 in reply to: ↑ 29 Changed 6 years ago by jdemeyer

Replying to charpent:

Is that enough for a positive review ?

Who are you asking? Essentially, that's up to you to decide.

comment:31 follow-up: Changed 6 years ago by jdemeyer

Note that this depends on #19373. So, it would be nice if you could also review that.

comment:32 in reply to: ↑ 31 ; follow-up: Changed 6 years ago by charpent

Replying to jdemeyer:

Note that this depends on #19373. So, it would be nice if you could also review that.

Is this dependance somehow managed by git ?

In other terms, when I checked #19371 out, did that also applied #19373 ?

If so, that means that my tests show that (#19371+#19373) passes ptestlong. I cannot speak for a Jupyter hub, since I don't have such a beast in my zoo...

Otherwise, I'll have to check out #19373 and merge both branches, then recompile and ptestlong (about 4 hours on my Ubuntu netbook...).

Or would it be enough to positively review #19371, and leave #19373 to someone equipped to do so ?

comment:33 in reply to: ↑ 32 ; follow-up: Changed 6 years ago by jdemeyer

Replying to charpent:

In other terms, when I checked #19371 out, did that also applied #19373 ?

Yes, this branch here is based on #19373.

If so, that means that my tests show that (#19371+#19373) passes ptestlong.

Indeed.

comment:34 in reply to: ↑ 33 Changed 6 years ago by charpent

  • Reviewers set to Emmanuel Charpentier
  • Status changed from needs_review to positive_review

Replying to jdemeyer:

Replying to charpent:

In other terms, when I checked #19371 out, did that also applied #19373 ?

Yes, this branch here is based on #19373.

If so, that means that my tests show that (#19371+#19373) passes ptestlong.

Indeed.

OK. I give a positive review to #19371. I'll leave to someone having the possibility to really test #19373 the responsibility of validating it.

comment:35 Changed 6 years ago by vbraun

  • Branch changed from u/jdemeyer/install_jupyter_kernel_spec_in__sage_local to 4f06d3df612175c6fceb7f564495aff7ce52d7bf
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.