Add a way to run a command before starting the kernel itself

[jankatins]

[From the gitter chat at Febr. 16 22:46]
Usecase: an environment needs to be activated to get the path set correctly. This can in some cases be tricky, as e.g. conda lets you add batch files which will be run on activate and can manipulate the path (that's at least my understanding and I would like to try that for a miktex package, so that miktex is in path when the env is activated).

So in this case, it would be nice if the actual kernel command which is run would be activate env & python ..... This can be achieved by two ways:

• add shell=True to the Popen call which starts the kernel (according to https://stackoverflow.com/questions/17742789/running-multiple-bash-commands-with-subprocess)
• first run the first command, export the env and then pass this env to the second command (https://stackoverflow.com/questions/1214496/how-to-get-environment-from-a-subprocess-in-python)

Both need changes in jupyther-client: either a way to get the shell=True parameter into the Popen call or the handling of env export (and a flag in the kernel spec to trigger it).

A third option is using batch files as a kernel startup command in the kernel spec. The batch would first activate the env and then call the python with the kernel startup line and the command line parameter as given by the batch call. Unfortunately, this leads to #104 and it could be interesting how to handle it because this needs a writable place where the batch file is created :-/

I would like to get advise what would be the best way to go forward here :-)

[minrk]

a2km provides the third option, which I think is the best, by creating a script that wraps env activation and launching the kernel. The most important aspect of that particular implementation is the use of exec to launch the kernel, so that the kernel is not a subprocess of what's launched, but the process becomes the kernel, so it cannot work on Windows.

[jankatins]

so it cannot work on Windows

:-/ Also: ruby :-/

So the third option is the way to go? That would mean that https://github.com/Cadair/jupyter_environment_kernels (a different solution to the "one kernel per env" problem, adds a kernel on the fly without writing a kernel spec) would need to create a batch/bash script for each kernel and safe that somewhere :-/ bash could exec, windows would create a cmd - python process chain, so would need a fix for #104 (which the R kernel anyway needs...).

The "add a 'before startup command' + run it and read the env + set the env for the kernel" option is not acceptable, isn't it?

[minrk]

@JanSchulz If that project used shell=True and exec, it should be equivalent to the script approach. I don't know of any even potentially working solutions on Windows.

[jankatins]

That project doesn't do anything to the launch sequence, only adds the environment entries when asked about the installed kernels (by extending the KernelSpecManager; you then have to set that as a config value to be used as a new kernel spec manager...).

What would be needed is a way to tell launch_kernel to use shell=True. Unfortunately, the **kw argument seems to be unused in that function (although further up the stack, it is implied that they are all passed to POpen) :-/

There also seems to be no way to add to the kw args from the kernel spec, as start_kernel uses the arguments from the start_kernel call to set these :-(

One way to solve this is simple scan the kernel start line in launch_kernel and if a & or ; is found, set shell=True. On *ix the kernel startup line could then be source activate <envname>; exec <old kernel start line> and on windows we omit the exec and simple keep cmd open.

[takluyver]

I don't think adding any way to use shell=True is a good idea. It has a great big code smell.

[jankatins]

I guess that the env thingy is also not acceptable... :-)

Ok, thanks for the advise here, will take it back to https://github.com/Cadair/jupyter_environment_kernels and there we will probably need to generate sh/cmd files :-/

[takluyver]

We allow declaratively setting extra environment variables, but not modifying the existing content.

Maybe we could special case adding directories to $PATH with a list in the kernelspec. But there are several other path-like things that you might want to modify similarly, and I don't see how we could cover them all.

[jankatins]

That would help (e.g. env could be set to PATH="c:\\conda\env\whatever\Scripts;{PATH}"), but it does not cover the whole range of things which activating an env can do (e.g. adding a path in the env, which is not Scripts -> getting miktex in a conda packages recognised by pandoc and in turn nbconvert). And I don't think reimplementing that logic in python makes any sense :-/

[takluyver]

If you're starting a Python kernel, you could of course just write a Python script which sets environment variables and then calls ipykernel.kernelapp.main()

[jankatins]

Unfortunately, this would then need to be installed in every environment, so that it is started with the right python :-(

[jankatins]

This is now solved in https://github.com/Cadair/jupyter_environment_kernels: it has a proxy dict implementation which is used as the env dict which loads the real env values of a activated env when first accessed.

[Carreau]

This is now solved in https://github.com/Cadair/jupyter_environment_kernels: it has a proxy dict implementation which is used as the env dict which loads the real env values of a activated env when first accessed.

That's cool !