local notebook execution connected to enterprise gateway in kubernetes through port-forward

[teaglebuilt]

Description

Prerequisites - The notebook and gateway are working as intended.

so just using this example from the documentation, to execute a notebook that is running locally but connected succesfully to the enterprise gateway which is running on kubernetes locally.

By simply port forwarding, my notebook has access to the gateway, yet I am not able to execute the notebook through the nbclient, and it really just does not find the kernel.

jupyter_client.kernelspec.NoSuchKernel: No such kernel named python_kubernetes

I did notice this conversation Kubernetes Ingress, but i have a successful connection to use enterprise gateway, is there some configuration that i am missing for the RemoteKernelManager to be able to do this?

Environment

virtualenv

JUPYTER_GATEWAY_URL=http://127.0.0.1:8888
VIRTUAL_ENV=/Users/dteagle1/workspace/fullflow/env
KG_URL=http://127.0.0.1:8888
_=/usr/bin/printenv

version

jupyter core     : 4.7.1
jupyter-notebook : 6.2.0
qtconsole        : not installed
ipython          : 7.20.0
ipykernel        : 5.4.3
jupyter client   : 6.1.11
jupyter lab      : 3.0.9
nbconvert        : 5.6.1
ipywidgets       : 7.6.3
nbformat         : 5.1.2
traitlets        : 5.0.5

pip

jupyter-client==6.1.11
jupyter-contrib-core==0.3.3
jupyter-contrib-nbextensions==0.5.1
jupyter-core==4.7.1
jupyter-enterprise-gateway==2.5.0
jupyter-highlight-selected-word==0.2.0
jupyter-latex-envs==1.4.6
jupyter-lsp==1.1.4
jupyter-nbextensions-configurator== 0.4.1
jupyter-packaging== 0.7.12
jupyter-resource-usage==0.5.1
jupyter-server== 1.4.1
jupyterlab==3.0.9
jupyterlab-lsp==3.4.1
jupyterlab-pygments==0.1.2
jupyterlab-server== 2.3.0
jupyterlab-widgets==1.0.0

Screenshots / Logs

*Snippet

import nbformat
from nbclient import NotebookClient
from enterprise_gateway.services.kernels.remotemanager import RemoteKernelManager


with open('notebooks/Untitled.ipynb') as f:
    test_notebook = nbformat.read(f, as_version=4)

client = NotebookClient(nb=test_notebook, kernel_manager_class=RemoteKernelManager)
client.execute()

Traceback

Traceback (most recent call last):
  File "test.py", line 10, in <module>
    client.execute()
  File "/Users/dteagle1/workspace/fullflow/env/lib/python3.8/site-packages/nbclient/util.py", line 74, in wrapped
    return just_run(coro(*args, **kwargs))
  File "/Users/dteagle1/workspace/fullflow/env/lib/python3.8/site-packages/nbclient/util.py", line 53, in just_run
    return loop.run_until_complete(coro)
  File "/usr/local/Cellar/[email protected]/3.8.7_2/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/Users/dteagle1/workspace/fullflow/env/lib/python3.8/site-packages/nbclient/client.py", line 524, in async_execute
    async with self.async_setup_kernel(**kwargs):
  File "/Users/dteagle1/workspace/fullflow/env/lib/python3.8/site-packages/async_generator/_util.py", line 34, in __aenter__
    return await self._agen.asend(None)
  File "/Users/dteagle1/workspace/fullflow/env/lib/python3.8/site-packages/nbclient/client.py", line 482, in async_setup_kernel
    await self.async_start_new_kernel(**kwargs)
  File "/Users/dteagle1/workspace/fullflow/env/lib/python3.8/site-packages/nbclient/client.py", line 394, in async_start_new_kernel
    await ensure_async(self.km.start_kernel(extra_arguments=self.extra_arguments, **kwargs))
  File "/Users/dteagle1/workspace/fullflow/env/lib/python3.8/site-packages/nbclient/util.py", line 85, in ensure_async
    result = await obj
  File "/Users/dteagle1/workspace/fullflow/env/lib/python3.8/site-packages/enterprise_gateway/services/kernels/remotemanager.py", line 362, in start_kernel
    self._get_process_proxy()
  File "/Users/dteagle1/workspace/fullflow/env/lib/python3.8/site-packages/enterprise_gateway/services/kernels/remotemanager.py", line 580, in _get_process_proxy
    process_proxy_cfg = get_process_proxy_config(self.kernel_spec)
  File "/Users/dteagle1/workspace/fullflow/env/lib/python3.8/site-packages/jupyter_client/manager.py", line 89, in kernel_spec
    self._kernel_spec = self.kernel_spec_manager.get_kernel_spec(self.kernel_name)
  File "/Users/dteagle1/workspace/fullflow/env/lib/python3.8/site-packages/jupyter_client/kernelspec.py", line 235, in get_kernel_spec
    raise NoSuchKernel(kernel_name)
jupyter_client.kernelspec.NoSuchKernel: No such kernel named python_kubernetes

[teaglebuilt]

Oh I forgot to mention that the kernel is running and is returned from /api/kernels with the name python_kubernetes

[kevin-bates]

Hi @teaglebuilt - thanks for opening the issue.

Oh I forgot to mention that the kernel is running and is returned from /api/kernels with the name python_kubernetes

This information is coming from an EG server in which the python_kubernetes kernel was started from a traditional front-end, not nbclient - correct? nbclient-run kernels will not show up via /api/kernels results - so I just want to understand this statement.

When you run nbclient and receive the missing kernelspec error, are you running nbclient from within Kubernetes as well? And, does that image have the EG kernelspec files installed? If nbclient is not running within Kubernetes, you will not be able to spawn pod-based kernels.

[teaglebuilt]

@kevin-bates , no i am not. I am port-forwarding using kubectl to have access to EG running in kubernetes namespace running on a local cluster. aka Docker Desktop. The python script with nbclient is not in kubernetes, but the lab is also not in kubernetes and is working with the EG fine through the port-forwarded connection with the command below

kubectl port-forward svc/enterprise-gateway port:port

So with jupyterlab running locally and executing cells in a notebook that is assigned to a remote kernel, i would now like to test executing that notebook or cell outside of the notebook. Thats where this error occurs. Could it be that RemoteKernelManager could not work this way with nbclient? Before going further down a rabbit hole, i thought i would ask first to make sure I am not missing any basics.

[teaglebuilt]

@kevin-bates also, this is not intended to always run outside of kubernetes. I am just learning and exploring

[teaglebuilt]

@kevin-bates

If nbclient is not running within Kubernetes, you will not be able to spawn pod-based kernels.

Why is that? If the connection is accesible through the notebook, why not the client? Also, enterprise gateway did not spin up a pod as a remote kernel, I plan to configure it that way, but I think it started the kernel with a local process. I did not see a pod spawned on activation of the kernel, so iis it the remotekernelmanager or the nb client?

Also, I really like this project. I will launch this in a pod to test nbclient from inside the cluster as well, but I also would like to target where the issue is on why nbclient cannot read through the proxied connection to the kernel but the notebook can.

[kevin-bates]

If nbclient is not running within Kubernetes, you will not be able to spawn pod-based kernels.

Why is that?

The design of EG is that EG essentially resides within the resource-managed cluster. For Kubernetes, that means it runs as a k8s service and spawns pods for each kernel.

If the connection is accesible through the notebook, why not the client?
Please see my response on the nbclient issue you openened. nbclient is not a client of notebook server.

Also, enterprise gateway did not spin up a pod as a remote kernel, I plan to configure it that way, but I think it started the kernel with a local process. I did not see a pod spawned on activation of the kernel, so iis it the remotekernelmanager or the nb client?

If EG is configured within a Kubernetes cluster it can spin up kernels within its pod, but that is not the desired intention. Where/how a kernel is launched is a function of the process_proxy stanza configured in the kernel's corresponding kernel.json file. If no process_proxy stanza exists, the LocalProcessProxy will be automatically configured, resulting in the default behavior of kernels running local to (and on) the EG server. This is purely for backward compatibility purposes with Jupyter Kernel Gateway. The real point of EG is to spawn kernels into the resource-managed cluster, letting the Resource Manager decide on which node the kernel will run. As a result, kernel.json files configured with the, say, a KubernetesProcessProxy will result in the kernel being launched within a pod onto a node determined by the Kubernetes Resource Manager.

Please feel free to ask questions, they are greatly appreciated, but it would be best to keep EG-relevant discussions here, in a single location until there's a need to span into other repositories due to the complexity of this topic and wide-range of jupyter applications - most all of which only know how to deal with local kernel management.

Thanks.

[kevin-bates]

I'm closing this issue, please re-open if you feel your question/use-case was not adequately addressed. Thanks.