Python processes still running after closing VSCode

[FedericoAzzi]

Applies To

• Notebooks (.ipynb files)
• Interactive Window and/or Cell Scripts (.py files with #%% markers)

What happened?

I am running a Python script with GridSearchCV and n_jobs > 1. This spawns N threads. After closing VSCode, these threads remain in memory.
This only happens with the Jupyter extension for VSCode (Interactive Window / .ipynb files). If I run the script in Terminal ("Run Python File in Terminal") or directly in Jupyter Notebook (outside of VSCode), the threads are closed properly.

I have tried some suggestions on similar issues (#5485, #1626, microsoft/vscode-python #19347, conda/conda #11569, microsoft/vscode #72769, scikit-learn/scikit-learn #12817) but none have worked for me.

Before closing VSCode

After closing VSCode

VS Code Version

1.69.2

Jupyter Extension Version

v2022.6.1201981810

Jupyter logs

Visual Studio Code (1.69.2, undefined, desktop)
Jupyter Extension Version: 2022.6.1201981810.
Python Extension Version: 2022.10.1.
No workspace folder opened.
info 9:21:57.461: ZMQ install verified.
El usuario pertenece al grupo de experimentos 'jupyterTestcf'
El usuario pertenece al grupo de experimentos 'jupyterEnhancedDataViewer'
info 9:21:57.843: LSP Notebooks experiment is disabled -- not in treatment group
info 9:21:57.982: Experiment status for python is {"enabled":true,"optInto":[],"optOutFrom":[]}
info 9:22:2.416: Process Execution: > c:\ProgramData\Anaconda3\python.exe -m pip list
> c:\ProgramData\Anaconda3\python.exe -m pip list
info 9:22:20.901: Starting interactive window for resource 'c:\Users\federico.azzi\Desktop\tree\arbol_decision.py'
info 9:22:21.26: Preferred Remote kernel for Interactive-1.interactive is undefined
info 9:22:21.204: Starting Jupyter Session id = 'startUsingPythonInterpreter:.jvsc74a57bd0ad2bdc8ecc057115af97d19610ffacc2b4e99fae6737bb82f5d7fb13d2f2c186.c:\ProgramData\Anaconda3\python.exe.c:\ProgramData\Anaconda3\python.exe.-m#ipykernel_launcher' (Python Path: c:\ProgramData\Anaconda3, EnvType: Conda, EnvName: 'base', Version: 3.9.12) for 'Interactive-1.interactive' (disableUI=false)
info 9:22:21.206: Creating raw notebook for resource 'c:\Users\federico.azzi\Desktop\tree\arbol_decision.py'
info 9:22:21.206: Computing working directory for resource 'c:\Users\federico.azzi\Desktop\tree\arbol_decision.py'
info 9:22:21.220: Process Execution: > c:\ProgramData\Anaconda3\python.exe ~\.vscode\extensions\ms-toolsai.jupyter-2022.6.1201981810\pythonFiles\normalizeSelection.py
> c:\ProgramData\Anaconda3\python.exe ~\.vscode\extensions\ms-toolsai.jupyter-2022.6.1201981810\pythonFiles\normalizeSelection.py
info 9:22:21.226: Starting raw kernel 'base (Python 3.9.12)' for interpreter c:\ProgramData\Anaconda3\python.exe
info 9:22:21.250: Process Execution: > c:\ProgramData\Anaconda3\python.exe -c "import ipykernel; print(ipykernel.__version__); print("5dc3a68c-e34e-4080-9c3e-2a532b2ccb4d"); print(ipykernel.__file__)"
> c:\ProgramData\Anaconda3\python.exe -c "import ipykernel; print(ipykernel.__version__); print("5dc3a68c-e34e-4080-9c3e-2a532b2ccb4d"); print(ipykernel.__file__)"
info 9:22:21.280: Kernel launching with ports 9000,9001,9002,9003,9004. Start port is 9000
info 9:22:21.302: Creating daemon process for c:\ProgramData\Anaconda3\python.exe with env variables count 61
info 9:22:21.316: Process Execution: > c:\ProgramData\Anaconda3\python.exe -m vscode_datascience_helpers.daemon --daemon-module=vscode_datascience_helpers.kernel_interrupt_daemon -v --ppid 3976
> c:\ProgramData\Anaconda3\python.exe -m vscode_datascience_helpers.daemon --daemon-module=vscode_datascience_helpers.kernel_interrupt_daemon -v --ppid 3976
info 9:22:21.597: Registering dummy command feature
info 9:22:21.653: get interrupthandle daemon
info 9:22:21.665: Process Execution: > c:\ProgramData\Anaconda3\python.exe -m ipykernel_launcher --ip=127.0.0.1 --stdin=9003 --control=9001 --hb=9000 --Session.signature_scheme="hmac-sha256" --Session.key=b"da1cfb1e-1b2a-4ce0-87b1-8bca0202c5eb" --shell=9002 --transport="tcp" --iopub=9004 --f=c:\Users\federico.azzi\AppData\Roaming\jupyter\runtime\kernel-v2-39765MEwELewjoOT.json
> c:\ProgramData\Anaconda3\python.exe -m ipykernel_launcher --ip=127.0.0.1 --stdin=9003 --control=9001 --hb=9000 --Session.signature_scheme="hmac-sha256" --Session.key=b"da1cfb1e-1b2a-4ce0-87b1-8bca0202c5eb" --shell=9002 --transport="tcp" --iopub=9004 --f=c:\Users\federico.azzi\AppData\Roaming\jupyter\runtime\kernel-v2-39765MEwELewjoOT.json
info 9:22:21.665: Process Execution: cwd: ~\Desktop\tree
cwd: ~\Desktop\tree
info 9:22:21.915: ipykernel version 6.9.1 for c:\ProgramData\Anaconda3\python.exe
info 9:22:21.915: ipykernel location c:\ProgramData\Anaconda3\lib\site-packages\ipykernel\__init__.py for c:\ProgramData\Anaconda3\python.exe
warn 9:22:23.953: StdErr from Kernel Process c:\ProgramData\Anaconda3\lib\site-packages\traitlets\traitlets.py:2202: FutureWarning: Supporting extra quotes around strings is deprecated in traitlets 5.0. You can use 'hmac-sha256' instead of '"hmac-sha256"' if you require traitlets >=5.
  warn(

warn 9:22:23.953: StdErr from Kernel Process c:\ProgramData\Anaconda3\lib\site-packages\traitlets\traitlets.py:2157: FutureWarning: Supporting extra quotes around Bytes is deprecated in traitlets 5.0. Use 'da1cfb1e-1b2a-4ce0-87b1-8bca0202c5eb' instead of 'b"da1cfb1e-1b2a-4ce0-87b1-8bca0202c5eb"'.
  warn(

info 9:22:24.37: Kernel Output: NOTE: When using the `ipython kernel` entry point, Ctrl-C will not work.

To exit, you will have to explicitly quit this process, by either sending
"quit" from a client, or using Ctrl-\ in UNIX-like environments.

To read more about this, see https://github.com/ipython/ipython/issues/2049


To connect another client to this kernel, use:
    --existing c:\Users\federico.azzi\AppData\Roaming\jupyter\runtime\kernel-v2-39765MEwELewjoOT.json

info 9:22:24.254: Se inició el kernel base (Python 3.9.12)., (Raw session started and connected)
info 9:22:24.255: Finished connecting 31d765e2-0562-40a8-92b6-5871213e0687
info 9:22:24.258: UpdateWorkingDirectoryAndPath in Kernel
info 9:22:24.259: Executing silently Code (idle) = import types as _VSCODE_types\nimport os\nimport hashlib as _VSCODE_hashlib\nfrom IPython import get_ip
info 9:22:24.307: Executing silently Code (completed) = import types as _VSCODE_types\nimport os\nimport hashlib as _VSCODE_hashlib\nfrom IPython import get_ip with 1 output(s)
info 9:22:24.307: Waiting for idle on (kernel): f8315e67-d9d1-45e7-bd9e-8cf4ee0cb757 -> idle
info 9:22:24.307: Finished waiting for idle on (kernel): f8315e67-d9d1-45e7-bd9e-8cf4ee0cb757 -> idle
info 9:22:24.313: Executing silently Code (idle) = __file__ = 'c:\\Users\\federico.azzi\\Desktop\\tree\\arbol_decision.py'
info 9:22:24.319: Widget Message: Received IPyWidgets_kernelOptions in ScriptManager
info 9:22:24.319: Widget Message: Received IPyWidgetMessages.IPyWidgets_kernelOptions
info 9:22:24.323: Executing silently Code (completed) = __file__ = 'c:\\Users\\federico.azzi\\Desktop\\tree\\arbol_decision.py' with 0 output(s)
info 9:22:24.332: Generated code for 1 = <ipython-input-1-03c6b825e39d> with 206 lines
info 9:22:24.389: Widget Message: data-base-url set to https://file%2B.vscode-resource.vscode-cdn.net/c%3A/Users/federico.azzi/.vscode/extensions/ms-toolsai.jupyter-2022.6.1201981810/tmp/scripts/a8afd25f292aa84774f0bcc2eab57fcaf24523317a2271647f91f5b9f6c47a9c/jupyter
info 9:22:35.618: Cell 1 executed with state Success

Coding Language and Runtime Version

Python 3.9.12

Language Extension Version (if applicable)

v2022.10.1

Anaconda Version (if applicable)

conda 4.12.0

Running Jupyter locally or remotely?

Local

[rchiodo]

Thanks for the issue.

Do you have some example code that repros the problem?

[rchiodo]

My only guess as to how this might be a VS code issue is that we don't wait for the kernel to shutdown on exit. We just kill the kernel. Jupyter may be waiting for shutdown.

The way to verify that would be to kill the jupyter server (when using just jupyter) and see if the processes leak there too.

[FedericoAzzi]

Thanks for the issue.

Do you have some example code that repros the problem?

Thanks for the quick reply. This is a short example that can be used to reproduce the problem:

import numpy as np
import pandas as pd
from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split, GridSearchCV, RepeatedKFold
from sklearn.ensemble  import RandomForestRegressor
import multiprocessing
import warnings
warnings.filterwarnings('ignore')

boston = load_boston(return_X_y=False)
df = np.column_stack((boston.data, boston.target))
df = pd.DataFrame(df,columns = np.append(boston.feature_names, "MEDV"))
X_train, X_test, y_train, y_test = train_test_split(
                                        df.drop(columns = 'MEDV'),
                                        df['MEDV'],
                                        test_size = 0.33,
                                        random_state = 123
                                    )
param_grid = {
    'max_features': [5, 7, 9],
    'max_depth' : [1, 3, 5]
    }

grid = GridSearchCV(
        estimator  = RandomForestRegressor(random_state=123),
        param_grid = param_grid,
        scoring    = 'neg_root_mean_squared_error',
        n_jobs     = multiprocessing.cpu_count() - 1,
        cv         = RepeatedKFold(n_splits = 5, n_repeats = 2, random_state = 123),
        refit      = True,
        verbose    = 0,
        return_train_score = True
       )

grid.fit(X = X_train, y = y_train)

result = pd.DataFrame(grid.cv_results_)
result.filter(regex = '(param.*|mean_t|std_t)') \
    .drop(columns = 'params') \
    .sort_values('mean_test_score', ascending = False) \
    .head(4)

[rchiodo]

Thanks that helps. I'm reproing. The processes don't die during shutdown in VS code but do in Jupyter regardless of whether or not I kill them in the UI or just shutdown jupyter altogether.