Toggle LSP for current buffer

Manually

To start an LSP client for the current buffer — or stop an attached one — you can use a function like the following:

local toggle_lsp_client = function()
  local buf = vim.api.nvim_get_current_buf()
  local clients = vim.lsp.get_clients({ bufnr = buf })
  if not vim.tbl_isempty(clients) then
    vim.cmd("LspStop")
  else
    vim.cmd("LspStart")
  end
end

This could be mapped to a keybinding for convenience.

Automatically

Alternatively, you can use lsp-timeout.nvim plugin, to automatically shutdown and startup servers upon focus. Make sure you read the troubleshooting subsection on plugins that you use that may misbehave (e.g. null-ls or efm-language-server).