Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Install go using aqua broke gopls LSP server #710

Closed
budimanjojo opened this issue May 9, 2022 · 50 comments
Closed

Install go using aqua broke gopls LSP server #710

budimanjojo opened this issue May 9, 2022 · 50 comments

Comments

@budimanjojo
Copy link

budimanjojo commented May 9, 2022

aqua version

Please use the latest version.

$ aqua -v
aqua version 1.5.1 (aaf1d8beb6965d785e0b04de4ab93021f8634640)

Environment

⚠️ aqua doesn't support Windows.

  • OS (Linux, macOS, etc): Linux
  • CPU Architecture (amd64, arm64, etc): amd64

Overview

How to reproduce

  • aqua.yaml
# aqua.yaml
registries:
  - type: standard
    ref: v2.13.0    # renovate: depName=aquaproj/aqua-registry

packages:
  - name: twpayne/[email protected]
  - name: junegunn/[email protected]
  - name: starship/[email protected]
  - name: ajeetdsouza/[email protected]
  - name: BurntSushi/[email protected]
  - name: sharkdp/[email protected]
  - name: sharkdp/[email protected]
  - name: ogham/[email protected]
  - name: blacknon/[email protected]
  - name: FiloSottile/[email protected]
  - name: golang/[email protected]
  - name: cli/[email protected]
  - name: go-task/[email protected]
  - name: direnv/[email protected]
  - name: kubernetes/[email protected]
  - name: fluxcd/[email protected]
  - name: kubernetes-sigs/kustomize@kustomize/v4.5.4
  - name: mozilla/[email protected]
  - name: helm/[email protected]
  - name: siderolabs/[email protected]

Actual Behaviour

When golang binary is in /usr/local/go/bin/go, My neovim which is running gopls LSP client is not crashing on package or import completion. But when golang binary is in ~/.local/share/aquaproj-aqua/bin/go, it will crash neovim without any log at all.

Important Factoids

References

@suzuki-shunsuke
Copy link
Member

suzuki-shunsuke commented May 9, 2022

It's hard to reproduce the problem because the information of How to reproduce isn't enough.
This depends on NeoVim and gopls LSP client.

@budimanjojo
Copy link
Author

Yes, but it's also happening with gopls running outside of neovim though.
I also just find out something here. If I set my $PATH to $HOME/.local/share/aquaproj-aqua/pkgs/http/golang.org/dl/go1.18.1.linux-amd64.tar.gz/go/bin then it works fine again.

So the problem is only happening when the exec is handled by aqua-proxy. Maybe there's something that aqua-proxy does that can cause this?

@suzuki-shunsuke
Copy link
Member

@budimanjojo
Copy link
Author

Very weird, I'm looking at aqua exec and it basically just run the program using os/exec, Can we have aqua to not use aqua-proxy for several command in the yaml file? maybe just symlink $ROOT_DIR/pkgs/.../binaryfile to $ROOT_DIR/bin/binaryfile

I just tried symlinking $HOME/.local/share/aquaproj-aqua/pkgs/http/golang.org/dl/go1.18.1.linux-amd64.tar.gz/go/bin/go to $HOME/.local/share/aquaproj-aqua/bin/go and it runs fine again.
Can you show me what I should do to debug this?

@suzuki-shunsuke
Copy link
Member

suzuki-shunsuke commented May 9, 2022

Can we have aqua to not use aqua-proxy for several command in the yaml file?

You can execute tools installed by aqua without aqua-proxy by aqua exec.

e.g.

$ aqua exec -- go version
go version go1.18.1 darwin/arm64

@suzuki-shunsuke
Copy link
Member

You can reproduce the problem by running gopls -rpc.trace -v check main.go, right?

@budimanjojo
Copy link
Author

You can reproduce the problem by running gopls -rpc.trace -v check main.go, right?

I use this golang/go#51643 (comment)

I run gopls daemon, then in my LSP client (which is neovim), I use gopls -remote=localhost:8091 to connect to the server. It crashes only when I'm completing package and import with the LSP client.
I just tried to compare the output of gopls -rpc.trace -v check main.go with and without aqua-proxy and they all produce the same output except GOROOT is different. Completing everything works fine except for package and import, which is weird.

@budimanjojo
Copy link
Author

@suzuki-shunsuke what code editor are u using? Maybe I can try to reproduce this in vscode too?

@suzuki-shunsuke
Copy link
Member

suzuki-shunsuke commented May 9, 2022

I'm using NeoVim.

macOS arm64

$ nvim --version
NVIM v0.7.0
Build type: Release
LuaJIT 2.1.0-beta3
Compiled by [email protected]
$ aqua which go
/Users/shunsukesuzuki/.local/share/aquaproj-aqua/pkgs/http/golang.org/dl/go1.18.1.darwin-arm64.tar.gz/go/bin/go

$ go version
go version go1.18.1 darwin/arm64

$ which go
/Users/shunsukesuzuki/.local/share/aquaproj-aqua/bin/go
$ gopls version
golang.org/x/tools/gopls v0.7.5
    golang.org/x/tools/[email protected] h1:8Az52YwcFXTWPvrRomns1C0N+zlgTyyPKWvRazO9GG8=

@budimanjojo
Copy link
Author

Try this:

  1. Backup your ~/.config/nvim/
  2. Put this into ~/.config/nvim/init.lua:
local on_windows = vim.loop.os_uname().version:match 'Windows'

local function join_paths(...)
  local path_sep = on_windows and '\\' or '/'
  local result = table.concat({ ... }, path_sep)
  return result
end

vim.cmd [[set runtimepath=$VIMRUNTIME]]

local temp_dir = vim.loop.os_getenv 'TEMP' or '/tmp'

vim.cmd('set packpath=' .. join_paths(temp_dir, 'nvim', 'site'))

local package_root = join_paths(temp_dir, 'nvim', 'site', 'pack')
local install_path = join_paths(package_root, 'packer', 'start', 'packer.nvim')
local compile_path = join_paths(install_path, 'plugin', 'packer_compiled.lua')

local function load_plugins()
  require('packer').startup {
    {
      'wbthomason/packer.nvim',
      'neovim/nvim-lspconfig',
      'hrsh7th/nvim-cmp',
      'hrsh7th/cmp-nvim-lsp',
      'saadparwaiz1/cmp_luasnip',
      'L3MON4D3/LuaSnip'
    },
    config = {
      package_root = package_root,
      compile_path = compile_path,
    },
  }
end

_G.load_config = function()
  vim.lsp.set_log_level 'trace'
  if vim.fn.has 'nvim-0.5.1' == 1 then
    require('vim.lsp.log').set_format_func(vim.inspect)
  end
  local nvim_lsp = require 'lspconfig'
  local on_attach = function(_, bufnr)
    local function buf_set_keymap(...)
      vim.api.nvim_buf_set_keymap(bufnr, ...)
    end
    local function buf_set_option(...)
      vim.api.nvim_buf_set_option(bufnr, ...)
    end

    buf_set_option('omnifunc', 'v:lua.vim.lsp.omnifunc')

    -- Mappings.
    local opts = { noremap = true, silent = true }
    buf_set_keymap('n', 'gD', '<Cmd>lua vim.lsp.buf.declaration()<CR>', opts)
    buf_set_keymap('n', 'gd', '<Cmd>lua vim.lsp.buf.definition()<CR>', opts)
    buf_set_keymap('n', 'K', '<Cmd>lua vim.lsp.buf.hover()<CR>', opts)
    buf_set_keymap('n', 'gi', '<cmd>lua vim.lsp.buf.implementation()<CR>', opts)
    buf_set_keymap('n', '<C-k>', '<cmd>lua vim.lsp.buf.signature_help()<CR>', opts)
    buf_set_keymap('n', '<space>wa', '<cmd>lua vim.lsp.buf.add_workspace_folder()<CR>', opts)
    buf_set_keymap('n', '<space>wr', '<cmd>lua vim.lsp.buf.remove_workspace_folder()<CR>', opts)
    buf_set_keymap('n', '<space>wl', '<cmd>lua print(vim.inspect(vim.lsp.buf.list_workspace_folders()))<CR>', opts)
    buf_set_keymap('n', '<space>D', '<cmd>lua vim.lsp.buf.type_definition()<CR>', opts)
    buf_set_keymap('n', '<space>rn', '<cmd>lua vim.lsp.buf.rename()<CR>', opts)
    buf_set_keymap('n', 'gr', '<cmd>lua vim.lsp.buf.references()<CR>', opts)
    buf_set_keymap('n', '<space>e', '<cmd>lua vim.diagnostic.open_float()<CR>', opts)
    buf_set_keymap('n', '[d', '<cmd>lua vim.diagnostic.goto_prev()<CR>', opts)
    buf_set_keymap('n', ']d', '<cmd>lua vim.diagnostic.goto_next()<CR>', opts)
    buf_set_keymap('n', '<space>q', '<cmd>lua vim.diagnostic.setloclist()<CR>', opts)
  end

  -- Add the server that troubles you here
  local name = 'gopls'
  local cmd = { 'gopls', 'serve' } -- needed for elixirls, omnisharp, sumneko_lua
  if not name then
    print 'You have not defined a server name, please edit minimal_init.lua'
  end
  if not nvim_lsp[name].document_config.default_config.cmd and not cmd then
    print [[You have not defined a server default cmd for a server
      that requires it please edit minimal_init.lua]]
  end

  util = require 'lspconfig/util'
  nvim_lsp[name].setup {
    cmd = cmd,
    on_attach = on_attach,
    filetypes = { 'go', 'gomod' },
    root_dir = util.root_pattern('go.work', 'go.mod', '.git'),
  }

  -- luasnip setup
  local luasnip = require 'luasnip'

  -- nvim-cmp setup
  local cmp = require 'cmp'
  cmp.setup {
    snippet = {
      expand = function(args)
        luasnip.lsp_expand(args.body)
      end,
    },
    mapping = cmp.mapping.preset.insert({
      ['<C-d>'] = cmp.mapping.scroll_docs(-4),
      ['<C-f>'] = cmp.mapping.scroll_docs(4),
      ['<C-Space>'] = cmp.mapping.complete(),
      ['<CR>'] = cmp.mapping.confirm {
        behavior = cmp.ConfirmBehavior.Replace,
        select = true,
      },
      ['<Tab>'] = cmp.mapping(function(fallback)
        if cmp.visible() then
          cmp.select_next_item()
        elseif luasnip.expand_or_jumpable() then
          luasnip.expand_or_jump()
        else
          fallback()
        end
      end, { 'i', 's' }),
      ['<S-Tab>'] = cmp.mapping(function(fallback)
        if cmp.visible() then
          cmp.select_prev_item()
        elseif luasnip.jumpable(-1) then
          luasnip.jump(-1)
        else
          fallback()
        end
      end, { 'i', 's' }),
    }),
    sources = {
      { name = 'nvim_lsp' },
      { name = 'luasnip' },
    },
  }

  print [[You can find your log at $HOME/.cache/nvim/lsp.log. Please paste in a github issue under a details tag as described in the issue template.]]
end


if vim.fn.isdirectory(install_path) == 0 then
  vim.fn.system { 'git', 'clone', 'https://github.com/wbthomason/packer.nvim', install_path }
  load_plugins()
  require('packer').sync()
  vim.cmd [[autocmd User PackerComplete ++once lua load_config()]]
else
  load_plugins()
  require('packer').sync()
  _G.load_config()
end
  1. Now, open neovim and it will install LSP and completion module then install gopls. Exit neovim.
  2. Create a new go-project, mkdir -p ~/go/src/goproject && cd ~/go/src/goproject
  3. go mod init github.com/hello/goproject && nvim main.go
  4. In neovim, press q to quit packer window and in insert mode type package and press tab, it should give you completion selection, select one and press enter. If it doesn't crash, press backspace and try again, it will crash after 2 or 3 tries. Or if it doesn't crash, continue to type import " and type some characters to trigger autocompletion, like encoding/, press tab to select from menu. Sometimes pressing tab continuously also crash neovim.

Here's my recording of the crash: https://asciinema.org/a/emF4nCQnWJpLgV2qiSkdntmPq

@budimanjojo
Copy link
Author

I tried VSCode and VSCode also crash immediately.

@frezbo
Copy link
Contributor

frezbo commented May 10, 2022

I can confirm vscode also crashes. Cross linking the issue from vscode-go extension: golang/vscode-go#2229

@frezbo
Copy link
Contributor

frezbo commented May 10, 2022

I wonder if using exec.Command from os/exec is causing issue, since it'd be a child process as opposed to syscall.Exec

@suzuki-shunsuke
Copy link
Member

Oh, I didn't know this package. This is interesting.
But this is low level API, so I don't know whether we should use this.
I'll take a look.

https://pkg.go.dev/golang.org/x/[email protected]/unix#pkg-overview

The primary use of this package is inside other packages that provide a more portable interface to the system, such as "os", "time" and "net". Use those packages rather than this one if you can.

https://pkg.go.dev/golang.org/x/[email protected]/unix#Exec

syscall package is locked down. https://pkg.go.dev/syscall#pkg-overview

Deprecated: this package is locked down. Callers should use the corresponding package in the golang.org/x/sys repository instead. That is also where updates required by new systems or versions should be applied. See https://golang.org/s/go1.4-syscall for more information.

@frezbo
Copy link
Contributor

frezbo commented May 10, 2022

so i think the problem is that when aqua executed a package it's a subprocess as seen from the process tree

tilix─┬─fish───viddy─┬─aqua─┬─viddy───13*[{viddy}]

as opposed to calling the binary directly from the path,

─tilix─┬─fish───viddy───9*[{viddy}]

so if using the exec sycall the calling binary is replaced by the actual program and this would also help that aqua-proxy doesn;t have to deal with environment variables, stdout, stderr etc, the aqua-proxy is replaced by the actual program. Also aqua proxy doesn't have to then deal with exit codes and process reaping

@frezbo
Copy link
Contributor

frezbo commented May 10, 2022

syscall package is locked down. https://pkg.go.dev/syscall#pkg-overview

thanks for finding that, I need to update some code (forgot it was moved to sys package)

@budimanjojo
Copy link
Author

Here is how my pstree looks like:

        │         ├─tmux: server─┬─fish─┬─pstree
        │         │              │      └─{fish}
        │         │              ├─fish
        │         │              └─fish───nvim─┬─gopls───8*[{gopls}]
        │         │                            └─{nvim}

gopls is the one calling go command in subprocess of aqua in the background (maybe). But it's not in the processtree.

@suzuki-shunsuke
Copy link
Member

WIP aquaproj/aqua-proxy#25

@suzuki-shunsuke
Copy link
Member

Maybe we can replace os.Exec in aqua exec command too.

func (ctrl *Controller) execCommand(ctx context.Context, exePath string, args []string, logE *logrus.Entry) error {
logE = logE.WithField("exe_path", exePath)
logE.Debug("execute the command")
for i := 0; i < 10; i++ {
logE.Debug("execute the command")
cmd := exec.Command(exePath, args...)
cmd.Stdin = ctrl.stdin
cmd.Stdout = ctrl.stdout
cmd.Stderr = ctrl.stderr
runner := timeout.NewRunner(0)
if err := runner.Run(ctx, cmd); err != nil {
exitCode := cmd.ProcessState.ExitCode()
// https://pkg.go.dev/os#ProcessState.ExitCode
// > ExitCode returns the exit code of the exited process,
// > or -1 if the process hasn't exited or was terminated by a signal.
if exitCode == -1 && ctx.Err() == nil {
logE.WithField("retry_count", i+1).Debug("the process isn't started. retry")
if err := wait(ctx, 10*time.Millisecond); err != nil { //nolint:gomnd
return err
}
continue
}
logerr.WithError(logE, err).WithField("exit_code", exitCode).Debug("command was executed but it failed")
return ecerror.Wrap(err, exitCode)
}
return nil
}
return nil
}

@frezbo
Copy link
Contributor

frezbo commented May 10, 2022

Maybe we can replace os.Exec in aqua exec command too.

That'd be cool, less code to maintain

@suzuki-shunsuke
Copy link
Member

WIP #715

@suzuki-shunsuke
Copy link
Member

It works well. #715 aquaproj/aqua-proxy#25

The process tree is good.

$ nvim
$ pstree -s nvim                                  
-+= 00001 root /sbin/launchd
 \-+= 00743 shunsukesuzuki /System/Applications/Utilities/Terminal.app/Contents/MacOS/Terminal
   \-+= 54801 root login -pf shunsukesuzuki
     \-+= 54802 shunsukesuzuki -zsh
       \--= 46635 shunsukesuzuki nvim

@frezbo
Copy link
Contributor

frezbo commented May 10, 2022

Awesome work, really appreciate fixing it this fast ❤️

@suzuki-shunsuke
Copy link
Member

I've published prerelease versions.

I'd like to verify them for a while.


Please update aqua to v1.6.0-0.

$ aqua -v
aqua version 1.6.0-0 (da98e9d04300ed3daad3710389430193bf4b2248)

aqua.yaml

registries:
- type: standard
  ref: v2.14.0 # renovate: depName=aquaproj/aqua-registry
packages:
- name: neovim/[email protected]
$ aqua i # aqua-proxy would be updated to v1.1.0-0
INFO[0000] download and unarchive the package            aqua_version=1.6.0-0 package_name=aqua-proxy package_version=v1.1.0-0 program=aqua registry=
$ nvim
$ pstree -s nvim
-+= 00001 root /sbin/launchd
 \-+= 00743 shunsukesuzuki /System/Applications/Utilities/Terminal.app/Contents/MacOS/Terminal
   \-+= 75216 root login -pf shunsukesuzuki
     \-+= 75217 shunsukesuzuki -zsh
       \--= 90024 shunsukesuzuki nvim aqua.yaml

@budimanjojo
Copy link
Author

Thanks! I just tested it and it works now.
Before:

             └─fish───hwatch─┬─aqua─┬─hwatch───2*[{hwatch}]
                             │      └─16*[{aqua}]
                             └─6*[{hwatch}]

After:

             ├─fish───hwatch───2*[{hwatch}]

You can close this issue or I can close it now.

@suzuki-shunsuke
Copy link
Member

suzuki-shunsuke commented May 11, 2022

You are using Linux but I'm using macOS amd64 and arm64.
The difference of OS may be related, because golang.org/x/sys/unix is low level API.

@budimanjojo
Copy link
Author

I see there's a commit by renovate that update the golang/x/sys digest. Can you try building from that version?

@budimanjojo
Copy link
Author

You were using version from 9 releases old.
https://pkg.go.dev/golang.org/x/sys?tab=versions
ce5d925

@suzuki-shunsuke
Copy link
Member

I build the latest main branch b6bd86d , but the problem isn't solved.

@suzuki-shunsuke
Copy link
Member

suzuki-shunsuke commented May 11, 2022

This is just an idea, but I'm considering to release this feature as an experimental feature.
This is disabled by default, but you can enable this with environment variable.

e.g.

$ export AQUA_EXPERIMENTAL_FEATURES=x_sys_exec # comma separated

@budimanjojo
Copy link
Author

This is just an idea, but I'm considering to release this feature as an experimental feature. This is disabled by default, but you can enable this with environment variable.

e.g.

$ export AQUA_EXPERIMENTAL_FEATURES=x_sys_exec # comma separated

I'm fine with this

@suzuki-shunsuke
Copy link
Member

I launched Linux amd64 on macOS amd64 with lima-vm and tested, then it works good.

$ limactl --version
limactl version 0.10.0
shunsuke-suzuki@lima-default:/Users/shunsuke-suzuki/repos/src/github.com/aquaproj/aqua-registry$ hyperfine 'tfcmt -v'
Benchmark 1: tfcmt -v
  Time (mean ± σ):      60.6 ms ±   4.5 ms    [User: 19.6 ms, System: 49.8 ms]
  Range (min … max):    52.5 ms …  70.6 ms    41 runs

shunsuke-suzuki@lima-default:/Users/shunsuke-suzuki/repos/src/github.com/aquaproj/aqua-registry$ hyperfine 'ci-info -v'
Benchmark 1: ci-info -v
  Time (mean ± σ):      62.1 ms ±   7.1 ms    [User: 21.2 ms, System: 52.8 ms]
  Range (min … max):    51.4 ms …  88.5 ms    41 runs
 
shunsuke-suzuki@lima-default:/Users/shunsuke-suzuki/repos/src/github.com/aquaproj/aqua-registry$ hyperfine 'tfmigrator -v'
Benchmark 1: tfmigrator -v
  Time (mean ± σ):      76.3 ms ±  36.1 ms    [User: 27.1 ms, System: 52.4 ms]
  Range (min … max):    58.9 ms … 200.9 ms    14 runs
 
  Warning: The first benchmarking run for this command was significantly slower than the rest (200.9 ms). This could be caused by (filesystem) caches that were not filled until after the first run. You should consider using the '--warmup' option to fill those caches before the actual benchmark. Alternatively, use the '--prepare' option to clear the caches before each timing run.

@budimanjojo
Copy link
Author

budimanjojo commented May 11, 2022

So the problem is only on darwin, both x86_64 and arm64? Maybe we should report to upstream? Or maybe there's a way to make os/exec work without spawning child process?

@suzuki-shunsuke
Copy link
Member

So the problem is only on darwin, both x86_64 and arm64?

The issue is reproduced on both darwin amd64 and darwin arm64.

@suzuki-shunsuke
Copy link
Member

Maybe we should report to upstream?

To report to upstream, we want to find the procedure to reproduce the issue with less dependencies.

@frezbo
Copy link
Contributor

frezbo commented May 11, 2022

sorry I couldn't be of much help with darwin

@suzuki-shunsuke
Copy link
Member

suzuki-shunsuke commented May 11, 2022

I've released the prerelease version aqua-proxy v1.1.0-1 and aqua v1.6.0-2.

https://github.com/aquaproj/aqua-proxy/releases/tag/v1.1.0-1
https://github.com/aquaproj/aqua/releases/tag/v1.6.0-2

Please set the environment variable AQUA_EXPERIMENTAL_X_SYS_EXEC to use the feature.

#725

$ export AQUA_EXPERIMENTAL_X_SYS_EXEC=true

The problem has been reproduced with GitHub Actions.
https://github.com/suzuki-shunsuke/aqua-benchmark/runs/6389050750?check_suite_focus=true

budimanjojo added a commit to budimanjojo/nix-config that referenced this issue May 11, 2022
frezbo added a commit to frezbo/dotfiles that referenced this issue May 12, 2022
Use aqua experimental `exec` mode

Ref: aquaproj/aqua#710 (comment)

Signed-off-by: Noel Georgi <[email protected]>
@frezbo
Copy link
Contributor

frezbo commented May 12, 2022

I've tested with aqua version v1.6.0-2 and setting the AQUA_EXPERIMENTAL_X_SYS_EXEC to true. Seems to work fine

@suzuki-shunsuke
Copy link
Member

Thank you for your confirmation.
I'll release aqua-proxy v1.1.0 and aqua v1.6.0.

@suzuki-shunsuke
Copy link
Member

Released.

https://github.com/aquaproj/aqua/releases/tag/v1.6.0
https://github.com/aquaproj/aqua-proxy/releases/tag/v1.1.0

And I've created an issue to track the above problem. #729

Let me close this issue now. Thank you for your contribution!

@suzuki-shunsuke
Copy link
Member

suzuki-shunsuke commented May 7, 2023

@budimanjojo @frezbo
I've released aqua v2.5.0.
https://github.com/aquaproj/aqua/releases/tag/v2.5.0
This release would solve this issue by default in other than Windows without any setting such as AQUA_EXPERIMENTAL_X_SYS_EXEC.
For details, please see the release.

@budimanjojo
Copy link
Author

@suzuki-shunsuke Tested and it works fine without the env variable on v2.5.0. Thanks!

@frezbo
Copy link
Contributor

frezbo commented May 7, 2023

Thanks for following this up after a long time ❤️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants