Unable to Autocomplete Citations with Nvim-Cmp + Cmp-Omni

[benbrastmckie]

Description

I've been having troubles trying to get VimTex + nvim-cmp + cmp-omni to work well in Lua, and have been reading everything I can find to try to sort out the problem. I've followed the VimTex help docs and believe that I have it all configured in the right way, and get most of the functionality that I'm looking for, just no access to my .bib file in the autocomplete menu, and so I can't autocomplete citations.

Here are some other relevant issues:

• Vimtex + nvim-cmp not working. #2453
• vimtex support hrsh7th/nvim-cmp#258
• Vimtex support isn't complete hrsh7th/cmp-omni#1

Steps to reproduce

Here is a MWE which I named minimal.lua

-- PLUGINS --

vim.cmd [[packadd packer.nvim]]

-- Packer 
require('packer').startup(function(use)
  -- Have packer manage itself
  use { "wbthomason/packer.nvim" }
  -- Useful lua functions used by lots of plugins
  use { "nvim-lua/plenary.nvim" }


-- Cmp 
  use { "hrsh7th/nvim-cmp" }
	use { "hrsh7th/cmp-nvim-lsp" }
  -- buffer completions
  use { "hrsh7th/cmp-buffer" }
  -- helps Vimtex completions
  use { "hrsh7th/cmp-omni" }


-- LSP
  -- enable LSP
	use { "neovim/nvim-lspconfig" }
  -- simple to use language server installer
  use { "williamboman/mason.nvim" }
  use { "williamboman/mason-lspconfig.nvim" }


-- LaTeX
  -- Vimscript
  use { "lervag/vimtex" } -- Vimscript


-- Treesitter
	use { "nvim-treesitter/nvim-treesitter", commit = "8e763332b7bf7b3a426fd8707b7f5aa85823a5ac" }

end)


-- CMP -----------------------------------------------------------

local cmp = require("cmp")

cmp.setup {
  sources = {
    { name = "omni" },
    { name = "nvim_lsp" },
    { name = "buffer", keyword_length = 3 },
  },
}

-- VimTeX -----------------------------------------------------------

vim.g.vimtex_view_method = "zathura"
vim.g.vimtex_syntax_enabled = 0
vim.g.vimtex_quickfix_enabled = 0

I was able to reproduce the observations above by running nvim -u minimal.lua minimal.tex. In case it is helpful, I also created a MW branch of my full config (I'm on the Lua branch until I can get full functionality out of VimTex). The same observations given above continue to apply. Any help or pointers about how to test the problem would be greatly appreciated.

NOTE: My config includes some files for tmux, alacrity, etc., so as to provide a complete package for those just starting off (i.e., academics with no experience working with the terminal etc., and so are stuck inside TexShop or Overleaf). I have not removed all of these files from the MWE since I can't imagine they are causing the problem.

Expected behavior

I expect to be able to autocomplete citations in the manner that I had working for my vimscript config.

Actual behavior

Here are some observations:

• I don't get VimTex autocompletion for Bibtex sources upon writing \citet{} as expected.
• I do get VimTex completions via Omni when writing \usepackage{}, so cmp-omni does seem to be working.
• The VimTex menu also works great, so I can open pdfs while hovering over citations etc.
• Upon triggering completion with when my cursor is here \citet{|}, I get the following error:

Error detected while processing function vimtex#complete#omnifunc[31]..167[6]..<SNR>107_filter_with_options[19]..<lambda>403:
line    1:
E716: Key not present in Dictionary: "mstr"

Do you use a latexmkrc file?

No

VimtexInfo

System info:
  OS: Garuda Linux
  Vim version: NVIM v0.8.0
  Has clientserver: true
  Servername: /run/user/1000/nvim.1726766.0

VimTeX project: LogicReligion
  base: LogicReligion.tex
  root: /home/benjamin/Documents/Philosophy/Papers/LogicReligion
  tex: /home/benjamin/Documents/Philosophy/Papers/LogicReligion/LogicReligion.tex
  main parser: current file verified
  document class: article
  packages: adjmulticol amsbsy amsfonts amsgen amsmath amsopn amssymb amstext atbegshi atbegshi-ltx atveryend atveryend-ltx auxhook bigintcalc bitset colonequals enumitem epstopdf-base etexcmds etoolbox fancyhdr float fontenc footmisc geometry gettitlestring graphics graphicx hycolor hyperref iftex ifvtex infwarerr intcalc keyval kvdefinekeys kvoptions kvsetkeys letltxmacro ltxcmds mathabx mathrsfs microtype multicol nameref natbib pdfescape pdftexcmds pgf pgfcomp-version-0-65 pgfcomp-version-1-18 pgfcore pgffor pgfkeys pgfmath pgfrcs pgfsubpic pgfsys pgftree refcount rerunfilecheck scrbase scrextend scrkbase scrlfile scrlfile-hook scrlogo setspace stmaryrd tikz tikz-qtree tipa titlesec tocloft trig uniquecounter url verbatim xcolor xstring
  compiler: latexmk
    engine: -pdf
    options:
      -verbose
      -file-line-error
      -synctex=1
      -interaction=nonstopmode
    callback: 1
    continuous: 1
    executable: latexmk
  viewer: Zathura
    xwin id: 0
  qf method: LaTeX logfile

[lervag]

First: I'm sure we will be able to figure this out. But I'm sorry that I don't have the time to help much right now. I'll follow up as soon as possible.

In the meantime, some minor comments:

• For testing, use nvim --clean -u minimal.lua minimal.tex - the --clean option is useful to ensure you are not loading unexpected user config.
• You should reconsider using VimTeXs syntax highlighting. As far as I understand, Tree-sitter is not better than VimTeX and will probably not be in quite some time.
• You should also reconsider the quickfix thing; it's a very good built-in tool that can be very helpful. If you find it noisy, then there are other options to help reduce the noise and only keep the important error messages and similar.
• Also, I see you add comment about VimTeX using vimscript. It's true, of course, but I don't see how it makes any major difference and how it warrants a comment in your config. ;)
• I notice I wrap the cmp sources like this: https://github.com/lervag/dotnvim/blob/95531578ec802422fd575e4e1f7b717f0b442d50/init/plugins/nvim-cmp.lua#L68
  You don't use the cmp.config.sources wrapper; perhaps it is no longer necessary? Or perhaps that's the issue?
• Finally: the error you get with the \citet completion - are you able to create a minimal LaTeX example with a bib file that you can share for which you get this error?

[benbrastmckie]

Thanks so much! I have changed my cmp.lua file to include cmp.config.sources matching the example you linked and removed the treesitter plugin, though it is required for lukas-reineke/indent-blankline.nvim and JoosepAlviste/nvim-ts-context-commentstring. Though the former adds a nice visual guide, I think I can do without the latter. However, I suspect that more and more plugins will draw on treesitter. Do you know if I can leave treesitter in place, and yet still use VimTex for syntax highlighting in LaTeX files?

I have also attempted the test you mentioned, including the --clean option and used a local .bib file, but could not avoid an error that I am struggling to interpret. Here is what I have:

minimal.lua

-- PLUGINS --

vim.cmd [[packadd packer.nvim]]

-- Packer 
require('packer').startup(function(use)
  -- Have packer manage itself
  use { "wbthomason/packer.nvim" }
  -- Useful lua functions used by lots of plugins
  use { "nvim-lua/plenary.nvim" }


-- Cmp 
  use { "hrsh7th/nvim-cmp" }
	use { "hrsh7th/cmp-nvim-lsp" }
  -- buffer completions
  use { "hrsh7th/cmp-buffer" }
  -- helps Vimtex completions
  use { "hrsh7th/cmp-omni" }


-- LSP
  -- enable LSP
	use { "neovim/nvim-lspconfig" }
  -- simple to use language server installer
  use { "williamboman/mason.nvim" }
  use { "williamboman/mason-lspconfig.nvim" }


-- LaTeX
  use { "lervag/vimtex" } 

end)


-- CMP -----------------------------------------------------------

local cmp = require("cmp")

cmp.setup({
  sources = cmp.config.sources({
    { name = "omni" },
    { name = "nvim_lsp" },
    { name = "buffer", keyword_length = 3 },
  }),
})

-- VimTeX -----------------------------------------------------------

vim.g.vimtex_view_method = "zathura"
vim.g.vimtex_syntax_enabled = 0
vim.g.vimtex_quickfix_enabled = 0

minimal.tex

%%%%%%%%%%%%%%%%%%%%%%%% PREAMBLE %%%%%%%%%%%%%%%%%%%%%%%%

%%% FORMATTING %%%
\documentclass[11pt]{article} %%Font size and document presets
\usepackage[a4paper, margin=4cm]{geometry} %Annina style

%%% CITATIONS %%%
\usepackage[round]{natbib} %%Or change 'round' to 'square' for square backers
\setcitestyle{aysep={}}

%%%%%%%%%%%%%%%%%%%%%%%%% TITLE %%%%%%%%%%%%%%%%%%%%%%%%%


\begin{document}
\title{\sc TITLE} %\thanks{}
\author{\it Benjamin Brast-McKie} 
\date{\today}
\maketitle
\thispagestyle{empty}
% \vspace{.1in}


\begin{abstract}
\noindent
ABSTRACT
\end{abstract}



%%%%%%%%%%%%%%%%%%%%%%%%% DOCUMENT %%%%%%%%%%%%%%%%%%%%%%%%%

\section{Introduction}\label{Intro}

\cite{Acharya2013}



%%%%%%%%%%%%%%%%%%%%%%% BIBLIOGRAPHY %%%%%%%%%%%%%%%%%%%%%%%


\newpage
\begin{footnotesize} %%Makes bib footnotesize text size
\bibliographystyle{Phil_Review} %%bib style found in bst folder, in bibtex folder, in texmf folder.
\setlength{\bibsep}{5pt} %%Changes spacing between bib entries
\bibliography{test.bib} %%bib database found in bib folder, in bibtex folder
\thispagestyle{empty} %%Removes page numbers
\end{footnotesize} %%End makes bib small text size

\end{document}

test.bib

@book{Olivelle1998,
  address =       {{New York}},
  author =        {Olivelle, Patrick},
  edition =       {Scholar's},
  month =         sep,
  publisher =     {{Oxford University Press}},
  title =         {The {{Early Upanishads}}: {{Annotated Text}} and
                   {{Translation}}},
  year =          {1998},
  abstract =      {This is the full edition of the early Upanisads, the
                   central scriptures of Hinduism. Featuring Patrick
                   Olivelle's acclaimed new English translation (Oxford,
                   1996), it also includes the complete Sanskrit text,
                   as well as variant readings, scholarly emendations,
                   and explanations of Olivelle's choices of particular
                   readings. The volume also contains a concordance of
                   the two recensions of the Brhadaranyaka Upanisad, and
                   an extensive bibliography.},
  isbn =          {978-0-19-512435-4},
}

@article{Acharya2013,
  address =       {{The Netherlands}},
  author =        {Acharya, Diwakar},
  journal =       {Indo-Iranian Journal},
  number =        {1},
  pages =         {3--39},
  publisher =     {{Brill}},
  title =         {N\'eti N\'eti {{Meaning}} and {{Function}} of an
                   {{Enigmatic Phrase}} in the
                   {{G\=argya-Aj\=ata\'satru}} Dialogue of {{B\d{r}had
                   \=Ara\d{n}yaka Upani\d{s}ad II}}.1 and {{II}}.3},
  volume =        {56},
  year =          {2013},
  abstract =      {This article argues that it is necessary to recognise
                   Sections II.1 and II.3 of the B\d{r}had
                   \=Ara\d{n}yaka Upani\d{s}ad as one discourse. Since
                   nobody among traditional or modern scholars has paid
                   attention to this structure, many problems have been
                   neither seen nor solved. This article does so, and
                   further, exposes textual problems embedded there and
                   analyses them. In this very discourse the expression
                   n\'eti n\'eti is introduced as an \=ade\'sa.
                   Therefore, this article investigates the meaning and
                   function of this expression there. This article also
                   muses on possible implications of the entire
                   discourse of G\=argya and Aj\=ata\'satru.},
  doi =           {10.1163/001972412-620402},
  issn =          {0019-7246},
}

Error

Although I could build the pdf just fine with my normal config, nvim --clean -u minimal.lua minimal.tex produced the following error:

Error detected while processing /home/benjamin/Documents/Philosophy/Papers/Logic
Religion/minimal.lua:
E5113: Error while calling lua chunk: ...in/Documents/Philosophy/Papers/LogicReligion/minimal.lua:3: Vim(packadd):E919: Directory not found in 'packpath': "pack/*/opt/packer.nvim"
stack traceback:
        [C]: in function 'cmd'
        ...in/Documents/Philosophy/Papers/LogicReligion/minimal.lua:3: in main c
hunk
Press ENTER or type command to continue

Hitting ENTER opened the file with syntax highlighting, but the autocomplete menus did not seem to work in any capacity including \usepackage{} which was autocompleting in my normal config.

Thanks again for the tips! I really appreciate the help :)

[lervag]

Still don't have quite enough time to go deep, sorry! But some quick replies:

Thanks so much! I have … removed the treesitter plugin,

Sorry, I did not mean you should remove the Tree-sitter plugin. Only to disable Tree-sitter highlights for VimTeX. Please read :help vimtex-faq-treesitter, where I hope these things are well explained.

Also, notice, for VimTeX syntax highlighting, you should remove vim.g.vimtex_syntax_enabled = 0.

Now, for your example minimal.tex and test.bib: They do not compile properly on my end because I don't have the "Phil_Review" bib style. Could you simplify further such that you use a style that is more common?

Although I could build the pdf just fine with my normal config, nvim --clean -u minimal.lua minimal.tex produced the following error:

Error detected while processing /home/benjamin/Documents/Philosophy/Papers/Logic
Religion/minimal.lua:
E5113: Error while calling lua chunk: ...in/Documents/Philosophy/Papers/LogicReligion/minimal.lua:3: Vim(packadd):E919: Directory not found in 'packpath': "pack/*/opt/packer.nvim"

This error is due to your use of packadd. I'm not completely sure how that works, but since we are looking at a minimal example, let's instead just create it in a slightly simpler manner without packer. Something like this:

-- vim.opt.runtimepath:prepend "PATH/nvim-lua/plenary.nvim"
-- vim.opt.runtimepath:prepend "PATH/hrsh7th/nvim-cmp"
-- vim.opt.runtimepath:prepend "PATH/hrsh7th/cmp-nvim-lsp"
-- vim.opt.runtimepath:prepend "PATH/hrsh7th/cmp-buffer"
-- vim.opt.runtimepath:prepend "PATH/hrsh7th/cmp-omni"
-- vim.opt.runtimepath:prepend "PATH/neovim/nvim-lspconfig"
-- vim.opt.runtimepath:prepend "PATH/williamboman/mason.nvim"
-- vim.opt.runtimepath:prepend "PATH/williamboman/mason-lspconfig.nvim"
vim.opt.runtimepath:prepend "PATH/vimtex"
vim.opt.runtimepath:append  "PATH/vimtex/after"

vim.cmd[[filetype plugin indent on]]

local cmp = require("cmp")

cmp.setup({
  sources = cmp.config.sources({
    { name = "omni" },
    { name = "nvim_lsp" },
    { name = "buffer", keyword_length = 3 },
  }),
})

vim.g.vimtex_view_method = "zathura"
-- vim.g.vimtex_syntax_enabled = 0
-- I would also recommend to keep the quickfix stuff enabled!
-- vim.g.vimtex_quickfix_enabled = 0

Here PATH must be the correct path to where these plugins are installed by packer. I don't know exactly where this is, but it should be somewhere under ~/.local/share/nvim/ or ~/.config/nvim, I believe. You should be able to figure it out by inspecting runtimepath with your normal configuration (e.g. echo &runtimepath).

[benbrastmckie]

Thanks so much for this! I've also been busy travelling but now returning to try to straighten this out before diving back into work.

I've fixed up my minimal.tex as follows:

\documentclass[11pt]{article}
\usepackage[a4paper, margin=4cm]{geometry}
\usepackage[english]{babel}

\begin{document}
\title{\sc TITLE}
\author{\it Benjamin Brast-McKie}
\date{\today}
\maketitle
\thispagestyle{empty}

\begin{abstract}
\noindent
ABSTRACT
\end{abstract}

\section{Introduction}\label{Intro}

\cite{Acharya2013}

\bibliographystyle{plain}
\bibliography{minimal}
\thispagestyle{empty}

\end{document}

I've also included a minimal.bib as follows:

@article{Acharya2013,
  address =       {{The Netherlands}},
  author =        {Acharya, Diwakar},
  journal =       {Indo-Iranian Journal},
  number =        {1},
  pages =         {3--39},
  publisher =     {{Brill}},
  title =         {N\'eti N\'eti {{Meaning}} and {{Function}} of an
                   {{Enigmatic Phrase}} in the
                   {{G\=argya-Aj\=ata\'satru}} Dialogue of {{B\d{r}had
                   \=Ara\d{n}yaka Upani\d{s}ad II}}.1 and {{II}}.3},
  volume =        {56},
  year =          {2013},
  abstract =      {This article argues that it is necessary to recognise
                   Sections II.1 and II.3 of the B\d{r}had
                   \=Ara\d{n}yaka Upani\d{s}ad as one discourse. Since
                   nobody among traditional or modern scholars has paid
                   attention to this structure, many problems have been
                   neither seen nor solved. This article does so, and
                   further, exposes textual problems embedded there and
                   analyses them. In this very discourse the expression
                   n\'eti n\'eti is introduced as an \=ade\'sa.
                   Therefore, this article investigates the meaning and
                   function of this expression there. This article also
                   muses on possible implications of the entire
                   discourse of G\=argya and Aj\=ata\'satru.},
  doi =           {10.1163/001972412-620402},
  issn =          {0019-7246},
}

Lastly, I changed the minimal.lua as you suggested:

vim.opt.runtimepath:prepend "/home/benjamin/.local/share/nvim/site/pack/packer/start/cmp-omni"
vim.opt.runtimepath:append  "/home/benjamin/.local/share/nvim/site/pack/packer/start/cmp-omni/after"
vim.opt.runtimepath:prepend "/home/benjamin/.local/share/nvim/site/pack/packer/start/nvim-cmp"
vim.opt.runtimepath:append  "/home/benjamin/.local/share/nvim/site/pack/packer/start/nvim-cmp/after"
vim.opt.runtimepath:prepend "/home/benjamin/.local/share/nvim/site/pack/packer/start/vimtex"
vim.opt.runtimepath:append  "/home/benjamin/.local/share/nvim/site/pack/packer/start/vimtex/after"

vim.cmd[[filetype plugin indent on]]

local cmp = require("cmp")

cmp.setup({
  sources = cmp.config.sources({
    { name = "omni" },
    { name = "nvim_lsp" },
    { name = "buffer", keyword_length = 3 },
  }),
})

vim.g.vimtex_view_method = "zathura"

And indeed, the VimTex functionality seems to be back up and running. What I'm wondering is how to proceed. How can I use this minimal example to detect what is going wrong? Am I to slowly restore my config piece by piece looking for where things break? This is new territory for me, so I'm not sure if there are standard procedures for such things.

As an after thought... I found the following upon checking health:

## VimTeX
  - OK: Vim version should have full support!
  - WARNING: biber is not executable!
    - ADVICE:
      - Biber is often required so this may give unexpected problems.
  - OK: Zathura should work properly!
  - OK: Compiler should work!

I have tried installing biber but Pacman is failing to retrieve files. Not sure if this is relevant but thought I'd include it in case it is.

Thanks again for all of your help. I'm definitely getting closer!

[lervag]

I've fixed up my minimal.tex as follows: …

I've also included a minimal.bib as follows: …

Lastly, I changed the minimal.lua as you suggested: …

And indeed, the VimTex functionality seems to be back up and running.

Cool. That's good, and implies that your problems are due to some issue with your configuration or similar.

What I'm wondering is how to proceed. How can I use this minimal example to detect what is going wrong? Am I to slowly restore my config piece by piece looking for where things break?

Well, that's the good question. First, you need a simple way to test if things are not working. E.g., start with your current full config and a minimal LaTeX project and verify that you can still reproduce your problem.

Now, either:

1. Start with the minimal example and add your config part by part while testing regularly. Initially, things should work, but it should break when the problematic part is added.
2. The opposite is also possible. I.e., start with your current full config, then strip it down part by part until you are left with the minimal configuration. At some point, things should start to work.

I typically go with 2 in such cases, and I find it usually doesn't take too long. It can be boring, of course. I tend to comment out/remove big chunks at a time, sort of like doing a bisection search.

As an after thought... I found the following upon checking health:

  - WARNING: biber is not executable!

If you don't know that you need biber, then it is probably not a big problem. biber is a modern alternative to bibtex, basically, and if you use biblatex you would typically also use biber. I can recommend it, but again, it is not necessarily important.

I have tried installing biber but Pacman is failing to retrieve files.

Did you remember to refresh your packages (e.g. pacman -Syy)?

[benbrastmckie]

I think I found the problem, and I suppose as is often the case, it is a lot more basic than I was worried it might be. The problem is fixed on my normal config so long as the .bib file is in the working folder. However, I prefer to draw on my full Zotero .bib file which lives in texmf/bibtex/bib and stays updated as I add papers to it etc. Is there a way to have my cake and eat it too? I'd love to have the VimTex autocompletion working for citations from my full bib file. Any thoughts would be greatly appreciated.

[lervag]

Ah, yes. First, I believe it should be possible to create a minimal example of the situation you are describing; if you do, then I can look into it more specifically and perhaps see if it is possible to fix things.

However, as a workaround, I believe you can e.g. use an absolute path to the bibliography. I think the problem is that VimTeX can't locate the bibliography file. That is, you have \bibliography{test.bib}, then test.bib would be searched for within the root of the current project first. If not found, VimTeX would also try to find it with kpsewhich - you can see how this command works with e.g. kpsewhich biblatex-examples.bib, wich would typically output something like /usr/share/texmf-dist/bibtex/bib/biblatex/biblatex/biblatex-examples.bib.

Notice, though, that this is quite a lot more complicated. The relevant code for finding the bib files is here:

vimtex/autoload/vimtex/bib.vim

Lines 7 to 41 in bd0173e

	function! vimtex#bib#files() abort " {{{1
	if has_key(b:vimtex.packages, 'biblatex')
	let l:file = b:vimtex.get_aux_file('bcf')
	if filereadable(l:file)
	let l:bibs = map(
	\ filter(readfile(l:file), "v:val =~# 'bcf:datasource'"),
	\ {_, x -> matchstr(x, '<[^>]*>\zs[^<]*')})
	for l:f in filter(copy(l:bibs), {_, x -> x =~# '[*?{[]' })
	let l:bibs += glob(l:f, 0, 1)
	endfor
	if !empty(l:bibs) | return s:validate(l:bibs) | endif
	endif
	endif
	let l:file = b:vimtex.get_aux_file('blg')
	if filereadable(l:file)
	let l:bibs = map(
	\ filter(readfile(l:file), 'v:val =~# ''^Database file #\d'''),
	\ {_, x -> matchstr(x, '#\d\+: \zs.*\ze\.bib$')})
	" Ignore '{name}-blx.bib' file (created by biblatex)
	if has_key(b:vimtex.packages, 'biblatex')
	call filter(l:bibs, 'v:val !~# ''-blx$''')
	endif
	" Ignore '{name}Notes.bib' file (created by revtex4)
	if b:vimtex.documentclass =~# '^revtex4'
	call filter(l:bibs, 'v:val !~# ''.Notes$''')
	endif
	if !empty(l:bibs) | return s:validate(l:bibs) | endif
	endif
	return s:validate(s:files_manual())
	endfunction

[benbrastmckie]

For good measure, I tried putting a copy of my full .bib file in the project folder for my minimal example, and no dice. The .bib file is huge (14976 lines), so started deleting chunks. I found that it works if I delete everything after line 13000, and stops working if I delete everything after line 13500 (I didn't persist to find the exact maximum number of lines, or entries as it may be, that it will accommodate). So the length looks to be the problem since I'm using my standard config for all of this, and got the same cut-off when using the minimal config.

Next I put a shortened version of my full .bib file in the /texmf/bibtex/bib directory, and indeed VimTex can draw from it no problem. So the length does seem to be the issue.

Do you know why VimTex struggles to handle such a big .bib file (assuming that it is VimTex's fault)? Is there anything I can do to fix that? I simply love the elegance of being able to add new citations with the Zotero browser plugin which updates my master .bib file automatically so that I can cite those papers immediately without ever having to edit a .bib file.

[lervag]

I found that it works if I delete everything after line 13000, and stops working if I delete everything after line 13500 (I didn't persist to find the exact maximum number of lines, or entries as it may be, that it will accommodate).

Perhaps there is something bad in your bib file that breaks the parsing? If you delete e.g. the first 2000 lines (make sure to not break a bib entry, of course) - does it work now?

Do you know why VimTex struggles to handle such a big .bib file (assuming that it is VimTex's fault)? Is there anything I can do to fix that?

No, and I don't think VimTeX actually do struggle here. There's a test file that's even larger than yours: huge.bib. It is used in some tests and it is handled properly.

I simply love the elegance of being able to add new citations with the Zotero browser plugin which updates my master .bib file automatically so that I can cite those papers immediately without ever having to edit a .bib file.

Yes, I also like the integration between Zotero and VimTeX. However, I stopped having a master bib file at some time. Instead, I export smaller collections relevant to the current paper/stuff I'm working on, and I export it to the particular project I'm working on. I found that works better for me, especially when I collaborate with other people.

[benbrastmckie]

Good thought! I've tried deleting the first 2k lines, and that also makes it work just fine. So the .bib file must be error free. I'm glad to know that VimTex works for large .bib files. Maybe it is nvim-cmp or cmp-omni? It's hard to imagine that they'd struggle for files with more than 13500 lines, but maybe that's the issue?

Out of curiosity, when you export a collection for some project but then need to add another paper to it, what does that entail? I suppose you'd have to add the new paper to Zotero via the browser plugin, then include it in the collection, then export that collection again? Or will the collection auto update? If the latter, then I guess the only extra step is creating and exporting a collection for each project and adding new papers to the relevant collection in addition to adding them to Zotero. I might give this a try until I can get my full .bib to work properly. I can definitely see how this would be better for collaboration.

[ThSGM]

I have been following this discussion with interest.

@benbrastmckie has asked about how to work between local and global bib files. I wanted to note that there is another nvim plugin that is starting to develop on the side of fuzzy-find citations. In addition, I have asked the author for functionality that splits local and global files.

For example:

• Feature request: calling with the local bib files vs. the global bib files
• [extension request] Yanking the entire bibtex entry

The second question also deals with the recent discussion about how we get references from a global bib file and move that reference to a local file. So my workflow was to basically use Telescope to fuzzy find a reference, then allow me to yank that reference. I guess you could develop scripts for this as well.

Just to be clear, I was turning to Telescope because I was somewhat frustrated by the inability to easily set up fuzzy find on citations (as you would get if you used Sublime Text or similar).

To me, what is the optimal workflow?

1. Use Zotero to update a master bib file
2. Within neovim, have a command that searches for a reference, either in the local bib file (at which point it adds the citation)
3. Or...find it in the global bib file, then pastes the bib entry into the local bib file.

[lervag]

Good thought! I've tried deleting the first 2k lines, and that also makes it work just fine. So the .bib file must be error free.

I'm not 100% convinced. If you don't mind, could you perhaps share a minimal LaTeX project and your large .bib file, as well as a short set of steps to reproduce your issue? If you do, then I can try and reproduce on my end and look into what might be the issue here.

Maybe it is nvim-cmp or cmp-omni? It's hard to imagine that they'd struggle for files with more than 13500 lines, but maybe that's the issue?

I also find it hard to believe the problem is with these.

---

Out of curiosity, when you export a collection for some project but then need to add another paper to it, what does that entail?

The same workflow you already outlined, except that you must add papers to the specified (sub)collections in Zotero. If you right-click on a subcollection, then choose "Export collection" and use "Better BibTeX", you will have the choice to export and continuously update with "Keep updated".

Many references are relevant for several papers/manuscripts; I just copy the items between collections as necessary. I don't mind overlap between the collections.

I guess the only extra step is creating and exporting a collection for each project and adding new papers to the relevant collection in addition to adding them to Zotero.

Yes, more or less. With the zotero extension for Firefox, I get a menu like this when I save items:

I just choose which collection to save to. It stays the same on subsequent saves, so I only need to change the collection every once in a while.

[lervag]

@benbrastmckie has asked about how to work between local and global bib files. I wanted to note that there is another nvim plugin that is starting to develop on the side of fuzzy-find citations. …

Thanks for sharing!

To me, what is the optimal workflow?

1. Use Zotero to update a master bib file

I already mentioned that I prefer to use Zotero to update local bib files. I don't see a reason to have both a full bibliographic collection in Zotero and a mirrored bib file version. But: We each have our own preferences, and that's fine!

2. Within neovim, have a command that searches for a reference, either in the local bib file (at which point it adds the citation)

I find writing \cite{XY with VimTeXs omnicompletion combined with an autocomplete function is usually very smooth. And this could be made more efficient by e.g. mapping ;c to \cite{ with a snippet or similar (I don't tend to optimize at these small scales myself, but I know some people do and enjoy it).

But, again, I don't mind that people want and choose other workflows! I'm glad there are more alternatives available that allows these differing workflows as well :)

[benbrastmckie]

@lervag Here is the directory where I have my minimal.lua, minimal.tex, and minimal.bib file which is identical to the Zotero.bib file that lives in texmf/bibtex/bib:

https://github.com/benbrastmckie/.config/tree/lua/nvim/minimal

Thanks for checking it out, and for the notes on your workflow. I think I'll institute that for the time being.

---

@ThSGM Thanks for the links and for promoting those feature requests. Perhaps you are already aware of bibexport but this is what I have been using to generate a local .bib file from my master .bib file. You can look up further details with bibexport --help, but here is the command that I have bound to a WhichKey binding to paste all citations present in my current paper into a new local .bib file:

bibexport -o %:p:r.bib %:p:r.aux

My previous workflow was to:

1. Cite freely from my master .bib while writing.
2. Run bibexport once I'm ready to submit my paper to a journal.

I suppose what you are proposing is that this process happen in the background as you cite new papers that have not already been cited in the current file. Plus fuzzy finding. Seems like a great idea :)

However, the workflow that @lervag describes also strikes me as a natural way to proceed. It does mean a little more care must go into maintaining a collection for each paper, something that I am not accustomed to, using collections instead to organise papers by subject-matter (not that these are incompatible).

[ThSGM]

@benbrastmckie

However, the workflow that @lervag describes also strikes me as a natural way to proceed. It does mean a little more care must go into maintaining a collection for each paper, something that I am not accustomed to, using collections instead to organise papers by subject-matter (not that these are incompatible).

I think @lervag noted the problem in this workflow, which is the same problem I encounter when collaborating with many people---which is that you need to create the local bib files as you are editing collaboratively. This often occurs over something like Overleaf or via version control.

@lervag

I already mentioned that I prefer to use Zotero to update local bib files. I don't see a reason to have both a full bibliographic collection in Zotero and a mirrored bib file version. But: We each have our own preferences, and that's fine!

From a VimTeX perspective, I guess the answer is that using a master bib file would then allow you to interface directly with the bib file. So here is the dream workflow:

1. First, fuzzy-finding on local bib files when doing \cite
2. Another hotkey to fuzzy-find/search through a global bib file for a reference; once selected, the script then copies that entry to the local bib files and completes the citation request

Also, it may interest you, if you weren't already aware of gscholar.

So you can essentially run a script to output the first paper on Google Scholar under a keyword. (I'm not sure how to do this in vim but I assume there is a way). I also tried https://github.com/mreppen/vim-scholar but had mixed results.

The dream would be a hotkey that then gives you a listing of say, 10 articles. Then you can select which one you want. Then it automatically inserts the intro into your local bib file.

[lervag]

@benbrastmckie

@lervag Here is the directory …

Thanks, I'll check it out and get back to you!

@ThSGM

I think @lervag noted the problem in this workflow, which is the same problem I encounter when collaborating with many people---which is that you need to create the local bib files as you are editing collaboratively. This often occurs over something like Overleaf or via version control.

Yes, in these cases I often tend to have one separate personal bib file for the project with which I work as previously indicated. I then synchronize the contents with the shared "collaborative" bib file after doing a writing session.

From a VimTeX perspective, I guess the answer is that using a master bib file would then allow you to interface directly with the bib file. So here is the dream workflow: …

Ok, I can see that this workflow may be useful. Still, I believe workflows like this may be equally useful as separate bib-specific plugins. Further, I'm quite at a lack of time, and so when prioritizing between issues I want to follow up, I'm sorry to say I will not prioritize this personally.

Also, it may interest you, if you weren't already aware of gscholar.

Thanks for sharing!

[lervag]

@benbrastmckie

@lervag Here is the directory …

Thanks, I'll check it out and get back to you!

It seems the issue was that you had an entry that was not correctly parsed:

@article{Thieme1968,
  title = {\=Ade\'sa},
  author = {Thieme, Paul},
  year = {1968},
  journal = {M\'elanges d' indianisme: \`a la m\'emoire de Louis Renou},
  pages = {pp.\textasciitilde 715-723}
}

Specifically, the tex codes \=A and \'s were not handled, which means the titlestring \=Ade\'sa was used verbatim as the substitute string in a substitute() call, where \= has a special meaning. This resulted in an error.

I believe this is fixed now, could you test?

[benbrastmckie]

That fixed it! Thanks so much for the help. Out of curiosity, how were you able to locate that error? I'm interested to improve my own troubleshooting skills.

[lervag]

That fixed it! Thanks so much for the help.

Glad to hear it!

Out of curiosity, how were you able to locate that error?

I downloaded your files, then I created a minimal test.vim file like this:

set nocompatible
set runtimepath^=~/.local/plugged/vimtex
set runtimepath+=~/.local/plugged/vimtex/after
filetype plugin indent on
syntax enable

silent edit minimal.tex

I started neovim with nvim --clean -u test.vim and added \cite{ inside the document, then typed <c-x><c-o> in insert mode to start omni completion. It immediately displayed an error message that told me there was an error in the completion file.

Now I added a few unsilent echo ... lines to investigate the state that errored. The test loop is 1) changing the source files with these echo statement, then 2) in another terminal running the nvim --clean -u test.vim to test. With that, I quickly realize it was the Thieme entry in the bib file, which made it easy to simplify the test files somewhat. This allowed me to fix the problem in the end by updating the vimtex#util#tex2unicode function that is used for parsing the bib entries.

To speed up the test loop, I automate more of the testing with the following test.vim file - now started with nvim --clean --headless -u test.vim.

set nocompatible
set runtimepath^=~/.local/plugged/vimtex
set runtimepath+=~/.local/plugged/vimtex/after
filetype plugin indent on
syntax enable

silent edit minimal.tex

let s:candidates = vimtex#test#completion('\cite{', '')
echo len(s:candidates)

quitall!

[benbrastmckie]

Thanks so much!