Nvim-cmp's keyword_pattern and Vimtex's Omnifunc #2786
Comments
|
@hrsh7th I think some of the points I discussed in this issue could interest you. Feedback from both ends will be beneficial. |
|
@micangl Sorry for not responding sooner! I've been busy with a lot of things, including playing a computer game - which I seldom do. As I don't have much spare time, this means the time for working with VimTeX has been at a minimum for a few weeks. I'll try to step it up and I'll begin reading your issue post in full now. |
Thanks; I appreciate your effort and I recognize that you've worked quite hard on this!
My initial thought is that I'm using nvim-cmp and I feel like I have very good VimTeX support. The following is my personal configuration (the relevant snippets):
For me, this "just works"™. I don't specify any Still, I'll give some initial thoughts to your investigations:
Ah, yes, that's a valid point. Completion triggers for me after typing two letters (due to my settings).
Cool, thanks! That seems useful!
Working with VimTeX has made me quite adept at writing regexes. ;)
I would ask: do we need that? Could we first establish the current real pain points? Let's say we recognize some real pain points: I believe a possible better way forward would be to write a full source for nvim-cmp. I believe we could use the existing completion function and possibly the existing regexes for that. I remember that I did attempt this once, but I didnt immediately figure things out and I realized that things worked well enough that I just stopped.
or by adjusting the regex (make a new one) that is magic mode. It shouldn't be too hard.
<3
It would not be surprising to me if such a proposed changed implies other changes that may not be consistent - but I wouldn't know as I've not studied this code. |
|
The main issue, currently, is that one has to add the It would also be very nice if said source would configure formatting for the completion menu, like the solution discussed in issue #2215: local cmp = require("cmp")
cmp.setup.buffer({
formatting = {
format = function(entry, vim_item)
vim_item.menu = ({
omni = (vim.inspect(vim_item.menu):gsub('%"', "")),
luasnip = "[LuaSnip]",
buffer = "[Buffer]",
})[entry.source.name]
return vim_ite
end,
},
})I've tried the pattern with vanilla nvim-cmp and it didn't work, as expected. But it does match without the leading backslashes (which is in accord with that offset issue I've talked about). But as you pointed out, it's just not worth the headache to try making the pattern work with cmp. The custom source would be nice, though, to set the |
Ok, but then it also seems as adding the addition config you suggest more or less fixes this, right?
Yes, one of the reasons I have a lot of config for nvim-cmp is precisely to improve the formatting of e.g. VimTeX completion with nvim-cmp.
Well, yes, I still agree. And I would be happy to help figure out these things. Though, I don't have much time. If someone else (you?) would take some initiative, then I could be a "passenger" and would likely be less of a "blocker". It would be fine by me to consider a PR for this, although I may end up to suggest to add it as an extension for nvim-cmp instead of a part of VimTeX itself. |
|
Yes, what I meant was that we should just create a nvim-cmp extension (plugin). I'm working on it. |
|
Great! Let me know if/when you need input/advice/help! |
|
Well, the hope of creating a custom source for cmp has already faltered. Unfortunately, there's no way for a source to set a format function in a non-destructive way (i.e. without overwriting the user's implementation). |
Ah, no, you are probably right. However, I think the conclusion may still be wrong. That is, I don't think we want a source to have a custom formatter, instead, we want it to provide the metadata in a way that is compatible with the nvim-cmp standard format. That is, the lsp completer looks like this on my end:
I believe we can have a VimTeX completer for nvim-cmp that translates information for the completion items from the omnicomplete function into a data model that fits with nvim-cmp. |
|
The custom format function which I planned to implement simply removed double quotes from the completion items. Adding it, gets you this result (no quotes surrounding the book titles:
What would you like to achieve, exactly? |
|
On my end, without changing the formatter option, I get a completion menu that looks like this:
I've made some adjustments which basically just changes
I'm not very familiar with the spec for nvim-cmp, but I believe it could be possible to leverage it to get a better experience. E.g., we could move the full info into a preview window. The complete function should call a callback function with one or more completion items according to this spec: There may be some better docs for this that I have not found yet, but I think this may point to a direction of what I want to achieve. |
The citations section of the menu looks strange; mine, even without the I've added an option to display the full I think that the result is quite lame, especially considering the huge amount a I've noticed that the
Should I add an option to set this too? Another game-changing option is the ability to fuzzy match, not only against the citation keywords, but also against the data contained in the menu field (like author, publication date, and title): P.S. I still haven't uploaded the repository on Github, but I will soon (I need to make the codebase presentable). |
|
Another big problem has arised: cmp-omni is unlicensed. I could open an issue, but @hrsh7th, the author, seems to have been inactive with regard to nvim-cmp for some weeks. The only contact information he provides is his twitter account, but I don't use it. |
|
Could you explain why it is an issue that cmp-omni is unlicensed? |
|
Because the cmp source I was working on for Vimtex is based on cmp-omni. |
|
Ok, I see. There are a lot of other sources that could be suitable as a reference, though. I bet some of them have a license..? |
|
Not really, since Vimtex still uses omnicompletion. Cmp-omni is the source for omnifuncs in nvim-cmp. I've commented under the last of @hrsh7th tweets but he hasn't answered. |
|
IMHO, the only thing about omnicompletion that is relevant here is how to fetch the completion candidates. Basically, this is all documented under My idea for how to create a custom source would be to:
|
|
Yeah, we could do that. For a skeleton source we could choose anything, such as cmp-latex-symbols or cmp-buffer. I'll still wait a little though, in case I hear from hrsh7th. |
|
He updated the license! Here's my repository: https://github.com/micangl/cmp-vimtex. Now it's ready (also added to nvim-cmp's sources page). |
What do you think about this? |
Cool thanks! You're fast! (Sorry about not being quicker in my replies. Busy life!) I'll look at your repo. Feel free to announce it on reddit (r/neovim) - it may help you get more users on board, which will help in discovering issues and thus improve things if needed.
Ah, sorry, I overlooked that comment. I'll read it and respond asap. |
Cool, that's a very nice proof of concept! I believe you can have the preview formatted by using Markdown (e.g.
Yes: The VimTeX completion functions were never developed with a preview window in mind (the concepts barely existed at the time and I've never personally had much use of it).
Perhaps. Notice that process of populating the completion candidate list with info can be somewhat slow. Fetching more data may lead to a significant slow down. One possibility would be to fetch the wanted extra data on demand. I think that could be fast enough and possibly more practical. I do this for the context menu. Pseudo vimscript code that will get a dictionary with the bibtex info for a specified "KEY": " Get all bibtex entries into a list. The pushd is to ensure we are at the root
" directory when we are locating bib files.
call vimtex#paths#pushd(b:vimtex.root)
let l:entries = []
for l:file in vimtex#bib#files()
let l:entries += vimtex#parser#bib(l:file, {'backend': 'vim'})
endfor
call vimtex#paths#popd()
" Get the desired entry
call filter(l:entries, {_, x -> x.key ==# "KEY"})
let l:entry = get(l:entries, 0, {})It should not be too hard to do something like the above in Lua. Notice, though, that for very large projects or projects with very large bibfiles, this could be slow. It could be of interest to implement a caching mechanism for the
Yes, or at least, I would prefer not to have "Text" as the indicated type. A symbol for TeX or VimTeX or something like that would be more correct IMHO.
I personally don't like fuzzy matching that much. With VimTeX and the omnicompletion, you can actually match against the menu field with regexes, but I think that has become an esoteric thing. Still, I know a lot of people like fuzzy matching, so I agree it is nice to have! |
That would surely help! But I don't use it; if you wish, feel absolutely free to make a post though, I'd appreciate it!
I hadn't though of this! It will definitely increase readability. It's a shame that Github's markdown flavor doesn't support colors, as having the specifiers highlighted would be even better. Parsing the bibtex files with that snippet is indeed slow, so a cache will be absolutely essential. Sorry if I'm dragging this on, it's my first plugin (and open source project, for the matter). I hope I'm not abusing your willingness to help too much. Here are my questions:
|
|
Okay, I've made some progress, in the meantime. Currently, I'm parsing the Bibtex files on startup (with an autocmd for I've explored the possibility of creating a separate thread, but I can't pass or receive lua tables, which makes the functionality useless for this application. Do you have any idea on how I could proceed? (At this point, I'm starting to think that I'll have to do the parsing in lua, which doesn't seem that fun...). |
|
@lervag After two weeks, I finally managed to implement the asynchronous parser. I've had to port it to lua, though. Everything, luckily, is licensed under the MIT License. So, how should I specify the copyright (i.e. what should I write in license file)?
|
|
I'm very sorry for not having had the time to reply. Life has been very busy lately. I really appreciate that you are working on this, and I would love to be able to help more. I will try to make more time to follow up more closely in the coming week. I still don't have the time to fully read your previous comment and give a detailed reply, but I'll make a quick remark to your latest update.
My suggestion would be to publish your work under the MIT license as well and add an acknowledgement in your README file where you refer to the source you've based your work on. Similar to what you've written here. By the way: would it make sense to adopt the Lua code for the parser into VimTeX? Then it could be made the default choice (if it is fast enough) for neovim users. |
When things are ready, I will test it personally, then write a section in the docs and post on Reddit. Let's keep the current issue open until I do, if you don't mind.
Is this still true with your new Lua parser?
No, not at all! I'm sorry for not having made time to follow up from my end!
The current one.
That's a hard question. I don't have the code in my "brain buffer". I can investigate and figure out a good answer if this is still relevant?
Yes; it is hard to make this stuff fully asynchronuous I think without calling out to Lua. From Lua, I think it may be possible to start a separate thread with coroutines.
Ah, yes; you're clearly way ahead of me already. And if I understood correctly, you already did this. You are still working with this repo, right? I didn't see the parser code, is it in a separate repo? Would you say it is a good time for me to test things? |
For large files, like the one Vimtex uses for tests, it still may not be instantaneous. So, the parsing is started asynchronously on a
I investigated a little and there doesn't seem to be a way, with Neovim's api, to check the last-modified timestamp of a file. But it should be easy to just call a terminal program (like stat).
Currently, I was keeping the changes on my local repo. I'll update it today. Also, the configuring options have changed as you'll be able to see in the README. About the asynchronous execution of the parsing, it was so messy to figure out... Luckily, hrsth7th, for his Also, there are more features that, in the future, I'd like to implement:
This last feature, I think, could be very useful to check things on the fly when writing a paper. Thanks a lot for your support! |
Cool, but I think this also holds for the other parsing methods. I have a test for this (
Nice. Notice, when I talk about caching, what I mean is to store the result of parsing a single file. If the file was not changed, then instead of parsing it, we can just load the stored result.
Oh, yes, there are apis for this. Since Neovim's Lua APIs allow direct access to the builtin functions, you can just use them. E.g. vimtex/autoload/vimtex/cache.vim Lines 224 to 227 in cbb2064
Great, I look forward to having the time to test this! Hopefully within this week.
Nice. Notice that the context menu already provides some of this (see One last thing: You didn't respond to one of my questions:
|
If I understand correctly the point you are raising, the parsing is not done after every BufWinEnter event, but only one time. The results then get stored in the lua table and, when another event takes place, the autocmd's callback first checks if the file was already parsed (if this is the case, then, the file doesn't get parsed again, as it would be useless and inefficient).
Cool, this criteria will definitely be added for determining if a file should be reparsed.
The use-case I thought of is the following: you're inserting a citation and you can't remember which is the correct one, as there may be many with slightly different titles/keys/etc. So, you select one in the completion menu and, with a shortcut, open the bibtex file(for modification), or the Doi, or the pdf (exactly as the context-menu does), or you can search for it on a specific research database or through a standard search engine (maybe even opening Zotero could be implemented). The key difference, here, is that the databases/engines should also be specifiable by the user (a feature which could also be implemented in Vimtex), and that this operations would be done on-the-fly, while you're choosing the citation and not after you've chosen it. This, I think, could turn out to be very useful for people who write papers and have to deal with a lot of literature.
Sorry, my mistake. I don't really see why it couldn't be. The delay for opening the context-menu up, when dealing with very big bibtex files, is consistent. Lua will probably speed it up, as testing will might indicate. The "main" parser function has been adapted to be called asynchronously, but it could easily be reverted to just being a simple translation of the vimscript code into lua. |
Ok. This is not a feature I personally want or need as I can't really say I have this problem very often. But people are different and I guess someone would like this very much. And I think it does work out nicely as a feature from a VimTeX compatible completion plugin!
I think the asynchronous part could just as well be moved one step up, right? |
|
I've started to test things now. My first two comments:
|
A nice thing to do, in the future, could be to add the online databases search functionality to Vimtex, so that it could be used in the context menu, allowing cmp-vimtex to call this functions directly from Vimtex, so as not to have redundant code. But this is just a thought as, currently, I don't have a lot of time available.
The problem with Lua, in such a context, is that it's not really asynchronous. Lua provides coroutines, which give the semblance of asynchronousness by allowing software to schedule execution of a function on the main event-loop. If the scheduled functions are small there is no harm in this, but, if they take some time to complete (like for parsing), the main event-loop is occupied with this task, effectively freezing Neovim. The problem is solved by parsing a small chunk of the file, and then scheduling execution of the same function to continue. But, if you want, I can provide you with a literal translation of your integrated parser; the function I need to change is only the main one. |
|
I've mostly converted your lua parser to a synchronous version now, see here: I'll test it some more and see how it compares to the other methods in terms of speed when I have time to continue this. |
I've tested it in a VM with the following minimal configuration:
I agree. As you can see, the setup function registers the source with EDIT: I've added this feature. |
|
Thanks for the quick and detailed response! I'll check things again later when I get the time. I've continued work on the lua parser in VimTeX now and it should work. Performance wise it seems very good. On the speed test I have running, it seems to be faster than the previous fastest parser (which is, fascinatingly, to run bibtex and then source the output). I think I will end up using this as the default backend for neovim users in not too long. :) |
|
The speed test is also running in the ci pipeline, so you can see the results here: https://github.com/lervag/vimtex/actions/runs/6606130297/job/17942017427?pr=2816#step:4:102 |
|
@micangl @lervag I just want to add that your work on the recent cmp plugin has been a huge quality of life upgrade for me, and has made a huge difference to some of the issues I've had in using autocompletion with vimtex. Part of my issue is not being able to articulate all the little issues I've had in the past (things like autocompletion not triggering on '{' and also the lack of fuzzy finding on bibliography data (this has been asked multiple times by people on Reddit and otherwise). Please continue the development of the cmp plugin and also @lervag I hope you also work to integrate cmp-vimtex. Thank you!! |
|
@ThSGM Thanks a lot! If you feel like there are some features missing, don't hesitate to open an issue. I've also checked your nvim-cmp config, lervag, and I can't understand why you're missing information from the completion menu. Maybe something unexpected is happening in the |
This is similar to the Vimscript parser ("vim"), but since it is in Lua
it is much faster and comparable to the current fastest parser
("bibtex").
refer: #2786
This is similar to the Vimscript parser ("vim"), but since it is in Lua
it is much faster and comparable to the current fastest parser
("bibtex").
refer: #2786
Credit goes to @micangl here, I've just helped on conceptual stuff really. But I'm glad to hear feedback from other users; I can see how this gives a noticable improvement to many users.
I will integrate it in the sense that I will refer to it in the documentation. Perhaps @ejmastnak may be interested in updating his tutorial as well, at least when cmp-vimtex has reached some more maturity?
Thanks; I'll look into this. Now, I want to bring your attention to #2816 - a PR where I add support for the Lua backend to VimTeX. I've adjusted your implementation somewhat and included it, and as I reported earlier it seems to work quite well. I would be very glad to get some feedback on it before I merge. In particular, I've removed the license header from the source file as I instead use a global license file and a simple header for all the source files. Before I merge I also want to check if I can further improve the parser somewhat. I'm not quite happy with the code, it doesn't feel quite right. |
|
Regarding my problem with getting the preview window: it seems to be some strange issue with plugin loading. I use lazy.nvim, and I loaded cmp-vimtex as a dependency for lazy.nvim. I tested with I've opened an issue for that: micangl/cmp-vimtex#1. |
This is similar to the Vimscript parser ("vim"), but since it is in Lua
it is much faster and comparable to the current fastest parser
("bibtex").
refer: #2786
This is similar to the Vimscript parser ("vim"), but since it is in Lua
it is much faster and comparable to the current fastest parser
("bibtex").
refer: #2786
This is similar to the Vimscript parser ("vim"), but since it is in Lua
it is much faster and comparable to the current fastest parser
("bibtex").
refer: #2786
|
Thanks to some useful comments by @clason, I've managed to further improve the performance of the Lua bib parser. It is now significantly faster than the bibtex parser (which was usually the fastest one). On my end it is often near twice as fast. 🥳 |
|
The main improvements come from using the Lua pattern matching instead of regexes. Regular expressions are powerful, but Lua patterns are powerful enough in most cases and much faster. |
|
If you want to check on your side, go to the VimTeX folder, then to (The 2nd and 3rd runs are just for checking that the caching of completion items work as expected.) |
|
I realize it is much easier to follow up on the remainings tasks from this thread by opening a new issue for that, so I just did: #2818. I'll therefore close this issue now. |
Description
Vimtex support in nvim-cmp is an issue which has been extensively discussed. These are just some of the related issues:
cmp-omnisource not displayed whennvim_lspis enabled hrsh7th/nvim-cmp#833The problem explained in those issues dealt mainly with the Vimtex-provided neocomplete pattern (https://github.com/lervag/vimtex/blob/master/autoload/vimtex/re.vim#L35), why it didn't work, and why cite-completion wasn't triggered automatically (i.e. the completion menu appeared) as soon as a
\cite{statement was typed.I may have found some workarounds and/or hints towards a possible solution. I'll explain them in order:
Cite-completion isn't triggered automatically as soon as a
\cite{statement is issued. Differently from other engines, with nvim-cmp the user first has to type a character to have the menu pop up and show bilbiographic references. This is obviously annoying, as the user may not always remember, at the top of his head, exactly which reference to insert (or the key name). A workaround which I don't think was ever discussed is to use atrigger_characterin nvim-cmp's configuration, as such:{ name = "omni", trigger_characters = { "{" } },Correctness of the Vimtex-provided Neocomplete pattern. When the discussions in the previously linked issues were held, the validity of said pattern was questioned and, if I recall correctly, noone ever managed to make it work with nvim-cmp. The pattern, though, is indeed correct. This can be tested by opening the following file in Neovim and sourcing (
:so):And now the hardest issue: Making that pattern work with nvim-cmp. I've only partially achieved this goal, and I write this issue with the hope that someone more expert than me will be able to help.
First thing, nvim-cmp modifies the
keyword_patternprovided by the user:https://github.com/hrsh7th/nvim-cmp/blob/5dce1b778b85c717f6614e3f4da45e9f19f54435/lua/cmp/context.lua#L77
This is necessary to ensure that only the last-typed text will be matched, as discussed in Custom completion source - completion popup not showing up hrsh7th/nvim-cmp#1273. The problem is that such a modification assumes to be in magic mode, whilst the keyword pattern we would be using set
\vvery-magic mode. This can be fixed by either adding a\mat the end of the pattern we pass to nvim-cmp, as suchor by correcting nvim-cmp's handling. I still haven't made a PR, but it should be like this:
After making such an addition, nvim-cmp still doesn't match. It took me multiple hours of debugging but, by modifying cmp/source.lua
nvim-cmp start finally to match correctly and show the completion menu. The issue I've found is that the completion menu, with such a modification, doesn't appear after typing a
\cite{statement. I've tried to troubleshoot this, but I'm not a programmer and the code in that nvim-cmp's function is just too complicated for me.If someone who has the expertise would like to help, I'd be happy to contribute where I can. I honestly hope that we will, finally, be able to solve this stubborn issue.
Finally, sorry for bearing with me 'till the end of this message.
VimtexInfo
The text was updated successfully, but these errors were encountered: