Add support for local bibs

[bdarcus]

Sometimes, specially when working collaboratively in a paper, it is useful to have the .bib file of the current document (that sometimes is a local file and not your bib database) added to the bibtex-completion-bibliography.

Originally posted by @ramos in #148

Not sure what's involved, or when I'll get to it.

Should work both for latex-mode, but also for other relevant options; certainly (the soon to be merged) org-cite. Pandoc?

A PR is welcome.

[bdarcus]

https://github.com/tmalsburg/helm-bibtex/blob/master/ivy-bibtex.el#L156

[jsilve24]

Just want to bump this - this would be nice.

[bdarcus]

Yes, will get to it, though there are some nuances to figure out with org-cite.

One user posted this suggestion as a starting point, BTW:

https://gist.github.com/bdarcus/241e24ed568817a7229fa237bb4c5f84

[bdarcus]

I just put this on the "roadmap" for 1.0, just so people know it's a priority.

I just have a lot going on, so a PR would definitely be appreciated.

[bdarcus]

Someone just pointed out a potentially obvious solution for org-cite, which is just to use org-cite-list-bibliography-files.

Can it possibly be that easy?

In related news, bibtex-completion recently added support for org-cite bibs.

tmalsburg/helm-bibtex@6dd1702

[bdarcus]

@tmalsburg - can you help me understand how to think about the local-bib case?

I want to support both latex and org-cite, with a higher priority on the later.

Does it matter that org-cite can mix local and global bib files?

How does your recent commit for bibtex-completion impact all this on my end?

Any tips appreciated.

[aikrahguzar]

I have been working on this for latex. But my elisp skills are weak and I would like to understand the case better too. The scenario that seems the best ux wise to me is the following,

1. there is a buffer local cache containing the combined entries from local and global bib files, this seems doable.
2. we somehow keep track if an entry is global, local or both. I don't am not sure how easy this is.
3. If a global only key is inserted into the document we add it to the local file too. This seems doable but might prove beyond me.

In this scenario we don't have to have separate local versions of the commands.

The other lazy scenario is to just add the commands that insert keys into the document for local bibliography. This enough for my use case and gets the ball rolling and I have a working version of this.

The last scenario involves duplicating everything between local and global bibliographies and that is doable but seems least appealing to me.

[aikrahguzar]

I guess another option is to just do 1) from the list above and leave the user to deal with missing entries in the local bibliography. This is doable and works fine if you have exclusively a global or local bibliographies. It is not ideal but seems useful enough.

[bdarcus]

Hi @aikrahguzar!

I'd actually like to step back further, and think about how this should work from a user POV first.

Let's review:

org-cite

In org-cite, local and/or global bib loading is configured by the user, via a setq and/or local document settings, and org-cite then automates which files to load via org-cite-list-bibliography-files.

I have not yet tested, or figured out, how this commit impacts this, but I think this is how it works:

tmalsburg/helm-bibtex@6dd1702

It's just that in ivy-bibtex and helm-bibtex the user has the choose to load local bib files (see below for why I wonder whether that's necessary).

biblatex

Biblatex basically does similar, where all bib files are listed in the document, and where there really is no distinction between "local" and "global".

\addbibresource{/home/bruce/Code/org-mode/test.bib}
\addbibresource{~/org/bib/newer.bib}
\addbibresource{~/Code/org-mode/test.bib}

latex

It seems the same?

\bibliography{file1,file2}

Thoughts

From a user POV in bibtex-actions, then, couldn't this all be transparent?

If I have a buffer loaded that specifies only to use a local bib file, then that's what I should see, without any (at least required) intervention in bibtex-actions?

The obvious advantage is the user sees the same list of files that the processor (org-cite, bibtex, etc.) sees, which is what we want in the end.

EDIT: only wrinkle I see ATM is what if I haven't test specified this?

Does all of this make sense so far?

If we can settle the high-level UX question, then should be easier to settle the best technical approach.

[aikrahguzar]

Hi @bdarcus, this seems like the last scenario and I think that will work. I actually don't have a global collection, so I was trying to imagine what someone who has both would like. There the best ux seems to allow them to build up the local files by inserting the entries they need from the global one into both the document and the local bibliography. There can be multiple files so we somehow have to figure out where to insert these entries but we can just prompt in that case.

That can be left as something to work out later and if we just restrict ourselves to working with the files in the document like you are suggesting, I think I can get a working version.

[bdarcus]

I actually don't have a global collection, so I was trying to imagine what someone who has both would like.

Good; I'm the opposite, so between us we should be able to figure it out!

Might be good to get some more input from people here.

There the best ux seems to allow them to build up the local files by inserting the entries they need from the global one into both the document and the local bibliography. There can be multiple files so we somehow have to figure out where to insert these entries but we can just prompt in that case.

This is an additional requirement that would complicate things if we were to support it. My impulse is to say we shouldn't worry about it initially. WDYT?

That can be left as something to work out later and if we just restrict ourselves to working with the files in the document like you are suggesting, I think I can get a working version.

Well, to get a little into implementation details, my hunch is:

1. For org-cite it may be possible to get that module to use org-cite-list-bibliography-files, and it will handle all this without any code needed from us, except possibly for knowing when to refresh the cache. As in, if a user has org-cite-global-bibliography specified, that list of files will be appended to whatever local ones might be present.
2. For LaTeX, I guess you'll just need to align this somehow with 1, so it's similarly straightforward for the user. I'd suggest you look at the recent changes in bibtex-completion here. It might be not hard at all.

I created a local-bibs branch, that you can open a PR (draft if you wish) against when you have something, and we can go from there.

[bdarcus]

On my org-cite "hunch", yeah, this is what John Kitchin appears to be doing in org-ref-cite:

https://github.com/jkitchin/org-ref-cite/blob/6fd4ed70ae6311f535c5fa2c5de70a6eb44eca1b/org-ref-cite-follow.el#L48

[bdarcus]

So for org-cite, maybe something like this logic in the end:

• set org-cite-global-bibliography to the list value of bibtex-completion-bibliography
• update the latter to use org-cite-list-bibliography-files, which will fold in any local files
• if it's not set, then set to
  org-cite-list-bibliography-files

That should be it I think.

[aikrahguzar]

This is an additional requirement that would complicate things if we were to support it. My impulse is to say we shouldn't worry about it initially. WDYT?

I agree, best to not worry about it now for now.

So the summary seems to me to be,

1. Change cache to be buffer-local.
2. Have a major-mode dependent way of populating it. For org we rely on org-cite-global-bibliography, for latex we instead do the opposite of what we are doing now and use the local bibliography exclusively. This seems the correct way to me since the generated document wouldn't be able to use global bib anyway.

Any other major modes to worry about?

This seems doable, I think I should be able to have a working version no later than this weekend, though I am still figuring this git thing out.

[bdarcus]

Yes, but I don't think you'll need 1. I think we just need to refresh the cache when opening a new document?

We should also support markdown with pandoc, at least at some point.

I may get started on the org angle, which involves me finding a way to do this, dependent on the major-mode:

(let ((bibtex-completion-bibliography (org-cite-list-bibliography-files)))

So we could extend that to work for the other modes, perhaps put behind a simple function? Like:

(defun bibtex-actions-list-bibliography-files (mode)
  "Return complete list of global and local bibliography files for MODE."
  ;; conditional statements based on major-mode/featurep

[aikrahguzar]

Yes, but I don't think you'll need 1. I think we just need to refresh the cache when opening a new document?

The reason I would like it to be buffer local is that I can have a document with different local bibs open at the same time sometimes. I actually have a hook that generates the local bib when on an idle timer when I open a latex document.

(let ((bibtex-completion-bibliography (org-cite-list-bibliography-files)))

I think we only need this in the bibtex-actions-refresh with how we are doing it now though I obviously need to test it.

So we could extend that to work for the other modes, perhaps put behind a simple function? Like:

(defun bibtex-action-list-bibliography-files ()
  "Return complete list of global and local bibliography files."
  ;; conditional statements based on major-mode/featurep

Another way would to use an alist of major modes and corresponding functions, which would allow it to be extended from outside the package.

[bdarcus]

We might need the extra cache; I'm not sure either.

I'll experiment without it and see where that gets me.

[bdarcus]

I think the only place indirectly tied to bibtex-completion-bibliography is bibtex-actions--format-candidates; here specifically.

https://github.com/bdarcus/bibtex-actions/blob/1119f98d5d31dc715957b6c52e6b51d59122f601/bibtex-actions.el#L251

[aikrahguzar]

I think the only place indirectly tied to bibtex-completion-bibliography is bibtex-actions--format-candidates; here specifically.

https://github.com/bdarcus/bibtex-actions/blob/1119f98d5d31dc715957b6c52e6b51d59122f601/bibtex-actions.el#L251

The only function that call ibtex-actions--format-candidates is bibtex-actions-refresh and it was smaller and less scary for me to fiddle with. And bibtex-actions-refresh set the cache so I needed two versions of it with how I was doing thing before.

[bdarcus]

I've not tested it, and the code itself probably needs work, but something like this is what I'm thinking to start with for org-cite, that can probably be generalized for latex and markdown/pandoc?

https://github.com/bdarcus/bibtex-actions/pull/208/files

I haven't thought in detail about the cache though. Ideally, we only want it regenerated when needed, and without user needing to manually do it.

[tmalsburg]

@tmalsburg - can you help me understand how to think about the local-bib case?

I want to support both latex and org-cite, with a higher priority on the later.

I'm not sure what you're trying to achieve exactly but I think it's sufficient to grab the bibliographies from (bibtex-completion-find-local-bibliography) and let-bind them to bibtex-completion-bibliography before retrieving candidates via (bibtex-completion-candidates).

Does it matter that org-cite can mix local and global bib files?

In the case of org files, bibtex-completion-find-local-bibliography will return all bibs, local and global. Not sure whether that's what you want but it's consistent with org-cite I think.

Hope that helps.

[bdarcus]

I think that's exactly what we need; thanks @tmalsburg!

[bdarcus]

OK, I reverted my initial commit and added one based on Titus' suggestion. It's just a start.

53073d9

[bdarcus]

In case anyone can help with testing, this is now working on the local-bibs branch.

The intended behavior is that any file you visit, at least in latex and org-cite, bibtex-actions-refresh will get you the intended mix of local and global bib candidates automatically.

If feedback, please post to #208.

[aikrahguzar]

The intended behavior is that for any file you visit, at least in latex and org-cite, bibtex-actions-refresh will get you the intended mix of local and global bib candidates automatically.

I think refresh is not strictly needed. The local candidates should be generated the first time a bibtex-action command needing them is called. Though this call will be slower than the subsequent calls.

[jsilve24]

Just tried this branch out. I just tried playing around with it and from my initial tests all looks good!