support local bibs

[bdarcus]

This adds support for local bib files, by adding a buffer-local cache.

When visiting a buffer, you should get the correct mix of local and global bib files loaded automatically.

Also adds a hidden is:local string to local candidates, so that you can filter local items like so:

Open questions:

1. Should local candidates already included in the global cache be removed from the local cache?
2. Best options for auto-freshing of cache so users don't have to think about this? Maybe we include an example auto-timer hook in the README? settled; working on this
3. Should there be some kind of UI hint for local candidates, building on the above filtering? We discuss below grouping as one option, but there are more subtle alternatives. Think, for example, how presence of related PDFs or notes is currently indicated in the affixation prefix (though if we go this way, we need a good default symbol). For example:

... or, different symbol:

Fix #150.

[bdarcus]

Do we agree at this point the solution to getting the right files is solved, and the remaining piece is just how to make sure the candidates are appropriately in sync with whatever document we're working on?

For you, that means the candidate list will change, and so necessarily the cache refresh, depending on which document you have open; for me, usually not.

But no with a buffer local cache, which seems ideal to me.

If you still think this is an idea you want to champion, how would the buffer local cache work? Does that mean a separate candidates list? What makes it "ideal", with what cost, if any?

I can also see it being annoying for some one with global bib file but it should be about the same as refreshing it one opening a new buffer in terms of time spent doing it.

Ideally, none of us find anything about the ultimate solution "annoying"; it should be elegant, and we basically shouldn't have to think about it. Oh, and easy to understand and maintain.

That's the goal anyway.

[bdarcus]

In any case, can I delegate this piece to you? Might be I can merge this piece first, and you could follow-up with a separate PR focused on the syncing piece.

But whether that makes sense depends on the solution.

Could be input from the doom discord might be valuable here?

[bdarcus]

Idea: rather than cache candidates, cache bib file names.

We then know we need to reset the cache if the file list of the current buffer differs from the file list cache.

Just a question of how to initiate the check automatically. From a quick check, it seems like the mode hooks for latex, markdown, and likely org will run when the mode is initiated. Shouldn't that work here?

Or is that what you have in mind?

Alternately, we can just not worry about this part. As you note, one can already refresh cache manually (with a prefix arg, etc).

[aikrahguzar]

Do we agree at this point the solution to getting the right files is solved, and the remaining piece is just how to make sure the candidates are appropriately in sync with whatever document we're working on?

Yes

If you still think this is an idea you want to champion, how would the buffer local cache work? Does that mean a separate candidates list? What makes it "ideal", with what cost, if any?

I have been using a buffer local cache (i.e with cache variable defined by defvar-local instead of defvar) in my config. The reason this is ideal to me that I can work with latex documents with different bib files have the correct cache by construction. The only downside I can think of is memory usage because the candidate list is unique to the buffer. For me the effect has been negligible with 2-3 latex-mode buffers with bib files containing about 1500 entries. On the positive it does save time spent on refreshing the cache and we don't have to worry about refreshing it when the focused document changes.

I have a hook that run the refresh command in my latex-mode-hook and it works pretty well

Ideally, none of us find anything about the ultimate solution "annoying"; it should be elegant, and we basically shouldn't have to think about it. Oh, and easy to understand and maintain.

That's the goal anyway.

To me the defvar-local seems the simplest in terms of maintainability but then I have never really maintained software used by other than me.

[aikrahguzar]

Alternately, we can just not worry about this part. As you note, one can already refresh cache manually (with a prefix arg, etc).

This would be my second preferred solution.

[aikrahguzar]

And asking on doom discord seems like a good thing. While trying to familiarize myself with how bibtex-completion works under the hook, I saw that reftex which is used under the hood has a function for generating a new bib file consisting of all the entries cited in the document from a global bib file. Some people might have a workflow built around that.

[bdarcus]

I have been using a buffer local cache (i.e with cache variable defined by defvar-local instead of defvar) in my config. The reason this is ideal to me that I can work with latex documents with different bib files have the correct cache by construction.

But what's the practical difference between the global cache that is refreshed vs a local one? Isn't it the same UX in the end?

If we do add this, how would we integrate it with this PR? I assume in bibtex-actions--get-candidates?

[aikrahguzar]

But what's the practical difference between the global cache that is refreshed vs a local one? Isn't it the same UX in the end?

If we do add this, how would we integrate it with this PR? I assume in bibtex-actions--get-candidates?

The only change needed would be to change

(defvar bibtex-actions--candidates-cache nil)

to

(defvar-local bibtex-actions--candidates-cache nil)

I don't think there would be any practical difference in ux but it would be much less code.

[bdarcus]

I don't think there would be any practical difference in ux but it would be much less code.

I can make the change here and test it (edit: just pushed).

How would it impact the case of the user who doesn't use local bibs; only global?

[aikrahguzar]

How would it impact the case of the user who doesn't use local bibs; only global?

From how the cache is computed in this pr, I think the only effect would be that they would get the same cache duplicated in different buffers. So higher memory usage is all I can see.

But even for them compared to global but refreshed cache it would smoother time wise if they switch buffers frequently since the cache wouldn't be refreshed every time.

[bdarcus]

How would it impact the case of the user who doesn't use local bibs; only global?

From how the cache is computed in this pr, I think the only effect would be that would get the same cache duplicated in different buffers. So higher memory usage is all I can see.

Higher memory usage, and a brief pause when changing buffers. Could be an issue if one has a very large global bibliography?

But even for them compared to global but refreshed cache it would smoother time wise if they switch buffers frequently since the cache wouldn't be refreshed every time.

OIC. You're saying the local caches would be maintained across the buffers?

If yes, I get it now.

But the above issue seems like something worth thinking about.

[aikrahguzar]

Higher memory usage, and a brief pause when changing buffers. Could be an issue if one has a very large global bibliography?

There shouldn't any pause when changing a buffer.

OIC. You're saying the local caches would be maintained across the buffers?

Yes

If yes, I get it now.

I am using this and it seems to be working great for me.

[bdarcus]

There shouldn't any pause when changing a buffer.

Say I'm working on a book, with a single large global bibliography.

I have two buffers open, representing two chapters.

Would not they each get their own duplicate buffer-local cache, created upon opening those buffers (though not when changing between them)?

Maybe I can just merge this change and document it, in case we need to address this later?

[bdarcus]

Maybe I can just merge this change and document it, in case we need to address this later?

No; I think we need to figure this out.

Consider someone using org-roam, where they have many buffers opening.

[aikrahguzar]

I have two buffers open, representing two chapters.

Would not they each get their own buffer-local cache, created upon opening those buffers (though not when changing between them)?

This will happen (assuming a hook runs then). I have one on in my latex-mode-hook that runs with an idle timer so actual bibtex-actions ui always seems fast. This is the hook,

(defun gen-bib-cache-idle ()
  "Generate local bibtex cache with idle timer"
  (run-with-idle-timer 0.5 nil #'bibtex-actions-refresh))

Maybe I can just merge this change and document it, in case we need to address this later?

This seems good to me.

I ran (memory-report-object-size bibtex-actions--candidates-cache) on a buffer with almost 1500 entries, I see 723275 bytes ~ 0.7 MiB. On another buffer with about 300 entries I see 155518 bytes ~ 0.15 MiB which doesn't seem too bad.

Though the entries are pretty simple and might not be representative.

[aikrahguzar]

No; I think we need to figure this out.

Consider someone using org-roam, where they have many buffers opening.

One solution would be to use a global cache containing the entries from global file only and one buffer local cache containing the local bib only and concat them before offering candidates. This will work if concatenation is fast. But the complication would be that we can no longer simply usebibtex-completion-find-local-bibliography since it return both local and global files for org.

[bdarcus]

Oops; cross-posting. Hmm ...

I ran (memory-report-object-size bibtex-actions--candidates-cache) on a buffer with almost 1500 entries, I see 723275 bytes ~ 0.7 MiB. On another buffer with about 300 entries I see 155518 bytes ~ 0.15 MiB which doesn't seem too bad.

But if, as with the org-roam case, there are 20 buffers open, with a 2000 item global bib?

If this is worth addressing, it seems that would suggest your original two caches idea, which I didn't understand earlier:

(defvar bibtex-actions--global-candidates-cache nil)
(defvar-local bibtex-actions--local-candidates-cache nil)

We then only use the later when it's needed; e.g. when the local bib list differs from the global one.

Right?

This will happen (assuming a hook runs then). I have one on in my latex-mode-hook that runs with an idle timer so actual bibtex-actions ui always seems fast. This is the hook ...

We should probably add this to the README regardless.

[aikrahguzar]

But if, as with the org-roam case, there are 20 buffers open, with a 2000 item global bib?

Even 20MiB doesn't seem too bad to me.

If this is worth addressing, it seems that would suggest your original two caches idea, which I didn't understand earlier:

(defvar bibtex-actions--global-candidates-cache nil)
(defvar-local bibtex-actions--local-candidates-cache nil)

We then only use the later when it's needed; e.g. when the local bib list differs from the global one.

Right?

This does seem worth doing and I will try to get something like this working over the weekend.

And my idea earlier was wound up in too many details and without a good ux but I think we have settled on a good one here.

We should probably add this to the README regardless.

Please go ahead.

[bdarcus]

Even 20MiB doesn't seem too bad to me.

OK, we can try if you can't figure out a straightforward way to do the below.

This does seem worth doing and I will try to get something like this working over the weekend.

👍

We then only use the later when it's needed; e.g. when the local bib list differs from the global one.

Something like this on bibtex-actions--get-candidates:

  ;; code to determine which cache to use; if nil, use local
  (let ((global-cache (bibtex-actions--use-global-cache))) ; return t if local and global file lists are the same

We may also want a defcustom in the end.

[aikrahguzar]

Question: What is our working definition of global bib files? The simplest answer seem to be whatever is in bibtex-completion-bibliography and I am inclined to work with it for the time being at least. But I don't know what if any is the org's idea of global bib files.

[bdarcus]

Question: What is our working definition of global bib files? The simplest answer seem to be whatever is in bibtex-completion-bibliography and I am inclined to work with it for the time being at least. But I don't know what if any is the org's idea of global bib files.

Exactly.

It probably won't work as is, but here's what I was playing with; need to think more on org though.

(defun bibtex-actions--normalize-paths (file-paths)
  "Return a list of FILE-PATHS sorted and normalized with truename."
  (cl-sort
   (delete-dups
    (mapcar (lambda (p) (file-truename p)) file-paths)) 'string-lessp))

(defun bibtex-actions--use-global-cache ()
  "When non-nil, use the global cache."
  (let* ((local-bib-files
          (bibtex-actions--normalize-paths (bibtex-completion-find-local-bibliography)))
         (global-bib-files
          (bibtex-actions--normalize-paths bibtex-completion-bibliography)))
    (equal local-bib-files global-bib-files)))

[aikrahguzar]

What I was thinking of was actually slightly different, we should always use both caches but concatenate them when offering candidates to the user. The global caches is populated with candidates from global files, while the local cache contains the differences of local files and global files, i.e morally

(cl-set-difference local-files global-files)

This seems ideal in terms of memory usage. If there are 20 buffers each with the same big global file but small local files the big global file is present in a single cache.

This is assuming that while constructing the cache most of the time is spent formatting and the concatenation is quick.

I think we should take the same approach with the latex too. if the user has a declared a global file they probably want to use it and reftex can be used to extract a locally used entries from the global file and we should put in the README explaining how to do that.

[bdarcus]

(cl-set-difference local-files global-files)

Seems we want seq-difference though? I'm agnostic on the details. Whatever is cleanest (including easiest to understand), and works the best.

If there are 20 buffers each with the same big global file but small local files the big global file is present in a single cache.

Yes, that definitely makes sense.

reftex can be used to extract a locally used entries from the global file

You'd want to confirm bibtex-completion doesn't already have some feature for this. I actually don't know.

[bdarcus]

Do note, though, that I think bibtex-completion normalizes across bib files, removing duplicates and such. Not sure it matters, but ...

[bdarcus]

the local cache contains the differences of local files and global files,

So more like this (seq-difference in this sequence seems what we need here?):

(defun bibtex-actions--local-files-to-cache ()
  "The local bibliographic files not included in the global bibliography."
  (let* ((local-bib-files
          (bibtex-actions--normalize-paths (bibtex-completion-find-local-bibliography)))
         (global-bib-files
          (bibtex-actions--normalize-paths bibtex-completion-bibliography)))
    (seq-difference local-bib-files global-bib-files)))

I pushed a commit with these two small functions, though I didn't figure out how to hook them up.

[aikrahguzar]

(cl-set-difference local-files global-files)
Seems we want seq-difference though? I'm agnostic on the details. Whatever is cleanest (including easiest to understand), and works the best.

I don't know emacs api that well and don't understand the differences between data structures manipulation modules, I picked up cl-set-difference because it seemed to do what we want and there is a lot of cl- stuff in the file.

[aikrahguzar]

Do note, though, that I think bibtex-completion normalizes across bib files, removing duplicates and such. Not sure it matters, but ...

Duplication is definitely a concern, from going through the function, it seems like it does deduplications by comparing hashes derived from bib data. We should be still able deduplicate by first constructing the global cache and checking against it in addition while constructing the local cache. Though I think this should be a second order concern

[bdarcus]

Last change for the day, I think:

I separately culled some outdated org-cite code no longer needed, and rebased everything here on top of that, and merged it all together into one commit.

Also, I updated the main post with the current status, and a screenshot with "local" filtering.

[bdarcus]

BTW, on your hook, can you just include the code you use to run the hook?

Maybe we can include an example that works for all three primary modes: latex, org, markdown?

[aikrahguzar]

BTW, on your hook, can you just include the code you use to run the hook?

It is this

(add-hook! 'LaTeX-mode-hook #'outline-minor-mode #'outline-hide-body #'auto-fill-mode #'gen-bib-cache-idle)

Though add-hook! is a doom convenience.

[bdarcus]

OK, updated the README with the basics, and the hook example.

Does it need anything more?

[aikrahguzar]

This hook won't help with stale cache, I include it so that the cache is generated when a new buffer is opened making the ui seem more responsive.

For stale cache filenotifyis the solution though it won't work with local cache as is.

[bdarcus]

For stale cache filenotifyis the solution though it won't work with local cache as is.

As in, they conflict, or (as I assume) they are complementary? If the latter, I'll add it back and adjust the language.

[aikrahguzar]

They are complemtary. I think another hook will be needed that adds a watch on the files returned by bibtex-completion-find-local-bibliography to make it work with local cache.

[bdarcus]

Can you just use "suggested changes" to suggest the specific line changes then? Just click the left most icon on the toolbar.

Allows me to just click to accept, and then you get credit in the git history.

[aikrahguzar]

I adjusted the language, I will have to look into filenotify a bit to write and test the hook for cache update.

[localauthor]

I just gave the local-bibs branch a shot and figured I'd report that all worked for me as expected, straightaway. All it took was adding a local bib file to the #+bibliography: header argument and refreshing the setup (C-c C-c with point on header arguments). And I made sure the local bib had some entries overlapping with the global bib and some unique entries. Results:

• After calling org-cite-insert, all local and global candidates were listed together, without any duplicates.

• After removing the local #+bibliography: header argument and refreshing the setup, the cites from the local bib are marked as no longer valid (face changes to error), while those from the global bib, including the cites listed in both bibs, remain valid.

• Added entry to the local bib, refreshed setup (C-c C-c on the header args), and found the new entry upon calling org-cite-insert. (Surprisingly, it wasn't necessary to call bibtex-actions-refresh. Is that expected?) Same in reverse, for removing entries.

• Exports generate as expected, no missing item data.

Any other testing you're interested in while I'm on this branch? Seems to be in order, from here. In any case, thanks for your work on this package!

[bdarcus]

I just gave the local-bibs branch a shot and figured I'd report that all worked for me as expected, straightaway.

Great; thanks for confirming!

• Added entry to the local bib, refreshed setup (C-c C-c on the header args), and found the new entry upon calling org-cite-insert. (Surprisingly, it wasn't necessary to call bibtex-actions-refresh. Is that expected?)

I'm not sure actually. I think it will create the local cached data the first time you run a bibtex-actions command on a "new" buffer with local bib. Does that explain it?

Any other testing you're interested in while I'm on this branch? Seems to be in order, from here. In any case, thanks for your work on this package!

Not that I can think of.

Do you have any input on the "open questions" in the main post; particularly 1 and 3 (we have already started to add 2 to the README)?

[localauthor]

Does that explain it?

Yes, that makes sense.

input on the "open questions"

On question 1, not really; I don't really know how caching works, or what's at stake in removing entries from local cache.

On question 3, some way of distinguishing local from global would be nice:

• Grouping seems the more actionable option, in terms of quickly narrowing to either local or global entries. I can see if I were to collaborate with someone, and wanted to use their bib entries in a document without 'polluting' my precious global bib, it might be helpful to be able to select from a grouping of their bib entries. But a complication might be: which group would any common entries be in? Could they be in both?

• Affixation prefixes are nice, but mostly visual, rather than actionable, right? Or, would there be a way to narrow to just local or just global entries? As for indicators, rather than try to find properly communicative symbols, why not simply letters: l, g, or l/g?

[bdarcus]

Per the screenshot, you can already filter the local candidates from the prompt. The ideas are just designed to visually indicate that.

[aikrahguzar]

I changed the handling to setting caches but mostly superficially. But I think they made the code more readable and simpler. I have tested these changes and they work (apart from any mistakes I might have introduced going through them on github piecewise).

[aikrahguzar]

2. ~Best options for auto-freshing of cache so users don't have to think about this? Maybe we include an [example auto-timer hook](https://github.com/bdarcus/bibtex-actions/pull/208#issuecomment-892589239) in the README?~ settled; working on this

I got it work but it is somewhat non-trivial

(defun bib-referesh-add-watches (func) "Add watches for the files that contribute to the local cache"
       (let ((buffer (buffer-name)))
         (seq-map (lambda (bibfile)
                    (file-notify-add-watch bibfile '(change)
                                           (lambda (x) (unless (eq 'stopped (cadr x))
                                                    (with-current-buffer buffer (funcall func))))))
                  (bibtex-actions--local-files-to-cache))))

(defun bib-refresh-filenotify (func)
  "Hook to add watches on buffer local bib files and remove them when the buffer is killed"
  (let ((descs (bib-referesh-add-watches func)))
    (add-hook 'kill-buffer-hook
              (lambda () (seq-map #'file-notify-rm-watch descs)) nil t)))

Adding

(lambda () (bib-refresh-filenotify #'bibtex-actions-refresh))

to the appropriate mode hook will auto-refresh the cache when a local bib file changes.

While something

(lambda () (bib-refresh-filenotify (lambda () (setq bibtex-actions--candidates-cache-local 'uninitialized))))

will invalidate the cache so that it gets refreshed the next time a bibtex-actions command is called.

Right now one problem with this is that bibtex-actions--local-files-to-cache is not autloaded so the user has to do the autolading.

[bdarcus]

2. ~Best options for auto-freshing of cache so users don't have to think about this? Maybe we include an [example auto-timer hook](https://github.com/bdarcus/bibtex-actions/pull/208#issuecomment-892589239) in the README?~ settled; working on this

I got it work but it is somewhat non-trivial
...
Right now one problem with this is that bibtex-actions--local-files-to-cache is not autloaded so the user has to do the autolading.

1. Should we include this, or should it go in the README?
2. Either way, maybe we can save this for a separate PR, and you can submit that? E.g. I merged this without this piece?
3. Should it be autoloaded?

[aikrahguzar]

1. Should we include this, or should it go in the README?

filenotify stuff is in core eamcs so it seems ok to include this and I think it is complicated enough that I would rather not have it in my personal configuration if possible.

2. Either way, maybe we can save this for a separate commit, and you can submit that as a PR?

I need a git tutorial so I can probably do it this weekend unless I have to do some work stuff.

3. Should it be autoloaded?

bibtex-actions--local-files-to-cache probably not but if we include something like bib-refresh-filenotify then that should be. Or we can document how to autoload it in the readme. I am agnostic.

[aikrahguzar]

This comment is outdated now. I forgot to remove it earlier.

[bdarcus]

Oops; I merged it.

You can remove it when you do the related PR :-)

[aikrahguzar]

That was quick. I hope these late changes don't introduce any errors. They are pretty trivial but still fingers crossed.

[bdarcus]

I need a git tutorial so I can probably do it this weekend unless I have to do some work stuff.

FWIW, for the PRs, you don't need to worry about advanced stuff like rebase, as I usually just "squash rebase" anyway (e.g. bundle whatever commits into one single commit). Just make your changes and commit, then push, them, to your fork.

[bdarcus]

And asking on doom discord seems like a good thing. While trying to familiarize myself with how bibtex-completion works under the hook, I saw that reftex which is used under the hood has a function for generating a new bib file consisting of all the entries cited in the document from a global bib file. Some people might have a workflow built around that.

FYI, parsebib just merged major enhancements, that could maybe allow bypassing bibtex-completion for parsing. It handles deduping, error-handing, etc.

Here's it parsing two files with same data; one biblatex, and the other CSL JSON.

ELISP> (defvar my/bibtest3 (parsebib-parse '("~/org/bib/newer.json" "~/org/bib/newer.bib")))
my/bibtest3
ELISP> my/bibtest3
#<hash-table equal 234/325 0x1e587cd>
ELISP> (gethash "low2001" my/bibtest3)
(("subtitle" . "gated communities and the discourse of urban fear")
 ("title" . "The Edge and the Center")
 ("number" . "1")
 ("langid" . "english")
 ("urldate" . "2016-02-18")
 ("doi" . "10.1525/aa.2001.103.1.45")
 ("issn" . "1548-1433")
 ("pages" . "45--58")
 ("volume" . "103")
 ("journaltitle" . "American Anthropologist")
 ("date" . "2001-03-01")
 ("author" . "Low, Setha M.")
 ("shorttitle" . "The Edge and the Center")
 ("=type=" . "article")
 ("=key=" . "low2001"))