Files exported with Better BibLaTeX format are not watchable for changes without polling

[alexanderchr]

Support log ID:

L3N8A9ZR-euc

Exporter used:

BibLaTeX / Better BibLaTeX

Expected behavior:

I use the LaTeX Workshop extension for VS Code. In my regular workflow, I export my Zotero library using the BibLaTeX format. When I do this, LaTeX Workshop detects that the .bib file was changed (using the file watching library Chokidar, see here: https://github.com/James-Yu/LaTeX-Workshop/blob/master/src/components/managerlib/bibwatcher.ts) and parses the updated file so that the autocomplete suggestions for citations are up to date. This is a very nice workflow, and I would like the Better BibLatex format to work in the same way.

Actual behavior:

When I export using the "Better BibLaTeX" format, the changes are only picked up the first time I export my library. After that, LaTeX workshop (in effect Chokidar) does not detect any changes. I found a workaround, setting latex-workshop.latex.watch.usePolling to true which sets usePolling: true on the Chokidar watcher. However this is slower and drains CPU, so it's not a great solution. And since watching works without polling for the BibLaTeX format, it does not seem unreasonable to expect the Better BibLaTeX format to also work without polling.

[retorquere]

You'd have to ask chokidar why it isn't detecting changes for BBT files. If I know that, maybe I can change something to accommodate it.

[alexanderchr]

I dug a bit further. Here is a minimal example demonstrating the bug in node:

const chokidar = require('chokidar');

const watcherOptions = {
  useFsEvents: false,
  usePolling: false,
  interval: 300,
  binaryInterval: 1000,
  awaitWriteFinish: { stabilityThreshold: 250 }
}

const path = '/Users/alexander/notes/bibliography.bib';
const watcher = chokidar.watch(path, watcherOptions);

watcher.on('raw', console.log);
watcher.on('change', () => console.log('change event emitted'));

If I export twice with the BibLaTeX formatter this gives me

% node index.js
change bibliography.bib { watchedPath: '/Users/alexander/notes/bibliography.bib' }
change bibliography.bib { watchedPath: '/Users/alexander/notes/bibliography.bib' }
...
change bibliography.bib { watchedPath: '/Users/alexander/notes/bibliography.bib' }
change bibliography.bib { watchedPath: '/Users/alexander/notes/bibliography.bib' }
change event emitted
change bibliography.bib { watchedPath: '/Users/alexander/notes/bibliography.bib' }
change event emitted
change bibliography.bib { watchedPath: '/Users/alexander/notes/bibliography.bib' }
change bibliography.bib { watchedPath: '/Users/alexander/notes/bibliography.bib' }
...
change bibliography.bib { watchedPath: '/Users/alexander/notes/bibliography.bib' }
change bibliography.bib { watchedPath: '/Users/alexander/notes/bibliography.bib' }
change event emitted
change bibliography.bib { watchedPath: '/Users/alexander/notes/bibliography.bib' }
change event emitted

However doing two exports with the Better BibLaTeX formatter yields only one change event:

% node index.js
rename bibliography.bib { watchedPath: '/Users/alexander/notes/bibliography.bib' }
change event emitted

I think what is happening here is that BibLaTeX writes directly to the .bib whereas Better BibLaTeX writes to a temporary file and then replaces the .bib with the temp file. This causes Chokidar to lose track of the file when useFsEvents: false. This would also explain why polling works.

If my explanation is correct this sounds more like a bug in Latex workshop. Because you most likely write to a temp file for a reason, and file changes due to swapped files is something Latex workshop should be able to handle.

[retorquere]

Right, that analysis correct. Overlapping auto-exports could in principle lead to corrupted bib files, so that's one reason why it's desirable, but it's mostly that the background exporter uses an api that only documents atomic writes by create-and-move.

If you disable background exports, you'll get the behavior you want, because that uses the regular zotero export facility. If your exports are large, zotero will block during exports, that's just how regular zotero export works. The background export reimplements part of that facility to put it in a different thread.

WRT the file watcher - it has requested a notification on the inode of the bib file from the OS, and during the atomic write, the original inode just disappears. It should rather watch the inode of the directory and watch for file creation.

[alexanderchr]

Ok, thanks! Then this is clearly not an issue here so I'm closing this and creating an issue in latex workshop.

[github-actions]

Thanks for the feedback; there's no way you could have known, but @retorquere prefers to keep bugreports/enhancements open as a reminder to merge the change into a new release.

[retorquere]

Cool. Can you let me know what the outcome is?

[alexanderchr]

Sure!

[alexanderchr]

See James-Yu/LaTeX-Workshop#2833. They consider polling to be a fix and are not interested in fixing it.

[retorquere]

Shame. If your exports aren't too big you can disable background exports. If not, I'll see if I can do in-place writes, but that does come with the risk of overlapping writes.

[alexanderchr]

I did not realize I could disable background exports. This works great for me since my collections are quite small. Thanks!

By the way it did not seem to be possible to drag to slider for number of workers to 0 to disable background exports, as mentioned in the docs. I had to set that value to 0 through Zotero's config editor.

[retorquere]

A new build will drop here in about 10 minutes that may fix the problem without having to disable background export; I'd appreciate it if you could test whether that works (with BG export on, natch).

[retorquere]

By the way it did not seem to be possible to drag to slider for number of workers to 0 to disable background exports, as mentioned in the docs. I had to set that value to 0 through Zotero's config editor.

That is correct -- I don't encourage disabling background export usually, since the most frequent complaint about BBT was that it would cause Zotero to freeze up during exports.

[github-actions]

🤖 this is your friendly neighborhood build bot announcing test build 5.4.28.1297 ("atomic without tmp")

Install in Zotero by downloading test build 5.4.28.1297, opening the Zotero "Tools" menu, selecting "Add-ons", open the gear menu in the top right, and select "Install Add-on From File...".

[alexanderchr]

Works beautifully with background exports reenabled. Thanks!

% node index.js
change bibliography.bib { watchedPath: '/Users/alexander/notes/bibliography.bib' }
change event emitted
change bibliography.bib { watchedPath: '/Users/alexander/notes/bibliography.bib' }
change event emitted

[retorquere]

Super -- you can just keep using this, I'll probably roll out a new release end of this week, and then you'll be upgraded automatically.

[alexanderchr]

Amazing - thanks again for fixing this and so quickly.