fix: Don't crash indexer on an "UNKNOWN: unknown error, lstat" error

[symwell]

Attempts to fix #897. Created from branch sw/disable_watching_on_ENOSPC_error

This error is an exception thrown by chokidar while file watching.

I wasn't able to reproduce this error. I created symlinks and deleted them programmatically when the queue processes the file but this produced an ENOENT error.

File /home/test/tmp/too_many_open_files/minitest/Static_pages_controller_should_get_help.appmap.json does not exist or is not a file.
Deleted File successfully.
[Error: ENOENT: no such file or directory, lstat '/home/test/tmp/too_many_open_files/minitest/Static_pages_controller_should_get_home.appmap.json'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'lstat',
  path: '/home/test/tmp/too_many_open_files/minitest/Static_pages_controller_should_get_home.appmap.json'
}

There's a claim this UNKNOWN: unknown error, lstat error is the result of file corruption down at the NTFS level.
nodejs/node#43439

[ahtrotta]

I don't think we should catch this error if we don't understand why it's happening. It's possible it could be a UTF-8 problem. Or it could be the corrupt file like you linked to above. Or it might have to do with a faulty npm cache. Bottom line is, I think we shouldn't mess with it until we understand it. It does seem like it's a Windows problem, so maybe someone with a Windows machine should tackle this.

[dustinbyrne]

I'd be fine with not propagating unknown errors into a panic as long as we emit the exception as a telemetry event:

appmap-js/packages/cli/src/fingerprint/fingerprintWatchCommand.ts

Lines 58 to 59 in dd80f92

	// let it crash if it's some other error, to learn what the error is
	throw error;

Is there a chance this method would be invoked repeatedly on an unreadable file? If so, that's a bigger issue.

[symwell]

Added index:watcher_error:unknown telemetry event. Also added in telemetry spreadsheet.

Is there a chance this method would be invoked repeatedly on an unreadable file? If so, that's a bigger issue.

If you mean getting stuck on an unreadable file and not processing other files, I think yes. Running ls on mounted images that contain unreadable files on Linux gets stuck, pressing ctrl-c doesn't stop.

This error was encountered 2 times in telemetry. Maybe it's not worth adding logic that will bypass such unreadable files right now.

[symwell]

I don't think we should catch this error if we don't understand why it's happening.

Maybe we can do both. Catch this error, to reach the goal of not letting the indexer crash. And send telemetry with the error message, to help understand why it's happening.

[ahtrotta]

If you mean getting stuck on an unreadable file and not processing other files, I think yes. Running ls on mounted images that contain unreadable files on Linux gets stuck, pressing ctrl-c doesn't stop.

I think this is a problem. I don't think we should release it in its current form if this is the behavior. Maybe you could console.warn and send a telemetry event like you do now, and then add throw error after so that it doesn't get stuck on an unreadable file. That way we could get more information on the error and not have the program get stuck.

Alternatively, maybe we could catch the error, then add the file to a list of files to skip while watching/polling.

[symwell]

Added the ability to save unreadable files in a Set and ignore them.

Added logic in testcases. Also tested by copying and executing this logic for ENOSPC which is easily reproducible. It showed files were added to unreadableFiles correctly.

If we throw error then the indexer will exit.

[appland-release]

🎉 This PR is included in version @appland/appmap-v3.55.0 🎉

The release is available on:

• @appland/appmap-v3.55.0
• GitHub release
• @appland/appmap-v3.55.0

Your semantic-release bot 📦🚀