Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Indexer extract errors are not output to logging #132

Open
ross-spencer opened this issue Nov 15, 2024 · 2 comments
Open

Indexer extract errors are not output to logging #132

ross-spencer opened this issue Nov 15, 2024 · 2 comments

Comments

@ross-spencer
Copy link
Collaborator

Given the command below, with gocfl compiled with extensions:

./gocfl   add /tmp/gocfl-test /home/ross-spencer/git/ocfl/corpus/xml   \
 -u "test"   \
 -a "ross@gocfl"   \
 -m "test-file"   \
 --object-id 'test:dir:xml1'   \
 --log-level DEBUG   \
 --ext-NNNN-metafile-source file:///home/ross-spencer/git/ocfl/gocfl/gocfl/info-json/info.json

We see errors in the indexer packages test_dir_xml1/v1/content/metadata/indexer_v1.json:

{
  "Path": "v1/content/data/sample_data.xml",
  "Indexer": {
    "errors": {
      "identify": "error executing (convertc [XML:- json:-]) for file 'data/sample_data.xml': : exec: \"convertc\": executable file not found in $PATH",
      "siegfried": "cannot identify file data/sample_data.xml: empty source",
      "tika": "status not ok - http://localhost:9998/meta -> 422 Unprocessable Entity: "
    },
    "mimetype": "",
    "mimetypes": [],
    "pronom": "",
    "pronoms": [],
    "size": 0,
    "metadata": {},
    "type": "",
    "subtype": ""
  }
}

But the errors aren't making it into the logs here:

2024-11-15T15:56:32+01:00 INF indexer action siegfried added host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 INF indexer action xml added host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 INF indexer action ffprobe added host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 INF indexer action identify added host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 INF indexer action tika added host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 INF indexer action fulltext added host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
opening '/tmp/gocfl-test'
2024-11-15T15:56:32+01:00 INF opening '/tmp/gocfl-test' host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG initializing ExtensionFactory host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG adding creator for extension initial host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG adding creator for extension NNNN-gocfl-extension-manager host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG adding creator for extension 0001-digest-algorithms host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG adding creator for extension 0002-flat-direct-storage-layout host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG adding creator for extension 0003-hash-and-id-n-tuple-storage-layout host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG adding creator for extension 0004-hashed-n-tuple-storage-layout host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG adding creator for extension 0006-flat-omit-prefix-storage-layout host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG adding creator for extension 0007-n-tuple-omit-prefix-storage-layout host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG adding creator for extension 0011-direct-clean-path-layout host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG adding creator for extension NNNN-direct-clean-path-layout host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG adding creator for extension NNNN-direct-path-layout host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG adding creator for extension NNNN-pairtree-storage-layout host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG adding creator for extension NNNN-content-subpath host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG adding creator for extension NNNN-metafile host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG adding creator for extension NNNN-indexer host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG adding creator for extension NNNN-migration host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG adding creator for extension NNNN-thumbnail host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG adding creator for extension NNNN-filesystem host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG adding creator for extension NNNN-mets host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG [/tmp/release/gocfl/pkg/ocfl/extensionFactory.go:101] [storage root 'osFSRW(/tmp/gocfl-test)'] Validation Warning #W013 - ‘In an OCFL Object, extension sub-directories SHOULD be named according to a registered extension name.’ (https://ocfl.io/1.1/spec/#W013) [extension 'NNNN-direct-clean-path-layout' is not registered] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG [/tmp/release/gocfl/pkg/ocfl/extensionFactory.go:101] [storage root 'osFSRW(/tmp/gocfl-test)'] Validation Warning #W013 - ‘In an OCFL Object, extension sub-directories SHOULD be named according to a registered extension name.’ (https://ocfl.io/1.1/spec/#W013) [extension 'NNNN-direct-path-layout' is not registered] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG [/tmp/release/gocfl/pkg/ocfl/extensionFactory.go:101] [storage root 'osFSRW(/tmp/gocfl-test)'] Validation Warning #W013 - ‘In an OCFL Object, extension sub-directories SHOULD be named according to a registered extension name.’ (https://ocfl.io/1.1/spec/#W013) [extension 'NNNN-gocfl-extension-manager' is not registered] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG test:dir:xml1 host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'test-file' / 'test' / 'ross@gocfl' host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 INF adding file metadata:[info.json] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG [test:dir:xml1] adding '[metadata/info.json]' -> 'v1/content/metadata/info.json' host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/info.json' [11894e007658bd9617bf81fcf7aa4d23d2a8cd83863424f37c9ed110438643dbbb5cec0a908c7dc05962649295b6bdbbaf00d36cbc6f995e38803582f91f86e2] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/info.json' - duplicate false host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/info.json' [11894e007658bd9617bf81fcf7aa4d23d2a8cd83863424f37c9ed110438643dbbb5cec0a908c7dc05962649295b6bdbbaf00d36cbc6f995e38803582f91f86e2] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/info.json' - update false host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG walking 'osFSRW(/home/ross-spencer/git/ocfl/corpus/xml)' host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 INF adding file content:. host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 INF adding file content:sample_data.xml host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 INF cannot decode image 'data/sample_data.xml': image: unknown format extension=NNNN-thumbnail host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG [test:dir:xml1] adding '[data/sample_data.xml]' -> 'v1/content/data/sample_data.xml' host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'data/sample_data.xml' [94f6efdb024e4886ebc24690b9b292a5c24e796dc1d52b4f9460054d2d2f74f035253eb26e738fd8974737eb7c4e9b37a8f2a9bb95c8346185c4edb15c9b23f8] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'data/sample_data.xml' - duplicate false host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'data/sample_data.xml' [94f6efdb024e4886ebc24690b9b292a5c24e796dc1d52b4f9460054d2d2f74f035253eb26e738fd8974737eb7c4e9b37a8f2a9bb95c8346185c4edb15c9b23f8] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'data/sample_data.xml' - update false host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 INF EndUpdate of object 'test:dir:xml1' host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 INF adding file full:README.md host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'README.md' [6f6a6418e72610c0021412627ac89a5d1acb3c90a6fdef7d8c408c96bdbaa2f528b307f99fb72dd83458d796a10601ce012d7749d4b4493c163cd09ec5ce9b89] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'README.md' - duplicate false host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 INF cannot decode image 'README.md': image: unknown format extension=NNNN-thumbnail host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG [test:dir:xml1] adding '[README.md]' -> 'v1/content/README.md' host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'README.md' [6f6a6418e72610c0021412627ac89a5d1acb3c90a6fdef7d8c408c96bdbaa2f528b307f99fb72dd83458d796a10601ce012d7749d4b4493c163cd09ec5ce9b89] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'README.md' - duplicate false host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'README.md' [6f6a6418e72610c0021412627ac89a5d1acb3c90a6fdef7d8c408c96bdbaa2f528b307f99fb72dd83458d796a10601ce012d7749d4b4493c163cd09ec5ce9b89] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'README.md' - update false host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 INF adding file metadata:[filesystem_v1.jsonl.gz] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG [test:dir:xml1] adding '[metadata/filesystem_v1.jsonl.gz]' -> 'v1/content/metadata/filesystem_v1.jsonl.gz' host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/filesystem_v1.jsonl.gz' [43b09b78fd66495dfe72b7d79200fef306ace61e4cc7e035770beca15bdd02e1c2c0018b50acc7241000cdbdd5401eaab7bca36f5c2e4cba087f93ef7edd33bb] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/filesystem_v1.jsonl.gz' - duplicate false host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/filesystem_v1.jsonl.gz' [43b09b78fd66495dfe72b7d79200fef306ace61e4cc7e035770beca15bdd02e1c2c0018b50acc7241000cdbdd5401eaab7bca36f5c2e4cba087f93ef7edd33bb] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/filesystem_v1.jsonl.gz' - update false host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 INF adding file metadata:[indexer_v1.jsonl.gz] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG [test:dir:xml1] adding '[metadata/indexer_v1.jsonl.gz]' -> 'v1/content/metadata/indexer_v1.jsonl.gz' host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/indexer_v1.jsonl.gz' [103a47bdbfb0149d41f88b0283d3b8766769e3588439f5ee87571814382c16a2a55f319928bcfda6f555c5c9afd31cb5aab56a9acfa8b455c6d2847d76f30162] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/indexer_v1.jsonl.gz' - duplicate false host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/indexer_v1.jsonl.gz' [103a47bdbfb0149d41f88b0283d3b8766769e3588439f5ee87571814382c16a2a55f319928bcfda6f555c5c9afd31cb5aab56a9acfa8b455c6d2847d76f30162] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/indexer_v1.jsonl.gz' - update false host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 INF adding file metadata:mets.xml host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG [test:dir:xml1] adding '[metadata/mets.xml]' -> 'v1/content/metadata/mets.xml' host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/mets.xml' [92a8654db0fa4791de6fdcaa608c90302adf0fa499a62cdc6326fb6e4ae9cda2a64877c876ef5226c54f912ad59c0f9a78679f543dd6c4f63575d07870838d49] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/mets.xml' - duplicate false host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/mets.xml' [92a8654db0fa4791de6fdcaa608c90302adf0fa499a62cdc6326fb6e4ae9cda2a64877c876ef5226c54f912ad59c0f9a78679f543dd6c4f63575d07870838d49] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/mets.xml' - update false host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 INF adding file metadata:[premis.xml] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG [test:dir:xml1] adding '[metadata/premis.xml]' -> 'v1/content/metadata/premis.xml' host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/premis.xml' [d3c65dbe1dd5fc93317bd9dbc5fe76602d0f0f7e7bb43651e616a89c8b4a2b83b5721fc40615d81eb11015f67a650adaf6738ac81482f9c0dcbd0506c3f9821c] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/premis.xml' - duplicate false host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/premis.xml' [d3c65dbe1dd5fc93317bd9dbc5fe76602d0f0f7e7bb43651e616a89c8b4a2b83b5721fc40615d81eb11015f67a650adaf6738ac81482f9c0dcbd0506c3f9821c] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/premis.xml' - update false host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 INF adding file metadata:schemas/mets.xsd host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/schemas/mets.xsd' [04e16f04d827e067ad59d981dd2904528adea03ce6e54e8e3d67829f32e3a084f6d681e26abecba9544d48e2a436308519527922b302657ab5c7ca533c214950] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/schemas/mets.xsd' - duplicate false host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG [test:dir:xml1] adding '[metadata/schemas/mets.xsd]' -> 'v1/content/metadata/schemas/mets.xsd' host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/schemas/mets.xsd' [04e16f04d827e067ad59d981dd2904528adea03ce6e54e8e3d67829f32e3a084f6d681e26abecba9544d48e2a436308519527922b302657ab5c7ca533c214950] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/schemas/mets.xsd' - duplicate false host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/schemas/mets.xsd' [04e16f04d827e067ad59d981dd2904528adea03ce6e54e8e3d67829f32e3a084f6d681e26abecba9544d48e2a436308519527922b302657ab5c7ca533c214950] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/schemas/mets.xsd' - update false host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 INF adding file metadata:schemas/premis.xsd host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/schemas/premis.xsd' [f0feb6d7d30a0f27ffd53dadb976ded9fde56adbe339fac0d2cb43352966c768103001c44d1dd0a013c89323f3ca81557f4ddf106629134d82ac9f0587de07c5] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/schemas/premis.xsd' - duplicate false host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG [test:dir:xml1] adding '[metadata/schemas/premis.xsd]' -> 'v1/content/metadata/schemas/premis.xsd' host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/schemas/premis.xsd' [f0feb6d7d30a0f27ffd53dadb976ded9fde56adbe339fac0d2cb43352966c768103001c44d1dd0a013c89323f3ca81557f4ddf106629134d82ac9f0587de07c5] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/schemas/premis.xsd' - duplicate false host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/schemas/premis.xsd' [f0feb6d7d30a0f27ffd53dadb976ded9fde56adbe339fac0d2cb43352966c768103001c44d1dd0a013c89323f3ca81557f4ddf106629134d82ac9f0587de07c5] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/schemas/premis.xsd' - update false host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 INF adding file metadata:schemas/xlink.xsd host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/schemas/xlink.xsd' [d524ed457c67a9e9e4de723519f26567e098c02b8db21fca6807683b30a0b9e477a230cef942e8245411f2a6ccb0f8a7df2bae18dc7363ec2810899d67064a32] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/schemas/xlink.xsd' - duplicate false host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG [test:dir:xml1] adding '[metadata/schemas/xlink.xsd]' -> 'v1/content/metadata/schemas/xlink.xsd' host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/schemas/xlink.xsd' [d524ed457c67a9e9e4de723519f26567e098c02b8db21fca6807683b30a0b9e477a230cef942e8245411f2a6ccb0f8a7df2bae18dc7363ec2810899d67064a32] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/schemas/xlink.xsd' - duplicate false host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/schemas/xlink.xsd' [d524ed457c67a9e9e4de723519f26567e098c02b8db21fca6807683b30a0b9e477a230cef942e8245411f2a6ccb0f8a7df2bae18dc7363ec2810899d67064a32] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/schemas/xlink.xsd' - update false host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 INF adding file metadata:[thumbnail_v1.jsonl.gz] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG [test:dir:xml1] adding '[metadata/thumbnail_v1.jsonl.gz]' -> 'v1/content/metadata/thumbnail_v1.jsonl.gz' host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/thumbnail_v1.jsonl.gz' [1d497009f3aabd230bcd055ef09fd180e63e330c47f4c1afdfc36172e5421a220b78e3e3ce30c9ae533c516accfa8976fd3f1198bd15ff79373d5fe87fc64cd2] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/thumbnail_v1.jsonl.gz' - duplicate false host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/thumbnail_v1.jsonl.gz' [1d497009f3aabd230bcd055ef09fd180e63e330c47f4c1afdfc36172e5421a220b78e3e3ce30c9ae533c516accfa8976fd3f1198bd15ff79373d5fe87fc64cd2] host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 DBG 'metadata/thumbnail_v1.jsonl.gz' - update false host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"
2024-11-15T15:56:32+01:00 INF Closing object 'test:dir:xml1' host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"

[storage root 'osFSRW(/tmp/gocfl-test)']
   #W013 - ‘In an OCFL Object, extension sub-directories SHOULD be named according to a registered extension name.’ [extension 'NNNN-direct-clean-path-layout' is not registered]
   #W013 - ‘In an OCFL Object, extension sub-directories SHOULD be named according to a registered extension name.’ [extension 'NNNN-direct-path-layout' is not registered]
   #W013 - ‘In an OCFL Object, extension sub-directories SHOULD be named according to a registered extension name.’ [extension 'NNNN-gocfl-extension-manager' is not registered]

no errors found
2024-11-15T15:56:32+01:00 INF Duration: 64.540185ms host=exponential-decay timestamp="2024-11-15 15:56:32.827323525 +0100 CET m=+0.027174484"

NB. we also exit okay with no errors found.

Providing I am reading through the code correctly, it is written into the indexer file here:

gocfl/pkg/extension/NNNN-indexer.go

Lines 357 to 369 in 714abed

if result != nil {
var indexerline = indexerLine{
Path: filepath.ToSlash(inventory.BuildManifestName(dest)),
Indexer: result,
}
data, err := json.Marshal(indexerline)
if err != nil {
return errors.Errorf("cannot marshal result %v", indexerline)
}
if _, err := sl.writer.Write(append(data, []byte("\n")...)); err != nil {
return errors.Errorf("cannot brotli %s", string(data))
}
}

E.g. for convert, tt's first recorded in the indexing tool here: https://github.com/je4/indexer/blob/a02d015d6289f5687d43af935eb064ef104615a6/pkg/indexer/actionIdentifyV2.go#L139-L141

And similar lines in the other runners (in this case tika and siegfried),

Based on discussions with the team, I think we need this information to be exposed to the logging to be able to be better analysed down-stream. Will bring this offline to discuss more.
@ross-spencer ross-spencer mentioned this issue Nov 23, 2024
Open
@je4
Copy link
Collaborator

je4 commented Dec 6, 2024 •
edited
Loading

since indexer errors are part of the indexer result structure, they are not yet logged. we have to decide whether we will log them where they appear (indexer) or just log the errors from the result structure

no errors found means, that there are no structural ocfl errors. eventually we should use a different wording

@ross-spencer
Copy link
Collaborator Author

I addressed it to a small extent here, given some of the errors analysed, it does seem beneficial to be able to log these: c184fc9

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ross-spencer @je4