Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Compactor AuthenticationErrorDetail: Request date header too old for Azure after upgrade to 0.29.0 #5961

Closed
bck01215 opened this issue Dec 13, 2022 · 4 comments
Closed

Compactor AuthenticationErrorDetail: Request date header too old for Azure after upgrade to 0.29.0 #5961

bck01215 opened this issue Dec 13, 2022 · 4 comments

Comments

@bck01215
Copy link

bck01215 commented Dec 13, 2022 •
edited
Loading

Thanos:v0.29.0:

Azure:

Compactor ran from 12 hours or so then stopped until restarted:

Compactor to keep running continuously:

How to reproduce it (as minimally and precisely as possible):
Begin the compactor writing to an Azure blob with version 0.29.0. Write for around 24 hours and this error should occur

Full logs to relevant components:

Logs 

level=info ts=2022-12-13T11:33:57.431064514Z caller=fetcher.go:475 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=13.908477057s duration_ms=13908 cached=29006 returned=29006 partial=0
level=error ts=2022-12-13T11:39:43.568649104Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="incomplete view: 2 errors: meta.json file exists: 01EMRH2104QRTFZJZFXYJPYEGF/meta.json: cannot get properties for Azure blob, address: 01EMRH2104QRTFZJZFXYJPYEGF/meta.json: ===== INTERNAL ERROR =====\ncontext deadline exceeded; meta.json file exists: 01F0HZ51VP45CYTACYQS7WZHK9/meta.json: cannot get properties for Azure blob, address: 01F0HZ51VP45CYTACYQS7WZHK9/meta.json: ===== INTERNAL ERROR =====\ncontext deadline exceeded"
level=info ts=2022-12-13T11:39:57.373112339Z caller=fetcher.go:475 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=13.804206007s duration_ms=13804 cached=29006 returned=29006 partial=0
level=info ts=2022-12-13T11:40:12.100686068Z caller=fetcher.go:475 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=14.697822284s duration_ms=14697 cached=29006 returned=29006 partial=0
level=info ts=2022-12-13T11:40:59.403629637Z caller=fetcher.go:475 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=15.882382907s duration_ms=15882 cached=29006 returned=29006 partial=0
level=info ts=2022-12-13T11:42:01.518760019Z caller=fetcher.go:475 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=17.993535778s duration_ms=17993 cached=29006 returned=29006 partial=0
level=info ts=2022-12-13T11:42:57.304027212Z caller=fetcher.go:475 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=13.778750204s duration_ms=13778 cached=29006 returned=29006 partial=0
level=info ts=2022-12-13T11:43:59.361113767Z caller=fetcher.go:475 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=15.840606298s duration_ms=15840 cached=29006 returned=29006 partial=0
level=info ts=2022-12-13T11:44:57.713912833Z caller=fetcher.go:475 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=14.188447966s duration_ms=14188 cached=29006 returned=29006 partial=0
level=info ts=2022-12-13T11:46:02.73691884Z caller=fetcher.go:475 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=19.213156457s duration_ms=19213 cached=29006 returned=29006 partial=0
level=info ts=2022-12-13T11:46:59.604444811Z caller=fetcher.go:475 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=16.07916049s duration_ms=16079 cached=29006 returned=29006 partial=0
level=info ts=2022-12-13T11:47:58.087985659Z caller=fetcher.go:475 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=14.562731136s duration_ms=14562 cached=29006 returned=29006 partial=0
level=info ts=2022-12-13T11:48:58.843307293Z caller=fetcher.go:475 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=15.31812656s duration_ms=15318 cached=29006 returned=29006 partial=0
level=warn ts=2022-12-13T11:49:47.02416205Z caller=intrumentation.go:67 msg="changing probe status" status=not-ready reason="syncing metas: filter metas: filter blocks marked for no compaction: get file: 01F16S6H3X8TKQ7R5X7WRHHQR1/no-compact-mark.json: cannot download blob, address: https://secret.blob.core.windows.net/thanos-snap/01F16S6H3X8TKQ7R5X7WRHHQR1/no-compact-mark.json: ===== RESPONSE ERROR (ErrorCode=AuthenticationFailed) =====\nDescription=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:e15da35d-a01e-00fe-09e8-0ed881000000\nTime:2022-12-13T11:49:47.0175287Z, Details: \n   AuthenticationErrorDetail: Request date header too old: 'Tue, 13 Dec 2022 11:34:47 GMT'\n"
level=info ts=2022-12-13T11:49:47.024405198Z caller=http.go:84 service=http/server component=compact msg="internal server is shutting down" err="syncing metas: filter metas: filter blocks marked for no compaction: get file: 01F16S6H3X8TKQ7R5X7WRHHQR1/no-compact-mark.json: cannot download blob, address: https://secret.blob.core.windows.net/thanos-snap/01F16S6H3X8TKQ7R5X7WRHHQR1/no-compact-mark.json: ===== RESPONSE ERROR (ErrorCode=AuthenticationFailed) =====\nDescription=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:e15da35d-a01e-00fe-09e8-0ed881000000\nTime:2022-12-13T11:49:47.0175287Z, Details: \n   AuthenticationErrorDetail: Request date header too old: 'Tue, 13 Dec 2022 11:34:47 GMT'\n"
level=error ts=2022-12-13T11:49:47.024772066Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="BaseFetcher: iter bucket: context canceled"
level=info ts=2022-12-13T11:49:47.024839651Z caller=http.go:103 service=http/server component=compact msg="internal server is shutdown gracefully" err="syncing metas: filter metas: filter blocks marked for no compaction: get file: 01F16S6H3X8TKQ7R5X7WRHHQR1/no-compact-mark.json: cannot download blob, address: https://secret.blob.core.windows.net/thanos-snap/01F16S6H3X8TKQ7R5X7WRHHQR1/no-compact-mark.json: ===== RESPONSE ERROR (ErrorCode=AuthenticationFailed) =====\nDescription=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:e15da35d-a01e-00fe-09e8-0ed881000000\nTime:2022-12-13T11:49:47.0175287Z, Details: \n   AuthenticationErrorDetail: Request date header too old: 'Tue, 13 Dec 2022 11:34:47 GMT'\n"
level=info ts=2022-12-13T11:49:47.024891203Z caller=intrumentation.go:81 msg="changing probe status" status=not-healthy reason="syncing metas: filter metas: filter blocks marked for no compaction: get file: 01F16S6H3X8TKQ7R5X7WRHHQR1/no-compact-mark.json: cannot download blob, address: https://secret.blob.core.windows.net/thanos-snap/01F16S6H3X8TKQ7R5X7WRHHQR1/no-compact-mark.json: ===== RESPONSE ERROR (ErrorCode=AuthenticationFailed) =====\nDescription=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:e15da35d-a01e-00fe-09e8-0ed881000000\nTime:2022-12-13T11:49:47.0175287Z, Details: \n   AuthenticationErrorDetail: Request date header too old: 'Tue, 13 Dec 2022 11:34:47 GMT'\n"

Anything else we need to know:
This error occurred directly after an upgrade to 0.29.0

Environment:

  • OS (e.g. from /etc/os-release): Docker image on RHEL 7.9
  • Kernel (e.g. uname -a): Linux 3.10.0-1160.59.1.el7.x86_64 SMP Wed Feb 16 12:17:35 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

@vglafirov
@bck01215
Copy link
Author

https://github.com/thanos-io/thanos/pull/5707/files#diff-33ef32bf6c23acb95f5902d7097b7a1d5128ca061167ec0716715b0b9eeaa5f6L114-R118 possibly related

@bck01215
Copy link
Author

bck01215 commented Dec 14, 2022 •
edited
Loading

New error occurred in logs today causing a shutdown:

level=error ts=2022-12-14T13:25:49.640524543Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="incomplete view: 2 errors: meta.json file exists: 01EK33A3VQ15QKRNQ2HDTZXSKA/meta.json: cannot get properties for Azure blob, address: 01EK33A3VQ15QKRNQ2HDTZXSKA/meta.json: ===== INTERNAL ERROR =====\ncontext deadline exceeded; meta.json file exists: 01EP1ZEKJ1YEW12RQ2BZYWSEEV/meta.json: cannot get properties for Azure blob, address: 01EP1ZEKJ1YEW12RQ2BZYWSEEV/meta.json: ===== INTERNAL ERROR =====\ncontext deadline exceeded"
level=info ts=2022-12-14T13:26:06.236241296Z caller=fetcher.go:475 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=16.595427245s duration_ms=16595 cached=29045 returned=29045 partial=0
level=warn ts=2022-12-14T13:26:58.262391529Z caller=intrumentation.go:67 msg="changing probe status" status=not-ready reason="syncing metas: incomplete view: 2 errors: meta.json file exists: 01EK33A3VQ15QKRNQ2HDTZXSKA/meta.json: cannot get properties for Azure blob, address: 01EK33A3VQ15QKRNQ2HDTZXSKA/meta.json: ===== INTERNAL ERROR =====\ncontext deadline exceeded; meta.json file exists: 01EP1ZEKJ1YEW12RQ2BZYWSEEV/meta.json: cannot get properties for Azure blob, address: 01EP1ZEKJ1YEW12RQ2BZYWSEEV/meta.json: ===== INTERNAL ERROR =====\ncontext deadline exceeded"
level=info ts=2022-12-14T13:26:58.26255649Z caller=http.go:84 service=http/server component=compact msg="internal server is shutting down" err="syncing metas: incomplete view: 2 errors: meta.json file exists: 01EK33A3VQ15QKRNQ2HDTZXSKA/meta.json: cannot get properties for Azure blob, address: 01EK33A3VQ15QKRNQ2HDTZXSKA/meta.json: ===== INTERNAL ERROR =====\ncontext deadline exceeded; meta.json file exists: 01EP1ZEKJ1YEW12RQ2BZYWSEEV/meta.json: cannot get properties for Azure blob, address: 01EP1ZEKJ1YEW12RQ2BZYWSEEV/meta.json: ===== INTERNAL ERROR =====\ncontext deadline exceeded"
level=info ts=2022-12-14T13:26:58.262933587Z caller=http.go:103 service=http/server component=compact msg="internal server is shutdown gracefully" err="syncing metas: incomplete view: 2 errors: meta.json file exists: 01EK33A3VQ15QKRNQ2HDTZXSKA/meta.json: cannot get properties for Azure blob, address: 01EK33A3VQ15QKRNQ2HDTZXSKA/meta.json: ===== INTERNAL ERROR =====\ncontext deadline exceeded; meta.json file exists: 01EP1ZEKJ1YEW12RQ2BZYWSEEV/meta.json: cannot get properties for Azure blob, address: 01EP1ZEKJ1YEW12RQ2BZYWSEEV/meta.json: ===== INTERNAL ERROR =====\ncontext deadline exceeded"
level=info ts=2022-12-14T13:26:58.262971177Z caller=intrumentation.go:81 msg="changing probe status" status=not-healthy reason="syncing metas: incomplete view: 2 errors: meta.json file exists: 01EK33A3VQ15QKRNQ2HDTZXSKA/meta.json: cannot get properties for Azure blob, address: 01EK33A3VQ15QKRNQ2HDTZXSKA/meta.json: ===== INTERNAL ERROR =====\ncontext deadline exceeded; meta.json file exists: 01EP1ZEKJ1YEW12RQ2BZYWSEEV/meta.json: cannot get properties for Azure blob, address: 01EP1ZEKJ1YEW12RQ2BZYWSEEV/meta.json: ===== INTERNAL ERROR =====\ncontext deadline exceeded"
level=error ts=2022-12-14T13:26:58.265708645Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="BaseFetcher: iter bucket: context canceled"

The error in the main post occurred twice more as well

@phillebaba
Copy link
Contributor

I think it would be best to move this issue to the objstore repository. While doing that could you also specify which authentication method you are using towards the Azure Storage Account.

@bck01215
Copy link
Author

Not found in newest version

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@phillebaba @bck01215