Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Record chunked response size before sending last chunk #102586

Conversation

DaveCTurner
Copy link
Contributor

We expect that the HTTP response stats should reflect the responses
received by the client, but today we record that a chunked-encoded
response was sent after sending the last chunk, which might mean that
the client can receive the complete response body and then retrieve
stats that do not include that response. With this commit we record the
stats before sending the last chunk.

Closes #102547

We expect that the HTTP response stats should reflect the responses
received by the client, but today we record that a chunked-encoded
response was sent _after_ sending the last chunk, which might mean that
the client can receive the complete response body and then retrieve
stats that do not include that response. With this commit we record the
stats before sending the last chunk.

Closes elastic#102547
@DaveCTurner DaveCTurner added >non-issue :Distributed Coordination/Network Http and internode communication implementations v8.12.0 labels Nov 24, 2023
@DaveCTurner DaveCTurner requested a review from ywangd November 24, 2023 12:21
@elasticsearchmachine elasticsearchmachine added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label Nov 24, 2023
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@DaveCTurner
Copy link
Contributor Author

DaveCTurner commented Nov 24, 2023

Calling this a >non-issue because really stats are allowed to be a little stale, it just seems easier to make them more consistent rather than make the test more lenient.

Copy link
Member

@ywangd ywangd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Thanks for fixing it. I am amazed how quickly you spotted the issue. Did you manage to reproduce it or did you just discover it by reading the code?

@DaveCTurner DaveCTurner merged commit 8914314 into elastic:main Nov 27, 2023
@DaveCTurner DaveCTurner deleted the 2023/11/24/response-stats-before-response-completion branch November 27, 2023 13:36
@DaveCTurner
Copy link
Contributor Author

Did you manage to reproduce it or did you just discover it by reading the code?

I spotted that the missing response stats were always the GET _cat/nodes call, which was the last one. Pretty sure it wasn't actually getting lost, probably just delayed, so suspected a race. So then I went to check whether there was any guarantee that the stats would be updated before the client requested them, and saw the problem.

Didn't actually reproduce it, but I imagine it would happen every time with a sufficient delay just before updating the stats.

timgrein pushed a commit to timgrein/elasticsearch that referenced this pull request Nov 30, 2023
We expect that the HTTP response stats should reflect the responses
received by the client, but today we record that a chunked-encoded
response was sent _after_ sending the last chunk, which might mean that
the client can receive the complete response body and then retrieve
stats that do not include that response. With this commit we record the
stats before sending the last chunk.

Closes elastic#102547
@DaveCTurner DaveCTurner restored the 2023/11/24/response-stats-before-response-completion branch June 17, 2024 06:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Distributed Coordination/Network Http and internode communication implementations >non-issue Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. v8.12.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[CI] HttpStatsIT testClusterInfoHttpStats failing
3 participants