Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat] keep track of bytes read when max_bytes exceeded #31824

Merged
merged 2 commits into from
Jun 7, 2022
Merged

[Filebeat] keep track of bytes read when max_bytes exceeded #31824

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bff444b
[Filebeat] keep track of bytes read when max_bytes exceeded
liuwenping Jun 4, 2022
74d34d3
add test code for exceeding max bytes in filebeat
liuwenping Jun 7, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion libbeat/reader/readfile/line.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,7 @@ func (r *LineReader) skipUntilNewLine() (int, error) {

if idx != -1 {
r.inBuffer.Write(r.tempBuffer[idx+len(r.nl) : n])
skipped += idx
skipped += idx + len(r.nl)
} else {
skipped += n
}
Expand Down
13 changes: 11 additions & 2 deletions libbeat/reader/readfile/line_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -364,9 +364,13 @@ func TestMaxBytesLimit(t *testing.T) {
}

// Read decodec lines and test
var idx int
var (
idx int
readLen int
)

for i := 0; ; i++ {
b, _, err := reader.Next()
b, n, err := reader.Next()
if err != nil {
if err == io.EOF {
break
Expand All @@ -387,11 +391,16 @@ func TestMaxBytesLimit(t *testing.T) {
break
}

readLen += n
s := string(b[:len(b)-len(nl)])
if line != s {
t.Fatalf("lines do not match, expected: %s got: %s", line, s)
}
}

if len(input) != readLen {
t.Fatalf("the bytes read are not equal to the bytes input, expected: %d got: %d", len(input), readLen)
}
rdner marked this conversation as resolved.
Show resolved Hide resolved
}

// test_exceed_buffer from test_harvester.py
Expand Down