Skip to content

Commit

Permalink
Grow buffers exponentially, not one byte at a time (#121)
Browse files Browse the repository at this point in the history
* Grow buffers exponentially, not one byte at a time

To check if a TranscodingStream is eof, it first checks if it has any unused
bytes in its buffer. If not, it will attempt to fill the buffer, and only return
true if no bytes were filled.
If there is no margin in the buffer to fill, it will call `makemargin(buffer, 1)`
to ensure at least 1 byte of space.
In the happy case, this will remove used up bytes.
Unfortunately, buffers do never move data pointed to by their mark. So, when no
space is available due to the entire buffer being filled up with marked data,
the buffer will grow one byte at a time in a loop until the data fills.
This is unacceptable as megabytes of data can theoretically be marked.

This commit changes the growth behaviour to request at least half the buffer's
length in the margin. In most cases, this will do nothing. But in the unhappy
case mentioned above, it will cause the buffer to grow expoentially, until a
proper size is quickly reached.

* Address suggestions
  • Loading branch information
jakobnissen authored Aug 18, 2022
1 parent 38b969d commit fde7eea
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 2 deletions.
2 changes: 1 addition & 1 deletion src/noop.jl
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ function fillbuffer(stream::NoopStream; eager::Bool = false)
end
nfilled::Int = 0
while ((!eager && buffersize(buffer) == 0) || (eager && makemargin!(buffer, 0, eager = true) > 0)) && !eof(stream.stream)
makemargin!(buffer, 1)
makemargin!(buffer, max(1, div(length(buffer), 2)))
nfilled += readdata!(stream.stream, buffer)
end
buffer.transcoded += nfilled
Expand Down
2 changes: 1 addition & 1 deletion src/stream.jl
Original file line number Diff line number Diff line change
Expand Up @@ -575,7 +575,7 @@ function fillbuffer(stream::TranscodingStream; eager::Bool = false)
end
callstartproc(stream, :read)
end
makemargin!(buffer2, 1)
makemargin!(buffer2, max(1, div(length(buffer2), 2)))
readdata!(stream.stream, buffer2)
_, Δout = callprocess(stream, buffer2, buffer1)
nfilled += Δout
Expand Down

0 comments on commit fde7eea

Please sign in to comment.