Skip to content

Commit

Permalink
Flush buffer in streaming interface before writing zip data
Browse files Browse the repository at this point in the history
We ran into a really obscure issue when working on ansible/receptor#683.

I'll try to make this at least somewhat digestable.

Due to a bug in Kubernetes, AWX can't currently run jobs longer than 4 hours when deployed into Kubernetes. More context on that in ansible/awx#11805

To address this issue, we needed a way to restart from a certain point in the logs. The only mechanism Kubernetes provides to do this is by passing "sinceTime" to the API endpoint for retrieving logs from a pod.

Our patch in ansible/receptor#683 worked when we ran it locally, but in OpenShift, jobs errored when unpacking the zip stream at the end of the results of "ansible-runner worker". Upon further investigation this was because the timestamps of the last 2 lines were exactly the same:

```
2022-11-09T00:07:46.851687621Z {"status": "successful", "runner_ident": "1"}
2022-11-08T23:07:58.648753832Z {"zipfile": 1330}
2022-11-08T23:07:58.648753832Z UEsDBBQAAAAIAPy4aFVGnUFkqQMAAIwK....
```

After squinting at this code for a bit I noticed that we weren't flushing the buffer here like we do in the event_handler and other callbacks that are fired in streaming.py. The end. Ugh.
  • Loading branch information
shanemcd committed Nov 9, 2022
1 parent 6f96ff5 commit 64c9465
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions ansible_runner/utils/streaming.py
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ def stream_dir(source_directory, stream):
else:
target = stream
target.write(json.dumps({"zipfile": zip_size}).encode("utf-8") + b"\n")
target.flush()
with Base64IO(target) as encoded_target:
for line in source:
encoded_target.write(line)
Expand Down

0 comments on commit 64c9465

Please sign in to comment.