Skip to content

Commit

Permalink
scripts: Wait for a single Salt Master container
Browse files Browse the repository at this point in the history
Sometimes, if kubelet restarted the `salt-master` static Pod after an
operation, two containers matching the usual selector will co-exist for
a small time window.
If we use the `scripts/common.sh:get_salt_container` function at that
point in time, we may return a string with two container IDs instead of
just one, and subsequent commands will fail.
Instead, we now wait for a single container to exist (and also add a
sleep time between two attemps, which we didn't before).

Fixes: #2434
  • Loading branch information
gdemonet committed Apr 20, 2020
1 parent 6d50ead commit 17183dd
Showing 1 changed file with 11 additions and 3 deletions.
14 changes: 11 additions & 3 deletions scripts/common.sh
Original file line number Diff line number Diff line change
Expand Up @@ -313,13 +313,21 @@ pre_minion_checks() {
get_salt_container() {
local -r max_retries=10
local salt_container='' attempts=0
local -a found_containers=()

while [ -z "$salt_container" ] && [ $attempts -lt $max_retries ]; do
salt_container="$(crictl ps -q \
while [[ $attempts -lt $max_retries ]]; do
IFS=$'\n' read -r -d '' -a found_containers < <(crictl ps -q \
--label io.kubernetes.pod.namespace=kube-system \
--label io.kubernetes.container.name=salt-master \
--state Running)"
--state Running && printf '\0')

if [[ "${#found_containers[@]}" -eq 1 ]]; then
salt_container=${found_containers[0]}
break
fi
echo "Invalid number of candidates: ${#found_containers[@]}" >&2
(( attempts++ ))
sleep 3
done

if [ -z "$salt_container" ]; then
Expand Down

0 comments on commit 17183dd

Please sign in to comment.