x-pack/filebeat/input/{cel,httpjson}: fix flaky test (#41358) (#41406)

Do not use uncontrolled randomisation in tests where it is not necessary. The
retry tests in the httpjson and cel packages were using a randomised 5xx HTTP
status code to trigger the retry behaviour of the the go-retryablehttp
package. This has the unfortunate consequence of causing 2% of test runs to
fail.

The reason for this is given in the go-retryablehttp documentation[1]:

	Mainly, if an error is returned by the client (connection errors,
	etc.), or if a 500-range response code is received (except 501),
	then a retry is invoked after a wait period.

Since the package is already tested, and is documented to accept all 5xx
status codes except 501 to cause a retry, just use 500.

[1]https://pkg.go.dev/github.com/hashicorp/[email protected]#section-readme

(cherry picked from commit 794b832)

Co-authored-by: Dan Kortschak <[email protected]>

CHANGELOG-developer.next.asciidoc

@@ -106,6 +106,7 @@ The list below covers the major changes between 7.0.0-rc2 and main only.
 - Close connections properly in Filbeat's HTTPJSON input. {pull}39790[39790]
 - Add the Offset property to libbeat/reader.Message to store the total number of bytes read and discarded before generating the message. This enables inputs to accurately determine how much data has been read up to the message, using Message.Bytes + Message.Offset. {pull}39873[39873] {issue}39653[39653]
 - AWS CloudWatch Metrics record previous endTime to use for next collection period and change log.logger from cloudwatch to aws.cloudwatch. {pull}40870[40870]
+- Fix flaky test in cel and httpjson inputs of filebeat. {issue}40503[40503] {pull}41358[41358]
 
 ==== Added
 

x-pack/filebeat/input/cel/input_test.go

@@ -10,7 +10,6 @@ import (
 	"flag"
 	"fmt"
 	"io"
-	"math/rand"
 	"net/http"
 	"net/http/httptest"
 	"net/url"
@@ -1947,7 +1946,8 @@ func retryHandler() http.HandlerFunc {
 			w.Write([]byte(`{"hello":"world"}`))
 			return
 		}
-		w.WriteHeader(rand.Intn(100) + 500)
+		// Any 5xx except 501 will result in a retry.
+		w.WriteHeader(500)
 		count++
 	}
 }

x-pack/filebeat/input/httpjson/input_test.go

@@ -8,7 +8,6 @@ import (
 	"context"
 	"fmt"
 	"io"
-	"math/rand"
 	"net/http"
 	"net/http/httptest"
 	"os"
@@ -1724,7 +1723,8 @@ func retryHandler() http.HandlerFunc {
 			_, _ = w.Write([]byte(`{"hello":"world"}`))
 			return
 		}
-		w.WriteHeader(rand.Intn(100) + 500)
+		// Any 5xx except 501 will result in a retry.
+		w.WriteHeader(500)
 		count += 1
 	}
 }