Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-enable resource leak tests, change timeout, improve message #4454

Merged
merged 7 commits into from
Mar 22, 2024
Merged

Re-enable resource leak tests, change timeout, improve message #4454

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
b8e6d43
enable test, change timeout, error message
fearful-symmetry Mar 21, 2024
62d1832
change id name
fearful-symmetry Mar 21, 2024
0cc66f0
Merge branch 'main' into flaky-test-handle-leaks
pierrehilbert Mar 21, 2024
c71aaff
fix syntax
fearful-symmetry Mar 21, 2024
cd3e6e8
Merge remote-tracking branch 'origin/flaky-test-handle-leaks' into fl…
fearful-symmetry Mar 21, 2024
6f157c9
try to make the unit IDs more generic
fearful-symmetry Mar 21, 2024
e5ab543
Merge remote-tracking branch 'upstream/main' into flaky-test-handle-l…
fearful-symmetry Mar 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions .buildkite/pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -223,15 +223,15 @@ steps:
provider: "gcp"
machineType: "n1-standard-8"

# - label: "Extended runtime leak tests"
# key: "extended-integration-tests"
# command: ".buildkite/scripts/steps/integration_tests.sh stateful integration:TestForResourceLeaks"
# artifact_paths:
# - "build/TEST-**"
# - "build/diagnostics/*"
# agents:
# provider: "gcp"
# machineType: "n1-standard-8"
- label: "Extended runtime leak tests"
key: "extended-integration-tests"
command: ".buildkite/scripts/steps/integration_tests.sh stateful integration:TestForResourceLeaks"
artifact_paths:
- "build/TEST-**"
- "build/diagnostics/*"
agents:
provider: "gcp"
machineType: "n1-standard-8"

- label: "Integration tests"
key: "integration-tests"
Expand Down
8 changes: 5 additions & 3 deletions testing/integration/agent_long_running_leak_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ func TestLongRunningAgentForLeaks(t *testing.T) {
}

suite.Run(t, &ExtendedRunner{info: info,
healthCheckTime: time.Minute * 3,
healthCheckTime: time.Minute * 6,
healthCheckRefreshTime: time.Second * 20,
resourceWatchers: []StatusWatcher{ // select which tests to run
&handleMonitor{},
Expand Down Expand Up @@ -229,13 +229,14 @@ func (runner *ExtendedRunner) TestHandleLeak() {
// CheckHealthAtStartup ensures all the beats and agent are healthy and working before we continue
func (runner *ExtendedRunner) CheckHealthAtStartup(ctx context.Context) {
// because we need to separately fetch the PIDs, wait until everything is healthy before we look for running beats
compDebugName := ""
require.Eventually(runner.T(), func() bool {
allHealthy := true
status, err := runner.agentFixture.ExecStatus(ctx)

apacheMatch := "logfile-apache"
foundApache := false
systemMatch := "metrics-default"
systemMatch := "system/metrics"
foundSystem := false

require.NoError(runner.T(), err)
Expand All @@ -253,11 +254,12 @@ func (runner *ExtendedRunner) CheckHealthAtStartup(ctx context.Context) {
}
runner.T().Logf("component state: %s", comp.Message)
if comp.State != int(cproto.State_HEALTHY) {
compDebugName = comp.Name
allHealthy = false
}
}
return allHealthy && foundApache && foundSystem
}, runner.healthCheckTime, runner.healthCheckRefreshTime, "install never became healthy")
}, runner.healthCheckTime, runner.healthCheckRefreshTime, "install never became healthy: components did not return a healthy state: %s", compDebugName)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: printing out the entire last agent status (in a structured way) may be more useful for debugging

}

/*
Expand Down
Loading