Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(docker_logs source): add assert_source_compliance calls to integration tests #14431

Merged
merged 4 commits into from
Sep 15, 2022

Conversation

StephenWakely
Copy link
Contributor

Ref #14411

This adds assert_source_compliance calls to the integration tests for the docker logs source.

There is also a fix to the docker-compose file. It appears the integration tests weren't running at all without this.

@netlify
Copy link

netlify bot commented Sep 15, 2022

Deploy Preview for vector-project canceled.

Name Link
🔨 Latest commit b89c8c8
🔍 Latest deploy log https://app.netlify.com/sites/vector-project/deploys/632352acaa6ec60008138ffa

@github-actions github-actions bot added domain: ci Anything related to Vector's CI environment domain: sources Anything related to the Vector's sources labels Sep 15, 2022
@github-actions
Copy link

Soak Test Results

Baseline: bcbcb2e
Comparison: 37dc768
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
http_pipelines_no_grok_blackhole 234.52KiB 2.16 100.00% 10.59MiB 550.35KiB 11.24KiB 0 0.0507236 10.82MiB 1.05MiB 21.89KiB 0 0.0970981 False False
datadog_agent_remap_blackhole_acks 994.05KiB 1.63 100.00% 59.67MiB 4.35MiB 90.67KiB 0 0.0729486 60.64MiB 2.82MiB 58.93KiB 0 0.0464485 False False
syslog_splunk_hec_logs 162.73KiB 1.03 100.00% 15.48MiB 687.23KiB 13.99KiB 0 0.0433478 15.64MiB 380.87KiB 7.79KiB 0 0.0237797 False False
datadog_agent_remap_blackhole 352.72KiB 0.62 99.98% 55.69MiB 3.58MiB 74.54KiB 0 0.0642089 56.04MiB 2.85MiB 59.39KiB 0 0.0507881 False False
http_pipelines_blackhole_acks 6.01KiB 0.49 96.57% 1.2MiB 115.41KiB 2.35KiB 0 0.0936836 1.21MiB 78.19KiB 1.59KiB 0 0.0631649 False False
splunk_hec_route_s3 69.1KiB 0.36 70.49% 18.65MiB 2.29MiB 47.66KiB 0 0.122651 18.72MiB 2.18MiB 45.64KiB 0 0.116576 False False
http_pipelines_blackhole 2.41KiB 0.14 66.13% 1.64MiB 47.91KiB 1002.84B 0 0.0285074 1.64MiB 113.6KiB 2.32KiB 0 0.0675002 False False
syslog_regex_logs2metric_ddmetrics 8.38KiB 0.07 42.33% 12.44MiB 523.89KiB 10.68KiB 0 0.0411081 12.45MiB 517.49KiB 10.55KiB 0 0.0405792 False False
splunk_hec_to_splunk_hec_logs_noack 16.67KiB 0.07 83.97% 23.82MiB 476.27KiB 9.72KiB 0 0.0195219 23.84MiB 333.34KiB 6.81KiB 0 0.013654 False False
syslog_humio_logs 12.29KiB 0.07 99.99% 16.71MiB 105.59KiB 2.16KiB 0 0.00616864 16.72MiB 104.0KiB 2.13KiB 0 0.00607151 False False
splunk_hec_to_splunk_hec_logs_acks 6.89KiB 0.03 23.17% 23.76MiB 829.66KiB 16.88KiB 0 0.034096 23.76MiB 794.35KiB 16.17KiB 0 0.0326357 False False
enterprise_http_to_http -1.9KiB -0.01 20.60% 23.85MiB 249.73KiB 5.1KiB 0 0.0102246 23.84MiB 253.96KiB 5.2KiB 0 0.0103985 False False
file_to_blackhole -47.8KiB -0.05 38.09% 95.34MiB 2.96MiB 61.26KiB 0 0.0309875 95.3MiB 3.56MiB 74.11KiB 0 0.0373811 False False
splunk_hec_indexer_ack_blackhole -17.71KiB -0.07 49.17% 23.76MiB 893.1KiB 18.17KiB 0 0.0367057 23.74MiB 966.36KiB 19.65KiB 0 0.0397453 False False
http_text_to_http_json -29.98KiB -0.08 78.64% 38.46MiB 825.72KiB 16.86KiB 0 0.0209601 38.43MiB 844.15KiB 17.23KiB 0 0.0214442 False False
socket_to_socket_blackhole -26.82KiB -0.11 100.00% 24.13MiB 139.93KiB 2.86KiB 0 0.00566062 24.11MiB 202.34KiB 4.13KiB 0 0.00819421 False False
http_to_http_json -38.71KiB -0.16 99.71% 23.85MiB 336.52KiB 6.87KiB 0 0.013778 23.81MiB 538.61KiB 11.0KiB 0 0.0220871 False False
syslog_log2metric_splunk_hec_metrics -34.55KiB -0.19 91.82% 17.97MiB 576.49KiB 11.75KiB 0 0.0313245 17.94MiB 786.42KiB 16.0KiB 0 0.0428115 False False
fluent_elasticsearch -187.66KiB -0.23 100.00% 79.47MiB 54.36KiB 1.1KiB 0 0.00066789 79.29MiB 1.7MiB 35.05KiB 0 0.0214797 False False
http_to_http_noack -78.68KiB -0.32 99.72% 23.82MiB 612.27KiB 12.5KiB 0 0.0250965 23.74MiB 1.11MiB 23.12KiB 0 0.046689 False False
datadog_agent_remap_datadog_logs_acks -248.06KiB -0.39 96.71% 61.67MiB 3.45MiB 72.02KiB 0 0.0559074 61.43MiB 4.38MiB 91.22KiB 0 0.0713273 False False
syslog_log2metric_humio_metrics -55.93KiB -0.42 100.00% 12.87MiB 218.29KiB 4.46KiB 0 0.0165634 12.81MiB 489.16KiB 9.97KiB 0 0.0372745 False False
datadog_agent_remap_datadog_logs -281.55KiB -0.46 99.79% 60.36MiB 1.23MiB 25.72KiB 0 0.0203314 60.09MiB 4.22MiB 87.77KiB 0 0.070142 False False
syslog_loki -68.47KiB -0.47 99.99% 14.34MiB 421.42KiB 8.62KiB 0 0.0286924 14.27MiB 751.56KiB 15.28KiB 0 0.0514097 False False
http_to_http_acks -136.25KiB -0.77 43.02% 17.35MiB 8.17MiB 170.85KiB 0 0.470934 17.22MiB 8.06MiB 168.1KiB 0 0.467811 True True

src/sources/docker_logs.rs Outdated Show resolved Hide resolved
src/sources/docker_logs.rs Show resolved Hide resolved
Signed-off-by: Stephen Wakely <[email protected]>
@StephenWakely StephenWakely enabled auto-merge (squash) September 15, 2022 16:28
@StephenWakely StephenWakely merged commit fb4ae49 into master Sep 15, 2022
@StephenWakely StephenWakely deleted the stephen/docker_source_assert branch September 15, 2022 17:58
@github-actions
Copy link

Soak Test Results

Baseline: 2db1aab
Comparison: b89c8c8
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
http_pipelines_no_grok_blackhole 180.95KiB 1.69 100.00% 10.43MiB 431.42KiB 8.81KiB 0 0.0403886 10.61MiB 1.11MiB 23.14KiB 0 0.104798 False False
http_to_http_acks 242.12KiB 1.39 68.79% 17.04MiB 8.12MiB 169.85KiB 0 0.476813 17.27MiB 8.09MiB 168.83KiB 0 0.468324 True True
http_pipelines_blackhole_acks 10.57KiB 0.88 100.00% 1.17MiB 97.59KiB 1.99KiB 0 0.0810949 1.19MiB 61.01KiB 1.24KiB 0 0.0502529 False False
datadog_agent_remap_blackhole_acks 495.39KiB 0.78 100.00% 61.99MiB 4.11MiB 85.65KiB 0 0.0663225 62.48MiB 2.88MiB 60.29KiB 0 0.0461386 False False
syslog_humio_logs 71.28KiB 0.43 100.00% 16.22MiB 240.18KiB 4.9KiB 0 0.0144584 16.29MiB 241.05KiB 4.94KiB 0 0.0144484 False False
syslog_log2metric_splunk_hec_metrics 73.16KiB 0.42 97.10% 17.2MiB 1.06MiB 22.14KiB 0 0.0616423 17.28MiB 1.21MiB 25.12KiB 0 0.0698095 False False
datadog_agent_remap_blackhole 238.27KiB 0.4 94.70% 57.63MiB 4.56MiB 94.94KiB 0 0.0790877 57.87MiB 3.76MiB 78.41KiB 0 0.0649469 False False
syslog_regex_logs2metric_ddmetrics 44.75KiB 0.35 98.67% 12.36MiB 670.65KiB 13.65KiB 0 0.0529807 12.4MiB 581.05KiB 11.84KiB 0 0.0457407 False False
syslog_splunk_hec_logs 30.1KiB 0.18 88.00% 16.29MiB 730.14KiB 14.86KiB 0 0.0437534 16.32MiB 607.61KiB 12.4KiB 0 0.0363454 False False
splunk_hec_to_splunk_hec_logs_noack 17.53KiB 0.07 86.66% 23.82MiB 466.3KiB 9.52KiB 0 0.0191116 23.84MiB 331.23KiB 6.76KiB 0 0.013566 False False
datadog_agent_remap_datadog_logs_acks 17.7KiB 0.03 11.82% 60.7MiB 3.74MiB 78.11KiB 0 0.0616787 60.72MiB 4.31MiB 89.74KiB 0 0.070992 False False
splunk_hec_indexer_ack_blackhole 6.89KiB 0.03 21.95% 23.75MiB 874.21KiB 17.79KiB 0 0.035932 23.76MiB 843.85KiB 17.17KiB 0 0.0346746 False False
splunk_hec_to_splunk_hec_logs_acks -2.69KiB -0.01 7.80% 23.77MiB 938.7KiB 19.1KiB 0 0.0385573 23.77MiB 968.1KiB 19.69KiB 0 0.0397693 False False
enterprise_http_to_http -5.12KiB -0.02 50.53% 23.85MiB 262.96KiB 5.37KiB 0 0.0107658 23.84MiB 255.94KiB 5.24KiB 0 0.0104809 False False
file_to_blackhole -46.88KiB -0.05 39.15% 95.34MiB 3.06MiB 63.45KiB 0 0.0320955 95.3MiB 3.17MiB 65.95KiB 0 0.0332589 False False
socket_to_socket_blackhole -29.28KiB -0.12 67.23% 23.23MiB 930.22KiB 18.99KiB 0 0.0391006 23.2MiB 1.11MiB 23.11KiB 0 0.0476335 False False
http_to_http_json -28.74KiB -0.12 98.45% 23.85MiB 329.1KiB 6.72KiB 0 0.0134732 23.82MiB 478.19KiB 9.78KiB 0 0.0196001 False False
datadog_agent_remap_datadog_logs -80.62KiB -0.13 50.36% 59.98MiB 3.07MiB 64.29KiB 0 0.0510931 59.9MiB 4.78MiB 99.55KiB 0 0.079762 False False
fluent_elasticsearch -189.27KiB -0.23 100.00% 79.47MiB 54.07KiB 1.09KiB 0 0.000664326 79.29MiB 1.72MiB 35.5KiB 0 0.0217403 False False
http_text_to_http_json -107.83KiB -0.28 100.00% 37.98MiB 868.96KiB 17.74KiB 0 0.0223392 37.87MiB 886.19KiB 18.09KiB 0 0.0228456 False False
http_to_http_noack -81.1KiB -0.33 99.86% 23.82MiB 555.39KiB 11.34KiB 0 0.0227609 23.75MiB 1.09MiB 22.77KiB 0 0.0459805 False False
syslog_log2metric_humio_metrics -46.19KiB -0.36 99.90% 12.54MiB 369.24KiB 7.54KiB 0 0.0287577 12.49MiB 580.91KiB 11.83KiB 0 0.0454069 False False
http_pipelines_blackhole -6.66KiB -0.4 99.42% 1.63MiB 63.8KiB 1.3KiB 0 0.0381959 1.62MiB 99.55KiB 2.03KiB 0 0.0598421 False False
splunk_hec_route_s3 -85.69KiB -0.45 80.26% 18.6MiB 2.29MiB 47.74KiB 0 0.123257 18.52MiB 2.21MiB 46.25KiB 0 0.119429 False False
syslog_loki -199.15KiB -1.33 100.00% 14.63MiB 309.52KiB 6.34KiB 0 0.0206634 14.43MiB 718.28KiB 14.6KiB 0 0.0485977 False False

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain: ci Anything related to Vector's CI environment domain: sources Anything related to the Vector's sources
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants