Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(aws_kinesis_firehose sink): File cleanup and use SinkRequestBuildError #14478

Merged
merged 3 commits into from
Sep 20, 2022

Conversation

spencergilbert
Copy link
Contributor

Signed-off-by: Spencer Gilbert [email protected]

Closes #14203

Majority of errors are handled by the new sink code, or the encoder.

@spencergilbert spencergilbert added ci-condition: integration tests enable Run integration tests on this PR domain: observability Anything related to monitoring/observing Vector sink: aws_kinesis_firehose Anything `aws_kinesis_firehose` sink related labels Sep 20, 2022
@spencergilbert spencergilbert self-assigned this Sep 20, 2022
@netlify
Copy link

netlify bot commented Sep 20, 2022

Deploy Preview for vector-project canceled.

Name Link
🔨 Latest commit 85d561e
🔍 Latest deploy log https://app.netlify.com/sites/vector-project/deploys/6329c3661787260008af6358

@github-actions github-actions bot added the domain: sinks Anything related to the Vector's sinks label Sep 20, 2022
Copy link
Contributor

@neuronull neuronull left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one comment similar to cloudwatch PR, otherwise LGTM 👍

src/sinks/aws_kinesis_firehose/service.rs Outdated Show resolved Hide resolved
@spencergilbert spencergilbert enabled auto-merge (squash) September 20, 2022 14:52
@spencergilbert spencergilbert merged commit 512da40 into master Sep 20, 2022
@spencergilbert spencergilbert deleted the spencer/audit-aws_kinesis_firehose-sink branch September 20, 2022 15:13
@github-actions
Copy link

Soak Test Results

Baseline: 14fefee
Comparison: 85d561e
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
socket_to_socket_blackhole 632.95KiB 2.66 100.00% 23.22MiB 755.65KiB 15.42KiB 0 0.0317732 23.84MiB 796.39KiB 16.26KiB 0 0.0326179 False False
splunk_hec_route_s3 363.49KiB 1.89 100.00% 18.81MiB 2.2MiB 45.9KiB 0 0.117137 19.17MiB 2.17MiB 45.38KiB 0 0.113222 False False
syslog_loki 238.49KiB 1.66 100.00% 14.02MiB 437.96KiB 8.96KiB 0 0.0304893 14.26MiB 737.22KiB 14.99KiB 0 0.0504846 False False
datadog_agent_remap_blackhole_acks 798.58KiB 1.34 100.00% 58.38MiB 4.39MiB 91.51KiB 0 0.0752559 59.16MiB 3.51MiB 73.45KiB 0 0.0593451 False False
syslog_regex_logs2metric_ddmetrics 120.15KiB 0.95 100.00% 12.41MiB 590.25KiB 12.03KiB 0 0.046445 12.53MiB 585.24KiB 11.93KiB 0 0.0456195 False False
syslog_log2metric_splunk_hec_metrics 160.78KiB 0.89 100.00% 17.7MiB 607.9KiB 12.39KiB 0 0.0335406 17.85MiB 802.11KiB 16.33KiB 0 0.0438665 False False
syslog_humio_logs 144.98KiB 0.88 100.00% 16.09MiB 245.38KiB 5.01KiB 0 0.0148946 16.23MiB 262.28KiB 5.37KiB 0 0.0157811 False False
http_pipelines_blackhole_acks 10.55KiB 0.88 100.00% 1.17MiB 101.23KiB 2.06KiB 0 0.0846564 1.18MiB 71.27KiB 1.45KiB 0 0.0590855 False False
syslog_log2metric_humio_metrics 9.01KiB 0.07 51.54% 12.7MiB 274.45KiB 5.6KiB 0 0.0210927 12.71MiB 570.27KiB 11.61KiB 0 0.0437983 False False
splunk_hec_to_splunk_hec_logs_noack 17.58KiB 0.07 86.09% 23.82MiB 474.01KiB 9.67KiB 0 0.0194279 23.84MiB 337.68KiB 6.9KiB 0 0.0138304 False False
http_pipelines_blackhole 1.25KiB 0.07 35.22% 1.65MiB 11.44KiB 239.42B 0 0.00675466 1.65MiB 134.0KiB 2.73KiB 0 0.0790625 False False
enterprise_http_to_http -1.27KiB -0.01 13.87% 23.85MiB 249.51KiB 5.09KiB 0 0.0102157 23.85MiB 253.56KiB 5.19KiB 0 0.0103821 False False
datadog_agent_remap_blackhole -13.17KiB -0.02 10.61% 59.66MiB 3.43MiB 71.64KiB 0 0.0575467 59.65MiB 3.26MiB 67.93KiB 0 0.0546026 False False
file_to_blackhole -28.6KiB -0.03 24.25% 95.33MiB 3.2MiB 66.3KiB 0 0.0335626 95.31MiB 3.11MiB 64.71KiB 0 0.0326429 False False
http_to_http_acks -5.87KiB -0.03 2.01% 17.17MiB 7.81MiB 163.29KiB 0 0.454907 17.16MiB 7.97MiB 166.23KiB 0 0.464098 True True
splunk_hec_to_splunk_hec_logs_acks -16.54KiB -0.07 50.99% 23.76MiB 797.97KiB 16.24KiB 0 0.0327847 23.75MiB 866.26KiB 17.62KiB 0 0.0356145 False False
datadog_agent_remap_datadog_logs -43.53KiB -0.07 36.28% 62.74MiB 825.1KiB 16.9KiB 0 0.0128406 62.69MiB 4.36MiB 90.73KiB 0 0.0694948 False False
splunk_hec_indexer_ack_blackhole -19.75KiB -0.08 56.36% 23.76MiB 843.48KiB 17.17KiB 0 0.034658 23.74MiB 918.45KiB 18.68KiB 0 0.037769 False False
http_to_http_json -37.95KiB -0.16 99.73% 23.85MiB 334.22KiB 6.82KiB 0 0.0136822 23.81MiB 521.92KiB 10.66KiB 0 0.0213996 False False
fluent_elasticsearch -160.99KiB -0.2 100.00% 79.47MiB 52.5KiB 1.06KiB 0 0.000644962 79.32MiB 1.48MiB 30.57KiB 0 0.0187088 False False
syslog_splunk_hec_logs -57.77KiB -0.36 82.27% 15.49MiB 1.51MiB 31.35KiB 0 0.097119 15.44MiB 1.4MiB 29.16KiB 0 0.0903801 False False
http_to_http_noack -94.61KiB -0.39 99.94% 23.82MiB 599.06KiB 12.25KiB 0 0.0245545 23.73MiB 1.18MiB 24.49KiB 0 0.0495119 False False
datadog_agent_remap_datadog_logs_acks -375.23KiB -0.59 99.84% 62.32MiB 3.6MiB 75.25KiB 0 0.0578258 61.95MiB 4.42MiB 91.96KiB 0 0.0712906 False False
http_text_to_http_json -280.16KiB -0.69 100.00% 39.71MiB 1.08MiB 22.65KiB 0 0.0272841 39.43MiB 1.09MiB 22.71KiB 0 0.0275357 False False
http_pipelines_no_grok_blackhole -101.53KiB -0.92 100.00% 10.72MiB 36.66KiB 766.36B 0 0.00333742 10.62MiB 1.06MiB 22.03KiB 0 0.0995921 False False

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci-condition: integration tests enable Run integration tests on this PR domain: observability Anything related to monitoring/observing Vector domain: sinks Anything related to the Vector's sinks sink: aws_kinesis_firehose Anything `aws_kinesis_firehose` sink related
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Verify aws_kinesis_firehose sink
2 participants