v68

Warning

BREAKING CHANGE

Starting in the v67 version, the Next Generation Datadog Lambda Extension is the default, to force launch of the compatibility (previous) Extension, set the DD_EXTENSION_VERSION=compatibility

The Next-Gen Datadog Lambda Extension is designed specifically for Lambda and makes different tradeoffs around the lifecycle and processing of data than the original, converted datadog-agent. While we expect the features and configuration to be compatible, there are timing changes to when data flushes, or aggregation changes which are best suited to the Lambda lifecycle. As a result, this is a major version change and we strongly recommend testing this release in dev/staging/pre-production environment before moving to production.

Read about the Next Generation Datadog Lambda Extension.

Important

Compatibility (previous) Extension Agent version is v7.60.1

arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-Extension:68
arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-Extension-ARM:68
arn:aws-us-gov:lambda:us-gov-<AWS_REGION>:002406178527:layer:Datadog-Extension:68
arn:aws-us-gov:lambda:us-gov-<AWS_REGION>:002406178527:layer:Datadog-Extension-ARM:68

What's Changed

Next Generation Extension:

• fix version parsing for number by @alexgallotta in #492
• debug log error for reports by @alexgallotta in #490
• fix: fallback on intake urls by @duncanista in #495
• fix: Decode error message and error type by @shreyamalpani in #500
• fix: set dogstatsd timeout by @alexgallotta in #497
• chore: update dependencies by @duncanista in #501
• fix: turn headers serializer into a deserializer by @duncanista in #502

Compatibility (Previous) Extension:

• [serverless] decode lambda error.msg and error.type by @shreyamalpani in DataDog/datadog-agent#32231
• [serverless] Support trace context propagation for ALB target groups with MultiValueHeaders by @purple4reina in DataDog/datadog-agent#31542

Full Changelog: v67...v68

v67

Warning

BREAKING CHANGE

This version sets the Next Generation Datadog Lambda Extension as default, to force launch of the compatibility (previous) Extension, set the DD_EXTENSION_VERSION=compatibility

The Next-Gen Datadog Lambda Extension is designed specifically for Lambda and makes different tradeoffs around the lifecycle and processing of data than the original, converted datadog-agent. While we expect the features and configuration to be compatible, there are timing changes to when data flushes, or aggregation changes which are best suited to the Lambda lifecycle. As a result, this is a major version change and we strongly recommend testing this release in dev/staging/pre-production environment before moving to production.

Read about the Next Generation Datadog Lambda Extension.

Important

Compatibility (previous) Extension Agent version is v7.59.0

arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-Extension:67
arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-Extension-ARM:67
arn:aws-us-gov:lambda:us-gov-<AWS_REGION>:002406178527:layer:Datadog-Extension:67
arn:aws-us-gov:lambda:us-gov-<AWS_REGION>:002406178527:layer:Datadog-Extension-ARM:67

What's Changed

Next Generation Extension:

• rename failover to fallback by @alexgallotta in #465
• feat(bottlecap): add runtime tag by @duncanista in #464
• feat(bottlecap): handle timeouts by @duncanista in #466
• feat(bottlecap): handle OOM exceptions by @duncanista in #467
• fix(bottlecap): fallback when otel set by @duncanista in #470
• fix(bottlecap): fix aws.lambda service naming by @duncanista in #471
• fix(bottlecap): try kms decrypt without context as in goagent by @alexgallotta in #472
• feat(bottlecap): fallback on opted out only by @duncanista in #473
• fix(bottlecap): fallback on yaml otel config by @duncanista in #474
• feat(bottlecap): honor serverless_logs by @astuyve in #475
• chore: update Go version in multiple places by @duncanista in #477
• fix(bottlecap): priority of decrypt: kms > arn > use dd_api_key by @alexgallotta in #478
• feat(bottlecap): Flush timeouts by @astuyve in #480
• fix(bottlecap): dogstatsd to filter invocations from lambda library by @duncanista in #481
• feat(bottlecap): support adaptive flush strategy again by @astuyve in #482
• feat(bottlecap): 1min default by @astuyve in #484

Compatibility (Previous) Extension:

• change env variable for bottlecap failover->fallback DataDog/datadog-agent@877d60d
• run remote config service in extension and start trace agent config endpoint DataDog/datadog-agent@53070e2

Full Changelog: v66...v67

v66

Important

Agent version is v7.59.0

arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-Extension:66
arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-Extension-ARM:66
arn:aws-us-gov:lambda:us-gov-<AWS_REGION>:002406178527:layer:Datadog-Extension:66
arn:aws-us-gov:lambda:us-gov-<AWS_REGION>:002406178527:layer:Datadog-Extension-ARM:66

What's Changed

Main Extension:

• Move flags for stripping and scrubbing keys to agent func DataDog/datadog-agent@e1ea1f1
• Read 128 bit trace ID in End Invocation DataDog/datadog-agent@5cfbd26
• [Serverless] Reset both inferred spans on each invocation. DataDog/datadog-agent@9b6d767
• [serverless] Extract EventBridge from SNS/SQS Payloads and Create EventBridge Spans DataDog/datadog-agent@024af28
• Fix port collision in serverless daemon tests. DataDog/datadog-agent@231be18
• [serverless] Add EventBridge Inferred Spans DataDog/datadog-agent@be76ade
• Enhanced Metrics Fix DataDog/datadog-agent@0df804c
• [SVLS-5034] Create trace context from Step Function execution details DataDog/datadog-agent@a4534bc
• [SVLS-5551] Add lambda threads enhanced metrics DataDog/datadog-agent@e99d8db

Next Generation Extension:

Important

Customers using .NET, Java, Ruby, and Go can now opt-in to take advantage of the performance improvements.

• chore: decouple hyper from trace_processor by @duncanista in #390
• fix: sync with fixed dogstatsd by @alexgallotta in #402
• feat: support HTTPS_PROXY by @astuyve in #381
• fix: add ap-southeast-5 region by @duncanista in #411
• fix: display correct Agent version in Alpine builds by @nhulston in #422
• fix: aj/honor both proxies in order by @astuyve in #410
• fix: bump go version in alpine image to 1.23.2 by @duncanpharvey in #426
• chore: adding static flag to build parameters for alpine images by @TophrC-dd in #438
• feat(bottlecap): add Universal Instrumentation by @duncanista, @alexgallotta, @shreyamalpani, and @astuyve in #374

New Contributors

• @nhulston made their first contribution in #422
• @TophrC-dd made their first contribution in #438

Full Changelog: v65...v66

v65

What's Changed

Agent Tag
Agent Version: 7.57.0

arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-Extension:65
arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-Extension-ARM:65
arn:aws-us-gov:lambda:us-gov-<AWS_REGION>:002406178527:layer:Datadog-Extension:65
arn:aws-us-gov:lambda:us-gov-<AWS_REGION>:002406178527:layer:Datadog-Extension-ARM:65

Main Extension:

• [SVLS-5257] Add lambda file descriptor enhanced metrics DataDog/datadog-agent@0dea3e9
• [ASCII-2222] Fix serverless log level logic DataDog/datadog-agent@849a13c
• [SVLS-5268] Disable flush ticker for AWS Lambda DataDog/datadog-agent@381e420

Next generation Datadog Extension:

• chore(bottlecap): fast failover by @duncanista in #371
• aj/fix log level casing by @astuyve in #372
• chore(bottlecap): update dev image by @duncanista in #375
• feat: break if we shouldn't wait for the end by @astuyve in #377
• feat: increase to 10k contexts by @astuyve in #380
• Use upstream dogtstatsd by @alexgallotta in #368
• fix: licenses by @astuyve in #386
• feat: Specify aws_lc_rs as our default crypto provider by @astuyve in #385
• feat: Remove unused deps, drop minor version pins and update by @astuyve in #387
• feat: failover on dd proxy by @astuyve in #391

Full Changelog: v64...v65

New Contributors

• @ichinaski made their first contribution in #379

v64

What's Changed

Agent Tag
Agent Version: 7.55.3

arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-Extension:64
arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-Extension-ARM:64
arn:aws-us-gov:lambda:us-gov-<AWS_REGION>:002406178527:layer:Datadog-Extension:64
arn:aws-us-gov:lambda:us-gov-<AWS_REGION>:002406178527:layer:Datadog-Extension-ARM:64

Main Extension:

• fix: get lambda runtime API endpoint dynamically DataDog/datadog-agent@1d07d29

Next generation Datadog Extension:

• feat: Use stable instead of nightly by @astuyve in #355
• feat: Allowlist additional env vars by @astuyve in #354
• chore(bottlecap): reduce allocations for logs processing by @duncanista in #345
• aj/allow apm replace tags array by @astuyve in #358
• fix: update docs by @astuyve in #360
• fix(bottlecap): set explicit deny list and allow yaml usage by @duncanista in #363
• Modularize dogstatsd by @alexgallotta in #361

Full Changelog: v63...v64

v63

What's Changed

Agent Tag
Agent Version: 7.55.3

arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-Extension:63
arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-Extension-ARM:63
arn:aws-us-gov:lambda:us-gov-<AWS_REGION>:002406178527:layer:Datadog-Extension:63
arn:aws-us-gov:lambda:us-gov-<AWS_REGION>:002406178527:layer:Datadog-Extension-ARM:63

Datadog Agent:

• [Bugfix] [SLES-1749] histogram_aggregates and histogram_percentiles added to serverless in… DataDog/datadog-agent@94c9e78

• [serverless] Add debugging log line for OOM substrings DataDog/datadog-agent@3b686ac

Rust:

• feat: bias main select by @astuyve in #339
• fix(bottlecap): update build images by @duncanista in #340
• fix(bottlecap): remove nightly in favor or stable by @duncanista in #341
• feat: Remove excepts from secrets, drop into idle loop when local by @astuyve in #342
• feat: Open up more env vars which we don't rely on by @astuyve in #344
• Scan latest releases for vulnerabilities. by @purple4reina in #343
• feat: Allow trace disabled plugins by @astuyve in #348

Full Changelog: v62...v63

v62

What's Changed

⚠️ Container Images

A change was made a while ago in our documentation which requires to update which artifacts are copied from our datadog/lambda-extension image. Ensure you modify it from the previous instrumentation to the new one. Starting v61 this is required to get your instrumentation working properly.

Expected:

# now
COPY --from=public.ecr.aws/datadog/lambda-extension:<TAG> /opt/. /opt/

Previous:

# previous
COPY --from=public.ecr.aws/datadog/lambda-extension:<TAG> /opt/extensions/ /opt/extensions

---

Agent Tag
Agent Version: 7.55.1

arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-Extension:62
arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-Extension-ARM:62
arn:aws-us-gov:lambda:us-gov-<AWS_REGION>:002406178527:layer:Datadog-Extension:62
arn:aws-us-gov:lambda:us-gov-<AWS_REGION>:002406178527:layer:Datadog-Extension-ARM:62

Datadog Agent:

• feat(serverless): temporary set dogstatsd socket to empty DataDog/datadog-agent@a5c3d2b

Rust:

• fix(bottlecap): remove log that might confuse customers by @duncanista in #333
• fix: image builder agent resolution by @duncanista in #332
• fix dogstatsd multiline by @alexgallotta in #335
• chore(bottlecap): add cold start environment variables by @duncanista in #337

Full Changelog: v61...v62

v61

---

⚠️ Container Images

A change was made a while ago in our documentation which requires to update which artifacts are copied from our datadog/lambda-extension image. Ensure you modify it from the previous instrumentation to the new one. Since v61 requires this to work properly.

Expected:

# now
COPY --from=public.ecr.aws/datadog/lambda-extension:<TAG> /opt/. /opt/

Previous:

# previous
COPY --from=public.ecr.aws/datadog/lambda-extension:<TAG> /opt/extensions/ /opt/extensions

🚨 There is a known issue with this version. We recommend using v62:

• An issue was encountered too when using container images, we published a faulty image to the latest, latest-alpine, 61, and 61-alpine images. But this has been remediated and fixed.
• This version reports the agent version as 6.0.0 instead of the expected.
• An UDS error message appears on initialization, it's benign, but doesn't provide a great experience for customers.

---

Agent Tag
Agent Version: 7.55.1

arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-Extension:61
arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-Extension-ARM:61
arn:aws-us-gov:lambda:us-gov-<AWS_REGION>:002406178527:layer:Datadog-Extension:61
arn:aws-us-gov:lambda:us-gov-<AWS_REGION>:002406178527:layer:Datadog-Extension-ARM:61

What's Changed

🔔 This version was release off of the main branch, unlike previous releases where commits are cherry-picked. Below are Serverless specific changes:

Datadog Agent:

• [SVLS-5070][serverless] Fix Serverless logs request sync with flush DataDog/datadog-agent@6332366
• [SVLS-5001] Add tmp_used and tmp_max enhanced lambda metrics DataDog/datadog-agent@293e98e
• Support noop trace agent for serverless. DataDog/datadog-agent@6cdd940
• bump go-libddwaf to v3.3.0 DataDog/datadog-agent@946cd11
• feat(serverless): send bottlecap failover metric DataDog/datadog-agent@a0f8078
• [SVLS-4912] Generate CPU utilization metrics DataDog/datadog-agent@827a81b
• [SVLS-4936] Add lambda network enhanced metrics DataDog/datadog-agent@1cfabef
• [SVLS-4640][serverless] Fix dropped logs in AWS Lambda DataDog/datadog-agent@7e6dc75
• [SVLS-3697] Fix missing post runtime duration metrics and log values after a shutdown DataDog/datadog-agent@6e4f4f8
• Serverless Agent supports stream-logs DataDog/datadog-agent@4660cce

Rust:

• feat(bottlecap): warn on hello by @astuyve in #314
• feat(bottlecap): warn by default by @astuyve in #316
• chore(bottlecap): add [email protected] in telemetry api by @duncanista in #312
• chore(bottlecap: )add mock for integration test for metrics by @alexgallotta in #310
• fix(bottlecap): hyper refactor removed an error type by @alexgallotta in #321
• chore(documentation): Fix dead link for contributing.md by @lym953 in #322
• feat(bottlecap): multi-metric support by @astuyve in #317
• chore(bottlecap): add license by @alexgallotta in #320
• chore(bottlecap): fallback on datadog.yaml usage by @duncanista in #326
• feat(bottlecap): add runtime tag by @alexgallotta in #327
• feat(bottlecap): add dd_extension_version to tags by @duncanista in #328
• fix(bottlecap): filter debug logs from external crates by @duncanista in #329
• fix(bottlecap): use /proc/pid/environ to read the env variable for runtime by @alexgallotta in #330
• chore(bottlecap): switch flushing strategy to race by @astuyve in #318
  and many more...

New Contributors

• @lym953 made their first contribution in #322

Full Changelog: v60...v61

v60

Agent Tag
Agent Version: 7.54.0

arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-Extension:60
arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-Extension-ARM:60
arn:aws-us-gov:lambda:us-gov-<AWS_REGION>:002406178527:layer:Datadog-Extension:60
arn:aws-us-gov:lambda:us-gov-<AWS_REGION>:002406178527:layer:Datadog-Extension-ARM:60

What's Changed

🔔 This version is a duplicate of v58.

• Enable to set the buffering config of Lambda Logs API via env vars DataDog/datadog-agent@1eb4f52

• [Serverless] Reuse DD_ENHANCED_METRICS for new system enhanced metrics DataDog/datadog-agent@62532f6

• Improve logging in the Lambda Logs API handler DataDog/datadog-agent@e5b69ff

• [SVLS-4850] Generate lambda enhanced CPU system, user, and total metrics DataDog/datadog-agent@fa5ba1b

• [SVLS] Mark serverless unit test TestLoadConfigShouldBeFast as flaky DataDog/datadog-agent@ebe363c

• Set _dd.p.dm trace field to null in Serverless Integration Tests Snapshots DataDog/datadog-agent@f20521b

• [ASCII-1218] Only initialize needed configs for serverless DataDog/datadog-agent@09161cd

• [SVLS-4563] Adds provided.al2023 as a runtime tag DataDog/datadog-agent@3b65821

• [SVLS] Add AWS Account Id Fields to Credit Card Obfuscation Exclusion DataDog/datadog-agent@527cc4f

v59

🚨 There is a known issue with this version. We recommend using another version as we continue to investigate.

Agent Tag
Agent Version: 7.54.1

arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-Extension:59
arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-Extension-ARM:59
arn:aws-us-gov:lambda:us-gov-<AWS_REGION>:002406178527:layer:Datadog-Extension:59
arn:aws-us-gov:lambda:us-gov-<AWS_REGION>:002406178527:layer:Datadog-Extension-ARM:59

What's Changed

🔔 This version was release off of the main branch, unlike previous releases where commits are cherry-picked. Below are Serverless specific changes:

• feat(serverless): send bottlecap failover metric DataDog/datadog-agent@a0f8078

• [SVLS-4912] Generate CPU utilization metrics DataDog/datadog-agent@827a81b

• [serverless-build-workflow] Compare with merge-base instead of just base ref DataDog/datadog-agent@b950f74

• [SVLS-4936] Add lambda network enhanced metrics DataDog/datadog-agent@1cfabef

• [SVLS-4640][serverless] Fix dropped logs in AWS Lambda DataDog/datadog-agent@7e6dc75

• [Serverless] Maintain serverless PR comments independently. DataDog/datadog-agent@4683873

• [Serverless] Post binary size change when serverless deps file changes. DataDog/datadog-agent@f0722c6

• Move serverless vuln scan to extension repo. DataDog/datadog-agent@41f10b8

• [SVLS-3697] Fix missing post runtime duration metrics and log values after a shutdown DataDog/datadog-agent@6e4f4f8

• Serverless Agent supports stream-logs DataDog/datadog-agent@4660cce