Bug: sendDistributionMetricWithDate custom date doesn't have any effect in DataDog

[PatrykMilewski]

Expected Behavior

When I send data point using sendDistributionMetricWithDate function, I want to see in DataDog data point with matching timestamp.

Actual Behavior

When I send data point using sendDistributionMetricWithDate function, DataDog ignores date provided and displays data point with the ingestion time.

Steps to Reproduce the Problem

1. Call sendDistributionMetricWithDate with Date from the past, for example 2 hours behind.
2. Wait for data point to appear in DataDog.

Specifications

• Datadog Lambda Layer version: "datadog-lambda-js": "2.20.0"
• Node version: 12.x

More details

I observed the same behaviour for setting 30 minutes in the past and 10 minutes into the future. All of the data points were displayed in DataDog with ingestion date instead of specified date.

Logs and screenshots

Log from Lambda in debug mode:

2020-04-02T12:44:57.304Z	375befed-1693-4bd6-9fd4-45139104c9b9	DEBUG	[dd.trace_id=5038852051485810583 dd.span_id=1784722096679938107] 
{
    "status": "debug",
    "message": "datadog:sending payload with body {\"series\":[{\"metric\":\"mux.test.viewer_experience_score\",\"points\":[[1585821600545,[0.0]]],\"tags\":[\"dd_lambda_layer:datadog-nodev12.16.1\"],\"type\":\"distribution\"}]}"
}

Timestamp is in UTC:

> new Date(1585821600545)
2020-04-02T10:00:00.545Z

Parsed JSON:

{
  "series": [
    {
      "metric": "mux.test.viewer_experience_score",
      "points": [
        [
          1585821600545,
          [
            0.0
          ]
        ]
      ],
      "tags": [
        "dd_lambda_layer:datadog-nodev12.16.1"
      ],
      "type": "distribution"
    }
  ]
}

I was looking as far as 15 months into the past but couldn't find any data points.

I believe something is wrong with the API, since the request according to the logs looks completely fine.

[sfirrin]

Hi @PatrykMilewski, thanks for the detailed report.

I'm able to reproduce and I've found the bug in the layer that's causing this. I'm working on a fix now and I'll update you here when I'm able to merge it.

[PatrykMilewski]

@sfirrin Thank you!

[PatrykMilewski]

I see, that a patch is merged. I will update my lambdas today and let you guys know, if it's fixed.

[sfirrin]

👍 sounds good

[tianchu]

@PatrykMilewski did you get a chance to test out the fix?

[tianchu]

I'm closing this issue and assuming the problem has been fixed. Feel free to reopen if that's not the case.

[PatrykMilewski]

@tianchu I'm sorry that I didn't responded, but we started using another library for metrics ingestion into DataDog and I didn't really had a chance to check if this bug is fixed for me.

[pinkerton]

@PatrykMilewski are you using a third-party library for submitting custom metrics from Lambda functions? Asking because we've researched most of these and they rely on a deprecated feature for submitting metrics: https://docs.datadoghq.com/integrations/amazon_lambda/?tab=awsconsole#deprecated-using-cloudwatch-logs

[PatrykMilewski]

@pinkerton I'm using this one:
https://www.npmjs.com/package/dogapi
It's not the best maintained one, but since I use it only for sending gauge and count custom metrics, I'm fine with replacing it with my implementation, if it's going to break in the future.

I really appreciate what you do guys with this library, but I think it's not supposed to be used with our use case.

We use Lambdas to integrate services, that doesn't have built-in integration with DataDog. So basically what we do is reading metrics in Lambda and then instantly ingesting it into DataDog. We are not interested about monitoring the Lambda by itself (but in the future we plan to monitor the monitors, so maybe I will get back to it 😄).

I believe it's a right move, since I'm going to request for stuff, that shouldn't be really in this library, since it's designed for monitoring the Lambda, not just use Lambda as a proxy between 3rd party services and DataDog.