Marks: First measurement decides if field is mapped to long or float

[roncohen]

When sending up marks with transactions we currently use a dynamic mapping which lets Elasticsearch determine the mapping when it sees a new mark for the first time. Unfortunately, JSON only has Numbers, it doesn't distinguish between int/long and floats. That means that when we send up a new mark, even if it's measured as a float in the agent, it can happen that the first measurement just happens to be a number with no significant decimal digits.

This will cause the field will be marked as a long in the Elasticsearch mapping. If we subsequently send up measurements that have significant decimal digits, they will all be truncated to integer. Since these are millisecond measurements, it might not matter too much, but we should at least make sure they are consistently mapped. Here's an example of a mapping that can happen today:

{
    "marks": {
        "dynamic": "true",
        "properties": {
            "navigationTiming": {
                "properties": {
                    "appBeforeBootstrap": {
                        "type": "float"
                    },
                    "navigationStart": {
                        "type": "long"
                    },
                    "thing1": {
                        "type": "long"
                    }
                }
            },
            "performance": {
                "type": "object"
            }
        }
    }
}

In this example, the first value of thing1 was 1. This caused the mapping to become long. Subsequent values like 1.2, 1.2333 will be truncated to 1.

[graphaelli]

@roncohen what do you think of making this specific to navigationTiming? Something like:

- name: marks.navigationTiming
  type: object
  object_type: scaled_float
  scaling_factor: 1000000
  dynamic: true
  description: >
    navigationTiming marks in milliseconds with nanosecond precision.

Users could extend fields.yml (or the new append_fields?) with any custom defined fields in the same way.

Note that object_type: scaled_float is not yet supported in beats, working on getting that in the meantime.

[graphaelli]

some form of elastic/beats#6691 is required for that approach.

[roncohen]

sounds good to me @graphaelli. Good job on the beats PR

[graphaelli]

In #828 (comment) @felixbarny brings up a good point:

mark values should always be a timestamp in ms, right? Shouldn't we map it to a date then?

Not my area of expertise here, but https://www.w3.org/TR/navigation-timing/#performancetiming seems to indicate all of these are times and not durations - should we map these to date with format epoch_millis? My understanding is that resolution is not nearly millisecond to mitigate security issues.

[simitt]

ping @jahtalab regarding the suggestion of treating mark values as timestamps.

[hmdhk]

@simitt , We convert navigation timings to duration since it is more useful that way. Of course the UI or the user can make this conversion but it's easier to query and compare transactions with durations since the exact timestamp is not relevant for these comparisons.

We can treat them as long values considering the original resolution varies in different browsers and it's generally not higher than 1 millisecond.

[roncohen]

to elaborate a bit: we have them as relative time offsets, relative to when the navigation was started (ish). It's useful to be able to aggregate across them e.g. show me the average time till the page load event fires. This would be way harder if they were stored as dates.

[beniwohli]

Would it make sense to use the same storage type and unit as duration, microseconds as integers? They are on the same scale as transaction durations, so using two different formats seems confusing.

Also, I don't think resolution limits of browsers should be a deciding factor, since backend agents have uses for marks as well (e.g. I have an experimental branch that marks start and end of garbage collection).

[graphaelli]

I've put #915 to address the immediate need. Namely, that specially stores transaction.marks.navigationTiming.* as scaled_floats- this doesn't impose the time unit at all but does limit the precision reasonably. Turns out elastic/beats#6843 was implemented while we were making decisions, making this implementation straightforward.

Is that sufficient for this or do we want to make sure any number stored under transaction.marks is a float? That's not supported in beats yet.

Also, I agree with @beniwohli that it's confusing to store durations as microseconds but marks as milliseconds. So far this data has been opaque to the APM ui but in case we will rely on it soon, I agree we should change the description to microseconds.

[simitt]

pinging @jahtalab and @roncohen regarding the question whether we could treat marks as microseconds instead of milliseconds

[beniwohli]

I vote to handle them exactly as transaction/span durations: agents send them as milliseconds with type float, and the apm-server stores them as longs representing microseconds

[hmdhk]

I'm ok with apm-server converting the payload to microseconds! So the agent payload stays the same but it is stored in microseconds in ES and the UI will show them in milliseconds again!

Pinging @elastic/apm-ui , since this change in the data (milliseconds to microseconds) affects the UI as well.

[formgeist]

@jahtalab When is this change due? In order for us to change it in the UI, we need to have it added in the next stack release.

[beniwohli]

I was under the impression that marks aren't implemented in the UI yet? So we can decide on a unit/format somewhat independently of the UI team (assuming that stored as long microsecond isn't a fundamental problem for the UI team, given that it's already used for durations)

[formgeist]

@beniwohli @jahtalab I was probably too quick to jump the shark; marks are not yet supported, so there's no issue with UI at the moment. Thought it was related something else than marks