[RFC] abstract clock subsystem: counter syntonization

[ghost]

Proposes a well-defined intermediate concept of scalar syntonized nanosecond resolution time with overflow protection above low-level counters/cycles/ticks and below higher level time abstractions (timescales, calenders, etc.). Details see the discussion below.

Currently this happens in the net subsystem but in principle this can and may be promoted to something more generic as it is an implementation of layers 1 (counter) and 2 (syntonization) of the proposed clock subsystem RFC which is itself relevant to the POSIX roadmap.

The infrastructure discussed in this RFC will first be used for TSCH TDMA.

[ghost]

@rlubos @jciupis @edmont relevant to you in the context of #59245

[ghost]

@cfriedt relevant to you in the context of POSIX clocks

[andyross]

No real objections to the typedef, but it seems a little nonorthogonal? Right now kernel time is in ticks, because a "tick" is what kernel time is measured in by definition. Conversion of ticks to ns (both numerically and via the existing timeout mechanism) is already provided for anyone who needs it, and the utilities to do so are uniformly 64 bit (there are 32 bit converters for smaller units, but not ns).

Having a separate "kind" of time seems more confusing than helpful to me, honestly. There aren't any APIs that use this yet. What's the end goal here?

[cfriedt]

@andyross - I would almost suggest a 1:1 with Florian. I had a few broader picture discussions with him about some hi-res timer / clock things and was sold on it. Aside, those old tickets might actually get closed soon!

[andyross]

Sure. There's just a long graveyard of attempts in this space, and I'd much rather we see an implementation/prototype arrive before making any changes to the public API. In particular an attempt to tie kernel timekeeping to a real unit is something we spilled a lot of blood trying to get away from, given that so many of our platforms use counters that don't convert cleanly into decimal units.

[ghost]

I'd much rather we see an implementation/prototype arrive before making any changes to the public API

@andyross : 100% agreed - the onus is on me to prove that it makes sense to decouple certain uses of time from existing kernel concepts (ticks/cycles) and that ktime_t is the right candidate to do so. I agree with @cfriedt that a F2F would be helpful.

In any case I provide some more thoughts in writing - depending on your communication preference no need to read the following fine print, though, if you prefer F2F anyway...

Conversion of ticks to ns (both numerically and via the existing timeout mechanism) is already provided for anyone who needs it

The new type is not meant to represent kernel time "on the fly" but to represent time in structures and variables independently of the underlying clock source which more often than not will not be the kernel cycle counter.

given that so many of our platforms use counters that don't convert cleanly into decimal units.

That's correct, rounding from/to the ktime_t domain must be dealt with. Clock sources will have to provide precise cycle values, similarly to what the kernel timer does (and the kernel timer is the default clock source). Rounding results in +/- 0.5 ns jitter or alarm errors but error does not accumulate over time.

There aren't any APIs that use this yet.

@cfriedt , @andyross : Do you agree that the waiting PR justifies ktime_t at least for the network subsystem? Would you prefer that we "hide" ktime_t somewhere in the network subsystem until we provide PRs in the POSIX/libc/TSCH area using that type in a broader sense?

IMO the ktime_t concept has been very useful to refactor time across the network subsystem (including but not limited to the IEEE 802.15.4 subsystem):

• Attempts to use generic types to represent time in the network subsystem failed in the past as a precise common specification of time was never provided. Fields representing time were used inconsistently within the same subsystem and across drivers due to lack of a well-defined concept of high-resolution scalar time.
• Standardizing around kernel cycles in net_pkt or the radio API wouldn't work as kernel cycles are not guaranteed to be high resolution enough to represent network time plus the clock source is almost never kernel time (but comparison to / synchronization with kernel time is still needed).
• Other existing notions of time would be inadequate to represent time in the network subsystem as well. The rationale is given in the description of ktime_t for all such types representing time in Zephyr so far.

What's the end goal here?

The ultimate end goal is defined in an RFC.

As soon as this PR is accepted the waiting PR will be added to the code base immediately introducing ktime_t as network time.

But that alone would not justify introducing ktime_t in such an exposed place. We all need to be 100% convinced that at least the following features make sense in Zephyr in the short to mid term, should be decoupled from kernel time and use ktime_t-based intermediate clock sources instead:

1. high-resolution POSIX clocks and libc time sharing a common adjusted and drift compensated low-power (tickless) hybrid RTC/high-res counter clock source which may or may not be (partially) based on the kernel timer and can be configured in DT to use other peripherals instead.
2. hybrid nanosecond resolution clock sources for TDMA / timed TX/RX protocols in the IEEE 802.15.4 stack (namely CSL and TSCH) based on the combination of a low-power, always-on RTC and a radio timer providing high resolution during RX/TX windows which will usually not involve kernel time at all
3. synchronization/compensation of the main internal clock source (and thereby POSIX/libc clocks) with low-frequency, high-precision "pulse-per-second" devices like always-on RTCs, NTP, PTP, GPS, clocks derived from TDMA radio or industrial automation protocols or even atomic clock modules.

These use cases have in common:

• They all rely on "hybrid" clock sources combining at least two hardware peripherals or drivers of which the kernel timer is only one (if used at all).
• They share a couple of algorithms (timer multiplexing, drift compensation, leap second support, epoch offsets, etc.) which should be re-usable across all of them as a shared clock subsystem and which cannot sensibly be based on ticks/cycles for sufficient precision and configurability.

Next steps that follow immediately will be:

• I'll provide PRs for the basics of a clock subsystem with support for generic hybrid clock sources, drift adjustment + well-defined timescales (this is when ktime_t would have to be moved from the network subsystem to something shared).
• @cfriedt , if I understood you correctly you'll then be able to use that clock subsystem to solve the current layering problem in POSIX and libc plus introduce a DT-based configuration approach for the hybrid POSIX/libc clock source.
• At the same time I'll start to base TSCH timing on the same common clock subsystem.

[ghost]

@kartben The build suddenly fails with "API_DEFINE: do not specify a non-Zephyr API for libc File:include/zephyr/sys/ktime.h". Any idea what that means? The only thing that changed is the comments.

[kartben]

@kartben The build suddenly fails with "API_DEFINE: do not specify a non-Zephyr API for libc File:include/zephyr/sys/ktime.h". Any idea what that means? The only thing that changed is the comments.

@keith-packard would know :)
See #60487

[keith-packard]

@kartben The build suddenly fails with "API_DEFINE: do not specify a non-Zephyr API for libc File:include/zephyr/sys/ktime.h". Any idea what that means? The only thing that changed is the comments.

@keith-packard would know :) See #60487

Whoa. Looks like that new check is busted. PR #60551 should help here.

[andyross]

So... I won't stand in the way here as a simple typedef isn't a big deal. I'll just say I remain deeply suspicious that this is going to help and not hurt. Real hardware on real devices doesn't work with abstracted fine grained timekeeping due to the aliasing issues with e.g. 32768 Hz clocks. Real apps need to know what the tick rate is and tune their timer math to that using APIs that expose the conversions explicitly. Real libraries and subsystems[1] need to be aware of this limitation and not just paper it over.

This is a pretty API. The hardware isn't pretty. The API is making promises the hardware can't keep. While, again, it's just a typedef, my suspicion is that creeping/contagious use of this kind of abstraction (vs. k_timeout_t, which requires explicit conversion at creation time) is going to take us right back to the world of constant annoying bad-timer-assumption bugs on nRF et. al.

[1] In particular, the network layer is in fact a pretty bad citizen here precisely because it's not tick-aware! Now... it never will be?

tl;dr: No -1 from me. But... I really think this needs to wait on some major subsystem work using it to land to see what the impact is going to be.

[cfriedt]

tl;dr: No -1 from me. But... I really think this needs to wait on some major subsystem work using it to land to see what the impact is going to be.

I wonder if it's possible to isolate things to a testsuite or sample application first, get some of these concepts upstream that way without it being public API, and then create a separate PR that essentially flips a switch (moves the headers into public API and demonstrates the MVP use case). That way, it's still a 2-way door.

A testsuite would kind of be ideal because we could test things on various pieces of hw, with e.g. a 32k timer.

[ghost]

I really think this needs to wait on some major subsystem work using it to land to see what the impact is going to be.

@andyross I think you're right. We need a deeper understanding of how this type is going to be embedded in a larger subsystem based on real code. I have the same concerns re aliasing and abuse of pseudo-precise nanosecond values. These problems need to be addressed right away - not later on. I propose I write more code and draw some pictures, add it all into this PR and then we continue the discussion.

I wonder if it's possible to isolate things to a testsuite or sample application first, get some of these concepts upstream that way without it being public API

@cfriedt Yes, this goes in the direction I proposed above when I said that we should probably hide this typedef in the network subsystem while it's only used there. I believe it is provably useful there - it's a big step forward, even if it never makes its way out of the network subsystem. Before we use the concept in POSIX clocks or libc, @andyross 's concerns need to be addressed anyway.

UPDATE: Done - see #60401.

In particular, the network layer is in fact a pretty bad citizen here precisely because it's not tick-aware!

@andyross Do you propose to base TX/RX timestamps, timed TX and RX windows on opaque counter cycle values? I actually considered this, but so far I assumed that this would be a bad idea:

• None of the standards relevant in the area (IP derivatives, gPTP, IEEE 802.15.4, Thread, Bluetooth) use counter values. They all use nanosecond values, microsecond values or timespec-like structs with well defined tolerances concerning drift and jitter (which usually excludes 32k oscillators anyway). Then these standards deal explicitly with aliasing problems but hide them behind MAC (L2) or PHY (L1) APIs.
• Timestamps in the network subsystem often come from different low-level counters and must still be comparable (e.g. when ETH and some radio are used together which is totally legit and happens often).
• Timestamps and clocks need to be syntonized/synchronized across device boundaries where only timestamps with well defined tolerances rather than precise cycle values are available.

Therefore I currently assume that using nanoseconds + timespec as the common denominator for network APIs is the right approach even in the long term. But I might be blind here because I read these standards all day - so what is your opinion about this? It's important because it influences the clock subsystem design quite a bit.

[keith-packard]

I have watched other projects adopt a general 64-bit nanosecond time mechanism with good results; it avoids numerous errors where scaling is performed incorrectly or lost entirely. Plus it encourages the use of 64-bit types instead of various kludges to try and fit times into 32-bit values. As long as the model exposes the resolution of every time source, it seems like the resulting programming model is generally easier to use correctly.

[andyross]

For clarity here: I'm not saying an API that exposes fine-grained monotonic time is a bad idea. I'm not saying it doesn't belong in Zephyr somewhere. I'm saying it's a poor fit for our hardware and that it almost certainly doesn't belong in the kernel as the "official" way we keep time.

Again, we spent years dealing with test code that would naively compute timeouts in milliseconds in an obvious way and then fall over mysteriously on Nordic because something happened too early or took too long. And the solution was to say "No, don't give me an integer. Instantiate a timeout once and the system will ensure it's an opaque struct so you don't shoot your foot trying to do more math on it."

If we could assume all systems had a MHz-scale timekeeping device, my answer would be different. But... we can't.

[andyross]

@andyross Do you propose to base TX/RX timestamps, timed TX and RX windows on opaque counter cycle values? I actually considered this, but so far I assumed that this would be a bad idea:

To be blunt: if you're basing it on something other than ticks, regardless of what your API says, you probably have bugs somewhere. The result of k_uptime_ticks() is the official kernel monotonic timestamp source.

[github-actions]

This pull request has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this pull request will automatically be closed in 14 days. Note, that you can always re-open a closed pull request at any time.