Idempotent Publishing Support: Client Infrastructure

[jsquire]

Summary

When publishing events to Event Hubs, timeouts or other transient failures may introduce ambiguity into the understanding of whether a batch of events was received by the service. Because Event Hubs has an at-least-once guarantee for delivery and consumers are strongly encouraged to be idempotent in their processing, the common approach is to resend any batch where the status of receipt was unknown.

In some specialized scenarios, producers have a need to make efforts to avoid publishing duplicate events. To support these scenarios, the Event Hubs service is adding support for annotating messages with metadata to indicate the sequence in which events were intended to be published and to use that as an indicator of which events were already received by a the target partition. This functionality comes at a performance cost, however, and requires producers to follow a strict set of semantics to allow the service to perform server-side deduplication based on the intent of the producer.

Scope of Work

• Implement the design for idempotent publishing.

Success Criteria

• The design has been implemented; developers can opt-into the idempotent publishing, the producer client tracks the appropriate state.

• The tests necessary for its validation have been created or adjusted and pass reliably.

• The existing test suite continues to produce deterministic results and pass reliably.

Out of Scope

• Service integration; this is covered under a dedicated issue.

Related Issues and References

• Idempotent Publishing Support: Service Integration (#14710)
• Design Proposal for Idempotent Producer