Apache TubeMQ (InLong) Adapter & Sink

[SteveYurongSu]

Purpose

TubeMQ support for StreamPipes.

Background

Apache TubeMQ (now is a submodule of Apache Inlong) is a trillion-records-scale distributed messaging queue (MQ) system, focuses on data transmission and storage under massive data. Compared to many open source MQ projects, TubeMQ has unique advantages in terms of stability, performance, and low cost.

It offers a variety of features:

• Pluggable transport protocols, such as TCP, SSL
• Support big-data and streaming ecosystem integration
• Message retroactivity by time or offset
• Efficient pull and push consumption model
• Flexible distributed scale-out deployment architecture
• Feature-rich administrative dashboard for configuration
• Authentication and authorization

For more information:

• TubeMQ VS Kafka

Approach

Connect Adapter:
Introduce the TubeMQProtocol for reading data from TubeMQ brokers.

Sink:
Introduce the TubeMQPublisherSink for writing data to TubeMQ brokers.