Real-time data distribution made easy with efficient use of new edge tooling

by Vishal Joshi | on

Historically, real-time data delivery was looked at as moving the metaphorical elephant through the pipe, with levels of support at both the ports of shipping and landing. The challenges involved included setting up the infrastructure to send and receive data and ensuring the security and performance on both sides. This was achieved by either having low-level network connectivity or through a publish/subscribe messaging queue with virtual private networks between sender and receiver.

SmartStream made this complex problem easy…

The answer lies in the efficient use of tooling available in Kafka, REST APIs, Keycloak and AWS ecosystems. Your application publishes the message on a Kafka topic, and your consumer on the other side of this topic converts the message into Flux (Reactor-Kafka).

Here on your Kafka, maintain the last read by for each client in the form of Kafka offset. This guarantees no message loss in the event that your client application had a blip. For access security, suggest making use of Keycloak basic HTTP authentication on your side.

Now comes the light-weight mechanism of delivering messages with an added layer of security. On your side, a simple controller implementation with REST API end point allows client applications to subscribe to that end point. To make this end point behave as a SSE (Server-Side Event), you simply need to define the Content-Type as event/text-stream.

Remember, we do not want to expose our private servers. Neither do we want to set up virtual private networks or let our clients worry about setting up heavy queuing infrastructure. For that purpose, our recommendation is to use AWS ALB (Application Load Balancer) exposed as HTTPS endpoint. This acts as the interface between the client application and you. The only task for your client is to subscribe to your end point over HTTPS while you publish messages to them real-time.

Voila! By abstracting the complexity from your clients point of view, you just delivered high performance, real-time data with multiple layers of security.

2021-08-02 Topic: RDU Real Time Distribution Diagram