Confluent + Kinetica

A high-throughput, low-latency platform for handling real-time data feeds with Kinetica

Confluent is the company behind the open-source streaming platform, Apache Kafka. Kafka provides a unified, high-throughput, low-latency platform for handling real-time data feeds. Ingest streaming data from sensors, mobile apps, connected devices, and social media into Kinetica, combine it with data at rest, and analyze it in real time to improve customer experience, deliver targeted marketing offers, and for operational efficiencies.

Certified Kafka Connector

Kinetica’s certified Kafka connector makes it easy to stream data directly between Kafka and Kinetica. Organizations can ingest real-time data streams from Apache Kafka and provide a means for analysis and immediate action on incoming data within Kinetica.

Kafka architecture

The Kafka connector has been certified by Confluent and has met criteria for code development best practices, schema registry integration, security, and documentation.

The connector contains two classes that integrate Kinetica database with Kafka:

  • KineticaSourceConnector A Kafka Source Connector, which receives a data stream from the Kinetica database via table monitor. Data is streamed in flat Kafka Connect “Struct” format with one field for each table column. A separate Kafka topic is created for each database table configured.
  • KineticaSinkConnector A Kafka Sink Connector, which receives a data stream from a Kafka Source Connector and writes it to the Kinetica database. Streamed data must be in a flat Kafka Connect “Struct” that uses only supported data types for fields (BYTES, FLOAT64, FLOAT32, INT32, INT64, and STRING). No translation is performed on the data and it is streamed directly into a table. The target table and collection will be created if they do not exist.

The Kinetica Connector can be deployed into any Confluent cluster from the Control Center GUI or command line using the Kafka Connect RESTful API. The Kafka Connect API ensures fault tolerant integration between the Kafka topic stream and the Kinetica instance. For example, retailers can use the Kinetica connector to capture real-time, streaming geospatial data from shopper’s mobile phones as Kafka streams, combine it with customer loyalty data in Kinetica, and push out targeted, personalized, location-based offers through mobile apps.

Combined with the Confluent enterprise-grade streaming data platform, this powerful solution will help you capitalize on streaming data to power real-time decision making and drive your business results.

Resources

Recently in the Blog

In the press...