Connector Kafka
The Kafka-to-CDP connector enables the loading of messages from Apache Kafka to Meiro CDP.
Learn more: about Apache Kafka here and here.
Requirements
In terms of authentication and communication encryption, the component supports the following modes of operation:
-
PLAINTEXT: No authentication and no encryption
-
SSL: Authentication and broker communication encryption using SSL/TLS certificates
Features
Only Avro messages payloads can be loaded from Apache Kafka to Meiro CDP.
Data In/Data Out
|
Data In
|
N/A
|
|
Data Out
|
N/A as messages taken from Kafka are processed and uploaded directly as events to CDP
|
Learn more: about folder structure please go to this article.
Parameters
Connection settings
To connect Kafka you need to know its Server and Security protocol.
|
Server(s)
(required)
|
List of Kafka brokers the connector should attempt initial connection with
If multiple hosts are provided the connector will attempt to establish a connection in the same order as brokers are specified.
Format: host:port
Example: kafka-broker1.meiro.io:9094
|
|
Security protocol
(required)
|
Authentication and encryption protocol the connector should use for communication with Kafka brokers
Possible values:
|
|
Connection log level
|
Log level for diagnostic logging on Kafka connection. Value is passed to configuration property debug for librdkafka.
Possible options:
-
Default (normal level of logging)
-
Security (authentication and connection initiation messages)
-
All (full diagnostic logging, very granular)
|
Security settings
|
Broker’s CA certificate
|
CA certificate string (PEM format) for verifying the broker's key.
Passing a directory value (for multiple CA certificates) is not supported.
Value are passed to ssl.ca.pem in librdkafka
|
|
Client’s private key
|
Client's private key string (PEM format) used for authentication.
Value is passed to ssl.key.pem in librdkafka
|
|
Client’s public key
|
Client's public key string (PEM format) used for authentication.
Values are passed to ssl.certificate.pem in librdkafka
|
Schema Registry
|
Schema Registry URL
|
Avro schema registry URL http(s)://kafka:8081
|
|
Schema Registry CA certificate
|
CA certificate string (PEM format) for verifying the Schema Registry's key.
Required when HTTPS is enabled.
|
|
Schema Registry client’s private key
|
Client's private key string (PEM format) used for authentication.
|
|
Schema Registry client’s public key
|
Client's public key string (PEM format) used for authentication.
|
Topics (list)
Define which topic you wish to upload to CDP.
|
Topic name
(required)
|
Name of the Kafka topic, where the messages will be consumed from.
The topic can be specified for each separate output event stream to CDP. If topics use different schemas, schemas will be dynamically discovered and used accordingly.
|
|
Event Time Path
(required)
|
JSON Path expression to search for in consumed Kafka messages to use as an CDP event timestamp (e.g. /changeTimestamp)
|
|
User Identifier Path
(required)
|
JSON Path expression to search for in consumed Kafka messages to use as an CDP event user identifier (e.g. /accountId)
|
|
Source ID
(required)
|
CDP Event Source ID corresponding to a Source defined in Meiro CDP.
|
|
Source Event type
(required)
|
For example: set, created, updated, moved, deleted, etc.
|
|
Event Version
(required)
|
CDP Event version corresponding to an Event Version defined in Meiro CDP.
|
Connection to CDP settings
|
CDP API URL
(required)
|
Typically it is an endpoint feed. For example, the absolute URL is https://instance_name.meiro.io/api
|
|
CDP API Credentials User Name
(required)
|
Account name
|
|
CDP API Credentials User Name Password
(required)
|
Account password
|
