Loader Kafka

Loader Kafka sends input data as messages to Kafka brokers.

The component can produce two types of message payloads

• JSON message payloads 

• Avro message payloads 

The component builds upon the Python package for Kafka client. Specifically C/C++ implementation of librdkafka and confluent-python module.

In terms of authentication and communication encryption the component supports the following modes of operation

• PLAINTEXT: No authentication and no encryption

• SSL: Authentication and broker communication encryption using SSL/TLS certificates 

• SASL_SSL: Authentication using Kerberos (GSSAPI) with encryption using SSL/TLS certificates

• SASL_PLAINTEXT: Authentication using Kerberos (GSSAPI) without communication encryption

Visit the relevant section of documentation as configuration parameters  might have specific meaning or behavior depending on message payload type or authentication scheme

Warning: Known Limitations:

The topic must exist prior to loading. The component does not create a target topic automatically.

The component only supports schema discovery and message production for Avro payloads.

Avro schemas are only supported for message values, not for message keys

The component was tested against the Confluent Kafka Schema Registry. Other registry implementations were not tested but may work.

Data In/Data Out

Data In	The component reads the configuration, loads the data from the input files and then sends data to the Kafka server. Files that need to be uploaded should be in the folder /data/in/files/folder_name The format of files should be newline delimited JSON files.
Data Out	N/A

Learn more: about the folder structure here.

Parameters

Bootstrap.Server (required)	Kafka's server address.
Path (required)	Path to data files. Example: *.data/in/files/folder_name/file_name.ndjson
Topic (required)	Name of topic in Kafka for which you want to assign loaded files.
Key (required)	A specific key name will be used in the data file. The key will be used in Kafka.