ePrivacy and GPDR Cookie Consent by Cookie Consent Skip to main content

Loader Kafka

Loader Kafka sends input data as messages to Kafka brokers.

The component can produce two types of message payloads

  • JSON message payloads 

  • Avro message payloads 

The component builds upon the Python package for Kafka client. Specifically C/C++ implementation of librdkafka and confluent-python module.

In terms of authentication and communication encryption the component supports the following modes of operation

  • PLAINTEXT: No authentication and no encryption

  • SSL: Authentication and broker communication encryption using SSL/TLS certificates 

  • SASL_SSL: Authentication using Kerberos (GSSAPI) with encryption using SSL/TLS certificates

  • SASL_PLAINTEXT: Authentication using Kerberos (GSSAPI) without communication encryption

Visit the relevant section of documentation as configuration parameters  might have specific meaning or behavior depending on message payload type or authentication scheme

Warning: The topic must exist prior to loading. The component does not create a target topic automatically.

Warning: The component only supports schema discovery and message production for Avro payloads.

Warning: Avro schemas are only supported for message values, not for message keys

Warning: The component was tested against the Confluent Kafka Schema Registry. Other registry implementations were not tested but may work.

Data In/Data Out

Data In

The component readssearches the configuration, loads the data from thefor input files andin thendirectory sends data/data/in/files to the Kafka server.

FilesThe that need to be uploadedfiles should be in thenewline-delimited folderJSON /data/in/files/folder_nameformat (*.ndjson). 

Each input JSON object found in an input file is translated into a single Kafka message.

The formatexpected structure of filesthe should be newline delimitedinput JSON files.object depends on the chosen message payload format.

For JSON message format the input JSON object is used simply as a message payload. 

For Avro message format the input JSON object must provide properties required by Avro schema. The final message is then created according to the discovered schema by populating it with values from the JSON object.
Data Out N/A

Learn more: about the folder structure here.

Parameters

Connection Settings


kafka.png

authentication
Bootstrap.Server

Server(s)

(required)

 Kafka's server

List address.of Kafka brokers the loader should attempt initial connection with

If multiple hosts are provided the loader will attempt to establish connection in the same order as brokers are specified.

Format: host:port

Example: kafka-broker1.meiro.io:9094
Path

Security protocol

(required)

 Path to

Authentication dataand files.encryption Example:protocol *.data/in/files/folder_name/file_name.ndjsonthe loader should use for communication with Kafka brokers

Possible values:

  • plaintext

  • ssl

  • sasl_ssl

  • sasl_plaintext
Topic (required)

Connection log level

 Name

Log level for diagnostic logging on Kafka connection. Value is passed to configuration property debug for librdkafka.

Possible options:

  • Default (normal level of topiclogging)

    in
    • Kafka
  • for which you want to assign loaded files.
Key

Security (required)

 Aand specificconnection keyinitiation namemessages)

will be
  • used

    All in(full thediagnostic datalogging, file. Thevery keygranular)

    will
    • be used in Kafka.


    Back to top