Skip to content

kafka python

Apache Kafka is an open-source streaming platform that is commonly used for building real-time data pipelines and streaming applications. If you want to work with Kafka in Python, you can use the confluent-kafka-python library. This library provides a Python client for Apache Kafka and allows you to produce and consume messages from Kafka topics.

Here are the steps to get started with Kafka in Python using the confluent-kafka-python library:

  1. Install the confluent-kafka-python library using pip:
pip install confluent-kafka
  1. Import the necessary modules in your Python script:
from confluent_kafka import Producer, Consumer
  1. Create a Kafka producer to produce messages:
# Define producer configuration
producer_config = {
    'bootstrap.servers': 'your_kafka_broker',  # Replace with your Kafka broker(s) address
    '': 'python-producer'

# Create a Kafka producer instance
producer = Producer(producer_config)

# Produce a message to a Kafka topic
topic = 'your_topic_name'
message = 'Hello, Kafka!'
producer.produce(topic, key=None, value=message)
  1. Create a Kafka consumer to consume messages:
# Define consumer configuration
consumer_config = {
    'bootstrap.servers': 'your_kafka_broker',  # Replace with your Kafka broker(s) address
    '': 'my-consumer-group',
    'auto.offset.reset': 'earliest'  # You can set the offset behavior as per your requirement

# Create a Kafka consumer instance
consumer = Consumer(consumer_config)

# Subscribe to a Kafka topic
topic = 'your_topic_name'

# Consume messages from the topic
while True:
    message = consumer.poll(1.0)  # Adjust the timeout as needed
    if message is None:
    if message.error():
        if message.error().code() == KafkaError._PARTITION_EOF:
            print("Reached end of partition")
            print(f"Error while consuming message: {message.error()}")
        print(f"Received message: {message.value().decode('utf-8')}")

# Close the consumer when done

Make sure to replace 'your_kafka_broker' with the address of your Kafka broker(s) and 'your_topic_name' with the name of the Kafka topic you want to produce to or consume from.

These are the basic steps to get started with Kafka in Python using the confluent-kafka-python library. You can further customize the configurations and error handling according to your application’s requirements.

Leave a Reply

Your email address will not be published. Required fields are marked *


Enjoy this blog? Please spread the word :)