KAFKA

Topic:  A particular stream of data. It is identified by its name.

Topics are split in partitions. Each message within a partion gets an incremental id (offset).

Data is kept only for a limited time (one week by default)

Written data is immutable.

Brokers: 

A kafka cluster is composed of mutliple brokers (servers)

Each broker contains certain topic partitions.

 

Topic Replication:

Each partition has one leader and multiple ISR (in sync replica). The leader and ISR is dertermined by the keeper.

 

Producer:

Producers writes data to topics.  Producers can choose to recieve acknowledgment  of data writes. (acks=0 wont wait, acks=1 by default will wait for broker ack, acks=all leader and all replicas ack)

• A producer can choose to send a key with the message. A key is used if need message ordering for a specific field.

 

Consumers:

Consumers read data from a topic identified by its name.

 

Consumer offset:

Kafka stores the offsets at which the groupe of consumer has been reading. 

The offsets commited live in kafka topic named __consumer_offsets

 

Zookeeper:

• it manages brokers
• It helps performing leader election for partions.
• It sends notifications to kafka in case of changes.

Start Zookeeper:

$zookeeper-server-start  config/zookeeper.properties

Start kafka server:

$kafka-server-start config/server.properties

Create a topic:

$kafka-topics --zookeeper 127.0.01:2181 -topic first_topic --create --partitions 3 --replication-factor 1
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic first_topic.

List topics:

$kafka-topics --zookeeper 127.0.01:2181 --list
first_topic

Describe topic:

$kafka-topics --zookeeper 127.0.01:2181 --topic first_topic --describe
Topic: first_topic	PartitionCount: 3	ReplicationFactor: 1	Configs: 
	Topic: first_topic	Partition: 0	Leader: 0	Replicas: 0	Isr: 0
	Topic: first_topic	Partition: 1	Leader: 0	Replicas: 0	Isr: 0
	Topic: first_topic	Partition: 2	Leader: 0	Replicas: 0	Isr: 0

Create second_topic and delete it:

$kafka-topics --zookeeper 127.0.01:2181 --topic second_topic --delete                                
Topic second_topic is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.

Create a producer:

$kafka-console-producer --broker-list 127.0.0.1:9092 --topic first_topic
>Hello Meriem
>awsome course!
>learning kafka
>another message

Create a consumer:

 $kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic first_topic 
hello there

Getting all the messages:

 $kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic first_topic --from-beginning
awsome course!
just for fun
learning kafka
some message that is acked
hi how are
Hello Meriem
another message
hello there