OVERVIEW
Apache Kafka is a distributed data store optimized for ingesting and processing streaming data in real-time. Kafka provides three main functions to its users:
Publish and subscribe to streams of records
Effectively store streams of records in the order in which records were generated
Process streams of records in real time
Kafka is primarily used to build real-time streaming data pipelines and applications that adapt to the data streams. It combines messaging, storage, and stream processing to allow storage and analysis of both historical and real-time data. It uses a partitioned log model, which combines messaging queue and publish subscribe approaches. It uses binary protocol over TCP.
GOALS
One-click launch
Easy configurable AMI with pre-installed Apache Kafka
INSTALLATION VIA AWS MARKETPLACE
Kindly click on the below link to install the server via AWS Marketplace:
Pending..
GET STARTED
Apache Kafka provides:
Allows data to be distributed across multiple servers, making it scalable beyond what would fit on a single server.
Kafka decouples data streams so there is very low latency, making it extremely fast
Partitions are distributed and replicated across many servers, and the data is all written to disk.
Topics are automatically replicated, but the user can manually configure topics to not be replicated
Step 1. Enable Ports on Instance
Enable the ports 2181 and 9092 port on the instance
Step 2. Login via Kafka user
Login with kafka user using this command
su kafka
Enter Instance ID as password
password- <aws_instance_id>
Step3. Start the Apache Kafka Server
Perform the below steps to enable the Apache Kafka on the server,
sudo systemctl start zookeeper.service
sudo systemctl start kafka.service
Open new terminal and run the below command to process the producer in Apache Kafka,
~/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic testTopic
~/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic testTopic
Open another terminal on the same path and run the below command to process the consumer in Apache Kafka
~/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic testTopic --from-beginning
Comments