|
==History==
Kafka was originally developed at [[LinkedIn]], and was subsequently open sourced in early 2011. [[Jay Kreps]], [[Neha Narkhede]] and [[Jun Rao]] helped co-create Kafka.<ref name="ForbesKreps">Li, S. (2020). He Left His High-Paying Job At LinkedIn And Then Built A $4.5 Billion Business In A Niche You've Never Heard Of. Forbes. Retrieved 8 June 2021, from [https://www.forbes.com/sites/stevenli1/2020/05/11/confluent-jay-kreps-kafka-4-billion-2020/?sh=1a82e619709d Forbes_Kreps] {{Webarchive|url=https://web.archive.org/web/20230131210616/https://www.forbes.com/sites/stevenli1/2020/05/11/confluent-jay-kreps-kafka-4-billion-2020/?sh=1a82e619709d |date=2023-01-31 }}.</ref> Graduation from the Apache Incubator occurred on 23 October 2012.<ref>{{Cite web|title=Apache Incubator: Kafka Incubation Status|url=https://incubator.apache.org/projects/kafka.html|access-date=2022-10-17|archive-date=2022-10-17|archive-url=https://web.archive.org/web/20221017081525/https://incubator.apache.org/projects/kafka.html|url-status=live}}</ref> Jay Kreps chose to name the software after the author [[Franz Kafka]] because it is "a system optimized for writing", and he liked Kafka's work.<ref>{{cite book |last1=Narkhede |first1=Neha |last2=Shapira |first2=Gwen |last3=Palino |first3=Todd |title=Kafka: The Definitive Guide |date=2017 |publisher=O'Reilly |isbn=978-1-4919-3611-5 |url=https://books.google.com/books?id=dXwzDwAAQBAJ |chapter=Chapter 1 |quote=People often ask how Kafka got its name and if it has anything to do with the application itself. Jay Kreps offered the following insight: "I thought that since Kafka was a system optimized for writing using, a writer's name would make sense. I had taken a lot of lit classes in college and liked Franz Kafka."}}</ref>
== Release historyOperation ==
In contrast, Apache Kafka is a distributed log-based messaging system that guarantees ordering within individual partitions rather than across the entire topic. Unlike queue-based systems, Kafka retains messages in a durable, append-only log, allowing multiple consumers to read at different offsets. Kafka uses manual offset management, giving consumers control over retries and failure handling. If a consumer fails to process a message, it can delay committing the offset, preventing further progress in that partition while other partitions remain unaffected. This partition-based design enables fault isolation and parallel processing while allowing ordering to be maintained within partitions, depending on consumer handling.<ref>{{Cite book |last=Narkhede |first=Neha |title=Kafka: the definitive guide: real-time data and stream processing at scale |last2=Shapira |first2=Gwen |last3=Palino |first3=Todd |date=2017 |publisher=O'Reilly Media |isbn=978-1-4919-3616-0 |___location=Sebastopol, CA |oclc=933521388}}</ref> {{page needed|date=May 2025}}▼
In 2025, Apache Kafka introduced "Queues for Kafka" through KIP-932,<ref>[https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka KIP-932: Queues for Kafka]</ref> adding share groups as an alternative to consumer groups. This feature enables queue-like semantics where consumers can cooperatively process records from the same partitions, with individual message acknowledgment and delivery tracking. Unlike traditional consumer groups where partitions are exclusively assigned, share groups allow the number of consumers to exceed partition count, making it ideal for work-queue patterns while maintaining Kafka's durability and scalability benefits. This development addresses the common challenge of "over-partitioning" that many Kafka users face. {{cn|date=May 2025}}▼== Release history ==
{| class="wikitable sortable"
|-
! Version !! Release date !! Notable changes
|-
| 4.0.0 <ref name="kafka-downloads">{{cite web |url=https://kafka.apache.org/downloads |title=Apache Kafka Downloads |access-date=2025-05-14}}</ref>|| March 18, 2025 ||
* First major release operating entirely without Apache ZooKeeper (KRaft mode by default)
* General availability of new consumer group protocol (KIP-848) for improved rebalance performance
* Early access to Queues for Kafka (KIP-932)
* Updated Java requirements (Java 11 for clients/streams, Java 17 for brokers/connect/tools)
|-
| 3.8.1 || October 29, 2024 || Latest in 3.8 line, primarily bugfix release
|-
| 3.7.2 || December 13, 2024 || Bugfix and incremental improvements
|-
| 3.7.0 || February 27, 2024 || Significant new features and fixes
|-
| 3.6.2 || April 4, 2024 || Bugfix release
|-
| 3.5.0 || June 15, 2023 || Major new features and improvements
|-
| 3.4.1 || June 2023 || Bugfixes
|-
| 3.4.0 || February 2023 || New features
|-
| 3.3.1 || October 3, 2022 || Marked KRaft as production-ready, introduced KRaft-to-KRaft upgrades
|-
| 3.2.0 || May 17, 2022 || Replaced log4j 1.x with reload4j, new features for KRaft, error handling, and Kafka Streams enhancements
|-
| 3.1.0 || January 24, 2022 || Added Java 17 support, new metrics, and protocol improvements
|-
| 3.0.0 || September 21, 2021 || Major version milestone marking the start of the 3.x series
|-
| 2.3.0 || June 25, 2019 || Added incremental cooperative rebalancing, improvements to Connect REST API, new metrics, and enhanced error handling
|-
| 1.1.0 || March 28, 2018 || Introduced controller improvements, dynamic broker configs, delegation token authentication, and enhancements to Connect and Streams APIs
|}
== Comparison with Queue-Based Messaging Systems ==
[[Amazon Simple Queue Service|Amazon SQS]] FIFO and Azure Service Bus sessions are queue-based messaging systems that provide ordering guarantees within a message group or session attempt but do not necessarily guarantee ordered delivery in cases of retries or failures. In SQS FIFO, messages in the same message group are processed in order, with subsequent messages held until the preceding message is successfully processed or moved to the dead-letter queue (DLQ). Once a message is placed in the DLQ, it is no longer retried, creating a gap in the sequence. However, the remaining messages continue to be delivered in order.<ref>{{Cite web |title=FIFO queue delivery logic in Amazon SQS - Amazon Simple Queue Service |url=https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-understanding-logic.html |access-date=2025-03-22 |website=docs.aws.amazon.com}}</ref><ref>{{Cite web |title=Using dead-letter queues in Amazon SQS - Amazon Simple Queue Service |url=https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html |access-date=2025-03-22 |website=docs.aws.amazon.com}}</ref><ref>{{Cite web |title=Amazon SQS FIFO queues - Amazon Simple Queue Service |url=https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-fifo-queues.html |access-date=2025-03-22 |website=docs.aws.amazon.com}}</ref>
[[Microsoft Azure|Azure]] Service Bus sessions function similarly by maintaining ordering within a session, provided a single consumer processes messages sequentially. The implementation differs from SQS FIFO but follows the same fundamental ordering principle.<ref>{{Cite web |last=spelluru |date=2025-03-21 |title=Azure Service Bus message sessions - Azure Service Bus |url=https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-sessions |access-date=2025-03-22 |website=learn.microsoft.com |language=en-us}}</ref><ref>{{Cite web |last=spelluru |date=2025-02-07 |title=Service Bus dead-letter queues - Azure Service Bus |url=https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dead-letter-queues |access-date=2025-03-22 |website=learn.microsoft.com |language=en-us}}</ref>
▲In contrast, Apache Kafka is a distributed log-based messaging system that guarantees ordering within individual partitions rather than across the entire topic. Unlike queue-based systems, Kafka retains messages in a durable, append-only log, allowing multiple consumers to read at different offsets. Kafka uses manual offset management, giving consumers control over retries and failure handling. If a consumer fails to process a message, it can delay committing the offset, preventing further progress in that partition while other partitions remain unaffected. This partition-based design enables fault isolation and parallel processing while allowing ordering to be maintained within partitions, depending on consumer handling.<ref>{{Cite book |last=Narkhede |first=Neha |title=Kafka: the definitive guide: real-time data and stream processing at scale |last2=Shapira |first2=Gwen |last3=Palino |first3=Todd |date=2017 |publisher=O'Reilly Media |isbn=978-1-4919-3616-0 |___location=Sebastopol, CA |oclc=933521388}}</ref>
▲In 2025, Apache Kafka introduced "Queues for Kafka" through KIP-932,<ref>[https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka KIP-932: Queues for Kafka]</ref> adding share groups as an alternative to consumer groups. This feature enables queue-like semantics where consumers can cooperatively process records from the same partitions, with individual message acknowledgment and delivery tracking. Unlike traditional consumer groups where partitions are exclusively assigned, share groups allow the number of consumers to exceed partition count, making it ideal for work-queue patterns while maintaining Kafka's durability and scalability benefits. This development addresses the common challenge of "over-partitioning" that many Kafka users face.
==Kafka APIs==
{{Portal|Free and open-source software}}
*[[RabbitMQ]]
*[[Apache Pulsar]]
*[[Redis]]
*[[NATS Messaging|NATS]]
| 