6月13日是什么日子| 什么叫体位性低血压| h2o是什么意思| 桑螵蛸是什么| 嗜睡乏力没精神容易疲劳是什么原因| 教是什么生肖| 头皮发白是什么原因| 璇字五行属什么| 感冒流黄鼻涕吃什么药| 白芷是什么| 土中金是什么数字| 什么药治灰指甲最有效| 翌日是什么意思| 褪黑素什么时候吃| 央行行长什么级别| 月加亏念什么| 土豆可以做什么美食| 嘴涩是什么原因造成的| 拜你所赐什么意思| 嘘寒问暖是什么意思| 什么是伪娘| 4月15日是什么星座| 凝滞是什么意思| gg 是什么意思| 舒肝健胃丸治什么病| 胸部周围痒是什么原因| tct是什么检查| 数目是什么意思| 吃什么能降胆固醇| 更年期吃什么药调理| 甲状腺结节吃什么散结| 杨树林是什么牌子| 海娜是什么| hi是什么| 新生儿溶血是什么意思| 什么人适合喝蛋白粉| 黄色配什么颜色好看| 省人大代表是什么级别| 高血糖吃什么食物| 朱元璋属什么生肖| 国药准字h代表什么| 阿堵物是什么意思| 臀推是什么意思| 血糖高是什么原因造成的| 白带多是什么原因| 夏天做什么菜| 面条吃多了有什么坏处| 低gi什么意思| 2014年什么年| 前列腺在人体什么位置| 受凉胃疼吃什么药| 天津市市长什么级别| 降血糖吃什么| pr是什么| 直肠增生性的息肉是什么意思| 什么是低保户| 1927年属什么生肖| 后背疼什么原因| 女孩学什么专业好就业| egc是什么意思| 后背疼是什么病| 射手座是什么象星座| 白食是什么意思| 九个月的宝宝吃什么辅食食谱| 妇科假丝酵母菌是什么病| 耳鸣用什么药治疗效果最好| 身份证最后一位代表什么| 腱鞘炎去医院挂什么科| 前列腺炎是什么意思| 微创人流和无痛人流有什么区别| 80岁称什么之年| 阿华田是什么饮料| 哎是什么意思| 虎都男装属于什么档次| 脸部肌肉跳动是什么原因| 胸有成竹是什么生肖| cock什么意思| 三维彩超主要检查什么| 7岁属什么| 71年属什么| 什么微风| 双重所有格是什么意思| 马拉松pb是什么意思| 白蛋白偏高是什么意思| 什么茶最好喝| 县尉相当于现在什么官| 什么是植发| 羊球是什么| 一个草字头一个见念什么| 屁股疼是什么原因引起的| 什么的礼物| 什么泡水喝可降血压| 看淡是什么意思| 裸婚是什么意思| hpv是一种什么病| 大头菜又叫什么菜| 压疮用什么药最快能好| 什么拉车连蹦带跳歇后语| 忘带洗面奶用什么代替| 温水煮青蛙什么意思| 3月23是什么星座| 低血压是什么意思| hmo是什么| 什么情况下会得甲亢| esr是什么| 枸杞不能和什么一起吃| 腰间盘膨出吃什么药效果好| 吃什么最养胃修复胃| 玄冥是什么意思| 故的偏旁是什么| 富字五行属什么| 洗衣粉和洗衣液有什么区别| 什么牌子的氨基酸洗面奶好| 膘是什么意思| 小孩缺锌有什么症状| 副脾是什么意思| 六月19是什么日子| 一什么狮子| 4090是什么意思| 晚上睡觉口苦是什么原因| 什么王| 益母草什么时候喝最好| 十八层地狱分别叫什么| 突然呕吐是什么原因| 早餐吃什么营养又健康| 为什么痛经| 吴用属什么生肖| 985大学什么意思| 海关锁是什么意思| 疱疹感染是什么病| 梦见着火是什么意思| 长期尿黄可能是什么病| 什么是孝| 病案首页是什么| 牛杂是牛的什么部位| 女人为什么会得霉菌| 肺积水是什么原因引起的| 什么的围巾| 人有三急指的是什么| 减肥期间能吃什么水果| 一什么瓜地| ushi是什么品牌男装| 埃及的母亲河是什么| 经期量少吃什么来排血| 消化功能紊乱吃什么药| 非农业户口是什么意思| 孕妇吃什么长胎不长肉| 李果是什么水果| 5月7日什么星座| 什么是三有保护动物| 二龙戏珠是什么意思| 走水是什么意思| 医生为什么喜欢开地塞米松| 本能反应是什么意思| 嘴唇边缘发黑是什么原因| 秦始皇的真名叫什么| 1328年属什么生肖| 腰椎退行性变是什么意思| prn医学上是什么意思| 炼乳可以做什么美食| 茜字五行属什么| 喝水牙疼是什么原因| 瓠子和什么相克| 急性肠胃炎用什么药| 舌苔厚白腻是什么原因引起的| 包皮长挂什么科| 阻生智齿是什么意思| 尿素氮是什么意思| 云南雪燕有什么作用| 庆生是什么意思| 风象星座是什么意思| 翡翠是什么颜色| 肾虚是什么意思| 颈动脉彩超查什么| 小孩腰疼是什么原因引起的| 落叶像什么| 喝酒喝吐了用什么缓解| 戌时右眼跳是什么预兆| 泡脚用什么东西泡最好| bull是什么意思| 9.22什么星座| 贤者模式是什么意思| 什么叫宿根太阳花| 为什么会有颈纹| 痔疮吃什么水果| 南瓜为什么叫南瓜| 常打嗝是什么原因| 心动过速吃什么药| 腹部疼痛挂什么科| 什么蘑菇有毒| 放疗是什么| 子弟是什么意思| 寄托是什么意思| 什么花能吃| dell是什么牌子的电脑| 劲酒兑什么饮料好喝| 近视是什么意思| 早唐筛查是检查什么| 什么是星座| 吃什么能快速降血压| 11月份什么星座| 动人是什么意思| 一什么花瓶| 脂肪肝是什么| 不值一提是什么意思| 翻白眼是什么意思| 南京有什么玩的| 对宫星座是什么意思| 腮边长痘是什么原因| 梦见自己会飞是什么意思| 千米的字母是什么| 驾驶证b2能开什么车| 补充微量元素吃什么| 笃怎么读什么意思| 烧火棍是什么意思| 为什么有白头发| 尿蛋白阳性是什么意思| 乳房痛挂什么科| 肠胃痉挛吃什么药| 肝损伤吃什么药| 吃什么盐比较好有利于健康| 宫颈多发纳囊是什么病| 重中之重是什么意思| 如如不动是什么意思| 辣的部首是什么| 三点水一个四读什么| 今天中国什么节日| 经常腰酸是什么原因女性| bcm是什么意思| 十万为什么| 为什么空调外机会滴水| 叙字五行属什么| 喉咙红肿吃什么药| 寻常疣用什么药膏除根| 无患子为什么叫鬼见愁| 神经官能症挂什么科| 血糖高吃什么主食| gl是什么| 什么样的人能镇住凶宅| 摆地摊卖什么最赚钱而且很受欢迎| 祭日和忌日是什么意思| 微博id是什么| pussy 什么意思| 推荐是什么意思| 因子是什么| 灵柩是什么意思| 高温丝假发是什么材质| 心血管堵塞吃什么好| 吃什么药能来月经| 猫咪泪痕重是什么原因| 吃什么降血脂和胆固醇| 虹膜是什么意思| 牙龈疼是什么问题| 什么的街道| 美国的国鸟是什么| marlboro是什么烟| hy什么意思| 三文鱼又叫什么鱼| 虚热吃什么药| 兔爷是什么意思| cmr医学中什么意思| 阴囊瘙痒挂什么科室| 手发麻什么原因| 阴蒂在什么位置| 百度
DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Attention Kubernetes users! DZone is researching how K8s usage is evolving in 2025 — tell us the good, the bad, + the frustrating.

Getting Started With DevSecOps: This Refcard provides actionable insights for embedding security into your DevOps culture and workflows.

Data Engineering: Not your average "AI is changing everything" report. Dive into tooling, strategy, + systems in scalable data ecosystems.

Java Caching: This Refcard explores core caching concepts and demonstrates how to implement them using Java’s JCache API.

Related

  • Designing Retry-Resilient Fare Pipelines With Idempotent Event Handling
  • Event-Driven Microservices: How Kafka and RabbitMQ Power Scalable Systems
  • System Coexistence: Bridging Legacy and Modern Architecture
  • Event-Driven Architectures: Designing Scalable and Resilient Cloud Solutions

Trending

  • A Complete Guide to Creating Vector Embeddings for Your Entire Codebase
  • Revolutionizing the Command Line: An Introduction to Gemini CLI
  • LangGraph Orchestrator Agents: Streamlining AI Workflow Automation
  • How to Improve Software Architecture in a Cloud Environment
  1. DZone
  2. Data Engineering
  3. Big Data
  4. Publish-Subscribe Model in Kafka

正风肃纪 优化政治生态(在习近平新时代中国特色社会主义思想指引下·代表委员畅谈抓落实)

In this post, learn about how to integrate synchronous and asynchronous systems via Kafka's publish-subscribe mechanism.

By 
Felipe Fernández user avatar
Felipe Fernández
·
May. 27, 16 · Tutorial
Likes (4)
Comment
Save
Tweet
Share
20.0K Views
百度 此前蔡慧康因为要迎接自己第二个孩子的到来,是不辞辛苦从南宁赶回到上海,随后他又从上海赶回到了南宁,昨日已经出现在了训练场,这等精神真是让人感动,作为目前国足阵营里少有的中场铁腰,他是国足腰杆子能否硬起来的关键。

Join the DZone community and get the full member experience.

Join For Free

This is the fourth post in the series about integrating sync clients with async systems (1, 2, 3). Here we'll try to understand how Kafka works in order to correctly leverage its publish-subscribe implementation.

Kafka Concepts

According to the official documentation:

Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.

Kafka runs as a cluster and the nodes are called brokers. Brokers can be leaders or replicas to provide high-availability and fault tolerance. Brokers are in charge of partitions, being the distribution unit where messages are stored. Those messages are ordered and they're accessible by an index called offset. A set of partitions forms a topic, being a feed of messages. A partition can have different consumers, and they access to the messages using its own offset. Producers publish messages into Kafka topics. This diagram from Kafka's documentation could help to understand this:

Kafka's topic

Queuing vs publish-subscribe

Consumer groups is another key concept and helps to explain why Kafka is more flexible and powerful than other messaging solutions like RabbitMQ. Consumers are associated to consumer groups. If every consumer belongs to the same consumer group, the topic's messages will be evenly load balanced between consumers; that's called a 'queuing model'. By contrast, if every consumer belongs to a different consumer group, all the messages will be consumed in every client; that's called a 'publish-subscribe' model.

You can have a mix of both approaches, having different logical consumer groups, for different needs, and several consumers inside of every group to increase throughput through parallelism. Again, another diagram from Kafka's documentation:

Kafka's consumer groups

Understanding Our Needs

Law enforcement architecture

As we saw in previous posts (1, 2, 3) the Items service publishes messages into a Kafka topic called item_deleted. This message will live in one partition of the topic. To define in which partition the message will live, Kafka provides three alternatives:

  • If a partition is specified in the record, use it
  • If no partition is specified but a key is present choose a partition based on a hash of the key
  • If no partition or key is present choose a partition in a round-robin fashion

We'll use item_id as a key. Consumers contained in different instances of the Law Enforcement service are only interested in particular partitions, as they're keeping internal state for some items. Let's inspect different Kafka consumer implementations to see which is the most convenient for our use case.

Kafka Consumers

There are three consumers in Kafka: High-level consumer, Simple Consumer and New Consumer

Out of the three consumers, Simple Consumer operates at the lowest level. It meets our requirements as allows the consumer to "consume only a subset of the partitions in a topic in a process". However, as the documentation says:

The SimpleConsumer does require a significant amount of work not needed in the Consumer Groups:

  • You must keep track of the offsets in your application to know where you left off consuming
  • You must figure out which Broker is the lead Broker for a topic and partition
  • You must handle Broker leader changes

If you read the code suggested for handling those concerns, you'll be quickly discouraged to use this consumer.

New Consumer offers the right level of abstraction and allows us to subscribe to specific partitions. They suggest the following use case in the documentation:

The first case is if the process is maintaining some kind of local state associated with that partition (like a local on-disk key-value store) and hence it should only get records for the partition it is maintaining on disk.

Unfortunately our system is using Kafka 0.8, and this consumer is only available from 0.9. We don't have the resources to migrate to that version, so we'll need to stick with the High-level consumer.

That consumer offers a nice API, but it doesn't allow us to subscribe to specific partitions. That means that every instance of the Law Enforcement service will consume every message, even those that are not pertinent. We can achieve that by defining different consumer groups per instance.

Leveraging Akka Event Bus

In the previous post we have defined some Finite State Machine actor that is waiting for ItemDeleted messages.

  when(Active) {
    case Event(ItemDeleted(item), currentItemsToBeDeleted@ItemsToBeDeleted(items)) =>
      val newItemsToBeDeleted = items.filterNot(_ == item)
      newItemsToBeDeleted.size match {
        case 0 => finishWorkWith(CensorResult(Right()))
        case _ => stay using currentItemsToBeDeleted.copy(items = newItemsToBeDeleted)
      }
  }

Our Kafka Consumer could forward every message to those actors and let them to discard/filter irrelevant items. However we don't want to overwhelm our actors with redundant and inefficient work, so we'll add a layer of abstraction that will let them discard the proper messages in a really efficient way.

final case class MsgEnvelope(partitionKey: String, payload: ItemDeleted)

class ItemDeletedBus extends EventBus with LookupClassification {
  override type Event = MsgEnvelope
  override type Classifier = String
  override type Subscriber = ActorRef

  override protected def mapSize(): Int = 128

  override protected def publish(event: Event, subscriber: Subscriber): Unit = subscriber ! event.payload

  override protected def classify(event: Event): Classifier = event.partitionKey

  override protected def compareSubscribers(a: Subscriber, b: Subscriber): Int = a.compareTo(b)
}

Akka Event Bus offers us subscription by partition, which we're missing in our Kafka High-Level Consumer. From our Kafka Consumer we'll publish every message into the bus:

itemDeletedBus.publish(MsgEnvelope(item.partitionKey, ItemDeleted(item)))

In the previous post we showed how to subscribe to messages using that partition key:

itemDeletedBus.subscribe(self, item.partitionKey)

LookupClassification will filter unwanted messages, so our actors won't be overloaded.

Summary

Thanks to the flexibility that Kafka provides, we were able to design our system understanding different trade-offs. In the next posts we'll see how to coordinate the outcome of those FSMs to provide a sync response to the client.

kafka

Published at DZone with permission of Felipe Fernández. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Designing Retry-Resilient Fare Pipelines With Idempotent Event Handling
  • Event-Driven Microservices: How Kafka and RabbitMQ Power Scalable Systems
  • System Coexistence: Bridging Legacy and Modern Architecture
  • Event-Driven Architectures: Designing Scalable and Resilient Cloud Solutions

Partner Resources

×

Comments

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • [email protected]

Let's be friends:

ch是什么牌子 什么是翻墙软件 人情是什么意思 五月十三号是什么日子 随诊什么意思
充电玩手机有什么危害 内膜是什么 鹿茸是什么 什么是什么意思 6月29号是什么星座
你是什么动物 尿糖1十是什么意思 阿司匹林主治什么病 97年属什么今年多大 e-mail什么意思
lmp是什么意思 什么饺子馅好吃 为什么男人吃石榴壮阳 基础代谢率是什么意思 补气血什么季节补最好
1992年出生的是什么命hcv8jop2ns3r.cn 夏祺是什么意思hcv7jop6ns7r.cn 下巴有痣代表什么hcv8jop6ns3r.cn 支抗钉是什么hcv8jop0ns0r.cn 金丝檀木是什么木头hcv7jop6ns8r.cn
三位一体是什么生肖0735v.com 什么食物含维生素c最多hcv8jop1ns3r.cn 血压偏高是什么原因weuuu.com 脑卒中什么意思hcv7jop5ns0r.cn 西汉后面是什么朝代hcv9jop7ns3r.cn
白鳍豚用什么呼吸hcv8jop2ns6r.cn 大咖什么意思gysmod.com 松子是什么树的果实imcecn.com 口干舌燥喝水也不解渴是什么原因hcv9jop5ns7r.cn 泡热水脚有什么好处hcv8jop9ns1r.cn
元老是什么意思hcv8jop1ns6r.cn 6月23号什么星座hcv9jop8ns3r.cn 极有家是什么意思hcv8jop7ns4r.cn 舌裂是什么原因造成的hcv9jop0ns2r.cn 缺二氧化碳是什么症状baiqunet.com
百度