current position:Home>Message queue Kafka "retrieval component" is heavily online!

Message queue Kafka "retrieval component" is heavily online!

2022-05-07 15:14:31Alibaba cloud native

author :Kafka&Tablestore The team


When still used for message queuing , Can't efficiently check repeated and failed messages ?

When still used for message queuing , Can't locate the problem and the content accurately ?


Message queue Kafka「 Retrieve components 」 To help you ~

This paper focuses on message queue Kafka「 Retrieve components 」 Detailed introduction , Firstly, it introduces the pain points in the use of message queue , Then put forward corresponding solutions to the pain point problem , And interpret the key technologies , Designed to help you understand the message queue Kafka「 Retrieve components 」 More familiar with the characteristics and use of , In order to help you more effectively solve the pain points encountered in the process of message troubleshooting .

Introduction to pain points

In the use of message queue , The default in the industry is to assume that after the message enters the message queue , The news is reliable , The probability of loss is also low . But in practical application, we will face a variety of problems :

Pain points in application

  • Due to the characteristics of distributed system , Message failure 、 Repetition is inevitable , For failed and repeated troubleshooting , It usually depends on the log of the client to deduce , But if it's huge , It will also be very difficult for customers to do this manually , This will challenge the reliability of the message ;

  • Besides , Larger projects are usually completed by multiple people or teams , The code implementation methods of message sending and consumption are also different , This will challenge whether the message will succeed in completing its mission ;

  • In addition to the troubleshooting of the problem results , Will the message not meet expectations when it is generated ? This is also one of the difficulties perplexing customers . From the current system of message queue , There is no way to check by content , As a result, it is difficult to check the correctness of the business .

Simply speaking , In the message field, each message can represent specific meaning and action , In the event of failure 、 Loss and error , Under the current situation of message queuing in the industry , It's hard to check specific problems , This will lead to the problem of locating the whole upstream and downstream links .

Pain points faced by the technical side

These are the problems that customers will face in the scenario of message application . Problems based on application scenarios , There will also be many pain points on the technical side , When handling message troubleshooting :

  • First, we need to invest in R & D code 、 Deployment and operation and maintenance , At the same time, the operation and maintenance personnel need to be familiar with Kafka Use , You need to use Kafka The client carries out consumer consumption , Then confirm the message by traversal , To confirm the existence of the message ;

  • In addition to the code investment required for research and development 、 Deployment and operation and maintenance , Other products may need to be introduced , Such as docking flow calculation , Through flow calculation, traverse messages, etc .

What's more troublesome is , At present, this kind of investigation is often very frequent , Often in weeks 、 Even in days , Will make R & D 、 High time cost for deployment and operation and maintenance ; At the same time, the meta information to be confirmed is different every time , It will lead to large investment , And the flexibility is not high .

In conclusion , When troubleshooting problems such as failure and repetition during the use of message queue , First, there is no better tool and way to complete the content retrieval , It is difficult to check , Accuracy and ease of use are insufficient ; Second, it needs to invest high time and labor cost , Large investment and inflexible . These problems will bring many troubles to users when troubleshooting message problems .

Kafka Retrieval component introduction

Through the introduction of the above pain points, we can see , At present, in the field of information , There are a lot of pain points in message troubleshooting , To solve the above problems , Alicloud message queue Kafka Version heavy launch message retrieval component . The content of the component is described in detail below :

Retrieve component Introduction

Message queue Kafka「 Retrieve components 」 It's a full trust 、 High elasticity 、 Interactive retrieval component , It has the second level response capability of trillion level message content retrieval .

  • It is mainly for troubleshooting and recovery scenarios of operation and maintenance personnel , Used for message related full link message troubleshooting , Including the sending of messages 、 Repeated production and missing verification ; The main functions include supporting messages by Topic Partition 、 Site range and time range retrieval , At the same time, it supports by message Key and Value Keyword search, etc ;

  • It is mainly used to solve the problem that message products in the industry do not support retrieving message content .

Message queue Kafka「 Message retrieval 」 With the help of Kafka Connect Function and table storage (Tablestore) Realization , adopt Connector Yes Topic Dump messages in , Then send it to the data table in the table storage , Finally, the ability of message retrieval is provided through the table storage index function .

Its core is to provide a complete message content retrieval capability , Can quickly locate problems , At the same time, it is convenient to operate 、 Save manpower ; When users use , After completing the message queue Kafka After instance creation , It takes only five simple steps to achieve Kafka Retrieve the application of the component :


The following is a brief introduction to message queuing Kafka The operation steps of version message retrieval are introduced .

Retrieval component operation introduction

1) Open message retrieval

First, open an instance Topic Message retrieval function , So that it can be adjusted as needed Topic To retrieve messages in . Steps are as follows :

  • Log in to message queuing Kafka Version of the console ;
  • In the resource distribution area of the overview page , Choose the region ;
  • On the left navigation bar , Click message retrieval ;
  • On the message retrieval page , Select the instance to be retrieved from the drop-down list of the selected instance Topic The instance to which the message belongs , Then click open message retrieval ;
  • Open the message retrieval panel , Fill in the opening parameters , Then click OK .


2) Test sending messages

After opening message retrieval , You can send messages to the message queue Kafka Version of the data source Topic Send a message , This is used to schedule tasks and test whether message retrieval is successfully created .

  • On the message retrieval page , Find the target to test Topic, Operate at the corresponding position according to the task status ;
  • Send a test message in the quick experience messaging panel .


3) Search message

  • On the message retrieval page , Find the target Topic, In its action column , Click search ;
  • In the search panel , Set search criteria , Select the search term to be added from the search term drop-down list , Click add search term , Add a search term and set the search information in the value column , Then click ok .



4) View message retrieval task details

  • After opening message retrieval , You can view the automatically created Topic、Group、 Table storage instance name 、 The table stores detailed information such as data table and table name , You can also directly enter the table in the details to store the data table ;
  • On the message retrieval page , Find the target Topic, In its action column , Click Details ;
  • On the task details page, you can view the target Topic Details of related message retrieval ; It can also be in the target service column in the basic information area , Click table storage , You can enter the data sheet details page to view .


5) View consumption details

Support to view the current subscription Topic On line Group stay Topic Consumption progress of each partition , Understand the consumption and accumulation of messages .

  • On the message retrieval page , Find the goal that needs to check the consumption progress Topic, In its action column , Click consumption progress ;
  • Here's the picture , On the consumption details page , You can see Topic Consumption by district :


In addition to the above functions , When running the message retrieval function , You can also pause the message retrieval task 、 Enable the message retrieval task and delete the message retrieval task .

Kafka Retrieval component technology interpretation

Previous message queuing Kafka The message retrieval method of version only supports searching according to the consumption location or creation time , rely on Kafka The system itself cannot well support the user's demand for retrieving messages through keywords .

In order to solve this problem better ,Kafka And Tablestore Strong alliance , take Kafka Message through Connector Import Tablestore In the data sheet of , utilize Tablestore The ability to achieve keyword retrieval .


The key technologies are explained below :

Kafka Connect

Kafka Connect The core of is to solve the synchronization problem of heterogeneous data . The solution is to add a layer of message middleware between various data sources , All data is stored and distributed through message oriented middleware .


The benefits of doing so are the following two points :

1) Do asynchronous decoupling through message oriented middleware , All systems only communicate with message oriented middleware ;

2) Number of parsing tools to be developed , Also from the original n Square one , Become linear 2*n individual ;Kafka Connect It is used to connect the message system and data source , According to the flow direction of data , Connections can be divided into Source Connector and Sink Connector.

The principle is very simple ,Source Connector Responsible for analyzing source data , Messages converted to standard format , adopt Kafka Producer Send to Kafka Broker in . Empathy ,Sink Connector Through Kafka Consumer Consumption corresponds to Topic, Then deliver it to the target system . In the whole process ,Kafka Connect Unified solution of task scheduling 、 Interact with the messaging system 、 Automatic volume expansion and shrinkage 、 Fault tolerance and monitoring , Greatly reduce duplication of labor .

Message queue Kafka Version provides full hosting 、 Free of operation and maintenance Kafka Connect, For message queuing Kafka Data synchronization between version and other Alibaba cloud services . As shown in the figure below , You can see the message queue Kafka Version supports table storage Tablestore、Mysql Source Connector、OSS Sink Connector、MaxCompute Sink Connector as well as FC Sink Connector And other mainstream Connector. If users want to use these Connector Data synchronization , Only in message queues Kafka Make several configurations on the graphical interface of the console , You can pull it up with one key Connector Mission .


Form storage Tablestore

Form storage Tablestore It is a massive structured data storage service built on Alibaba cloud Feitian distributed system . Based on Feitian Pangu distributed file system as storage base , Using storage computing separation architecture , Elastic shared resource pool design , Implemented a cloud native Serverless Storage products . Built in distributed index system , It can automatically expand the computing resources needed to build the index according to the write traffic , Support extremely high write traffic . At the same time, the index structure is optimized , It can support faster fuzzy query . Memory computing separation architecture 、 High throughput, real-time indexing and other key capabilities make Tablestore Able to support Kafka Writing and efficient search of China shipping volume data , Help quickly and effectively retrieve the required information .


Technology leadership

Kafka+Kafka Connect+ Form storage Tablestore Cloud native data application solution , adopt Kafka Connect As a trigger for real-time processing tasks , It can receive new data sent to the message queue cluster in real time , Then forward to the table store Tablestore.

As a link in the subsequent data flow ,Kafka Connect In addition to ensuring the real-time performance of data , It also solves the problem of task scheduling 、 Interact with the messaging system 、 Automatic volume expansion and shrinkage 、 Fault tolerance and monitoring , Greatly reduce duplication of labor . The data is stored in the table Tablestore in the future , With the help of distributed storage of table storage and powerful index engine , Able to support PB Levels of storage 、 Ten million TPS And millisecond delay , At the same time, it supports full hosting 、 High elasticity 、 Interactive retrieval component , Thus, the trillion level message content retrieval capability of second level response can be realized .

User oriented values and advantages

Kafka The retrieval component function not only has strong technical advantages , At the same time, it can also bring more convenience to the actual work of users :

1、 The cost of troubleshooting is low

It only needs a simple configuration of the console to achieve Kafka View all messages in the server cluster ;

2、 Fast troubleshooting

No development 、 Free of resource assessment 、 Avoid deployment 、 No operation and maintenance ; As long as the search conditions are established , The second level query response can be realized ;

3、 High troubleshooting accuracy

The retrieval component function is provided by the message commercialization team and Tablestore The core R & D team jointly builds , Relying on Alibaba cloud's native capabilities , High retrieval accuracy , Reliability and availability can be well guaranteed .

In conclusion ,Kafka The retrieval component function has the following advantages in practical business :

  • Fast location problem , It can realize the failure of upstream and downstream products 、 Abnormally fast recovery , Reduce business asset losses ;
  • Save enterprise cost , Reduce operation and maintenance 、 R & D and other personnel investment ;
  • Reduce learning costs , The understanding mechanism requirements for message products are reduced .



Alicloud message queue Kafka「 Retrieve components 」 It is the first component in the field of message queuing to support interactive message content retrieval , Development free 、 No operation and maintenance 、 The characteristics of high elasticity . For in the message domain 、 For heavy users , Alicloud message queue Kafka「 Retrieve components 」 Is the existence of daily troubleshooting messages & The sharp weapon of correctness .


Click on here , Go to the relevant product documentation for details !

copyright notice
author[Alibaba cloud native],Please bring the original link to reprint, thank you.

Random recommended