Kafka Complete Guide - ksqlDB
This course is designed to help you learn the use of ksqlDB and its core mechanisms through various hands-on exercises. After completing the course, you will be able to easily and quickly build a real-time streaming data analysis system based on Kafka.
This course is prepared for Intermediate Learners.
What you will learn!
From KSQLDB's basic concepts to advanced architecture
Difference between Stream and Table, and Stateful Streaming Processing Mechanism
Information on the creation and management of KSQLDB's main objects, understanding of various data types
RocksDB operation mechanism in KSQLDB
Understanding various query syntax and functions of KSQLDB
Understanding and using Group by and Mview, and its specificity and limitations
Understanding and utilizing various types of KSQLDB joins, as well as their specificities and limitations
Understanding of various types of Windows and the operation mechanism of Time-based Window Aggregation and Window Join
Using Connect in KSQLDB
Integration of KSQLDB and Elasticsearch and visualization of analysis results through Kibana
A large-scale real-time streaming analytics system,
Easy and powerful with Kafka + ksqlDB!
Real-time streaming data analysis
Want to achieve efficiency and scalability?
If you are using Kafka , the easiest and fastest way to implement a large-scale real-time streaming analytics system is to use ksqlDB .
ksqlDB, which is installed and operated integrated with Kafka, can easily process/transform/analyze real-time streaming data with just a few lines of SQL code without using complex streaming APIs.
Excerpt from the official Confluent page (link)
Leading domestic and international companies are already facing the need to analyze streaming data in real time with incomparable large volumes and fast latency compared to the past and to immediately reflect the results, and are actively introducing ksqlDB to this end.
In the past, the complex Kafka Streams API was used to process/transform/analyze real-time streaming data based on Kafka, but now ksqlDB, which allows you to build a streaming analysis system easily and quickly with just simple queries, is becoming the trend for building real-time streaming data analysis systems.
ksqlDB is growing fast, and there is a shortage of experts
ksqlDB is rapidly replacing the existing Kafka Streams API due to its many advantages, including its easy and convenient SQL-based implementation.
However, it is very difficult to find field personnel with the skills to use ksqlDB in practical work . This is because ksqlDB is a solution that has emerged relatively recently, and most of the materials and lectures covering ksqlDB are composed of superficial concepts, which are insufficient for building the skills required in practical work.
Real-time, high-volume streaming data processing applications made easy and fast.
'Kafka Complete Guide - ksqlDB' is
This course is designed to be a practical course that will help you grow into a ksqlDB expert. Our goal is to help you grow into a ksqlDB expert that companies want .
✅ Those who are repeatedly blocked by the high wall of ksqlDB
✅ Anyone who wants to understand the core mechanism of ksqlDB
✅ Those who want to immediately use ksqlDB for work
Accordingly, this lecture is filled with the contents that students must acquire in order to utilize ksqlDB in practical work.
Unique features of this course
Check it out.
Learn the core operation mechanisms of key ksqlDB components through detailed visual aids and hands-on exercises.
ksqlDB has some similarities to general RDBMS, but there are many differences. Therefore, in order to handle ksqlDB well, you must have a detailed understanding of the operation mechanisms of the main components, such as Stream, Table, Query, Mview, and RocksDB. This lecture will help you learn the core mechanisms of ksqlDB through detailed visual materials and practice.
Various functions unique to ksqlDB, joins, Group by, Window utilization, and practical training for practical use
In order to effectively utilize ksqlDB in practice, you must be familiar with the various functions provided by ksqlDB, such as Join, Group by, and Window usage. In particular, ksqlDB's Join, Group by, and Window have different restrictions than SQL, and you cannot properly utilize ksqlDB without understanding them. In this lecture, you will be able to clearly understand the difference by directly performing these elements through many practical exercises.
In addition, through the separately provided 'Online Shoe Shop' practical training section, we will guide you to a level where you can efficiently apply various real-time analyses in ksqlDB in practice.
Using Connect in ksqlDB, and then linking with Elasticsearch and visualizing with Kibana
We will explain how to integrate and utilize Connect in ksqlDB. You will also learn through practice how to save the analysis results of ksqlDB to Elasticsearch through Connect and visualize them through Kibana.
We will guide you from beginner to expert level in ksqlDB.
⚙️
Core mechanisms of ksqlDB's main components
🔎
Differences between Stream and Table and how to use them, creation and management of major objects
🧰
Understanding various query syntax and main functions of ksqlDB through practice
📊
Understanding and utilizing ksqlDB's unique Group by, MView, and Join, differences and limitations from RDBMS
🔐
Understanding of various types of Windows and practicing the operation mechanism of Time-based Window Aggregation and Window Join
💾
Integration with Connect and collection and visualization of analysis results through Elasticsearch and Kibana
ksqlDB We have put a lot of effort into covering the core content and usage methods that cannot be found in any lectures or online materials so far, rather than just skimming the surface. We have also filled the curriculum with various practical classes so that you can learn the theory more naturally through practice.
After completing this course, you will find yourself becoming a ksqlDB expert who can compete with anyone else.
We've organized the content so that you can fully understand it with detailed explanations and various practical exercises.
(We provide students with a PDF of lecture materials of more than 100 pages.)
💡 Please note before taking the class!
- This lecture only covers ksqlDB and does not cover Kafka Streams.
- Virtualbox VM compatibility practice may be difficult on Macbook M1/M2 model PCs.
Practice environment
Server OS
We use Ubuntu Linux 20.04 on Oracle VirtualBox VM as the Kafka server OS. Although it uses Linux, it is run on a virtual machine basis, so it can be configured on both Windows/macOS environments.
VirtualBox can be installed in almost all Windows/macOS environments. However, in the case of Mac, VirtualBox is not installed in the latest M1 model, so you must install Ubuntu using a virtual environment such as UTM. For M1 models, please make sure that Ubuntu is installed in a virtual environment before selecting a lecture.
Confluent Kafka
Community Edition
Kafka uses Confluent Kafka Community Edition version 7.1.2, not Apache Kafka.
Confluent is a company founded by the core people who created Kafka, and provides enterprise Kafka that is more advanced in terms of performance and convenience for corporate customers. It is 100% compatible with Apache Kafka, but you can use more diverse Kafka modules and integrated binaries. Use the powerful distributed system Kafka in a more elastic and scalable form with Confluent. It will help you reduce the burden of infrastructure construction and maintenance, and help you develop faster.
Recommended PC Specifications
A full lab environment configuration may require a PC environment with 20-30 GB of storage capacity and 4 GB or more of RAM .
Check out the Q&A 💬
Q. Should I take the previous lecture, Kafka Complete Guide - Core or Connect?
Two sections of this lecture cover the integration of ksqlDB and Connect. Even if you have not taken the Complete Guide to Kafka - Connect, you should have a basic understanding of Connect and practical experience to understand the practical exercises in this section.
Understanding Kafka Core is essential. It is recommended that you take the previous lecture, Kafka Complete Guide - Core. However, even if you did not take the lecture, if you have experience using Kafka's basic Broker, Producer, and Consumer and have a good grasp of the core concepts, you can sufficiently take this lecture.
Q. Do I need to have RDBMS SQL experience to take this course?
Many of the exercises in this course are query-based, so you should have experience using basic RDBMS SQL syntax and Group by and Join.
Recommended for
these people!
Who is this course right for?
Anyone who wants to understand the main components of KSQLDB easily and deeply
Data engineer who wants to quickly and effectively build a large-scale real-time streaming data processing/transformation analysis system based on Kafka
Analysts and data scientists who need to leverage real-time streaming data analytics
Developers who want to migrate applications from existing Producer/Consumer-based or Kafka Streams-based to KSQLDB-based
Need to know before starting?
(It would be best if you took the Kafka Complete Guide - Core, but if not) you need a solid basic knowledge of topics/producers/consumers.
Basic knowledge of Kafka Connect
Many of the exercises are query-based. Basic SQL knowledge is required to understand joins and Group By.
Hello
This is 권 철민
Students
23,091
Reviews
1,060
Rating
4.9
Courses
12
(전) 엔코아 컨설팅
(전) 한국 오라클
AI 프리랜서 컨설턴트
파이썬 머신러닝 완벽 가이드 저자
Curriculum
All
139 lectures ∙ (21hr 31min)
are provided.