BEST

The Complete Guide to Spark Machine Learning - Part 1

Name: The Complete Guide to Spark Machine Learning - Part 1
Price: 99000 KRW

If you want to be recognized as a machine learning expert based on large-scale data, from understanding the core framework of Spark machine learning, to SQL-based data processing through difficult practical problems, to data analysis through business domain analysis, and to the ability to implement optimized machine learning models, please join this course.

(4.9) 25 reviews

916 students

dooleyz3525

Apache Spark

Machine Learning(ML)

Big Data

Data Engineering

This course is prepared for Intermediate Learners.

What you will learn!

Implementing Machine Learning Models in Spark
Detailed understanding of DataFrame, the foundation of Spark's data processing
Understand the various technical elements that make up the Spark Machine Learning Framework
Mastering Spark's Machine Learning Pipeline
Ability to use SQL for data analysis
SQL-based Feature Engineering Techniques
Implementing models with XGBoost and LightGBM in Spark
Model hyperparameter tuning method based on Bayesian optimization
Improve your data analysis and ML model implementation skills simultaneously through challenging real-world problems.
Data analysis method based on analysis domain
Various data visualization techniques

Data analysis + feature engineering + ML implementation,
Grab three competencies at once.

With Apache Spark
Machine learning meets.

Apache Spark, the leader in open source large-scale distributed processing solutions, has met with Machine Learning .

Many large domestic companies and financial institutions are using Apache Spark to analyze large amounts of data and create machine learning models. Since Spark is based on a distributed data processing framework, it can process large amounts of data and create ML models by expanding capacity on a few to dozens of servers . Therefore, it can overcome the limitations of scikit-learn, which can only implement machine learning models on a single server.

Also good at data processing/analysis
As a machine learning expert
I will help you grow.

The 'Spark Machine Learning Complete Guide - Part 1' course will help you grow into a machine learning expert who is skilled in data processing and analysis beyond learning how to implement machine learning models in Spark.

In order to grow into a true machine learning expert, it is very important not only to have ML implementation skills, but also to have the ability to process and combine business data to create ML models. To this end, you will learn how to process data using SQL, which is most commonly used in large-scale data processing in practice, and data analysis techniques based on business domain analysis through practice.

_{It is designed to help you develop data processing/analysis and ML implementation capabilities through detailed theoretical explanations and practical training.}

The problems you will face
We will solve it for you.

Implementing machine learning models on Spark is not easy. This is because it faces many problems that existing data scientists or machine learning experts have not experienced, such as unique machine learning APIs and frameworks based on the Spark architecture, and data processing based on SQL.

In this course, The Complete Guide to Spark Machine Learning, we will develop your ability to solve the problems you encounter .

The first half of the lecture 'Spark Machine Learning Complete Guide - Part 1'

The first half of the lecture consists of detailed theoretical explanations and abundant practical exercises on various elements that make up the Spark Machine Learning Framework, such as DataFrame, SQL, Estimator, Transformer, Pipeline, and Evaluator. Through this, you will be able to implement ML models in Spark easily and quickly .

We will also go into detail about how to use XGBoost and LightGB in Spark, and how to tune hyperparameters using HyperOpt based on Bayesian optimization.

The second half of the lecture 'Spark Machine Learning Complete Guide - Part 1'

The second half of the lecture will improve your real-world data processing/analysis skills and machine learning model implementation skills at the same time through hands-on practice on Kaggle's Instacart Market Basket Analysis competition . The Kaggle Instacart competition is a difficult competition, and the data set consists of e-commerce order processing tables (products, orders, and order products).

Through this data set, you will learn in detail how to process and analyze business data based on SQL, perform feature engineering, how to derive analysis domains from business, and how to create models based on the derived features.

This is Part 1 of the 'Spark Machine Learning Complete Guide' course that is being released this time. Part 2 of the course will be released later, and will cover text analysis, recommendations, and time series analysis.

💻 Please check before taking the class!

All of the practical codes in this lecture are based on Python. Scala is not covered, so please refer to this before selecting a lecture.

The practice environment
Please check.

The hands-on training uses Databricks. Databricks provides a notebook environment that allows you to create Spark-based applications on the cloud without installing Spark.

Databricks is officially available for free use for 14 days as a Community version.
And in the video lecture ' Managing Spark Clusters on Databricks and Using Databricks Even After 2 Weeks of Signing Up ' in Section 0, I explain how you can continue to use it for free after 14 days, so please watch that video carefully (for explanation about the Databricks Community version, please refer to the link ).

You can download the lecture practice code and lecture explanation materials from ‘Download the practice code and explanation materials.’

Player knowledge
This is a required course.

This course is designed assuming that students have knowledge of Chapter 5 (Regression) of the Complete Guide to Python Machine Learning or equivalent, and that they have a very basic understanding of SQL . Please refer to the above when selecting a course.

It would be helpful to know the basics of Spark, but even if you don't, you will have no problem following the lecture.

Please check out the player lecture!

The Complete Guide to Python Machine Learning

Stop teaching theory-based machine learning.
From core machine learning concepts to practical skills, easily and accurately.

_{Are you curious about the interview with the knowledge sharer? (Click)}

Recommended for
these people!

Who is this course right for?

Anyone who wants to implement machine learning using Spark
Those who want to implement machine learning based on large-scale data
Anyone who wants to improve their data processing techniques for machine learning using SQL
Anyone who wants to learn the entire process of processing data into the desired format and creating an ML model based on it in practice
Anyone who wants to improve data analysis, feature engineering capabilities, and ML implementation

Need to know before starting?

Understanding up to Chapter 5 (Regression) of the Complete Guide to Python Machine Learning or equivalent prior knowledge
Understanding SQL Basics

Hello
This is dooleyz3525

24,943

Students

1,160

Reviews

3,916

Answers

4.9

Rating

Courses

(전) 엔코아 컨설팅

(전) 한국 오라클

AI 프리랜서 컨설턴트

파이썬 머신러닝 완벽 가이드 저자

Curriculum

All

117 lectures ∙ (24hr 27min)

Course Materials:

Lecture resources

Section 1. Course Introduction and Practice Environment Setup

6 lectures ∙ (50min)

Section 2. Spark Overview

4 lectures ∙ (52min)

7. Distributed Data Architecture and Spark Overview
15:14
8. Spark Architecture and RDD Overview
12:43
9. Spark's RDD, DataFrame, SQL Comparison
11:48
10. Spark Machine Learning Overview
12:28

Section 3. Understanding Spark DataFrame - 01

10 lectures ∙ (2hr 11min)

Section 4. Understanding Spark DataFrame - 02

9 lectures ∙ (1hr 57min)

Section 5. Spark SQL Overview

4 lectures ∙ (49min)

Section 6. Understanding Spark Machine Learning - 01

14 lectures ∙ (3hr 7min)

Section 7. Understanding Spark Machine Learning - 02

5 lectures ∙ (1hr 22min)

Section 8. Spark ML Classification - 01

12 lectures ∙ (2hr 9min)

Section 9. Spark ML - Classification - 02

7 lectures ∙ (1hr 23min)

Section 10. Spark ML Regression

4 lectures ∙ (44min)

Section 11. Understanding Data Analysis for Practical Machine Learning

6 lectures ∙ (1hr 9min)

Section 12. Practical Machine Learning - Kaggle Instacart Market Basket Analysis Competition Overview

5 lectures ∙ (1hr 1min)

Section 13. Practical Machine Learning - SQL-based EDA Analysis of Kaggle Instacart Data

13 lectures ∙ (3hr 4min)

Section 14. Practical Machine Learning - Kaggle Instacart Feature Engineering, Model Training, Evaluation, Tuning - 01

12 lectures ∙ (2hr 23min)

Section 15. Practical Machine Learning - Kaggle Instacart Feature Engineering, Model Training, Evaluation, Tuning - 02

6 lectures ∙ (1hr 19min)

Published: 12/08/2021

Last updated: 05/13/2025

Reviews

All

25 reviews

4.9

25 reviews

gomjong
Reviews 8
∙
Average Rating 4.9
01/14/2022
5
100% enrolled
Thanks to you, I learned about Spark and gained confidence in Kaggle challenges. Thank you!
freedom07
Reviews 7
∙
Average Rating 5.0
02/04/2022
5
93% enrolled
I first got to know Professor Kwon Chul-min through the Complete Guide to Python Machine Learning. Thanks to that lecture, I, a non-major, was able to not give up on this field that I had been thinking of giving up on. I am currently working in this field and studying steadily by taking Infraon lectures. I wanted to thank the teacher, so I first thanked the teacher in the Q&A session, and the teacher encouraged me that if I continued to study, I would be able to achieve what I had worked for. I plan to continue to listen to the teacher's lectures in the future. ^^ㅎㅎ He really teaches so well. Professor Kwon Chul-min, I would like to take this opportunity to sincerely thank you.
- dooleyz3525
  Instructor
  02/04/2022
  I am even more impressed that you left such a touching review. I think I should be the one to thank you for the writing that instantly rewards the hard work you put into creating the lecture. If you continue to work hard like this, you will definitely achieve everything you want. Thank you.
indizz4933
Reviews 1
∙
Average Rating 5.0
01/03/2022
5
100% enrolled
Thank you for explaining it step by step.
yhzzz123
Reviews 12
∙
Average Rating 5.0
02/07/2022
5
54% enrolled
I am a student who has been attending Kwon Chul-min's lecture series! Thank you for continuing to provide high-quality lectures! And I have seen several Spark lectures in Scala and Java, but this is the first time I have seen a lecture that teaches Spark in Python, so I think it was even better! Although I have not completed the course yet, I still like how he tries to teach simple grammar as easily as possible! And I also like how he provides various practice materials to encourage repeated mastery! I look forward to other lectures in the future!
egs41
Reviews 53
∙
Average Rating 5.0
02/09/2022
5
10% enrolled
It was good to focus on the instructor's diction and voice, and the content was solid. Please continue to make good lectures. Thank you.

dooleyz3525's other courses

Check out other courses by the instructor!

Deep Learning CNN Complete Guide - Pytorch Version

dooleyz3525

$84.70

Basic / Deep Learning(DL), PyTorch, CNN, Kaggle, Computer Vision(CV)

5.0

(7)

200+

Update

Active Replies

From core theories of deep learning and CNN to various CNN model implementation methods, and practical deep learning development know-how through real-world problems, If you want to become a deep learning CNN technology expert based on Pytorch, join us in this lecture :)

Basic

Deep Learning(DL), PyTorch, CNN

[Revised Edition] The Complete Guide to Python Machine Learning

dooleyz3525

$77.00

Basic / Python, Machine Learning(ML), Statistics

4.9

(394)

8,000+

Update

Active Replies

We will help you easily understand the core concepts of machine learning and acquire the ability to implement practical machine learning applications by moving away from theory-based machine learning courses.

Basic

Python, Machine Learning(ML), Statistics

Data Analysis SQL Fundamentals

dooleyz3525

$42.90

Basic / SQL, PostgreSQL, DBMS/RDBMS

4.9

(130)

2,400+

Update

Active Replies

Through detailed lectures and hands-on practice on the core elements of SQL, we will provide you with a solid foundation to grow into a SQL analysis expert.

Basic

SQL, PostgreSQL, DBMS/RDBMS

[Revised Edition] A Complete Guide to Deep Learning Computer Vision

dooleyz3525

$93.50

Intermediate / Machine Learning(ML), Tensorflow, Python, Deep Learning(DL), Keras, Computer Vision(CV)

4.9

(145)

3,700+

Update

Active Replies

This course will help you become a deep learning-based computer vision expert who is needed in the field by providing a deep theoretical explanation of object detection and segmentation and practical examples that can be used immediately in the field.

Intermediate

Machine Learning(ML), Tensorflow, Python

Kafka Complete Guide - Core

dooleyz3525

$77.00

Intermediate / Kafka, Data Engineering

4.9

(75)

1,600+

Update

Active Replies

From the core of Kafka to in-depth content on internal mechanisms, the course is structured so that even beginners can quickly reach the expert level through detailed theoretical explanations, hands-on practice, and practical Kafka application development practice.

Intermediate

Kafka, Data Engineering

Oracle Performance Analysis and Instance Tuning Core Guide

dooleyz3525

$59.40

Intermediate / Oracle, DBMS/RDBMS

4.8

(98)

1,200+

Update

Active Replies

It provides a key guide to understanding the internal mechanisms of Oracle DB architecture and growing as a performance tuning and performance analysis expert.

Intermediate

Oracle, DBMS/RDBMS

FastAPI Complete Guide

dooleyz3525

$59.40

Intermediate / Python, FastAPI, SQL, SQLAlchemy

4.8

(24)

500+

Update

Active Replies

This course is designed to help you learn the core functions of FastAPI, as well as the entire process of web service development. Through this course, we will help you become a FastAPI expert developer needed in the field.

Intermediate

Python, FastAPI, SQL

SQL data analysis learned through various cases

dooleyz3525

$68.20

Intermediate / SQL, PostgreSQL, DBMS/RDBMS, Performance Marketing, Data Engineering

5.0

(40)

1,500+

Update

Active Replies

By implementing various practical data analysis cases using SQL, you can simultaneously improve your data analysis and SQL utilization skills.

Intermediate

SQL, PostgreSQL, DBMS/RDBMS

A Complete Guide to Deep Learning CNN - TensorFlow Keras Version

dooleyz3525

$84.70

Basic / Deep Learning(DL), CNN, Tensorflow, Keras, Kaggle, Machine Learning(ML)

5.0

(114)

2,000+

Update

Active Replies

From core theories of Deep Learning and CNN to implementation methods of various CNN models, and practical Deep Learning development know-how through real-world problems, If you want to become a Deep Learning CNN technology expert, join us in this lecture :)

Basic

Deep Learning(DL), CNN, Tensorflow

Kafka Complete Guide - Connect

dooleyz3525

$77.00

Intermediate / Kafka, Data Engineering

4.9

(23)

600+

Update

Active Replies

Through in-depth theoretical explanations of Kafka Connect and detailed practical training that can be used immediately in the field, we will help you grow into an expert in building data linkage and data pipelines based on Kafka Connect that are needed in the field.

Intermediate

Kafka, Data Engineering

Kafka Complete Guide - ksqlDB

dooleyz3525

$77.00

Intermediate / Kafka, ksqlDB, Data Engineering

4.8

(16)

300+

Update

Active Replies

This course is designed to help you learn the use of ksqlDB and its core mechanisms through various hands-on exercises. After completing the course, you will be able to easily and quickly build a real-time streaming data analysis system based on Kafka.

Intermediate

Kafka, ksqlDB, Data Engineering

Kaggle Advanced Machine Learning Practical Crash Course

dooleyz3525

$51.70

Intermediate / Machine Learning(ML), Kaggle

4.9

(69)

1,400+

Update

Active Replies

This course is designed to upgrade your skills as a practical machine learning development expert by implementing the machine learning model of the Home Credit Default Risk competition on Kaggle.

Intermediate

Machine Learning(ML), Kaggle

Similar courses

Explore other courses in the same field!

Learning by doing: Practical Spark Part 1

nexthumans

$77.00

Basic / Apache Spark, Big Data, Machine Learning(ML), data-transformation

Update

By the end of this course, you will be able to implement Apache Spark projects in your organization.

Basic

Apache Spark, Big Data, Machine Learning(ML)

Kafka Complete Guide - Core

dooleyz3525

$77.00

Intermediate / Kafka, Data Engineering

4.9

(75)

1,600+

Update

Active Replies

Intermediate

Kafka, Data Engineering

mongoDB from basics to practice (feat. Node.js)

sihoon

$59.40

16%

$49.50

Basic / MongoDB, REST API, Node.js, AWS, JavaScript, DBMS/RDBMS, Data Engineering

4.7

(86)

1,300+

Update

mongoDB, NoSQL You hear a lot these days, but is it still an unfamiliar database? Aren't you using it like a relational database (RDS/SQL)? No matter how good the technology is, if it's not used correctly, it will have the opposite effect. That's why we often see cases of failure in using MongoDB. This lecture will teach you everything from basic concepts to practical know-how so that you can use MongoDB as MongoDB-like as possible.

Basic

MongoDB, REST API, Node.js

Database - SQL

kjlee

$42.90

Basic / SQL, Data Engineering

4.8

(59)

700+

Update

Active Replies

This course is for those who are learning database programming for the first time or those who have some knowledge but want to learn systematically. From concepts to practical exercises, you will master the database query language by organizing examples of types that frequently occur in real life.

Basic

SQL, Data Engineering

[Management Course #3] DE, DBA (SSIS, SSAS, MachineLearning, BI, ETL)

vmproductor0202

$42.90

Basic / Big Data, ssis, ssas, Machine Learning(ML), etl

5.0

(2)

Update

Active Replies

SSIS, SSAS, MachineLearning, BI, ETL. You can learn important technologies that cannot be found in domestic books, YouTube, lectures, blogs, and academies. I also recommend it to those who are interested in employment at domestic large companies, American large companies, and American state funding agencies.

Basic

Big Data, ssis, ssas

Reinforcement learning made easy with R

coco

$42.90

Intermediate / Machine Learning(ML), R, Reinforcement Learning(RL)

4.3

(3)

We will learn about Q-learning and Deep Q-learning, and have time to implement reinforcement learning in R. We will cover the entire reinforcement learning content, from Deep Q-network to Self-imitation learning and Random Network Distillation.

Intermediate

Machine Learning(ML), R, Reinforcement Learning(RL)

Machine Learning That Even My Mom Can Do (Basic Practice)

$23.10

23%

$17.60

Beginner / EDA, Machine Learning(ML)

Update

The first step in learning AI: We've created the perfect curriculum for AI beginners! Second, match the theories and code you've learned! Now it's time for the real world! This is the stage to solidify the foundation and confidence in your knowledge by unraveling the theories you've learned into code.

Beginner

EDA, Machine Learning(ML)

Big Data Analytics Professional Practical Exam R All-in-One: Complete Preparation in 3 Weeks

masocampus

$77.00

30%

$53.90

Basic / Engineer Big Data Analysis, Big Data, R, Machine Learning(ML)

5.0

(2)

New

Active Replies

Ace the Big Data Analysis Engineer certification, a must-have credential that's far from easy to self-study, in a short amount of time by mastering the data analysis tool R, known for its ease of use!

Basic

Engineer Big Data Analysis, Big Data, R

MariaDB client development, HeidiSQL

swa

$8.80

Intermediate / SQL, MariaDB, HeidiSQL, DBMS/RDBMS, Data Engineering

4.9

(29)

1,100+

Learn how to write SQL queries and work with databases using the MariaDB client development tool HeidiSQL.

Intermediate

SQL, MariaDB, HeidiSQL

[Tensorflow2] Complete conquest of Python machine learning - Marathon record prediction project

nomad

$26.40

Intermediate / Tensorflow, Machine Learning(ML), Keras, Computer Vision(CV)

4.1

(25)

300+

This is a comprehensive machine learning project course that learns various useful machine learning regression and classification projects along with theory using Python and TensorFlow 2 based on Boston Marathon big data.

Intermediate

Tensorflow, Machine Learning(ML), Keras

SQL Basics from a Silicon Valley Data Leader

keeyonghan9539

$55.00

Basic / SQL, Data literacy, Data Engineering, Big Data, DBMS/RDBMS, duckdb

5.0

(2)

Update

If you work with data, SQL is a fundamental skill you must know. In this course, we will learn SQL from a data analysis perspective, focusing on hands-on practice. The practice will be conducted in Google Colab using DuckDB.

Basic

SQL, Data literacy, Data Engineering

Logpresso Enterprise - Dashboard

logpresso

$26.40

Basic / Big Data

5.0

(7)

200+

This course covers how to use queries to build dashboards in Logpresso. We will discuss how to visualize real-time data, trigger alerts, and dynamically drill down and filter as needed.

Basic

Big Data

Linear Algebra Leading to Machine Learning/Deep Learning

dlbro

$42.90

Beginner / Linear Algebra, Machine Learning(ML), Deep Learning(DL)

4.5

(41)

600+

Update

Covers linear algebra, which is essential for machine learning/deep learning research.

Beginner

Linear Algebra, Machine Learning(ML), Deep Learning(DL)

Resignation Time if You Don't Know? The Standard Guide to Data Engineering

altoformula

$84.70

24%

$63.80

Basic / Big Data, Data Engineering, Architecture

5.0

(2)

100+

Update

Uncover true value in the age of data! 📊 Designing applications focused on data has now become essential. Develop the insight and skills your company wants with the latest trends and practical case studies. Start now with efficient data processing and design secrets! Your next step, leap into the data-driven world!

Basic

Big Data, Data Engineering, Architecture

[2025] Yellow Book with 176 Problems: SQLD Solutions for Those Who Find It Difficult

datacodelab

$30.80

14%

$26.40

Basic / SQL, SQLD, Big Data, Oracle, MSSQL

5.0

(1)

Update

SQLD Yellow Book 176 Practice Problem Course for those who studied but can't solve the problems. After completing the course, you will become an expert beyond just passing. SQLD All-in-One Pass!

Basic

SQL, SQLD, Big Data

Time series analysis and forecasting

루비네코딩

$34.10

Intermediate / Probability and Statistics, Python, Big Data, AI, arima, kalman-filter

4.5

(2)

Update

Time series lecture to become a data analysis expert! Let's become an analyst strong in theory and practice!

Intermediate

Probability and Statistics, Python, Big Data

Java Machine Learning Weka Beginner

javaraml

Free

Basic / Java, Machine Learning(ML), Weka

4.9

(19)

1,900+

There is no need to ignore or fear machine learning. Machine learning is easy, but let's think about where to apply machine learning. The direction of machine learning expansion in Java is expected to be similar to the current Python, R strength, and future coexistence of Python, R, and Java, like iOs. The purpose of this lecture is to build a DIKW collaboration system between domain experts and IT professionals.

Basic

Java, Machine Learning(ML), Weka

ICBM technology of the 4th industrial revolution that anyone can easily understand

joshuachun

$68.20

Beginner / IoT, cloud-computing, Machine Learning(ML), Autonomous Driving, 5g

4.5

(2)

We will learn the core technologies of the 4th industrial revolution, such as ICBM (IoT, Cloud, Big Data, ML) and related technologies such as self-driving cars and metaverse, from the basics to develop the ability to prepare for future technological changes.

Beginner

IoT, cloud-computing, Machine Learning(ML)

Complete Guide to Unity Machine Learning Agents (Basics)

kyushik

$51.70

Basic / Reinforcement Learning(RL), Machine Learning(ML), Unity, Unity ML-Agents

4.2

(20)

400+

Update

Through this course, students will learn various reinforcement learning theories and implement them themselves, as well as create a reinforcement learning environment to test the reinforcement learning algorithm implemented using Unity Machine Learning Agents.

Basic

Reinforcement Learning(RL), Machine Learning(ML), Unity

AI Web Programming (Streamlit)

bestdjango

$34.10

Beginner / streamlit, Machine Learning(ML), Django, Seaborn, jupyter-notebook

5.0

(1)

Create AI data analysis and modeling results into a web program.

Beginner

streamlit, Machine Learning(ML), Django

$77.00

The Complete Guide to Spark Machine Learning - Part 1

This course is prepared for Intermediate Learners.

What you will learn!

With Apache Spark Machine learning meets.

Also good at data processing/analysis As a machine learning expert I will help you grow.

The problems you will face We will solve it for you.

The first half of the lecture 'Spark Machine Learning Complete Guide - Part 1'

The second half of the lecture 'Spark Machine Learning Complete Guide - Part 1'

The practice environment Please check.

Player knowledge This is a required course.

Please check out the player lecture!

The Complete Guide to Python Machine Learning

Recommended for these people!

HelloThis is dooleyz3525

Curriculum

Reviews

dooleyz3525's other courses

Similar courses

With Apache Spark
Machine learning meets.

Also good at data processing/analysis
As a machine learning expert
I will help you grow.

The problems you will face
We will solve it for you.

The practice environment
Please check.

Player knowledge
This is a required course.

Recommended for
these people!

Hello
This is dooleyz3525