Member-only story
Setup a Streaming Data Pipeline With Apache Kafka on GCP
Hi All,
Today , We will go through a hands on labs on GCP. We will Setup a Streaming Data Pipeline With Apache Kafka on Confluent hosted in GCP.
Overview
In this lab, we will create a streaming data pipeline with Kafka providing a hands-on look at the Kafka Streams API. You will run a Java application that uses the Kafka Streams library by showcasing a simple end-to-end data pipeline powered by Apache Kafka.
Objectives
In this lab GSP730, we will:
- Start a Kafka cluster on a Compute Engine single machine
- Write example input data to a Kafka topic, using the console producer included in Kafka
- Process the input data with a Java application called WordCount that uses the Kafka Streams library
- Inspect the output data of the application, using the console consumer included in Kafka
Setup and requirements
Before you click the Start Lab button
Read these instructions. Labs are timed and you cannot pause them. The timer, which starts when you click Start Lab, shows how long Google Cloud resources will be made available to you.
This hands-on lab lets you do the lab activities yourself in a real cloud environment, not in a simulation or demo environment. It does so by giving you new, temporary credentials that you use to sign in and access Google Cloud for the duration of the lab.
To complete this lab, you need:
- Access to a standard internet browser (Chrome browser recommended).
Note: Use an Incognito or private browser window to run this lab. This prevents any conflicts between your personal account and the Student account, which may cause extra charges incurred to your personal account.
- Time to complete the lab — -remember, once you start, you cannot pause a lab.
Note: If you already have your own personal Google Cloud account or project, do not use it for this lab to avoid extra charges to your account.
