Print Friendly and PDF

Data Engineering Streaming

onDemand | Data Engineering | Self-Paced | 10.5.1

Course Overview

Gain the skills necessary to execute end-to-end data engineering streaming use cases. Learn to prepare, process, enrich, and maintain streams of data in real time using Informatica Developer Tool, Kafka, and Spark. This course is applicable to software version 10.5.1. 

Objectives

After successfully completing this course, students should be able to:

  • Discuss streaming
  • Describe Kappa architecture
  • List the types of streaming data
  • List the DES key features
  • Describe the DES component architecture
  • Describe Kafka data objects
  • Create Kafka connections
  • Discuss and list sources, and targets in a streaming mapping
  • Discuss lookup sources
  • Execute a streaming mapping
  • Monitor logs and troubleshoot streaming mappings

Target Audience

  • Developer
  • Administrator

Prerequisites

    AND
Agenda
Module 1: Streaming Overview
  • Key differences between batch and streaming
  • Streaming Data Management use cases
  • Streaming architecture
  • Kappa architecture
  • End-to-end Streaming Data Management
  • Types of streaming data
  • Benefits of streaming
  • Lab: Getting Started
Module 2: Data Engineering Streaming Overview
  • Data Engineering Streaming overview
  • Stream Data Processing with Spark streaming
  • DES component architecture
  • DES key features
Module 3: Kafka Overview
  • Kafka Concepts
  • Kafka core APIs
  • Topics in Kafka
  • Kafka models
  • Kafka use cases
  • Lab: Install and Configure Kafka
  • Lab: Create a Kafka connection
Module 4: Streaming Mappings
  • Sources in a streaming mapping
  • Targets in streaming Mapping
  • Lookup sources
  • DQ Transformations in streaming mappings
  • Kafka Data Object Properties
  • DES Transformations
  • Lab: Create a Mapping with Kafka Source and HDFS Target
  • Lab: Create a Mapping with Kafka Source and Kafka Target
  • Lab: Enhance Mapping Using Filter and Expression Transformations
  • Lab: Enhance Mapping Using Window and Aggregator Transformations
  • Lab: Create a Mapping Using Kafka Source and Kudu Target
  • Lab: Create a Mapping to write standardized data using Kafka Source and Hive Target
  • Lab: Create a Mapping Using Parser Transformation
  • Lab: Create a Mapping Using Classifier Transformation
Module 5: Monitoring Logs and Troubleshooting
  • Spark Monitoring
  • Viewing Logs
  • Troubleshooting
  • Lab: Monitor a DES Mapping
Module 6: Performance Tuning and Best Practices
  • Tune performance of Spark jobs
  • List some best practices while working with streaming data

Enroll Now

Back to Course Overview

Power User Axon for Community Users (Instructor Led or onDemand) Axon Content Curation (Instructor Led) Axon for Power Users (Instructor Led) Axon Data Governance (Professional Certification) Axon Data Governance (Professional Certification) Axon Data Governance (Professional Certification) Some more content to make this bigger asdf asdf asdf

Informatica offers programs to extend learning in convenient and economic packages. Programs include self-paced subscriptions as well as bundled instructor led training and certifications. Each program is curated around a specific skillset to enable customer success.

365University Data Governance Annual Subscription

Informatica MasterPass Education Subscription

Informatica Learning Library

Data Governance & Privacy Journey Master

View Full Course Offerings