Course Overview Print

This intermediate two-day course equips learners with the skills to design, build, secure, and automate reliable data pipelines using Apache Spark on the Databricks Free Platform. It extends beyond basic notebook execution by incorporating structured software engineering practices. The course integrates data collection and analysis techniques with modern data architecture concepts (e.g., Lakehouse and Data Mesh), as well as operational practices such as FinOps, observability, and data governance. Learners will also apply agile development methodologies to design, enhance, and deploy data-driven applications.

Learning Outcomes:

  • Explain software development standards, protocols, and best practices in data engineering solutions.
  • Implement data quality and validation frameworks to maintain dataset integrity.
  • Deploy appropriate automation solutions for repetitive and routine tasks in software development and data pipeline deployment processes.

Course Outline

 Module 1: The Engineering Landscape & Theory
Module 2: Storage Strategy & Economics
Module 3: Engineering Lifecycle
Module 4: Advanced Modelling & Migration
Module 5: The Secure Data Lakehouse - Architecture Data Warehouse principles & Medallion
Module 6: Robust Pipeline Construction
Module 7: Observability & Automation
Module 8: Streaming & Real-Time Challenges 

Target Audience and Pre-requisites

This course is targeted at intermediate to senior data professionals, including Data Engineers, Analytics Engineers, and Data Platform Engineers, who want to design and operate production-ready data pipelines using Apache Spark and Databricks.

 

Minimum entry requirement

Leaners are strongly recommended to have the following skills and knowledge before starting this course:

General knowledge of databases and analytics tools. Basic understanding of statistics and data visualization.

Minimum Education: Diploma or Professional certifications.

ST Engineering e-Services will determine the suitability of the learners if entry requirement is not met.

SWDA Funding Requirements

  • Trainees must scan their attendance twice daily using the SingPass application.
  • Trainees must attain at least 75% attendance.
  • Trainees must pass the in-house assessment to be eligible for funding.
  • Trainee and/or sponsoring company is/are required to meet all SWDA-mandated eligibility criteria and requirements for funding.  For more information, please refer to SkillsFuture homepage.

Appeal Policy and Procedure

  • As a candidate in this course assessment, you may appeal your results if you disagree with them.
  • To do so, submit your written appeal request via email to esv_comat_cse@stengg.com within 3 working days from date of assessment.

Cancellation, Postponement and Refund Policy

  • Request for cancellation or postponement must be submitted in writing more than 4 weeks before the class start date to avoid any charges.
  • Written notice for cancellation or postponement received 2 to 4 weeks before class start date will incur Late Cancellation Charge - 50% of course fee.
  • Written notice for cancellation or postponement received less than 2 weeks before class start date will incur Late Cancellation Charge - 100% of course fee.
  • If payment has been made and ST Engineering e-Services Pte Ltd accepts the trainee's written notification to cancel or withdraw from the course, ST Engineering e-Services Pte Ltd will issue a refund, less any applicable Late Cancellation Charges.

Feedback Policy and Procedure

  • You may submit feedback via email to esv_comat_cse@stengg.com or your servicing Account Manager.
  • Any formal feedback will be handled within 10 working days from receipt with a written reply given.  An interim reply will be provided should more time be required.
*Important Note : Fees are subject to Singapore's prevailing Goods and Services Tax (GST).

Funding available

Course fee funding

Pay or claim with

Eligibility conditions apply. Fees & Funding →
Data Engineering - Foundations & Operations
Fees & Funding Enquire Now

Have Question?

We’re here to help — reach out anytime.

By submitting this form, you consent to be contacted via email and/or your mobile number regarding your enquiry. You consent to the collection, use, disclosure and processing of your personal data in accordance with our Personal Data Policy.