Software Development Engineer II – Data Engineer (DSP / AdTech)

Client: Leading Ad-Tech Company / Platform

Experience: 4-6 years

Role Overview

We are looking for a Data Engineer with strong expertise in large-scale data processing, performance optimization, SQL, and data modeling. The role focuses on building efficient data pipelines and implementing deterministic, rule-based business logic. Machine learning is not part of the core requirement.

Required Skills

1. 4-6 years of experience in Data Engineering or distributed data processing.
2. Strong hands-on experience with Python/PySpark, Apache Spark, Airflow and SQL.
3. Familiarity with cloud platforms such as Microsoft Azure (preferable), AWS or Google Cloud.
4. Good understanding of schema design, data modeling, and partitioning strategies.
5. Practical experience in performance tuning and processing large datasets.
6. Strong analytical, problem-solving, and debugging skills.
7. Understanding of ETL pipelines, distributed systems, and data quality.

Good to Have

1. Experience with Scala, Kafka, Parquet, Iceberg, or Delta Lake.
2. Experience in Ad-Tech, advertising analytics, or high-volume event processing.

Key Responsibilities

1. Build and maintain scalable data pipelines using Apache Spark/PySpark and SQL.
2. Tune and optimize Spark jobs and SQL queries to improve performance and resource utilization.
3. Design efficient data schemas based on data volume, access patterns, and business requirements.
4. Define appropriate partitioning strategies to improve query performance and processing efficiency.
5. Troubleshoot data skew, shuffle overhead, slow queries, and pipeline bottlenecks.
6. Implement rule-based data transformations, validations, and business logic.
7. Analyze data issues, identify root causes, and validate solutions using metrics and execution plans.

Key Skill Areas

1. Performance tuning: Identify bottlenecks and optimize Spark jobs/SQL queries.
2. Large datasets: Handle high data volumes efficiently.
3. Schema design: Design schemas based on business and query requirements.
4. Partitioning: Select suitable partition keys and resolve skew or small-file issues.
5. Analytical skills: Debug issues systematically and justify solutions with data.
6. Rule-based processing: Implement accurate, deterministic business logic without ML dependencies.