Role Overview
We are hiring a Senior/Staff Data Engineer to lead the architecture and development of enterprise-scale streaming data pipelines, with Kafka as the core technology. This is a hands-on builder and owner role involving architecture decisions, production support, and direct collaboration with onsite US clients.
Key Responsibilities
- Architect, build, and operate real-time and near-real-time Kafka-based pipelines
- Lead development using Confluent Kafka (core focus)
- Design and implement stream processing using:
- Apache Flink (preferred) or
- Apache Spark Structured Streaming
- Define Kafka best practices for:
- Topic design, partitioning, and delivery semantics
- Error handling and retries
- Backpressure management and performance tuning
- Design and maintain batch and hybrid ETL pipelines using SQL Server
- ETL tools preferred: SSIS
- Own data quality, reliability, and observability across streaming pipelines
- Work directly with onsite clients to:
- Translate business requirements into technical solutions
- Defend architectural decisions
- Troubleshoot complex Kafka and streaming production issues
- Mentor engineers and raise overall engineering standards
- Collaborate with DevOps and platform teams in containerized, distributed environments
Required Skills & Experience
- 8+ years of hands-on data engineering experience
- Deep expertise in Confluent Kafka:
- Architecture and cluster design
- Topic and partition strategy
- Delivery semantics (exactly-once, at-least-once, etc.)
- Strong hands-on experience with:
- Apache Flink (preferred) or
- Apache Spark Structured Streaming
- Excellent SQL skills (complex analytics, performance tuning)
- Strong experience in Python, Java, or .NET (C#)
(At least one OOP language is mandatory) - Solid understanding of:
- Streaming and event-driven architectures
- ETL / ELT patterns
- Data modeling and orchestration
- Experience with SQL Server
We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.