Job Description :-
Job Summary:
We are seeking a highly skilled and experienced Test Architect / Test Manager to drive data engineering and migration initiatives using Azure Databricks, Python, and PySpark. The ideal candidate will lead the team in testing of scalable data pipelines and oversee the migration of legacy systems to modern cloud-based architectures.
Key Responsibilities:
Manage Testing of end-to-end data pipelines using Azure Databricks, Python, and PySpark.
Collaborate with cross-functional teams including data scientists, analysts, and business stakeholders to understand data requirements.
Ensure data quality, integrity, and governance across all migration and transformation processes.
Mentor junior engineers and provide technical leadership within the team.
Develop reusable frameworks and automation tools to streamline migration and data processing tasks.
Stay updated with the latest trends and best practices in cloud data engineering and migration.
Required Skills & Qualifications:
Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.
12+ years of experience in data engineering with expertise in Python and PySpark.
3+ years of experience working with Azure Databricks and other Azure data services (e.g., Azure Data Lake, Azure Synapse).
Proven experience in ETL and data migration projects, including strategy, execution, and validation.
Strong understanding of distributed computing, data modeling, and ETL/ELT processes.
Experience with CI/CD pipelines and version control tools (e.g., Git, Azure DevOps).
Excellent problem-solving and communication skills.
Ability to lead and manage multiple projects simultaneously.
We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.