Job Description :
Role : Senior Big Data Engineer Scala/Spark
Location: [Fort Mill /NY/NJ/or Remote, who can work in EST time zone]
Experience Level: Senior level only - 13+ years
Role Overview:
We are looking for a highly skilled Senior Big Data Engineer with strong expertise in Scala and Apache Spark to support our data engineering team. The ideal candidate should be able to read and refactor existing Scala Spark code, develop enhancements, create robust unit tests, and help migrate code into a modern cloud-native environment.
Key Responsibilities:
Analyze and refactor existing Scala Spark code to support migration to new environments.
Write and enhance Scala Spark code with a focus on performance and maintainability.
Develop comprehensive unit tests to ensure code quality and stability.
Work with data stored in Amazon S3 (CSV and Parquet formats).
Integrate and manipulate data from MongoDB.
Independently set up environments to validate and test code changes.
Execute both manual and automated testing procedures.
Collaborate within an Agile team and contribute to sprint planning, story grooming, and reviews.
Maintain clear and proactive communication with team members, project managers, and stakeholders.
Technical Skills & Qualifications:
Proficiency in Scala and Apache Spark for large-scale data processing.
Working knowledge of Python for scripting and data transformation.
Hands-on experience with Amazon Web Services (AWS), including:
S3
AWS Glue
Step Functions
Experience working with MongoDB and structured/unstructured data.
Familiarity with SQL and BDD frameworks.
Experience with Terraform for infrastructure-as-code is a plus.