Job Details
Job Description:
- Primarily responsible for managing and administering in house product and workflows.
- Experinece with programming in Python within AWS Glue.
- Overseeing legacy integrations that are currently being managed by offshore resources; legacy integrations built in SSIS - would be a nice to have skill. Offshore team is still primarily managing these, but this Engineer will work closely with them.
- Data sources are primarily on-prem, SQL, Oracle and SAP moving into AWS data lake. The ODAP tool generates a customized pipeline (Python/Glue) to bring data from source to data lake.
- Largely transactional data movements right now, but moving toward more real time/near real. Skills in real-time pipeline development would be the ideal profile to help with some of these migrations.
- Orchestrations are running in Airflow, but another tool would be acceptable here.
- Primarily responsible for maintaining existing ETL jobs and integrations, but if there is a feature enhancement or performance improvement that can be made, they will be responsible for that.
- A Mid level should be okay to start, open to them growing into the job role, but will eventually be serving in somewhat of a lead capacity.
- Will be responsible for CI/CD deployments - open to any tooling i.e. Jenkins, Github Actions
Must Have:
- Python
- AWS Glue
- ETL development & management of ETL jobs
- ETL pipeline orchestration (Airflow preferred, other tooling/Step Functions okay)
- CI/CD
Nice to Have:
- Real-Time streaming (experience migrating batch/transactional data workloads to real/near real time)
- SSIS
We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.