BigData Developer with Airflow - StateStreet
Client: StateStreet
Remote until Covid Boston, MA
Long term contract or Fulltime
· Overall 7+ years of data warehousing experience
· Minimum 2-3 years of experience on Apache Airflow in recent project
· Develop guidelines for Airflow cluster and DAGs. Performance tuning of the DAGs and task implementation
· Develop DAG - data pipeline to on-board and change management of datasets
· Experience with Apache Airflow, including installing, configuring and monitoring Airflow cluster
· Understanding of airflow rest services and integration of airflow of platform eco-system
· Orchestrating the Airflow / workflow in hybrid cloud environment is big plus
· Python & Pandas, with demonstrated experience in ETL
· Ability to maintain clean and secure data environments
· Working in a hybrid cloud environment
· Must have hands on experience with advanced sql
· Must be experienced in Lambda, SNS and SQS
· Redshift experience is desired