Job Description :
Hadoop Developer

Duration: 12 + months with expectation to go full 24 months



As Hadoop Developer in Risk & Finance Core Services, the ideal candidate for this developer role should be able to:

- Build high performing data models on big-data architecture as data services.

- Build high performing and scalable data pipeline platform using Hadoop, Apache Spark and Amazon S3 based object storage architecture.

- Partner with Enterprise data teams such as Data Management & Insights and Enterprise Data Environment (Data Lake) and identify the best place to source the data

- Work with business analysts, development teams and project managers for requirements and business rules.

- Collaborate with source system and approved provisioning point (APP) teams, Architects, Data Analysts and Modelers to build scalable and performant data solutions.

- Effectively work in a hybrid environment where legacy ETL and Data Warehouse applications and new big-data applications co-exist

- Work with Infrastructure Engineers and System Administrators as appropriate in designing the big-data infrastructure.

- Work with DBAs in Enterprise Database Management group to troubleshoot problems and optimize performance

- Support ongoing data management efforts for Development, QA and Production environments

- Utilizes a thorough understanding of available technology, tools, and existing designs.

- Leverage knowledge of industry trends to build best in class technology to provide competitive advantage.



Required Qualifications

- ETL (Extract, Transform, Load) Programming experience

- Experience in Hadoop ecosystem tools for real-time batch data ingestion, processing and provisioning such as Apache Flume, Apache Kafka, Apache Sqoop, Apache Flink, Apache Spark or Apache Storm

- Java or Python experience

- Agile experience

- Design and development experience with columnar databases using Parquet or ORC file formats on Hadoop

- -Apache Spark design and development experience using Scala, Java, Python or Data Frames with Resilient Distributed Datasets (RDDs)

- Deliver the data services on container-based architecture such as Kubernetes and Docker



Desired Qualifications

- Excellent verbal, written, and interpersonal communication skills

- Ability to work effectively in virtual environment where key team members and partners are in various time zones and locations

- Knowledge and understanding of project management methodologies: used in waterfall or Agile development projects

- Knowledge and understanding of DevOps principles

- Reporting experience, analytics experience or a combination of both

- Experience designing and developing data analytics solutions using object data stores such as S3

- A BS/BA degree or higher in information technology

- Knowledge of Oracle Financial Services Data Foundation (FSDF) Banking & Insurance industry standard data mod