Job Description :
Core Responsibilities
Analyzes and determines data integration needs
Evaluates and plans software designs, test results and technical manuals using Big Data (Hadoop) ecosystem
Reviews literature, current practices relevant to the solution of assigned projects in Data Warehousing and Reporting areas
Programs new software using Spark, Scala, Kafka, Sqoop, SQL
Supports existing and new applications and customization of current applications
Responsible for administration, monitoring, tuning, optimizing, governing Hadoop Cluster and Hadoop components
Design and implement high availability options for critical component like Kerberos, Ranger, Ambari, Resource Manager
Collaborate with various cross functional teams; infrastructure, network, database and application for various activities: deployment of new software, environment, capacity, uplift
Performance tuning of Hadoop clusters and various Hadoop components and routines
Monitor job performances, file system/disk-space management, cluster and database connectivity, log files, management of backup/security and troubleshooting various user issues
Hadoop cluster performance monitoring and tuning, disk space management
Harden the cluster to support use cases like self-service in 24x7 model and apply advanced troubleshooting techniques to critical, highly complex customer problems
Contribute to the evolving Hadoop architecture of our services to meet changing requirements for scaling, reliability, performance, manageability and price
Setup monitoring and alerts for the Hadoop cluster, creation of dashboards, alerts and weekly status report for uptime, usage, issues
Work with various teams to setup new Hadoop users, security and platform governance
Create and execute capacity planning strategy process for the Hadoop platform
Edits and reviews technical requirements documentation
Displays knowledge of software engineering methodologies, concepts, skills and their application in the area of specified engineering specialty (like Data warehousing)
Displays knowledge of, and ability to apply, process software design and redesign skills
Displays in-depth knowledge of, and ability to apply, project management skills
Works independently, assumes responsibility for job development and training, researches and resolves questions and problems, requests supervisor input and keeps supervisor informed required
Consistent exercise of independent judgment and discretion in matters of significance
Regular, consistent and punctual attendance. Must be able to work nights and weekends, variable schedule(s) as necessary
Other duties and responsibilities as assigned