Job Description :
Manage large scale multi-tenant Hadoop cluster environments.
Handle all Hadoop environment builds, including design, security, capacity planning, cluster setup, performance tuning and ongoing monitoring.
Perform high-level, day-to-day operational maintenance, support, and upgrades for the Hadoop Cluster.
Research and recommend innovative, and where possible, automated approaches for system administration tasks.
Deploy new/upgraded hardware and software releases and establish proper communication channels.
Assist in effort estimating, forecasting, and work planning for resources
Promote a “one-team” approach for sharing common processes and practices
Builds a network of strong, collaborative relationships (e.g., with customers, employees, business colleagues, and external partners) that increases collaboration and communication – Product Development Teams, Architects and Data Architects.
Performs Capacity Planning for storage and performance; includes proper monitoring, trend analysis, and modeling.
Performance-tuning consultation by working closely with software vendors to tune the database and meet application specific needs
Ensures creation of runbooks for vendor partners and service providers in the assembly, configuration and administration of the application database
Develop and enforce standards, policies, and procedures for managing databases
Challenges established ways of doing things to improve the organization’s capabilities around commitments, actions and results
Minimum Requirements
8 – 11 years of experience managing a multi-tenant Hadoop eco-system (HDFS, HBase, YARN, Hive, Spark, Kafka) environment
5 years of experience as a Systems Engineer with Linux Administration
Understanding of Hadoop internals HDFS, YARN, Hive, Spark, cluster connectivity, security and the factors that affect distributed system performance
Proven experience with identifying and resolving hardware and software related issues;
Experience working with HDFS, YARN, Hive, Spark.
Experience supporting BigData clusters with PetaBytes of data
Experience with building/upgrading/patching BigData clusters
Knowledge of best practices related to security, performance, and disaster recovery;
Expert experience with at least two of the following languages; SQL, Python, Java, Scala, Spark or Bash.
Experience with complex networking infrastructure including firewalls, VLANs, and load balancers.
Working knowledge of operating system internals (Shell scripting, monitoring, debugging) in Linux environments is required
Experience with automated database migration techniques
Experience with Agile development methodologies is highly desirable
Experience with cloud based datadases is big plus
Experience with virtualization in continuous integration environment is highly desirable
Minimum of Bachelor’s degree in MIS/IT, business administration, or related discipline