Job Description :
Job Title: Senior Hadoop Engineer USC/GC/GCEAD
Location: Holmdel, NJ
Duration: 6 months plus, this is a contract to hire role
The Senior Hadoop Engineer will assist in the setup and production readiness
of Client''s Data Lake. The candidate will work on the installation and
configuration of Pivotal HC 3.0, utilizing Open Source components such as
Ranger and Ambari. The candidate should have a knowledge of concepts such
as LDAP integration, Kerberos and highly available architectures.
What the Resource will do
* Responsible for setup, administration, monitoring, tuning, optimizing,
governing Hadoop Cluster and Hadoop components
* Design and implement new components and various emerging technologies
in Hadoop Echo System and successful execution of various Poof-Of-Technology
(PoT)
* Design and implement high availability options for critical component
like Kerberos, Ranger, Amabari, Resource Manager, MySQL repositories
* Collaborate with various cross functional teams; infrastructure,
network, database and application for various activities: deployment new
hardware/software, environment, capacity, uplift, etc.
* Work with various teams to setup new Hadoop users, security and
platform governance
* Create and execute capacity planning strategy process for the Hadoop
platform
* Work on cluster maintenance as well as creation and removal of nodes
using tools like Ganglia, Nagios, Cloudera Manager Enterprise, Ambari, etc.
* Performance tuning of Hadoop clusters and various Hadoop components
and routines
* Monitor job performances, file system/disk-space management, cluster
and database connectivity, log files, management of backup/security and
troubleshooting various user issues
* Hadoop cluster performance monitoring and tuning, disk space
management
* Harden the cluster to support use cases and self services in 24x7
model and apply advanced troubleshooting techniques to critical, highly
complex customer problems
* Contribute to the evolving Hadoop architecture of our services to meet
changing requirements for scaling, reliability, performance, manageability
and price
* Setup monitoring and alerts for the Hadoop cluster, creation of
dashboards, alerts and weekly status report for uptime, usage, issues, etc
* Design, implement, test and document performance benchmarking strategy
for platform as well or each use case
* Act as liaison between the Hadoop cluster administrators and the
Hadoop application development team to identify and resolve issues impacting
application availability, scalabilty, performance and data throughput
* Research Hadoop user issues in a timely manner and follow up directly
with the customer with recommendations and action plans
* Work with project team members to help propagate knowledge and
efficient use of Hadoop tool suite and participate in technical
communications within the team to share best practices and learn about new
technologies and other ecosystem applications
* Automate deployment and management of Hadoop services including
implementing monitoring
* Drive customer communication during critical events and
participate/lead various operational improvement initiatives
What the Resource needs to succeed
* Bachelor''s degree in Computer Science, Information Science,
Information Technology or Engineering/related field
* 3 years of strong Hadoop/Big Data experience
* Strong experience on administration and management of large-scale
Hadoop productions clusters
* Able to deploy Hadoop cluster, add and remove nodes, keep track of
jobs, monitor critical parts of the cluster, configure high availability,
schedule and configure and take backups
* Strong experience with Hortonworks (HDP) or Pivotal (PHD) Hadoop
Distribution and Core Hadoop Echo System components: MapReduce and HDFS
* Strong experience with Hadoop cluster management / administration /
operations using Oozie, Yarn, Zookeeper, Tez, Slider
* Strong experience with Hadoop Security & Governance using Ranger,
Falcon Kerberos, Security Concepts-Best Practices
* Strong experience with Hadoop ETL/Data Ingestion: Sqoop, Flume, Hive,
Spark
* Experience in Hadoop Data Consumption and other components; Hive,
HUE,HAWQ,Madlib, Spark, Mahout, Pig
* Prior working experience with AWS, any or all of EC2, S3, EBS, ELB,
RDS
* Experience monitoring, troubleshooting and tuning services and
applications and operational expertise such as good troubleshooting skills,
understanding of system''s capacity, bottlenecks, basics of memory, CPU, OS,
storage and networks
* Experience with open source configuration management and deployment
tools such as Puppet or Chef and Scripting using Python / Shell / Perl /
Ruby / Bash
* Good understanding of distributed computing environments
* Education: Bachelor''s Degree or equivalent work experience
* Experience: Minimum 3 years as a Hadoop Engineer or Hadoop
Administrator