Job Description :
Hadoop Data Engineer (Hadoop Developer)
Franklin Lakes NJ
5+ years of work Experience in BigData / Hadoop Technologies.
Job Description:
Desirable Skills:
1. Experience building and optimizing scalable ''big data'' data pipelines, architectures and data sets.
Should have hands on experience in writing applications in Spark/MapReduce.
2. Understanding of Hadoop architecture and core concepts of Hadoop Distributed File System as well as skillset in manipulating, processing, extracting value from data stored on HDFS.
3. Extensive experience in extracting data to-fro from relational database systems using Sqoop and incorporating performance optimized incremental load stratgegies.
4. Should be well versed with Unix/Linux environment and should have hands-on experience in writing shell scripts using Unix.
5. Experience in writing user defined functions using PigLatin to perform various data transformation functions.
6. Build hadoop processes supporting data transformation, metadata, other dependencies and workload management.
7. Experience in real time analytics applications, should be able to build and deploy pipelines to collect real time streaming data using Kafka.
8. Advanced knowledge and experience working with Hive and other relational databases. Possess skills to transform the incoming data using Hive Query Language according to business needs. Skillset of understanding various Formats of the tables.
9. Strong analytic skills related to working with unstructured datasets as well as techniques to extract it from disparate data sources and storing it into HDFS or on Cloud.
10. Strong knowledge and experience building applications associated with NoSql databases like HBase, Cassandra. Well versed with storing and fetching data out of these NoSql databases.
11. Experience in scheduling the jobs using ControlM.
Strong experience with Hadoop ETL/Data Ingestion: Sqoop, Flume, Hive, Spark
Experience in Hadoop Data Consumption and other components; Hive,Ambari, Spark,Kafka.
Experience monitoring, troubleshooting and tuning services and applications and operational expertise such as good troubleshooting skills, understanding of system''s capacity, bottlenecks, basics of memory, CPU, OS, storage and networks
Experience with open source configuration management and deployment tools such as Puppet or Chef and Scripting using Python / Shell / Perl / Ruby / Bash
Good understanding of distributed computing environments