Job Description :
Job title: Hadoop lead with Java, Kafka and Spark
Location; Atlanta,GA and Chicago, IL
Duration: Long Term
Senior Scala/Spark Engineer
As a Senior Scala/Spark Engineer, you will work with the team to design and develop a new cutting-edge
Data Ingestion Pipeline Platform as well as several new and exciting products in the Big Data/Advanced
Analytics space. You will operate in a fast-paced environment where multiple project deliverables are
coordinated within specified deadlines. This is a great opportunity to work with cutting edge technologies
as well as propel your career.
Responsibilities
Design and develop Scala/Spark core code to ingest data from different data sources, transform
using data frames (in memory processing), write sink processes to write data into multiple targets
(RDBMS, Hive, HDFS, Hbase etc
Develop solutions in Scala and Java using best practices around enterprise architecture such as
dependency management, design patterns, complexity estimation & persistence layers.
Deliver high quality code deliverables for modules, lead validation for all types of testing and
support activities related to implementation, transition and warranty.
Requirements
MUST: Solid understanding (5+ years) in Scala, Java, JSON, XML
MUST: (5+ years) Experience working with large data sets and pipelines using tools and libraries
of Hadoop ecosystem such as Spark, HDFS, YARN, Hive and Oozie
MUST: (5+ years) of experience and working knowledge of distributed/cluster computing concepts.
MUST: Solid understanding of SQL, relational and nosql databases
MUST: Solid understanding (5+ years) in multi-threaded applications; Concurrency, Parallelism,
Locking Strategies and Merging Datasets. * MUST: Solid understanding (5+ years) in Memory
Management, Garbage Collection & Performance Tuning.
MUST: Solid understanding (5+ years) experience in Linux environments; strong knowledge of
shell scripting and file systems.
Experience working in cloud based environment like AWS.
Knowledge of CI tools like Git, Maven, SBT, Jenkins, and Artifactory/Nexus
Self-managed and results-oriented with sense of ownership is required
Excellent analytical, debugging and problem solving skills is required
Experience with Agile/Scrum development methodologies a plus
Minimum Bachelors degree in CS or equivalent with 8 - 10 years industry experience