Job Description :
Responsibilities:
Design and implement data ingestion techniques for real time and batch processes for a variety of sources into Hadoop ecosystems and HDFS clusters
Define and document architecture roadmaps and standards
Drive use case analysis and architectural design around activities focused on determining how to best meet customer requirements within the tools of the ecosystem
Ensure scalability and high availability, fault tolerance, and elasticity within big data ecosystem
Provide technical leadership; Guide/Mentor/Assist in development of the proposed architectural solution
Architect and develop ELT and ETL solutions focused on moving data from highly diverse data landscape into a centralized data lake
Manage all activities centered on obtaining data and loading into an Enterprise Data Lake
Serve as an expert in efficient ETL, data quality, and data consolidation
Stay current with vendor/product roadmaps and make recommendations for adoption (Informatica BDM, Cloudera, MongoDBetc)
Develop automation for the setup and maintenance of the Big Data Platform
Lead performance tuning activities of the system
Need to take end-to-end ownership for solution components and bring in design best practices and tools
Basic Qualifications:
8 years IT experience
3+ years of experience building large-scale data solutions involving data architecture, data integration, business intelligence and data analytics
2+ year of experience working on large scale Big Data projects
Deep technical knowledge of most components contained within the Hadoop ecosystem (MapReduce, HDFS, YARN, Hive, HBase, Sqoop, etc, preferable with Hortonworks distribution
Hands on experience with Scala/Spark.
Experience with RDBMS platforms