Job Description :
Open Source Lead (Azure Data Lake)
10+ years as an IT Professional
2+ years of experience on MS Azure or other cloud based big data solution HD Insights, Azure Data Lake, BLOB, Azure Events
5+ years of experience on Hadoop, Spark, Hive, Pig, Paraquete, Yarn, Avro, NoSQL databases, Hadoop, Hive, HBase, Pig, Spark, Elasticsearch
8+ years of experience working on large-scale data warehousing and analytics projects leveraging Big Data, Hadoop, Spark, end to end Data Warehousing, BI projects
Experience designing and implementing Azure Big Data solutions with both large volumes of batch & real time
Experience on capacity and infrastructure involving cluster sizing, VM set up
Experience with Data Repositories - Azure SQL DB, SQL Server, Blob, ADL
Experience with at least one implementation on cloud based NoSQL databases, Hadoop, Hive, HBase, Pig, Spark, Elasticsearch
Experience with PolyBase and/or T-SQL
Exposure to SQL and SSIS
Understanding of Python, Java, or other programming languages
Effectively analyze complex issues, identify and provide technical solution design, develop code with minimal defects, review teams code and solution, performance tuning of large volumes of data (batch and real time), assist in integration testing, deployments, Analyze and debug critical production defects
Work independently to manage and prioritize your own work load and teams work load
Work in a highly efficient manner to complete project assignments on-time
Build and migrate complex ETL pipelines from on premise /cloud based sources to cloud and Hadoop/Spark to allow the solution to grow elastically
Identify data quality issues and address them immediately to ensure a high quality user experience
Extract and integrate data from various heterogeneous data sources
Create a solution that will allow ad-hoc access to large datasets
Model data and metadata to support machine learning
Experience working in Agile / Super Agile environment
Ability to collaborate and communicate openly with team.
Collaborate effectively with teams - both onshore and offshore
Ability to multi task and priority work based on key support issues and development tasks
Ability to work on super agile projects with multiple sprints in parallel
Work closely with Business Analyst, Solution Architect, Project Manager, and client to ensure all project deliverables are met
Microsoft Azure Certification is a big plus
Bachelor Degree in Computer Science or related field
Mandatory Skills:
Azure HD Insights, Azure Data Lake, Spark, Azure Event, Cluster, VM