Job Description :
The Data engineer will work on analyzing the data needs, migrating the data into an Enterprise data lake, build data products and reports.
The role requires experience with building real time and batch based ETL pipelines with strong understanding of big data technologies and distributed processing frameworks with.Skill Needso
Expertise working with large scale distributed systems (Hadoop, Spark
Strong understanding of the big data cluster, and its architectureo Experience building and optimizing big data ETL pipelines.
Advanced programming skills with Python, Java, Scala
Good knowledge of spark internals and performance tuning of spark jobs.o Strong SQL skills and is comfortable operating with relational data models and structure.
Capable of accessing data via a variety of API/RESTful services.o Experience with messaging systems like Kafka.
Experience with No SQL databases. Neo4j, mongo, etc.o Expertise with Continuous Integration/Continuous Delivery workflows and supporting applications.o Exposure to cloud environments and architectures. (preferably Azure)
Ability to work collaboratively with other teams. o Experience with containerization using tools such as Docker.