Job Description :
The Job:
· As a Big Data Engineer, you will be an integral member of our Big Data & Analytics team responsible for design and development
· Partner with data analyst, product owners and data scientists, to better understand requirements, finding bottlenecks, resolutions, etc.
· Design and develop different architectural models for our scalable data processing as well as scalable data storage
· Build data pipelines and ETL using heterogeneous sources
· You will build data ingestion from various source systems to Hadoop using Kafka, Flume, Sqoop, Spark Streaming etc.
· You will transform data using data mapping and data processing capabilities like MapReduce, Spark SQL
· You will be responsible to ensure that the platform goes through Continuous Integration (CI) and Continuous Deployment (CD) with DevOps automation
· Expands and grows data platform capabilities to solve new data problems and challenges
· Supports Big Data and batch/real time analytical solutions leveraging transformational technologies like Apache Beam
· You will have the ability to research and assess open source technologies and components to recommend and integrate into the design and implementation
· You will work with development and QA teams to design Ingestion Pipelines, Integration APIs, and provide Hadoop ecosystem services
The Skills:
· 4+ years of experience with the Hadoop ecosystem and Big Data technologies
· Ability to dynamically adapt to conventional big-data frameworks and tools with the use-cases required by the project
· Hands-on experience with the Hadoop eco-system (HDFS, MapReduce, Hbase, Hive, Impala, Spark, Kafka, Kudu, Solr)
· Experience with building stream-processing systems using solutions such as spark-streaming, Storm or Flink etc
· Experience in other open-sources like Druid, Elastic Search, Logstash etc is a plus
· Knowledge of design strategies for developing scalable, resilient, always-on data lake
· Experience in agile(scrum) development methodology
· Strong development/automation skills. Must be very comfortable with reading and writing Scala, Python or Java code.
· Excellent inter-personal and teamwork skills
· Can-do attitude on problem solving, quality and ability to execute