Job Description :
Job title : Big Data Engineer
Location : Santa Clara , CA
Duration : 6 MONTHS
Job description :
As a Big Data Engineer, you will be an integral member of our Big Data & Analytics team responsible for design and development scalable data processing and storage systems.
Partner with data analyst, product owners and data scientists, to better understand requirements, finding bottlenecks, resolutions, etc.
Suggest and implement improvements to development tools and processes.
Build data pipelines and ETL using heterogeneous sources
You will build data ingestion from various source systems to Hadoop using Kafka, Flume, Sqoop, Spark Streaming etc.
You will transform data using data mapping and data processing capabilities like Map Reduce, Spark SQL
Expands and grows data platform capabilities to solve new data problems and challenges
Supports Big Data and batch/real time analytical solutions leveraging transformational technologies like Apache Beam
You will have the ability to research and assess open source technologies and components to recommend and integrate into the design and implementation
You will work with development and QA teams to design Ingestion Pipelines, Integration APIs, and provide Hadoop ecosystem services
3-5 years of experience with the Hadoop ecosystem and Big Data technologies
Career level software development experience
Ability to dynamically adapt to conventional big-data frameworks and tools with the use-cases required by the project
Hands-on experience with the Hadoop eco-system (HDFS, Map Reduce, H Base, Hive, Impala, Spark, Kafka, Kudu, Solr)
Experience with building stream-processing systems using solutions such as spark-streaming, Storm or Flink etc
Experience in other open-sources like Druid, Elastic Search, Log stash, CICD and cloud-based deployments is a plus
Knowledge of design strategies for developing scalable, resilient, always-on data lake
Knowledge of agile(scrum) development methodology is a plus
Strong development/automation skills. Must be very comfortable with reading and writing Scala, Python or Java code.