Job Description :
Position : Big Data Engineer
Location : San Jose, CA
Duration of contract : Long term
Job Description:
As a Big Data Engineer, you will be an integral member of our Big Data & Analytics team responsible for design and development
Help build the technical foundations of our Data Governance Engineering team
Be constantly challenged by tough engineering and design tasks
Build data profiling and ETL jobs using heterogeneous sources
Partner with data analyst, product owners and data scientists, to better understand requirements, finding bottlenecks, resolutions, etc.
Design and develop different architectural models for our scalable data processing and profiling and create certified feature sets to be consumed by our data science team
Transform data using data mapping and data processing capabilities like MapReduce, Spark SQL
Supports Big Data and batch/real time analytical solutions leveraging transformational technologies
Create complex integrations, between data governance platform and Hadoop Data Lake, SAP HANA, Salesforce etc.
Enable users to access enterprise wide data definitions and business glossary using REST API services
Have basic level of expertise in the areas of data governance, data quality, MDM, information management, metadata management and information exchange
You will have the ability to research and assess open source technologies and components to recommend and integrate into the design and implementation
You will also have opportunity to work closely with our data science team and work on various business critical predictive use-cases
Desired Skillset:
1+ year of experience with the Hadoop ecosystem and Big Data technologies
Strong knowledge of JAVA, data structure and algorithms
Basic understanding of SPARK jobs and prior experience with Cloudera Navigator
Experience in working with REST/SOAP API services
Experience in working with ETL tools like Spark, Mulesoft etc.
Nice to have previous experience in big data open source products like Apache Kafka, Confluent Centralized Schema Registry, etc.
Bonus points for previous experience with Data Governance and Data Quality tracking processes/systems and tools, such as Collibra
Understanding of data governance and data profiling concepts
Ability to work in a fast-paced environment and manage multiple simultaneous priorities
Effective problem solving and analytical skills