Job Description :
position:Big Data Developer
Location:100% Remote
Duration:6+ Months
1. Big Data
a. Scala is used for data pipelines with spark
b. Pulling data out of JSON data docs
c. On prem Hadoop experience is a must
d. 70% of their time will be spent writing code and automating the data pipelines
i. Existing pipeline, creating new stream for a new product they are developing
1. Will need to partner with that other team on collaboration
RESPONSIBILITIES:
Building the data pipeline using big data technologies
More than 70%+ of the time spent on coding and/or hands-on technical implementation of scalable quality software solutions
Lead the design, coding, and unit tests through consistent development practices with a focus on building CI/CD, automated testing, and monitoring
Works with large, complex data sets such as JSON data documents to build data driven analytical products
Perform various technical aspects of software development for assigned tasks including design, developing prototypes, and coding assignments
Lead thorough code reviews
Lead the deployment, support, and monitoring of software across development, integration, and production environments
Empower implementation teams by building scalable implementations by minimizing custom code per implementation
All phases of software engineering including requirements analysis, data architecture, solution design, code development and testing
Designing and implementing product features in collaboration with other technical teams, business and IT stakeholders
Designing reusable technical frameworks, libraries and components
Working very closely with Data Architecture group
REQUIRED SKILLS:
Proven experience with Spark on Big Data platform working with large data sets
Big Data Technologies (Spark, Hadoop, HDFS, Hive, SQL)
Strong analytical skills and programming skills (preferably in Scala)
Unix, Python scripts
Agile/Scrum methodology
Experience with Azure Cloud such as the Blob Storage (nice to have)
8+ years software engineering
5+ years hands on expertise with Unix, Scala, SQL
3+ years Big data technologies