Job Description :
u>Data Engineers:/u>
Glider test: Data Engineer
Notes:
Going to be integrating with teams that are already running
5+ years of experience
Migrating legacy platform to the cloud platform
Migrating and modernizing in AWS
For example: Ab Initio – reverse engineering and bringing back to the cloud
ul>
li>Ab initio is a big legacy platform that they have modernized for/li>
/ul>
Skillset:
Language: Python/Scala/Java
ul>
li>Python is required/li>
li>Pyspark is a big one to have/li>
/ul>
Spark or Scala – required
Analytics of data
ul>
li>Analyzing results to compare with the legacy platforms/li>
li>Being engraved in the Data/li>
/ul>
AWS – EC2, EMR, Lambda
CI/CD – nice to have
ul>
li>Jenkins, and various deployment pipelines/li>
/ul>
Data Engineer Job Description for Data Transformation:
We are looking for a candidate with 5+ years of experience in a Data Engineer role, who has
attained a Graduate degree in Computer Science, Statistics, Informatics, Information Systems
or another quantitative field. They should also have working experience using the followingu>/u>software/tools:
? Experience with data pipeline and workflow management tools:
? Experience with AWS cloud services: S3, EC2, EMR, RDS, Redshift
? Experience with stream-processing systems: Storm, Spark-Streaming, etc.
? Experience with object-oriented/object function scripting languages: Python,
Java, C++, PySpark, Scala, etc.
Responsibilities for Data Engineer:
? Create and maintain optimal data pipeline architecture,
? Assemble large, complex data sets that meet functional / non-functional businessu>/u>requirements.
? Build analytics tools that utilize the data pipeline to provide actionable insights intocustomer acquisition, operational efficiency and other key business performance metrics.
? Work with stakeholders including the Executive, Product, Data and Design teams tou>/u>assist with data-related technical issues and support their data infrastructure needs.
? Keep our data separated and secure across national boundaries through multiple datau>/u>centers and AWS regions.
? Create data tools for analytics and data scientist team members that assist them inu>/u>building and optimizing our product into an innovative industry leader.
Qualifications for Data Engineer
? Advanced working SQL knowledge and experience working with relational databases,u>/u>query authoring (SQL) as well as working familiarity with a variety of databases.
? Experience building and optimizing ‘big data’ data pipelines, architectures and data sets.
? Experience performing root cause analysis on internal and external data and processesu>/u>to answer specific business questions and identify opportunities for improvement.
? Strong analytic skills related to working with unstructured datasets.
? Build processes supporting data transformation, data structures, metadata, dependencyu>/u>and workload management.
? A successful history of manipulating, processing and extracting value from largeu>/u>disconnected datasets.
? Working knowledge of message queuing, stream processing, and highly scalable ‘bigdata’ data stores.