Job Description :
Data Engineer with DevOps
Mclean, VA/Richmond, VA/Wilmington, DE
Long term
Job Description:
5+ years of experience (Sr-level) Strong Programming experience with object-oriented/object function scripting languages: Python with Spark
5+ years of strong DevOps Engineering experience (Sr-level)
Experience (Mid-level) with big data tools: Hadoop, Apache Spark, Kafka, etc
Strong Technical Experience with AWS cloud services and DevOps engineering: S3, IAM, EC2, EMR, Lambda RDS, Redshift, CloudWatch
Experience using modern build and deployment tools such as Jenkins, CICD, Docker, Kubernetes etc.
Experience with Terraform, Chef, Ansible or Puppet
Experience with stream-processing systems: Storm, Spark-Streaming, etc. (Nice to have)
1+ Years of experience with relational SQL, Snowflake and NoSQL databases, including Postgres and Cassandra.
Responsibilities:
Create and maintain optimal data pipeline architecture, Assemble large, complex data sets that meet functional / non-functional business requirements.
Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability etc.
Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS ‘Big data’ technologies.
Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency and other key business performance metrics.
Work with stakeholders including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs.
Create data tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader.
Troubleshooting problems, involving the appropriate resources and driving resolution of issues with a focus on minimizing impact to our customers.
Participate in the Agile DevOps design, development, testing, and release of new capabilities and features with focus on release and post-production support.
Represent production support for the suite of apps in the domain in Agile stand-ups, planning sessions and deployment activities.
Developing and enabling continuous integration/continuous deployment (CI/CD) for search system components.
Drive improvements to processes and design enhancements to automation to continuously improve the production environment.
Coordinates, approves, and/or performs changes to the production environment