Title- Data Engineer
Location- Plano, TX- Fully Onsite From Day-1
Type- Contract - C2C Works
Job Description:
We are seeking a highly skilled and experienced Data Engineer to join our team. The ideal candidate should possess expertise in a variety of technologies and tools, with a particular focus on Databricks, Python, Pyspark, MongoDB, Redis, Kubernetes, and containerization deployment.
Key Responsibilities:
Develop and maintain data pipelines using Databricks, Python, and Pyspark.
Implement Python API-based project development using REST APIs for efficient data processing.
Utilize MongoDB and Redis for data storage and retrieval.
Deploy and manage applications using Kubernetes and containerization technologies.
Collaborate with the Data Science and Machine Learning teams to develop and deploy DS/ML pipelines.
Explore and implement Delta Live Tables for real-time data processing and analytics.
Participate in the design, development, and optimization of data architecture.
Troubleshoot and resolve data-related issues in a timely manner.
Stay updated with emerging technologies and industry trends to continuously improve data engineering processes.
Skills Required:
Proficiency in Databricks, Python coding, and workflows.
Strong experience with Pyspark coding for data processing tasks.
In-depth knowledge of MongoDB and Redis for data storage and retrieval.
Familiarity with Kubernetes and containerization deployment for application management.
Experience in DS/ML pipeline development and deployment is highly desirable.
Knowledge of Delta Live Tables for real-time data processing is a plus.
Familiarity with Azure DevOps for continuous integration and deployment.
Understanding of real-time streaming concepts and technologies.