Azure Data Engineer
Plano ,TX
6+ Months
Position's Overview
Overall 2+ years’ experience as Data & Intelligence field with min 2 years as Data Engineer using Big Data and cloud based data platforms (Azure or AWS)
- Experience with Azure Data Factory Pipelines and Azure services: Azure Data Lake Gen 2, Data factory, Data Flows, and Synapse (Azure SQL DW).
- Experience with Azure SQL DB/DW, Azure Logic Apps/API Management/Service Bus, Azure AD, Azure Resource Manager,
- Expert use of SQL and experience with relational & No SQL databases
- Expertise managing large datasets - SQL and UNIX scripting skills.
- Expertise in a scripting language like Python (or similar) - Development experience in Scala
- Minimum 2 years’ experience as a cloud-based Data Engineer
- Minimum 2 years’ experience and strong knowledge of Azure Data Services such as Azure Data Lake Storage, Blob Storage, Azure Data Factory,
- Minimum 2 years’ experience on Apache Spark or Azure Databricks. Good hands on skills working with Python, PySpark, and SparkSQL
- Good understanding of data integration and warehousing (ETL, ELT) processes
- Strong knowledge of Database concepts, SQL and experience in working with Database clients like, TOAD, SQL developer, SQL Server Management Studio
Position's General Duties and Tasks
Diverse Data Engineering activities on Azure Data platform.
- Build & Deliver Data pipeline connecting various enterprise data sources both RDBMS, NoSQL & APIs. Develop data mappings to existing data sources.
- Lead the data identification, data analysis efforts for data sources and work directly with data owner teams
- Develop data requirements for new data sources.
- Define new business data models.
- Design and development of data extraction, data ingestion, data quality rules implementation
- Scripting using programming language
- Clean and process the data for Machine Learning consumption.
- Provide Business Intelligence (PowerBI) and Data Warehousing (DW) solutions and support by leveraging project standards and leading analytics platforms
- Evaluate and define functional requirements for BI and DW solutions
- Design and development of data extraction, data ingestion, data quality rules implementation
- Involvement in architecture and design of data