Job Description :
Azure Senior Data Engineer
Houston, TX
Duration : 12+months
JD : : Data Factory and Data Bricks
Oil and Gas client looking for an Azure Senior Data Engineer with experience in designing data centric applications in Azure leveraging the following:
Platform: SQL Databases, ELT and ETL tools, Microsoft Azure: Azure SQL Database, Data Factory, SQL Data Warehouse, App Services (Web, API, Logic apps), Azure Functions, Log Analytics, networking & security, Azure storage, Service Bus, ExpressRoute/VPN, Azure Active Directory, Azure Dev Ops, Azure ARM templates
Technologies: SQL, PolyBase, Python, PySpark, Spark internals, PowerShell, Bash, C#, JavaScript, Entity Framework, RESTful services, Microservices, Visual Studio
Qualifications
12+ years of professional solution delivery experience and a Bachelor of Computer Science, MIS or equivalent degree
10+ years of experience as an ETL developer in 1 or more: Informatica, SSIS, Ab Initio, SAP Data Services
3-5 years of designing and building modern data pipelines and data streams in the Azure/Databricks environment and deployed the solution in Production
3-5 years of designing Azure data storage solutions (SQL Database, SQL Data Warehouse, Data Lake Storage)
3-5 years of developing data ingestion, data processing and data optimization (Databricks, Data Factory, Informatica, PolyBase)
3-5 years of developing complex scripting leveraging Python, Pyspark and SQL
Experience in delivering solutions within a Massively Parallel Processing (MPP) environment
Experience in solving complex ASDW query performance and defining execution rules
Experience in curation process and leveraged Data Flows, Data Bricks, Delta lake for Slowly Changed dimensions
Experience in handling various file formats viz. Avro, Parquet, Delta
Azure Certification
Experience in designing and developing User interfaces using .NET technologies
Understating of Master Data Management (MDM) and data quality tools and processes
Understanding of Lambda and Kappa architecture patterns
Understanding of DevOps including CI/CD
Experience developing software architectures and key software components
High-level understanding of common authentication patterns and flow including single sign-on and OAuth
Experience with Agile/Scrum methodology
Ability to apply technology and consulting to solve a client business problem
Sufficient depth and breadth of technical knowledge to be individually responsible for the design and scope of deliverables of a data warehouse solution
Able to communicate and present complex issues with preciseness, assurance, and confidence
A disciplined approach to software development and problem solving
Passion for technology and a high technical aptitude
Insightful and always looking for breakthrough ideas
Self-sufficient, high integrity, more than just competent
Responsibilities
Collaborate with data engineers, data architect, business analysts to build a modern data warehouse application on Azure
Perform POCs on various concepts using Azure services as required
Ingest, curate data using Data Factory and Data Bricks
Design, develop dimensional model in MPP environment using Azure Synapse Analytics
Understand the data architecture design, solution structures and component design
Ensure that technical software development process is followed on the project, be familiar with industry best practices for software development and agile practices
Understand a broad spectrum of technology in order to provide part or all of a detailed technical design which meets customer requirements
Act as senior data engineer on customer projects to deliver a wide-range of PaaS-focused solutions
Participate in standups, agile breakout sessions and independently review the deliverables in a timely manner
Communicate across the client’s community – consistently viewed as adding value
Contribute knowledge, tools, and positivity to the rest of internal Data Engineer community