Job Description :
Azure Data Engineer
Location- local to Houston, TX as on-site 1 - 2days is required.
LinkedIn is mandatory
Backfillrole- Urgent role
Main required skills:
1. Azure Data Lake
2. Azure Data Factoring
3. Azure Data Bricks
4. Python
The candidate needs to understand how Data Scientists work but does not have to be a data scientist
They are moving Data to the cloud (Azure) and cleansing it
The initial project is around pipeline integrity by taking data, identifying threats and creating actionable info.
*I changed the years of experience to 10 below.
A master’s degree is preferred but not required as they will take experience and a 4 yr. degree. But if 2 candidates are equal they will lead to the one with a graduate degree.
O&G is a very nice to have
There are only a total of 6 vendors
Cultural fit is real important, being collaborative, focus on users, embrace risk and being entrepreneurial.
They want people with new ideas that embrace cutting edge technology.
Agile experience is important
They will be in the office 1 to 2 days a week rotating. No more than 8 to 10 people per day and the room accommodates 40 people so plenty of social distancing
Big Data tools important
As a Data Engineer, you’ll help ingest, transformand store clean and enriched data in ready for business intelligenceconsumption.
Who you are
You’ll haveexperience in a Data Engineer role (10 years), with a Graduate degree inComputer Science, Statistics, Informatics, Information Systems or anotherquantitative field
You buildand maintain optimal data pipeline architecture.
You assemblelarge, complex data sets that meet functional / non-functional businessrequirements.
Youidentify, design, and implement internal process improvements: automatingmanual processes, optimizing data delivery, re-designing infrastructure forgreater scalability, data quality checks, minimize Cloud cost, etc.
You buildthe infrastructure required for optimal extraction, transformation, and loadingof data from a wide variety of data sources using SQL, Data Bricks, No-SQL
you buildanalytics tools that utilize the data pipeline to provide actionable insightsinto customer acquisition, operational efficiency and other key businessperformance metrics.
You documentand communicate standard methods and tools used.
You workwith other data engineers, data ingestion specialists, and experts across thecompany to consolidate methods and tool standards where practical.
You’reexperienced using the following software/tools:
Big datatools: Hadoop, HDI, & Spark
RelationalSQL and NoSQL databases, including COSMOS
Datapipeline and workflow management tools: Data Bricks (Spark), ADF, Dataflow
MicrosoftAzure
Stream-processingsystems: Storm, Streaming-Analytics, IoT Hub, Event Hub
Object-oriented/objectfunction scripting languages: Python, Scala, SQL
What you’lldo
You’ll workindependently on complex data engineering problems to support data sciencestrategy of products
You’ll usebroad and deep technical knowledge in the data engineering space to tacklecomplex data problems for product teams, with a core focus on using technicalexpertise
You’llimprove the data availability by acting as a liaison between Lab teams andsource systems
You’llcollect, blend, and transform data using ETL tools, database management systemtools, and code development
You’llimplement data models and structures data in ready-for business consumptionformats
You’llaggregate data across various warehousing models (e.g. OLAP cubes, starschemas, etc for BI purposes
You’llcollaborate with business teams and understand how data needs to be structuredfor consumption