· Direct experience of building data pipelines using Azure Data Factory and Apache Spark (preferably Databricks).
· Microsoft Azure Big Data Architecture certification (preferred)
· Delivering and presenting key components to project stakeholders.
· Developing scalable and re-usable frameworks for ingesting of data sets
· Integrating the end to end data pipeline to take data from source systems to target data repositories ensuring the quality and consistency of data is maintained at all times
· Working with event based / streaming technologies to ingest and process data
· Working with other members of the project team to support delivery of additional project components (API interfaces, Search)
· Working within an Agile delivery / DevOps methodology to deliver proof of concept and production implementation in iterative sprints.
· Strong knowledge of Data Management principles
· Experience using geospatial frameworks on Apache Spark and associated design and development patterns