Tops 3 Skills Needed:
| 1 | Strong/expert Spark (PySpark) Using Jupyter Notebooks or DataBricks | 7+ years |
| 2 | Strong analytical and design skills | 7+ years |
| 3 | Hands on data pipeline development, ingest patterns in Azure | 7+ years |
Years experience:
Developing, implementing, and sustaining test automation processes, practices, and controls in support of application and system requirement, development and test activities throughout the software development and sustainment lifecycles. Leads and consults on test automation strategy, requirement, design, implementation and execution. 7 years
Key projects: Employee Data Analytics
This position is responsible for design, development, testing and support for data pipelines to enable continuous data processing for data exploration, data preparation and real-time business analytics.
Daily schedule:
1. Demonstrate deep knowledge the data engineering domain to build and support non-interactive (batch, distributed) & real-time, highly available data, data pipeline and technology capabilities
2. Build fault tolerant, self-healing, adaptive and highly accurate data computational pipelines
3. Provide consultation and lead implementation of complex programs
4. Develop and maintain documentation relating to all assigned systems and projects
5. Tune queries running over billion of rows of data running in a distributed query engine
6. Perform root cause analysis to identify permanent resolutions to software or business process issue
Skills/Qualifications
- We are looking for strong hands on knowledge (more than 4/5) in the following:
- 1. Strong/expert Spark (PySpark) Using Jupyter Notebooks or DataBricks
- 2. Hands on data pipeline development, ingest patterns in Azure
- 3. Orchestration tools, ADF or Airflow
- 4. SQL
- 5. Denormalized Data modelling for big data systems
What determines the best candidate over an average candidate?:
- Cloud-Computing (Azure) experience is going to be a differentiator
Unique selling points?/Value added:
- Contribute to applications that are being utilized by millions of customers
- Resume builder- solve problems at scale- live production systems
- Huge footprint with massive amount of visibility
- Working with a major brand