Azure Data Architect

Overview: We are looking for Azure Data Scientist Architect who specializes in designing and implementing data science solutions on the Microsoft Azure cloud platform. This role requires a deep understanding of both data science concepts and the Azure ecosystem. Will be responsible for creating end-to-end data science solutions that involve data collection, storage, processing, analysis, and visualization, all while leveraging the capabilities of Azure's services.

Key Skills:

• ADF/ADB Pipeline design, Pyspark
• ADF/ADB Pipeline Implementation
• ADF/ADB Pipeline CI/CD pipeline
• Data Quality metrics adherence
• Data security Implementation
• Data Consumption provisioning

Responsibilities: Years of Experience: 15-20.

Data Architecture Design:

• Collaborate with stakeholders to understand business requirements and translate them into architecture designs that meet those needs using Azure services
• Design scalable, efficient, and reliable solutions that align with the organization's data strategy and future growth needs.
• Develop data integration strategies that ensure data consistency, integrity, and security.
• ETL Development:
• Build, test, and maintain ETL processes to extract data from various source systems, transform it into the desired format, and load it into target databases or data warehouses.
• Develop ETL workflows and scripts using industry-standard tools and programming languages (e.g., SQL, Python, Apache Spark)
• Optimize ETL processes for performance and efficiency, considering factors like data volume, latency, and resource utilization.
• Data Quality and Governance:
• Implement data quality checks and validation rules to ensure the accuracy and reliability of data during the ETL process.
• Establish data governance practices, including metadata management, data lineage tracking, and documentation of ETL processes.
• Work with data stewards and business analysts to resolve data quality issues and maintain data consistency.

Collaboration and Communication:

• Collaborate with cross-functional teams including data engineers, data scientists, analysts, and business stakeholders to understand data requirements and deliver effective solutions.
• Communicate design choices, technical constraints, and recommendations to both technical and non-technical stakeholders.
• Performance Monitoring and Optimization:
• Monitor jobs and processes to ensure they run smoothly and meet performance benchmarks.
• Identify bottlenecks, inefficiencies, and areas for optimization within workflows and take proactive steps to enhance their efficiency.
• Continuous Learning and Innovation:
• Stay updated on industry trends, best practices, and emerging technologies in data integration, ETL, and data architecture.
• Identify opportunities to innovate and improve existing ETL processes, tools, and methodologies.

Qualifications:

• Bachelor’s or Master’s degree in computer science, Information Systems, or a related field.
• Proven experience designing and implementing Azure services and data integration solutions.
• Proficiency in programming languages like SQL, Python, and familiarity with ETL tools such as Apache Spark, Talend, Informatica, etc.
• Strong understanding of data modeling, data warehousing, and data architecture principles.
• Knowledge of data governance, data quality, and metadata management practices.
• Excellent problem-solving skills and ability to troubleshoot and optimize ETL processes.
• Effective communication skills to convey technical concepts to non-technical stakeholders.
• Experience with cloud platforms (e.g., AWS, Azure, GCP) and big data technologies is a plus.