Job Description :
Job Description:
Pyspark and Python expert for design and coding
10+ years of lead developer experience
4+ years of Pyspark and Python hands coding experience
Responsible for design and development. Also must have design approach to build the reusable components
Setup Azure Python/PySpark on data bricks environment and optimization
Must have experience in converting existing tradition ETL code (data stage, informatica) into databricks PySpark/Python
Should able to cover perl code into PySpark
Implement complex ETL by leveraging PySpark and Python
Must able to load data from Azure blob or ADLS or Data bricks to SQL DW system. Load Performance is a key element
Must able to implement complex ETL login by leveraging Spark/scala
Good Hive experience
Need to work with customer directly with minimal directions
Need to mentor new teams with different background
Must need to work from client location
Good to have:
Good to have Spark/Scala experience
Experience in building reusable frameworks (data quality, audit trails, error logs etc)
Experience to handle job restartability (start a job from the failure, not from the beginning)