Job Titles: DataBricks Data Engineer
Location: Atlanta, GA or Chicago, IL Hybrid need candidate under 50 miles
Duration: FTE conversion after 3 - 06months
Interview:- Virtual
Required Skills
Azure Databricks data engineering (Databricks Data Engineer Professional)
Understanding of Databricks Unity Catalog (including Delta Lake/Delta tables)
Understanding of structured, unstructured, and semi-structured data processing
Understanding of financial services cyber security and fraud data domain
Highly proficient Python and SQL skills
PySpark and Spark SQL
Preferred Experience:
3+ years of Databricks data engineering experience (Databricks Data Engineer Professional)
7+ years of experience in data engineering pipelines for analytics-must include batch and streaming data
2+ years working in a financial services or security related data domain
Key Talents
Adaptability: Data engineering requirements will change frequently as analytical needs and priorities flex. The data engineer who fills this role must be able to understand and quickly adapt to new requests
Conscientiousness: it is critical the person who fills this role takes in upon themselves to ensure that the output of their pipelines are what is practically needed by the business users and analytics engineers downstream. This person must be able to interpret specification in the context of business needs and work to clarify specifications to ensure that they serve the practical business need."
Job Responsibilities
Work closely with business stakeholders, analytics architects, and AI engineers to help exploratory and production application data preparation needs. Data customers will include human analysts, analytics engineers, AI and analytical models, dashboarding applications, and data stores.
Set up pipelines to ingest, store, clean, merge, transform, and aggregate data from multiple external and internal data sources
Set up both batch and streaming pipelines
Build data pipelines and transformations in Azure Databricks