Job Description :
Technical Experience Required: AWS,SCALA,KAFKA,Python, Apache Spark, and Hadoop
The client is looking for a Big Data Engineer to build and maintain an infrastructure for delivering customer insights from structured and unstructured data sources. The Data Engineer will implement analytical solutions with a focus on collecting, managing, modeling, analyzing, and visualizing data and develop batch and real-time processes to move data. Work in an Agile Scrum driven environment to deliver new and innovative products for Analytics customers, and keep up-to-date with relevant technology in order to maintain and improve functionality. Must be able to work across multiple divisions and teams to ensure the end result of the product.
Responsibilities:
Ability to work with senior IT and Business leaders providing technical guidance related to data architecture, data models and metadata management
Works closely with database teams on topics related to data requirements, cleanliness, and accuracy
Interacts with Business divisions to understand all data requirements to develop business insights and translates them into data structures and data model requirements to IT
Codes, test, and documents new or modified data systems to create robust and scalable applications for data analytics
Develop standards and processes for integration projects and initiatives
Collects, parses, manages analyzes and visualizes large sets of data using multiple platforms
Ensures that data pipelines are scalable, repeatable, secure, and can serve multiple users within the company
Qualifications
Bachelor's or Master's Degree in technical or business discipline or related experience; Master's Degree preferred
Experience of working with Big Data Systems such as Hadoop, Hive, Spark, etc.
Strong SQL Skills
Programming/scripting experience with languages such as Scala or Python
Demonstrated ability to manage and prioritize workload and roadmaps
Solid written and verbal communication skills
Familiarity with AWS solutions such as EMR, S3, and Redshift
3+ years of professional experience in data engineering or comparable position
5+ years of experience with advanced data management systems (Teradata, Hadoop)
5+ years of experience in developing applications in high volume data staging/ETL environments and is proficient in advanced SQL experience
Technical Experience Required: Python, Apache Spark, and Hadoop
A clear understanding of different data domains in operations and has experience in master data management
Strong analytical and problem-solving experience paired with the ability to develop creative and efficient solutions; tolerance in dealing with bad quality data
Knowledge of developing and maintaining a formal document that describes data and data structures including data modeling
Comfortable working in an unstructured environment that will require self-starters/motivators to achieve benchmarks
Agile and collaborative environment working to support the whole division
Nice to have technical experience: JIRA, AWS Glue, Redshift .