AWS Data Engineer

Role: AWS Data Engineer

Location: Torrance , CA (Day 1 Onsite)

Rate : $75/hr

Job Description:

• 5+ years of experience in data engineering, database design, ETL processes, and data warehousing.
• 3+ years of experience with AWS tools and technologies (S3, EMR, Glue, Athena, RedShift, RDS, Spectrum and Airflow)
• 2+ years of experience with CI/CD tools.
• Strong knowledge of data storage and processing technologies, including databases and data lakes based distributed computing frameworks (e.g., Hadoop, Spark).
• 3+ in programming languages such as Python, Java, or Scala.
• Nice to have Informatica Cloud experience (IDMC)

Detailed Responsibilities:

Develop and Maintain Data Integration Solutions:

o Design and implement data integration workflows using AWS Glue/EMR,AWS MWAA(Airflow), Lambda, Redshift

o Demonstrate proficiency in Pyspark, Apache Spark and Python for data processing large datasets

o Ensure data is accurately and efficiently extracted, transformed, and loaded into target systems.

Ensure Data Quality and Integrity:

o Validate and cleanse data to maintain high data quality.

o Ensure data quality and integrity by implementing monitoring, validation, and error handling mechanisms within data pipelines

Optimize Data Integration Processes:

o Enhance the performance, optimization of data workflows to meet SLAs, scalability of data integration processes and cost-efficiency on AWS cloud infrastructure.

o Identify and resolve performance bottlenecks, fine-tuning queries, and optimizing data processing to enhance Redshift's performance

o Regularly review and refine integration processes to improve efficiency.

Support Business Intelligence and Analytics:

o Translate business requirements to technical specifications and coded data pipelines

o Ensure timely availability of integrated data for business intelligence and analytics.

o Collaborate with data analysts and business stakeholders to meet their data requirements.

Maintain Documentation and Compliance:

o Document all data integration processes, workflows, and technical & system specifications.

o Ensure compliance with data governance policies, industry standards, and regulatory requirements.

Do

1. Instrumental in understanding the requirements and design of the product/ software

• Develop software solutions by studying information needs, studying systems flow, data usage and work processes
• Investigating problem areas followed by the software development life cycle
• Facilitate root cause analysis of the system issues and problem statement
• Identify ideas to improve system performance and impact availability
• Analyze client requirements and convert requirements to feasible design
• Collaborate with functional teams or systems analysts who carry out the detailed investigation into software requirements
• Conferring with project managers to obtain information on software capabilities

2. Perform coding and ensure optimal software/ module development

• Determine operational feasibility by evaluating analysis, problem definition, requirements, software development and proposed software
• Develop and automate processes for software validation by setting up and designing test cases/scenarios/usage cases, and executing these cases
• Modifying software to fix errors, adapt it to new hardware, improve its performance, or upgrade interfaces.
• Analyzing information to recommend and plan the installation of new systems or modifications of an existing system
• Ensuring that code is error free or has no bugs and test failure
• Preparing reports on programming project specifications, activities and status
• Ensure all the codes are raised as per the norm defined for project / program / account with clear description and replication patterns
• Compile timely, comprehensive and accurate documentation and reports as requested
• Coordinating with the team on daily project status and progress and documenting it
• Providing feedback on usability and serviceability, trace the result to quality risk and report it to concerned stakeholders

3. Status Reporting and Customer Focus on an ongoing basis with respect to project and its execution

• Capturing all the requirements and clarifications from the client for better quality work
• Taking feedback on the regular basis to ensure smooth and on time delivery
• Participating in continuing education and training to remain current on best practices, learn new programming languages, and better assist other team members.
• Consulting with engineering staff to evaluate software-hardware interfaces and develop specifications and performance requirements
• Document and demonstrate solutions by developing documentation, flowcharts, layouts, diagrams, charts, code comments and clear code
• Documenting very necessary details and reports in a formal way for proper understanding of software from client proposal to implementation
• Ensure good quality of interaction with customer e-mail content, fault report tracking, voice calls, business etiquette etc
• Timely Response to customer requests and no instances of complaints either internally or externally

Deliver