APN Consulting Inc
Monmouth Junction, NJ
Post Resume to
View Contact Details &
Apply for Job
Job Description :
APN Consulting has an immediate need for a direct client requirement:
Title - Platform Automation Engineer
Location: Raritan/Bridgwater/Trenton NJ
Contract
Remote/Hybrid(1 day onsite)
Note - Candidate needs to be 1 day onsite to any of the location (Raritan/Bridgwater/Trenton)
About the Role
We are seeking a skilled Platform Automation Engineer to design, implement, and maintain automated workflows and scalable solutions for our Databricks and Data Lake environments. You will play a pivotal role in optimizing data platform performance, ensuring data integrity, and enabling efficient data pipelines through automation.
Key Responsibilities
  1. Platform Automation & Management:
    • Develop and maintain Infrastructure-as-Code (IaC) solutions for Databricks and cloud-based Data Lakes using tools like Terraform or CloudFormation.
    • Automate deployment, scaling, and management of Databricks clusters and services.
    • Implement monitoring, logging, and alerting systems for platform performance and uptime.
  2. Data Engineering Support:
    • Collaborate with data engineering teams to design and deploy automated ETL pipelines for data ingestion, transformation, and processing.
    • Optimize Data Lake storage patterns and structures for high performance and cost efficiency.
  3. Cloud Integration:
    • Integrate Databricks and Data Lake solutions with cloud platforms (AWS, Azure, GCP).
    • Configure secure connections for external data sources and downstream applications.
  4. Security & Compliance:
    • Implement automated security controls, ensuring compliance with organizational and regulatory standards.
    • Conduct regular audits to verify access controls, encryption, and data integrity.
  5. Continuous Improvement:
    • Evaluate new tools and technologies to enhance platform efficiency and automation.
    • Design and implement CI/CD pipelines for Databricks notebooks and data workflows.
Required Qualifications
  • Education: Bachelor's degree in Computer Science, Engineering, or related field.
  • Experience:
    • 3+ years of experience in platform automation, infrastructure engineering, or DevOps roles.
    • Hands-on experience with Databricks, Spark, and large-scale data platforms.
    • Expertise in Data Lake architecture and management (e.g., Delta Lake, Apache Iceberg).
    • Proficiency in scripting/programming languages (Python, Scala, Shell).
    • Strong knowledge of cloud platforms (AWS S3, Azure Data Lake, GCP BigQuery).
Preferred Skills
  • Familiarity with orchestration tools like Apache Airflow or Azure Data Factory.
  • Experience with containerization and Kubernetes for scalable application deployments.
  • Certification in Databricks, AWS, Azure, or GCP.


             

Similar Jobs you may be interested in ..