Job Description :
Role: Bigdata Engineer
Location: Morristown, NJ

Senior Data Engineer is responsible for supporting the data ingest and integration needs for the TPS Security.
The engineer must provide technical expertise and leadership on architecture, design, and integration of multiple data sets into our big data environment.

Contributes design, configuration, deployment, and documentation for components that manage data ingestion, real time streaming, batch processing, data extraction, transformation, enrichment, and loading of data into a variety of data platforms, including (Hadoop, AWS S3, Spark, Redshift, etc
Identifies gaps and improves the existing platform to improve quality, robustness, maintainability, and speed.
Mentors junior and senior engineers, provides code reviews, feedback, and enables professional growth. Required Experience
Experience building, maintaining, and improving Data Processing Pipeline / Data routing in large scale environments.
Fluency in common query languages, API development, data transformation, and integration of data streams.
Strong experience with large dataset platforms such as (Spark, hive, Impala etc
Fluency in multiple programming languages, such as Python, Shell Scripting, Regex, SQL, Java, or similar languages and tools appropriate for large scale data processing.
Experience with acquiring data from varied sources such as: API, data queues, flat-file, remote databases
Must have basic Linux administration skills and Multi-OS familiarity (Windows / OSX)
Data Pipeline and Data processing experience using common platforms and environments such as Apache Spark ML, Apache Nifi, Flink, Pentaho DI, Talend, etc.