Job Description :
Position: Sr. Data Engineer
Location: New York / New Jersey
Project
Architect and build data platform to support pre-trade and post-trade market transparency for equity and fixed income markets. The system is designed to process large volumes of trade
order data feeds from various applications, messaging platforms, files, and emails. The data
pipelines acquire data from many sources, ingest to a data lake, analyze in an in-memory data
processing cluster, and store in high performance columnar data stores. The challenge is to
converge batch and stream processing, high speed ingestion, and on demand data analysis.
Ideal Candidate
Expertise to architect with experience in data lakes
Hands-on experience with techniques for data pipeline acceleration
Expertise to architect high volume streaming applications
Hands-on experience in profiling to improve performance and latency
Requirements
10+ years of architecture of distributed systems and data warehousing
Strong hands-on programming using Java, Scala, RDBMS and SQL
3+ years of deep understanding and programming on Hadoop (HDP)
2+ years of programming using Spark 1.6 – 2.x (Core Spark, Spark SQL)
Expert in architecting streaming applications (Spark streaming and Kafka)
Expert in ingestion frameworks - Flume, Sqoop, Oozie, Hive, Nifi, Kafka connect
Expert in HBase, Phoenix, and data storage formats (parquet, avro, orc)
Experience in search (Solr) and Spark integration
Expertise in high performance REST based web service frameworks
Experience in CI (Jenkins, maven, gitflow) and CD (docker, kubernetes) pipelines
Familiarity with Agile development using SCRUM and XP methodologies
Nice to have
Certification in Hortonworks and Apache Spark 2.0
Certification in Confluent Kafka ecosystem