Job Description :
Summary:
Minimum 6 years’ experience developing scalable analyses in a real world problem domain.
Ability to contribute to multiple projects/demands simultaneously and self-prioritize. Knowledge of fundamental CS concepts, i.e., algorithms and data structures.
Cloud visualization API Processes the data in the Spark.
Takes SQL query as an input and generates the output of the processed data AWS S3 buckets. Implemented Kafka .
Published messages into JSON schema.
Scheduled Spark Launcher: Developed cloud automation framework .
Loads the data from S3 using AWS SDK API into Spark SQL for the Scala Spark job which processes the data and stores the target data into Mongo DB Strong math, statistics, analytical, and problem solving skills.
2+ Years of spark Experience .
Responsibilities:
Analyzing terabyte-scale datasets, both structured and unstructured, from a variety of sources.
Responsible for developing the real-time processing of Wi-Fi data logs using Spark. Should have a deep understanding of Java and is expected to perform complex data transformations in Spark using Scala language.
Experienced in handling large datasets using Partitions, Spark in Memory capabilities, Broadcasts in Spark, Effective & efficient Joins, Transformations and other during ingestion process itself.
Experienced in performance tuning of Spark Applications for setting right Batch Interval time, correct level of Parallelism and memory tuning.
Consumed the data from Kafka queue using Spark. Configured different topologies for spark cluster and deployed them on regular basis.
Used Java/J2EE for the development of business layer services.
Build a custom processer in Apache NIFI is must. Excellent Java development skills using J2EE, J2SE, Servlets, JSP, EJB, JDBC.
Experience in designing and developing POCs in Spark using Scala to compare the performance of Spark with Hive and SQL/Oracle.
End-to-end problem analysis, design, implementation, and presentation of findings Building maintainable data solutions, including documentation.
Working in Java/Scala functional classes development. Design and Development & Coding .