Job Description :
Big Data developer:
In depth knowledge of Big Data concepts and common components including Hadoop Components (Pig, Hive, Kafka, Storm, MapReduce, Hbase,), Spark, Cloud Components (Azure) and multiple languages (Java, Scala, Python
RESPONSIBILITIES
Design, build and maintain Big Data workflows/pipelines to process billions of records into and out of our data lake and Identity Graph
Fine tune application performance
Troubleshoot and resolve data processing issues
Engage in application design and data modeling discussions
Participate in developing and enforcing data security policies
Participate in capacity monitoring and planning
Build, maintain and execute unit test cases with high code coverage QUALIFICATIONS
BS/BA degree in Computer Science, Information Systems or related field
Expert in HIVE SQL and ANSI SQL - Great hands on in Data Analysis using SQL.
2 years programming in Spark, Java, Python
2 years developing on Hadoop/Spark
2 years developing on an RDBMS such as Microsoft SQL Server, MySQL or Oracle
Experience with large data sets
Exposure to data hygiene routines and models
Experience in database design, development and data modeling
Ability to identify problems, and effectively communicate solutions to team
Ability to work in a dynamic multi-team environment as well as independently ADDED VALUABLE SKILLS
Hadoop: HDFS, MapReduce, Hive, Pig
Experience on Azure platform
Data architecture
Database security
ETL using SQL or a scripting/programming language (DataFactory is a plus)
Experience with fuzzy-logic matching and tools
NoSQL: HBase, Azure Admin Responsibilities & Duties
Manage the Accelerated Analytics Azure environments (Development, Testing and Production
Monitor performance on all environments (CPU, disk space, etc to ensure smooth running of all environments.
Advise on, and implement, different environments for different requirements, including (but not limited to):
Different setups for development, testing and production environments.
Very intensive data processing environments.
Environments with and without external OLAP access.
Light data processing environments.
Small one-off projects.
Research and development projects.
Physical versus cloud-based SQL databases.
Scalable solutions.
All aspects of cloud security.
Manage remote access to servers, based on different user roles and access requirements.
Manage cloud-based FTP and other forms of file sharing.
Backups, restorations and disaster recovery.
Research into, and implementation of, new Azure tools to improve efficiency, performance, and cost-effectiveness.
Monitor and report to management on actual and projected monthly costs. Requirements
At least three years working as an Azure system administrator.
Experience implementing Microsoft SQL Server, SSIS, SSRS, SSAS, Power BI on Azure.
Experience with the full Software Development Life Cycle (SDLC
Excellent logical and problem-solving abilities.
Bachelor’s degree in computer science or mathematics.
Microsoft training and certification will be considered a strong asset.
Experience working in a fast-paced environment.
Excellent English communication skills – both verbal and written.