Job Description
Ayan Infotech are looking for an Apache Spark-Lead for a long term contract job opportunity in Sydney.
All applicants must have full work rights in Australia.
Title: Apache Spark-Lead
Location: Sydney
Type: Contract (6 months + extensions)
Required Skills and Experience:
- Creating Spark/Scala jobs for data aggregation and transformation, producing unit tests for Spark helper and transformations methods, using all code writing Scaladoc-style documentation, and design data processing pipelines.
- Must have worked exhaustively in creating Dataframes and RDDs.
- Must have good experience in connecting to various datasources like files, databases, json and other formats.
- Must have good knowledge in using Scala as a programming language for Spark.
- Have exposure to working in spark in EMR as a cluster.
- Expected to run data on distributed SQL, creating data pipelines, ingesting data into a database
- Have good experience in working in Parallelism and configuring the servers for effecient scaling.
- Clean, transform and analyze raw data from various mediation sources using Spark to provide ready to use data
- Use of refactoring code, so that joins are done efficiently and provide technical Spark platform architecture guidance.
Contact: 02 9412 4178 for more details.
www.ayaninfotech.com
