Apache Spark Lead - Sydney

Job Title: Apache Spark Lead - Sydney
Contract Type: Contract
Location: Sydney CBD, New South Wales
Salary: Negotiable
Reference: 4016_1615932622
Contact Name: Gaurav Bagga
Contact Email:
Job Published: March 17, 2021 09:10

Job Description

Ayan Infotech are looking for an Apache Spark-Lead for a long term contract job opportunity in Sydney.

All applicants must have full work rights in Australia.

Title: Apache Spark-Lead
Location: Sydney
Type: Contract (6 months + extensions)

Required Skills and Experience:

  • Creating Spark/Scala jobs for data aggregation and transformation, producing unit tests for Spark helper and transformations methods, using all code writing Scaladoc-style documentation, and design data processing pipelines.
  • Must have worked exhaustively in creating Dataframes and RDDs.
  • Must have good experience in connecting to various datasources like files, databases, json and other formats.
  • Must have good knowledge in using Scala as a programming language for Spark.
  • Have exposure to working in spark in EMR as a cluster.
  • Expected to run data on distributed SQL, creating data pipelines, ingesting data into a database
  • Have good experience in working in Parallelism and configuring the servers for effecient scaling.
  • Clean, transform and analyze raw data from various mediation sources using Spark to provide ready to use data
  • Use of refactoring code, so that joins are done efficiently and provide technical Spark platform architecture guidance.

Contact: 02 9412 4178 for more details.

Get similar jobs like these by email

By submitting your details you agree to our T&C's