Ayan Infotech are looking for multiple Pyspark Consultants for either full time permanent positions or long term contract job opportunities in Melbourne.
All applicants must have full unrestricted work rights in Australia.
Title: Pyspark Consultant
Location: Melbourne (on-site only)
Type: Permanent OR Long term contract (both options are available)
Required Skills and Experience:
- 5 or more years of experience in developing Spark applications using Spark-SQL in Databricks for data extraction, transformation, and aggregation from multiple file formats for Analyzing& transforming the data to uncover insights into the customer usage patterns.
- Extract Transform and Load data from sources Systems to Azure Data Storage services using a combination of Azure Data factory, T-SQL, Spark SQL, and U-SQL Azure Data Lake Analytics.
- Data ingestion to one or more Azure services (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in Azure Databricks
- Develop Spark applications using pyspark and spark SQL for data extraction, transformation, and aggregation from multiple file formats for analyzing and transforming the data uncover insight into the customer usage patterns
- Hands on experience on developing SQL Scripts for automation
- Responsible for estimating the cluster size, monitoring, and troubleshooting of the Spark databricks cluster
- Ability to apply the spark DataFrame API to complete Data manipulation within spark session
- Good understanding of Spark Architecture including spark core, spark SQL, DataFrame, Spark streaming, Driver Node, Worker Node, Stages, Executors and Tasks, Deployment modes, the Execution hierarchy, fault tolerance, and collection
Contact: 02 9412 4178 for more details.