AYAN InfoTech is looking for Databricks Data Engineer to join an exciting project based in Sydney. The role offers you the opportunity to contribute towards an extremely well structured and mature environment, working on sophisticated enhancement projects.
Role: Databricks Data Engineer
Location: Sydney
Contract Duration: 6 Months with high possible extensions
Experience: 5+ Years
Job Description:
- Minimum 5+ years of IT experience with 3+ years of hands-on experience in designing, developing, implementing, and maintaining data solutions with Databricks.
- This role requires close collaboration with various teams to gather data requirements, build scalable data pipelines, and ensure the overall solution is reliable, available, and optimized for high performance.
- This role requires close collaboration with various teams to gather data requirements, build scalable data pipelines, and ensure the overall solution is reliable, available, and optimized for high performance.
- Proven expertise with Lakehouse Architecture with Delta Tables - Delta tables, Schema Evolution, ACID compatibility, History capture etc.
- Integrating Databricks with various ETL/Orchestration tools and perform required operations.
- Strong expertise in fundamental areas like Spark ecosystems, Data frame API & Spark SQL with Python/Scala and SQL as language
- Knowledge of latest Databricks features.
- Hands on knowledge on Databricks native features like Workflows, Delta Live Tables, Unity CatLog etc.
- Expertise in workspace and cluster configuration, performance tuning, cost optimization, data security, and log monitoring.
- Experience with Spark Structured Streaming jobs with Autoloader integrating with different streaming platforms.
- Experience in building optimal data pipeline architecture with knowledge and best practices from Databricks platform.
- Fair understanding of Data Modelling and defining conceptual logical and physical data models.
- Knowledge of different Azure Could services
Contact: 61-(02) 7207 6926 for more details.
Please note we will be able to contact only shortlisted candidates for this role. We thank you in advance for your interest.
