Design and implement scalable data pipelines on Databricks, optimize data processing, collaborate with stakeholders, and ensure data quality and compliance.
Job Summary:
We are seeking a skilled and motivated Databricks Data Engineer to design, build, and maintain scalable data pipelines and data platforms using Databricks on Azure/AWS. You will work closely with data scientists, analysts, and business stakeholders to enable data-driven decision-making across the organization.
Key Responsibilities:- Design and implement robust, scalable, and high-performance data pipelines using Apache Spark on Databricks.
- Develop and maintain ETL/ELT processes using Databricks Notebooks, Delta Lake, and Spark SQL.
- Optimize and troubleshoot large-scale distributed data processing jobs.
- Collaborate with data scientists and analysts to deliver data products that support analytics, reporting, and machine learning use cases.
- Develop and enforce best practices in data engineering and DevOps processes.
- Implement CI/CD pipelines for Databricks workflows using tools such as Azure DevOps, GitHub Actions, or Terraform.
- Ensure data quality, integrity, security, and compliance across platforms.
- Support migration of data systems to the cloud or from legacy environments to Databricks.
Required Qualifications:
- Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
- 3+ years of experience in data engineering or related roles.
- Proficiency with Apache Spark and the Databricks platform.
- Strong experience with Python, SQL, and PySpark.
- Experience with Delta Lake, Lakehouse Architecture, and data partitioning strategies.
- Hands-on experience with cloud platforms (Azure, AWS, or GCP).
- Familiarity with Databricks Workflows, Unity Catalog, and DBFS.
- Experience working with structured and unstructured data at scale.
- Good understanding of data warehousing, data modeling, and data governance concepts.
- Databricks certification (e.g., Databricks Certified Data Engineer Associate or Professional).
- Experience integrating Databricks with Power BI, Tableau, or other BI tools.
- Experience with Terraform, Databricks REST APIs, and infrastructure-as-code (IaC).
- Knowledge of MLflow, Feature Store, and MLOps best practices is a plus.
- Experience with streaming technologies like Structured Streaming or Kafka.
Top Skills
Spark
AWS
Azure
Azure Devops
Databricks
Delta Lake
Elt
ETL
Github Actions
Pyspark
Python
Spark Sql
SQL
Terraform
Unison Consulting Singapore Office
1 Changi Business Park Crescent, , Plaza 8 #03-06 Tower A, Singapore, , Singapore, 486025
Unison Consulting Singapore Office
#12-00, 63 Market Street, Bank of Singapore Center, Singapore, , Singapore, 048942
Similar Jobs
Artificial Intelligence • Fintech • Payments • Financial Services • Generative AI
Lead the development of technical elements for the issuing network, focusing on high-quality infrastructure and API design while mentoring engineers and driving strategic technology adoption.
Top Skills:
AWSDockerGCPGoGrafanaJavaKafkaKubernetesPrometheusPythonSplunk
Financial Services
As a Software Engineer II, you'll enhance and deliver technology solutions, write and debug code, automate processes, and ensure software stability.
Top Skills:
AWSCloud FoundryGemfireJavaJava 17OracleSpringSpring Boot
Aerospace • Hardware • Internet of Things • Robotics • Wearables • App development • Automation
The Senior Frontend Application Developer will design and optimize user interfaces for mobile and web applications, lead frontend architecture, implement CI/CD automation, and collaborate across teams.
Top Skills:
AngularBabelCSSFlutterGitHTMLJavaScriptReactReact NativeVueWebpack
What you need to know about the Singapore Tech Scene
The digital revolution has driven a constant demand for tech professionals across industries like software development, data analytics and cybersecurity. In Singapore, one of the largest cities in Southeast Asia, the demand for tech talent is so high that the government continues to invest millions into programs designed to develop a talent pipeline directly from universities while also scaling efforts in pre-employment training and mid-career upskilling to expand and elevate its workforce.
