Design, build and maintain data processing pipelines using Databricks, collaborate with teams for data requirements, and ensure data quality.
As a Data Engineer specializing in Databricks, you will be instrumental in designing, building, and maintaining robust data processing pipelines leveraging the Databricks platform. You will work closely with data scientists and analysts to ensure the seamless integration and transformation of data to facilitate analytical needs and enhance decision-making across the organization.
Key Responsibilities:
- Data Pipeline Development: Design and implement scalable data pipelines using Databricks, ensuring efficient and reliable data processing from various sources.
- Collaboration: Work closely with data science and analytics teams to understand data requirements, providing support for data-driven projects and initiatives.
- Data Transformation: Utilize Spark SQL and Python to transform and clean large datasets, making them ready for analysis and reporting.
- Performance Optimization: Monitor and optimize the performance of data processing workflows within Databricks to ensure high availability and scalability.
- Data Quality Assurance: Implement data quality checks and validation processes to ensure the accuracy and integrity of data ingested into the system.
- Documentation: Create and maintain comprehensive documentation for data pipelines, architecture, and processes to facilitate knowledge sharing within the team.
- Bachelor's degree in Computer Science, Data Engineering, or a related field.
- Proven experience as a Data Engineer with a focus on Databricks and Spark.
- Strong proficiency in SQL and Python for data manipulation and analysis.
- Experience with data modeling, ETL processes, and data warehousing solutions.
- Familiarity with cloud platforms (AWS, Azure, or GCP) is preferred.
- Excellent problem-solving skills and attention to detail.
- Strong communication and collaboration abilities.
- Salary: SGD 6000 - 9000
- Great Work Culture
- Includes Standard Annual and Sick Leave
Top Skills
AWS
Azure
Databricks
GCP
Python
Spark Sql
SQL
Unison Consulting Singapore Office
1 Changi Business Park Crescent, , Plaza 8 #03-06 Tower A, Singapore, , Singapore, 486025
Unison Consulting Singapore Office
#12-00, 63 Market Street, Bank of Singapore Center, Singapore, , Singapore, 048942
Similar Jobs
Information Technology • Software • Financial Services
As a Quantitative Researcher, you will develop and enhance trading models, conduct statistical research, and drive innovation in automated trading strategies.
Top Skills:
Machine LearningMathematical ModelingNatural Language ProcessingPattern RecognitionStatistical TechniquesTime-Series Analysis
Information Technology • Software • Financial Services
As a Quantitative Researcher Intern, you'll develop trading models, conduct statistical analysis, and translate mathematical algorithms into code while collaborating with senior researchers.
Top Skills:
C++PythonR
Information Technology • Software • Financial Services
The Quantitative Research Analyst designs and tests automated trading strategies using statistical models and conducts research to enhance trading systems.
Top Skills:
Machine LearningNlpPattern RecognitionStatistical TechniquesTime-Series Analysis
What you need to know about the Singapore Tech Scene
The digital revolution has driven a constant demand for tech professionals across industries like software development, data analytics and cybersecurity. In Singapore, one of the largest cities in Southeast Asia, the demand for tech talent is so high that the government continues to invest millions into programs designed to develop a talent pipeline directly from universities while also scaling efforts in pre-employment training and mid-career upskilling to expand and elevate its workforce.
