The Data Engineer will develop and optimize ETL processes using PySpark and Databricks, design and implement data models in AWS, monitor data pipelines for quality, utilize GitHub for version control, and contribute to CI/CD practices and documentation.
- Assist in the development and optimization of ETL processes using PySpark and Databricks.
- Collaborate with team members to design and implement data models and architecture in AWS.
- Maintain and monitor data pipelines to ensure data quality and reliability.
- Utilize GitHub for version control and collaboration on codebases.
- Participate in code reviews and contribute to best practices in coding and data engineering.
- Support the team in implementing CI/CD practices for data workflows.
- Document processes and workflows for data engineering tasks.
- Familiarity with AWS services (e.g., S3, Redshift, Glue) and cloud computing concepts.
- Basic knowledge of Databricks and PySpark for data processing.
- Understanding of Git and experience with GitHub for version control.
- Awareness of DevOps principles and tools (e.g., CI/CD, Docker) is a plus.
- Databricks certification is a plus.
Top Skills
Pyspark
Unison Consulting Singapore Office
1 Changi Business Park Crescent, , Plaza 8 #03-06 Tower A, Singapore, , Singapore, 486025
Unison Consulting Singapore Office
#12-00, 63 Market Street, Bank of Singapore Center, Singapore, , Singapore, 048942
Similar Jobs
Be an Early Applicant
As a Lead Data Engineer, you will design and develop data infrastructure, manage a team of data engineers, and optimize data processing pipelines utilizing technologies like Spark and Elasticsearch. You will also oversee data engineering solutions on OpenShift, collaborate with various teams, and provide mentorship to junior members.
Be an Early Applicant
As a Junior Data Engineer, you will design, develop, and maintain data pipelines using Hadoop and Spark. You will collaborate with teams to ensure data quality, and implement CI/CD processes with DevOps. The role requires optimizing workflows for containerized deployment and mentoring junior members.
Be an Early Applicant
The Data Engineer (Python) will be responsible for ingesting data from various sources, curating data assets, collaborating with teams, deploying ML models, and architecting data pipelines to enable informed decision-making. The role focuses on enhancing the user experience through data-driven solutions.
What you need to know about the Singapore Tech Scene
The digital revolution has driven a constant demand for tech professionals across industries like software development, data analytics and cybersecurity. In Singapore, one of the largest cities in Southeast Asia, the demand for tech talent is so high that the government continues to invest millions into programs designed to develop a talent pipeline directly from universities while also scaling efforts in pre-employment training and mid-career upskilling to expand and elevate its workforce.
