The Data Engineer will design, develop, and maintain data pipelines using technologies like Hadoop and Spark. Responsibilities include ensuring data quality, optimizing workflows, deploying solutions on OpenShift, and collaborating with teams to implement monitoring and CI/CD pipelines. Mentoring junior team members is also part of the role.
We are seeking a skilled and motivated Data Engineer with expertise in Hadoop, Spark, OpenShift Container Platform (OCP), and DevOps practices. As a Data Engineer, you will be responsible for designing, developing, and maintaining efficient data pipelines, processing large-scale datasets. Your expertise in Hadoop, Spark, OCP, and DevOps will be crucial in ensuring the availability, scalability, and reliability of our ML Solutions.
Responsibilities:
- Implement data transformation, aggregation, and enrichment processes to support various data analytics and machine learning initiatives
- Collaborate with cross-functional teams to understand data requirements and translate them into effective data engineering solutions
- Ensure data quality and integrity throughout the data processing lifecycle
- Design and deploy data engineering solutions on OpenShift Container Platform (OCP) using containerization and orchestration techniques
- Optimize data engineering workflows for containerized deployment and efficient resource utilization
- Collaborate with DevOps teams to streamline deployment processes, implement CI/CD pipelines, and ensure platform stability
- Implement data governance practices, data lineage, and metadata management to ensure data accuracy, traceability, and compliance
- Monitor and optimize data pipeline performance, troubleshoot issues, and implement necessary enhancements
- Implement monitoring and logging mechanisms to ensure the health, availability, and performance of the data infrastructure
- Document data engineering processes, workflows, and infrastructure configurations for knowledge sharing and reference
- Stay updated with emerging technologies, industry trends, and best practices in data engineering and DevOps
- Provide technical leadership, mentorship, and guidance to junior team members to foster a culture of continuous learning and innovation to the continuous improvement of the analytics capabilities within the bank
- Bachelor's degree in Computer Science, Information Technology, or a related field
- Proven 2-5 Years of experience as a Data Engineer, working with Hadoop, Spark, and data processing technologies in large-scale environments
- trong expertise in designing and developing data infrastructure using Hadoop, Spark, and related tools (HDFS, Hive, Pig, etc)
- Experience with containerization platforms such as OpenShift Container Platform (OCP) and container orchestration using Kubernetes
- Proficiency in programming languages commonly used in data engineering, such as Spark, Python, Scala, or Java
- Knowledge of DevOps practices, CI/CD pipelines, and infrastructure automation tools (e.g., Docker, Jenkins, Ansible, BitBucket)
- Experience with Graphana, Prometheus, Splunk will be an added benefit
- trong problem-solving and troubleshooting skills with a proactive approach to resolving technical challenges
- Excellent collaboration and communication skills to work effectively with cross-functional teams
- Ability to manage multiple priorities, meet deadlines, and deliver high-quality results in a fast-paced environment
- Experience with cloud platforms (e.g., AWS, Azure, GCP) and their data services is a plus
Top Skills
Java
Python
Scala
Unison Consulting Singapore Office
1 Changi Business Park Crescent, , Plaza 8 #03-06 Tower A, Singapore, , Singapore, 486025
Unison Consulting Singapore Office
#12-00, 63 Market Street, Bank of Singapore Center, Singapore, , Singapore, 048942
Similar Jobs
Financial Services
As a Data Engineer III at JPMorgan Chase, you will design and maintain scalable and secure data solutions, develop critical data pipelines, and analyze data risks. Collaborating with various stakeholders, you will support analytics teams while utilizing Agile DevOps practices and cloud technologies. Your role emphasizes building robust systems capable of handling large volumes of data and ensuring data accuracy and timeliness. You will also document processes and enhance team culture.
Top Skills:
Python
Fintech • Information Technology • Payments
The Data Engineer will design and develop systems for Visa's Loyalty & Marketing Services, focusing on high-availability and performance. Responsibilities include developing new products, enhancing existing systems, coordinating CI activities, and instilling best practices in software development. The role requires collaboration with architects and stakeholders to ensure high quality and timely delivery of products.
Top Skills:
JavaScala
Information Technology • Consulting
The Data Engineer will work hands-on with Cloudera Data Platform and monitoring tools. Responsibilities include troubleshooting complex issues, implementing high-performance data systems, and collaborating with cross-functional teams. Familiarity with cloud technologies and middleware applications is advantageous.
Top Skills:
BashPython
What you need to know about the Singapore Tech Scene
The digital revolution has driven a constant demand for tech professionals across industries like software development, data analytics and cybersecurity. In Singapore, one of the largest cities in Southeast Asia, the demand for tech talent is so high that the government continues to invest millions into programs designed to develop a talent pipeline directly from universities while also scaling efforts in pre-employment training and mid-career upskilling to expand and elevate its workforce.
