The Lead Data Engineer is responsible for designing, developing, and implementing data processing pipelines using Spark and Scala, integrating Elasticsearch, optimizing data workflows, and collaborating with multi-disciplinary teams to ensure efficient data infrastructure and performance.
- Design, develop, and implement Spark Scala applications and data processing pipelines to process large volumes of structured and unstructured data.
- Integrate Elasticsearch with Spark to enable efficient indexing, querying, and retrieval of data.
- Optimize and tune Spark jobs for performance and scalability, ensuring efficient data processing and indexing in Elasticsearch.
- Collaborate with data engineers, data scientists, and other stakeholders to understand requirements and translate them into technical specifications and solutions.
- Design and deploy data engineering solutions on OpenShift Container Platform (OCP) using containerization and orchestration techniques.
- Optimize data engineering workflows for containerized deployment and efficient resource utilization.
- Collaborate with DevOps teams to streamline deployment processes, implement CI/CD pipelines, and ensure platform stability.
- Monitor and optimize data pipeline performance, troubleshoot issues, and implement necessary enhancements.
- Strong expertise in designing and developing data infrastructure using Hadoop, Spark, and related tools (HDFS, Hive, Ranger, etc)
- Experience with containerization platforms such as OpenShift Container Platform (OCP) and container orchestration using Kubernetes.
- Proficiency in programming languages commonly used in data engineering, such as Spark, Python, Scala, or Java.
- Knowledge of DevOps practices, CI/CD pipelines, and infrastructure automation tools (e.g., Docker, Jenkins, Ansible, BitBucket)
- Experience with Grafana, Prometheus, Splunk will be an added benefit
Top Skills
Java
Python
Scala
Spark
Unison Consulting Singapore Office
1 Changi Business Park Crescent, , Plaza 8 #03-06 Tower A, Singapore, , Singapore, 486025
Unison Consulting Singapore Office
#12-00, 63 Market Street, Bank of Singapore Center, Singapore, , Singapore, 048942
Similar Jobs
Information Technology • Consulting
As a Lead Data Engineer, you will design and develop data infrastructure, manage a team of data engineers, and optimize data processing pipelines utilizing technologies like Spark and Elasticsearch. You will also oversee data engineering solutions on OpenShift, collaborate with various teams, and provide mentorship to junior members.
Top Skills:
JavaPythonScalaSpark
Mobile • Software
The Lead Data Engineer will lead the development of data warehouse systems, ensuring scalability and integration with various business lines. Key responsibilities include designing technical specifications, guiding a team of data engineers, establishing best practices, and creating dashboards for system metrics.
Top Skills:
PythonScalaSQL
Software
The Principal Technical Lead will oversee the Aviatrix Data Pipeline team, focusing on designing scalable systems for collecting, processing, and analyzing network data. Responsibilities include leading a team, optimizing data pipelines, ensuring data quality, and integrating analytics tools.
Top Skills:
GoPython
What you need to know about the Singapore Tech Scene
The digital revolution has driven a constant demand for tech professionals across industries like software development, data analytics and cybersecurity. In Singapore, one of the largest cities in Southeast Asia, the demand for tech talent is so high that the government continues to invest millions into programs designed to develop a talent pipeline directly from universities while also scaling efforts in pre-employment training and mid-career upskilling to expand and elevate its workforce.
