The Data Engineer will design, develop, and implement Spark Scala applications to process large datasets, integrate Elasticsearch for data retrieval, and optimize Spark jobs for performance. This role involves collaborating with stakeholders to define requirements, implementing data transformations, maintaining scalable applications, and troubleshooting data processing issues.
- Design, develop, and implement Spark Scala applications and data processing pipelines to process large volumes of structured and unstructured data
- Integrate Elasticsearch with Spark to enable efficient indexing, querying, and retrieval of data
- Optimize and tune Spark jobs for performance and scalability, ensuring efficient data processing and indexing in Elasticsearch
- Collaborate with data engineers, data scientists, and other stakeholders to understand requirements and translate them into technical specifications and solutions
- Implement data transformations, aggregations, and computations using Spark RDDs, DataFrames, and Datasets, and integrate them with Elasticsearch
- Develop and maintain scalable and fault-tolerant Spark applications, adhering to industry best practices and coding standards
- Troubleshoot and resolve issues related to data processing, performance, and data quality in the Spark-Elasticsearch integration
- 5+ Years as Data Engineer
- 5 Years experience in Spark, Scala, Elastic Search
- Develop and maintain scalable and fault-tolerant Spark applications, adhering to industry best practices and coding standards
- Monitor and analyze job performance metrics, identify bottlenecks, and propose optimizations in both Spark and Elasticsearch components
- Stay updated with emerging trends and advancements in the big data technologies space to ensure continuous improvement and innovation
Top Skills
Scala
Spark
Unison Consulting Singapore Office
1 Changi Business Park Crescent, , Plaza 8 #03-06 Tower A, Singapore, , Singapore, 486025
Unison Consulting Singapore Office
#12-00, 63 Market Street, Bank of Singapore Center, Singapore, , Singapore, 048942
Similar Jobs
Be an Early Applicant
As a Senior Data Engineer, you will design, develop, and optimize big data solutions using Apache Spark, Scala, and Elasticsearch. You will implement data transformation processes, collaborate with teams to meet data requirements, and ensure data quality and integrity. The role requires optimizing job performance, deploying data engineering solutions on OpenShift, and monitoring data pipelines.
Be an Early Applicant
As a Data Engineer at our company, you will design, develop, and optimize big data solutions using technologies like Apache Spark and Scala. Your responsibilities include creating scalable data processing pipelines, ensuring data quality, optimizing performance, and collaborating with cross-functional teams to meet data requirements, all while utilizing Quantexa and Elasticsearch optimally.
Be an Early Applicant
The Data Engineer will design and implement Spark Scala applications and data processing pipelines, integrate Elasticsearch for efficient data querying, and optimize performance. Responsibilities include collaborating with stakeholders, developing fault-tolerant applications, and troubleshooting data quality issues.
What you need to know about the Singapore Tech Scene
The digital revolution has driven a constant demand for tech professionals across industries like software development, data analytics and cybersecurity. In Singapore, one of the largest cities in Southeast Asia, the demand for tech talent is so high that the government continues to invest millions into programs designed to develop a talent pipeline directly from universities while also scaling efforts in pre-employment training and mid-career upskilling to expand and elevate its workforce.
