The Data Engineer will design, develop, and maintain scalable data pipelines and ETL processes using Hadoop and Spark, write optimized data processing jobs in Scala, and utilize SQL for data querying. Collaboration with data scientists is essential to deliver solutions that meet business needs, monitor workflows for performance, and document processes for knowledge sharing.
- Design, develop, and maintain scalable data pipelines and ETL processes using Hadoop and Spark.
- Write efficient and optimized data processing jobs in Scala to transform and load data.
- Utilize SQL for querying and manipulating data across various databases.
- Collaborate with data scientists and analysts to gather requirements and deliver data solutions that meet business needs.
- Monitor and troubleshoot data processing workflows to ensure high performance and reliability.
- Implement best practices for data quality, governance, and security.
- Document data engineering processes, workflows, and architecture for team knowledge sharing.
- Stay updated on industry trends and emerging technologies to continuously improve our data strategies.
- Bachelor’s degree in Computer Science, Engineering, or a related field.
- 3+ years of experience as a Data Engineer or in a similar role.
- Strong proficiency in Hadoop ecosystem components (e.g., HDFS, Hive, Pig).
- Hands-on experience with Apache Spark for big data processing.
- Proficient in Scala programming for data transformation tasks.
- Advanced SQL skills for querying and managing large datasets.
- Experience with data modeling, data warehousing, and ETL processes.
- Familiarity with cloud platforms (AWS, GCP, or Azure) is a plus.
- Strong analytical and problem-solving skills.
Top Skills
Hadoop
Scala
Spark
SQL
Unison Consulting Singapore Office
1 Changi Business Park Crescent, , Plaza 8 #03-06 Tower A, Singapore, , Singapore, 486025
Unison Consulting Singapore Office
#12-00, 63 Market Street, Bank of Singapore Center, Singapore, , Singapore, 048942
Similar Jobs
Be an Early Applicant
As a Data Engineer at our company, you will design, develop, and optimize big data solutions using technologies like Apache Spark and Scala. Your responsibilities include creating scalable data processing pipelines, ensuring data quality, optimizing performance, and collaborating with cross-functional teams to meet data requirements, all while utilizing Quantexa and Elasticsearch optimally.
Be an Early Applicant
As a Quantitative Researcher at Citadel's Global Quantitative Strategies, you will conduct advanced research utilizing mathematical, statistical, and computational methods to develop predictive models and optimize multi-asset portfolios. The role demands a strong background in data analysis and a passion for financial markets to drive impactful research solutions.
Be an Early Applicant
As a Trading Strategist Analyst at J.P. Morgan, you will analyze data to uncover revenue opportunities, develop models for trading problems, back test trading strategies, assess trading performance, and improve trading software. You will work closely with the trading desk to drive revenue in commodities markets.
What you need to know about the Singapore Tech Scene
The digital revolution has driven a constant demand for tech professionals across industries like software development, data analytics and cybersecurity. In Singapore, one of the largest cities in Southeast Asia, the demand for tech talent is so high that the government continues to invest millions into programs designed to develop a talent pipeline directly from universities while also scaling efforts in pre-employment training and mid-career upskilling to expand and elevate its workforce.
