Develop and optimize scalable data pipelines using AWS and Python, maintaining data quality and collaborating with analytics teams for business intelligence.
JOB DESCRIPTION
- Develop, maintain scalable data pipelines and build out new integrations to support continuing increases in data volume and complexity.
- Develop and maintain scalable, optimized data pipelines leveraging Python and AWS services to support increasing data volume and complexity, while ensuring seamless integration with AI platforms like Bedrock and Google. Further enhance data accessibility and drive data-driven decision making by collaborating with analytics and business teams to refine data models for business intelligence tools.
- Develop, maintain, and optimize scalable data pipelines using Python and AWS services (e.g., S3, Lambda, ECS, EKS, RDS, SNS/SQS, Vector DB).
- Rapidly developing next-generation scalable, flexible, and high-performance data pipelines.
- Collaborate with analytics and business teams to create and improve data models for business intelligence.
- End-to-end ownership of data quality in our core datasets and data pipelines.
- Participate in code reviews and contribute to DevOps / DataOps / MLOps.
JOB REQUIREMENTS
Requirements:
- Bachelor's degree in Computer Science, Engineering, or a related field.
- 2-3 years of experience in data engineering or a similar role.
- Strong programming skills in Python, SQL, AWS and related tech stack.
- Experience with building scalable data pipelines with technologies such as Glue, Airflow, Kafka, Spark etc..
- Experience using Snowflake, DBT, Bedrock is a plus.
- Good understanding of basic machine learning concepts (Sagemaker)
#LI-JL2
BUSINESS SEGMENT
CorporatePLATFORMOperating Division
Top Skills
Airflow
AWS
Bedrock
Dbt
Glue
Kafka
Python
Sagemaker
Snowflake
Spark
SQL
Keppel Singapore Office
1 HarbourFront Avenue, Level 2, Keppel Bay Tower, Singapore, 098632
Similar Jobs
Information Technology • Consulting
The Data Engineer will design and implement data platforms, manage data pipelines, ensure data quality, and mentor junior engineers, focusing on efficient data workflows and governance.
Top Skills:
AirflowAmundsenAWSBeamBigQueryBigtableDatabricksDataplexDbtDynamoDBFlinkGlue CatalogJavaKafkaMongoDBMySQLPostgresPower BIPubsubPythonQuicksightRedshiftScalaSnowflakeSparkSQLSupersetTableau
Sports
As a Data Engineer at Sporty, you will design and maintain data pipelines, ensure data quality, and work on data applications and architectures for machine learning initiatives.
Top Skills:
Apache AirflowSparkAws AthenaAws Ec2Aws EksAws EmrAws LambdaAws RdsAws RedshiftAws S3PythonSQL
Gaming • Hardware
The Data Engineer Intern will enhance data pipelines, perform data profiling, and ensure data accuracy for analytics purposes while learning data engineering skills.
Top Skills:
AirflowAWSDbtGoogle Cloud PlatformPythonSQLTerraform
What you need to know about the Singapore Tech Scene
The digital revolution has driven a constant demand for tech professionals across industries like software development, data analytics and cybersecurity. In Singapore, one of the largest cities in Southeast Asia, the demand for tech talent is so high that the government continues to invest millions into programs designed to develop a talent pipeline directly from universities while also scaling efforts in pre-employment training and mid-career upskilling to expand and elevate its workforce.
