The Cloud Data Engineer designs and manages data storage solutions, optimizes data pipelines, integrates diverse data sources, and develops ETL processes using AWS and Databricks technologies.
- Design and architect data storage solutions, including databases, data lakes, and warehouses, using AWS services such as Amazon S3, Amazon RDS, Amazon Redshift, and Amazon DynamoDB, along with Databricks' Delta Lake. Integrate Informatica IDMC for metadata management and data cataloging.
- Create, manage, and optimize data pipelines for ingesting, processing, and transforming data using AWS services like AWS Glue, AWS Data Pipeline, and AWS Lambda, Databricks for advanced data processing, and Informatica IDMC for data integration and quality.
- Integrate data from various sources, both internal and external, into AWS and Databricks environments, ensuring data consistency and quality, while leveraging Informatica IDMC for data integration, transformation, and governance.
- Develop ETL (Extract, Transform, Load) processes to cleanse, transform, and enrich data, making it suitable for analytical purposes using Databricks' Spark capabilities and Informatica IDMC for data transformation and quality.
- Monitor and optimize data processing and query performance in both AWS and Databricks environments, making necessary adjustments to meet performance and scalability requirements. Utilize Informatica IDMC for optimizing data workflows.
- Good experience in data engineering, with expertise in AWS services, Databricks, and/or Informatica IDMC.
- Proficiency in programming languages such as Python, Java, or Scala for building data pipelines.
- Evaluate potential technical solutions and make recommendations to resolve data issues especially on performance assessment for complex data transformations and long running data processes.
- Strong knowledge of SQL and NoSQL databases.
- Familiarity with data modeling and schema design.
- AWS certifications (e.g., AWS Certified Data Analytics - Specialty, AWS Certified Data Analytics - Specialty), Databricks certifications, and Informatica certifications are a plus.
Top Skills
Aws Data Pipeline
Aws Dynamodb
Aws Glue
Aws Lambda
Aws Rds
Aws Redshift
Aws S3
Databricks
Informatica Idmc
Java
NoSQL
Python
Scala
SQL
Unison Consulting Singapore Office
1 Changi Business Park Crescent, , Plaza 8 #03-06 Tower A, Singapore, , Singapore, 486025
Unison Consulting Singapore Office
#12-00, 63 Market Street, Bank of Singapore Center, Singapore, , Singapore, 048942
Similar Jobs
Big Data • Cloud • Internet of Things
The Principal Cloud Engineer will design and maintain cloud architectures across multiple platforms, focusing on AI and data-intensive workloads, while optimizing resilience and performance through automation and collaboration.
Top Skills:
AirflowApache KafkaAWSAzureBashDataformDbtGCPGoGoogle Pub/SubJavaKubernetesPythonSparkSQLTerraform
Software
Manage cloud infrastructure on AWS focusing on reliability and performance, build Data Lake architecture, administer databases, and ensure data quality and compliance.
Top Skills:
Amazon S3AnsibleAthenaAWSAws CloudformationAws GlueBashDynamoDBEmrGitGoJenkinsLake FormationPythonRdsRedshiftTerraform
Financial Services
The Cloud Data Platform Engineer designs, builds, and maintains data infrastructure on Azure and Snowflake, optimizing for performance, security, and scalability while collaborating with cross-functional teams.
Top Skills:
AdoAzureBashCi/CdFivetranLogic AppsPowershellQlikSnowflakeTalendTerraform
What you need to know about the Singapore Tech Scene
The digital revolution has driven a constant demand for tech professionals across industries like software development, data analytics and cybersecurity. In Singapore, one of the largest cities in Southeast Asia, the demand for tech talent is so high that the government continues to invest millions into programs designed to develop a talent pipeline directly from universities while also scaling efforts in pre-employment training and mid-career upskilling to expand and elevate its workforce.
