The Senior Data Engineer develops and maintains scalable data infrastructure and pipelines, ensuring data quality and accessibility for data-driven decision-making.
The Senior Data Engineer plays a critica role in designing, developing, and maintaining scalable data infrastructure and pipelines. This position supports data-driven decision-making across the organization by ensuring data quality, accessibility, and performance.
Roles and Responsibilities
- Data Pipeline Development: Design, build, and maintain robust, scalable data pipelines using tools like Apache Airflow, Kafka, and Spark.
- Data Architecture: Develop and optimizedata models, data warehouses, and ETL processes to support analytics and machine learning initiatives.
- Collaboration: Work closelywith data scientists, analysts, and software engineers to understand data requirements and deliver tailored solutions.
- Data ǪualityG Governance: Ensure data integrity, implement validation checks, and enforce data governance policies.
- Performance Optimization: Monitor and improvedata processing speed,system uptime, and query performance.
- Leadership G Mentorship: Guide junior engineers, promote best practices, and foster a culture of continuous learning.
- Innovation G Research: Stay updated with emerging technologies in AI, automation, and data streaming; lead initiatives to integrate them into existing systems.
Profile
- Bachelor's or Master's in computer science, data science, information technology, or a related field, with 6+ years of experience.
- Proficiency in SǪL, Python, and ETL/ELT tools
- Experience in Microsoft solutions are as follows:
- Azure Data Factory
- Microsoft Fabric
- Azure Synapse Analytics
- Azure Databricks
- Power BI
- Azure SǪL Database/ SǪL Server
- Azure Purview
- Experience with big data technologies (Hadoop, Spark)
- Familiarity with cloud platforms(Azure, AWS, and GCP)
- Strong understanding of data modelling, data warehousing, and stream processing
- Knowledge of data security, compliance, and governance
- Knowledge of CI/CD pipelines using Azure DevOps
- Understanding of data security and compliance(e.g., GDPR, HIPAA)
Top Skills
Apache Airflow
AWS
Azure Data Factory
Azure Databricks
Azure Devops
Azure Purview
Azure Sql Database
Azure Synapse Analytics
Ci/Cd
ETL
GCP
Hadoop
Kafka
Microsoft Fabric
Power BI
Python
Spark
SQL
SQL Server
Similar Jobs
Cloud • Information Technology • Software
The Data Engineer will design, build, and maintain data systems and pipelines, collaborate with teams, and ensure data quality for analytics.
Top Skills:
AthenaAws DmsAws LambdaAws S3CassandraEcsEventbridgeGlueHadoopKafkaMongoDBMySQLPandasPostgresPythonRRabbitMQRest ApisSparkSpringSQLSqliteSsisVoltdb
Financial Services
Data Developers at Rimes build and maintain ETL pipelines for financial datasets, ensuring data accuracy and managing integrations with multiple data vendors.
Top Skills:
DatabricksFoundryGitParquetPythonRest ApisSnowflakeSQLXML
Information Technology • Consulting
The Data Engineer is responsible for designing and implementing modern data platforms, developing ETL pipelines, ensuring data quality, and supporting ETL pipeline issue resolution.
Top Skills:
Amazon S3Aws AthenaAws GlueAws LambdaAws Step FunctionsPysparkPythonSQL
What you need to know about the Singapore Tech Scene
The digital revolution has driven a constant demand for tech professionals across industries like software development, data analytics and cybersecurity. In Singapore, one of the largest cities in Southeast Asia, the demand for tech talent is so high that the government continues to invest millions into programs designed to develop a talent pipeline directly from universities while also scaling efforts in pre-employment training and mid-career upskilling to expand and elevate its workforce.
