The Senior Data Engineer develops and maintains scalable data infrastructure and pipelines, ensuring data quality and accessibility for data-driven decision-making.
The Senior Data Engineer plays a critica role in designing, developing, and maintaining scalable data infrastructure and pipelines. This position supports data-driven decision-making across the organization by ensuring data quality, accessibility, and performance.
Roles and Responsibilities
- Data Pipeline Development: Design, build, and maintain robust, scalable data pipelines using tools like Apache Airflow, Kafka, and Spark.
- Data Architecture: Develop and optimizedata models, data warehouses, and ETL processes to support analytics and machine learning initiatives.
- Collaboration: Work closelywith data scientists, analysts, and software engineers to understand data requirements and deliver tailored solutions.
- Data ǪualityG Governance: Ensure data integrity, implement validation checks, and enforce data governance policies.
- Performance Optimization: Monitor and improvedata processing speed,system uptime, and query performance.
- Leadership G Mentorship: Guide junior engineers, promote best practices, and foster a culture of continuous learning.
- Innovation G Research: Stay updated with emerging technologies in AI, automation, and data streaming; lead initiatives to integrate them into existing systems.
Profile
- Bachelor's or Master's in computer science, data science, information technology, or a related field, with 6+ years of experience.
- Proficiency in SǪL, Python, and ETL/ELT tools
- Experience in Microsoft solutions are as follows:
- Azure Data Factory
- Microsoft Fabric
- Azure Synapse Analytics
- Azure Databricks
- Power BI
- Azure SǪL Database/ SǪL Server
- Azure Purview
- Experience with big data technologies (Hadoop, Spark)
- Familiarity with cloud platforms(Azure, AWS, and GCP)
- Strong understanding of data modelling, data warehousing, and stream processing
- Knowledge of data security, compliance, and governance
- Knowledge of CI/CD pipelines using Azure DevOps
- Understanding of data security and compliance(e.g., GDPR, HIPAA)
Top Skills
Apache Airflow
AWS
Azure Data Factory
Azure Databricks
Azure Devops
Azure Purview
Azure Sql Database
Azure Synapse Analytics
Ci/Cd
ETL
GCP
Hadoop
Kafka
Microsoft Fabric
Power BI
Python
Spark
SQL
SQL Server
Similar Jobs
Artificial Intelligence • Fintech • Payments • Business Intelligence • Financial Services • Generative AI
Design and implement data models, build ETL pipelines, manage data quality and governance, and leverage AI in data engineering.
Top Skills:
AirflowBigQueryGoogle Cloud PlatformMySQLOraclePostgresPythonRSQL
Cloud • Information Technology • Software
As a Data Engineer at Xtremax, you'll design, build, and maintain data pipelines, collaborate with various teams, and transform data for analytics.
Top Skills:
AthenaAws DmsAws LambdaAws S3CassandraEcsEventbridgeGlueHadoopKafkaMongoDBMySQLPandasPostgresPythonRRabbitMQSparkSpringSQLSQLSqliteSsisVoltdb
Information Technology • Consulting
The role requires designing and developing data infrastructure, utilizing Hadoop and Spark in large-scale environments, while also leveraging containerization and automation tools.
Top Skills:
AnsibleBitbucketControl-MDockerGraphanaHadoopJavaJenkinsKubernetesOpenshift Container PlatformPrometheusPythonScalaSparkSplunk
What you need to know about the Singapore Tech Scene
The digital revolution has driven a constant demand for tech professionals across industries like software development, data analytics and cybersecurity. In Singapore, one of the largest cities in Southeast Asia, the demand for tech talent is so high that the government continues to invest millions into programs designed to develop a talent pipeline directly from universities while also scaling efforts in pre-employment training and mid-career upskilling to expand and elevate its workforce.
