Data Engineer
Role Overview:
As a vital member of the team, the Data Engineer will be responsible for managing and optimising diverse data sets from various systems, extracting valuable business insights. This role involves developing raw data pipelines and using Azure Cloud tools to ensure seamless integration and optimisation. The candidate will also play a key role in migrating on-premises databases to the Azure Cloud, utilising their expertise in cloud architecture, database management, and change management to ensure a smooth transition for stakeholders.
Responsibilities:
Data Architecture: Develop, build, test, and maintain data architectures within large-scale data processing systems.
Cloud Migration: Migrate existing on-premises databases to the Azure Cloud.
Pipeline Management: Develop and manage data pipelines using Azure Data Factory, Delta Lake, and Spark, ensuring secure, reliable, and accessible data sets.
Scalable Solutions: Design and implement scalable data solutions leveraging Azure Cloud architecture.
Data Manipulation: Utilise Spark, SQL, Python, R, and other data frameworks to manipulate data and gain insights into dataset characteristics.
API Interaction: Interact with API systems to query and retrieve data for analysis.
Stakeholder Collaboration: Work closely with Business Analysts, IT Ops, and other stakeholders to understand and meet data needs.
Governance and Quality: Ensure compliance with data governance and data quality principles.
Data Modelling: Design robust data models to enhance data accessibility and facilitate deeper analysis.
Documentation: Maintain technical documentation for the entire codebase.
Lifecycle Ownership: Take end-to-end ownership of the Data Engineering Lifecycle.
ETL Processes: Implement and manage Fivetran for efficient and reliable ETL processes.
Desired Experience:
Bachelor's degree in computer science, engineering, or equivalent experience.
Extensive experience as a Senior Data Engineer/Cloud Data Architect, or similar role.
Deep knowledge of Azure Cloud architecture, Azure Databricks, DevOps, and CI/CD.
Extensive experience migrating on-premises data warehouses to the cloud.
Proficiency with Spark, SQL, Python, R, and other data engineering development tools.
Experience with metadata-driven pipelines and SQL serverless data warehouses.
Extensive knowledge of querying API systems.
Extensive experience building and optimising ETL pipelines using Databricks.
Understanding of data governance and data quality principles.
Experience implementing and managing Unity Catalogue for data governance.
Familiarity with Fivetran ETL tool for seamless data integration.