About the role:
An exciting opportunity has arisen for an experienced Lead Data Engineer to join the Engineering team within the Digital & Product Department. In this role, the successful candidate will lead a team of data engineers in designing and creating scalable and repeatable data pipelines. This hands-on position involves discussing requirements, designing solutions, and writing code to automate the ingestion, cleansing, and transformation of data to achieve the target format.
You will collaborate with the Products, Engineering, Data, and Business Analytics teams to understand source and target data requirements and develop data integration solutions for necessary migrations. The Data Engineer will support software developers, PowerBI developers, data analysts, and data scientists on data initiatives, ensuring optimal data delivery architecture across ongoing projects. You must be self-directed and comfortable supporting the data needs of multiple teams, systems, and products.
Responsibilities
- Develop and maintain ETL and data integration solutions using various technologies and data sources.
- Oversee and update the Data Engineering Architecture.
- Analyze and organize raw data from diverse sources.
- Investigate methods to improve data quality and reliability.
- Identify, design, and implement internal process improvements, such as automating manual processes and optimizing data delivery.
- Utilize cloud-based infrastructure (Azure) for hosting data solutions and applications.
- Collaborate with architects, data analysts, and data scientists to achieve business objectives.
Requirements
- Previous experience as a data engineer.
- Proven track record in developing and maintaining ETL/ELT processes within a medallion architecture.
- Solid experience in data modeling within a cloud-based data platform.
- Extensive experience with SQL Server and relational databases.
- Strong knowledge of the Azure data engineering stack, including Azure Synapse and Azure Data Lake.
- Proficiency in programming with Python, PySpark, and T-SQL.
- Familiarity with wider Azure data solutions, such as Azure ML Studio.
- Previous experience with Azure DevOps and a good understanding of CI/CD.
- Analytical skills for working with structured and unstructured datasets.
- Excellent written and verbal communication skills.
- Experience in supporting and collaborating with cross-functional teams in a dynamic environment.