Data Engineer
6 Months
Inside IR35
Hybrid
Our client in the Further Education Sector is currently looking to recruit a Data Engineer to join them on a 6-month contract. As a Data Engineer, experience of creating a low-level Design for a Data Lake solution in Azure is required. In-depth knowledge of Data engineering practices and principles and programming skills and ability to develop data pipelines through code.
Technical Skills
- In-depth knowledge of relevant Azure services such as Azure Data Lake Storage, Azure Data Factory, Azure Databricks, Azure SQL Database, Azure Stream Analytics, etc.
- Microsoft Fabric knowledge would be an advantage but not a must.
- Python skill, with Spark / PySpark experience. Ability to code independently based on a design or requirement is a must.
- Good knowledge of Terraform
- Experience with relational databases and SQL Skills.
- Significant experience and in-depth knowledge of creating data pipelines and associated design principles, standards, Data modelling concepts, ETL procedures, and all steps of data production process
- Experience with unit and integration testing, and data quality frameworks ideally.
- Experience working with continuous integration and continuous deployment (CICD) including Git
- An understanding of data architecture and data designs patterns (medallion, Kimball etc)
- Experience in Data Migration, Data Profiling and Data Quality enhancement.
- Apache Spark: Deep understanding of Apache Spark for large-scale data processing.
- Azure Blob Storage and Azure Data Lake Storage: Expertise in setting up and optimizing Azure Data Lake Storage for storing and managing large volumes of data.
- Azure Security and Compliance: Knowledge of Azure security features and compliance standards.
- Monitoring and Logging: Familiarity with Azure Monitor and Azure Log Analytics for monitoring and logging.
- Experience working with different data formats particularly JSON and DELTA.
- Knowledge and experience of Rest API.
- Knowledge of Azure Function and Azure API Management will be a big advantage.
- Azure Data Factory: Advanced knowledge of Azure Data Factory for orchestrating and automating data workflows.
- Any exposure to Data Governance and related platform like Purview or Collibra would be an advantage.
Ways of working:
- Experience in Agile (Scrum) way of working
- Understanding of CI/CD process & ways of working including Git
- Understanding of software development life cycle
- Knowledge of software engineering best practices for the full software development life cycle, including Agile methodologies
Carbon60, Lorien & SRG - The Impellam Group STEM Portfolio are acting as an Employment Business in relation to this vacancy.