We are partnered with a fast growing Data and AI specialist consultancy firm that are looking to expand their data engineering team. As a Data Engineer you will have extensive experience working with AWS cloud services, Terraform, SQL, and Python for automation and data processing. This role involves designing, building, and maintaining scalable data pipelines, optimising data storage solutions, and ensuring data integrity and availability across their clients platforms.
Key Responsibilities:
- Develop, maintain, and optimize ETL processes using AWS services such as Glue, Lambda, and Step Functions.
- Create and manage data workflows that extract data from various sources, transform it, and load it into our data warehouses and data lakes.
- Utilise Terraform to automate the provisioning and management of AWS resources.
- Ensure that infrastructure is designed for scalability, security, and cost-effectiveness.
- Design and optimise data storage solutions using AWS services such as S3, Redshift, RDS, and DynamoDB.
- Ensure data is stored efficiently and securely, with appropriate backup and disaster recovery strategies.
- Develop Python scripts to automate data processing tasks and improve workflow efficiency.
Experience:
- 3+ years of experience as a Data Engineer or in a similar role.
- Proven experience with AWS services (e.g., S3, Redshift, RDS, Glue, Lambda, IAM).
- Strong expertise in Terraform
- Proficient in SQL for querying relational databases and handling large datasets.
- Experience with data pipeline orchestration tools (e.g., Apache Airflow, AWS Step Functions).
- Familiarity with CI/CD pipelines and version control systems (e.g., Git).
- Knowledge of data warehousing concepts and best practices.
Benefits:
- Competitive salary and bonus structure.
- Opportunities for professional development and certification.
- Flexible work hours and remote work options.