Job Title: Data Engineer
Location: Hybrid (2 days in office, Glasgow)
Day Rate: £390/day (Inside IR35)
About the Role:
We are seeking an experienced Data Engineer to join our blue-chip client's dynamic team. The ideal candidate will have extensive hands-on experience in data engineering, particularly with Spark, Python, and Java. A background in banking or financial services is highly desirable, as you will work on projects that require domain-specific knowledge. If you have a deep understanding of big data ecosystems, enjoy solving complex problems, and thrive in a hybrid work environment, this role is for you!
Key Responsibilities:
- Develop and optimize scalable ETL pipelines using Spark and other big data technologies.
- Work on large datasets using HDFS, Hive, and similar tools in the big data ecosystem.
- Implement and maintain CI/CD pipelines to automate data workflows.
- Collaborate with cross-functional teams to design data solutions and support ongoing analytics efforts.
- Ensure data quality, availability, and performance in production environments.
Core Skills & Experience Required:
- Minimum of 8+ years of experience in data engineering.
- Hands-on experience with Spark 2.x/3.x.
- Proficiency in Python (or Scala/Java), with a preference for Python.
- Strong knowledge of HDFS, SQL, and experience in building ETL pipelines.
- Familiarity with big data ecosystems including data nodes/edge nodes, Hive, and similar technologies.
- Experience in data wrangling and working with large datasets.
- Competency with CI/CD pipelines and experience in deploying scalable data solutions.
Desirable:
- Previous experience in banking or financial services industries.
- Familiarity with financial data structures and regulatory requirements.