Role: Software Dev Engineer
Duration: 6 Month Contract
Location: London
Rate: Up to £420 a day
Job summary
This project consists of transforming our AWS Redshift cluster (UDM) from a production-level system used across multiple orgs to a reporting-level system used within CF only. It consists of several tasks that focus on i) reducing the surface area, ii) deprecating and migrating core pipelines off of UDM and iii) upgrading the cluster configuration based on the new needs. In addition, third-party vendor is also expect to address company-wide mandates related to the Digital Markets Act (DMA) and security initiatives
Deliverables
UDM Migration
1. Offboard external teams
2. Offboard internal teams' production use cases
3. Migrate key pipelines off of UDM
4. Migrate derived datasets off of UDM
5. Clean up the cluster
6. Update cluster configuration
7. Automate cluster management and maintenance
DMA and Security
1. Move UDM to private subnet
2. DMA Tagging of Customer Behavior Datasets
Skills
Candidates are expected to have 2+ years of experience in 5 out of the 6 following domains.
- Knowledge of distributed systems as it pertains to data storage and computing
- Experience with data modeling, warehousing and building ETL pipelines
- Experience with big data technologies such as: Hadoop, Hive, Spark, EMR
- Experience programming with at least one programming language such as C++, C#, Java, Python, Golang, PowerShell, Ruby
- Experience with AWS technologies like Redshift, S3, AWS Glue, EMR, Kinesis, FireHose, Lambda, and IAM roles and permissions
- Experience with non-relational databases / data stores (object storage, document or key-value stores, graph databases, column-family databases)