Mandatory Skills
You need to have the below skills.
- At least 12+ Years of IT Experience with Deep understanding of component understanding around Spark Data Integration (PySpark, scripting, variable setting etc.), Spark SQL, Spark Explain plans.
- Spark SME - Be able to analyse Spark code failures through Spark Plans and make correcting recommendations.
- To be able to traverse and explain the architecture you have been a part of and why any particular tool/technology was used.
- Spark SME - Be able to review PySpark and Spark SQL jobs and make performance improvement recommendations.
- Spark - SME Be able to understand Data Frames / Resilient Distributed Data Sets and understand any memory related problems and make corrective recommendations.
- Monitoring -Spark jobs using wider tools such as Grafana to see whether there are Cluster level failures.
- Cloudera (CDP) Spark and how the run time libraries are used by PySpark code.
- Prophecy - High level understanding of Low-Code No-Code prophecy set up and its use to generate PySpark code.
- Ready to work at least three days from Sheffield (UK) office and accept changes as per customer policies.
Good to have skills.
Ideally, you should be familiar with
- Collaboration with multiple customer stakeholders
- Knowledge of working with Cloud Databases
- Excellent communication and solution presentation skills.