Senior Technical Lead

About the role

We are looking for an experienced hands-on Senior Data Engineer to support the migration of existing PBWM PEGA-related data pipelines away from legacy SAS and Teradata processing and onto the AWS-based data platform. The engineer will pick up work already progressed earlier this year, complete the remaining SAS-to-PySpark migration, and implement the producer-side changes required to supply data into the agreed PEGA integration pattern.

What you'll do

• Understand the existing SAS/Teradata pipelines supporting PBWM PEGA data provisioning and baseline the remaining migration scope
• Convert remaining SAS transformation and business logic into production-quality PySpark
• Develop and enhance batch data pipelines using Python/PySpark and AWS, particularly Glue and S3
• Maintain equivalent business logic and data structures where appropriate to support a controlled migration
• Reconcile PySpark outputs against the existing SAS/Teradata processes and support parallel-run/data validation

What you need

• Python
• SQL
• AWS
• Git
• ETL
• SAS
• Any graduate

Good to have

• Snowflake
• Airflow
• Spark
• Data structures
• CI/CD
• Reconciliation