Design, build and optimize scalable AWS-based data pipelines (batch and real-time) to supply analytics and ML tools such as SageMaker. Implement orchestration, containerisation, IaC, cost and performance optimisations, and embed DLP and governance controls to protect and deliver reliable datasets for downstream consumption.
Manchester, Cheltenham or London
We are seeking a skilled Data Engineer to join our Engineering team, responsible for designing, building, and optimising scalable data pipelines that power advanced analytics and machine learning solutions. You will play a key role in enabling data-driven decision-making by delivering high-quality, reliable datasets to tools such as Amazon SageMaker and other analytics platforms.
Key Responsibilities
We are looking for a Data Engineer to work closely with the Data Science team to develop robust data pipelines that feed analytics and machine learning tools such as Amazon SageMaker and third-party platforms like Databricks. You will leverage AWS technologies such as EMR, S3, EKS and Airflow to process and orchestrate high-volume datasets, ensuring solutions are scalable, resilient and cost-efficient. You will also play a key role in embedding data loss prevention (DLP) principles and controls into data pipelines to protect sensitive information, while ensuring data is reliable, accessible, well-governed and optimised for downstream consumption.
Skills, Knowledge & Expertise
Essential
- Strong experience in data engineering within AWS cloud environments.
- Hands-on experience with AWS big data technologies such as EMR, S3 and SageMaker.
- Proficiency in Python for building scalable data pipelines and processing frameworks. Experience with Apache Spark for distributed data processing.
- Experience designing and maintaining scalable batch and real-time data pipelines.
- Solid understanding of ETL/ELT design patterns and data modelling techniques.
- Experience with workflow orchestration tools such as Apache Airflow (ideally deployed on AWS).
- Familiarity with containerisation and orchestration using Docker and Kubernetes (EKS).
- Experience with infrastructure as code (e.g. Terraform) and CI/CD/GitOps practices.
- Proven ability to optimise performance and reduce cloud costs through partitioning, clustering and workload management.
- Understanding of data security principles, including data loss prevention (DLP).
Desirable
- Experience with Databricks or similar third-party big data platforms.
- Knowledge of real-time streaming technologies (e.g. Kafka, Kinesis).
- Experience implementing data governance and compliance frameworks.
- Familiarity with monitoring and observability tools in AWS environments.
- Exposure to Lakehouse or modern data platform architectures.
Job Benefits
- Flexible Working: Balance your work and personal life with our flexible working options.
- Generous Holiday Allowance: Enjoy 25 days of holiday, plus bank holidays, with the option to buy up to 5 additional days of annual leave.
- Medicash & Critical Illness Scheme
- Financial & Investment Benefits: Enjoy peace of mind with our Pension, Life Assurance, and Share Save Scheme.
- Community & Volunteering Programmes: Make a difference in your community with our volunteering opportunities.
- Green Car Scheme: Drive green and save money with our eco-friendly car scheme.
- Cycle Scheme: Stay fit and healthy with our cycle-to-work scheme.
- Special Time Off: Take time off for those big moments in life, like getting married/entering into a civil partnership, becoming a grandparent, and welcoming home a new pet.
- Family Planning: Benefit from our generous maternity and paternity leave, as well as time off and support for those undergoing fertility treatments.
About
We assess, develop and manage cyber threats across our increasingly connected society. We advise global technology, manufacturers, financial institutions, critical national infrastructure providers, retailers and governments on the best way to keep businesses, software and personal data safe.With our knowledge, experience and global footprint, we are best placed to help businesses identify, assess, mitigate & respond to the risks they face.We are passionate about making the Internet safer and revolutionising the way in which organisations think about cyber security.Headquartered in Manchester, UK, with over 35 offices across the world, NCC Group employs more than 2,000 people and is a trusted advisor to 15,000 clients worldwide.
Similar Jobs
Information Technology • Professional Services • Software • Cybersecurity
Design, build and optimize scalable batch and real-time data pipelines on AWS to deliver reliable datasets for analytics and ML (e.g., SageMaker). Implement orchestration (Airflow), containerisation (Docker/EKS), IaC (Terraform), cost and performance optimisation, and embed data loss prevention and governance controls.
Top Skills:
Amazon S3Amazon SagemakerApache AirflowSparkAws EmrCi/CdDockerGitopsKubernetes (Eks)PythonTerraform
Information Technology • Professional Services • Software • Cybersecurity
Design, build and optimise scalable batch and real-time data pipelines on AWS to deliver reliable datasets for analytics and machine learning (e.g., SageMaker). Embed DLP and governance controls, improve performance and cost-efficiency, and collaborate with Data Science to support Databricks/third-party platforms.
Top Skills:
Amazon EksAmazon EmrAmazon S3Amazon SagemakerApache AirflowSparkCi/CdDockerGitopsKubernetesPythonTerraform
Information Technology • Legal Tech • Analytics
Design and deliver scalable Customer Data Platform solutions (Oracle Unity), build and operate data pipelines and Customer 360 models, integrate CDP with data lake/CRM/marketing systems, lead architecture and data modelling workshops, ensure data quality, governance, security, and support CDP operations, and partner with Oracle implementation teams.
Top Skills:
Adobe Experience PlatformAWSAzureDatabricksETLJSONOracle CloudOracle Unity CdpPythonRest ApisSalesforce Data CloudSnowflakeSQLTealium
What you need to know about the Bristol Tech Scene
Along with Gloucester, Swindon and Bath, Bristol is part of the "Silicon Gorge" tech hub, a region in the U.K. renowned for its high-tech and research-driven industries, with a particular emphasis on sustainability and reducing environmental impact. As the European Green Capital, Bristol is home to 25,000 cleantech companies, including Baker Hughes and unicorn Ovo Energy. The city has committed to achieving net-zero emissions within the next decade.
