Principal Data Platform Engineer, Java, Kafka, AWS Full Time NEW

Digital Skills Ltd

Principal Data Platform Engineer

Java, Kafka, AWS

Remote (UK-based)

Permanent – Up to £130,000 + Benefits

Security Clearance Requirement

For this role, you must be eligible for SC Clearance. Due to the secure nature of the work, this requires British Citizenship, as some data is UK Eyes Only .

Overview

We are seeking an experienced Principal Data Platform Engineer to play a key technical leadership role within a high-impact programme delivering secure, scalable, Real Time data platforms for critical UK Government and national infrastructure clients.

This is a hands on senior engineering role with significant responsibility for technical direction, architecture design, client assurance, and mentoring. You will work closely with both internal teams and client stakeholders, helping shape solutions from early concept through to production delivery.

The Role

As a Principal Data Platform Engineer, you will provide technical leadership across multiple data platform initiatives, designing and assuring highly available, low-latency, and scalable batch and Real Time data solutions.

You will act as a trusted technical authority, engaging directly with clients, producing high-quality architecture designs, and ensuring proof-of-concept solutions are robust, secure, and production-ready.

Key Responsibilities

• Lead the design and delivery of complex batch and Real Time data pipelines and Back End services
• Produce high-level and low-level architecture designs, ensuring alignment with security, scalability, and performance requirements
• Provide technical assurance for solution designs and proof-of-concepts prior to production implementation
• Act as a client-facing technical lead, working alongside customer teams to define requirements, shape solutions, and build confidence
• Mentor and coach Lead and Senior Engineers, setting technical standards and raising overall engineering capability
• Drive and promote best engineering practices, including TDD, CI/CD, DevOps, and clean, maintainable code
• Influence technology choices and architectural patterns across the data platform estate
• Collaborate with architects, analysts, security specialists, and testers in Agile/BDD environments
• Oversee services in production, ensuring resilience, performance, and operational excellence
• Contribute to technical strategy, estimation, delivery planning, and roadmap definition
• Champion continuous improvement across data platform tooling, processes, and ways of working

Required Experience

• Extensive commercial experience with Java in production environments
• Strong experience designing and building Kafka-based streaming platforms
• Hands-on experience with Spark, Flink, Kafka Streams, or similar technologies
• Experience with Spring Boot and/or Python
• Deep experience working with AWS, including services such as Lambda, S3, and cloud-native architectures
• Experience integrating with data stores such as PostgreSQL, Redis, or similar
• Strong understanding of data pipelines, data modelling, and distributed systems
• Proven experience leading technical designs and mentoring engineers
• Comfortable working in client-facing roles, explaining complex technical concepts clearly
• Strong background in Agile, DevOps, and modern software delivery practices

Desirable Skills

• Experience with Elasticsearch, OpenSearch, or Solr
• Infrastructure-as-Code tools such as Terraform, Ansible, or Packer
• CI/CD tooling such as Jenkins, Drone, or similar
• Experience working within highly secure or regulated environments

The Company

You will be joining a UK-based SME (circa 300 people) that designs, builds, and integrates highly secure data platforms for UK Government and national infrastructure clients.

Teams work in integrated, collaborative environments, partnering closely with clients and internal subject-matter experts. The organisation is known for its high technical standards, strong engineering culture, and genuinely supportive working environment .

What’s on Offer

• Fully remote working (UK-based) for security reasons
• Competitive £130,000 salary plus benefits
• Opportunity to work on nationally critical systems
• Access to exceptionally talented engineers and architects
• A role with real technical influence and long term impact

If this role is of interest, please apply.