Senior Data Engineer Full Time

City, Leeds
The Bridge IT Recruitment

Purpose of the Job:

• Design, build, and maintain robust data systems and pipelines that support data storage, processing, and analysis on the Cloud.
• Work with large datasets, ensuring data quality, scalability, and performance, while collaborating.
• closely with data scientists, analysts, and other engineering teams to understand their data needs and provide them with high-quality, accessible data.
• They are responsible for ensuring that the underlying data infrastructure supports the organizations broader data and business goals, enabling more effective data-driven decision-making.

Key Accountabilities:

• Design and implement scalable, efficient, and secure data architectures, ensuring optimal data flow across systems in order to achieve high service levels of support, maintenance and development
• You will own development and change projects to ensure requirements are met in the most cost-effective manner while minimising associated risk to expected standards.
• Responsibly for cloud data platform development, data modelling, shaping and technical planning
• You will be a mentor among the owning decision making and evaluation of requirement suitability, facilitate reliable estimates, technical project management, stakeholder management with a project
• Ensure that resource requirements are understood and planned/estimated effectively against demand, including identification of additional temporary resource capability within projects
• Maintain appropriate process procedures, compliance and service level monitoring, performance reporting and vendor management.
• Implementing best practices around data security, privacy, and compliance for the teams compliance with cyber security and data protection and supporting along with BI lead
• Strong stakeholder management will be required for maintaining relationships with our business users to clarify and influence requirements. Including liaising with internal business departments and functions to manage the service level expected from the data team.
• Collaborating with external organisations and third-party software/service suppliers for ongoing support, maintenance and development of systems.
• You will be able to demonstrate you are quality focused to ensure that they solutions are built to an appropriate standard whilst being balanced with a drive to deliver against tight deadlines. Support in developing and implementing best practices and process across the team along with BI lead.
• Influence the evolution of business and system requirements and contribute to the design of technical solutions to feed a delivery pipeline that increasingly employs Agile methods such as SCRUM and Kanban
• You will be required to develop unit tested code and then support test cycles including post implementation validation. You will be required to contribute to the transition into service and ongoing support of the applications in the area which provides the opportunity to reduce technical debt and rationalise our technical footprint
• Mentor data engineers, supporting their professional growth and development

Outcome, Results and Key Performance Indicators:

• Delivery of projects to expected timely, cost and quality standards
• Excellent levels of application availability and resilience as required by business operations.
• Necessary governance and control requirements defined – design, code and test standards and guidelines.
• Ensure data systems comply with necessary governance and control requirements.
• Internally-developed data solutions are fit for purpose and fit correctly within the data architecture. Built and tested to user requirements, performing to defined performance and capacity requirements.
• Company data is secure, accurate, maintained and available according to requirements.
• Technical risks and issues correctly mitigated and managed on Projects and Production support.
• High quality software delivered in to production – zero critical and high defects before production release.

Dimensions of Job:

• This role is part of a well-established data team, the role offers a great opportunity for the right candidate to hone their modern data management skills in a friendly and supportive environment.
• This role requires attendance to a Leeds based office as often as needed with a minimum 2 days a week.
• Able to work effectively as part of a remote team.
• A great opportunity for a motivated data engineer seeking a new opportunity with a friendly, newly formed data team and able to contribute to the team’s growth with their technical expertise

Key Relationships:

• Internal: Wider technical teams (including apps, test, dev ops and more), Project managers, business SME’s, data teams and communities , Data scientists, BI Lead, Head of Data
• External: software & service suppliers, consultants.

Knowledge and Skills:

Knowledge
– Broad data management technical knowledge so as to be able to work across full data cycle.
– Proven Experience working with AWS data technologies (S3, Redshift, Glue, Lambda, Lake formation, Cloud Formation), GitHub, CI/CD
– Coding experience in Apache Spark, Iceberg or Python (Pandas)
– Experience in change and release management.
– Experience in Database Warehouse design and data modelling
– Experience managing Data Migration projects.
– Cloud data platform development and deployment.
– Experience of performance tuning in a variery of database settings.
– Experience of Infrastructure as code practises.
– Proven ability to organise and produce work within deadlines.

Skills
– Good project and people management skills.
– Excellent data development skills.
– Excellent data manipulation and analysis skills using a variety of tools including SQL, Phyton, AWS services and the MSBI stack.
– Ability to prioritise and be flexible to change those priorities at short notice.
– Commercial acumen.
– Able to demonstrate a practical approach to problem solving.
– Able to provide appropriate and understandable data to a wide ranging audience.
– Well-developed and professional communication skills.
– Strong analytical skills – ability to create models and analyse data in order to solve complex problems or reinforce commercial decisions.
– Able to understand business processes and how this is achieved/influenced by technology.
– Must be able to work as part of a collaborative team to solve problems and assist other colleagues.
– Ability to learn new technologies, programs and procedures.

Technical Essentials:
– Expertise across data warehouse and ETL/ ELT development in AWS preferred with experience in the following:
– Strong experience in some of the AWS services like Redshift, Lambda,S3,Step Functions, Batch, Cloud formation, Lake Formation, Code Build, CI/CD, GitHub, IAM, SQS, SNS, Aurora DB
– Good experience with DBT, Apache Iceberg, Docker, Microsoft BI stack (nice to have)
– Experience in data warehouse design (Kimball and lake house, medallion and data vault) is a definite preference as is knowledge of other data tools and programming languages such as Python & Spark and Strong SQL experience.
– Experience is building Data lake and building CI/CD data pipelines
– A candidate is expected to understand and can demonstrate experience across the delivery lifecycle and understand both Agile and Waterfall methods and when to apply these.

Experience:

This position requires several years of practical experience in a similar environment. We require a good balance of technical and personal/softer skills so successful candidates can be fully effective immediately.

– Proven experience in developing, delivering and maintaining tactical and enterprise data management solutions.
– Proven experience in delivering data solutions using cloud platform tools.
– Proven experience in assessing the impact of proposed changes on production solutions.
– Proven experience in managing and developing a team of technical experts to deliver business outcomes and meet performance criteria.
– Exposure to Energy markets, Energy Supply industry sector
– Developing and implementing operational processes and procedures.