Data engineering DENG
Designing, building, operationalising, securing and monitoring data pipelines and data stores.
Revision notes
Updates for SFIA 9
- There is an updated version of this skill for SFIA 9.
- Theme(s) influencing the updates for this skill: Continued refinement for data and analytics skills, Making SFIA easier to consume (enhance readability/guidance/descriptions), Support for cyber security working practices (both specialised and general).
- Content changes have been made to levels 2, 3, 4, 5, and 6.
- You can move to SFIA 9 when you are ready - SFIA 8 skill descriptions will still be available to use.
- Previous SFIA assessments or skills mapping are not impacted by this change.
Guidance notes
Activities may include — but are not limited to:
- identifying data sources, data processing concepts and methods
- evaluating, designing and implementing on-premise, cloud-based and hybrid data engineering solutions
- structuring and storing data for uses including — but not limited to — analytics, machine learning, data mining, sharing with applications and organisations
- harvesting structured and unstructured data
- integrating, consolidating and cleansing data
- migrating and converting data
- applying ethical principles in handling data
- ensuring appropriate storage of data in line with relevant legislation
- building in security, compliance, scalability, efficiency, reliability, fidelity, flexibility and portability.
Levels of responsibility for this skill
| 2 | 3 | 4 | 5 | 6 |
Data engineering: Level 1
This skill is not typically observed or practiced at this level of responsibility and accountability.
Data engineering: Level 2
Assist in developing and implementing data pipelines and data stores.
Performs administrative tasks to provide accessibility, retrievability, security and protection of data.
Data engineering: Level 3
Designs and implements data pipelines and data stores to acquire and prepare data.
Applies data engineering standards and tools to create and maintain data pipelines and extract, transform and load data.
Carries out routine data quality checks and remediation.
Data engineering: Level 4
Designs, implements, and maintains complex data engineering solutions to acquire and prepare data.
Creates and maintains data pipelines to connect data within and between data stores, applications and organisations.
Carries out complex data quality checking and remediation.
Data engineering: Level 5
Plans and drives the development of data engineering solutions ensuring that solutions balance functional and non-functional requirements.
Monitors application of data standards and architectures including security and compliance.
Contributes to organisational policies, standards, and guidelines for data engineering.
Data engineering: Level 6
Leads the selection and development of data engineering methods, tools and techniques.
Develops organisational policies, standards, and guidelines for the development and secure operation of data services and products.
Ensures adherence to technical strategies and architectures.
Plans and leads data engineering activities for strategic, large and complex programmes.
Data engineering: Level 7
This skill is not typically observed or practiced at this level of responsibility and accountability.