Data engineering DENG
(unchanged)
Designing, building, operationalising, securing and monitoring data pipelines and data stores.
SFIA 9 is in development
- SFIA 9 beta due in early July 2024
- SFIA 9 planned for publication October 2024
This is a prototype for SFIA 9. It is subject to change before publication.
Guidance notes
(modified)
Activities may include, but are not limited to:
- identifying data sources, data processing concepts and methods
- evaluating, designing and implementing on-premise, cloud-based and hybrid data engineering solutions
- structuring and storing data for uses including, but not limited to, analytics, machine learning, data mining, sharing with applications and organisations
- harvesting structured and unstructured data
- integrating, consolidating and cleansing data
- migrating and converting data
- applying ethical principles in handling data
- ensuring appropriate storage of data in line with relevant legislation
- building in security, compliance, scalability, efficiency, reliability, fidelity, flexibility and portability.
Understanding the responsibility levels of this skill
Where lower levels are not defined...
- Specific tasks and responsibilities are not defined because the skill requires a higher level of autonomy, influence, and complexity in decision-making than is typically expected at these levels. You can use the essence statements to understand the generic responsibilities associated with these levels.
Where higher levels are not defined...
- Responsibilities and accountabilities are not defined because these higher levels involve strategic leadership and broader organisational influence that goes beyond the scope of this specific skill. See the essence statements.
Developing skills and demonstrating responsibilities related to this skill
The defined levels show the incremental progression in skills and reponsibilities.
Where lower levels are not defined...
You can develop your knowledge and support others who do have responsibility in this area by:
- Learning key concepts and principles related to this skill and its impact on your role
- Performing related skills (see the related SFIA skills)
- Supporting others with tasks (generic examples are provided by the essence statements for each level)
Where higher levels are not defined...
- You can progress by developing related skills which are better suited to higher levels of organisational leadership.
Levels
Defined at these levels: | 2 | 3 | 4 | 5 | 6 |
Click to learn why SFIA skills are not defined at all 7 levels.
Show/hide extra descriptions and levels.
Level 1
Data engineering: Level 2
(unchanged)
Assist in developing and implementing data pipelines and data stores.
Performs administrative tasks to provide accessibility, retrievability, security and protection of data.
Data engineering: Level 3
(modified)
Follows standard approaches and established design patterns to create and implement simple data pipelines and data stores to acquire and prepare data.
Applies data engineering standards and tools to create and maintain data pipelines and extract, transform and load data.
Carries out routine data quality checks and remediation.
Data engineering: Level 4
(unchanged)
Designs, implements, and maintains complex data engineering solutions to acquire and prepare data.
Creates and maintains data pipelines to connect data within and between data stores, applications and organisations.
Carries out complex data quality checking and remediation.
Data engineering: Level 5
(unchanged)
Plans and drives the development of data engineering solutions ensuring that solutions balance functional and non-functional requirements.
Monitors application of data standards and architectures including security and compliance.
Contributes to organisational policies, standards, and guidelines for data engineering.
Data engineering: Level 6
(unchanged)
Leads the selection and development of data engineering methods, tools and techniques.
Develops organisational policies, standards, and guidelines for the development and secure operation of data services and products.
Ensures adherence to technical strategies and architectures.
Plans and leads data engineering activities for strategic, large and complex programmes.