The global skills and competency framework for the digital world

Data engineering DENG Beta

(modified)

Designing, building, operationalising, securing and monitoring data pipelines, stores and real-time processing systems for scalable and reliable data management.

SFIA 9 is in development

  • SFIA 9 planned for publication October 2024.
  • The content of this skill may change before publication.

Guidance notes

(modified)

Activities may include, but are not limited to:

  • identifying data sources, data processing concepts and methods
  • evaluating, designing and implementing on-premise, cloud-based and hybrid data engineering solutions
  • structuring and storing data for analytics, machine learning, data mining and sharing with applications and organisations
  • harvesting structured and unstructured data
  • integrating, consolidating and cleansing data
  • implementing real-time and batch data processing pipelines
  • ensuring compliance with data governance, security and privacy standards, including encryption and secure multi-tenancy
  • managing continuous integration, deployment and monitoring of data pipelines (DataOps)
  • migrating and converting data
  • applying ethical principles in handling data
  • ensuring data storage aligns with relevant legislation
  • building in security, compliance, scalability, efficiency, reliability, fidelity, flexibility and portability to data engineering solutions.

Understanding the responsibility levels of this skill

Where lower levels are not defined...
  • Specific tasks and responsibilities are not defined because the skill requires a higher level of autonomy, influence, and complexity in decision-making than is typically expected at these levels. You can use the essence statements to understand the generic responsibilities associated with these levels.
Where higher levels are not defined...
  • Responsibilities and accountabilities are not defined because these higher levels involve strategic leadership and broader organisational influence that goes beyond the scope of this specific skill. See the essence statements.

Developing skills and demonstrating responsibilities related to this skill

The defined levels show the incremental progression in skills and responsibilities.

Where lower levels are not defined...

You can develop your knowledge and support others who do have responsibility in this area by:

  • Learning key concepts and principles related to this skill and its impact on your role
  • Performing related skills (see the related SFIA skills)
  • Supporting others who are performing higher level tasks and activities
Where higher levels are not defined...
  • You can progress by developing related skills which are better suited to higher levels of organisational leadership.

Show/hide extra descriptions and levels.

2 3 4 5 6

Levels of responsibility for this skill

Data engineering: Level 2

Level 2 - Assist: Essence of the level: Provides assistance to others, works under routine supervision, and uses their discretion to address routine problems. Actively learns through training and on-the-job experiences.

(modified)

Assists in developing and implementing data pipelines and data stores.

Performs administrative tasks to provide data accessibility, retrievability, security and protection.

Supports the monitoring of data pipeline operations, identifying issues and escalating as needed.

Participates in data migration and conversion tasks under routine supervision.

Data engineering: Level 3

Level 3 - Apply: Essence of the level: Performs varied tasks, sometimes complex and non-routine, using standard methods and procedures. Works under general direction, exercises discretion, and manages own work within deadlines. Proactively enhances skills and impact in the workplace.

(modified)

Follows standard approaches and established design patterns to create and implement simple data pipelines and data stores to acquire and prepare data.

Applies data engineering standards and tools to create and maintain data pipelines and perform extract, transform and load (ETL) processes, incorporating security and data integrity practices.

Contributes to data migration and conversion projects, ensuring data integrity and consistency.

Conducts routine data quality checks and remediation.

Data engineering: Level 4

Level 4 - Enable: Essence of the level: Performs diverse complex activities, supports and guides others, delegates tasks when appropriate, works autonomously under general direction, and contributes expertise to deliver team objectives.

(modified)

Designs, implements and maintains complex data engineering solutions to acquire and prepare data.

Creates and maintains data pipelines to connect data across data stores, applications and organisations. Builds in compliance with data governance and security standards.

Supports the development of continuous integration and deployment practices. Monitors and optimises pipeline performance and scalability.

Conducts complex data quality checking and remediation. Leads data migration and data conversion activities.

Data engineering: Level 5

Level 5 - Ensure, advise: Essence of the level: Provides authoritative guidance in their field and works under broad direction. Accountable for delivering significant work outcomes, from analysis through execution to evaluation.

(modified)

Plans and drives the development of data engineering solutions, balancing functional and non-functional requirements.

Monitors application of data standards, architectures and security, ensuring compliance and scalability.

Develops and promotes continuous integration, deployment and monitoring practices.

Contributes to organisational policies, standards and guidelines for data engineering.

Data engineering: Level 6

Level 6 - Initiate, influence: Essence of the level: Has significant organisational influence, makes high-level decisions, shapes policies, demonstrates leadership, promotes organisational collaboration, and accepts accountability in key areas.

(modified)

Leads the selection and development of data engineering methods, tools and techniques.

Develops organisational policies, standards and guidelines for the development and secure operation of data services and products.

Ensures adherence to technical strategies and architectures.

Plans and leads data engineering for strategic, high-impact, large and complex programmes ensuring alignment with organisational objectives and industry practices.

Keywords
Small change