In GAMSO v1.1, under Corporate Support, there is a box called "Manage Quality"
Manage quality
These activities cover developing and administering a quality framework and cross-cutting work with tools to assure quality, i.e. compliance with the quality framework that should cover quality linked to the organisational framework, processes and products. Such tools comprise quality indicators (including balancing quality components), user surveys, self-assessments, quality reviews or audits, certification and labelling of statistics. Quality documentation here refers to the organisational level and covers quality declarations, policies and relevant guidelines such as guidelines on handling of errors and revisions. These include:
- Manage quality framework
- Manage quality assurance tools
- Manage quality documentation
In GSBPM v5, Quality is an overarching process.
13. Over-arching processes with a statistical component include the following. The first four are most closely related to the model. Quality and Metadata management are specifically highlighted in model diagrams and are elaborated further in Section VI.
- Quality management - This process includes quality assessment and control mechanisms. It recognises the importance of evaluation and feedback throughout the statistical business process;
Quality Management
106. Quality concerns organisations, processes and products. In the present framework, quality management over-arching process refers mainly to product and process quality.
107. The main goal of quality management within the statistical business process is to understand and manage the quality of the statistical products. There is general agreement among statistical organisations that quality should be defined according to the ISO 9000-2005 standard: "The degree to which a set of inherent characteristics fulfils requirements"1 . Thus, product quality is a complex and multi-faceted concept, usually defined in terms of several quality dimensions. The dimensions of quality that are considered most important depend on user perspectives, needs and priorities, which vary between processes and across groups of users.
108. In order to improve the product quality, quality management should be present throughout the statistical business process model. It is closely linked to Phase 8 (Evaluate), which has the specific role of post-evaluating individual instances of a statistical business process. However, quality management has both a deeper and broader scope. As well as evaluating iterations of a process, it is also necessary to evaluate separate phases and sub-processes, ideally each time they are applied, but at least according to an agreed schedule. Metadata generated by the different sub-processes themselves are also of interest as an input for process quality management. These evaluations can apply within a specific process, or across several processes that use common components.
109. In addition, a fundamental role in quality management is played by the set of quality control actions that should be implemented within the sub-processes to prevent and monitor errors. The strategy could be reported in a quality assurance plan.
110. Within an organisation, quality management will usually refer to a specific quality framework, and may therefore take different forms and deliver different results within different organisations. The current multiplicity of quality frameworks enhances the importance of the benchmarking and peer review approaches to evaluation, and whilst these approaches are unlikely to be feasible for every iteration of every part of every statistical business process, they should be used in a systematic way according to a pre-determined schedule that allows for the review of all main parts of the process within a specified time period2 .
111. Broadening the field of application of the quality management over-arching process, evaluation of groups of statistical business processes can also be considered, in order to identify potential duplication or gaps.
112. All evaluations result in feedback, which should be used to improve the relevant process, phase or sub-process, creating a quality loop.
113. Examples of quality management activities include:
- Setting and maintaining of the quality framework;
- Setting of global quality criteria;
- Setting process quality targets and monitoring compliance;
- Seeking and analysing user feedback;
- Reviewing operation and documenting lessons learned;
- Examining process metadata and quality indicators;
- Internal or external auditing on statistical processes.
114. Quality management also involves institutional and organisational factors. Such factors are included in other GSBPM over-arching processes (e.g. Human resources management, Statistical programme management) although they can have an impact on quality.
Questions: How do we manage having Quality in both GAMSO and GSBPM? Does the inclusion in GAMSO mean that Quality should not be in GSBPM? What does that mean for all the quality indicator work that has been done?
6 Comments
user-8e470
Jenny Linnerud - Maybe you know some background that would help us understand this issue?
Jenny Linnerud
Quality is and should be everwhere.
A simple division could be
Guidelines, support & training in quality issues are in GAMSO under Corporate support. Also choice of a quality framework.
Monitoring and measuring the quality of an individual product, individual production (sub-)process belong naturally to GSBPM.
The distinction is less clear for groups of products/processes e.g. demographic statistics/data collection by interview.
Development of the quality capability in the form of guidelines, support & training for all or a group of products/processes
would be in Capability development in GAMSO and then moved to Corporate Support when they have been tested and approved for use.
Thereafter they woould be implemented, monitored & measured in the production of an individual product/process or group thereof.
Feedback based on experience in production in the form of one or more evaluation reports could then be given to Corporate Support
e.g as a request for more training, an update to the guidelines to make them more understandable etc.
Ultimately quality is about meeting the needs of the users. So who are the users of the product, the process, the institution? End users,
employees, management all have different needs and quality issues need to be addressed at different activity levels.
Hope that helped a bit. I can ask our quality experts for more input if it is needed ...
In a way, this reminds me of the request at every international meeting for a definitive list of metadata types. The answer would always be
that it depends on what are you going to use the list for - communicating to end users, improving your processes etc.
Jenny Linnerud
Regarding the quality indicators: It may well be that some of them should be part of the Corporate Support.
Perhaps it would be possible in a future revision to have different chapters for different levels (institution, product, process).
Marina Signore
I agree with Jenny's comments.
We could define the three levels as follows:
Quality at Institutional level
Quality at Institutional level concerns the adoption of a Quality Policy or a Quality Assurance Framework as an overarching tool to guide quality management and ensure quality of statistical activities and outputs
Process quality
Refers to the quality of the operations and instruments used in the data production process. To monitor and improve the statistical production process is a way to improve data quality. The Quality Indicators (QIs) defined for the GSBPM allow for a monitoring and assessment of each step of the production process. Thus, they support a process-oriented approach.
Product quality
Refers to the quality of statistical output (or products). It is usually expressed as a set of components which have some attributes or requirements that are to be met (e.g. accuracy, relevance,…). It focuses more on the errors (e.g. sampling and non-sampling errors) and on measuring their impact on final estimates or data. Some well-known output quality indicators (e.g. sampling errors, non-response rate,…) are included in the QIs for the GSBPM since they also provide information on process steps (e.g. data collection)
How to describe Quality in GAMSO and GSBPM?
Following the general definitions:
Examples
Examples of Quality activities/indicators at the Institutional level
a) Defining and adopting a Quality Framework (e.g. UN NQAF, ES CoP)
b) Defining and adopting a procedure for internal or external auditing for statistical processes
Examples of Quality activities/indicators at the process and product level
a) Process quality monitoring through the analysis of QIs (e.g. non response rate) in different phases of the GSBPM
b) Definition of an action plan for improvements in the “evaluate” phase of the GSBPM (phase 8). This applies to recurrent statistical processes.
Essi Kaukonen
Dear Colleagues,
Quality is an important issue in the statistical production. Quite many organisations already use GSBPM but do not implement GAMSO. Hence, in our opinion, Quality Management should be clearly visible in the GSBPM-model as it is today in the version 5.0.
How about the idea of giving those Corporate Support activities, which are most closely related to the statistical production a special place in the GAMSO-model nearby “Production”? Following, these activities could then be included in both models?
Essi Kaukonen
Statistics Finland / Standards and Methods
InKyung Choi
(Feedback from Australian Bureau of Statistics (ABS); 2 October, 2017)
Quality - GAMSO/GSBPM
In the ABS, Statistical Production Activity Model (SPAM) is an element of our broader Business Activity Model (BAM). ABS BAM provided a starting point for Statistical Network BAM which then informed GAMSO.
Where GAMSO is much less detailed than GSBPM (although broader in scope) the other three elements of ABS BAM are similar in detail to SPAM.
ABS finds this approach works well for us but we understand why GSBPM might not be "absorbed" into GAMSO the same way as SPAM sits within BAM for us. It does mean, however, that ABS observations are on a slightly different basis when talking about SPAM compared with the rest of BAM as distinct from GSBPM vs GAMSO.
In regard to Quality, we broadly agree with the sentiment "quality is everywhere".
There are aspects of quality managed within the ABS equivalent of GSBPM (ie in regard to the operations and outputs of a specific statistical program) and there are also aspects managed beyond SPAM/GSBPM.
In addition, aspects of quality management occur throughout SPAM/GSBPM.
For example,
In addition, ABS places quality gates at other points of business processes modelled in accordance with SPAM.
Describing quality as an over-arching process should include an impression that this includes the fact that "quality" is (or should be) weaved into every activity explicitly described in GSBPM as opposed to quality being purely a separate "over-arching" activity somehow separate and additional to the business activities characterised in each phase of GSBPM.
Within the ABS, business areas have asked similar questions about quality and ABS Information Model (AIM)/GSIM. Some look for classes labelled "quality x" (eg quality declaration, quality statement, quality target, quality measure, quality indicator) within AIM/GSIM but don't find them. We continue to believe it is appropriate for these to be built up as purpose specific applications of classes defined more generically within AIM/GSIM (eg Variables, Referential Metadata, Parameters etc) rather than adding a whole set of classes that are similar to - but purpose specific compared with - classes already defined in AIM/GSIM.
Nevertheless, both in regard to GSBPM and GSIM such questions are commonly asked. This suggests a supporting guide on application of Modernstats Standards to measuring and managing quality might be useful? Alternatively (or additionally) some simple case studies could be provided so the approach taken by specific agencies is visible.
In the latter case, some aspects of any case study are likely to be agency specific while other aspects may, implicitly, represent a useful general practice that other agencies could adopt. A coherent guide is probably more valuable, but also entails more work to build and agree?