1. Many statistical organizations are facing common challenges. There are major threats to the continued efficient and effective supply of core statistics that come from within statistical organizations.
2. Existing threats to statistical organisations are:
- Rigid processes and methods and
- Inflexible ageing technology environments
3. Emerging Threats include:
- The need to be able to quickly respond to emerging information needs
- Challenges (and opportunities) of increasing use of administrative data and the move to harness alternative sources of data (sensor, satellite etc)
- The ability to attract and retain critically skilled staff in a competitive market.
Background to existing threats
4. Over the years, through many iterations and technology changes, statistical organizations have built up their organizational structure, production process, enabling statistical infrastructure, and technology. The cost of maintaining this business model and the associated asset bases (process, statistical, technology) is becoming insurmountable and the model of delivery is not sustainable.
Background to emerging threats
5. Statistical organizations are being increasingly challenged to respond quickly to emerging information needs. Criticisms levelled at statistical organizations include;
- The inability of underlying statistical models such as classifications and frameworks to remain relevant to modern information needs
- Difficulties in producing statistics that are coherent across information domains
- Difficulty in producing richer insights into key priority areas where traditional statistical outputs are not sufficient
6. For most statistical organizations the underlying model for statistical production is sample survey based. Increasingly there is a need for organizations to make use of administrative or alternative source data to deliver efficiencies, reduce provider burden and make richer use of existing information sources. This requires significant new capabilities that do not exist within the majority of statistical organizations.
7. The skill-sets that underpin statistical organizations are becoming increasingly valuable in the wider market. It is becoming difficult for statistical organizations to compete to attract and retain these skills in this environment.
Describing the current state
8. Historically, statistical organizations have developed their own business processes and IT-systems for producing statistical products. Although the products and the processes conceptually are very similar, the individual solutions are not (as represented by the different shapes in Figure 1). Every technical solution was built for a very specific purpose with little regard for ability to share information with other adjacent applications in the statistical cycle and with limited ability to handle similar but slightly different processes and tasks. This can be referred to as 'accidental architecture' as the process and solutions were not designed from a holistic view.
Figure 1: Accidental Architectures
9. Often it is difficult to replace even one of the components supporting statistical production. Use of these processes, methods and an inflexible and aging technology environment mean that statistical organizations find it difficult to produce and share between systems data and information aligned to modern standards (for example, Data Documentation Initiative (DDI) and Statistical Data and Metadata eXchange (SDMX)). Process and methodology changes are time consuming and expensive resulting in an inflexible, unresponsive statistical organization.
10. Many statistical organizations are modernizing and transforming their organizations using enterprise architecture to underpin their vision and change strategy. An enterprise architecture aims to create an environment which can change and support business goals. It shows what the business needs are, where the organization wants to be, and ensures that the IT strategy aligns with this and supports the implementation of service-oriented architecture approaches both on-site and in the cloud to improve the flexibility, robustness and sustainability of their technology environments. They. Enterprise architecture helps to remove silos, improves collaboration across an organization and ensures that the technology is aligned to the business needs. This work will enable them to standardize their organizations. This is shown in Figure 2 where, as opposed to Figure 1, the countries have standardized their components and interfaces.
Figure 2: The result of standardization within an organization
11. Statistical organizations have attempted many times over the years to share their processes, methodologies and solutions, as it has long been believed that there is value in this. The mechanism for sharing has historically meant an organization taking a copy of a component and integrating it into their environment. Examples include CANCEIS (CANadian Census Edit and Imputation System) and Banff (an editing and imputation system for business surveys). However, most cases of sharing have involved significant work to integrate the component into a different environment with regards to processes, methodology and IT.
12. Figure 3 attempts to explain why the difficulty in sharing or reuse occurs. The figure assumes that the two statistical organizations in the figure develop all their business capability and supporting components and interfaces in a standard way (i.e. they have an Enterprise Architecture as shown in Figure 2). Each organization has standardized within their own organization but not in the same way as the other organization. As shown in Figures 2 and 3 where each country has a different shape of component - Canadian components have a zig zag shape and Sweden have components with slanted edges. There are differences in the way statistical information is structured, conceptual differences between methodologies and differences in technology. If Sweden needs a new component, ideally they need a component with a slanted edge. It can be seen in the third row of Figure 3 that while a component from Canada might support the same process and incorporate robust statistical methodologies, it will not be simple to integrate it into the Swedish environment.
Figure 3: Why sharing /reuse is hard now