A. The product challenge
1. Statistical organisations no longer have a monopoly on the means to inform society about social and economic developments. Others are starting to create outputs in competition with ours, and although we may question the accuracy of some of these outputs, it is undeniable that they are often more timely, more easily accessible, and better promoted. We risk losing relevance. We could react defensively and try to maintain our foothold by legislation etc. Instead, we propose to actively pursue a course in which we use our strengths and resources to create the statistical outputs that will give our stakeholders a clear perspective on what is happening in the world today.
2. Our output has traditionally been determined by the demands of our respective governments and other customers. The process is one of reasoning back from the output desired to survey design because often few or no pre-existing data were available. This paradigm has shaped the way official statistics are designed and produced. We have elaborate coding schemes like the International Standard Industrial Classification (ISIC) on which we have based carefully designed surveys.
3. Statistical organizations, have traditionally looked at data through a filter that discards all data without the stamp "officially sourced" or "officially collected" as generally not fit for our use. In the past, there was little of that type of data anyway. Within the constraints of legislation and policy, many of us have been able to introduce administrative data in our daily work, either to replace or supplement survey data.
4. In household expenditure surveys in many countries, people kept track of their daily spending in a small book. This information was then processed into statistics on household spending. While it was feasible to ask people to do this 10 or 20 years ago, the dynamics of current society are such that:
a) People are conducting many more transactions daily;
b) They have less patience with statistical organizations compared to other claims on their time and attention;
c) Representativeness is becoming problematic because some groups in society are not willing to cooperate at all.
5. In the future it will become increasingly unrealistic to expect meaningful statistics from this approach, even when results are collected and transmitted electronically. This is a typical statistical product from a past era, when this was the only way to get at these data.
6. In an information society there is an abundance of data. To a statistical organization this represents opportunities as well as threats. We have to learn to look at the available data and find the opportunities hidden in there. We also see new and interesting uses of statistics, prompted by the availability of so much data. Another important point is that nowadays it is much easier to get data that cover more than the traditional national statistics users would need. We do not, however, have the mechanisms in place to make full use of these data.
7. Perhaps one advantage that we retain, at least for now, is our wide range of outputs. We are able to integrate and reconcile data from diverse sources into a consistent and comprehensive picture of society, rather than focusing on single issues.
8. The above is a strong indication that we have to rethink our products at the risk of becoming obsolete. For example, some products should be designed and created as international statistics in international collaboration right from the start.
B. The process challenge
9. The changes in our society drive the need for more and quicker statistics. Quality is multi-faceted, with different users placing different emphasis on dimensions such as accuracy and timeliness. The challenge for statistical organizations is to be sufficiently flexible and agile to provide statistics according to user needs, at an acceptable cost. Statistical organizations are starting to acknowledge that it is becoming too expensive for each and every one of them to individually change their tailored production systems to meet user expectations.
10. Statistical organizations have a long experience in harmonising statistical products and regulating requirements within the different statistical domains. Providing statistical results has been regarded as statistical production for decades, but international cooperation has not concentrated much effort on harmonising production processes, leaving many possibilities for increased standardisation of production means.
11. However, the last few years have been a wake-up call for statistical organizations. New technologies and communication facilities have sprung up and are reforming the landscape in which we do our business. For national statistical organisations, there are often two dimensions to consider. Collaboration and harmonsiation can be across government at the national level, or they can be across the “official statistical industry” at the international level. Whilst recognising that reconciling these two approaches can be challenging, this vision focuses on the international dimension, for the simple reason that considerations at the national level are often very country-specific.
12. Most international collaboration in the area of information and communication technology (ICT) and the automation of processes and statistical methods has been between specialists. The common objectives have been to share experiences and best practices, and occasionally carry out research and demonstrate innovation. However, collaboration and the common uptake of results in this area have proven to be difficult. The specialists have the power to agree, but lack the authority to initiate substantial changes in their organizations.
13. For real progress, senior and top level management have to step in and drive the changes, actively request organizational innovation and show commitment to international cooperation.
14. The High-level Group for the Modernisation of Statistical Production and Services (HLG) has been established to add power and governance to the various groups working under the umbrella of the Conference of European Statisticians, and other groups working in the area of modernising statistical production. It provides strategic overview and guidance to support the realisation of a common vision.
15. We need to define our strategy in two directions:
(a) Statistical outputs: New and better products and services more tuned to the way the world is operating today, and created from an integrated, global perspective. They will help us to stay relevant;
(b) Production methods: Different and better processes and methods tuned to delivering our products at minimal cost with greater flexibility and in cooperation between institutions. This will help us to improve efficiency and effectiveness.
A. On products
16. Statistics can be a most exciting business, showing causality where there was none expected and providing insight into the inner workings of our society. It used to be very difficult to fulfil that role, data being scarce and expensive; a survey is a costly instrument. However, with the transformation into an information society, data are everywhere and are much cheaper than they used to be. Slowly people are beginning to understand the real value of this fact. For example, although they did not start out as such, social networking sites and search engines are now perceived as data collection devices. We, as impartial organizations with legislation on our side, are in a unique position to connect to the data of the emerging information society and transform them into something useful.
17. As the global dimension of events gains importance we can no longer work on a national level only and rely on international organizations alone to consolidate. We need to expand our work and deliver products that explain what is happening on a multinational level. In some specific statistical domains, only cross border data make sense, for example globalisation, enterprise groups and climate change.
18. The raw materials, the data we use to create our products, will need to be found in the data that are already available throughout society. The opportunities these data represent will need to be transformed into concrete statistical products and data-based services. The active pursuit of data and the creation of products and services that give insight from an impartial perspective, our unique selling point, will be our new mission. It will undoubtedly mean that our organizations will have to leave their comfort zone and will have to question the work that seems so normal at present. This will require new communications strategies focussed on brand and highlighting to users what it is that makes our products and services stand out from those of other data providers.
B. On processes
19. The production of statistics should be based on common and standardised processes, transforming raw data into statistical products according to generic and commonly accepted information concepts. In some cases, standardisation of processes and the availability of international data sources could lead to statistical production at a multi-national level, for example perhaps price statistics could be produced in a single production process for more than one country. The increased harmonisation of production methods will increase the potential for common solutions provided by the statistical community or by commercial partners. Whilst this could imply shared production systems, the scenario of common solutions being implemented in organisation-specific environments is equally valid, particularly given current constraints relating to data confidentiality.
20. We view this as the industrialisation and standardisation of statistics production.
21. Each statistical organization is a factory of statistical information. Together they form the “official statistics industry”. Like any established industry, the production of official statistics should have its own industrial standards. On the one hand, this will provide a necessary foundation for development and exchange of the means of production among the statistics producers, and potentially create a market of commercial interest to our benefits. On the other hand, this will consolidate the use of our statistical outputs in the global information community, making them readily accessible, interpretable and comparable.
III. The road to new statistical products
22. Key to a new understanding of our profession is the fact that the changeover to an information society will cause a profound change in our business.
23. Common wisdom states that you need to research the market for what it needs and then produce what is needed. That is not the way the automobile was born, or the “smart phone”. The fact is that these artefacts were not needed at all; market research would not have revealed them as opportunities. What happened is that the presence of enabling technology and innovative thinking created a product that was at first only of any importance in the eyes of the innovators and their funders. They struggled considerably in early incarnations before the general public caught on. Once manufactured in a sufficiently mature form they became “must haves”. The process of innovation at work is not only imaginative design of new products, but also relies on well-tuned consciousness of enabling technologies, availability of raw materials and conceptual readiness.
24. Obviously we cannot force our organizations to create successful innovative products. We can, however, create an environment and conditions in which the innovators will thrive and task them with research in the right direction. The existence of the HLG, and its programme of work, can help here. The HLG provides a focus for modernisation and a point of reference globally, which can be used to help identify organisation-specific needs and justify the investment to address them.
25. We believe that the amount of available data is another key factor. These data have to be actively sought, and their possibilities researched. We need to assess whether this is a viable direction for our trade before we think about full commitment. For the coming years we propose to delve deeper into the opportunities we perceive and thus demonstrate their added value.
IV. The road to standards-based modernisation
26. The strategic goal of modernisation involves aligning a complex set of prerequisites.
27. In industries that have embraced standardisation, it is not important by whom and where parts are produced. Parts are cheap and available as variants of the same basic design. It is all about reducing the cost of the production process. Cost is defined as human labour, materials and duplication of efforts.
28. In Figure 1, the objective of standards-based modernisation is symbolised by the blue square. In this square we have fewer but more standardised and cheaper ways of producing statistics, in contrast to the present statistical production approach with lots of different methods and tools and a much larger cost. The basic aim is to converge around a core set of common standards, and decrease the unnecessary diversity and duplication within the official statistics industry.
29. Standardisation is equally applicable with regards to methodology. It is not necessarily about imposing a single solution. It implies adoption of common solutions where possible, and that any variations must be justified based on a rigorous evaluation of the alternatives. This also provides an impetus for auditing existing practices.
30. Over time, standardisation will result in a methods library, from which sensible solutions can be fetched, configured and implemented according to the actual needs.
31. The new statistical process (the blue square) is to be seen as the area where the statistical production is compliant with four constraints. The Generic Statistical Business Process Model (GSBPM) describing processes, the Generic Statistical Information Model (GSIM) describing the information objects that flow between processes, standardised methods and common technology. These cornerstones of standards-based modernisation form the basis for the Common Statistical Production Architecture
32. To enable statistical organizations to produce statistics in a standardised way, we first need to agree at the conceptual level. We have to bring our concepts within the blue square under the umbrella of the GSBPM and the GSIM. This is a very high ambition which will take time. A first goal for the models is to act as a common language. We are lost if we cannot communicate properly.
33. It is obvious that these standards will need to evolve further over time, in the same way that common industrial standards evolve. The HLG needs to actively promote the use of common standards and ensure they are updated as necessary to ensure they remain relevant.
34. Having united on concepts, this will allow us to start moving our diverse production systems slowly into the area where we are building our Common Statistical Production Architecture, and reducing labour costs.
35. There should be a requirement for any international collaborative initiative in this area to describe how their efforts contribute to the overall goal of modernisation, either by enhancing the frameworks, models and standards at the conceptual level, or by enhancing methods of work or technological solutions at the implementation level. Achieving this, any accepted and agreed international collaboration would contribute to the overall, global goal of modernising statistical production. Overlaps could be avoided, and any initiative could be appreciated, rather than being overwhelmed by scepticism.
36. The increased cost effectiveness represented by the modernisation of statistics should be realised by dividing the whole process in four phases:
(a) Product design; we need to start designing statistical products to make use of common, re-usable tools and processes;
(b) Process design; the statistical production process (manual and automated) should be designed by configuring components that are modular in nature and exchangeable between organisations, and as independent as possible from subject-matter constraints;
(c) Production: the statistical process should be executed by machines, with as little human intervention as possible, and with short turnaround times (close to real time should be possible), to minimise operational costs;
(d) Analysis: Statistical subject-matter specialists should use outputs and intermediate results to publish articles and do research with advanced tools and as little human intervention as possible.
37. Although, in the short-term, production and analysis will probably be carried out separately by each organisation, design should be done in collaboration between organisations, and enterprise architectures will need to be aligned with an “industry architecture”. This architecture should be flexible enough so that it doesn’t become a constraint, or “straightjacket”, instead, it must be able to evolve over time to accommodate new types of services and products.
38. This change is urgently needed. The demands of modern society are completely different from those of 20 years ago and we risk becoming obsolete. Our organisations have a reputation to uphold for producing statistics of indisputable quality, which inevitably implies a time lag. However, we should quality assure our processes to ensure that our outputs are more timely.
A. Energise innovation
39. We need to establish a culture for change. Among our most important assets are our human resources. That is where we keep our knowledge and our culture. In most organisations there is a good supply of forward-thinking people. The challenge is to unlock this potential. We should encourage an entrepreneurial attitude and look for ways to change the culture in our organisations where necessary.
40. Another important asset is our official status, backed by legislation. This distinguishes us from most other statistics producers. In the past, this asset has led to big advantages in data collection. We could collect data from sources that were not accessible to other parties. This advantage is declining, however. There are already risks that our main sponsors may turn elsewhere. Our challenge is to use the combination of "informal" and "formal" data to create products that have an added value for our governments and other important parties.
41. We need to systematically evaluate new data sources, and learn how to use them. Finding new ways to produce statistics from data that are already available is challenging. Potential new data sources can be found either from administrative registers, or the rapidly growing private sector data repositories. The era of “Big Data” is here, and we need to explore the potential benefits it offers. The “Semantic Web” could have significant impacts on data management and dissemination. At first sight these look like technology issues, but they are not. New competencies are needed, and research must be undertaken to understand how best to exploit these developments and rethink our products in a truly innovative way.
42. Innovation must be a management driven part of our core business. In order to drive our workforce outside its comfort zone and try new ways of producing statistics, ownership and responsibility for innovation in statistics must be clearly defined and mandated. In a world that is in "fast forward" mode, we have to become more outward looking.
43. True innovation takes nothing for granted. We need to think out of the box. Looking back at previous attempts to create standardisation in the statistical system, only minor results have been achieved. Most of the attempts lost momentum by taking the existing situation and underlying statistical processes for granted. Our challenge will be to do the opposite and ask ourselves the hard questions. If we keep doing what we have always done, we will not advance.
44. Being part of an information network must become part of our culture. Instead of being self-contained and self-sufficient as a statistical system, we must realise that we are becoming part of a more complicated network of data providers and information producers. We need to be aware of the processes that create the data we are interested in. This means that we must take a place at the negotiating table on numerous occasions to fight for our position in the value chain of information producers. We must show our added value as professionals in statistical data processing.
B. Standardise to improve flexibility
45. Standardisation is an enabler for change. The objectives of standardisation are to gain efficiency, to be able to automate processes and to prepare for added value of various kinds. Through standardisation we can re-allocate resources, change the focus from production to products and become more agile to respond to the needs of our users and societies.
46. To get product quality without a huge labour cost, process quality is needed. Our challenge is to standardise our processes and achieve better and more uniform product quality, whilst reducing costs. We may need to learn to negotiate product quality with our customers and find economic optimums. We might even consider a number of standard quality levels for our products.
47. Standardisation is not a goal in itself. It needs well-defined business cases showing costs and benefits. Accordingly, when reaching the objectives, the profit from successes must be captured and re-allocated.
VI. Conclusion and pressing forward
48. The changes we are proposing are profound and it will be a challenge to achieve them, but the world in which we used to define our role in no longer exists. For us, as statistical organisations, there is no other way forward than to adapt. This will take a lot of time and effort, and we will have to collaborate to retain the relevance we need. But then, our world is changing and we have to change with it.
49. We, the HLG, seek your commitment to implement this vision in the coming decade and together create a statistical industry that can keep up and even be ahead of its time.