18. A number of statistical organisations have developed or are developing international, national and/or organisation wide strategies for data integration.
19. Some of the most common activities when planning for data integration include the use of co-operation agreements for transferring data, preparation of legal documents for establishing and/or maintaining use of the data, and developing long-term partnerships (formal or informal) which consist of two or more institutions using the same data.
A. Access to data
20. A key requirement is to have access to the desired data. In most countries, data can be accessed freely by the statistical organisations. In some countries and cases, data are only available with payment and in some cases the data exists but cannot be accessed.
21. A legal basis is often important to access the data for statistical purposes. A sound approach is to ensure national legislation is aware of already existing administrative sources rather than recollecting data. The usage of administrative sources is often stated in a Statistical Act. It is needed to consider various statistical, methodological, legal and ethical issues.
22. Practical work to develop common approaches starts with data. Some types of data can be used without much difficulty, like web-scraped data or social media sources, government-owned open data or public statistical outputs. In other cases, already available public use files or IPUMS data may be made available. It is also possible to systematically anonymize real data from surveys, censuses, government registrations or privately held big data sources.
23. At the beginning of new collaborative data integration projects, use of experimental data sets makes this easier. The creation and documentation of a set of synthetic datasets allows countries to collaborate on developing common methods, removing issues of confidentiality and encouraging use of the same data formats. As suitable methods, processes and tools are developed in a collaborative way, they can be moved to the secure environments of individual organisations for further testing on real data. There is potential to bring some of these data providers and a group of statistical organisations together to explore mutual benefits and potentially develop global or regional agreements for data supply.
24. An example where many countries have similar challenges which can be assisted with data integration is improving the quality of the Consumer Price Index (CPI) in terms of coverage and real-time quantity. Several data providers operate in many countries and there is an opportunity to develop a common approach that can be used in multiple countries. Some organisations have significant data holdings (for example IRi with 5 years of supermarket scanner data).
25. There are other issues to be considered in securing access to data. These include:
- Access to application programming interfaces (API) can be restricted under terms and conditions
- Some countries may not have access to the same data sources or there may be different formats, various levels of detail or aggregation across countries
- Data may no longer be representative of current situation
- Commercial companies may not be interested in partnering with statistical organisations unless a compelling case is made
- There is potential for interrupted supply
- If the statistical organisation can't get data, they could may still be able to generate and provide meta-information about the data
- Use of a common area for the lodgement and storage of data (not necessarily in statistical organisations) is helpful
- There is a need for secure and efficient file transfer mechanisms for data used in production processes.
B. Partnerships
26. The importance of establishing and maintaining effective partnerships for data integration should be recognised. The types of partnerships include:
- collaboration and sharing experience, approaches and standards with other official statistical organisations
- partnerships with data providers
- cooperation among institutions within countries (e.g. tax offices, employment office) – together deciding the methodology, concepts and classifications
- public/private partnerships with technology organisations, research initiatives and academia.
27. There are many factors which encourage effective partnership. These include:
- establishing personal and friendly connection between organisations - relationship management
- providing feedback on the data regarding its usefulness for official statistics needs
- promoting the goals of the data integration project to the providers and jointly clarifying mutual benefits
- understanding and managing barriers such as costs, capacity and risks
- establishing formal agreements.
C. Skills
28. Some of the essential skills needed for integrating data are:
- leadership and negotiation skills are useful for participating in policy development and in discussions with data providers
- legal skills relate to the legal basis for obtaining data, data protection and a co-operation agreement between the administrative authority and the statistical organisation
- subject-matter statistician skills cover expertise in knowing data content, understanding and analysing data, knowing the statistical process and dissemination methods
- methodological skills relate to all statistical processes such as sampling frame preparation and selection of observation units, data linkage and matching, weighting, time series analysis and seasonal adjustment, data protection, etc.
- programming, software and database skills are needed for construction of microdata databases and for establishing and maintaining generic and non-generic process programs (e.g. for data editing and imputations, validation, aggregation and tabulation, micro and macro data analysis, data protection).
29. The Survey on Data Integration asked organisations about the level of skills and interest in obtaining or providing skills development related to data integration. Interestingly, organisations indicated the areas where training was most required with methods and quality frameworks. Information on these topics can be found in Sections V and VI.