Case study: Statistics Norway

Metadata strategy

Statistics Norway has in the course of time developed many different metadata systems. This led to the same information being stored several times in several places making the availability of updated and consistent information difficult. In later years, there has been a strong focus on the need to link existing systems and a requirement that new metadata systems should not be built in isolation. To facilitate this, Statistics Norway developed a metadata strategy, which was approved early in 2005. The strategy focuses on establishing a conceptual framework, clear roles and responsibilities, and a stepwise development involving integration and linkage of systems. For more information on our metadata strategy see: http://www.ssb.no/english/subjects/00/doc_200508_en/doc_200508_en.pdf

Current situation

The objective of Statistics Norway's current work on metadata is to develop an integrated metadata system that will contribute to effective statistics production and dissemination, in addition to improved quality of statistics. Different metadata systems are being linked together making the metadata more easily accessible for all users. Metadata should be updated only in one place. Metadata projects completed after 2005 include: Documentation of key metadata concepts Metadata portal on the Internet and the intranet Variables documentation system About the statistics as a content management system instead of just a text document Analysis of the end-to-end creation and re-use of metadata in one production cycle for one statistic. A service library for master metadata systems Improved access to micro-data for researchers (About the data collections) Administrative system for projects, products and processes Current/planned metadata projects include: A master metadatabase for questionnaires is under development.

Metadata Classification

The only metadata classification we have needed in-house so far is the distinction between conceptual and contextual metadata. This probably reflects the fact that we are only just beginning to try to integrate the metadata in the production lifecycle.

Metadata system(s)

Datadok - File descriptions (implemented) We document all permanent archive data files in our file documentation database Datadok. The database was built in 1998 but wasn't mandatory until 2002. Vardok - Variables documentation system (implemented) The overall purpose of the variables documentation system is to document variables in a central location, accessible by all, and to function as a tool for harmonising names and definitions. There is a two way link between Vardok and Datadok (file descriptions database), a one-way link from Vardok to Stabas (standard classifications database), a two way link between Vardok and StatBank (dissemination database), a two way link between Vardok and Metadb (system for documentation of event history data) and a one way link from About the statistics, About the data collections and the statistical metadata portal to Vardok, via web services. 2006 was the last year in the development phase for the Vardok-project. Stabas - Standard classifications database (implemented) The overall aim of Stabas is: • To make work with and the use of standards simpler and more efficient • To ensure systematic use of standards across different statistical areas One main task is to make approved versions of the central statistical classifications available in a database system where they can be taken out at different aggregation levels, together with texts in different languages and relevant documentation, and where the classifications can be exported to other IT tools. 2004 was the last year in the development phase for the Stabas-project. Service library for metadata systems (implemented) The purpose of this project was to • Create a library of services for the master systems Vardok, Datadok, Metadb and Stabas. • Define a framework for the description and formulation of SSB's metadata based on international metadata models (e.g. Neuchâtel) and standards (e.g. ISO/IEC 11179). The project began in 2005 and ended in 2008. Metadata portal (implemented) The overall purpose of the metadata web page is to make Statistics Norway's metadata systems more accessible and easier to use. Both internal and external users will get easier access to the metadata by displaying the contents of these systems in a common web page. The project began in 2005 and ended in 2009. Metadata portal: http://www.ssb.no/english/metadata/ Metadb - metadatabase for event history data (implemented) Metadata for FD-Trygd (Social security database) and NUDB (Norwegian national Education Database). FD-Trygd: details on demography, social conditions, social security, employment, search for employment, government employees, income and wealth. Data from1992 to the present. Continuous regulatory and technical changes. NUDB : All individually based statistics on education from completed lower secondary education to tertiary education from 1970 to the present. System for questionnaires Systems exist but are being replaced. Administrative system for projects, products and processes (implemented) This administrative system can be used to take out reports that combine manhours and other administrative information. It includes important information on all products in Statistics Norway such as financing, response burden, responsible division and person, response rates, frequency, laws, EEA requirements, subject field etc. This system contains both metadata and data. About the data collections (implemented) Researchers frequently use data collections from Statistics Norway for their research. However, the process from finding out what you need, to actually getting the data, may be long and troublesome, especially for inexperienced researchers. Statistics Norway has therefore (with support from the Research Council of Norway) developed a website to make information about this process more easily available. Among other things, this page provides the users with documentation of several data collections. Each data collection has a general description e.g. of data quality, and it also contains a list of relevant variables, including variable documentation from Vardok. A new system is being scoped, hopefully with even more automatic solutions. About the statistics (implemented) About the statistics is metadata that describes each statistics that is published by Statistics Norway. It contains administrative information, information about statistics production, variables, concepts, sources of errors and uncertainty, comparability, coherence and availability. About the statistics now uses a CMS (Content Management system)-platform. CMS makes it possible to link About the statistics to Vardok and Stabas. StatBank - dissemination database (implemented) StatBank Norway is a service where you may select scope and content of each table, and then may export the result in various formats to your own PC. This system contains both metadata and data.

Costs and Benefits

Examples of costs: Metadata strategy: A total of 1420 man-hours have been used in preparing the metadata strategy with ca. 35% of resources from IT. Vardok: A total of 12690 man-hours have been used in development with ca. 70% of resources from IT. A total of 476 man-hours from standards were used in 2007 for continued harmonisation of names and definitions, and training of personnel in the six new divisions. 294 IT man-hours were used in 2007 for maintenance and minor changes to the system. Stabas: A total of 7200 man-hours have been used in development 2002-2004 with ca. 75% of resources from IT. However these man-hours do not include the development performed by Statistics Denmark on the editing application. A rough guess for this would be 2500 man-hours. The system required approximately 1000 man-hours in production each year from 2005-2007 with ca. 70% from IT. We are now planning a new version of the editing application that we hope will be more flexible and less costly in production. Benefits: Statistics Norway's Strategies 2007 emphasise systematic quality control of products and processes. Statistics Norway's IT-strategies 2007 emphasise that - metadata systems contribute to simplifying, improving and re-use of work processes - data that are disseminated and exchanged must in addition to an agreed structure have sufficient metadata to give them meaning - use of metadata systems are a pre-condition for the development of efficient data capture solutions according to Statistics Norway's data capture strategy.

Implementation strategy

All our metadata projects are based on a step-wise approach.

IT Architecture

Statistics Norway's technical solutions shall be built mainly upon the principles of service-oriented architecture. Guidelines on this are presented in Norway's eGovernment plan. All solutions for external users and most solutions for internal users shall: • Have support for open standards. • Be platform independent. • Be component based. • Have support for the packing in of data and functions in the form of services (web services). These are central principals in service-oriented architecture. By applying these principles, applications and services can reuse existing functionality/components completely independent of the system they were developed in. In addition, by use of this technique, we can extend the lifetime of older applications, which have important functionality we wish to expose, just by creating a service layer on top of these. This increases the possibilities for collaboration between old and new applications in a completely new way, which gives benefits in the form of shorter development time, increased reuse and more consistent systems. This also enables us to replace systems behind the scenes, because communication with these is not directly exposed to the users. For more information see: http://www.ssb.no/english/about_ssb/strategy/it_strategy.pdf System architects System architects are introduced for each of the following areas in the top-level information architecture: data collection, metadata and dissemination. The mandate for this role supports the system architect's responsibility to ensure that IT development projects are in line with the IT strategy. Solution architects Solution architects are appointed for each new development project to ensure that the systems development is in accordance with the current IT-strategy, architecture principles,recommended tools and practices.

Metadata Management Tools

All our master metadata systems have Oracle databases. Links are mostly hard-links between these databases. Our web applications obtain metadata from our service library for metadata systems.

Standards and formats

Our classifications system is an implementation of Neuchâtel Terminology Model Part 1 Classifications v2.0 with the addition of an attribute on the Correspondence Item for Item Change from v2.1. We are contributing to a task force looking at a revision of the Neuchâtel Terminology Model Part 1 Classifications v2.1 Our variables system is a partial implementation of Neuchâtel Terminology Model Part 2 Variables. The extent to which we follow ISO/IEC 11179, is best described by the figure in the attachment to this chapter where the number of instances of the objects per 2012 are given in the brackets.The figure in the attachments illustrates that there is little re-use of data elements or value domains in the current archive. This is a situation that we hope to address in the coming years. We are considering using DDI in connection with micro-data for researchers. We have used definitions of key metadata concepts from SDMX MCV where possible. We have contributed to the development of GSIM (Generic Statistical Information Model) v1.0. We are contributing to a task force looking at the flow of GSIM v1.0 information objects in GSBPM v4.0.

Version control and revisions

Metadata have valid to, valid from and last updated. Program code for web-services is checked in and out of subversion.

Outsourcing versus in-house development

Improved editing tool for our classification database - previous editor was developed out-of-house. Analysis of the end-to-end creation and re-use of metadata in one production cycle for one type of statistics - no system development required (purely analysis). Service library for master metadata systems was developed in-house. Metadata portal on the Internet and intranet was developed in-house. Variables documentation system was developed in-house About the statistics as a content management system instead of just a text document was developed in-house Improved access to micro-data for researchers (About the data collections) was developed in-house Adminstrative system for projects, products and processes was mainly being developed in-house, but we had substantial help from external consultants on-site. Continuing development of a master metadatabase for questionnaires - in-house development, but with help from external consultants on-site.

Sharing software components of tools

 .

Overview of roles and responsibilities

Subject matter statistician, survey manager, metadata manager, senior advisers in standards, IT developers, system architects, solution architects and web designers are all important roles in metadata/statistical lifecycle management in Statistics Norway. List of metadata contacts Jenny Linnerud (jenny.linnerud@ssb.no) Anne Gro Hustoft (agt@ssb.no)

Metadata management team

Our core team for metadata systems consists of one senior adviser, one system architect for metadata systems and four programmers in the Department of IT. When necessary we draw on other IT expertise from the same department. Metadata system maintenance is carried out by two or three people in the Department of IT. Maintenance of the system contents, i.e. the metadata, is carried out by all statistical divisions with support from the senior adviser in the core metadata team. Details per system are as follows: Our systems for classifications and variables and our metadata portal are owned by the Department of IT. In-house development (2 developers and two customers) has been carried out by the Department of IT. Maintenance of the systems is carried out by the same department (1 developer, 1 system architect and 1 system owner) with the addition of two people from the Department of Data collection and statistical methods in the case of classifications. Maintenance of the metadata in the systems is the responsibility of all 20 statistical divisions. Our master metadatabase for questionnaires is being developed by 3 developers from the Department of IT, 2 external consultants and a project leader from the Department of Data Collection and statistical methods. This metadatabase will replace two previous metadatabases. Our service library for master metadata systems was developed by 7 developers from the Department of IT and is being maintained by one. Our Administrative system for products, projects and processes is owned by the Department of administrative affairs and director general. The previous system was outsourced by the previous owner but the new system was developed by 2 developers from the Department of IT in cooperation with the new system owner and one external consultant. Our system for file descriptions was developed and is being maintained in the Department of IT. Maintenance of the metadata in the system is the responsibility of all statistical divisions. Maintenance of the metadata in the system for event history metadata is the responsibility of two statistical divisions (Division for Social Welfare Statistics and the Division for Education Statistics).

Training and knowledge management

All new employees participate in a course designed to give them an overview of all areas in Statistics Norway. Most new employees attend this course within 3-6 months of starting work in Statistics Norway.

Partnerships and cooperation

Statistics Norway participates in relevant international meetings e.g. METIS, MSIS and informal groups e.g. Neuchâtel (classifications and variables). In addition, there is a strong tradition of Scandinavian collaboration (e.g. dissemination database StatBank) and collaboration within the Statistical Network (currently Australia, Canada, Italy, New Zealand, Norway, Sweden, United Kingdom).

Other issues

So long as metadata is not an integral part of the statistical production cycle it will be prioritised lower than the publication of statistics.

Lessons learned

Top management support is essential. Make a metadata strategy. It is important that we can refer to formal documents like the metadata- and IT-strategy (which has been approved by the board of directors) in our metadata work. In the same way it is useful that the list of key metadata terms promoted for use within the statistical office has an official "stamp". Use step-wise development of metadata systems with active user involvement and regular delivery of functionality. Ensure continuous follow-up of progress and quality with direct feedback to users and regular reports to middle and top management. One of the biggest challenges in management of metadata is allocating the necessary resources. Releasing good quality statistics within the planned time schedule is the primary task for the subject matter divisions and documentation will often have a lower priority. It is therefore crucial that the management stresses the importance of documentation and increases the status for this kind of work. Harmonising variables between subject matter divisions is also a considerable challenge and an important tool to improve the quality of metadata. Several subject matter divisions may use the same variable names, but define them differently. In some cases this is necessary because of laws and regulations, but this is not always the case. We had meetings where contact persons from divisions using variables with similar names came together and discussed the definitions, e.g. if a division could change the wording of their definition to such an extent that other divisions might use it as well, which would allow us to reduce the number of definitions to one instead of e.g. three. This is a time consuming work which requires a lot of resources, both to monitor where harmonisation is needed and to do the job. The possibility to release metadata on the Internet makes it easier to motivate subject matter divisions to document metadata and improve metadata quality. We think that to really make metadata work a natural part of everyday life in the subject matter divisions, we have to include the metadata systems in the production cycle. Then we can establish routines where the handling of metadata is included in all relevant production steps. So far the metadata work in Statistics Norway has been focused on implementing metadata systems and filling them with relevant content. Now we need to focus more on integrating the metadata(systems) in the production cycle.