Case study: Statistics New Zealand

Metadata strategy

Macro Rendering Error

Current situation

The BmTS programme was started in July 2004, three years on we have largely developed the 'core platform' and see a positive way forward. The BmTS is the main platform that provides the framework for projects related to metadata to develop to. Most metadata related projects are being undertaken within the BmTS suite of projects, but those that are not are governed by the BmTS principles. See 6. Lessons learned (New Zealand) for more information on our experiences developing and implementing this programme. Delivering benefits The BmTS is aimed at delivering a number of benefits to Statistics New Zealand, and provide a solid basis for growth and development, through: Abstracting the business users and their business processes from the underlying data structures and database systems, moving our statistical staff up the analytical 'value' chain and providing an environment that would facilitate the more challenging data integration and data analysis necessary to meet the increasingly complex policy and research needs of government and the wider research community. Creating the flexibility to respond to changes in user needs and demands, to make use of new data sources or methods and to provide a flexible range of information access methods; while also providing the ability to more easily match and confront data in order to increase the quality of Statistics NZ information. Reducing the time to design, build and process information sources, providing more time for analytical and dissemination processes. Building a professional environment that creates a more satisfying working experience. Increasing the use of administrative data, reducing the number of individual collections or the need for new collections to create new statistics. Providing a standard environment and uniform systems that will allow staff to quickly get up to speed with new subject matter. This will also simplify the migration of data and systems as underlying technologies change, while reducing the maintenance cost of separate subject matter systems. Standardising the skills sets and professional development costs of our staff. Utilising a smaller number of larger projects that are more likely to have a real rate of return through the reuse of the investment in a number of business areas. Allowing Statistics NZ to provide standard information management tools and services for official statistical purposes. Metadata Strategy The Business Model Transformation Strategy (BmTS) is designing a metadata management strategy that ensures metadata: fits into a metadata framework that can adequately describe all of Statistics New Zealand's data, and under the Official Statistics Strategy (OSS) the data of other agencies documents all the stages of the statistical life cycle from conception to archiving and destruction is centrally accessible is automatically populated during the business process, where ever possible is used to drive the business process is easily accessible by all potential users is populated and maintained by data creators is managed centrally Established principles of metadata management metadata is centrally accessible metadata structure should be strongly linked to data metadata is shared between data sets content structure conforms to standards metadata is managed from end-to-end in the data life cycle. there is a registration process (workflow) associated with each metadata element capture metadata at source, automatically (where possible) establish a cost/benefit mechanism to ensure that the cost to producers of metadata is justified by the benefit to users of metadata metadata is considered active metadata is managed at as a high a level as is possible - managing at the lowest level is prohibitive metadata is readily available and useable in the context of client's information needs (internal or external) tracking the use of some types of metadata (eg. classifications)

Metadata Classification

The MetaNet Reference ModelTM (Version 2) categorises types of metadata in the following way: Conceptual Metadata describes the basic idea (concept) behind the metadata object e.g. conceptual data elements, classifications, measure units, statistical object types. This type of metadata can be context free (eg the variable 'income' as a concept) or context-related (e.g. 'income' collected in a particular survey). Operational Metadata are the metadata required to view the data from an operational point of view (e.g. record variables, matrix operations, statistical process). This includes all processes and configuration. In other words, the operational metadata is used to explain how the data was created or transformed. Operational Metadata is one of the links between the concepts and the physical data. Quality Metadata are the metadata for a particular instance e.g. response rates, status, weighting, versions. This provides the other link between concepts and physical data. It is worth noting that the processes involved in preparing quality metadata are considered operational metadata. Physical Metadata includes the physical, unique characteristics of the data which cannot be separated eg server locations, data base.

Metadata system(s)

The metadata infrastructure will be implemented within the 10 components covering the whole BmTS environment. The Metadata Broad Logical Design 2004 defined nine components of key metadata infrastructure which were needed to create the physical metadata environment. Note: In defining the relationships, the terms period dependent and period independent are used. Period dependent refers to metadata which is linked to a specific activity/collection (includes quality metadata). Period independent metadata has meaning while held separate from data and can be applied to several collections (includes operational and conceptual metadata). Search and discovery, Metadata and data access/ registration These components reflect the ways the user interacts with the metadata. Ideally, searching, registration and access should be possible directly with each component, or through a central portal. Metadata Storage Data Definition - The data definition component is the only infrastructure linked directly to the data. This is the primary store which defines and adds meaning to the data. This is a period dependent store which compiles the relevant metadata for a single collection. All other storage components link to the data via the data definition store. Passive Metadata Store - The passive metadata store is the next level removed from the data. It contains period dependent and period independent metadata about a collection of data (this includes survey collections and administrative data collections). Question Library - The question library should be period independent. It contains questions and variables which have been defined independent of the data. The question library and classification management store are linked through the classifications used in questions. Classification Management - The classification management store is another period independent store which manages the classifications used to define the data. It includes metadata linking classifications to each other (concordances) to allow more options when analysing and transforming the data. Business Logic - The final period independent metadata store is the business logic component. While business logic is not linked directly to the data, it is applied to change the data through it's various states. This contains details of the rules and processes that may be applied to the data. Business logic may also be referenced in the design and methodology content of the passive metadata store. The business logic component sits partly outside the storage environment due to the need for software to access the rules and processes (e.g. rules engines). Other Storage Frame and Reference Stores - While the Frame is not part of the metadata environment, it may contain information which is used to define the data. Hence there is a link between this component and the data definition component. Document Management - Document One is a tool for the management of documentation. As several reports and documents will be created during the business process, they are considered part of the wider metadata environment. Standards Framework - The standards framework represents a tool for the central storage of standards used in the generic business process. This includes a definition of processes and methodologies at high levels. It will also include statistical standards which define how classifications are applied. Similar to Document One, this should be considered part of the wider metadata environment. Logical View of Metadata Infrastructure and Relationships In 2007, further analysis was completed using the gBPM and the MetaNet Reference Model to build a more detailed understanding of the logical metadata stores and the key relationships with data. The reference metadata layer contains stores of metadata with similar characteristics which allow it to be managed in a consistent way. For instance, classifications are used at various stages throughout the statistical business process to define various types of data. By storing and managing classifications in a separate 'classification management' store, we are able to analyse usage and identify opportunities for further standardisation. In order to develop a fully integrated metadata environment, each metadata object will need to be linked with objects within the store, or in other reference stores. For instance, a description of business process will be stored in the 'standards and processes' component, this will also need to link to the 'operational metadata' component containing workflows and transformations which operationalise the process. The workflows and transformations may also reference 'business rules' which utilise concepts from the 'variable library'. Linking metadata objects allows the user to consider the full usage of each object and will enhance reuse and standardisation.The 'Structural Metadata Layer' is the mechanism for linking the reference metadata with the actual data. Each data item (or fact) within the data environment should contain a profile within the data definition store which identifies all the metadata relevant to that fact. Ideally this will consist of a map identifying the location of the relevant metadata in the reference stores. However, until all the reference stores exist, this component will contain snapshots of the relevant metadata. When translating from the logical view, to the physical design it is anticipated that the components will take a different shape to that referenced in the model above. For instance, 2008 will see the investigation of a single system to manage classifications, questions and variables.

Costs and Benefits

The high level benefits of undertaking the metadata programme were outline in the introduction. Additional benefits to consider are as follows: maximising the value of metadata through reuse. reducing the unnecessary duplication of metadata. providing a more comparable, central source of metadata to allow for improvements through standardisation. It is recognised that the development of a full metadata solution will require a large investment in the infrastructure of the organisation. There are also various levels of investment that could be applied to deliver the most practical solution (eg if the needs are a lower priority for one type of metadata, then the solution should be less complex). At this stage, the cost of delivery has not been calculated, however it is known the the following considerations will need to be addressed: Large amounts of metadata already exist in various systems will need to be transferred into new systems where practical. The principle of reuse, may require more effort in creating and storing metadata at early stages of survey development in order to reduce effort at later stages (essentially shifting the effort, rather than reducing it). There will need to be careful management to ensure the duplication of metadata is reduced (ie if it's already entered in the reference store, it should be selected and configured for current requirements rather than duplicated). Detailed versioning will be required to ensure that the metadata is relevent to the instance is relates to. This may increase the storage requirements.

Implementation strategy

The work of the metadata project during 2007 focused on identifying the high level needs of an integrated environment and the adaptation of a metadata conceptual model to understand the relationships and interactions with metadata. With the bigger picture in place, the intention is to focus on developing solutions for smaller components of the wider environment. This approach allows us to focus delivery in the areas which will provide the most gain, while still progressing along the path to delivering the fully integrated solution. It also allows us to assess the strategy at each stage to determine the most practical apporach and to minimise the risk of the delivering over-complicated solutions.

IT Architecture

The introduction of Service Oriented Architecture (SOA) into Statistics NZ was the culmination of researching industry trends and evaluating those trends against the new technical challenges that were arising in response to the BmTS. The BmTS has three core deliverables: A standard and generic end-to-end process or processes to collect, process, analyse and disseminate data (Value Chain). An approach to data management which is disciplined and consistent (Information Architecture). An agreed organisation-wide technical architecture as a framework for making system decisions. To support the first two deliverables and to ensure that the third deliverable is achieved Statistics NZ has adopted a Service Oriented Architecture (SOA) approach. A SOA resolves the behaviour of the organisation's IT assets into a set of common behaviours or services. Services can be business services and technical services. The SOA is a key enabler of BmTS exposing common services (business / statistical and technical) as an abstract, decoupled and consistent set of interfaces enabling the communication of as much of the process and data in Statistics NZ's core business. In addition, there are a number of benefits related to the incorporation of third party software; this includes off-the-shelf applications and providing and using services to and from other statistical agencies. Key aspects of the Statistics NZ SOA are that the consumer of the service can find and bind to services at runtime and the SOA extends to the development, deployment and management of services. This architecture will enable the transparent exchange of metadata between different systems and tools. The current service layer is supporting some existing metadata components like process management (workflow), business rules (rules engine) and the integration with main systems (CRM-based respondent management and call centre), tools (SAS, ETL, Blaise) and databases (SQL Server).

Metadata Management Tools

Macro Rendering Error

Standards and formats

When defining a concept based model for to be used as the overarching metadata framework, four concept based models were reviewed, specifically DDI, SDMX, MetaNet Reference Model v2.0 (MetaNet), and Neuchatel Terminology Model. In December 2006 a working group determined that no one model met all the needs of Statistics NZ. A blended model was recommended taking the best components of two models to create a single model - MetaNet and SDMX. Further analysis by the working group provided clarity for each model evaluated, including details of risks, impacts and gaps. They produced a second recommendation, this time a primary model - MetaNet with a secondary layer to treat any gaps - SDMX. Selection was based on simplicity, adaptability and integration and ability to support the business process. During this stage the MetaNet model was analysed and mapped against internal metadata stores to assess the usability within Statistics New Zealand. As a result of this process a series of recommendations were presented for adapting the model to better meet our needs. Work has been progressing to adapt this model with a revised version due for completion early in 2008.

Version control and revisions

Currently the broad principles for versioning have been developed, however the application will need to be part of the individual developments. Returning to the Logical View of our data and metadata stores in section 2, it is intended that versionning will be maintained within the 'reference metadata layer'. There will also need to be versions of the data definitions within the 'structural metadata layer', however these will essentially be linking structures which identify the relevant versions of reference metadata.

Outsourcing versus in-house development

The strategy for developing components of the metadata environment is still being developed, however in general a principle of enhance first, then buy before build is applied. A summary of the state of current developments is as follows: Currently the solution for our data definition is being developed in house as part of the wider development of the Input Data Environment (IDE). The decision has also been made that many of the business logic components will be dependent on the tools used to run transformations and workflows. Planning is underway to investigate the feasibility of enhancing our current classification management tool (CARS) to incorporate functionality for question and variable management. Longer term we also aim to redevelop our current survey metadata tool (SIM), however the form of this development is as yet undecided. Additional tools have also been adopted for managing other metadata components, eg documentation and report management is being enhanced through the implementation of Document ONE on top of our current Lotus Notes functionality.

Sharing software components of tools

 .

Overview of roles and responsibilities

Overview of metadata audiences and use of metadata To ensure that metadata is relevant and useful a high level analysis of the audiences of the metadata environment has been completed. The audiences have also been identified using several broad user groups - Public, Professional, Technical, System.

Metadata management team

While currently undefined, it is recognised that additional support will be required to maintain, monitor and improve the metadata usage in the organisation. Teams within our information management group and standards, solutions and capability group are currently involved in developing our metadata processes and infrastructure. These groups will also be made responsible for the ongoing training, knowledge management and support of the metadata solution. Over 2008, the metadata team completed the adaptation of MetaNet to produce the Statistics New Zealand Metadata Model. This is being consulted on internally and will be finalised in early 2009. This is being seen as the first step in building a more consistent knowledge of metadata across the organisation. Further strategies will be developed as the metadata infrastructure is completed and implemented.

Training and knowledge management

 .

Partnerships and cooperation

 .

Other issues

 .

Lessons learned

Macro Rendering Error