5.1 IT Architecture
CROMETA METADATA SERVER
CROMETA Metadata Server is a comprehensive solution that covers all aspects of a modern metadata repository. The server is not designed for methodologists, statisticians nor systems developers solely, but provides each key unit of a statistical office with appropriate functionality through its various modules. Using the server, the methodologists can rest assured that a well-established and constantly developing theoretical model on statistical metadata has been brought into use. At the same time, the owners and producers of statistical surveys will enjoy a user-friendly Windows-like interface for maintaining and browsing their metadata. As far as concerns the system developers, an XML-based web service ensures a straightforward, practically code-free integration with existing as well as new metadata consuming applications using the centrally stored metadata. Even the system administrators are not needed to intervene during the implementation. The whole server uses the latest and most established tools and techniques around.
The comprehensive CROMETA Metadata Server
CROMETA Metadata Notion
Basically, CROMETA Metadata Notion is the model on which the complete metadata server resides. This generic model inherits its theoretical base from the MetaNet Reference ModelTM on Metadata, which was originally developed within EUROSTAT in order to reach a common model on metadata in general, and a widespread terminology in particular. All terms used in order to describe various metadata concepts within CROMETA Metadata Notion are directly derived from the EUROSTAT model. Given the fact that the terminology was elaborated by experienced statisticians representing major national offices, it is fairly easy for any statistician to rapidly grasp and understand the terms as well as their conceptual meaning and relations within the model.
CROMETA Metadata Notion is purely conceptual, hence could be implemented on any technical platform.
CROMETA Metadata Storage Central
The conceptual metadata model covered through CROMETA Metadata Notion is implemented in practice though the CROMETA Metadata Storage Central. This is the basic storage point of metadata for any implementation of the metadata server.
A huge advantage of the storage engine is that it is not limited to run on a specific platform, furthermore, it is completely open to use for any purpose. In practice this means that the system developers may use the underlying data source directly in order to connect and integrate it with other solutions. Avoiding encapsulation into some hidden storage format enables the office to use the DBMS already purchased or to choose one in line with the budget. Moreover, built-in functionality of the DBMS chosen could be used for replicating and synchronizing data all over the system. Obviously, this approach also provides the office with possibilities to further customize the metadata model, should there be a need for it.
As mentioned, the CROMETA Metadata Storage Central could be customized for any database platform; however, the current implementation is developed for the Microsoft SQL Server platform. At present, the system runs on the Microsoft SQL Server 2005 DBMS.
CROMETA Metadata Business
CROMETA Metadata Business is a business tier that implements the conceptual model from CROMETA Metadata Notion as well as storage-specifics from CROMETA Storage Central through an object-oriented approach. Included in the business model is a fully documented UML class diagram, clearly displaying the metadata classes, objects, methods etc. available through the metadata server.
Basically, CROMETA Metadata Business puts all features such as version handling, multi-lingual support, etc. into play by applying the appropriate business rules. For the ordinary user, the use and work of this module is obviously completely invisible. However, when using the maintenance tool - CROMETA Metadata Manager for browsing or editing metadata, or when using metadata for tabulation from the website, CROMETA Metadata Business takes care of all the background processing enabling the various features to be put into play.
For the developer engaged with integrating various metadata consumers with the central metadata server, the CROMETA Metadata Business is essential. By using the CROMETA Metadata Business together with the CROMETA Metadata Consumer Services, the task of creating a real Integrated Statistical Information System (ISIS) has been considerably simplified. Connecting surrounding applications with the metadata server is basically just one click away. For additional information regarding integration and how to put metadata consumers in connection with the central repository, see CROMETA Metadata Consumer Services.
CROMETA Metadata Consumer Services
The CROMETA Metadata Consumer Services offers an uncomplicated and swift way for integrating existing as well as future metadata consumers with the metadata server. By exposing all metadata objects with belonging properties through an XML-based web service, the whole metadata repository is made available. Virtually any application requiring metadata can reach it in a practically code-free manner. Using the CROMETA Metadata Consumer Services, the system developers can connect to the metadata server and use its metadata just by referencing the web service and use the predefined methods for retrieving and editing metadata.
Consumer services that are considered important to be implemented soon:
- Exposing classifications through the official web site;
- Displaying the organization and areas of responsibility, contacts, etc;
- Publishing statistics using metadata for browsing purposes;
- Delivering publications including data and metadata;
All of these are features that could be implemented in a straightforward manner using the CROMETA Metadata Consumer Services.
CROMETA Metadata Manager
Most people will access the CROMETA Metadata Server through the CROMETA Metadata Manager. This is the tool which provides a memorable, graphical user-friendly interface for adding, browsing, editing and generally maintaining metadata. Basically everything that could be done to metadata could be achieved here; given you have the appropriate privileges to the metadata you are aiming for.
The CROMETA Metadata Manager tool runs on the following platforms: Microsoft Windows 2000, Microsoft Windows XP, Microsoft Windows 2003 Server, and Microsoft Windows Vista.
Basically, CROMETA Metadata Manager features a Windows-like interface. Any user familiar with using Microsoft Windows should be able to acquaint with the tool in no time. For a more detailed description of this tool, see Metadata Management Tools (4.2)
Technical Platform for CROMETA Metadata Server
As regards technical platform and development environment, the conceptual data model for CROMETA has been developed using Sybase Power Designer 9.5. The same software has been used for creating the physical data model that has been implemented on the Microsoft SQL Server 2000 RDBMS. Consequently, all database development has been carried out on the Microsoft SQL Server platform, while the CROMETA maintenance tool has been developed in Microsoft VB.NET. All object and use case modeling have been performed in Microsoft Visio. For managing the development in a shared environment, Microsoft Visual Source Safe has been used, while for project management purposes, Microsoft Project and SharePoint Portal Server have been applied. Since the development is still ongoing, the solution was upgraded and tested on Microsoft SQL Server 2005.
5.2 Metadata Management Tools
As part of the comprehensive CROMETA solutions, a maintenance tool has been developed, providing a user-friendly interface to the central metadata repository. It is expected for the tool to be developed further and extended with additional features that may be missing presently.
CROMETA Metadata Manager provides numerous functions, out of which the most obvious are to add new, edit and delete metadata from the central repository. It is difficult to describe the tool in words only; most probably the only way to get acquainted with it for real is to try it out in practice. However, some of the outstanding features of the tool are listed in this chapter.
Easy editing through graphical interface with multi-language support
CROMETA Metadata Manager supports an indefinite number of languages, meaning that the languages desired for entering and maintaining metadata could be defined through the interface. All textual properties will then be open for browsing as well as editing in the languages defined. In practice this means that the office may decide to enter metadata in five languages, for example Croatian, English, Russian, French and Spanish. Obviously, all statistics may then be published dynamically in any of these languages. Furthermore, also the interface itself supports several languages; hence it could be customized for any language.
All metadata objects may exist in numerous versions. Each version must be of one and only one state, however only one version of each object could be current/authorized. Versions will be described in detail in chapter 4.4.
All metadata, despite type or usage, could be added, edited or deleted using the exact same methods in the maintenance tool. Reusing methods between different types of metadata secures a tool that is recognizable and easy to use.
Add new version based on an existing version of a metadata object
Sometimes the differences between versions of a metadata object are minimal. For example, a questionnaire used within a survey one year, may only differ in terms of one or two new questions being added compared to the previous year.
To minimize the work within respect to this matter, CROMETA Metadata Manager supports "Add new based on", which means that all properties and connections of an existing version of an object is copied into a new object that could be further edited.
Taking the example with the questionnaire, it means that all questions, the questionnaire layout, validation rules, etc. are being copied from the previous questionnaire version. Obviously this saves a lot of time and effort. Basically the person responsible for the questionnaire only has to add the new questions in order to have it ready for processing.
Besides the version management, the history of metadata could be displayed through the update log. Every time a change is made to metadata, this is logged with modifying user and date. The modifying user may also optionally supply a description of the change made. Through the history management, metadata may be studied and followed over time, including all changes, explanations, involved users and its lifetime cycle.
In order to secure the metadata, a full-covering model for access and authorization is applied while using CROMETA Metadata Manager. First of all, the system supports a basic user group model where users can be divided into system administrators, object type experts, standard users etc. Secondly, all metadata created may be secured by the creator assigning privileges for how it may be used by other users. By default, all metadata kept within the system is read-only for all internal users.
In practice this means that when creating for example a new study, the creator may assign rights in any way he/she likes, using the tool, in order to ensure that only some persons are able to edit information, while others are only allowed to browse it, etc.
CROMETA Metadata Manager allows searching and filtering objects in various ways. Quick search based on titles is available as well as a full-scale advanced search including descriptions, keyword-based search functionality, etc.
General properties managed in a general way
When working with any metadata object, the general properties will always be available in the same way. In practice this means that footnotes, keywords, documents, etc. could be connected to all metadata objects in the same flexible way.
Subscription to metadata
In some occasions users may find it useful to get notifications when objects like for example classifications are updated. This is supported by CROMETA Metadata Manager through subscription-functionality. A user is allowed to sign-up to any object, specifying with which frequency he or she would like to be notified on changes. As soon as the metadata object, for example the NACE-classification is modified; the registered users are notified according to their independent frequencies.
Locking and unlocking of metadata
In order to ensure that the same metadata are not edited simultaneously by two different users, CROMETA Metadata Manager has built in functionality for exclusively locking and unlocking of metadata.
5.3 Standards and formats
The CROMETA model contains Reference ModelTM concepts extended and customized for CBS needs. As CBS had already implemented several other software packages containing metadata, e.g. Crometa#1, PC-Axis, BRIDGENA, those other metadata had to be mapped to the Reference ModelTM. Thorough analysis provided specifications how to integrate specifics of Crometa#1 and PC-Axis SQL Macrometa in the new model. The result was CROMETA - a comprehensive CBS metadata solution consisting of metadata objects and their relations. Naturally, the original model has been extended and revised in last two years, in order to better support the needs of statistical processing. The model is constantly evolving as new needs are discovered among the users of the metadata server, and as areas of statistical processing are further elaborated. The CROMETA model is considered to be 'implementation independent', i.e. it is a conceptual description of metadata as well as its internal relations, and hence it could be implemented on any technical platform. The complete model is available as UML and other formats as part of the CROMETA Metadata Server;
The SDMX standards will be implemented as soon as possible; presently it is still under consideration.
5.4 Version control and revisions
Each metadata object can have several versions that are valid in exclusive periods of time. There is only one version valid in any moment. This is determined by the Status of the metadata object as follows:
1. Under development
When a metadata object is created it gets this status by default. (This could of course be changed by the person creating it). This is the only status that allows the user to completely delete the metadata object from the system. A metadata object having this status could be viewed, edited and deleted physically.
When a metadata object has been completely created, it is normally being released. This means that it is finalized but not yet authorized, i.e. it has not been pointed out as the current version. This could also be described as ready for review. A metadata object having this status could be viewed, edited and deleted (not shown).
After being released a metadata object is usually reviewed by a reference group, or the creator alone. When authorizing it, it is automatically displayed as the current version. The metadata object previously displayed as current version is at this point automatically moved into status 4, Archived. A metadata object having this status could be viewed, edited and deleted (not shown).
When a metadata object is being replaced as current version by a successor, the state of it is automatically changed to Archived. This shows that the metadata object has been used as current version, but has now been succeeded by another version. A metadata object having this status could be viewed, edited and deleted (not shown).
Metadata object that has been used as the current version is automatically given the status Archived. However, this object could still be edited. At some occasions you may not want to allow editing of old versions, and if so, the archived metadata object is given the status Frozen.
Only metadata objects of the first status, Under development, could be physically deleted. However, metadata object of statuses two-five could be put into this status in order not to be shown in the metadata system. Metadata object of this status thereby appear to be deleted but could in fact be restored by the administrator.
Whenever some significant change on metadata must be applied, it is recommended to create a new version. If changes are small, e.g. spelling corrections, they can be applied without defining a new version. Such changes are kept in Update information, a general property of all metadata concepts in CROMETA.
Full access to all versions of metadatadata
5.5 Outsourcing versus in-house development
The detailed plan for CROMETA development was laid out in May 2004. Before that, in 2002 and 2003 plenty activities were performed such as on-site training courses, seminars, short-term visits, study visits, procurement, Crometa#1 development, etc. Participants in those activities were mainly IT people and some statisticians. It should be noted that CBS's IT people are more than well informed about statistical activities, methodologies and processes since they process or manage more than 100 statistical surveys on various platforms on a daily basis. The fact that the IT sector contained both knowledge about statistical methodologies and IT development made it natural that the central metadata repository should be developed in-house. That saved a lot of money since it was performed during regular working hours. On the other hand, the development time was significantly longer because there is no development team that could be appointed to this job exclusively.
There is also the crucial problem of 'brain drain' throughout CBS and particularly the IT sector. Some twenty people left IT sector in last three years, most of them were involved in CROMETA and ISIS development. This influenced a great deal the activities laid out in the plan since there was a constant shift of duties and responsibilities along with the constantly growing amount of regular production work.
Outsourcing would take a year or so for the software development itself, but it would take another year of work with statisticians to sort out, explain and document all the requirements. Given the complexity of CROMETA concepts it seems unlikely if not impossible to outsource the development and receive a comprehensive solution to be used and maintained by statisticians.