SDMX Reference Architecture
A more complex case at Eurostat is that of the SDMX Reference Architecture for NSIs, whose main objective is to provide a description/specification of a generalized architecture to be used partially or as whole by MSs interesting in starting SDMX projects and willing to participate in Eurostat's pull data transmission (an example could be the Census Hub).
In this framework several open source tools are made availabe to NSIs if they are willing to use them:
- SDMX Web Service Provider: This module is responsible for exposing the Dataset using a Web Service interface that provides SDMX-ML messages
- SDMX Query Parser: This module is responsible for getting the request from the "Web Service Provider" and populate the internal data model i.e. sdmx data model
- SDMX Data Retriever: This module is responsible for querying the dissemination database and getting the respective recordset
- SDMX Data Generator: This module is responsible, upon receiving the recordset and the respective mappings from the "Data Retriever", for generating an SDMX-ML Dataset message
- SDMX Mapping Assistant: This module is responsible for creating the mappings between an SDMX Data Structure Definition (DSD) and a DB schema (dissemination database) or a set of dissemination data files. It maps the DB schema from the database to the SDMX DSD ("SDMX Structure File" artefact). The Mapping Assistant is meant to facilitate the mapping between the structural metadata provided by an SDMX-ML Data Structure Definition (DSD) and those that reside in a Dissemination database of an NSI dissemination environment.
The above modules exists in both .NET and JAVA. Each component is released as open-source software (OSS), thus it can be either re-used as is (as a packaged dll or a packaged jar) or by using its source code (in case modifications are needed). Of course, the developers should have in mind to respect the EUPL license governing the source code.
It is a unified modern architecture which materialises Eurostat's vision for EU-wide data sharing which has been deployed based on SDMX guidelines and OSS technologies. The guiding design rational behind the platform's architecture is to facilitate interoperability and to allow for extensibility while preserving platform independence.
The development of the tools was funded by the IDABC framework.
IDABC: Decision 2004/387/EC of the European Parliament and of the Council of 21 April 2004 on interoperable delivery of pan-European eGovernment services to public administrations, businesses and citizens (IDABC).
X-DIS (XML for Data Interoperability in Statistics) is a Project of Common Interest in the framework of IDABC.
The SPC gave a unanimous positive opinion on the X-DIS project at its meeting in February 2005. The first revision of the IDABC multi-annual work programme, including the work programme entry for X-DIS was adopted as a Commission Decision on 8 November 2005.
Under IDABC rules, the SPC is designated as the "sectoral committee" for X-DIS and therefore the SPC is asked to give its opinion on the project at key milestones; this means that the SPC, supported by the STNE working group and ITDG, will be able to influence the content and orientation of the project throughout its lifetime.
The first such milestone, the Global Implementation Plan (GIP) was approved at the SPC of November 2005. After that, work on X-DIS started in late December 2005 with the first development contracts. From the first ideas for this project dating back to the year 2004, Member States have been consulted at the meetings of the relevant expert groups, STNE and ITDG; the comments received lead to today's focus of X-DIS.
Up to now Eurostat has implemented the tools but it would be good if a different approach, involving actively the wide SDMX community in a more open initiative, could be started but we don't have experience of such approach. So far this project has not been connected with the SDMX ESSnet but the approach outlined above will be followed by ESSnets on this topic. In addition, the tools already available allow for a quicker and cheaper introduction of advanced data sharing environment. The benefits for administrations are principally the general availability of a widely accepted standard for the exchange of data and metadata together with a vast set of tools. This standard enables the streamlining of statistical processes, the introduction of modern service-oriented IT architectures and the sharing of tools between different statistical administrations.
One example of future development in cooperation with MS is Essnet for SDMX Working Package 7: PC-Axis SDMX Integration.
Antonio Consoli - Eurostat