Discussion Notes: DDI – SDMX Meeting, December 9, 2010, Utrecht, at the European DDI Users Conference
December 13, 2010
This meeting was attended by:
Nicole Buysse - Eurostat/SDMX
Krassimir Ivanov - Eurostat/SDMX
Peter Boško - Infostat
Dušan Praženka - Infostat
Joachim Wackerow - DDI Alliance
Steven Vale - UN/ECE (METIS)
Arofan Gregory - representing Australian Bureau of Statistics
This was an informal meeting between representatives of the SDMX Initiative and the DDI Alliance, along with some other interested groups. The purpose of the meeting was to determine if any collaboration among these different groups might result in benefits to those organizations wishing to use both SDMX and DDI together, and, if a collaboration was deemed useful, to identify the goals of such work and the means of reaching those goals. The meeting resulted from an inquiry from the chair of the SDMX Secretariat to the vice-chair of the DDI Technical Implementation Committee.
This document describes some of the themes discussed, and the outcome of this discussion. It is notable that although the meeting took place in the margins of a DDI-focused event, there had been a 3-hour workshop introducing SDMX, taught on December 8 as part of the program, and a plenary presentation by Steven Vale about the possibility and investigation into the use of both SDMX and DDI as a way of supporting statistical production, as represented by the Generic Statistical Business Process Model (GSBPM).
The outcome of the meeting was generally successful – several possible goals were identified, and some of the challenges as well. It was felt that continuing the discussion in future is useful. In general terms, all participants felt that both standards bodies would be supportive of some form of collaborative work.
As background, two themes were discussed. First, a summary of events concerning both SDMX and DDI was presented; second, the various groups and organizations which might be interested in the alignment and use of both standards were identified.
There have been several points of intersect ion between the standards in the recent past. At the METIS meetings, there have been presentations about using the two standards together. Over the past two years, Eurostat has provided a course introducing both SDMX and DDI as part of its ESTP training program (with the next such course scheduled for early 2011). At the DDI-focused workshops given each autumn at Schloss Dagstuhl, in Germany, there have been several attendees from national statistical agencies. This has included an early mapping effort between SDMX and DDI lead by Bryan Fitzpatrick. Several statistical agencies – the “Statistical Network” have been exploring how SDMX and DDI can be used together to provide support for implementation of the GSBPM, which work may lead into the creation of a “Generic Statistical Information Model” as a companion piece. At the OECD, a group of interested statistical agencies met this past summer to discuss microdata access – this group is now becoming an OECD Expert Committee, and includes users of both SDMX and DDI (the latter being often used to manage secure microdata access, as was discussed at the meeting.) In summary, there has been growing interest in the use of SDMX and DDI together.
A number of different potential stakeholders in collaborations between the standards are evident. The DDI Alliance and the SDMX Initiative would be involved, and at this point both have expressed interest in exploring the possibility. Within the European Statistical System, there are several potentially interested groups – ESSNet’s CORA and CORE groups might be considered interested, but it is likely that the ESSNet SDMX group would be the most directly involved. Within the Statistical Network, the Australian Bureau of Statistics (ABS) – as leader of the initial work on a GSIM model within OCMIMF work – is a stakeholder. It is also notable that ABS and some other national statistical agencies are prototyping the use of these standards together in production systems. Clearly, METIS as a group is interested in developments here, and is the likely place where work on a combined information model such as GSIM might be published.
Possible Use Cases
Following this background, several possible use cases were discussed, as forming the basis on which a discussion of collaboration might be considered.
The list suggested at the meeting was not discussed in any depth, but it was agreed that it might represent initial cases for further discussion if collaborative work was agreed:
- Support for end-to-end statistical production process
- Management and dissemination of secure microdata, with tabulated results
- Survey data collection/documentation, to be reported as aggregates
- Exchange of metadata for surveys and data between organizations
- “Drill down” from aggregates to underlying microdata metadata
It was realized that identification of real use cases was dependent on identifying the goals of any collaboration, and that there may well be other use cases not mentioned at this meeting.
Objectives and Challenges
Three potential objectives for a collaboration were identified:
- To avoid duplication of effort by the two standards, and thus avoid confusion about which standards should be used for specific types of applications
- To provide reassurance to the user communities of both DDI and SDMX that the end-to-end statistical process can be managed, and that the standards bodies are both considering the needs of users in this area
- To provide specific technical guidance about the use cases and implementation of the standards for specific purposes
It was mentioned that these objectives are not only technical ones – users should be considered at both the business level and the technical level.
Some of the potential challenges were also mentioned:
- The DDI and SDMX communities use different (and sometimes overlapping) terminology. A glossary of some sort would be needed to provide clarity for any discussions. Such resources as the SDMX Metadata Common Vocabulary and the OECD Glossary of Terms might form an example for this type of glossary.
- An agreed view of the data production process would be required. The obvious candidate here in the GSBPM, since it already has a high-level mapping to the DDI Lifecycle model, and has been widely adopted within statistical agencies.
- The organizational basis of this work would need to be agreed, and is as yet unclear. METIS would seem to offer one good alternative, but this will need clarification moving forward.
In general terms, it was understood by participants at the meeting that some effort would be required to establish a common basis for any work between the two standards communities.
Next Steps and Possibilities for Further Discussion
It was agreed that notes from the meeting would be drafted and circulated, and then used as the basis for gathering information and opinions from the various interested parties. Possibilities for further discussion were also identified: one possibility would be the ESSNet SDMX meeting in Lisbon in March 2011, with the idea being that further informal discussion could take place in the margins of that event. Other possibilities included the SDMX Global conference in Washington DC, in May 2011, and the upcoming METIS meeting on the GSBPM in Geneva in October 2011.
The notes from the meeting will be forwarded to the organizations involved in this discussion, including the director of the DDI Alliance and the chair of the SDMX Secretariat. If agreement can be reached that the objectives described here are worthwhile, then further steps will be identified, based on input from the various stakeholders.
Some ideas for concrete action could include a statement from the standards bodies that they have initiated this collaboration, should the decision be made to work together in future. Specific outputs from this work would need to be identified.