PILOT REVIEW(S)

The pilot review(s) can be divided into three distinct phases:

• The COLLECT phase involves retrieving the feedback from pilot participants and adding the feedback to the central issue log.  
• The PROCESS phase involves consolidating all of the feedback in the issue log, removing duplicates and following up any issues that lack clarity.  
• The ANALYSIS phase is where the technical group steps through the issues log and formulates actions based on the issues.  For example, an issue may be that a concept does not fully describe the category, so the action may be to add some more codes to a Code List.

The pilot phase(s) is (are) needed to test the project deliverables "in the field" so that when the SDMX artefacts are released to production, many or all issues have been found and added to the issues log so that they may be rectified by the project's technical working group.  There is also the added bonus that, when the artefacts are released to production, the pilot phase participants will have already had experience in using the SDMX artefacts and implementing the data flows.  

At certain project milestones in the checklist, project deliverables are sent to a group of pilot phase participants that are external to the project members.  For example, for a global DSD, the pilot participants would typically be a cross-section of member countries from the ownership group's agencies constituencies.  For a regional DSD, the pilot participants would typically be some agencies that report data to the ownership group.  In both cases, it would be very useful to include some external agencies that consume the data.

There are two typical, separate pilot phases in the checklist:

• Content review: The aim of the content review is that the projects's DESIGN stage material is understandable, well-designed, and that it covers the scope of the project.  The material from the DESIGN stage is sent to the pilot participants for review.  The main deliverable for review is the "DSD matrix" as this contains the Concept Scheme, Code Lists, and DSDs. Also, if they exist, the "supporting artefacts" e.g. Provision Agreements should be included in the pilot.  The pilot participants review the material and send the feedback by adding to the issue log that was created in the PLAN & ORGANISE stage.
  
  

• Technical review: The aim of the technical review is to test that the project's artefacts can be implemented at system level as efficiently as possible. The updated deliverables from the DESIGN stage, and the BUILD stage deliverables, and test material such as example data messages and xml-schemas are sent to the pilot participants for review.  The SDMX artefacts should be available in a registry, and the implementation & usage guidelines are sent for documentation.  

The technical review is more involved than the content review, although pilot participants will by now have knowledge of the design material:

1. 1. The pilot participants review the material;
   2. implement their SDMX system in order to process the pilot material;
   3. perform the system testing with the pilot material, in particular testing that their data messages validate against the DSDs;
   4. send the feedback and data messages by adding to the issue log that was created in the PLAN & ORGANISE stage.

If two pilot phases are not possible then the technical review should be mandatory, otherwise there will be no external testing of the project deliverables before release, which is not project management best practice.