Anker
_Ch2 _Ch2

Online questionnaire

Our online questionnaire was designed and administered using SurveyMonkey. All members of the HLG-MOS Machine Learning project were encouraged to participate and also to forward the survey to colleagues with relevant expertise. Between September 15th and October 15th of 2020, 28 responses were collected and form the basis of this report. The questionnaire remains available online however at https://www.surveymonkey.com/r/6G5VVFH and additional responses are welcome and may be incorporated into future products. 

Our 28 respondents include representatives of national statistical organizations covering 14 countries and regions, all in either North America or Europe. Most report having a role of “Statistician / Data Scientist”, followed by “Analyst / Subject Matter Expert” and “Manager / Policy Maker.” Only one respondent reported a role of “Software Engineer / Information Technology Specialist”. Most respondents also report belonging to large national statistical organizations (54%) defined as those having more than 2000 employees, followed by 32% of respondents reporting the next largest grouping, between 500 and 2000 employees. See the appendix for details.

Image Added

What are the biggest challenges facing statistical agencies in ML? Our questionnaire divides this into two sub questions, one asking about “organizational issues” and the other about “technical issues.”

Image Added

Among organizational issues, “coordination between internal stakeholders” ranked among the largest challenges with 16/27 (59%) reporting this moderately limits, severely limits, or prevents use.

 Image Added

Among technical issues, “availability of staff with appropriate machine learning algorithm skills” was the most limiting factor with 10/28 respondents (36%) reporting that it severely limits use. The average score of 1.8 makes this the most problematic issue identified in our survey. 

Our survey ends with a question about which activities have been most useful. Collaboration with other statistical organizations ranks as the most useful, with 14/28 respondents indicating it is “very useful”, followed closely by external training programs  with 10/28 indicating “very useful”.

Anker
_Ch3 _Ch3

Long form investigation

While the short form questionnaire gives us a high level overview of the challenges and potential solutions, it cannot tell us much about the details. To compliment this information we asked project participants to describe how they were addressing six key questions. We received detailed responses from 4 organizations, the UK Office of National Statistics (ONS), the Australian Bureau of Statistics (ABS), Statistics Flanders, and the U.S. Bureau of Labor Statistics (BLS), and related comments from many others. The questions, and a high level overview of the responses are below.

Where should machine learning fit in a statistical organization?

Participants indicated 4 broad approaches:

• Machine learning as a branch of methodology - In Statistics Flanders, machine learning is an experimental branch of methodology. Machine learning techniques are clearly related to traditional statistical techniques so methodology is a reasonable starting point, especially for organizations still determining whether they want to use ML. Several other NSO’s reported similar models at least early in their investigation. It is of course not a complete solution to production deployment but not all projects are ready for that yet.
• Machine learning as a multidisciplinary collaboration - The Australian Bureau of Statistics approach emphasizes the importance of multidisciplinary collaboration. In this model different pieces of the organization play lead roles on different aspects of the project. Methodology or research often develop initial prototypes which are then handed off or co-owned by information technology and subject matter experts. An advantage is that many different pieces of the organization are involved. A frequent challenge is coordination. For example, the tools preferred by researchers and methodologists, such as R and Python, are often quite different from those preferred by software engineers. Another challenge can be in getting alignment with the needs and interests of subject matter experts, who are often the most direct users of the technology and often must also assume key roles in creating training and evaluation data. 
• Machine learning as decentralized process - Although the Bureau of Labor Statistics traditionally follows the multidisciplinary approach, in the case of machine learning it has instead adopted a largely decentralized approach in which the program offices assume primary ownership of machine learning systems and consults with methodology to verify the integrity of the system, IT to integrate the system with existing infrastructure, and field staff to facilitate data collection and processing activities as needed. This reduces the difficulty of aligning different divisions, but at the cost of the program office assuming a more active role in methodology, systems development and maintenance.
• Centers of excellence - For the Office of National Statistics, a key aspect of machine learning strategy is the Data Science Campus, a separate division made up of experts in data science and machine learning which provides advice on machine learning projects not just to ONS, but to many parts of the UK government and even other countries. This allows the sharing of often limited machine learning expertise across many areas. A number of NSO’s have recently developed their own versions of this approach, sometimes called the “Hub and Spoke” model, including INEGE (Mexico), Stats Canada, Statistics Finland, and Statistics Sweden. In some versions of this model the goal is to ultimately transfer most of the limited machine learning expertise from the hub (the center of excellence) to the spokes (the specific business areas), eventually resulting in many parts of the organization being familiar with machine learning.

What should the machine learning pipeline look like in regards to organizational structure? Where should projects start, who should control what aspects when?

Interestingly, the responses to this question resulted in two seemingly opposite ideas. One set emphasized the importance of starting with a business need, moving to R&D, producing a prototype and then bringing in other areas like IT. The second response however emphasizes the importance of building ML experience first, which in turn allows one to identify suitable business problems which might be solved by machine learning. 

It is clear, in hindsight, that both are needed. An organization cannot determine whether machine learning is suitable if it knows nothing about machine learning, but it is also clear that the ultimate goal is to serve business needs.

What machine learning skills are needed and where are they needed in the organization?

On this question, there was general agreement among the responses. In organizations that distribute machine learning responsibilities across many divisions, machine learning requires new skills in many areas. Specifically:

• Everyone must understand the basics, such as the key ideas and common terminology. This allows effective communication between the varying parties.
• Research and methodology often must become familiar with new algorithms and new tools, like R and Python, which are popular for machine learning.
• Information technology must learn how to integrate these tools and processes in existing systems. In some cases they must also support specific hardware needs, such as powerful Graphical Processing Units for training deep neural networks.
• Subject matter must understand their role in supporting, using, and maintaining these systems as they often play a lead role in creating the training and evaluation data.
• Management must understand the needs of ML teams, including the need for careful alignment and coordination across these activities.

An alternate approach is to centralize all or most of these functions in one or several “data science experts”, who assume ownership over many of these aspects simultaneously. This limits the amount of coordination and communication that must occur, but requires individuals with a broad range of skills.

How can organizations efficiently acquire the ML skills they need?

Responses identified 4 strategies

1. Acquire and train - In this strategy, an outside expert is hired permanently or temporarily and used to train additional experts. Statistics Flanders, ONS, and ABS all report using some variant of this approach.
2. External training - In the case of machine learning, many high quality trainings are available often for free, and many NSO’s report using these extensively. There are also increasingly suitable trainings available through academia.
3. Communities of practice - A community of practice is a group of individuals with a shared interest and willingness to share what they learn with their members. The HLG-MOS ML project is essentially at least partly a community of practice, but many organizations have also set up others internally. The BLS, for example, has a popular data science user’s group in which members share information about data science projects.
4. Research projects - At some point learning requires doing. Research projects play an important role in supporting skill acquisition.

How should organizations demonstrate and communicate the value-added of ML techniques?

One of the recurring challenges of working on projects involving many parties is the need to convince others to adopt or support new techniques. This is supported both by numerous anecdotes among participants in the ML group, and by questionnaire responses indicating coordination and resistance issues from internal stakeholders. Responses identified 3 potential strategies.

1. Clearly demonstrate value added - When replacing or augmenting an existing process, it is often easy to demonstrate speed and cost improvements with machine learning but quality is also an important consideration and frequently much harder to evaluate. In many cases, the most readily available evaluation data for a machine learning project is just a subset of the data currently produced by the existing process. In this case, standard quality metrics (accuracy, mean squared error, etc.) only measure how closely the machine learning approach matches the existing process, not the more relevant question of whether one is better or worse. One solution is to construct the evaluation data in such a way that it is independent of all processes being evaluated. This can be accomplished, for example, by asking a trusted panel of experts to reprocess the evaluation data without knowledge of how either the machine learning existing processes would handle it. The resulting “gold standard” can then be used to evaluate and directly compare both the existing process and the machine learning process. In the case of the BLS injury and illness coder, this comparison played a critical role in justifying the use of the machine learning option. 
2. Use ML as a decision-support, at least initially - Replacing an existing process with something new is also a potentially dangerous task. There is always the potential for some unanticipated issue to occur, and this is especially concerning to stakeholders who might have little familiarity with machine learning. One solution is to instead use machine learning as an assistive tool, at least initially. If we are automating an occupation classification task which was previously done manually, for example, we might start by only using machine learning to provide suggestions to a human coder. This allows stakeholders to get hands-on experience working with the machine learning model in a  low-risk setting.
3. Use ML for things that aren’t otherwise possible - Another way to introduce machine learning into a statistical organization is to use it for new projects, whether no other option is feasible. Analysis of satellite imagery is a good example, it simply is not possible to do this at scale and high frequency without extraordinary amounts of labor. Here, machine learning can make what would otherwise be an impossible task possible.

How should statistical organizations identify the right problems for machine learning?

Our investigation uncovered 3 strategies.

1. Learn from others - learning from the successes and failures of others working on machine learning is a relatively cheap and easy way to identify promising areas, and avoid less promising ones. The HLG-MOS ML project facilitates this work significantly for NSO’s.
2. Look for tasks that meet machine learning friendly criteria - Machine learning tends to be well suited for tasks that have certain characteristics, such as the following:
1. Stable over time (relatively same task year to year), this limits the amount of retraining that needs to occur, which can be costly and difficult to do correctly
2. Lots of training data showing all relevant input to task and desired outcome. Ultimately machine learning requires training data and that data must adequately describe the problem. The more you have the better it tends to do.
3. Start with lightweight research projects, proof of concepts - this provides a lower cost and lower risk way to explore initial ideas.

Anker
_Ch4 _Ch4

Appendix: SurveyMonkey results