Assessment and monitoring

Regular measurement and assessment of internal communications enables a statistical organization to analyze a variety of trends:

• Demonstrate the value of communication interventions
• Determine how and to what extent communications activities and products are making a difference for the targeted audiences
• Increase the skill and expertise of the organization’s communication staff through continuous learning
• Inform decision-making about future communications initiatives and outreach

Perhaps most importantly, measurement, assessment and monitoring can provide both immediate and long-term feedback on communications initiatives, allowing course corrections to be made in time to be effective.

In conducting assessments, there are two types of data to consider:

Quantitative data: information that can be measured. These data reflect ‘how many’.

Qualitative data: information that can be observed and expressed as descriptions, perceptions or feelings. These data reflect the ‘what’ and ‘why’.

There are many tools available that can help an organization collect valuable employee information, such as attitudinal surveys, specific assessments, structured interviews, ability tests, and focus groups. These tools can improve understanding, trust, commitment, and communications among team members and facilitate more productive teams throughout the project.

The following table proposed by Paul Barton offers suggestions on the strengths/weaknesses and best use of the various tools.

Back to top

Case study 5: United States Federal Employee Viewpoint Survey

The U.S. Federal Employee Viewpoint Survey (FEVS)[1] is a web-based survey administered to most U.S. government employees that measures their perceptions of whether, and to what extent, conditions characteristic of successful organizations are present in their agencies. The FEVS provides results at high levels of aggregation (government-wide, departmental, and agency-level), but also at the lowest level at which respondent confidentiality can be preserved, often work groups of 25 or so employees. The results provide agency leaders insight into areas where improvements have been made as well as areas where improvements are needed.

A precursor of the FEVS was first administered in 2002, and then again in 2004, 2006 and 2008. The survey was revised in 2010 to focus more on actionable items and has been administered annually since then.

Though the FEVS undergoes small periodic revisions, in 2018 it consisted of 94 items that measured Federal employees’ perceptions of broad topic areas:

• Personal Work Experiences
• Work Unit
• Agency
• Supervisor
• Leadership
• Work/Life Programs
• Satisfaction

Of interest to internal communications and employee engagement, the FEVS reports on several special indexes that aggregate responses to individual items.

One of these is the Employee Engagement Index, where engagement is defined as “An employee’s sense of purpose that is evident in their display of dedication, persistence and effort in their work or overall attachment to their organization and its mission.” This index is further broken down into three component sub-indexes: Leaders Lead, Supervisors, and Intrinsic Work Experience.

The Leaders Lead sub-index is intended to reflect employees’ perceptions of the integrity of leadership, as well as leadership behaviors, such as communication and workforce motivation. It is an aggregation of responses to:

• In my organization, senior leaders generate high levels of motivation and commitment in the workforce
• My organization's senior leaders maintain high standards of honesty and integrity
• Managers communicate the goals of the organization
• Managers support collaboration across work units to accomplish work objectives
• Overall, how good a job do you feel is being done by the manager directly above your immediate supervisor?

The Supervisors sub-index is intended to reflect employees’ perceptions of the interpersonal relationship between worker and supervisor, including trust, respect, and support. It is an aggregation of responses to:

• Supervisors in my work unit support employee development
• My supervisor listens to what I have to say
• My supervisor treats me with respect
• I have trust and confidence in my supervisor
• Overall, how good a job do you feel is being done by your immediate supervisor?

The Intrinsic Work Experience sub-index is intended to reflect the employees’ feelings of motivation and competency relating to their roles in the workplace. It is an aggregation of responses to:

• I feel encouraged to come up with new and better ways of doing things
• My work gives me a feeling of personal accomplishment
• I know what is expected of me on the job
• My talents are used well in the workplace
• I know how my work relates to the agency's goals

Other indexes include the Global Satisfaction Index (broken into Job Satisfaction, Pay Satisfaction, Organizational Satisfaction, and Recommend Organization) and the New Inclusion Quotient Index (broken into Fair, Open, Cooperative, Supportive, and Empowering).

Back to top

Discussion

Employee climate surveys are tools used to solicit and assess employee opinions, feelings, perceptions, and expectations regarding a variety of factors related to organizational health, such as opportunities for growth, relations with management, working relationships, and overall environment.[2] Climate surveys such as the United States FEVS can have significant value in measuring employee engagement over time. 

In order to have value, climate survey results must be taken seriously by senior management. The issues such surveys bring to light must be addressed. Without both the reality and the perception of responsive action, employees will quickly become disillusioned and the effort will have been wasted.

The FEVS, administered annually to approximately two million U.S. federal employees, is clearly a substantial undertaking. An effort of this magnitude would be excessive for most statistical organizations looking for a measurement tool to use internally. Nonetheless, the FEVS might serve as a useful point of departure for organizations just getting started, or as a comparative point of reference for organizations who are looking to improve the instrument they currently employ.

Though the FEVS allows U.S. statistical organizations to assess themselves against other parts of the U.S. federal workforce, this is actually of limited value. There can be a thrill in finding oneself ranked higher than a bureaucratic rival, but the department-level (or ministry-level) comparisons are only a very rough guide to whether internal communications and employee engagement initiatives as a whole have been relatively effective. 

Much more useful are the internal division-level and workgroup-level results, which allow mid-level managers and supervisors to assess their own effectiveness and give upper management some insight with which to examine potential differences in approaches and outcomes across organizational units.

Like with any survey, methodological matters, such as instrument design, and operational matters, such as response rates, play a huge role in the ultimate accuracy and representational quality of the results. Thus, any climate survey instrument needs to be evaluated by survey methodologists and field tested before widespread use. 

Respondent confidentially is also critical. Statistical organizations may want to consider having a climate survey administered by a third party to ensure that management cannot identify individual employee responses.

FEVS results can be fairly variable from year to year, especially at the workgroup level. With a sample as small as 25 respondents, one or two unhappy employees, or just a few new employees, can have a notable effect on an annual score.

One disadvantage of a survey as broad as the FEVS is that it cannot address the idiosyncrasies of any particular organization or drill down to give nuanced measures of individual initiatives. As such, a statistical organization considering a similar regular organization-based climate survey might want to include a set of broad questions, which are repeated every cycle, along with a set of specific questions addressing current issues and initiatives that change from cycle to cycle.

On a related note, statistical organizations must make decisions concerning the periodicity of climate survey administration. The benefits of annual or possibly more frequent administration should be weighed against the costs of operations and respondent fatigue.

Finally, results from a periodic climate survey are not sufficient in themselves to present a full picture of an organization’s internal communications and employee engagement. Ideally, the quantitative results of a climate survey should be coupled with qualitative results from structured focus groups or informal conversations. Note that the FEVS does not include any text entry fields for comments.  With a smaller survey population, statistical organizations might have the resources to read and interpret free-form input that would enable employees to express more nuanced views on matters of significance

[1] The survey instrument, methodological detail, and government-wide results going back to 2004 are available at https://www.opm.gov/fevs/

[2] Definition adapted from http://the-definition.com/term/climate-survey

Back to top