The Foreign Service Journal, December 2015

20 DECEMBER 2015 | THE FOREIGN SERVICE JOURNAL that evaluation experts recommend that “8-10 percent of the budget of any pro- gram should be invested in evaluation,” and that the Gates Foundation spends “a reported 15 percent on performance measurement.” While the commision’s report does stress that PD leadership has started to pay more attention to measurement and evaluation, Vice President Joe Biden’s oft-quoted admonition to federal agencies—“Don’t tell me what you value; show me your budget, and I’ll tell you what you value”—seems appropri- ate here. By budgetary metrics, we do not value evaluation. Moreover, the evaluations we do carry out often lack rigor. The report notes that many public diplomacy evalu- ations tend to focus more on outputs than outcomes, exaggerate results, and seem to be designed less as a tool for improving or discontinuing certain programs and more as an exercise in “placating Congress.” In Search of the Holy Grail One reason we haven’t been able to satisfactorily measure public diploma- cy’s impact is that doing so is extremely difficult, if not impossible. In fact, many public diplomacy scholars refer to evi- dence of the impact of PD as “the holy grail” of their profession. The evaluation guide mentioned above lists many of the problems that make PD programs so difficult to mea- sure. Here are two of the most intrac- table factors: ■ PD work involves intangibles. Documenting verifiable changes in awareness, perceptions and attitudes requires an investment of considerable time, effort and skill. Doing so over a long period of time amplifies the chal- lenge considerably. ■ Results may not be directly attributable to PD intervention. It is often difficult to draw a straight line of causation between a PD program and its desired result. Time, external events and other actors complicate the cause-effect equation. Related to these problems is the difficulty in establishing appropriate program objectives in the first place. While the PD training department at the Foreign Service Institute has done a great job teaching officers how to design “SMART” (Specific, Measurable, Attain- able, Relevant and Time-Bound) objec- tives, most objectives that meet those stringent criteria are measureable in terms of output (people trained, people reached, number of participants) rather than the impact we are ultimately look- ing for (understanding acquired, minds changed, etc.). Output is merely what you did. Impact is what you achieved . It is no doubt because of these chal- lenges that many PD officers tradition- ally do not value measurement and eval- uation. Why spend time and resources on an evaluation whose results will be, at best, indeterminate? Grander Objectives, Larger Target Audiences Another major challenge in assessing the impact of public diplomacy pro- grams is that we have increasingly set grander and more ambitious goals for our foreign policy in general, and our PD programs specifically. Over the years, PD work has become about much more than just increas- ing understanding of the United States and its values. Many PD programs are about trying to instill our values in other societies, remaking other cultures in our image. Reflecting this change in scope, today’s PD programs are increasingly in line with integrated country strate- gies (ICS). Practitioners try to “move the needle” on common ICS objectives like strengthening democratic norms and institutions, encouraging entrepreneur- ship and economic reform, and empow- ering girls and women. Take, for example, the Young South- east Asian Leaders Initiative and the Young African Leaders Initiative. Both programs target tens of thousands of 18- to 35-year-olds across large regions, with the objective of creating young leaders (through leadership training and profes- sional development), then empower- ing them to bring about fundamental changes in their societies (through grants and other funding). As a result of their participation in these programs, these youth are expected to start businesses, advance women’s rights, bring about democratic reforms, create initiatives to protect the environment and implement many other noble social projects in their home countries. Compare these grand objectives with the relatively modest aims of one of our longest-running public diplomacy pro- grams, the International Visitor Program (now the International Visitor Leader- ship Program). Created in the 1940s, the IVLP has the objective of “increasing mutual understanding” among a rela- tively narrow target audience of “up- and-coming leaders and elites” through a one-time guided tour of the United States. Ironically, as public diplomacy pro- grams have become more strategically focused, they’ve also become harder to manage and evaluate. Measuring an “increase in understanding” among a small defined group of elites and track- ing them into the future is difficult, but not impossible. But evaluating and