The Foreign Service Journal, July-August 2012

34 F O R E I G N S E R V I C E J O U R N A L / J U LY- A U G U S T 2 0 1 2 Foreign Service National train- ing is up by nearly 300 percent. SZ: The Government Ac- countability Office criticized FSI last year for not being able to prove that its training has an impact in the field. What did you think about that report? RW: I think we took the GAO report very, very seriously. And on the whole, it was very positive. One of the things we learned is it’s not so much what you do, or whether you do something, but whether you have the documen- tation. We shared with them a variety of ways in which we get feedback on training and whether it has an im- pact. But from their point of view, you need to do a for- mal evaluation of each course. We asked whether there was any agency in the federal government that met their standard on accountability, and they couldn’t name one. SZ: It was a pretty high bar? RW: Yes, but we have taken it very seriously. You do have to demonstrate the value that the training has. In the training world, and I think the GAO recognized this, there is a recognized scale of evaluating it, named after the per- son who invented it. On the Kirkpatrick Scale, the evalu- ation at level one is a kind of smiley face. At the end of the class, I give you a piece of paper. You say: “I really had a good time, enjoyed the class, learned a lot,” and then you go. It’s an immediate reaction. Level two is whether you think you really increased your skills. Level three, which is about as high on the scale as anyone in the training world gets, evaluates impact: “Have I been able to use this in my work?” We have a very broad evaluation process now, which we’ve put in place as a result of this feedback. Every year we are doing a level-three evaluation of 30 percent of our courses. This means going out three to six months after a class to try to capture the impact: “Have you been able to implement what you learned on the job?” I think we’ve re- ally tried to take this on, documenting the training and making changes depending on what we found out. TJ: That 30 percent of level-three evaluations reflects an industry standard to which Kirkpatrick says any train- ing organization should aspire. And this year, we’ve met that target. SZ: Is the training making a difference in the field? RW: Responding to our customer survey, 94 percent said they were satisfied or very satisfied. In the feedback that’s coming back in on impact, there are very high numbers of people saying: “I’m putting into practice what I’ve learned.” It varies widely. In a course like the consular course, you’re giving a test and you have a way to say this person mastered this material. We know they know the consular law and regulations. For leadership and soft skills, you don’t really have a test, but people tell us: “Yes, I’m using these skills.” We use people’s responses to continuously review the course curriculum. SZ: Sen. Tom Coburn, R-Okla., said at last year’s Sen- ate hearing on Foreign Service training that he thought the GAO report was reason not to give the State Depart- ment more funding. Do you think he was misinterpret- ing what GAO said? RW: Oh yes, of course I do. The GAO said we were doing an excellent job. SZ: How are instructors recruited, hired and evaluated at FSI? RW: We have about 600 direct-hire staff, both Foreign Service and Civil Service. Our Civil Service employees are for the most part on the training side. We have Foreign Service officers in regular assignments who bid on our jobs. We work very hard to recruit strong candidates in areas like political training or economic training or man- agement/consular training. They are experienced officers doing that work in the field. We pair them up with our Civil Service trainers, who are really training specialists with professional credentials in curriculum development, in adult learning, in course design and course evaluation. It’s a good mix of practi- tioners who know how the work is done in the field and expertise on the Civil Service side on adult learning. Our Foreign Service jobs are very heavily bid, so we are able to attract very high-quality folks. And we think our Civil Service staff gets extremely high marks, as well. TJ: The Foreign Service officers who come to be train- F OCUS “We asked [the GAO] whether there was any agency in the federal government that met their standard on accountability, and they couldn’t name one.”