No longer a pass/fail gateway to the Foreign Service, written test scores will be considered with other factors evaluated by means of a complex algorithm at the Qualification Evaluation Panel stage.

BY DEIDI DELAHANTY

istockphoto.com / sensvector

How do we ensure that the State Department attracts and hires the most competitive, qualified candidates for the Foreign Service? For the Foreign Service Board of Examiners (BEX), the focus is on assessment. In coordination with testing industry experts, BEX continuously reviews its various test components and materials, making data-driven adjustments and improvements as warranted. The most recent change, announced on April 26—factoring a candidate’s Foreign Service Officer Test (FSOT) score at the Qualification Evaluation Panel (QEP) step—is the result of in-depth analysis to improve the selection of candidates most likely to succeed in the Foreign Service. The change is designed to ensure BEX assessors receive a holistic view of each candidate’s qualifications.

Why this change?

To answer this, we need some historical context. The QEP was introduced in 2007 as an intermediate step between the FSOT and the time- and labor-intensive Foreign Service Oral Assessment (FSOA). At the QEP, BEX assessors review a candidate’s résumé, as well as answers to six personal narrative questions. Using a series of detailed rubrics tied to the 13 Dimensions (see box on page 40), candidate files are scored; and the strongest candidates in each career track from that testing cohort advance to the oral assessment. Considering a candidate’s experience as part of the threshold to the FSOA is in line with best practices in the private sector and makes sense for the Foreign Service, as well.

Prior to 2007, any candidate who achieved a passing score on the FSOT received an invitation to the FSOA. Under that process, the historical pass rate at the FSOA hovered at around 22 percent, hardly an efficient use of resources when you consider the time and effort involved in the administration of the FSOA. (The pass rate for the written exam between 1980 and 2006 averaged 23.6 percent, and only 25 percent of those candidates [5.9 percent] passed the FSOA.) Since the introduction of the QEP, the success rate at the FSOA has gone up to nearly 60 percent, while the FSOA itself and the scoring rubrics used at the FSOA remain essentially the same. In short, the introduction of the QEP achieved its intended effect of identifying better qualified candidates, who were then more successful at the FSOA.

The QEP, however, was also labor intensive. In 2015, on the advice of the Board of Examiners and industrial and organizational psychologists, BEX introduced a computer-based, deep textual analysis of candidate files at the QEP stage, referred to as the computer-QEP. It drew on roughly 42,000 candidate files that had been manually scored by BEX assessors between 2007 and 2015, with each file representing six scores as determined by three separate assessors. This amounted to roughly 756,000 individual data points reflecting the scoring efforts of more than 150 BEX assessors over a period of eight years. (The assessors reflect the diversity of the Foreign Service in terms of career track, gender, race, and ethnicity.) Using this data, the computer-QEP was trained to identify candidates with relatively strong or weak chances of proceeding to the FSOA.

The computer-QEP ranks candidates, and based on the number of candidates who signed up for a particular career track, the strongest candidates in each track from that cohort advance to the assessor-QEP. Assessors are not told the candidate’s ranking. In recent years, roughly 250 people from each career track advanced to the assessor-QEP. Where fewer than 250 candidates selected a particular career track, all candidates advanced to the assessor-QEP. BEX assessors then manually score those files, and depending on the anticipated hiring needs of the department, the top candidates by career track receive an invitation to the FSOA.

The computer-QEP only reviews text associated with a unique number assigned to each candidate. Neither the computer-QEP nor the assessor-QEP has any other identifying information about a candidate. As with all other aspects of the Foreign Service selection process, BEX assessors regularly test the program’s reliability and validity to ensure its accuracy, further improving its effectiveness by adding more scoring data with each cohort. BEX assessors also verify the computer-QEPs to ensure the score is valid. Feedback is given to the QEP assessors after each QEP. QEP panels consists of three assessors who are required to deliberate outlying scores. The scoring rubrics are continuously updated.

Let’s see the numbers.

In 2021, BEX ran the personal narrative files of all 6,514 FSOT takers through the computer-QEP and discovered that there were a fair number of candidates who did not “pass” the FSOT but, based on their background and experience as described in their QEP files, would have moved successfully through the computer-QEP to the assessor-QEP had they done so. (Note: We put “pass” in quotation marks, because since 2007 the “passing” score was not set at a specific qualifying score but rather at a score so that roughly half the test takers in each cohort advanced to the QEP.)

All FSOT Candidates Calendar Year 2021

In the chart above, the first column shows the theoretical result if we had let the computer-QEP look at the entire cohort and decide who we should have advanced to the assessor-QEP. The computer selected 3,611 candidates. The second column shows what happened in practice: 3,571 candidates “passed” the FSOT and were moved to the QEP phase.

Then we dug deeper to see who these 3,611 computer-QEP candidates were compared to the FSOT-generated 3,571. We found that roughly two-thirds of the population remained the same, but we would have lost 1,007 “real” candidates (that is, those who “passed” the FSOT) and gained 1,047 “theoretical” candidates (as selected by the computer) at the assessor-QEP under the revised system.

Because we could, in fact, track the progress of the 1,007 candidates who “passed” the FSOT and went on to the QEP, we dug even deeper to see how this group of candidates ultimately fared. Of the 1,007, only 11 advanced to the FSOA after the computer and assessor QEPs. Of those, only four achieved the minimum score of 5.25 on the oral assessment. Given their relatively weak scores, it is not clear that any of them will be hired.

But what about the 1,047 candidates we would have seen instead? On further analysis, including examining the FSOT scores of these candidates, we found that we would likely have advanced strong candidates to the assessor-QEP who were also more diverse ethnically and racially, and included more women.

Why not get rid of the FSOT entirely and just use the computer-QEP?

The FSOT provides insight into elements and skills the State Department seeks for future diplomats, characteristics that themselves are determined through extensive job analysis and surveys. But on its own, the FSOT was a poor predictor of success as a Foreign Service officer, as evidenced by the abysmal FSOA pass rate before 2007. Far from “lowering standards” or “dumbing down the FS,” as some have claimed, the new process still factors a candidate’s performance on the FSOT into the decision to advance them to the FSOA, but adds consideration of additional qualifying factors through the QEP, including work, volunteering, language knowledge, and experience.

In fact, while under the previous system a candidate’s FSOT score played no further role in the assessment process beyond pass/fail, now a candidate’s numerical FSOT score is combined with the computer-QEP score to create one overall score. Based on that combined score, top candidates advance to the assessor-QEP, where assessors determine who will be invited to the FSOA. The FSOA itself remains unchanged and is the final, rigorous stage of the assessment process.

There are many reasons why a candidate might not do well on the FSOT that do not reflect their potential to serve effectively in the Foreign Service. Continuing to rely on it as a pass/ fail gateway to Foreign Service hiring erected an unnecessary barrier to entry when the QEP process has proven over the past 15 years to be highly effective at identifying strong candidates for the FSOA. Combining these two processes, rather than running them consecutively, will give us the most holistic look possible at candidates.

Why now?

We didn’t have the data before. Before 2020, QEP files were only collected for those who “passed” the FSOT. A procedural change requiring all applicants to provide QEP materials at the same time they signed up for the FSOT meant that we now had files for both “passers” and “non-passers” to analyze. As soon as we had amassed sufficient data and discovered that this tweak in our process would move forward stronger candidates, we moved to implement this change. The time from discovery to implementation was six months.

How do you protect the process from partisan influence and future politicization?

Multiple safeguards are in place that make politicization or improper influence extremely unlikely, if not impossible. First, the assessment process is provided a safeguard against improper influence through regular review by the Board of Examiners for the Foreign Service, “the Board,” which comprises representatives of the foreign affairs agencies and qualified experts as outlined in the Foreign Service Act of 1980. The process itself is designed, reviewed, and continually tested by external industrial and organizational psychologists. And changes, like this most recent change, are approved by the Board.

“BEX,” which constitutes the staff of “the Board,” invests in extensive training of its assessors to mitigate bias and to apply well-defined and objective rubrics to each assessment. Assessor scores are monitored and evaluated for unusual tendencies, such as scoring too easy or too hard, or identifying differences in how an assessor may score certain groups; and assessors receive regular quantitative feedback.

Our goal, as always, is to advance the most qualified candidates into the Foreign Service and to remove as many unnecessary barriers to entry as possible.

The State Department makes public detailed information on the assessment process, including information on all required steps; the 13 Dimensions, which underpin all evaluations; and practice tests. We update our assessment materials to ensure they remain targeted to Foreign Service work based on the evaluation of external experts who review the assessment process in light of actual job requirements. In this way, “validation,” as it is called, ensures that the hiring process is fair, objective, and accurate based on the job requirements.

In the new process, the use of the FSOT score and a computer-QEP rank should further safeguard against politicization or undue influence on choosing the candidates who advance to the next stages of selection. The FSOT is a proctored standardized test. The use of the computer-QEP program, based on the raw data of thousands of applicants reflecting the judgments of a wide array of BEX assessors, is yet another safeguard.

How does this affect aspiring FSOs?

BEX recognizes that our decisions and processes affect people’s career opportunities, and we take that responsibility seriously. These changes were proposed and crafted by experienced career Foreign Service officers and fully vetted by the Board and academic and private sector experts. This change is data-driven and will allow us to better identify and hire the best possible Foreign Service officer candidates. BEX will continue to track closely and validate the performance of the automated deep textual analytics tool and the revised assessment process.

A virtual oral assessment?

Our goal, as always, is to advance the most qualified candidates into the Foreign Service and to remove as many unnecessary barriers to entry as possible. One major barrier to entry is the cost to the applicant to attend the oral assessments, be they specialists or generalists. To address this issue, in February 2022 all Foreign Service specialist oral assessments were permanently moved to virtual platforms. Now, applicants interested in more than 15 specialties can apply from anywhere in the United States and from most parts of the world. As of July 2022, BEX has assessed 685 specialists across 15 specialties and customer satisfaction survey respondents have lauded this move.

Moving forward, BEX is making plans to transition the generalist FSOA to a virtual platform, as well, with the first of several proposed changes anticipated in 2023.

Together, we are modernizing the Foreign Service assessment process to better serve the State Department’s needs and mission.

Deidi Delahanty, a retired Foreign Service officer, serves as senior adviser to the State Department Board of Examiners as a reemployed annuitant. During a 40-year career with the Department of State, she served overseas in Warsaw, Bridgetown, Vilnius, Tashkent, Kyiv, Ankara, and Seoul. In Washington she worked in the Comptroller's Office, the Press Office, the Operations Center, the Allowances Staff, the Bureau of Inter-American Affairs, the Bureau of European and Eurasian Affairs, the Bureau of East Asian and Pacific Affairs, and the Bureau of Global Talent Management. She was deputy director of the Office of Foreign Missions Midwest Regional Office, was detailed to the Department of Labor in Chicago, and is a 2007 graduate of the National War College.

THE 13 DIMENSIONS

istockphoto.com / dmitry kovalchuk

What qualities do we seek in a Foreign Service Officer (FSO) candidate? The successful candidate will demonstrate the following dimensions that reflect the skills, abilities, and personal qualities deemed essential to the work of the Foreign Service at the United States Department of State.

1. Composure. To stay calm, poised, and effective in stressful or difficult situations; to think on one’s feet, adjusting quickly to changing situations; to maintain selfcontrol.

2. Cultural Adaptability. To work and communicate effectively and harmoniously with persons of other cultures, value systems, political beliefs, and economic circumstances; to recognize and respect differences in new and different cultural environments.

3. Experience & Motivation. To demonstrate knowledge, skills, or other attributes gained from previous experience of relevance to the Foreign Service; to articulate appropriate motivation for joining the Foreign Service.

4. Information Integration & Analysis. By absorbing and retaining complex information drawn from a variety of sources; to draw reasoned conclusions from analysis and synthesis of available information; to evaluate the importance, reliability, and usefulness of information; to remember details of a meeting or event without the benefit of notes.

5. Initiative & Leadership. To recognize and assume responsibility for work that needs to be done; to persist in the completion of a task; to influence significantly a group’s activity, direction, or opinion; to motivate others to participate in the activity one is leading.

6. Judgment. To discern what is appropriate, practical, and realistic in a given situation; to weigh relative merits of competing demands.

7. Objectivity/Integrity. To be fair and honest; to avoid deceit, favoritism, and discrimination; to present issues frankly and fully, without injecting subjective bias; to work without letting personal bias prejudice actions.

8. Oral Communication. By speaking fluently in a concise, grammatically correct, organized, precise, and persuasive manner; to convey nuances of meaning accurately; to use appropriate styles of communication to fit the audience and purpose.

9. Planning & Organizing. As prioritizing and ordering tasks effectively, to employ a systematic approach to achieving objectives, to make appropriate use of limited resources.

10. Resourcefulness. To formulate creative alternatives or solutions to resolve problems; to show flexibility in response to unanticipated circumstances.

11. Working with Others. To interact in a constructive, cooperative, and harmonious manner; to work effectively as a team player; to establish positive relationships and gain the confidence of others; to use humor as appropriate.

12. Written Communication. To write concise, well-organized, grammatically correct, effective, and persuasive English in a limited amount of time.

13. Quantitative Analysis. To identify, compile, analyze, and draw correct conclusions from pertinent data; to recognize patterns or trends in numerical data; to perform simple mathematical operations.

—U.S. Department of State

Read More...

• “State Department rethinks how it vets Foreign Service candidates to diversify ranks,” by Jory Heckman, Federal News Network, June 2022
• “The Foreign Service by the Numbers: Where We Stand,” by Julie Nutter, The Foreign Service Journal, January-February 2020
• “Examining State’s Foreign Service Officer Hiring Today,” by Glenn J. Guimond, The Foreign Service Journal, July-August 2016
• “Who Is the Total Candidate? FSO Hiring Today,” by Shawn Dorman, The Foreign Service Journal, June 2008