The Foreign Service Journal, October 2022

THE FOREIGN SERVICE JOURNAL | OCTOBER 2022 39 percent, hardly an efficient use of resources when you consider the time and effort involved in the administration of the FSOA. (The pass rate for the written exam between 1980 and 2006 averaged 23.6 percent, and only 25 percent of those candidates [5.9 per- cent] passed the FSOA.) Since the introduc- tion of the QEP, the success rate at the FSOA has gone up to nearly 60 percent, while the FSOA itself and the scoring rubrics used at the FSOA remain essentially the same. In short, the introduction of the QEP achieved its intended effect of identifying better qual- ified candidates, who were then more successful at the FSOA. The QEP, however, was also labor intensive. In 2015, on the advice of the Board of Examiners and industrial and organiza- tional psychologists, BEX introduced a computer-based, deep textual analysis of candidate files at the QEP stage, referred to as the computer-QEP. It drew on roughly 42,000 candidate files that had been manually scored by BEX assessors between 2007 and 2015, with each file representing six scores as determined by three separate assessors. This amounted to roughly 756,000 individual data points reflecting the scoring efforts of more than 150 BEX assessors over a period of eight years. (The assessors reflect the diversity of the Foreign Service in terms of career track, gender, race, and ethnicity.) Using this data, the computer- QEP was trained to identify candidates with relatively strong or weak chances of proceeding to the FSOA. The computer-QEP ranks candidates, and based on the number of candidates who signed up for a particular career track, the strongest candidates in each track from that cohort advance to the assessor-QEP. Assessors are not told the candi- date’s ranking. In recent years, roughly 250 people from each career track advanced to the assessor-QEP. Where fewer than 250 candidates selected a particular career track, all candidates advanced to the assessor-QEP. BEX assessors then manually score those files, and depending on the anticipated hiring needs of the department, the top candidates by career track receive an invitation to the FSOA. The computer-QEP only reviews text associated with a unique number assigned to each candidate. Neither the computer-QEP nor the assessor-QEP has any other identifying information about a candidate. As with all other aspects of the Foreign Ser- vice selection process, BEX assessors regularly test the program’s reliability and validity to ensure its accuracy, further improving its effectiveness by adding more scoring data with each cohort. BEX assessors also verify the computer-QEPs to ensure the score is valid. Feedback is given to the QEP asses- sors after each QEP. QEP panels consists of three assessors who are required to deliber- ate outlying scores. The scoring rubrics are continuously updated. Let’s see the numbers. In 2021, BEX ran the personal narrative files of all 6,514 FSOT takers through the computer-QEP and discovered that there were a fair number of candidates who did not “pass” the FSOT but, based on their background and experience as described in their QEP files, would have moved successfully through the computer-QEP to the assessor-QEP had they done so. (Note: We put “pass” in quotation marks, because since 2007 the “passing” score was not set at a specific qualifying score but rather at a score so that roughly half the test takers in each cohort advanced to the QEP.) In the chart above, the first column shows the theoretical result if we had let the computer-QEP look at the entire cohort and decide who we should have advanced to the assessor-QEP. The computer selected 3,611 candidates. The second column shows what happened in practice: 3,571 candidates “passed” the FSOT and were moved to the QEP phase. Then we dug deeper to see who these 3,611 computer-QEP candidates were compared to the FSOT-generated 3,571. We found that roughly two-thirds of the population remained the same, but we would have lost 1,007 “real” candidates (that is, those who “passed” the FSOT) and gained 1,047 “theoretical” candidates (as selected by the computer) at the assessor-QEP under the revised system. Because we could, in fact, track the progress of the 1,007 can- didates who “passed” the FSOT and went on to the QEP, we dug even deeper to see how this group of candidates ultimately fared. Of the 1,007, only 11 advanced to the FSOA after the computer and assessor QEPs. Of those, only four achieved the minimum Test Result Computer-QEP only FSOT only Fail 2,903 2,943 Pass 3,611 3,571 Grand Total 6,514 6,514 All FSOT Candidates Calendar Year 2021 ISTOCKPHOTO.COM/SEN SVECTOR