The first major changes to language testing in a half-century are underway. Here’s the story.

BY DAVID B. SAWYER

istockphoto.com / lemono

The Foreign Service Institute’s School of Language Studies (FSI/SLS) is undertaking the most substantial reforms to State Department language testing in nearly 50 years. The goal of these changes is to ensure that language test scores are relevant, fair, and best serve the needs of both students and the department.

The U.S. State Department requires proficiency in foreign languages for many Foreign Service professionals. This requirement is established in law—Section 702(a) of the Foreign Service Act of 1980 states: “Foreign Service posts abroad will be staffed by individuals having a useful knowledge of the language or dialect common to the country in which the post is located.” Knowledge of foreign languages strengthens cultural adaptability, one of 13 essential Foreign Service attributes (“dimensions”) that include the ability to work and communicate effectively and harmoniously with persons of other cultures and value systems, and to recognize and respect differences in new and different cultural environments.

The State Department’s language proficiency tests are highly consequential for the careers of nearly every Foreign Service employee. The department uses test scores to make decisions regarding retention, tenure, assignments, promotion at all levels, and language incentive pay. Given its outsized impact on employees and their families, the Foreign Service Institute is cognizant of the importance of language testing to Foreign Service personnel—including assignment and transfer. That is why it is imperative that FSI maintain a language testing program that is fair, transparent, and relevant to the vital mission of America’s diplomats.

Changes Overdue

Changes to language testing were overdue. While FSI’s language test was groundbreaking when introduced in the 1950s and has been incrementally updated since then, FSI leadership acknowledged that the test had not kept pace with current developments in language assessment and that significant innovations in the field went well beyond any previous revisions.

Separately, questions and concerns had arisen about limitations or potential biases associated with the testing program, particularly against individuals who identify as native or heritage speakers—terms for which there is no single, set definition—and perceptions that the department was not fully valuing their contributions to the workforce. Other concerns were raised about potential bias in testing based on accents, language variety, or educational or demographic background. (Please go to the LTU intranet site—http://fsi.state.gov/SLS/6951—for detailed information.)

Finally, the nature of diplomacy and the work of Foreign Service professionals have changed significantly over the decades through the use of email, text, social media, and the internet, resulting in shifts in the language skills required in overseas postings.

FSI’s goal is to ensure that language testing reflects the real-world requirements of Foreign Service work.

To address these concerns, FSI in 2018 commissioned a major study by the National Academies of Sciences, Engineering, and Medicine (NAS). Following receipt of the report, FSI conducted outreach to stakeholders throughout the State Department and convened a 20-person Task Force on the Future of Language Testing. Based on its review of the report, the task force proposed changes to language testing structure, content, administration, and scoring.

The task force also discussed concerns about bias and made recommendations to address them, while FSI undertook a parallel yearlong effort to examine training and testing of heritage speakers to ensure that they are tested fairly and their skills are recognized, including addressing concerns about unconscious bias.

The task force produced a new Statement of Test Purpose, Use, and Impact, to serve as a guide for the test redesign process.

The Changes, in Detail

In July 2022, FSI’s Language Testing Unit (LTU) began implementing the changes recommended by the task force and will continue with substantial revisions through 2023. For test takers, changes are both to scoring and the format and content of the test. Changes to scoring include updates to the assessment criteria and greater transparency through score reports.

Scoring. Tests will be scored with a new, updated version of the 0-5 Interagency Language Roundtable (ILR), often referred to as the ILR Skill Level Descriptions (SLDs). The new SLDs no longer refer to the “native speaker,” as definitions vary, and native speakers show a range of language proficiency across ILR skill levels. FSI now caps scores at 4, with 4, 4+, and 5 collapsed into one category, because the department’s highest requirement is also ILR 4 and the distinctions between these levels are challenging to identify in testing. These scores now appear as “AP” for Advanced Proficiency in the language skills inventory and the employee’s profile. The LTU is developing Foreign Service–specific rubrics and scoring procedures that will be implemented in early 2023 to tailor the assessment more closely to Foreign Service work and capture more information about the test. These new scoring criteria give greater weight to listening comprehension on the speaking test and reweight scores to stress communicative effectiveness and fluency more than grammatical and lexical precision. The result is a combined speaking/listening score that emphasizes communicative ability.

Score reports. The Language Testing Unit now provides a tailored score report, something long sought by examinees seeking feedback on their performance. Changes to scoring procedures are capturing more information about each test, which is reflected in the score report. The score report format will evolve when the new Foreign Service–specific rubrics and scoring procedures are introduced.

Changes to the format and content of the test include changes to tasks on the speaking test and texts on the reading test.

Introductions during the speaking test. The introduction at the start of the test focuses on a professional, rather than personal, context to address concerns that personal biographical information may be a source of unconscious bias. It is up to examinees to decide how to introduce themselves and share information about their professional background as they would when interacting with a foreign counterpart.

Presentation during the speaking test. The second part of the speaking test previously began with the examinee providing a 3- to 5-minute explanation or introduction to a chosen topic. This task has been eliminated, because many Foreign Service employees indicated that they had never given such a presentation or had done so only with the assistance of locally employed staff.

Flow of the speaking test. Most of the speaking test occurs in a scenario-based, topical conversation that flows through a series of listening and speaking tasks. The scenario is a meeting or conference that is typical of the types of meetings that Foreign Service employees attend. The examinee will select the meeting topic from a list of options that reflect all career tracks. A program or agenda—in English—will be provided to the examinee during the test and serve as material for the conversation.

Listening on the speaking test. Roughly half of the speaking test will be devoted to measuring listening comprehension. Tasks will include the examinee responding to questions and answers about the conference program or meeting agenda and meeting sessions. The last section of the speaking test will follow a format very similar to the interview previously on the third part of the speaking test, where the examinee asks questions, gathers information, and reports it back in English. While there is not a separate score for listening, these changes give equal weight to speaking and listening and result in a combined speaking/listening score, reflecting the integrated nature of speaking and listening.

FSI undertook a parallel yearlong effort to examine training and testing of heritage speakers.

Texts on the reading test. While literary and cultural references may appear in texts that are representative of the type of materials Foreign Service employees may read for work-related tasks, reading materials will not include poetry as a text genre.

These changes are in various stages of implementation. For information on the latest status and to prepare for a test, please visit the LTU’s testing reforms website on State Department SharePoint and attend one of the LTU’s monthly online information sessions.

Testing Operations

The COVID-19 pandemic has also led to permanent changes in testing. Prior to March 2020, most testing occurred in-person in the LTU or in more than 20 percent of cases by digital video conference (DVC). The move to remote work necessitated a change to telephone testing; and as soon as bandwidth made it feasible, the Language Testing Unit moved to the governmentapproved WebEx platform. Since January 2021, the LTU has conducted all remote testing on this platform and, with the return to in-person testing post-pandemic, will retain this platform for all remote testing into the foreseeable future. For speaking-only tests, examinees can be at any location. For tests that involve reading, however, examinees must be in a department facility to maintain the security of the testing materials.

Quality Assurance

Since 2017, the Language Testing Unit’s Quality Assurance program has ensured the reliability of test scoring and provides an established baseline against which to measure the impact of impending changes to the test. Test review requests from students typically average about 8 percent of tests per year. Beyond that, the LTU has routinely reviewed more than 20 percent of tests, and that will expand to 50 percent this fall following the recommendations of the NAS report. The LTU has a strong track record, with the consistency of scoring meeting or exceeding testing industry standards. The Language Testing Unit also routinely conducts benchmarking activities with other federal agencies and its counterpart office in the Canadian Foreign Service Institute, with excellent results.

The LTU’s tester and examiner training and certification program is one of the most extensive in language testing and includes mandatory training in mitigating unconscious bias for all testing staff. On the test exit survey administered to all examinees from January through June 2022, 96 percent of examinees said they were satisfied with their testing experience. When answering a new question introduced in April 2022, 86 percent of respondents stated that the test reflects their future language proficiency needs. More than 70 percent of examinees who had received their test scores thought that their performance on the exam was an accurate reflection of their ability, and an even greater proportion agreed with their scores (79 percent). The Language Testing Unit is confident that the testing reforms will result in even stronger numbers.

FSI’s goal is to ensure that language testing reflects the real-world requirements of Foreign Service work and keeps pace with modern developments in applied linguistics and measurement. These are evidence-based, data-driven efforts that include incorporating diversity, equity, inclusion, and accountability principles and addressing the concerns of the workforce.

Dr. David B. Sawyer is director of the Foreign Service Institute’s Language Testing Unit within the School of Language Studies. Specialized in applied linguistics, language mediation, and educational measurement, he holds graduate degrees and a doctorate in these fields, has served on the faculty at three universities, and publishes widely on related topics. He joined the State Department’s Office of Language Services in 2003 as the principal interpreter for German and supported the most senior levels of the U.S. government in that capacity for more than a decade.

Read More...

• “What State Should Bring to the Table: Cultural and Language Expertise,” by Phil Skotte, The Foreign Service Journal, May 2018
• “The Foreign Service Institute at 70: Recalling a Proud History,” by Steven Alan Honley, The Foreign Service Journal, December 2017
• “Five Ways to Tackle a New Language,” by Elijah David Bush and Todd Hughes, The Foreign Service Journal, September 2016