Modern Psychometrics

The Science of Psychological Assessment

John Rust, Michal Kosinski, David Stillwell

• Publisher: Routledge
• Year: 2020
• ISBN:
  9781317268765
• Edition: 4
• Topic: Psychology
• Subtopic: History & Theory in Psychology
• Index: Psychology

This popular text introduces the reader to all aspects of psychometric assessment, including its history, the construction and administration of traditional tests, and the latest techniques for psychometric assessment online.

Rust, Kosinski, and Stillwell begin with a comprehensive introduction to the increased sophistication in psychometric methods and regulation that took place during the 20th century, including the many benefits to governments, businesses, and customers. In this new edition, the authors explore the increasing influence of the internet, wherein everything we do on the internet is available for psychometric analysis, often by AI systems operating at scale and in real time. The intended and unintended consequences of this paradigm shift are examined in detail, and key controversies, such as privacy and the psychographic microtargeting of online messages, are addressed. Furthermore, this new edition includes brand-new chapters on item response theory, computer adaptive testing, and the psychometric analysis of the digital traces we all leave online.

Modern Psychometrics combines an up-to-date scientific approach with full consideration of the political and ethical issues involved in the implementation of psychometric testing in today's society. It will be invaluable to both undergraduate and postgraduate students, as well as practitioners who are seeking an introduction to modern psychometric methods.

1 The history and evolution of psychometric testing

Introduction

People have always judged each other in terms of their skills, potential, character, motives, mood, and expected behavior. Since the beginning of time, skill in this practice has been passed on from generation to generation. Being able to evaluate our friends, family, colleagues, and enemies in terms of these attributes is fundamental to us as human beings. Since the introduction of the written word, these opinions and evaluations have been recorded, and our techniques for classifying, analyzing, and improving them have become not just an art but also a technology that has played an increasing role in how societies are governed. Triumphing in this field has been the secret of success in war, business, and politics. As with all technologies, it has been driven by science—in this case, the science of human behavior, the psychology of individual differences, and, when applied to psychological assessment, psychometrics.

In the 20th century, early psychometricians played a key role in the development of related disciplines such as statistics and biometrics. They also revolutionized education by introducing increasingly refined testing procedures that enabled individuals to demonstrate their potential from an early age. Psychometrics required statistical and computational know-how, as well as data on a large scale, before its impact could be felt. Today, we think of big data in terms of the technological revolution, but large-scale programs implementing the analysis of human data on millions of individuals date back to over 100 years ago in the form of early national censuses and military recruitment. These early scientists did not see their subject as just an interesting academic discipline; they were also fascinated by its potential to improve all our lives. And indeed, in most ways it has, but it has been a long and rocky road—with many false starts, and indeed disasters, on the way. In this chapter, we start with definitions, followed by an evaluation of future potential, a history, a warning about past missteps, applause for current successes, and an invitation to learn from history’s lessons. It is said that those who cannot remember the past are condemned to repeat it. Let us all make sure that this does not happen, but rather that we can bring about a future that sees human potential expand to the stars.

What is psychometrics?

Psychometrics is the science of psychological assessment, and has traditionally been seen as an aspect of psychology. But its impact has been much broader. The scientific principles that underpin psychometrics apply equally to other forms of assessment such as educational examinations, clinical diagnoses, crime detection, credit ratings, and staff recruitment. The early psychometricians were equally at home in all of these fields. Since then, paths have often diverged, but they have generally reunited as the importance of advances made in one context come to the attention of workers in other areas. Currently, great strides are being made in the application of machine-learning techniques and big-data analytics—particularly in the analysis of the digital traces we all leave online—and these are beginning to have a significant impact across a broad range of applications. These are both exciting and disturbing times.

We experience psychometric assessment in many of our activities, for example:

• We are tested throughout our education to inform us, our parents, teachers, and policy makers about our progress (and the efficiency of teaching).

• We are assessed at the end of each stage of education to provide us with academic credentials and inform future schools, colleges, or employers about our strengths and weaknesses.

• We must pass a driving test before we are allowed to drive a car.

• Many of us need to pass a know-how or skills test to be able to practice our professions.

• We are assessed in order to gain special provisions (e.g., for learning difficulties) or to obtain prizes.

• When we borrow money or apply for a mortgage, we must complete credit scoring forms to assess our ability to repay the debt.

• We are tested at work when we apply for a promotion and when we seek another job.

• Our playlists are analyzed to assess our music tastes and recommend new songs.

• Our social media profiles are analyzed—sometimes without our consent—to estimate our personality and choose the advertisements that we are most likely to click.

Assessment can take many forms: job interviews, school examinations, multiple-choice aptitude tests, clinical diagnoses, continuous assessment, or analysis of our online footprints. But despite the wide variety of applications and manifestations, all assessments should share a common set of fundamental characteristics: they should strive to be accurate, measure what they intend to measure, produce scores that can be meaningfully compared between people, and be free from bias against members of certain groups. There are good assessments and bad assessments, and psychometrics is the science of how to maximize the quality of the assessments that we use.

Psychometrics in the 21st century

Psychometrics depends on the availability of data on a large scale, and so it is no surprise that the advent of the internet has massively boosted its influence. If we had to date the internet, we would probably start at CERN, the European Organization for Nuclear Research, in Geneva, with Tim Berners-Lee’s invention of the World Wide Web in 1990; he linked the newly developed hypertext markup language (HTML) to a graphic user interface (GUI), thereby creating the first web pages. Since then, the web has expanded to make Marshall McLuhan’s “global village” a reality (McLuhan, 1964). The population of this global village grew from a handful of academics in the early 1990s to a diverse and vibrant community of one billion users in 2005, and to over four billion users (representing more than 50% of the world’s population) in 2020. Thus, within less than 20 years, the new medium of cyberspace came into existence, creating a completely new science with new disciplines, new experts, and, of course, new problems. Some aspects of this new science are exceptional. While the science of biology is only 300 years old, and that of psychology considerably younger, both their subjects of study—humans and life itself—have existed for millions of years. Not so the internet. Hence the cyberworld is unique, and it is hard to predict what to expect of its future. It is also a serious disruptor; it has completely changed the nature of its adjacent disciplines, especially computer and information sciences, but also psychology and its progeny, psychometrics.

By the year 2000, the migration of psychometrics into the online world was well underway, producing both new opportunities and new challenges, particularly for global examination organizations such as the Educational Testing Service (ETS) at Princeton and Cambridge Assessment in the UK. On the positive side, gone were the massive logistical problems involved in securely delivering and recovering huge numbers of examination papers by road, rail, and air from remote parts of the world. But the downside was that examinations needed to take place at fixed times during the school or working day, and it became possible for candidates in, say, Singapore to contact their friends in, say, Mexico with advance knowledge of forthcoming questions. Opportunities for cheating were rife. To counter these challenges, the major examination boards and test publishers turned to the advantages offered by large item banks and computer adaptive testing, the psychometricians’ own version of machine learning. However, it was the development of the app—an abbreviation of “application” used to describe a piece of software that can be run through a web browser or on a mobile phone—that was to prove the most disruptive to traditional ways of thinking about psychometric assessment.

One such app was David Stillwell’s myPersonality, published on Facebook in 2007 (Stillwell, 2007; Kosinski, Stillwell, & Graepel, 2013; Youyou, Kosinski, & Stillwell, 2015). It offered its users a chance to take a personality test, receive feedback on their scores, and share those scores—if they were so inclined—with their Facebook friends. It was similar to countless other quizzes widely shared on Facebook around that time, yet it employed an established and well-validated personality test taken from the International Personality Item Pool (IPIP), an open-source repository established in the 1990s for academic use as a reaction to test publishers’ domination of the testing world. The huge popularity of myPersonality was unforeseen. Within a few years, the app had collected over six million personality profiles, generated by enthusiasts who were interested to see the sort of results and feedback about themselves that had previously only been available to psychology professionals. It was one of psychometrics’ first encounters with the big-data revolution.

But the availability of psychometric data on such a grand scale was to have unexpected consequences. Many saw opportunities for emulating the procedure in online advertising, destined to become the major source of revenue for the digital industry. Once the World Wide Web existed, it could be searched or trawled by search engines, the most ubiquitous of which is Google. In the mid-1990s, search engines simply provided information. By 2010 they did so with a scope and accuracy that exceeded all previous expectations; information on anything or anyone was ripe for the picking. But those who wished to be found soon became active players on the scene—it was the advertising industry’s new paradise. The battle to reach the top in search league tables—or, at the very least, the first results page—began in earnest. Once online advertising entered the fray, it became a new war zone. The battle for the keywords had begun. Marketing was no longer about putting up a board on the high street; it was about building a digital presence in cyberspace that would bring customers to you in droves. By the early 2000s, no company or organization could afford not to have a presence in cyberspace. For a high proportion of customers, companies without some digital presence simply ceased to exist.

While web pages were the first universally available data source in cyberspace, social networks soon followed, and these opened a whole new world of individualized personal information about their users that was available for exploitation. Not only was standard demographic information such as age, marital status, gender, occupation, and education available, but there were also troves of new data such as the words being used in status updates and tweets, images, music preferences, and Facebook Likes. And these data sources soon became delicious morsels in a new informational feeding frenzy. They were mined extensively by tech companies and the marketing industry to hone their ability to target advertisements to the most relevant audiences—or, to put it another way, to those who might be most vulnerable to persuasion. The prediction techniques used were the same as those that had been used by psychometricians for decades: principal component analysis, cluster analysis, machine learning, and regression analysis. These were able to predict a person’s character and future behavior with far more accuracy than simple demographics. Cross-correlating demographics with traditional psychometric data, such as personality traits, showed that internet users were giving away much more information about their most intimate secrets than they realized. Thus, online psychographic targeting was born. This new methodology, creating clickbait and directing news feeds using psychological as well as demographic data, was soon considered to be far too powerful to exist in an unregulated world. But this will prove one day to have been just the midpoint in a journey that began many centuries ago.

History of assessment

Chinese origins

Employers have assessed prospective employees since the beginnings of civilization, and have generated consistent and replicable techniques for doing this. China was the first country to use testing for the selection of talents (Jin, 2001; Qui, 2003). Earlier than 500 BCE, Confucius had argued that people were different from each other. In his words, “their nature might be similar, but behaviors are far apart,” and he differentiated between “the superior and intelligent” and “the inferior and dim” (Lun Yu, Chapter Yang Huo). Mencius (372–289 BCE) believed that these differences were measurable. He advised: “assess, to tell light from heavy; evaluate, to know long from short” (Mencius, Chapter Liang Hui Wang). Xunzi (310–238 BCE) built upon this theory and advocated the idea that we should “measure a candidate’s ability to determine his position [in the court]” (Xun Zi, Chapter Jun Dao).

Thus, over 2,000 years ago, much of the fundamental thinking that today underpins psychometric testing was already in place, as were systems that used this in the selection of talents. In fact, there is evidence that talent selection systems appeared in China even before Confucius. In the Xia Dynasty (c. 2070–1600 BCE), the tradition of selecting officers by competition placed heavy emphasis on physical strength and skills, but by the time of the Zhou Dynasty (1046–256 BCE) the content of the tests had changed. The emperor assessed candidates not only based on their shooting skills but also in terms of their courteous conduct and good manners. From then on, the criteria used for the selection of talent grew to include the “Six Skills”: arithmetic, writing, music, archery, horsemanship, and skills in the performance of rituals and ceremonies; the “Six Conducts”: filial piety, friendship, harmony, love, responsibility, and compassion; and the “Six Virtues”: insight, kindness, judgment, courage, loyalty, and concord. During the Warring States period (475–271 BCE), oral exams became more prominent. In the Qin Dynasty, from 221 BCE, the main test syllabus primarily consisted of the ability to recite historical and legal texts, calligraphy, and the ability to write official letters and reports. The Sui (581–618 CE) and Tang Dynasties (618–907 CE) saw the introduction of the imperial examinations, a nationwide testing system that became the main method of selecting imperial officials. Formal procedures required—then as they do now—that candidates’ names should be concealed, independent assessments by two or more assessors should be made, and conditions of examination should be standardized. The general framework of assessment set down then—including a “syllabus” of material that should be learned and rules governing an efficient and fair “examination” of candidates’ knowledge—has not changed for 3,000 years. While similar but less sophisticated frameworks may have existed in other ancient civilizations, it was models based on the Chinese system that were to become the template for the modern examination system. The British East India Company, active in Shanghai, introduced the Chinese system to its occupied territories in Bengal in the early 19th ...