Testing in the Professions

Credentialing Policies and Practice

Susan Davis-Becker, Chad W. Buckendahl

• Publisher: Routledge
• Year: 2017
• eBook ISBN: 9781317615859
• Edition: 1
• Topic: Education
• Subtopic: Education General

Testing in the Professions focuses on current practices in credentialing testing as a guide for practitioners. With a broad focus on the key components, issues, and concerns surrounding the test development and validation process, this book brings together a wide range of research and theory—from design and analysis of tests to security, scoring, and reporting. Written by leading experts in the field of measurement and assessment, each chapter includes authentic examples as to how various practices are implemented or current issues observed in credentialing programs.

The volume begins with an exploration of the various types of credentialing programs as well as key differences in the interpretation and evaluation of test scores. The next set of chapters discusses key test development steps, including test design, content development, analysis, andevaluation. The final set of chapters addresses specific topics that span the testing process, including communication with stakeholders, security, program evaluation, and legal principles. As a response to the growing number of professions and professional designations that are tied to testing requirements, Testing in the Professions is a comprehensive source for up-to-date measurement and credentialing practices.

The Open Access version of this book, available at http://www.taylorfrancis.com, has been made available under a Creative Commons Attribution-Non Commercial-No Derivatives 4.0 license.

1 Credentialing

A Continuum of Measurement Theories, Policies, and Practices

Chad W. Buckendahl

Overview of Credentialing

In concept, testing in the professions may elicit images of candidates preparing for their medical boards or the bar exam. Although this is an accurate picture for these respective professions, the area of credentialing has greatly expanded. Shimberg (1982) noted that at the time, more than 800 professions were licensed and that it was a dramatic increase since the early 1900s when regulatory agencies began to engage in the credentialing process. Schmitt (1995) indicated that the number of professions offering a credential had again increased. More recently, the number of occupations that require a license is greater than 1,100 (Morath, 2015) with this figure representing approximately 30% of workers in the United States (Kearney, Hershbein, & Boddy, 2015). This expansion is not surprising as new or existing professions often try to emulate established professions such as, medicine, dentistry and law, that have long been concerned with developing and maintaining systematic processes for entry into the profession. Emerging professions have the responsibility of persuading their state legislators of the necessity to protect the citizenry from unlicensed or unregulated practitioners who do not have the requisite education, training or competence for safe and effective practice.

This public protection component of state licensure is required to establish the process, but the profession also benefits from a defined scope of practice and protection from competition from less skilled, educated or qualified practitioners. The success that the established professions have had in establishing the credibility of their credential and the brand associated with it has led to numerous other professions striving to build their own credentialing process and programs as a strategy for communicating to the public the importance and distinction of a particular set of knowledge, skills, and abilities. Similarly, earlier volumes have explored a range of test development and validation activities within occupations, particularly focusing on licensure and certification (Fortune, 1985; Impara, 1995; Knapp, Anderson, & Wild, 2009). Extending this work to broader treatments of credentialing is important. Throughout this chapter, the term “profession” is used to be broadly inclusive of all occupations that rely on credentials to help define it.

This expansion of the credentialing concept spans the continuum of measurement practices that may have previously been associated separately with education, licensure, certification, and employment eligibility. However, as is discussed in this chapter, the lines that distinguish these traditional disciplines in the credentialing field are blurring. The result is that practitioners and policymakers need to have a greater breadth of understanding about how measurement theories, policies, and practices apply within and across these areas. The risks of failing to gain this breadth of understanding are also discussed in this chapter as external forces—political and legal—can influence how measurement principles are applied and interpreted in practice. Within the chapter, the term “credentialing” is used to be inclusive of the range of credentials that different sponsors may offer. The distinctions among the types of credentials are discussed below. An important first step in understanding credentialing programs is how they are used and why individuals may seek to obtain a given credential.

Purposes

Similar to the range of credentials available in professions, there is a range of purposes as to why certain credentials exist and how scores from examinations should be interpreted and used. For many credentialing programs, the emphasis is on public protection, specifically to distinguish candidates who are at least minimally qualified from those who are not. The idea of public protection stems from a profession’s belief or regulatory authority’s responsibility that the members of the public would not be able to distinguish qualified versus unqualified performance. For professions involving risk to the public, such as those in architecture, aviation, education, healthcare, or law, an incompetent practitioner can produce lasting, negative consequences.

The concept of public protection is often associated with certain types of credentialing (e.g., licensure, certification), but some programs exist to certify or recognize individuals who have demonstrated specialized knowledge, skills, and abilities in a particular domain or subdomain within a profession (e.g., specialty certification). Other credentials are closely tied to education or training and designed to serve as evidence of learning a particular domain of knowledge or skills associated with something that would be valued in the market (e.g., assessment-based certificates). As an expansion of these historical purposes, an emerging, yet poorly defined class of credentials is also being observed (e.g., micro-credentials, badging). These newer credentials may provide recognition for a range of reasons and be only loosely connected to sound measurement principles. The range of options for candidates and the differentiation for consumers can make understanding credentialing challenging. Beyond programmatic and public purposes, there are some important incentives for participants to engage in a credentialing program.

From a candidate’s perspective, the purpose of seeking a credential may be to demonstrate sufficient competence to be eligible to practice in a given profession. Beyond the entry-level expectations of a profession, there are often market-based incentives for candidates to demonstrate a particular depth of abilities that distinguish themselves from others in their field. More broadly, a credential may be considered by employers as part of eligibility requirements for employment purposes.

Sponsors of Programs

Similar to the range of intended uses and interpretations of scores from credentialing examinations, there is a range of sponsors for these programs. Agencies or organizations that award and enforce credentials may assume responsibility for development and validation of tests associated with their credentials (e.g., National Commission on Certification of Physician Assistants [NCCPA]). At the same time, some professions have formed federations where one or more intermediary agencies have been formed to be responsible for development and validation of the program’s examination(s) (e.g., National Council of Architectural Registration Boards [NCARB], American Board of Dental Examiners [ADEX], Federation of State Boards of Physical Therapy [FSBPT]). Members of the federation (e.g., state licensing boards) contribute to the process and then agree to accept the results of the examination(s) to facilitate efficiency in the process.

For some credentials, membership associations or trade groups may serve as the sponsor for the credentialing program (e.g., American Physical Therapy Association [APTA] for physical therapy specialties, National Strength and Conditioning Association [NSCA] for personal training). For some professions, this may be a single association (e.g., APTA) whereas for others, there may be multiple membership associations or programs that compete for credibility and market share among members of the eligible population (e.g., NSCA). In addition, some credentialing programs are sponsored by organizations that are then seeking to assign or place individuals somewhere within divisions of the parent organization (e.g., agencies like the Army, Navy, Federal Bureau of Investigation, National Security Agency within the U.S. Department of Defense). However, sometimes these credentials begin to have value in the workforce or labor market beyond just the requirements of the sponsoring organization.

The sponsorship and development of credentialing programs by organizations that may be viewed as employers necessitates a discussion of how the intended uses of credentialing examinations can be positioned as intersecting with purposes of education programs and employment decisions.

Overlap with Education and Employment Testing

As is discussed in more detail in the next section, some credentials are closely associated with specific educational training programs or experiences. In these situations, the development and validation processes will often mimic the processes observed in educational achievement testing programs. In a more historical meaning of the term, however, the meaning of the credential is interpreted relative to what occurs in practice and not directly associated with a given curriculum, instructional or training program.

Because many credentials are used as an important source of evidence for demonstrating qualifications, it is not surprising that they are frequently included as part of eligibility requirements in an employment process. This raises the question of whether credentialing examinations should be interpreted as employment tests. Although the purpose of a credentialing examination may be to simply distinguish between those candidates who are minimally competent or not, in some cases, those individuals who do not possess the credential are deemed not to meet the requirements for employment. The Standards for Educational and Psychological Testing (AERA, APA, & NCME, 2014; Standards) make a distinction among tests that are used for selection, placement, and promotion in the employment process from those examinations that are used to make a determination of minimum qualifications that may be used to screen out candidates who do not meet these minimum expectations. Employment tests are sometimes used to rank candidates – for example, interviews may be granted only to those candidates in the higher score groups, especially if the number of available positions is very limited. This ranking function requires that the exam produce scores of adequate information and reliability across the entire score range, rather than just near the passing point. Although seemingly subtle, it is an important distinction that has implications for development, gathering validation evidence, and legal defensibility of the examinations (see Chapter 11).

Suffice it to say, the proliferation of credentials for a range of purposes has led to an increase in the gray area that may have historically distinguished educational assessments, credentialing examinations, and employment tests. Some of this overlap can be attributed to semantics and a lack of clarification of the intended purpose of each program. More concerning is that some of the overlap is attributable to programs expanding purposes beyond the limits that may have been originally intended. Within the educational assessment sector, we have observed policies that incorporate student achievement results for purposes of school and educator accountability. In higher education, the results from licensure and certification exams are used to evaluate or serve as a proxy for outcomes evidence as part of program accreditation. These additional uses can be particularly problematic when organizations or training programs make evaluations based on subscores that lack appropriate psychometric characteristics to support the use. Although perhaps reasonable as policy in concept, the measurement community has been appropriately critical of these expanded uses of assessment when the policies are implemented without the necessary evidence and validity evaluation to support the interpretations and uses of the scores and decisions.

The characterization of credentialing programs can often be confusing and interpreted to mean different things depending on the particular profession and context. Although attempting to explain how or why the confusion emerged would likely be futile, it is important for readers to understand the distinctions among the common types of credentials they may encounter in practice. The next section illustrates these types, highlighting similarities and differences. It should also be noted that because there are different considerations under each country’s respective legal and regulatory systems, the credentialing discussions in this chapter are drawn from interpretation and use in jurisdictions in the United States of America. Although some will be analogous to other countries, jurisdiction-specific requirements will be controlling.

Focus of Credentialing Programs

A variety of credentials are often consolidated into a single group of programs that have a testing component. Although convenient for communicating that the outcome of participating in such a process is a credential that communicates something to the public, all credentials are not equivalent in terms of what they mean and how they are interpreted. Further, semantics can interfere with understanding how credentials are administered and enforced. It also helps to understand how credentialing emerged as an important component in the development of professions.

Credentialing, particularly as it relates to licensure and certification has a long history. Schmitt (1995) provides a summary of how licensure specifically developed since approximately 2000 BC when tariffs were imposed on medical practitioners as part of an effort to regulate practice. Many of the early examples of attempts to regulate professions were related specifically to healthcare. Given that licensure and certification programs are often designed around efforts to protect the health, safety, and welfare of the public, it stands to reason that professionals in the healthcare sector would be associated with these criteria. However, the effort to define and regulate professions extended beyond government agencies.

The Guild system that emerged in the 13th to 15th centuries began to establish rules for membership, expectations for training through apprenticeships, and also member-determined expectations for entry (Schmitt, 1995). The purposes of these efforts were likely less about ensuring competency of members, but rather control of the profession and ultimately the market by limiting the number of individuals who were included to prop up prices and wages. Although the Guilds may have functioned more like trade unions, this initial shift from government regulation to regulation by a profession could be interpreted as the precursor to more self-regulation or certification programs that are sponsored by associations or similar membership organizations within professions. However, as particular trends ebb and flow, the push for regulation of professions has observed similar periods of stronger or weaker support.

One of the reasons for the push for deregulation of certain professional credentialing requirements in the mid-1800s was that for professions like physicians and attorneys, there was a perception that the education programs training these individuals were of much better quality than had been at the time of the initial implementation of the licensing laws. As a result, perhaps the need for regulation had become outdated. However, by the late 1800s, this belief seems to have diminished with the growth in the number of schools and training programs coupled with concerns about the quality and consistency of practitioners who were representing themselves as having a particular skill set (Schmitt, 1995). Since that time, the number of professions regulated by states or professional associations has continued to grow.

It seems to be a case of history repeating itself because in this discussion of reasons for developing and enforcing a credential, the conversation about the need has gone on for decades and yet many of the arguments for why some credentials exist today are still ongoing. Within many professions, there is a belief that graduation from an accredited training program should be sufficient evidence that candidates have achieved at least the knowledge, skills, and abilities to have earned an entry level credential. At the same time, there are stakeholders in these professions who argue that having matriculated through one of many potential educational programs alone is insufficient without an independent demonstration of minimum competence.

Considering the hundreds of professions, specializations, and skill sets that are now credentialed, at what point have we gone too far where the intent and the value of a credential is diluted? Take, for example, a case in which a woman challenged a state’s Barber, Cosmetology/Barber, Esthetics, Electrology and Nail Technology Licensing Board for the right to provide African hair-braiding services without a license after being ordered to cease and desist offering the service (Romboy, 2012). The court in this case noted that the specific hair-braiding service was beyond the scope of the public protection interests of the licensing board. The lingering measurement question is whether the collection of evidence supports the intended interpretation and use of the credential examination and its role in the credentialing program.

There is a difference between this extreme example and representing oneself in a profession without the proper abilities and credentials. However, because credentialing has expanded so much, so too, have the reasons for development of the credential. Where public protection was a historical, primary intent, today, professions without recognized, credible credentials are at risk of being seen as less than professional and are at further risk of allowing external forces to regulate their profession for them. Fortunately, psychometricians have not succumbed to the siren song of credentialing within its profession. This distinction between government-regulated versus profession-regulated is part of the next section that describes the types of credentials and related characteristics observed in practice

Licensure

Perhaps the most widely known credentials are those that can be classified into the licensure category. Most laypersons can name a number of professions that requir...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Table of Contents
• List of Contributors
• Preface
• Acknowledgments
• 1 Credentialing: A Continuum of Measurement Theories, Policies, and Practices
• 2 A Validation Framework for Credentialing Tests
• 3 Test Design: Laying out the Roadmap
• 4 Specifying the Content of Credentialing Examinations
• 5 Content Development and Review
• 6 Estimating, Interpreting, and Maintaining the Meaning of Test Scores
• 7 Data and Scale Analysis for Credentialing Examinations
• 8 Communication with Candidates and Other Stakeholders
• 9 Security Issues in Professional Certification/Licensure Testing
• 10 Using Standards to Evaluate Credentialing Programs
• 11 Legal Issues for Credentialing Examination Programs
• Index