Handbook of Research Methods in Industrial and Organizational Psychology

Steven G. Rogelberg, Steven G. Rogelberg

• Publisher: Wiley-Blackwell
• Year: 2015
• ISBN:
  9780470573716
• Edition: 1
• Topic: Psicologia
• Subtopic: Psicologia in industrie e organizzazioni

Handbook of Research Methods in Industrial and Organizational Psychology is a comprehensive and contemporary treatment of research philosophies, approaches, tools, and techniques indigenous to industrial and organizational psychology.

• Only available research handbook for Industrial & Organizational Psychology.
• Contributors are leading methodological & measurement scholars.
• Excellent balance of practical and theoretical insights which will be of interest to both novice and experienced organizational researchers.
• Great companion to the content-oriented Handbooks.
• Now available in full text online via xreferplus, the award-winning reference library on the web from xrefer. For more information, visit www.xreferplus.com

PART I
Foundations

CHAPTER ONE
History of Research Methods in Industrial and Organizational Psychology: Measurement, Design, Analysis

James T. Austin, Charles A. Scherbaum, and Robert A. Mahlman

Our aim in this chapter is to review the history of research methods. An underlying premise is that in so doing we can improve current research. Research methods and theories enable the description, prediction, and understanding of organizational behavior. Phenomena of longstanding concern to industrial and organizational (I-O) psychologists (Wilpert, 1997) pertain to broadly-defined behavior by groups and individuals, within organizations, and the interrelationships among these levels. It is clear that the evolution of research methods brought the wealth of choices available to I-O psychologists (MacCallum, 1998; Sackett and Larson, 1990). What important issues arise from studying the history of research methods? We used three strategies to accomplish this review. We examined published research and historical discussions (e.g., Cowles, 1989; Morawski, 1988; Owen, 1976); in addition, we coded 609 empirical articles over a period of time from the Journal of Applied Psychology (JAP) in order to track researcher choices (cf. Sackett and Larson, 1990).

A Time X Domain framework organizes this chapter. The levels of the time facet are intervals that span 1904 to 1935, 1936 to 1968, and 1969 to 2000. Intervals were selected to group time and also to identify landmarks and trends. In the year 1935, for example, Thurstone and colleagues founded the Psychometric Society and Fisher published Design of Experiments. In 1968, Statistical Theories of Mental Test Scores (Lord and Novick, 1968) appeared and Cohen (1968) brought the general linear model into wider view. Currently, there are several potential landmarks in the research methods literature. One is computational modeling (Ilgen and Hulin, 2000), another is an integration of test theory models (McDonald, 1999), and a third consists of volumes honoring and extending the work of Donald Campbell (Baum and McKelvey, 1999; Bickman, 2000a, 2000b) and Douglas Jackson (Goffin and Helmes, 2000).

Table 1.1 Representative important developments within a methods domain X temporal interval matrix

Method domain	Temporal interval facet
	1904–1935	1936–1968	1969–2000
Measurement	Classical test theory	Item response theory, construct validity	Generalizability theory, consequential validity
Design	Longitudinal, design of experiments	Internal, external validity	Multi-level designs, causal inference
Analysis	Inference, multiple regression analysis	Multivariate methods ANOVA/ANCOVA	Quantitative synthesis, covariance structure models

The levels of the second facet, research methods, are measurement, design, and analysis (Pedhazur and Schmelkin, 1991). Measurement consists of conceptualizing and scoring the attributes of entities. Design involves planning and executing research to support valid inferences that may generalize beyond the sample. Analysis is making sense of the resultant data from measurement and design. Choices made in all these domains clearly influence study interpretations. Table 1.1 presents the organizing framework with one or more representative developments for each domain-interval intersection.

Although presented orthogonally, the levels of the research methods facet are indeed closely interrelated. Consider warnings against substantive research without first establishing construct validity (Schwab, 1980). Consider situations in which poorly designed research aims to inform policy. Lastly, consider the clash about whether analysis depends upon level of measurement (Michell, 1986, 1999). We force the separation of levels for purposes of exposition.

Historically, research methods first developed in wider spheres. We thus provide a general history of each domain using the time intervals as rough boundaries before tracing developments within the I-O field. In part, this tactic recognizes that, during the formative years of I-O psychology, researchers and practitioners were trained in experimental psychology (Katzell and Austin, 1992). In fact, Walter Dill Scott and Hugo Münsterberg were trained in Europe by Wilhelm Wundt, a founder of experimental psychology. In part, this tactic recognizes the role of the diffusion of innovations, a process by which innovations spread through various information channels over time (Johns, 1993; Rogers, 1995). The process of innovation helped research methods permeate into new areas of psychology, in this case from experimental to industrial psychology. In general, innovations in measurement, design, and analysis have diffused – sometimes slowly, sometimes more quickly – from developers of research methods to researchers who actually implement those methods. We first describe our coding of JAP articles. Although not the only journal of the field, it does have a long publication history in the United States, and this figured in our choice. Our purpose is to amplify discussion in the sections that follow on measurement, design, and analysis.

Snapshots over Time from the Journal of Applied Psychology

Consider the first empirical article in JAP. Terman (1917) evaluated 30 candidates for municipal positions using a shortened Stanford-Binet (S-B) and 7 “pedagogical” tests. He described his sample (age, education), provided frequencies for mental age, IQ, and test scores, and presented a matrix of correlations among the predictors and reported salary (range .17 to .81; probable error = .078). Spearman-Brown reliability was estimated as .69 by split-halves. Criteria were candidates’ reports of past salary and occupational level. A cut-off of 80 (S-B) eliminated 10 candidates. Analyses presented correlations among predictors, and with salary and general or specific sets of the tests. Observations on individual cases concluded the report. A failure to study other assessments (medical and physical exams, moral qualities) was noted as a limitation by Terman.

To develop that snapshot further into a series, we coded 609 empirical articles from nine volumes of JAP (every tenth volume between 1920 and 2000). Our choice of journal was predicated on prestige and length of publication, while our sampling strategy provided representative coverage. Our coding was based on the research methods facet and on previous work (Sackett and Larson, 1990; Stone-Romero, Weaver, and Glenar, 1995). The coding sheet is reproduced in figure 1.1, and the sample and data are described in tables 1.2 and 1.3. Notice in table 1.2 the increasing proportion of what would be regarded as “current” I-O topics, the pronounced shift from single to multiple authors, and the relatively small fluctuations in median sample size. Table 1.3 provides percentage-use-indices (PUI) for each domain using the calculations of Stone-Romero et al. (1995).

Measurement

Measurement and scaling as we know them today grew from procedures used by Galton to study individual differences and by Fechner to study psychophysics. Despite its de-emphasis (Aiken, West, Sechrest, and Reno, 1990), measurement remains important in I-O education, practice, and research. The concept of measurement includes representing scientific concepts, developing instruments, and evaluating score inferences through capturing relevant and irrelevant sources of score variance. Browne (2000) used categories of mental test theory, factor analysis and rela...