Validity and Reliability in Built Environment Research

A Selection of Case Studies

Vian Ahmed, Alex Opoku, Ayokunle Olanipekun, Monty Sutrisna, Vian Ahmed, Alex Opoku, Ayokunle Olanipekun, Monty Sutrisna

• Publisher: Routledge
• Year: 2022
• ISBN:
  9780429516375
• Edition: 1
• Topic: Arquitectura
• Subtopic: Arquitectura general

This book aims to guide researchers who are engaged in social science and built environment research through the process of testing the reliability and validity of their research outputs following the application of different methods of data collection.

The book presents case studies that emphasize reliability and validity in different examples of qualitative, quantitative and mixed method data sets, as well as covering action research and grounded theory. The reader is guided through case studies that demonstrate:

• An understanding of the reliability and validity approaches from social science and built environment perspectives in alignment with the relevant research philosophies, approaches and data collection strategies
• Real research projects that have been conducted by expert researchers on topics such as Lean, BIM, Housing and Sustainability to answer specific or evolving questions in relation to the reliability and validity of research
• A simple and easy method that students at Masters and PhD levels can relate to in order to adopt a sound reliability and validity approach to their research

This book is the essential guide for researchers at undergraduate and postgraduate level who need to understand how to validate the quality of the empirical tests they conduct using different techniques. The book will also be a great asset to supervisors from different backgrounds who need a refresher on this key aspect of the research cycle.

Part IResearch reliability and validity

1Understanding reliability in research

Vian Ahmed, Ayokunle Olanipekun, Alex Opoku and Monty Sutrisna

DOI: 10.1201/​9780429243226-2

1.1 Measuring errors in research

An important part of Built Environment (BE) research is the measurement of social phenomena. The idea of measurement conforms with the positivist research paradigm, or the empirical analytic approach for discerning reality and explaining social phenomena in the process (Dainty, 2008; Drost, 2011). In line with the paradigm, measurement requires precise definitions of (social) conceptual meanings, and such concepts in the BE include safety, performance, in addition to design and social science concepts that are applicable to the functioning of the BE such as culture and motivation. These concepts are abstract concepts, which are only theoretically constructed (Kimberlin & Winterstein, 2008). Measurement involves the operationalization of the concepts into definite variables (or specific questions/items) and the application of measuring instruments (and scientific tests) to quantify them (Kimberlin & Winterstein, 2008). For example, successful project performance may be operationalized as “project completion within or under-budget” while the related measurement instrument may ascertain “data on cost information” of completed projects. A questionnaire (which may either structured or unstructured) is the most common measuring instrument for operationalizing abstract concepts in BE and consequently used to obtain data.

However, measurement errors are plausible when abstract concepts are operationalized and developed into questionnaires to obtain data in BE research. Compared with objective sources of data like laboratory test results, data obtained on abstract concepts using questionnaires involve greater subjectivity in judgement that leads to error in measurement (Kimberlin & Winterstein, 2008). Accordingly, Drost (2011) identifies two sources of measurement errors in research: systematic errors and random errors. For example, consider a bathroom scale, a systematic error is when the scale produces a consistent measure of a person's weight but was always 5lb higher than it should be. A random error is when the scale produces correct weight measure, but the person misread the weight value. Based on the classical test theory, measurement error is the difference between the true value and the measured value obtained by a measuring instrument (i.e. a questionnaire) (Kimberlin & Winterstein, 2008; Mohajan, 2017). In a standard BE research, the value obtained by using a questionnaire is the sum of both the “true value” which is naturally unknown, and the “error” in the measurement process (Mohajan, 2017). Essentially, the “true value” is obtained if a measurement is perfectly accurate (Kimberlin & Winterstein, 2008). Therefore, pertaining to the measurement of abstract concepts, minimizing measurement errors to the barest minimum by enhancing the reliability of the measurement instrument and validity of the measured concepts will increase the rigour of BE research on social phenomena (e.g. Bannigan & Watson, 2009).

1.2 Research reliability

In quantitative BE research, measurements of social concepts are carried out by using measuring instruments (i.e. questionnaire). The measuring instrument is reliable when it yields consistently the same or comparable results over repeated measures (Drost, 2011). That is, regardless of who performs the measurement, and the occasion and condition under which measurement was carried out, the results produced by the measuring instrument is consistent (or comparably consistent) (Brink, 1993; Mohajan, 2017). Therefore, reliability can be regarded as the accuracy of a measuring instrument in quantitative BE research (Heale & Twycross, 2015). Take the bathroom scale example, if it consistently produces a person's true weight (e.g. 35lb) over repeated times, then it is a reliable measuring instrument. Therefore, for the BE researcher, the challenge of reliability is to develop measuring instruments to obtain the true values of measured concepts to reduce error in measurement process. This requires the testing of reliability of measuring instruments (Heale & Twycross, 2015). The three attributes of reliability that are often tested are: stability, homogeneity or internal consistency and equivalence as outlined in Table 1.1.

Table 1.1 Attributes of reliability test

Attributes	Description
Stability	The consistency of results using an instrument with repeated testing
Homogeneity (or internal consistency)	The extent to which all the items on a scale measure one construct
Equivalence	Consistency among responses of multiple users of an instrument, or among alternate forms of an instrument
Source: Heale et al. (2015)

1.2.1 Stability

Stability refers to the ability of a measure to remain the same over time without controlling the testing conditions or respondent themselves (Mohajan, 2017). Therefore, a perfectly stable measuring instrument will produce the same results when administered time after time to collect data (Bannigan & Watson, 2009) and this is obtained by performing the test-retest reliability method.

1.2.1.1 Test-retest reliability method

The test-retest reliability refers to the temporal stability of test from one measurement session to another (Drost, 2011). It is obtained by administering the same test twice, or more over a period ranging from few weeks to months, on a group of individuals (respondents) (Mohajan, 2017) under similar circumstances (Heale & Twycross, 2015). The procedure is to administer the test to a group of respondents and then administer the same test to the same respondents later (Drost, 2011). Thereafter, a statistical comparison is made between participant's test scores (values) for each of the times they have completed it to provide an indication of the reliability of the instrument (Heale & Twycross, 2015). For example, construction workers may be asked to complete the same questionnaire about safety satisfaction twice in three months so that test results can be compared to assess stability of scores. The correlation coefficient calculated between two sets of data, and the higher the coefficient, the better the test-retest reliability (and stability).

(Mohajan, 2017)

Qu et al. (2009) studied the quality of life of migrant construction workers in Shenyang, China by using the 36-Item Short Form Health Survey (SF-36) questionnaire. The questionnaire is divided into eight domains of individual questions about the physical function, role limitations due to physical problems, bodily pain, general health, vitality, social functioning, role limitations due to emotional problems and mental health. The study was designed to evaluate the quality of life of the migrant construction workers at one-week time apart and therefore, the test-retest reliability method was performed to demonstrate the stability of the questionnaire over time. In the first time, a total of 1125 SF-36 questionnaires were administered to the migrant construction workers, and a week a later, the questionnaires were administered to 50 of them who were randomly selected. The correlation test was used to perform statistical comparison between the migrant workers’ responses to the questionnaires in the first time and the week later. As shown in Table 1.2, the retest of the correlation between the items showed that r > 0.70 could be achieved for all eight domains (P < 0.01) (Table 1.2), demonstrating relatively good stability for the SF-36 questionnaire. The high correlation values indicate that the responses of the migrant construction workers about their quality of life remained uniform/consistent despite responding at different times, and a low correlation value would suggest otherwise. According to Ajayi (2017), the high correlation signifies high reliability of the SF-36 questionnaire administered to the migrant construction workers.

Table 1.2 Test-retest reliability results

Domains	Test-retest reliability results (n = 50)
Physical function (PF)	0.801**
Role limitation due to physical problems (RP)	0.781**
Bodily pain (BP)	0.856**
General health (GH)	0.721**
Vitality (VT)	0.962**
Social functioning (SF)	0.841**
Role limitation due to emotional problems (RE)	0.78
Mental health (MH)	0.793**
^**P < 0.01 Source: Qu et al. (2009)

1.2.1.2 Limitation

Test-retest reliability is defined by the correlation between scores (values) on the identical tests given at different times (Drost, 2011) and this leads to some limitations. For instance, when the interval between the first and second t...