![]()
1. The Proposition’s Restrictions
To convey meaning
Context demands
The audience knows
And truly understands
Propositional restrictions refer to the characteristics of the problem being studied (i.e., the proposition) in combination with the characteristics of the available environment. These characteristics limit researchers’ abilities to derive results that stand the test of time—and many other researchers working on the same or a similar topic. These limits exist because certain things can be done, and others cannot (e.g., see the section on human participants); and certain things are known, and others are not (i.e., researchers never seem to be able to acquire all of the data on their "wish list"). Even things that researchers believe they know are rarely known with certainty or optimal precision. The result is that "perfect research" is most often an oxymoron, as has been seen countless times through the conflicting results touted in the media. These conflicting results, especially when published in a peer-reviewed journal, are not often a sign of poor research. Rather, they are the natural consequence of propositional restrictions, along with other restrictions detailed in later chapters.
The restrictions covered in this chapter are as follows:
| a. | Questions: the research questions driving the project. Good questionscan lead to good research, but poor questions seldom do. |
| b. | Reliability: repeatability or consistency. Whether the issue is test results (e.g., medical, education, etc.) or supposedly the same data extracted from two sources or in two different ways, the results should be about the same. Two "identical" blood tests from different labs should yield the same results from a split blood sample, as should two different IQ tests. Notice, no mention is being made of being correct in any aspect of reliability—simply repeatability or consistency. |
| c. | Validity: the extent to which the available data reflect the characteristics thought to be the ones being studied; the intersection of intent with process. Here is where correctness matters. Repeatability is not enough. Everyone can be wrong, as the history of science has shown. |
| d. | Generalizability: the extent to which research results can be trusted to be accurate for a parent population from which samples were derived. When entire populations are used, generalizability is not a problem. |
| e. | Assumptions: the conditions that are believed to be true without specific evidence. Although inevitable, assumptions are often handled by simply listing them in the research report. |
| f. | Bias: generally, the unconscious prejudicing of the study through researchers’ preconceptions or through methodological flaws in the research. Personal bias can be harder to overcome than methodological flaws because it tends to be less apparent to an independent reader. |
| g. | Confounds: the characteristics that might actually be responsible for the results but were not accommodated in the research. Confounds are a major contributor—some would say the major contributor—to research results being reversed or greatly modified over time. |
The remaining sections in this chapter each discuss these proposi-tional restrictions in turn and how they are accommodated by a high school principal, a director of public health at the state level, and a professor of sociology. Through the examples listed throughout these sections, it becomes clear that research, as it is conducted, is very different from what might be thought. It is most often confined to more limited and poorer quality information than would be optimal for lasting results. These issues set the stage for the fragility of results that is commonly the unspoken hallmark of research.
1a. Questions
Facts don’t quite fit
Not sufficiently pat
A revised look
Might answer that
Good research questions are a hallmark of good research. The reason is simple: Research questions motivate and define practically all ensuing aspects of a research design. Good research questions have well-defined terms and are objective, concrete, and answerable within the available resources.
The high school principal wants to know the extent to which extracurricular activities have an impact on grades. His superintendent has asked that the expense for the activities be justified or else the activities might be eliminated from a budget that seems to shrink every year. Although the principal genuinely believes in the value of these activities, his belief alone will not be sufficient to ensure their continuation.
In designing his research, he first drafted his question as, "Do students taking extracurricular activities get higher grades?" Upon reflection, he noticed that different types of students are engaged in extracurricular activities, and even these types of students differ according to the activities. For example, students from poorer families often have to work after school and do not attend extracurricular activities at all. Students on the math team seem quite different from stagehands for a school play in their aptitude for math. Furthermore, few people would assume that participation in a school play would be as effective at increasing math grades as would a math club. A good research question should be able to accommodate these issues without becoming overly long or complicated.
The principal revised his question to ask, “Is the addition of academic extracurricular activities associated with increased grades in associated courses?” The question has become better by being more restricted, but differences in students who are able to avail themselves of the opportunity to become involved in the relevant extracurricular activities must still be accommodated. Furthermore, the research results must also be adjusted for the initial achievement of the students (i.e., before becoming involved in the extracurricular activities). Nonetheless, the research question has become better by becoming more restricted. More restrictions will be added as other aspects of research are addressed.
The director of public health wants to know if public clinics are providing a substantial portion of the childhood immunizations that do not appear in her data. Unlike private physicians, clinics are not required to publicly report childhood immunizations in the state, and the legislature is reluctant to change the law. The governor wants to know the extent to which a related public health epidemic is plausible, given the numbers of children without a record of being immunized.
In designing her research, she originally asked the question “Do clinics immunize children?” Upon reflection, she sees that the stated question suggests a “yes” or “no” answer as the result. She is really interested in the extent to which children are being immunized in clinics. Her question becomes a bit more complicated: “What percentage of the childhood population receives immunizations from clinics, which do not publicly report the information?”
The data for answering her revised question likely will be difficult to obtain. Because the clinics are not required to publicly report the information, it is likely kept (if at all) in a manner that would be burdensome to retrieve. Yet the precision required of the project would likely allow for other methods to suffice. For example, clinics do need to track expenses and inventory. By asking for copies of the immunization shipment receipts and current inventories, estimates of the numbers of children immunized could be calculated by the number of doses no longer in inventory. The estimates would be less precise where multiple doses per child are needed, but the overall pattern of findings would likely be sufficient for her purpose.
The professor wants to know if matrilineal cultures have more equal rights for women than patrilineal cultures. Her personal interests lie with two west African cultures, and although she has been unable to secure funding for fieldwork abroad, she has arranged instead to meet with groups of immigrants in a nearby major metropolitan area. She has reliable access to several representatives from both west African ethnic groups. Furthermore, she can eat dinner at home with her family most evenings, so she is not entirely displeased with the research restriction of her not being able to travel to Africa at this time. Her original question was, “Do matrilineal cultures have more equal rights than patrilineal cultures?” After discussions with her colleagues and an assessment of her resources, she modified her question to be, “Do west African matrilineal cultures currently have more equal rights than west African patrilineal cultures?” The term equal rights is somewhat vague, but it appears to be well understood by her subjects to mean equal rights under the law, as well as by a cultural respect that is demonstrated by courteous behavior toward women.
Matrilineal cultures vary widely in the importance placed on females, and it is possible that our researcher will need to narrow her methodology to case studies of the two specific cultures she is researching. Nonetheless, some of her participants seem knowledgeable about their neighboring cultures, and our researcher is hoping to draw the broadest defensible generalizations from her project. She is reasonably confident that she can answer her questions in a scholarly manner without more costly fieldwork in Africa, at least for now.
1b. Reliability
We all agree
Yet can be wrong
Reliability without validity
Is a very sad song
Reliability is the extent to which different methods or people would arrive at the same data or results. On the surface, it would seem to be a critical aspect of data and results—and it is— when the data and the results are sufficiently correct. Unfortunately, correctness is a characteristic of validity, not reliability. Therefore, although the common conception of reliability is somewhat overrated, it forms the basis for validity, which is critical to good research. The reason is that validity cannot exist without reliability.
The question then becomes, "Why do researchers care about reliability?" The answer is that once something is repeatable, it is much easier to retarget or refocus than if it were not repeatable. For example, darts players first learn to throw darts with an identical motion time after time so the darts land in about the same spot on the board with practice (i.e., reliability). Only then do they shift where the darts will land (i.e., validity, when hit). Simply throwing darts at a board rarely results in a skilled player and most often serves to increase the randomness of the results (i.e., less reliability and validity).
The high school principal has two sources of data for achievement, because the older hard copy system was recently replaced by an electronic system. His first thought is to test the extent to which grades were properly entered into the electronic system (i.e., the reliability of the electronic system). To compare the results of the two systems, the principal met with the district statistician to discuss sample size requirements. The sampling was conducted, grades were abstracted from the hard copy records, and a comparison was prepared. Much to the principal’s surprise, only 97.8% of the grades matched. Some were off by more than a letter grade. Although 97.8% might sound like a high proportion (i.e., high reliability due to the high repeatability), the principal noted that it meant that approximately 264 grades were wrong in the population of grades, from a school with approximately 1,000 students. For most social science research, 97.8% reliability would be grounds for a celebration, but not for the principal. He had all of the hard copy grades checked, and those that did not agree were re-entered. A second check on the system revealed the reliability to be 99.7%, which the principal could accept.
The director of public health is willing to accept a far lower standard for reliability, in this case by necessity. The research requested by the governor does not need a precise answer and is really based on judgment coming from information that can be gathered. For example, her intended method for calculating doses of vaccine delivered to children (i.e., clinic receipts minus serum on hand) does not accurately account for some vaccines that require multiple doses. She knows from previous research that approximately 85% of children in her state who receive a single dose from a multiple protocol also receive the follow-up doses, so she can generate some estimates. Yet she also knows that vaccine sometimes expires without being used. Under that condition, state law requires the vaccine to be properly disposed of but does not require a record of the amount. The director of public health will also need to estimate clinic spoilage. These conditions require her acceptance of moderate reliability for the results—at best.
The professor has performed extensive reviews of historical documents on the subject, and she is conducting face-to-face interviews as well as observing the participants. She has arranged to live with two west African families, one matrilineal and one patrilineal, each for a week during her vacations. Qualitative research is often quite subjective, and even though she is attempting to be as objective as possible, she knows her small study may not be very reliable. She accommodates this by disclosing her restrictions and theoretical assumptions early in her writing. She also plans to focus on her personal interactions with each ethnic group, as she is an attractive young African American woman. The extent to which each group responds to her will be detailed in her findings.
1c. Validity
Certainty conferred
From a human source
Often proves false
When truth runs its course
In a very real sense, validity in research is the result of the intersection of our intent with the process of its implementation. Researchers believe that they know what they want to measure but often find that their available measures are somewhat compromised by being a blend of what they want to measure and something else. For example, when researchers want to know the impact of a certain drug on the course of a disease, they might be measuring not only the impact of the drug but also the impact of the potentially different lifestyle of people who would volunteer for a clinical trial, even when the volunteers are randomly divided into experimental ...
