Faithful Measures

New Methods in the Measurement of Religion

Christopher D. Bader, Roger Finke

• Publisher: NYU Press
• Year: 2017
• ISBN:
  9781479897629
• Topic: Theology & Religion
• Subtopic: Religion & Science

A venture into the art and science of measuring religion in everyday life In an era of rapid technological advances, the measures and methods used to generate data about religion have undergone remarkably little change. Faithful Measures pushes the study of religion into the 21st century by evaluating new and existing measures of religion and introducing new methods for tapping into religious behaviors and beliefs. This book offers a global and innovative approach, with chapters on the intersection of religion and new technology, such as smart phone apps, Google Ngrams, crowdsourcing data, and Amazon buying networks. It also shows how old methods can be improved by using new technology to create online surveys with experimental designs and by developing new ways of mining data from existing information. Chapter contributors thoroughly explain how to employ these new techniques, and offer fresh insights into understanding the complex topic of religion in modern life. Beyond its quantitative contributions, Faithful Measures will be an invaluable resource for inspiring a new wave of creativity and exploration in our connected world

SECTION II

Beyond Surveys

7.

From Documents to Data

CHRISTOPHER P. SCHEITLE

Several chapters in this book explore generating measures from different types of content, whether that content consists of product reviews on Amazon.com or digitized books in Google’s database. This chapter is similar in that it is focused on generating measures from types of content. However, the focus here is less on generating measures created from keyword searches or other types of computer-assisted measure generation and more on traditional coding of content found in types of documents. One might call this more “old school.” This does not mean that the web does not play an important role. As I will show, the increasing presence of all types of documents online means that even more traditional content analyses of documents will often begin on the web. In fact, the web is making access to historical and contemporary documents of all sorts easier.

Documents offer unique ways to examine religion’s role in society. In some cases the documents being examined represent units of analysis that are rarely examined in our usual individual- or congregation-focused surveys. In other cases, documents offer ways to generate measures of phenomena that surveys simply have not yet included or cannot include because the phenomena are historical in nature. On a more practical note, generating measures from documents, while often labor intensive, tends to be relatively inexpensive, which makes this method a potentially attractive option for graduate students. Finally, although I imagine some might disagree, I would also argue that exploring documents is simply fun and rewarding. While it can be fun and rewarding to develop a survey instrument, if that instrument is handed off to a survey firm for the actual data collection, then we do not get to see and interact with our research subjects. This is not the case when we are looking through and exploring documents.

Content analysis is often idiosyncratic. The measures, methods, and challenges found within a content-analysis project will be shaped by the type of content being analyzed. Issues like the representativeness of the content source and the reliability of the content coding should obviously always be considered. I will discuss some of these issues in the context of the examples I discuss in this chapter, but my primary goal is to more generally illustrate and hopefully inspire more use of this method in the social scientific study of religion. I offer a discussion of three types of documents along with a case study for each type that draws from my own research. The chapter ends with some discussion of other potential sources of religion-related measures from documents.

Government Documents

In our lives as employees, patients, business owners, nonprofit leaders, students, and citizens, we often bemoan having to fill out form after form before we can accomplish what seemed at first like an easy task. For social scientists, though, bureaucracies’ tendency to create paperwork presents an opportunity to generate measures from that paperwork. This is true even when one is studying religion.

It is common knowledge among social scientists who study religion, although more surprising to others, that the federal government does not collect much data on religion. Still, it is often assumed that there must at least be some registry where all congregations are counted. This registry is often hypothesized to exist with the Internal Revenue Service (IRS), since it is thought that congregations must, say, apply for tax exemption. Congregations, however, have a special and somewhat confusing status with the IRS that does not require them to officially gain tax-exempt recognition (Scheitle 2010; Scheitle, Dollhopf, and McCarthy 2016). This means that when a reporter or other individual asks us how many congregations exist in the United States, a state, or a city, we are forced to provide ranges or estimates with disclaimers.

These sorts of roadblocks might make many swear off government documents as a potential source of data in the study of religion. This would be a hasty decision, though, as there are possibilities vis-à-vis government documents for the study of religion. I illustrate one such possibility from my own work in detail here, and at the end of the chapter I provide some other ideas for sources of measures.

The case study I illustrate here involves tax return documents for religious nonprofits. As noted above, religious congregations are not required to file the same applications and annual returns with the IRS required of most other nonprofit organizations, or more formally, 501(c)(3) public charities.1 However, religious nonprofits that are not congregations are required to apply for tax exemption and file annual reports with the IRS.2 This means that religious nonprofits that are not a congregation or denomination do apply for tax exemption and file annual reports with the IRS.3 This became important for my own research when I realized that such religious nonprofits had been pointed to by both social scientists and church leaders as having greater significance in the American Christian religious market.

In his book The Restructuring of American Religion, Robert Wuthnow (1988) devoted a chapter to what he called “special purpose groups.” He noted that “literally scores of such organizations have emerged in American religion since World War II.… Their causes range from nuclear arms control to liturgical renewal, from gender equality to cult surveillance, from healing ministries to evangelism” (100–101). Writing almost exactly twenty years later, the sociologist D. Michael Lindsay (2007) observed that evangelical Christianity’s “center of gravity today is not found at the level of the local congregation. Instead, evangelical business leaders, working with entrepreneurial ministers, have organized the movement around a constellation of parachurch organizations” (194). This label, “parachurch organization,” is the one that is often used among writers within the Christian community to describe these special-purpose, noncongregational organizations (White 1983; Willmer, Schmidt, and Smith 1998).

Although social scientists had written about the apparent growth in numbers and influence of such parachurch organizations, and individuals like Lindsay (2007) had collected some extensive interview and observational data, there really had not been much, if any, systematic quantitative analysis of these organizations. The primary obstacle appeared to be simply that no one had any way to produce data on these organizations. When I looked at the types of organizations being pointed to, though, it became clear that many of them were 501(c)(3) public charities. This meant that most of them would probably be registered with and, depending on the amount of revenue they generate, filing tax returns with the IRS. The question then becomes, how can I access those documents? This is where the growing availability of documents on the web comes into play.

It turns out that there are multiple websites where a person can search for and access digital copies of nonprofits’ tax returns, called 990s, for free. These websites include foundationcenter.org and www.guidestar.org. Much of a 990 form is financial information, such as total revenue, total expenses, breakdowns of different types of revenue and expenses, and so forth. If one were just interested in this financial information, then one could acquire a database of many financial variables for nonprofit organizations from the National Center for Charitable Statistics. A finances-only file, though, would not contain many of the rich measures that could be produced from such a document.

I utilized several criteria in determining which organizations’ documents I would extract measures from. I chose to focus on Christian organizations because the overwhelming amount of discussion on parachurch organizations was in a Christian context, although one could conceivably think of Muslim parachurch organizations. I also chose to focus on larger orga...