Slavery before Race

Europeans, Africans, and Indians at Long Island's Sylvester Manor Plantation, 1651-1884

Katherine Howlett Hayes

• Publisher: NYU Press
• Year: 2013
• Print ISBN: 9781479802227
• eBook ISBN: 9780814770900
• Topic: History
• Subtopic: North American History
• Index: History

The study of slavery in the Americas generally assumes a basic racial hierarchy: Africans or those of African descent are usually the slaves, and white people usually the slaveholders. In this unique interdisciplinary work of historical archaeology, anthropologist Katherine Hayes draws on years of fieldwork on Shelter Island's Sylvester Manor to demonstrate how racial identity was constructed and lived before plantation slavery was racialized by the legal codification of races. Using the historic Sylvester Manor Plantation site turned archaeological dig as a case study, Hayes draws on artifacts and extensive archival material to present a rare picture of northern slavery on one of the North's first plantations. The Manor was built in the mid-17th century by British settler Nathaniel Sylvester, whose family owned Shelter Island until the early 18th century and whose descendants still reside in the Manor House. There, as Hayes demonstrates, white settlers, enslaved Africans, and Native Americans worked side by side. While each group played distinct roles on the Manor and in the larger plantation economy of which Shelter Island was part, their close collaboration and cohabitation was essential for the Sylvester family's economic and political power in the Atlantic Northeast. Through the lens of social memory and forgetting, this study addresses the significance of Sylvester Manor's plantation history to American attitudes about diversity, Indian land politics, slavery and Jim Crow, in tension with idealized visions of white colonial community.

1 / Tracing a Racialized History

When Nathaniel Sylvester and his young bride Grissell came to reside on Shelter Island sometime around 1652 or 1653, they might have spoken between themselves about how they had landed in a lonely place, feeling that the two of them had only one another in this unfamiliar land. Writing to his business colleague, Connecticut Colony Governor John Winthrop Jr., Nathaniel commented about his marriage, “I find my selfe very happie and I hope in God wee may be a Confort unto Each Other [sic].” Their comfort likely came from a shared background, shared values, and a shared understanding of their place in society. In the new colonies and in independent settlements like this one, colonists had to adjust to not only a new environment but also a social order without precedent or tradition. This was partly what they had gone to the New World to achieve, but they perhaps had not anticipated the variety of people that they would also have to accommodate in their society, particularly American Indians and Africans.¹

Nathaniel, one of four partners who sought to supply their sugar plantations on Barbados, had settled here because of a business venture, but he was the only one of the four who would live on this 8,000-acre island. Located between the two forks of eastern Long Island, Shelter Island was also wedged between the Dutch West India Company settlement based in Manhattan to the west and the Puritan New England colonies which had begun settling on the east end of Long Island in the prior decade. Sylvester had encountered diversity in his life; born in Amsterdam of English Separatists, he had traveled widely throughout the Atlantic world in the 1640s as a merchant in the family business, reaching many areas of Europe, the west coast of Africa, the Caribbean, and Virginia, before settling on Shelter Island to attend to the planter side of the business. He had no love for the Puritans despite his Separatist upbringing, and his wife Grissell was the daughter of King Charles II’s auditor, in exile with the king following the monarchy’s defeat at the hands of the Puritan Cromwell. Thus, to be surrounded by Puritans, as well as the Dutch West India Company, to which he was no more than competition, was not necessarily a friendly neighborhood. Furthermore, the east end was only recently opened to settlement by the English. Indigenous Algonquian tribes there had previously been isolated from colonizer interference while under the protection of the Pequot, but the Pequot War with the English in 1637 had left the Paumanoc (Long Island) confederacy unhappily resigned to the necessity of dealing directly with the English.

The Sylvesters might have been lonely—far from family and a friendly community—but they certainly were not alone. Not only were there settler villages in Southampton, Easthampton, and Southold and the indigenous Montaukett, Shinnecock, and Corchaug in the lands across a short stretch of water from Shelter Island, but they also had their own retinue of servants and laborers quite literally in their back yard. Archived documents and recorded anecdotes suggest that some of these servants and laborers were indentured, and some were enslaved persons captured from Africa, all perhaps arriving at this place via Barbados. By the time of Nathaniel’s death in 1680, he claimed to own twenty-three persons, most explicitly identified as Negro, constituting one of the largest holdings of enslaved persons in the New York colony at that time and larger even than the family of eleven children his wife bore who survived to adulthood. In addition, archaeological excavations have shown that the indigenous Manhanset were part of the labor of the plantation, despite documents and anecdotes suggesting their departure at the establishment of the estate. Their skills, technologies, and crafts contributed greatly to the success of the plantation, though their status as laborers—enslaved, indentured, coerced, or free-- remains unknown. The work of the plantation required many hands: construct buildings, tend livestock, plant and harvest crops, load and unload ships as goods of many sorts passed through often to or from Barbados. Nathaniel Sylvester was there to manage this work, not to undertake it himself. To manage effectively he and his partners assembled a heterogeneous group of laborers—African, Native American, and possibly poor English or Irish. Unlike the inhabitants of the Puritan colonies, on Shelter Island Sylvester was not surrounded by like-minded individuals who shared a religious foundation or even a similar background; rather, his was a precarious community brought together, by coercion if necessary, to serve his interests. The nature of his relationship with this community is unknown, but it cannot have been one of ready identification, as demonstrated in his letters to John Winthrop Jr.; he wrote of his family, business transactions, and international affairs, but never of the people who were physically closest to his family.

In this historical context, Sylvester would have sought to establish control over his place in society by constructing categories of affiliation and difference among the diversity of people who surrounded him. Indeed, this process is at the very heart of colonialism itself (Foucault 1970; Thomas 1994; Pels 1997; Smith 1999; Spivak 1999; Stoler 2002). While religion and class were hotly debated in the seventeenth-century North American colonies, perhaps the most contentious long-term issue was race (Smedley 2007). The classification and justification of racial categories may have been variously framed by colonists and foreign investors in terms of civilization, spirituality, and eventually blood, but the effective result of the categories was control over labor and land, through enslavement and allotment. The discourse of racial difference evolved greatly over the next three hundred years. But in the earliest years of the Sylvester plantation, still prior to the enactment of laws defining race-based slavery, lines of difference and affiliation were inchoate, open to negotiation. Colonists viewed the enslaved and the indigenous as different, but how? Was it class, spiritual belief, physical difference? Or simply an economic need for labor? If it were only a need for labor, then why did they distinguish between African and Indian—both being viewed as potential laborers? Perhaps most difficult to answer, how did the enslaved and the indigenous view one another? The case of Sylvester Manor’s plantation provides a rare place to study not only how racial categorization emerged and evolved but also, by combining archival and archaeological perspectives, allows the possibility of interpreting multiple experiences, not just those of the white settlers. Furthermore, the structuring and representation of racialized relations at Sylvester Manor have been written and overwritten in so many ways that its history, like its landscape, is a palimpsest. In this rewriting, we can see the evolution of attitudes about race.

The case of Sylvester Manor’s plantation is also significant because it pushes back at two prevalent and popular misconceptions of American history: first, slavery was never a true institution of the northern colonies; and second, American Indians had no connection to the history of plantation slavery and indeed little connection to colonial society as a whole. The effect of these ideas has been to create spurious associations of racialized groups with distinct and separate geographies and temporalities. Thus, according to this historical logic, the roots of white American society lie in New England, the origins of black Americans develop in the southern colonies, and Indians, viewed as a race whose demise begins upon the arrival of settlers, are regarded as irrelevant to the history of either. Each of these misconceptions has been challenged by various historical studies, yet there remains a steadfast belief in the mutual exclusion of racial histories as a rule. Such beliefs have ongoing consequences to the way that race is still conceptualized in American society, namely the notion that racial groups, given their relatively isolated histories, have actually been stable categories until recently. Such an idea underwrites the assumption that race and culture are much the same and that both are generally static rather than actively, constantly renegotiated and reproduced (e.g., Smedley 2007; Berlin 2003; Forbes 1993). The notion of racially isolated histories is profoundly misguided, but when we start to look closely at historical records we see that the idea itself has been actively promoted for very particular reasons. These reasons relate to the struggles of both African Americans and Native Americans (self-identified) for their rights of self-determination and the efforts of colonial planters and later the U.S. government to control the labor and land of these peoples. In other words, our misconceptions of racial histories are not simply products of contemporary attitudes; such misconceptions were also deliberately fostered in the past, by colonial authorities, settlers, and government agencies.

To mitigate this misrepresentation, as well as understand its achievement through specific acts, this book explores the plantation of Sylvester Manor through multiple lines of evidence. Archival resources, the mainstay of historians, can include a multitude of different perspectives on racial categorization, from censuses, court documents, personal letters, account books, and even “ethnographic” descriptions of indigenous and enslaved peoples. These representations may be read for hints and suggestions that the predominant views of settlers on race and diversity were not universally accepted. These records are, however, the views of white settlers, and the active voices of American Indians and enslaved Africans are very rarely and never directly heard in them. Furthermore, the records of the archive are necessarily items that at some point were consciously selected for curation out of the universe of records created, and thus their mere inclusion must be queried for the historical value they represent as well as the perspectives they express. Archaeological remains—the material traces of past buildings, landscapes, daily practices of eating, dressing, labor, and leisure—may be thought of, however, as “records” not selected. The bulk of archaeological evidence is derived from excavating structures that have been reduced to rubble or decayed through abandonment as well as the mundane rarely recorded trash from day-to-day activities. Different preservation biases impact these materials, but more importantly the materials result from the efforts of past peoples to cover and discard things that they do not wish to remember. For the Sylvesters and their descendants, the materials excluded from memory included the very structure of the plantation itself, in which such a diversity of people lived and worked.

Thus in combining archival and archaeological evidence, we are often able to juxtapose the conspicuous silences in one (for example, the absence of reference to Manhanset laborers) with evidence of activities in the other. Historical archaeology, using both archival and archaeological remains, is an approach that allows for a critical movement between memory and forgetting, curation and abandonment, representation and embodied experience. This means that we can see not only what the physical and material setting of plantation life was like but also how the physical remains of the plantation have been used or discarded as evidence in the production of numerous historical narratives about the plantation over time. Just as the notion of race as a “natural” category has proven mutable, histories of the colonial period are unstable and contingent representations of the past. Indeed, historical narratives arise from the willingness or desire of communities to remember or forget; subsequent generations must in turn rely on the memory and evidence transmitted to them and their own wishes to remember or forget. Unearthing discarded evidence allows us not only to recover excluded histories but also to investigate how and possibly why they were excluded.²

Remembering race

How communities share memory, with whom they share memory, and how they protect or broadcast that knowledge is culturally and historically variable. Memory practices range from the incorporated or embodied reproduction in ritual or daily practice to the inscribed, textual or representational (Mills and Walker 2008; Van Dyke and Alcock 2003). Commemoration in contemporary Western societies tends toward materially weighty texts and depictions, meant to be evident to wide audiences and with an air of permanence by alienating history from its own history of production (Connerton 1989 and 2009). But memory may also be tied to more intimate groups, evoked in embodied associations, understood not as an independent record but as knowledge handled in careful circulation and dependent upon transmission from person to person. Ethnographic examples from Melanesia (Küchler 2002; Harrison 2004) and West Africa (Ferme 2001) have shown how communities may regard some memory as too dangerous to be permanently inscribed, or too widely distributed, because some associations must be forgotten. Contemporary valuation of heritage in our society gives the impression that we do not value or seek purposeful, active forgetting, but many scholars would argue that forgetting is critically constitutive of memory and heritage (e.g., Lowenthal 1996, Rowlands 1999, Shackel 2000), including racial heritage.

As the durability of memory lies in entangled networks of people, objects, and landscapes, forgetting is equally complex. In a plural society with multiple remembrances of the past—not all public or inscribed or even recognized as historical evidence—the process impacts both the ongoing construction of history and memories and the recognition of “subaltern” historicity. In archives, for example, Trouillot (1995) has shown the numerous moments where silences—forgetting through failures of transmission—may be introduced, in the creation of historical facts, archives, and narratives. Material memory may be effaced either by outright destruction or iconoclasm or more subtly by separation or exclusion of important associations (Forty 1999). Forgetting may be commanded, by offering amnesty (linked to amnesia) in exchange for abandoning certain historical ties (Ricouer 2004). The very course of modernity, rupturing associations with tradition and dispersing people from familiar places, has been a strain of forgetting on a massive social scale (González-Ruibal 2008; Connerton 2009; Spyer 2000). Each case introduces an active disruption of the associative basis of memory, sometimes replacing one historical narrative with another. Forgetting is made even more complex in its longer-term effects. Subsequent generations may not recognize that rupture or loss, despite the traces of the past haunting the archive or landscape. Yet conflict over historical perspectives—for example, when degrading representations underpinning racism are challenged--may drive descendants to reassemble those associations found in dusty archival shelves or buried under our feet.

Racialized identity emerges through similar performances of association and exclusion. Race is a distinct category of identity assigned to bodies or persons based on a perception of physically evident differences, attributed to some kind of embodied, essentialized, and thus ahistorical quality. Even in historical circumstances when the term race was not part of conventionally understood language, there may be evidence of discrimination attributed to some bodily or essentialized characteristic. The source of this personal characteristic has been defined in various ways, such as the lack of a Christian soul, capability for civilized behavior, intelligence, evolution. The process of creating those categories, primarily for the purpose of holding other persons or groups in a subordinate position (see Harrison 1995 and 1998; Frederickson 1988 and 2002), has been termed “racialization” or “racial formation” (Omi and Winant 1994, 55). Social scientists suggest that racial formation takes place in both representations of difference and social structure. This dual formation is critical for the historical evolution and institutionalization of race categories.

To merely label or represent others as belonging to a distinct group is insufficient to debase those persons if a social structure exists that allows them any freedom to act and to choose. But representations may be used in conjunction with social structures to develop long-term structural inequalities. Racial slavery, for example, elicited debasement through physically demanding labor, punishment and surveillance, malnutrition and the destruction of social supports. Such structures were expensive and difficult to morally defend simply as a means of acquiring labor. The inflicted debasement was thus attributed to the essential nature of the enslaved people, a representation used as a means of justifying their enslavement. Likewise in New England, Indians were subjected to land removal and immoderately heavy fines for small legal transgressions, often preventing their ability to make an adequate living, and in turn their poverty was attributed to their “primitive” and uncivilized nature. This tautology serves to mask the white populace’s true reasons for their marginalization, which were fear and the desire for more land. Both the structure and representation over time came to be viewed as “natural,” as the conditions of their construction are forgotten. Such racial associations have become so ingrained, so primordial in the imagination of white U.S. society that even now, when bodily or biological explanations are long rejected as a basis for racial differences, the categories continue to have pervasive effects. Critical race theorists point out that neoliberal efforts to neutralize racial representations by adopting “colorblind” policies ignore the historic social structure of racism (Delgado and Stefancic 2001). As philosopher Charles W. Mills noted (2008, 242), “erasure of the history of Jim Crow makes it possible to depict the playing field as historically level, so that current black poverty just proves black unwillingness to work. As individual memory is assisted through a larger social memory, so individual amnesia is then assisted by a larger collective amnesia.”

The investigation of racial representation has long been a subject for historians, whether in support or in challenge. Early colonists’ observations often explicitly referenced skin color and other physical attributes when discussing Indians or enslaved Africans (Vaughan 1995). Legal statutes, census records, estate records, and other forms of documentation provide rich evidence of how racial categories were represented and negotiated in colonial and early Republic America (Johnston 1929; Lauber 1913; Porter 1932; Woodson 1920 are early compilations of such sources). Forbes, for example, offered an exhaustive deconstruction of racial naming, such as “mulatto, pardo, colored, free colored, negro, zambo or sambo, mustee and mestizo” (Forbes 1993, 3), and indicated the multitude of racial terms colonial authorities created to police the borders of white society. He noted that the uncritical reading of those terms by contemporary historians has obscured the enmeshed histories of African American, Native American, and European peoples. More recently scholars have explored intermarriage between racialized groups by attending to how subsequent generations struggle to define themselves in wider social settings (Mandell 1998 and 2005; Campisi 1991; Barsh 2002; Miles 2002 and 2005; Parm 2005). These studies show that gender roles are deeply entangled with and disruptive of racial categories, not least because often intermarriage occurred by necessity, as selective importation of African men coincided with the decimation of indigenous men through disease and warfare. Whether born of necessity or not, intermarriage demonstrated that racial differences were not the basis of exclusion from these marginalized communities. However, a reactionary effort on the part of colonial and U.S. authorities to regain control of racial representation can be seen in the ever-expanding legal codes that clarified the status and categorization of the children of inter-marriage or later banned miscegenation (Gross 2008; Pascoe 2009).

Historians have also shown that those communities sometimes born of mutual support by oppressed groups were not always able to evade hegemonic racial representations. Native American and African American pluralistic communities, created through adoptions and marriages, were later subjected to racial categorization as part of nineteenth-century federal and state efforts at detribalization and privatization of communal Indian properties. These efforts ultimately targeted social structures, especially communal ownership of property, but used racial representations to instill dissent and conflict among community members (Man-dell 1998; Halliburton 1977; Saunt 2004 and 2005; Sweet 2003; Chang 2010). This complex history, quite evident in the Northeast, plays a significant role in tribal politics even today. Unexamined racial representations still structure the work of some contemporary historical scholars as we continue to presume the association or exclusion of racialized groups with particular objects, places, and times. For example, popular histories reproduce the “invisibility” of enslaved Africans in northern colonies (Melish 1998), of Indians in settings of modernity (Deloria 2006; O’Brien 2010), or of the pluralistic communities forged between them (Mathis and Weik 2005). In fact, their supposed invisibility has less to do with their presence or absence than with our expectations.

Social structure, however, is not only reproduced in discursive representation, but it is also lived in embodied and material realms. Archival records often do contain references to aspects of social structure, but the most habitual and naturalized aspects of daily life are rarely noted. In this arena, archaeological remains can contribute perspectives on the experience of racial formation. Archaeological work at plantation sites in southern United States and the Caribbean has not only well demonstrated the repressive structuring of space and daily activities, but it has also shown many acts of resistance and covert cultural maintenance on the part of the enslaved (e.g., Franklin 2001; McKee 1992; Fennell 2007; Wilkie and Farnsworth 2005). These illustrate social structure in planter/enslaved relations. Social relations among disenfranchised individuals and communities have also been addressed in other colonial settings outside plantations. Interethnic/interracial unions have been explored at the household level in Spanish missions and Russian colonial settlements, showing a complex restructuring of living spaces and cultural practices like foodways and bodily adornment (Deagan 1983; Lightfoot and Martinez 1997; Lightfoot et al. 1998). Communities of fugitives from enslavement (maroon or cimarron) were very often assisted by free indigenous persons, or fugitives were simply taken into indigenous communities. Where those places are documented, archaeologists have been able to investigate the material conditions and contexts of their interactions (Sayers et al. 2007; Funari 1999; Weik 1997 and 2009; Deagan and MacMahon 1995). The spatial structuring of the plantation at Sylvester Manor, where Manhanset and enslaved African laborers worked in close proximity but in spaces designed by the Sylvesters, demonstrates a complex negotiation of social order and tactical resistance between planters and labor. On a smaller spatial scale, certain material items may lend themselves to questions of shared production practices or reuse of goods. For...

Table of contents

• Cover Page
• Title Page
• Copyright Page
• Dedication
• Contents
• List of Figures and Table
• Acknowledgments
• Prologue
• 1 Tracing a Racialized History
• 2 Convergence
• 3 Building and Destroying
• 4 Objects of Interaction
• 5 Forgetting to Remember, Remembering to Forget
• 6 Unimagining Communities
• Epilogue
• Notes
• Bibliography
• Index
• About the Author