![]()
1.1 Introduction
We were meant to offer here a theory of modes and multimodality. However, what we offer instead is a theory of how sensation and experience in the world relate to what we call “designed experience.” Our theory covers much the same ground as some work on multimodality, but in a different way. The theory is meant also to constitute a model for the analysis of designed experience (“multimodality”). We will call our theory the “Zhangee Sensational Theory of Designed Experience” (and, yes, we mean “sensational” in both senses, though you may not think so when you finish reading the chapter).
1.2 Caring
Caring – affect – creates the human mind. It did so in humanity’s past and does so today (Damasio 1995, 1999; LeDoux 1998, 2019). When we have an experience, what we care about during that experience is what we pay attention to. And, what we pay attention to is what gets retained from the experience by our mind (Chabris and Simons 2010; Eagleman 2020).
What we care about is partly determined by our nature as a particular type of creature, a human being. Nearly all people care about some of the same things because they are human. For example, they nearly all care about what others think about them and where they stand in status hierarchies (Jackson 2019).
People who belong to the same social group or culture will care about some of the same things thanks to shared experiences, practices, and values in their groups or cultures (Anderson 1983; Douglas 1986; Lent 2017; Tomasello 2019).
And each of us cares about some things thanks to our own unique trajectory through different experiences in life and different social groups and the positions we have occupied in families, societies, and institutions. A single love affair or chance encounter can change what one cares about for a lifetime.
1.3 Mattering
One thing almost all humans care about, because it is biologically embedded in what it means to be human – we are, after all, social animals – is the need to belong, to feel they matter to others and what they do makes a difference to others (Marmot 2004; Pickett and Wilkinson 2011; Wilkinson and Pickett 2018).
This need for mattering is so strong in many humans that, if they cannot find a sense of mattering and belonging with a good group, they will find it with a bad one – for example, white supremacists or violent gangs. Either way, they will gain a sense of belonging that enhances their lives. The problem is that this enhancement is bought at the cost of other people being able to flourish and is a type of enhancement that might lead to their own death.
The groups with which a person finds a sense of mattering and belonging are a large part of what determines what they care about, what they pay attention to, and, thus, how they see and otherwise sense the world.
Most, but not all, people matter to their families. After childhood, they are exposed to a myriad of groups that they could join, if allowed to. And many people join and leave different groups across their lifetime and belong to a number of different ones at one and the same time. Of course, conflicts can arise, and we humans can behave quite differently in different groups.
1.4 Cultural Scripts
The groups that matter to us and to whom we matter partly create what we care about and attend to in the world. Humans evolved into a social species and a species that has made more of culture than any other. So, culture and the influence of social groups are as natural to us as are sensation and our need to eat (Douglas 1986; Jackson 2019; Tomasello 2019).
The groups to whom we matter and that then matter to us give us social scripts. Social scripts are ways of thinking and valuing that become taken for granted – operate on automatic pilot – by human groups that use them as markers and tests of identity and loyalty. They are part of what we will call below “fast associating.”
Some cultures have a script that pigs are dirty or taboo. This affects how they sense pigs at all different levels, as it affects how they interact with them.
Different cultures have different scripts (expectations) for how people will act out lust, love, anger, or sadness. In some social groups in the US and elsewhere, it is utterly taboo to hit anyone, in anger or not, especially in the face. In others, hitting, even in the face, is an expected part of anger, at least between men.
At one time in history, both speaking to god and hearing god speak back were considered normal. But we now have a saying that says, “If you speak to god, you are praying; if he speaks back to you, you are crazy.” If Moses’s culture had told him that his sense that god was talking to him was not a sign of grace but of mental illness, three major world religions would never have arisen. This may have been for the good or not, but we are still living in the history and cultures he helped make.
1.5 Sensation
Humans sense the world as they do because of the senses they have. Other animals have different senses and sense different things (Birkhead 2012). No creature senses “the world”, the whole of it. They each sense “their world”, the aspects of the world that are available to their senses (Eagleman 2020).
When humans sense things, they sense them as present before them. They sense them as personal, subjective, immediate, and private. I cannot know what things look, smell, taste, and feel like to you, and you cannot know how they look, smell, taste, and feel to me.
A person born blind from birth cannot know what color is at the level of appearance and feeling. And the sighted cannot know what it feels like to sense using echoes, like a bat, but some blind people do, since they get this ability when they go blind.
If you have a passion for birds and walk in a park with someone who has no interest in birds, as you revel in the birds all around, the other person barely sees them. And, if they do, they do not register strongly enough to be remembered other than as “birds”. For you, each bird is both a vibrant being as an individual and a hallmark of a species with a noble history.
Birds exist in “your world” in a very different way than they do for your walking companion. Birds are a vibrant part of your experience, major players in your world. Why these differences? It all has to do with caring. When we experience the world, we do not just get sensations at random; we get different ones depending on what we care about.
What you care about directs your attention. It makes some sensations vivid, memorable, important in a very personal and intimate way and makes others passing moments or non-existent altogether. We all fail to see what is right in front of us when we do not care about it. In fact, the way the brain works is that we see it and forget it immediately. It is not important to us.
The power of caring goes further. It shapes our imagination, the power through which we build out futures. Consider this example: one day, when a friend of ours was out hiking, a very small young skunk was on the trail ahead of her.
The little skunk pranced in front of her as she followed behind him; every once in a while, he turned and stood up and looked at her. He tried to fluff himself up to look bigger and intimidating, though he was only a baby.
She thought perhaps she was scaring the little skunk, so she moved off his trail to another one. But he quickly moved forward and crossed over to her trail to get ahead of her so he could continue to turn around and look at her and keep trying to intimidate her with his fake size. Perhaps he was not scared. Perhaps he was playing as a form of practice. Or, perhaps, both of them – human and skunk – were just playing to play.
When our friend walked on that same trail again on another day, she was perfectly able to see and experience and get joy from the little skunk. He was right there in all his beauty, wonder, and cuteness because she was able to use her imagination to bring him back. She was augmenting reality, making her own augmented reality, as can any human. She can also readily bring back the original hike and the skunk, not augmented reality now, but a virtual reality she is generating entirely herself.
What you have cared about in your experiences in the world is what fuels your imagination. It is the material which you use to build virtual worlds in your head. You can imagine things you have never experienced, but only by mixing and matching elements of experiences you have had, either in reality or through media. And the ones you have are a product of how you cared and attended in the experience.
1.6 Affordances
All humans – indeed, all animals – are action oriented. Creatures, including humans, sense the world in order to do what they have to do to survive and flourish. So, they sense things in terms of their possible uses, what they are good for, how they can facilitate actions to accomplish goals.
Gibson (1977, 1979; see also Barsalou 1999; Glenberg and Gallese 2012) years ago made this point in terms of what he called affordances and effectivities. An affordance is anything a thing can be effectively used for, if a creature has the ability to engage in that sort of use (an effectivity). A rock is good for throwing as a weapon, but only if you are a creature that can throw. A hammer has been designed by humans to have affordances for hammering nails into wood and taking them out of wood, but you must have an opposable thumb to make use of these affordances. So, what is an affordance for one creature may not be for another.
Our human senses are set up to recognize affordances in the objects and processes around us. Sensing the affordances of things, as well as designing tools with specific affordances, has been crucial to human survival.
Humans do not see all the affordances a thing may have. Some are hidden because they afford uses humans do not yet know about. A good way to see this is with video games. If you play a game like SWAT 4, where you command a small SWAT team, after you have played for some hours you begin to see the world outside the game in a new way.
SWAT teams have to be continuously vigilant as they enter buildings. They can be attacked from any direction. They dare not leave a door open behind them – and they have a rubber door stop to stop it from being opened. They must enter rooms unseen or fully behind cover; they cannot shoot any hostages or other innocent people; and they must, if they can, warn a bad guy to put down his gun before sho...
