This is a collection of essays in honor of Philip Johnson-Laird, one of the founding figures in mental models. These essays represent application of his theory to several particular domains.

George Miller: Contextuality

This essay is about handling words and stituations with multiple meanings. The process of figuring out these meanings is contextualization, described as a basic cognitive process. This is closely related to Goffman’s frame analysis. The goal of contextualization is to resolve ambiguity that is heavily present in interpretation of everyday language and knowledge.

Computational linguistics is a tricky area in cognitive science and computation. It is deeply affected by the issue of context. Miller’s analysis focuses on linguistics exclusively (mirroring Johnson-Laird), as opposed to other sorts of ambiguous circumstances. Computational linguistics involves processing language and attempting to identify and process the correct word meanings from that language. Miller mentions Bar-Hillel (1960), who finds that this sort of language processing can identify correct meanings about 80% of the time. He estimates that this last bit could never be achieved without significant advances in AI.

Expert systems, which are the general approach for working with specialized knowledge, limit the domain of word meanings to a significant degree, but this still does not absolve the “Curse of Bar-Hillel.” Miller theorizes that context identification is the key to unlocking this last bit of meaning.

An aside to note is that Miller is a collaborator on WordNet.

Alan Garnham: The Other Side of Mental Models: Theories of Language Comprehension

This essay looks at language comprehension by examining issues of reference and inference. One key element to inference and communication is instantiation. Where an abstract idea is replaced by a more concrete (or other known) one. However, Garnham is concerned with the communication of abstracts, and notes that we communicate information about abstracts without instantiation.

Propositional relations are a strategy used frequently in AI for world modeling, and relate to information as discrete facts. Garnham gives an example which uses locational prepositions, things of the form: “The lamp is in front of the candle,” etcetera. In terms of these locational structure here, it seems dubious. There is a suggestion that mental models use a more analog depiction and representation of spatial relations.

It is true that we do use abstracts in communication, but models of communication that have emerged from Vygotsky indicate that communication emerges in development when social interaction transforms from something embodied and physical to something symbolic. If we follow Lakoff and Johson, then relations are all metaphorical and based ultimately in the body.

Paolo Legrenzi and Vittorio Girotto: Mental Models in Reasoning and Decision-making Processes

This essay discusses decision making according to psychological studies, and explained in terms of mental models. The interesting thing here is that totally rational decision making is not present, rather, decision making is based on the matter of focusing. This sounds a lot like priming and activation (related to neural networks). Models illustrate the construction of ideas, but neglect to factor how the focusing works intrinsically.

David Green: Models, Arguments, and Decisions

Green builds a theory of decisions (as derived from Craik, 1943) based on argument. Argument is done through warrants, which are bits of relevant information.This work is built from Toulmin’s scheme. Further, the goal here is to analyze argument through mental models. The interplay between observation and model mirrors warrant and argument. Warrants also relate to beliefs, which may be connectable to the belief, desire, and intention scheme in AI. We can also apply warrants to causal models.

The final conclusion in this section is that there is an interplay between argument and simulation, as well as decisions and commitment.

Keith Oatley: Emotions, Rationality, and Informal Reasoning.

Oatley’s focus here is on informal reasoning as it relates to emotions. Informal is opposed to logical or day-to-day. Oatley argues that emotion is critical to this sort of everyday conventional reasoning.

He opens with an analysis of Aristotle’s rhetoric, which discusses two types of reason. There is absolute mathematical reasoning, and also persuaded reasoning, where there is no demonstrable truth. Persuasion instead aims to achieve the best truth possible. The interesting example with this is that Aristotle’s writing is based on the sort of rhetoric used in his day, where law is extremely dependent on performance and dramatic emotional appeal.

Oatley makes some notes on Aristotle’s Rhetoric: The first is that persuasion applies to the imperfectly knowable, and the field of the imperfectly knowable is huge. AI, on the other hand, seeks to only understand that which is perfectly knowable, or that which can be logically concluded or deduced. The second point is that Aristotle explains emotion as a tool of judgement, as opposed to something that is bestial or irrational. Furthermore, there are three impediments to making rational decisions:

1. Limited knowledge and resources. Our mental models are incomplete to fully predict the effects of our actions.
2. Multiple goals. Multiple goals cannot always all be satisfied rationally.
3. Distributed agency. Actions are performed in relation to others, planning must occur among multiple agents, where the problem of limited knowledge becomes especially difficult.

Rationality depends on environment and context. Emotion is used as a form of feedback for goals. Oatley describes emotion as a heuristic function for potential actions.

Oatley makes a connection to Vygotsky and Hutchins. Emotions play a role in the distribution and extension of cognition. There is a connection between Aristotle and the Roman historian Quintillian, who documents the practice of law in the Roman court. This is an argument for the theatricality of reason, relating to the ideas of performance. The performance of law is an enactment and exaggeration of events. The social nature of the audience is essential.

Oatley follows this with the analysis of two experiments, where individuals change behavior based on emotional priming. Emotional induction proved to be immensely relevant in both examples. One of which consisted of examining decision making in judgement of evidence of a trial (after having viewed a happy or a sad film clip), and the other examined forward or reverse reasoning (after reading an angry or sad short story). The experimental corrolation was immensely strong in both examples.

Ciuliano Geminiani, Antonella Carassa, Bruno Bara: Causality by Contact

This essay is about the role of causality in reasoning. Causality is related to the construction of scientific models, but also is relevant from the perspective of narrative. Using causality implies the use of simulation mentally. Causality has an evolutionary basis that is associative (for instance, a rat who smells a type of food on a dead rat will not eat that type of food). This associative logic is also imaginably present in humans, but humans also do use causal reasoning, which comes with the demand for knowing why something occurs. This connects well to Vygotsky and development. The why relates to the narrative/linguistic model of thought.

An interesting note: In developmental study, causality is dependent on contact. Touching is necessary for causality to be interpreted by infants. Gradually, though, causality becomes analogically based. Causal models are a subset of dynamic models. The authors give a funny example of two narrative segments: “Cleopatra was bitten by an asp, Cleopatra died” versus “Cleopatra was bitten by an asp, a tourniquet was applied to her arm, Cleopatra was saved.” This example is a little strange, but is used to understand how people might model what happens to the poison. Mental imagery and metaphors are especially important: poison is a particle, poison is like paint, etc. The important thing to note here is that the example is fundamentally a narrative one.

To understand how models are formed and used, the authors give a three part theory for development of causal models: Construction, comparison, falsification. The construction phase involves taking the components (as a pre-model) and understanding them quantitatively. This is literally formulated as collecting symbols and describing them qualitatively. Next, qualities are quantified, fixing values and times. Finally, the model is simulated dynamically at a sub-cognitive level. The sub-cognitive simulation involves 1) activation of implicit knowledge, 2) generation of instantaneous changes in quantities according to the simulation, and 3) simulation of the temporal evolution of the model.

At the comparison phase, the effects of the mental model with the base model are compared. In this context, the base model is imaginably the observed phenomenon, which is the original story. This comparison intiates revisitations and inferences. Finally, in the falsification phase, plausibility and counterexamples are considered. This sort of analysis derives from Qualitative Process Theory (Forbus 1984), which seems like a good place to check the connection between narrative and models.

This approach is useful in looking at models of fiction as pertains to adaptation, especially in terms of emotional value and responses.

This was written as a problem formulation for one of my courses with Nancy Nersessian. I thought it was handy and relevant, so decided to put it up.

To open, let us consider an argument made by Keith Oatley (in “The Science of Fiction,” New Scientist, 2008): “Fiction is a simulation that runs on the software of our minds.” Oatley’s argument is based on studies of readers of fiction who demonstrate better emotional interpretation skills after reading fiction. The interpretation that he gives is that readers mentally predict the emotions of the fictional characters, and can compare the prediction with the result that actually plays out. This can be seen as either modeling and conceptual change, or a sort of psychological activation of emotional parts of the brain. It is hard to tell for certain, but let this contextualize what follows:

My personal research is in the adaptation of fiction though simulation of the fictional characters and worlds. When the characters themselves are embedded in an electronic simulation, the problem immediately rises of how to develop emotional responses of the characters. Previous work in simulating characters (Perlin, Crawford, and many others) have given varying approaches to modeling characters and communicating states. Game studies (from which my work is emerging) has borrowed significantly from the theories of cinema and theatre for how to represent appealing and believable characters. Much of what has emerged from this is centered on the issue of providing more visible, clearer characters. A great deal of attention is paid to faces (and the movement from text, to image, to animation), which are, admittedly important. Everything boils down to dramatic presentation, which comes from drama, which may be embodied and visceral, but is not an interactive medium.

A simulated fictional world, in which the player is a participant, must look beyond interaction, though. The player must be able to communicate with other agents, for the fictional experience to be enacted. Many times in games, players are in opposition to the game world and its designers, a situation that does not satisfy the potential for emotional relevance. Despite the significant advances in character depiction in many contemporary games, this issue remains a problem. Bonnie Nardi does not give a solution, but provides a way to understand the failure of graphical technology in developing emotional experiences.

In Bonnie Nardi’s ethnography, she studies the methods of communications of business employees, investigating specifically how they use mediated communication. She finds that there are three dimensions to communication: affinity, commitment, and attention. These are explained to make sense within the business model, but are all grounded in sociology and anthropology. Expressions of each of these elements can be seen in every form of communication, and they seem almost tribal. Clearly, there is some deep importance to these channels. Individuals make use of the affordances of mediated communication to work with these elements, but the elements are still deeply embodied. The reliance on information bandwidth is insufficient to operate on these channels.

With Nardi’s Beyond Bandwidth in mind, developments in technology in games are clearly matters of attempting to increase bandwidth. For an interactive experience with fictional characters to succeed, communication must be necessary, and this depends on the relationship of the player to other characters. This does explain why emotional experiences in games work independently of technology. Nardi’s dimensions are a starting point, and open a suite of complex issues: Mediated communication has affordances for the elements of communication, but what are those affordances in games and simulation? How can affinity, commitment, and attention be represented in an artificial world?

This paper is on computer mediated communication. The abstract presents the paper as a critique of some standard approaches, which emphasize the role of bandwidth as a means for understanding different different means of mediated communication. Instead, the subject should be the relationship between the communicators, and the focus on how the mediated communication affects that relationship. Nardi proposes a model that uses three fields of connection: affinity, commitment, and attention. These fields form the dimensions of a communication space, wherein the values of each change and evolve over the course of communication.

Nardi reviews some existing theory on computer mediated communication. These are nuanced, but all fall under the category of exploring bandwidth: Media richness theory (Daft and Lengel 1984), Social presence theory (Short et al. 1976), and others. All these are about understanding communication in context of more objective information about the communicators.

Nardi’s analysis instead looks at an ethnography of instant messaging in the workplace. She finds that what is important is not the bandwidth and or richness of information, but rather communication is about the feelings of connection and the sense of openness in interaction. These feelings were established by bodily interactions and “informal discourse of low information content.”

In a study of instant messaging, Nardi finds that well regarded executives perform quick and informal “purposeless” communication with colleagues through IM. Nonetheless, this informal communication does serve a function of staying connected. Activities of connection occur through computer mediated communication, but simply through different channels than in regular conversation (which has a lot of embodied and visual signals). This idea is interesting because it puts a mediated spin on the sociology of interaction.

In reviewing social presence theory, Nardi introduces a number of concepts from sociology. Particularly relevant to mediated communication is the idea of symbolic interaction, which lends the insight that different media may be chosen to communicate different things based on the social role of the medium. Interaction relies on cueing, and different media have varying affordances for cues.

Nardi’s analysis of communication is broken down into three dimensions: Affinity, commitment, and attention.

Affinity is a feeling of connection, and the degree of openness in interacting with another. Affinity may be derived from social bonding activities. The examples of social al bonding that Nardi provides are: Touch, eating and drinking, sharing experience in a common space, informal conversation. Bonding is heavily embodied. These experiences are also tightly connected to not only sociological traditions, but also anthropological ones.

As an aside, the idea of bonding might be an interesting thing to use to develop emotional significance with artificial agents. A bit of work has been done looking into the emotional appeal of games and game characters (which is especially significant in the cases where tragedy strikes), and this might be useful in other cases. It may also be a good frame to analyze the Sims.

The expression of commitment is important. Nardi explores some interviews and finds that commitment is very much about establishing a bodily presence. What is important is making some sort of visible expression to indicate just how committed the actor is. Commitment is about performance and ritual. In some cases, it is related to expenditure, but bodily presence is especially valuable. For example, flying out a long distance to meet clients for a day. Commitment can probably be compared to issues of investment and personal sacrifice. Again, this is a common practice that has deep roots in anthropology.

Procuring attention is about capturing the focus of a subject. In personal interaction, attention has a lot to do with eye contact. The gaze is another heavily embodied element of communication, and cannot adapt well to the bandwith model of communication. Attention is also about conveying availability for interaction.

One curious thing about Nardi’s analysis is that her experimental subjects are all modern professionals, but each of the elements of study are overwhelmingly anchored in anthropology. The effects of the examples of highly paid business executives seem right at home next to the effects of tribal rituals.

Using these three elements as essential parts of communication, Nardi explains that some of these elements can be carried over into mediated communication, but their operation is different. Mediated communication enables the elements of contact, but in a subtle and definitively weaker manner than in full face-to-face communication. Nardi’s point can be seen that improved bandwidth can not improve communication. Rather, communication might be improved by focus on the elements of connection and relationships.

Opens by outlining the general failure of theoretical psychology to apply to HCI. HCI is generally developed from practical knowledge. The problem is especially related to the psychological theory of information processing, deriving from cognitive science. Kuutti describes a movement in HCI that reacts against the symbolic/Cartesian model.

He examines Liam Bannon, who presented a critique of Human Factors (as likening humans to components to factor into a human-machine system), and further criticizes HCI as exploring only inexperienced users. This critique relates back to the idea of everyday life and practice. Experienced users make use of emergent practices that may augment or subvert the original model.

HCI is composed of three levels. The first is oriented toward ergonomics, perception and motor skills. The second is concerned with information processing and conceptual psychology. The last is an emerging level that addresses deeper complexity, and is the target for applying activity theory.

As background, activity theory originates in Kant and Hegel. This German philosophy developed against objective empiricism, the idea that meaning is external and may be discovered through experience, and replaced it with constructivism, where meaning is actively constructed. This idea was later adopted by Marx and Engels who applied construction and activity to political ends. Activity was finally developed by Vygotsky and his school of psychology. Kuutti suggests that Mead’s symbolic interaction (which led to Goffman) also followed a similar vein.

An activity is the basic unit of analysis. The idea is to define an activity as a subset of actions that has a minimal meaningful context. Activities are mediated, and are also permeable and flexible structures. It also provides an interesting approach to tool use. Tools are media that enable activities, but they limit the user in things that do not belong to that activity. This is especially important if we consider tools or mediators to be more than physical objects. Mediation can be through environment or other contexts. In this sense, the limiting nature of tools reflects pattern matching in phenomenology as well as the notion of keying in sociology.

Activity also works with the idea of a community formed around activities and the mediating object. This establishes the notion of a collaborative activity, where members may have different roles. Kuutti calls the division between these roles a division of labor, but I would argue that activity is not dependent on labor.

Kuutti does relate activity to planning, but as a step that is about modeling, not outlining the action. The initial phase of an activity is orientation, during which the agent models the world into consciousness. The execution of an activity is composed of “fluent” engagement with the world, which is composed of operations. This model allows for a hierarchical model of activity that derives from learning and practice. For a novice learning how to drive a car, the process of changing the gearbox is a highly conscious activity that requires a lot of attention to the individual elements of the task, but later on, these operations become fluent elements and moving the gearbox is merely an action in the larger activity of driving. Kuuti does not examine this hierarchy, though, and seems to assert that activity only happens at this higher level.

Activity relates to HCI by looking at how people go about using computers for activities. This exposes some of the ways in which traditional information processing psychology has failed to help HCI. Notably, it addresses the issues of complexity and use and practice.

Notes:

The authors set out to bridge what they consider to be a false division between intelligence and technology. The idea of technology is something that is artificial, and usually electronic. Intelligence is seen as something that enables reason and planning, and is a biological property of individuals. The authors’ notion of technology is a tool-mediated social practice, which enables a very different model of intelligence. In this context, intelligence is “a process of adaptation to, and transformation of, the conditions of life.” (p. 2)

Artifacts are both ideal and material. This idea derives from Dewey, and is supported by constructivist philosophy. This unity is also something that exists in cognition: material and symbolic are tied together in thought, departing from Cartesian dualism. The authors outline levels of artifacts as identified by Wartofsky, which move artifacts from primarily material to primarily symbolic.

The authors continue by looking at how artifacts augment cognition, and some of which are explicitly psychological tools. This notion comes from Vygotsky, but the idea that tools are used to augment understanding extend back from Francis Bacon. The cognitive prosthetic continues through Norman (and also through other figures in early computer science, notably Vannevar Bush and Norbert Weiner). Norman’s principles are bulletted here: (p. 6)

• A representation is a set of symbols that substitutes for the real event.
• Once we have ideas represented by representations, the physical world is no longer relevant.
• Representations are abstractions so good representations are those which abstract the essential elements of the event.
• The critical trick is to get the abstractions right, to represent the important aspects and not the unimportant. This allows everyone to concentrate upon the essentials without distraction from irrelevancies.
• Representations are important because they allow us to work with events and things absent in space and time, or for that matter, events and things that never existed — imaginary objects and concepts.
• A person is a system with an active, internal representation.

This is an interesting approach with which to analyze representation. Norman uses this but focuses exclusively on the ideal or conceptual level and denies the material element in the cognitive. Further, the authors criticize Norman’s neglecting of the environment and broader social context surrounding the use of artifacts.

The negotiation between intelligence and technology leads to a reconsideration of the role of culture in cognition. When intelligence and culture are bridged, artifacts become material and conceptual aids to cognition. The idea of culture as a large pool of knowledge is heavily challenged. The authors cite Geertz as circumventing the ideal/material dichotomy, using semiotics as a means to embed culture in material artifacts.

The authors continue exploring Geertz, specifically his theory that the nervous system requires culture in order for human development. This argument is followed by Quartz and Sejnowski, who encourage the neurological aspect of culture, and the extension of intelligence and cognition into the environment.

To bridge the connection between technology and intelligence through culture, the authors give the example of the use of the Abacus in Japan. The tool was delivered culturally and developed a cultural role and function. Additionally, there is a cultural intelligence associated with it that becomes evident in highly skilled abacus users.