This past Tuesday, Audrey and I went over to East Atlanta, to the Earl for a nice indie music concert. We knew we were in for something, but really weren’t quite prepared for it. The actual concert started at 9:30 and the doors opened at 9:00. Audrey unfortunately had a class that normally gets out at 8. That’s right, 8 PM. We graduate students just live it up, don’t we? Fortunately, she managed to get out a bit early, and we scooted over there and tried to find dinner.

Fortunately, and we noticed this once we started wandering around, the Earl is also a restaurant. They had a wide variety of menu items that seemed to cater very precisely to the our demographic: the pretentious indie music crowd. I mean this endearingly! I had a guacamole burger, which was outstanding. It was just very amusing. We sat next to a couple of guys who were going to do some photography and conduct an interview with M83 frontman Anthony Gonzalez.

The show itself opened with School of Seven Bells, which I had heard about a few days ago. Their music is sort of an ethereal and spacey, with an unusual strong sense of rhythm. Their music has a somewhat otherworldly quality to it, sounding like a soundtrack that scores a journey to some lush but remote and isolated destination. I heard about them a week before the concert, and then found out that they were opening for M83, which was a pretty nice coincidence. There are three members to the band. The vocalists are sisters and occupy guitar and keyboard. There is also another guitarist who spent a lot of time rocking out. The singers layered together creating a nice resonance, that created not just music, but really something more like a textured soundscape.

When School of Seven Bells finished, they dismantled their set, and the stage crew began setting up for M83. They brought up a large quantity of fascinating and delightful looking electronic equipment. The setup took quite a while, but eventually the band itself came on stage.

M83 opened with an extended version of Run Into Flowers. The case with this, and with all of the rest of the tracks they played, was that the original song was not a script so much as a guideline. No song really was close at all to the album versions. Because M83 exists in a complex hybrid between pure electronic music and Explosions In The Sky-style instrumental music, the actual set combined elements of performance of electronic music with more traditional stage performance. Guitars were layered on top of each other to create a texture, and the actual electronic elements, coming from keyboards as well as Gonzales’ laptop, came together to sound like they were being mixed live before us.

Each song was layered with a great deal of complexity. The guitars were primarily responsible for creating a sense of texture, and came together with such resonance that they faded into the background. On top of those, the keyboards and vocals gave shape to the auditory space. On more than one occasion, when the leading instruments stopped or subsided, it sounded like silence, until we gradually realized that the music was still there, that it had never left us. When the instruments picked up again, they had the feeling not of individual instruments constructing a whole, but of waves that washed over us.

The band consisted of Anthony Gonzalez, the mastermind behind M83, who alternated between keyboard and guitar. There was also another guitarist and a drummer, who hammered away behind his drum cage. However, there was also another keyboardist, Morgan Kibby, who contributed to vocals as well. The word “contributed” doesn’t really begin to describe it, though. Kibby was a ferocious bundle of energy. She sang a lot of the tracks from saturdays=youth, and pounded away at those keyboards like she was possessed. Both she and Gonzalez were brimming with enthusiasm and energy. It was impossible to see them without having that energy rub off.

I’ll leave you with an emblematic video of a live performance of Couleurs, though the sound quality doesn’t remotely do it justice.

I’ve become totally addicted to this game. It’s made by Tilted Mill, which is the wonderful studio responsible for Pharaoh, Caesar, Sim City Societies, and many other delightful city building games. Nile Online is very fascinating conceptually. It is a broswer-based casual MMOG. Contrary to many browser-based MMOGs, it is not implemented using Java or Flash, but rather PHP delivering dynamic HTML with Ajax. It is casual because player actions are implemented over time. Creating a building in the beginning may take 15 minutes, but later on, upgrading it to a higher level may take 6 hours.

The primary mechanic of the game is trade. Players can trade with each other, but much of the trading is unchecked, so it relies on trust and communication (via an in-game email/scroll system). One could probably say that it is about economies, but the way that the economy is implemented in game, it relies on issues of time and distance that make it unlike many contemporary economic games which are much more instantaneous. It’s a lot of fun.

Meanwhile: When did Open Office 3 come out? Sun really needs to figure out how to cultivate popular enthusiasm and support! Maybe I’ll give some sort of review later after I get it installed. It’s a great project, but needs more publicity for it to get recognized. I don’t want them to advertise. I hate advertisements passionately. But they could see about getting their product reviewed on blogs or on tech news sites.

Renowned media and culture scholar (and blogger) Henry Jenkins visited us this past week. He gave a lively presentation on media technology as used in the election campaign, and later met up with several of the research groups. I stuck around the group meeting with Janet’s narrative schema group where we looked at some of Sergio Goldenberg‘s eTV projects, as well as Hartmut Koenitz’s Advanced Stories Group. I was at this last meeting, but was not especially conversational, as one, I don’t have a huge amount to say about the projects, and two, I did not have much completed work of my own to show. Nonetheless, there were a few very interesting bits that I picked up from both events, and I’ll try to convey them here.

Politics and Media

I’ve studiously avoided discussing politics here. This is not to say that I don’t have my own strong beliefs, but I find nearly all political discussions to be exhausting and ultimately futile. Nonetheless, there are a number of very interesting and noteworthy things discussed in Jenkins’ talk about media in the election, and in political communication in general. So, I’ll try to review what I can.

The talk opened with a review of how media has been used in previous elections. The Lincoln-Douglas debate is an example which is frequently used as a non-mediated political event. The historical debate was dry, logical, and textual. However, the situation of the debate was still at a carnival. Bands played, there were sideshow performances and greased pig chasing contests. Spectacle was still very much an issue. Neil Postman claimed that the spectacular nature of television could never match the rationality of the Lincoln-Douglas debate, but this claim misses a complex relationship between politics and media.

Media has been used by all candidates in recent past, and the skilled use of media has tended to make for successful elections. Examples are FDR and radio, Nixon and Kennedy over television, Reagan and dramatic iconography, Clinton with cable, and finally Obama and the internet. This election is marked by a convergence of media. Obama’s skilled use at weaving many kinds of media together to create a coherent and consistent message is in part responsible for his overwhelming success.

The pivotal moment to Jenkins is the CNN-YouTube debate. This is revolutionary and pivotal because it marks a return of control from an institutionalized system (television or town-hall debates) to popular control. The clash between participatory (meaning user/popular controlled) culture with the authority of mass media. There were two sets of controversies that came out of this. The first was the reaction against the legitimacy of the YouTube questions, and the second was a reaction against the authority of CNN to filter and screen the questions in the first place.

The issue of legitimacy takes a complex spin when examined carefully. one of Jenkins’ examples is a snowman, asking about global warming. The example was decried somewhat as illegitimate, because it is ostensibly not serious. Literally, the snowman is fictional, and it is at odds for a candidate to be faced with a question from a fictional character. However, a snowman asking about global warming is still apt metaphorically. Furthermore, there is a deeper thread: The snowman speaks with a squeaky voice that is reminiscent of Mr Bill from Saturday Night Live. Mr Bill was, in turn, a “user-created” tape sent to SNL. This example thus taps into a somewhat deeper set of meanings than may first appear. User created questions work beyond the questions themselves, but also pull the weight of a larger set of cultural meanings.

The YouTube debates are one example in which users were able to “talk back” to the candidates and the media, but there are also other cases. One example is the “3-AM girl” who was in some stock footage used in an advertisement by the Clinton campaign, but then was able to post a response saying that she supported Obama. All of these cases are ways in which users have taken control of media and used the internet to talk back, taking control away from the usual media authorities.

In turn, existing media still does not go away, but rather takes on a new relationship to the internet and other media sources. Television broadcasts something, and this is reacted to on the internet, and is in turn broadcast out by television. There is a feedback cycle between blogs, YouTube, cable television, and national news and television.

Other Tidbits

On reality television: This has always had to do with ethics. While reality TV always focuses on conflict, or strives to create conflict, the conflict is ultimately about ethics. Characters have different ethics, and observers project their own values onto them. Jenkins explains that this is how some theorists have come to understand gossip: people project values onto characters, and talk about those characters to communicate their ethical values. In a sense, reality television is used by audiences to discuss their own ethics indirectly.

We were still discussing the eTV projects, in terms of exploring fictional worlds, specifically as relates to convergence, and an interesting point came up. Convergence is the process of revealing a fictional world through many different kinds of media. An example of how this works is when a franchise (for example, The Matrix) splits off into several media forms (beyond films: an animated series and some games), and while watching or consuming one form of media, there is a reference to something that occurs in the others. This sort of connecting process is an active function of the viewer, and helps build a better sense and knowledge of the fictional world. Jenkins called the sort of pleasure that results from this “epistemophilia,” which is a pleasure of knowing and connecting. Epistemophilia is associated with puzzles and transmedia works.

Epistemophilia is also conflicted with a different pleasure, the pleasure of immersion, even though the two are often confused. Immersion is the sense of being in a world and experiencing it viscerally. (Maybe I should call it “ontophilia”). Immersion is at odds with epistemophilia, which can be seen as a type of “spoiler” that undermines the sense of being. Immersion is normally associated with transparency and immediacy. To be in a fictional world, the media that exists between the user and the world must be overcome. On the other hand, with epistemophilia, the media is necessary. An epistemophilic desire is to take advantage of a medium in order to understand and piece together the world as an external observer. This can be done for instance with freeze frames in DVDs (to catch some subtle and impossible clue). Both of these desires relate to the relationship between the user, the medium, and the world.

Later, while discussing the advanced stories system, Jenkins warned us about the mechanics of fan fiction and interactive story systems. Character drives fan fiction. In this case, characters are used projectively: to explore values and idealizations. Branching narratives do not deal well with the complex matter of character motivation, which is what drives fan fiction. There could be branching narration (not narrative), which explores different perspectives. The culture of writing is compelled by character psychology. In rich environments and settings, the world is a character.

This last bit was somewhat troubling to me, though. My work is focused on adaptation of fiction, and is looking at Jane Austen specifically. Austen has a huge culture of recreation, and adaptation, much of which reads very much like fan fiction. However: my approach and focus has been on recreating the social world and model, rather than capturing the characters exactly. I could make the argument that real motivation is impossible without some social or cultural model (that expresses values), but I do not attempt to express the deep complexity of character. At least not yet, the essences and complexities of character are extremely hard to formalize.

Nonetheless, I did speak to him for a few minutes afterwards, in which I hurriedly (and possibly incoherently) explained my ideas, and he seemed to give me an endorsement, so that is a positive sign. Hooray!

This book is about a shift in the study of cognition into the idea of practice. Lave’s book is the construction of a complex argument to free the understanding of cognition from the limits imposed by traditional methods of study. As far as this goes, Lave asserts that the use of the laboratory and theory impose certain values into the study of cognition, classifying some uses of cognition as superior to others. Additionally, these approaches also yield incorrect results. For example, the first part of the book challenges the idea of learning transfer, which would be valid if we the functionalist theory of cognition held. Lave demonstrates that learning transfer clearly does not occur in practice.

A lot of ground is covered in this book, and Lave makes some very progressive and compelling arguments. Her work can be considered a criticism of functionalism in all its forms, which would include mental model theory. Having found model theory useful for many purposes, I would posit that practice can be understood in terms of models. This requires a subtle understanding of what a model is: a consistent system of meaning that comes with a way to interpret and translate things in the world into symbols in the model. Lave’s argument could easily be extended to argue that all theories are practices, but I would extend this further and say that theories and practices are both models in turn.

Psychology and Anthropology I

An opening example is how arithmetic is used in everyday life. Problems that ordinary people face are very unlike problems that might be specified in a classroom sort of setting. In the introduction, Lave gives an example of a shopper who is buying apples, and rationalizing how many apples to buy. As presented, arithmetic is a tool that is used in everyday life, not as an explicit approach, but rather as something like duct tape, to be used for estimations than exact calculations. Classical mathematics studies math as an artifact, a whole subject, rather than a tool.

Lave gives a solid critique of functionalist theory. This is characterized by several points (p. 8):

• Passivity of learner: The learner is passive and receives knowledge without having an active role in its learning or use.
• Isolation of skills: Skills may be separated and understood atomically. Separation is unproblematic, and usage is not dependent on context.
• Leads to meritocracy: The use and adeptness at arbitrary skills leads to a value system where proficiency at one skill indicates general superiority.
• Stress on rationality and cognitive life: This is as opposed to usage and practice.
• Cultural uniformity: If the functionalist theory holds, then skills and functions transcend culture, meaning that culture must have uniform basis.

The book is organized into two parts: The first is a critique of the practice of cognitive theory. This challenges the functionalist emphasis on learning transfer, and the artificiality of the laboratory method. The theme of this argument is around the idea of ecological validity, where a theory is ecologically invalid if it fails to be supported in a diversity of settings. The latter half of the book explores and emphasizes the study of practice in of itself.

Missionaries and Cannibals (Indoors)

The section here is trying to understand the role and concept of knowledge transfer, the process by which knowledge from one domain is transferred to another. Presumably this happens analogically. The actual results of experiments about knowledge transfer obtained mixed conclusions. These experiments give students a math problem, which may be solved algorithmically, and give the students other problems which are isomorphic in some way. Students do not make the necessary leaps in these sorts of problems, and generally seem very confused. A functionalist theory would support a sort of reductive understanding and transfer, but this does not hold. Incidentally, this type of transfer is exactly the sort of mechanism by which Newell’s GPS worked.

Lave gives a critique of the method by which students were presented with problems in the learning transfer studies. “A number of problem characteristics are common to all four papers and by extension to the genre more broadly. The puzzles or problems are assumed to be objective and factual. They are constructed ‘off-stage’ by experimenters, for, not by, problem solvers.” (p. 35) Thus, the problems are presented as tests, and come replete with the values of producing a correct solution. These have an overall trend of disempowering the subjects, depriving them of suitable context or situation.

The learning transfer experiments rely on a sense of knowledge as component based or modular. In this point of view, knowledge is put into domains, which function like disconnected islands. However, this point of view, and the problems themselves, are cultural artifacts.

Life After School

To find an alternative to traditional problem solving, Lave conducted a study called the Adult Math Project, which studied how ordinary people use mathematics in everyday life, and more specifically, grocery shopping. This study found that people are remarkably successful at math problems, (framed both on paper, as well as in the store) but they use dramatically different methods than those taught in school.

A problem I found with her study, though, is that it is placing supermarket decisions primarily in terms of cost. This relates back to the fallacious idea of the rational consumer. Supermarket choices may factor in cost, but also have a great deal to do with social and class awareness and attitudes, preferences, and identity.

Psychology and Anthropology II

Lave is attempting to undermine the false dichotomy of scientific and “everyday” thought. This descends from dualism and cultural colonialism. The central theme in this is that science itself is a practice, and that it is culturally constructed.

Inside the Supermarket (Outdoors) and From the Veranda

The goal in this section is to develop a theory of practice. The first part of this is to distinguish between practice and formal knowledge domains, or conceptual spaces. A primary difference is that practice occurs simultaneously within other activities, and may be synchronized with them to some degree. An example of this that Lave gives is knitting and reading. Instead of taking place individually, they are situated within each other and affect each other.

There is a concern over the form of scientific inquiry. This may be seen as a concern over model building. How do we decide what to model, or decide what is important or relevant? This is an authorial judgement, and comes in part with judgement of value. Lave asks, “Further, who is to decide what cognitive phenomena are significant objects of study, and how? Are guidelines to be found in normative models of cognition, in an investigation of the activities of peoples’ lives, in some combination, or in other sources altogether?”

To address this concern, she introduces the idea of ecological validity. Experiments are ecologically valid if they get the same results regardless of situation. This is important for the study of cognitive science, because the way people think within a laboratory setting may be different in some ways than how they think outside. If a cognitive experiment yields one set of results within the laboratory, but totally breaks down outside, then the theory on which that experiment is based must be questioned.

An experiment that Lave critiques a great deal in this section is another supermarket math experience, conducted by Capon and Kuhn. Their experiment is intrinsically biased towards the knowledge domain understanding of math. The experiment was set at a grocery store, but still carried the structure of laboratory problems: answers were right or wrong, and the type of reasoning was intended to be proportional. Instead of asking “what sort of math occurs in grocery shopping?” they tested whether subjects could perform a certain kind of math.

Out of Trees of Knowledge into Fields for Activity

This chapter frames the complexity in problem solving. Instead of problem solving existing at one small and discrete domain, it is situated within a broader context. “People experience ‘problems’ subjectively in the form of dilemmas and, so motivated, ‘problem-solving’ actively often leads to more or less enduring resolutions than precise solutions.” (p. 124)

An example that Lave focuses on is a study of practitioners of Weight Watchers, who incorporate the goals of Weight Watchers into their daily routines of shopping and food preparation. Thus, these activities are recast with a new set of motivations. I would argue that these practices are totally consistent with model theory. Weight Watchers has a model with one underlying principle: the quantization of food. This causes activities involving food to be understood as systems of quantities. These models and motivations are necessarily in conflict with others.

Outdoors: A Social Anthropology of Cognition in Practice

In the opening to her concluding chapter, Lave overviews some of the elements of her study of practice (p. 171):

1. The context of a study is important, (for example, in the supermarket), but the context surrounding the situation is also important. The supermarket is also contextualized in peoples’ lives, in which the shoppers’ dilemmas are construed.
2. Conventional premises and analytic questions must be understood critically. This book can be seen as a project which makes these assumptions to be objects of study.
3. The study focuses on “whole-person activity” rather than attempting to understand cognitive functions in isolation. This places cognition as dependent on on time and setting, within culture.
4. Accepting that activity is situated, then those activities are placed within some point in history and culture, and must be understood in that context. Thus, the dilemmas of the shopper (and hence the form of math and cognition used) in 1950s America may be different from those found in contemporary Beirut.

Lave’s final conclusion has a good review of her project as a whole: “I have tried to move the investigation of ‘cognition’ outdoors in several senses: out of the laboratory, out of the head, out of a confusion with a rationalistic ‘culture,’ out of conflation with conventional ‘knowledge structures,’ and out of the role of order-producing, primary constraints on activity in the world.” (p. 189-190)

I haven’t had the chance to do much reading this week. I really ought to do more, but it has been hard to fit it in. What I have been doing, oddly enough, is a lot of writing. Last Thursday, I gave a presentation overviewing my work on game adaptations of Pride and Prejudice to Janet’s narrative schema group, and it went alright. It was… tricky, though. She was expecting me to be further along than I was, and presenting slightly different material than I was, but there was a very productive discussion afterwards. A few new ideas floated up, but mostly I had the chance to “try out” some of the ideas I have been thinking about for a long time. Now that it is over, I have been writing an overview of the adaptation project. It started out as a small thing, but has quickly evolved into something quite big.

Rest assured, I will put it up when I am done. Whether anyone will find it interesting or not, well…. we’ll have to see.

One of my classes is cross listed with an undergraduate course, this is Nancy Nersessian‘s Cognition and Culture. One fun advantage to having a course with undergraduates is that they make a lot of interesting and occasionally profoundly brilliant observations. Not to say that us graduate students are incapable of insight, but we tend to be very bogged down by our own research objectives.

We have been discussing Jean Lave‘s book, Cognition in Practice, and came to a segment where Lave discusses how mathematics is a cultural artifact, but we view it as universal and supremely valuable. An example of this is that we “beam” the Pythagorean theorem into space, in hopes that, were the signal ever to be discovered by extraterrestrials, it would help communication because mathematics is a universal language, that transcends humanity. I didn’t find a source on this beaming precisely, but it seems like the sort of thing that people might do. Coming from a mathematical background, and moving into the complex and tricky field of cognitive science and cultural studies, I had very torn reactions to this conflict, and only realized how to articulate that reaction after the discussion ended. So, I present it here.

Mathematicians are extremely strange people. I don’t really identify as a mathematician anymore, but I still consider myself close to the culture, so I say this pridefully. The conclusions of mathematics are universal, and they are fundamental, but, and this is where things get difficult, these universal conclusions rely on premises. These premises are necessarily situational, and depend on other cultural factors. Furthermore, the practice of mathematics is also culturally relevant, and lots of mathematicians disagree, not on conclusions (a proof is a proof, after all), but on the relevance, importance, usefulness, and elegance of different practices of math. All of these terms are subjective, and while there are many common impressions of what elegance means, it is far from universal.

Generally issues regarding the practice of math applies to topics that are more sophisticated than the Pythagorean theorem. The Pythagorean theorem has to be universal because of its simplicity, elegance, and universality in almost all kinds of math that we use conventionally, right? Those aliens must use that kind of math too, right? Well, mostly. Even in this case, the situation is ambiguous, and that ambiguity arises from the premises under which the Pythagorean theorem is valid, namely: Euclidean geometry. If you are dealing with some other domain of planar geometry, (most notably, spherical or hyperbolic geometry), then the Pythagorean theorem breaks down. It has analogues (which are quite elegant, I might say), but the existence of these alternative types of geometries, and the ways in which the theorems are modified illustrates that our idyllic Euclidean world is not quite as simple or so complete as it first seemed. Space itself is non-Euclidean, according to both relativity and quantum mechanics. So perhaps the Pythagorean Theorem may actually have something to do with our experience as humans on Earth, and may not be quite so transcendent after all.

For real transcendence, we need Godel’s Incompleteness Theorem. That’ll do it for sure.

I found myself discussing Michael Mateas‘s paper Semiotic Considerations with Audrey a week or two ago, and it occurred to me to put the paper on my reading lists. It is an important work, and also ties in very nicely to some of my focus on models. Reading his essay in detail, now that I have a much more substantial backing in cognitive science and structuralist theory, I came to realize that we are approaching similar regions of focus, but from coming to them from very different perspectives.

Mateas’s actual paper is breathtakingly short, a mere four pages of 18 very explicitly defined points. This deceptively clear organizational structure conceals the real depth at play in the paper. The paper is roughly divided into three regions of focus, and I will explore these and summarize these sections in context. His paper has the form of a poetics of Expressive AI, rather than an aesthetics, or an approach for criticism of such works.

One of my interests here, although I’m not going to have the chance to get into it, is the relationship between model and simulation, as compared to Mateas’s rhetorical and computational machines.

One: The Experiences of AI and Art

Mateas’s paper is on the practice of creating AI based art, which he calls Expressive AI. In Mateas’s perspective, art is fundamentally semiotic in nature. Art involves negotiating flows of meaning or semiosis, which is very situationally dependent. Some examples of situations are “a busy freeway, an office in a large bureaucratic organization, a party, a riot, or, perhaps, an art gallery,” these all are both recipients to artistic intervention, and they turn around and affect the flow of meaning given by the art itself.

Art in this case is a proactive and assertive force, engaging with the world and reconfiguring it. Art is  intrinsically participatory, as semiosis requires active observers and interpreters. Art is dynamic, as it is subject to influence by the context in which it is situated, and necessarily affected by the flows of its participants. Even the term flow communicates dynamism.

Connected with AI practice, expressive AI is the combination of AI and  injects not only the affordances of participating or simulated computer controlled characters, but also a unique intersection of rhetoric and semiotic functions that are unavailable in other forms of art practice. An AI based artwork becomes itself an active participant in semiosis, and may engage in working with flows of meaning. In this sense, the AI based work shares the situation with the human participants or observers.

Because Expressive AI operates at this unique intersection, Mateas argues that we should think of it in a new light. We must use special rhetorical strategies for understanding the relationship between the computational and artistic dimensions of the artwork. We must share a language to address these different domains, which operate on very different symbolic terms, but Expressive AI is interesting precisely because it exists at this intersection. The issue then becomes how to understand and negotiate these two critical components of AI practice.

Two: The Computational and Rhetorical Machines

Computational artifacts, and AI especially, are subject to a set of semiotic properties that are unique to the computational form. Specifically, computation enables automatic symbolic processing. This is the property of the computer heralded by Turing and popularized by Herbert Simon and Alan Newell. In this tradition, symbol systems exist in a domain of pure or abstract reasoning, much like abstract sign systems in the semiotic tradition. What is interesting about symbols and signs, is that, in principle anyway, they are intrinsically arbitrary and meaningless. One sign or symbol may be used to denote or connote anything.

Rhetorical meaning derives from and requires human interpretation. The human observer is necessary for connecting signs to referents, and for extracting meaning from an arbitrary configuration or conglomeration of symbols. Interpretation is necessary for categorizing a program as intelligent in any form. The role of interpretation imposes a new complexity in the otherwise ideal symbolic world, because human observation involves partial observers and the revelation of functional values which may be intrinsically encoded within the symbolic systems.

The technical operation of the AI system involves a computational machine, which is responsible for the processing of symbols. However, under the lens of human interpretation the AI system becomes part of something else, a rhetorical machine, wherein the system is coupled with the world of meanings and referents. “Every system is doubled, consisting of both a computational and rhetorical machine.”

This doubling affects how AI systems are created and interpreted. A developer of an AI based artifact must be aware of the relationship between the rhetorical and computational machines, as technical decisions will ultimately reflect rhetoric. A creator of Expressive AI must construct not only an artifact which engages as a participant in semiosis and flows of meaning making, the creator must also inscribe artistic intentions into the dual machines in order to affect the possible outcomes of interaction with the system. This lays out new artistic affordances and challenges, the scope and breadth of which is not yet clear.

Three: Systems of Code and Execution

All computational programs work using two systems or planes of meaning. The first plane is the space of the written program. This is what is actually authored, and its content is not the artifact, but rather the set of meanings that will enable or allow the artifact. It signifies the space of all possible executions. The second plane is the actual plane of execution, on which the artifact may actually be engaged and may participate in the semiotic processes described above. Mateas calls the first system, the code system, system1; and the second system, the execution system, system2.

The division between these two planes is deeper than their functional dimensions, but extends to the rhetorical strategies for interpreting, understanding, and manipulating the artifact on those planes. The two systems share signs, and we use similar language for discussing them. They are paired with their own matching rhetorical systems, which are different, but interact closely with each other.

Both of these systems have what Mateas calls “iterpretive surpluses.” The system1 has an interpretive surplus for the author, and the system2 has an interpretive for the audience. The author’s surplus comes with a freedom to embed strategies and approaches into the system, and interpret the composition of the system creatively. This leads to (especially in Mateas’s own projects) new terminology for constructing content in the system1. The audience’s surplus is one that resembles more closely the interpretive surplus afforded by interaction with works of art and other artifacts. The domain of execution can (and must) make use of other established media strategies and traditions for expressivity, that can be used for the audience to better glean meaning from the work.

The arrangement so far presents a portrait of Expressive AI as a practice of negotiating and manipulating many flows of meaning, and many complex and interdependent systems. The issue of rhetoric and language is extremely important, and provides a methodology according to which one may author works of Expressive AI. Mateas does not lay down any specific language that he thinks should be used, but rather, examines the role which he thinks rhetoric and language should play in the development of such artifacts. Furthermore, because the systems are so intrinsically connected, he argues against the approach of AI as a means to an end. To achieve a system with certain rhetorical goals, it must employ a computational machine that mirrors that rhetoric.

Kenneth Burke was an American philosopher and literary critic writing in the mid 20th century. His principles have had influence on a great number of others: Edward Said, Clifford Geertz, Frederic Jameson, and Erving Goffman. The introduction to this book poses his investigation as similar to some standard sociological questions, but his inquiry is much more philosophical in nature.

Burke’s opening also gives a straightforward overview of the principles of dramatism: The explicit topic is understanding everyday action and motive. Burke’s solution is to pose a dramatic approach. This uses five key elements, which he calls the pentad: Act, Scene, Agent, Agency, and Purpose. Act represents the nature of the deed or action took place. Scene is the situation and background wherein the act takes place. The agent is the performer who enacted the act. Agency is the means and manner by which the agent could perform the act. Purpose is the most ambiguous term, reflecting why the agent performed the act in the first place.

Burke explains the challenge in using a particular philosophical idiom. Originally, the grammar of motives was intended to be a theory of comedy, and more precisely a rhetoric, but eventually became clear as a grammar. The idea behind creating the grammar is not to shoehorn all experience into the rigid structure of the grammar, but rather, expose and understand ambiguity.

A perfectionist might seek to evolve terms free of ambiguity and inconsistency (as with the terministic ideals of symbolic logic and logical positivism). But we have a different purpose in view, one that probably retains traces of its “comic” origin. We take it for granted that, insofar as men cannot themselves create the universe, there must remain something essentially enigmatic about the problem of motives, and that this underlying enigma will manifest istself in inevitable ambiguities and inconsistenceis among the terms for motives. Accordingly, what we want is not terms that avoid ambiguity, but terms that clearly reveal the strategic spots at which ambiguities necessarily arise. (p. xvii)

There may be overlap between the categories. For example: war may be seen as agency, act, purpose, scene, or agent, if personified. One feature of this categorization is that issues of the border between agent and instrument are disposed of. Relationships between an actor and an instrument, such as “the chisel is an instrument,” versus “the hand is an instrument,” cause problems in comparison to cognitive extension. Burke’s fomulation here encompasses these under the broad stroke of “agents,” whereas the term agency has a significantly more separate role. This approach reflects the idea of symbolic action as posed by Mead.

The Container and Thing Contained

Grammar implies a structure, and the first element of structure is containment. The scene contains the act, and it contains the agents. The idea here is that the scene encompasses and reflects the act dramatically. This is best evident in drama, but can be found in fiction as well as in real life. We reconstruct and recast scenes to reflect the nature of our actions within them. One of Burke’s goals in this section is to examine the ratios between scene, act, and agent. These ratios are the degrees of emphasis or dependency belonging to the elements in given actions. Burke’s postulate seems to be that acts may be analyzed dramatistically in terms of the ratios involved.

Any verb can be an act if it is willed. Verb and intention defines an act. This has significant ramifications for how we might read acts and intentionality with respect to agents. “As for ‘act,’ any verb, no matter how specific or how general, that has connotations of consciousness or purpose falls under this category.” (p. 14)

The ratios are not means of measurement, but rather a tool of analysis. Burke gives an example of examining nations and “democracy.” Employing different ratios yields different perspectives on the relations between the two. A scene-act ratio would cast nations as scenes or situations and these situations would enable acts that are democratic or otherwise. With an act-agent ratio, the people of nations would perform “democratic acts.” This example is a little convoluted, but presents an inherent ambiguity and ambivalence of the ratios as investigative tools.

Antinomies of Definition

The matter of definition exposes a paradox at the heart of trying to understand something. Definition requires context, but requires a differentiation from that context. “To tell what a thing is, you place it in terms of something else. This idea of locating or placing, is implicit in our very word for definition itself: to define or determine a thing, is to mark its boundaries, hence to use terms that posess, implicitly at least, contextual reference.” (p. 24) Here, Burke is borrowing significantly from Spinoza, who was critical that things could be observed alone in themselves. This idea resurfaces again in Stafford’s theory of likeness.

One central element of Burke’s analysis is the notion of dialectics. There is a necessity of opposition in dialectics. Rivals are at opposite banks of the same stream. This sense of opposition is one in which adversaries share the same scene or model. In these situations, the opposing parties on each side of the dielectic may vehemently disagree regarding their issue at stake, but they do necessarily agree that there is a world consistent save for this issue. An opposing view to the dialectic perspective is one in which differences arise from the use of different models. In these situations, no common ground may even be found for taking up opposition for argument.

Scope and Reduction

There is an interesting opening here, which applies very neatly to the construction of models. Forming a cohesive vocabulary is a similar (if not the same) process as constructing a model. Burke’s goal with dramatism is to form a procedure for developing a terminology (or calculus) of any given domain. The key here is the idea of a representative anecdote. In a model based perspective, a representative anecdote might be compared to some sort of representative or essential element.

Men seek for vocabularies that will be faithful reflections of reality. To this end, they must develop vocabularies that are selections of reality. Any selection of reality must, in certain circumstances, function as a deflection of reality. Insofar as the vocabulary meets the needs of reflection, we can say that it has the necessary scope. In its selectivity, it is a reduction. Its scope and reduction become a deflection when the given terminology, or calculus, is not suited to the subject matter which it is designed to calculate. (p. 59)

There is a focus on the nature and effects of reduction. Divisions, distinctions, measurements, quantifications, comparisons, and metaphors, are all reductions.

Understanding a scene in terms of reductions: The scene must reflect only the reduced domain when a reduction is at hand. Any scene that reflects elements that do not belong to the reduction will fail to make sense when the reduction is in place. The idea that seems to be here is the importance of consistency. The grammar of drama is invariant under reductions (even though it is a reduction itself), and accounts for the transformed nature of content. A motive is a reduction, and can be understood as an analytic tool that is consistent as long as the dramatistic elements are consistent under that motive. For instance: if the motive is power, then acts must be power related acts, scenes must be scenes where power is at play, agents must be making use of power, agency is how power is played, and purpose is necessarily the pursuit of power.

A reduction, model, or term, is a way for interpreting the world, but is also a way to generating new worlds, which are dependent on these interpretations. Burke explains:

In sum: In any term we can posit a world, in the sense that we can treat the world in terms of it, seeing as all emanations, near or far, of its light. Such reduction to a simplicity being technically reduction to a summarizing title or “God term,” when we confront a simplicty we must forthwith as ourselves what complexities are subsumed beneath it. For a simplicity of motive being a perfection or purity of motive, the paradox of the absolute would admonish us that it cannot prevail in the “imperfect world” of everyday experience. It exist not actually, but only “in principle,” “substantially.” (p. 105)

Four Master Tropes

Burke explains the four master tropes: metaphor, metonymy, synechdoche, irony. He reformulates these in terms of other formats: Metaphor is perspective, metonymy is reduction, synechdoche is representation, and irony is dialectic. (p. 503)

Last Thursday, Barbara Stafford came to visit LCC as part of our distinguished speaker series. I read her book Visual Analogy last year, although whether I understood it or not is a different matter entirely. Stafford’s background is in art history, although recently she has expanded into the study of the cognitive dimensions and neuroscience of images. Her talk had the quality of weaving in and out among a number of conceptual domains, putting together a complex web that related to a few specific themes.

Because I am boring and no fun, one of the things I am going to try to to is unravel Stafford’s carefully woven tapestry, and isolate her exploration into more explicit bullets and categories. My drive to do things like this may strike the more poetic minded as undercutting her message, but without finding a graspable end of the woven yarn, it is hard to get the message at all.

Stafford’s central theme in her talk is the notion of attentiveness. One of her goals is to devise a taxonomy of attentiveness. In the process of exploring the kinds of attentiveness, there are a few binary conflicts: Speed versus slowness, automaticity versus spontaneity, and focus versus attention. Her exploration weaves through each of these conflicts, valuing the virtues of slowness, spontaneity, and attentiveness. The last of these conflcits is perhaps the easiest to explain.

Attentiveness is not focus. It is related: a chiefly visual practice, but takes a different aim in mind. Attentiveness is embodied, whereas focus is disembodied. An observer is focused when engaged with something at a distance, and with a specific objective in mind. Attentiveness is aware of the subject in its surroundings. Attentiveness is has many channels, and makes use of emotion and the body. Attention encompases affectiveness and affection. Focus narrows both the observer and the object observed down to a single channel: The observer is a detached eye, and the observed is reduced into components and parts. The rhetoric of focus comes from many sources, and is found in a sort of postmodern criticism. Pathological focus is voyeuristic in nature: it is the subjugation of the heat of affective life to icy scrutiny.

Automaticity relates to focus. The rhetoric and language of automaticity emerged from cybernetics and computation. The converse of this is sponteneity, whose language comes from art and the life sciences, especially biology. Both automaticity and spontaneity are about reactions and behavior. The difference between the two reflects the difference between focus and attention. Automaticity is disembodied where sponteneity is embodied. Automaticity is about precision and correctness, where sponteneity is about naturalness and freedom. Automaticity is rational where sponeneity is emotional. Automaticity has infiltrated our lives through computation. Stafford explained, hearkening back to an argument that has been made since Heidegger, that while we transfer data to computers, computers transfer their way of thinking back to us. Cognitive science has been infiltrated with the language of automaticity, especially that which comes from economics: We talk about “cognitive productivity.” Parts of the brain or mind have been deregulated or privatized. Automaticity is a language of parts, sponteneity is a language of wholes.

The final binary separation is between speed and slowness. Computation and automaticity aims to reduce things in a way that make them more easily systematized and more efficiently computed. Movements in art have moved toward slowness. Slowness demands a certain hesitation, something which Stafford considers a lost concept. Slowness also encourages reflection, and awareness of circumstance. Much postmodern architecture encourages the aesthetic of speed: glass is used to reduce the time that is necessary to look at things. Stafford gave several examples of artists who used slowness as an aesthetic, but I was only able to capture two of them. One is a documentary by Steve McQueen, called Gravesend. The other artist is Andy Goldsworthy. Both of these artists encourage the viewer to slow down and reflect. Instead of emphasizing the degree of information that can be observed, more can be learned and understood through careful observation and attention.

Slowness, sponteneity, and attention are all the same kind of thing. The virtues are wholeness, affection, and living in the moment.

Stafford’s goal is to develop a taxonomy of attention. She explores these by examining several kinds of looks, all reminisent of certain kinds of attention. Stafford’s presentation made use of paintings, photographs, and some digitally edited photographs. The kinds of attention are represented both in the subjects of the images, but also are evidenced by our own reading of the images. Becaue I am a dork, I’m actually bulletizing these:

1. What is a critical/diagnostic look?
   Critical observation and decision making have been studied in great detail in cognitive science. How we plan and select actions relates to a critical and spontaneous moment where the decision is actually made. This is about a moment, extending beyond focus.
2. What is a comparative look?
   Comparision is beyond impulse or reflex, but about a slow consideration of alternatives.
3. What is a sorrowful look?
   In studying affect, it is easy to cognitively understand simple emotions like pain and pleasure. However, these means of study are ineffective at comprehending deeper, more complex emotions. Stafford showed us Caravaggio’s Penitent Magdalene as an example. The emotions of the girl are more than simple categories can explain. Like other kinds of slow art, we understand more of it the more we observe.
4. What is a ponderous look?
   Pondering relates to weight in the etymology of the word itself. A ponderous look reflects the weight of the subject being considered.
5. What is a sweet look?
   One of the more interesting categories used Correggio’s Jupiter and Io. Stafford asked the interesting question: What makes us want to be in the moment? What makes us want to remain awake? This issue goes back to the deeper issue of desire. Sweetness relates to desire, touch, and longing. This is different from the lengthy focus of the voyeur, but is a different eroticized yearning, which is another mixed and complex emotion.
6. What is an inattentive/distracted look?
   In contrast to some of the above, these are not complete. Inattentive or distracted looks have their own complexity. An inattentive look is about fading and drifting consciousness. It is not about identity or self, but how we inhabit or dwell in the self. Similarly, distraction is not about multitasking, but about dispersal. These looks reflect where our consciousness resides.

This past Thursday, after Barbara Stafford (which I will write about later), Audrey and I came home after a quick stop at the grocery store, and passed by an onion on our way to our apartment. It was outside, lying at the side of the road, without any obvious defect, other than it might have fallen from someone’s grocery bag and touched the ground.

So, we took it upstairs and cooked with it. Cooking with found food, I suppose. Delicious!