I found myself discussing Michael Mateas‘s paper Semiotic Considerations with Audrey a week or two ago, and it occurred to me to put the paper on my reading lists. It is an important work, and also ties in very nicely to some of my focus on models. Reading his essay in detail, now that I have a much more substantial backing in cognitive science and structuralist theory, I came to realize that we are approaching similar regions of focus, but from coming to them from very different perspectives.

Mateas’s actual paper is breathtakingly short, a mere four pages of 18 very explicitly defined points. This deceptively clear organizational structure conceals the real depth at play in the paper. The paper is roughly divided into three regions of focus, and I will explore these and summarize these sections in context. His paper has the form of a poetics of Expressive AI, rather than an aesthetics, or an approach for criticism of such works.

One of my interests here, although I’m not going to have the chance to get into it, is the relationship between model and simulation, as compared to Mateas’s rhetorical and computational machines.

One: The Experiences of AI and Art

Mateas’s paper is on the practice of creating AI based art, which he calls Expressive AI. In Mateas’s perspective, art is fundamentally semiotic in nature. Art involves negotiating flows of meaning or semiosis, which is very situationally dependent. Some examples of situations are “a busy freeway, an office in a large bureaucratic organization, a party, a riot, or, perhaps, an art gallery,” these all are both recipients to artistic intervention, and they turn around and affect the flow of meaning given by the art itself.

Art in this case is a proactive and assertive force, engaging with the world and reconfiguring it. Art is  intrinsically participatory, as semiosis requires active observers and interpreters. Art is dynamic, as it is subject to influence by the context in which it is situated, and necessarily affected by the flows of its participants. Even the term flow communicates dynamism.

Connected with AI practice, expressive AI is the combination of AI and  injects not only the affordances of participating or simulated computer controlled characters, but also a unique intersection of rhetoric and semiotic functions that are unavailable in other forms of art practice. An AI based artwork becomes itself an active participant in semiosis, and may engage in working with flows of meaning. In this sense, the AI based work shares the situation with the human participants or observers.

Because Expressive AI operates at this unique intersection, Mateas argues that we should think of it in a new light. We must use special rhetorical strategies for understanding the relationship between the computational and artistic dimensions of the artwork. We must share a language to address these different domains, which operate on very different symbolic terms, but Expressive AI is interesting precisely because it exists at this intersection. The issue then becomes how to understand and negotiate these two critical components of AI practice.

Two: The Computational and Rhetorical Machines

Computational artifacts, and AI especially, are subject to a set of semiotic properties that are unique to the computational form. Specifically, computation enables automatic symbolic processing. This is the property of the computer heralded by Turing and popularized by Herbert Simon and Alan Newell. In this tradition, symbol systems exist in a domain of pure or abstract reasoning, much like abstract sign systems in the semiotic tradition. What is interesting about symbols and signs, is that, in principle anyway, they are intrinsically arbitrary and meaningless. One sign or symbol may be used to denote or connote anything.

Rhetorical meaning derives from and requires human interpretation. The human observer is necessary for connecting signs to referents, and for extracting meaning from an arbitrary configuration or conglomeration of symbols. Interpretation is necessary for categorizing a program as intelligent in any form. The role of interpretation imposes a new complexity in the otherwise ideal symbolic world, because human observation involves partial observers and the revelation of functional values which may be intrinsically encoded within the symbolic systems.

The technical operation of the AI system involves a computational machine, which is responsible for the processing of symbols. However, under the lens of human interpretation the AI system becomes part of something else, a rhetorical machine, wherein the system is coupled with the world of meanings and referents. “Every system is doubled, consisting of both a computational and rhetorical machine.”

This doubling affects how AI systems are created and interpreted. A developer of an AI based artifact must be aware of the relationship between the rhetorical and computational machines, as technical decisions will ultimately reflect rhetoric. A creator of Expressive AI must construct not only an artifact which engages as a participant in semiosis and flows of meaning making, the creator must also inscribe artistic intentions into the dual machines in order to affect the possible outcomes of interaction with the system. This lays out new artistic affordances and challenges, the scope and breadth of which is not yet clear.

Three: Systems of Code and Execution

All computational programs work using two systems or planes of meaning. The first plane is the space of the written program. This is what is actually authored, and its content is not the artifact, but rather the set of meanings that will enable or allow the artifact. It signifies the space of all possible executions. The second plane is the actual plane of execution, on which the artifact may actually be engaged and may participate in the semiotic processes described above. Mateas calls the first system, the code system, system1; and the second system, the execution system, system2.

The division between these two planes is deeper than their functional dimensions, but extends to the rhetorical strategies for interpreting, understanding, and manipulating the artifact on those planes. The two systems share signs, and we use similar language for discussing them. They are paired with their own matching rhetorical systems, which are different, but interact closely with each other.

Both of these systems have what Mateas calls “iterpretive surpluses.” The system1 has an interpretive surplus for the author, and the system2 has an interpretive for the audience. The author’s surplus comes with a freedom to embed strategies and approaches into the system, and interpret the composition of the system creatively. This leads to (especially in Mateas’s own projects) new terminology for constructing content in the system1. The audience’s surplus is one that resembles more closely the interpretive surplus afforded by interaction with works of art and other artifacts. The domain of execution can (and must) make use of other established media strategies and traditions for expressivity, that can be used for the audience to better glean meaning from the work.

The arrangement so far presents a portrait of Expressive AI as a practice of negotiating and manipulating many flows of meaning, and many complex and interdependent systems. The issue of rhetoric and language is extremely important, and provides a methodology according to which one may author works of Expressive AI. Mateas does not lay down any specific language that he thinks should be used, but rather, examines the role which he thinks rhetoric and language should play in the development of such artifacts. Furthermore, because the systems are so intrinsically connected, he argues against the approach of AI as a means to an end. To achieve a system with certain rhetorical goals, it must employ a computational machine that mirrors that rhetoric.