Blogs
What Does It Mean To Prompt A Robot? | Blog by Aengus ten Cate-Schulte
On the 6th of February, Dramaturgy for Devices and Robotstories | Expanding Narratives organised a collaborative exploration of Prompts & Theater. With the advent of LLMs, the concept of “prompt” has entered the HRI vocabulary. However, in the theater domain, formulas and systems resembling what a prompt might be for AI have been in use for much longer.
For example, Forsythe’s Improvisation technologies or Laban’s movement notation. The concept of “prompt” also took on meaning in other domains. The aim of the Prompts & Theater Workshop was to introduce and explore these crossovers and perspectives, after which potential next steps were identified.
This knowledge-sharing event was a collaboration between Dramaturgy for Devices and Robotstories | Expanding Narratives, joined by two special guests: Dr. Efrosini Protopapa, a London-based choreographer and academic whose work focuses on experimental and conceptual practices across dance, theatre, dramaturgy, and performance, and Nuno Atalaia, theatre director and researcher, who recently defended his PhD thesis on theatre and digital voice assistants.
In a world that is becoming increasingly fixated upon Large Language models (LLMs) and ChatGPT, ‘prompting’ has been growing into a fairly ubiquitous expression. But what exactly does it mean? How does its use differ across disciplines? And how can it help with better understanding human-robot interaction? It was with these questions in mind that we hosted a discussion event dedicated to the topic on Friday the 6th of February, bringing together researchers from robotics, media, theatre, dance, and performance studies.
Our initial interest in the concept arose from a conversation between Maaike Bleeker, a Professor of Performance, Science and Technology at Utrecht University, and Jorrit Thijn, a writer and PhD researcher at the Vrije Universiteit in Amsterdam working on incorporating social robots into dementia treatment. They were both drawn to the idea that prompts, while outlining an objective, simultaneously allow for a certain freedom in how to achieve that goal.
However, a significant challenge is the lack of consensus regarding what a prompt means. It’s a particularly ephemeral term, perhaps appropriately given its status as somewhere between command and suggestion. Not only that, but its usage differs between disciplines, from the textual instruction of computer science to the more open call-and-response of improvisation studies. Our answer was to call upon these different perspectives and enable a flow of information between them, so as learn what we could learn from one another.
The first portion of the day was headed by Maaike, who discussed her work together with theatre maker Ulrike Quade and roboticist Rick van Dugteren on using insights from improvisational dance to generate robotic movement. This work specifically draws on William Forsythe’s improvisation technologies, a project that uses computation to visualise and trace the movements of the human body through space. In effect, these visualisations offer frameworks for actions, suggesting poses and movements without prescribing the exact means by which they should be performed. In applying these technologies to robots, Maaike and her collaborators came to the realisation that such visualisations can be used as prompts, outlining a specific outcome to be achieved but allowing the robot freedom in how it reaches this result.
When Jorrit heard about this realisation he was quick to relate it to his own work, which is what eventually lead to the genesis of this discussion event. In his presentation he explained that as part of his PhD research he was developing robots as conversation partners for people living with dementia, and was quickly confronted with a problem. Given the dynamic nature of conversations, any such robot would have to be able to respond spontaneously to unexpected moments. But it simultaneously had to maintain its ‘identity’ as a friendly conversation partner, so that its responses were not random but based upon acceptable social norms. The problem that Jorrit was facing, then, was a conflict between two kinds of prompts: one that determined the role from which the robot should perform, and another that indicated how it should perform that role.
This problem—and the solution that Jorrit reached—was displayed in a live demonstration in which a robot performed an improvised segment from Jorrit’s play De Moestuin | Robot op de vloer. In this segment the robot—playing as the character Stein—recites a monologue from Anton Chekov’s Uncle Vanya, which is subsequently commented upon by the human actor. The exact manner in which Stein responds is left open, but nevertheless follows certain logics: when Jorrit asked him to recite the passage again but in a lighter tone, he refused, arguing that the melancholy performance of the monologue imbued it with weight. This offered a useful insight into the relation between a user prompt, or the types of behaviour encouraged by a human, and a system prompt, the deeper conditioning that determines how an LLM expresses itself. In the case of the example above, Stein could have very easily done as Jorrit asked, but that would have interfered with the role that he was embodying. Prompts within the context of robotics therefore seem to leave space for a robot to express themself, rather than strictly confining them to a set outcome.
The second portion began with Aengus ten Cate-Schulte, an intern on the project and student of the research master Media, Art, and Performance Studies at Utrecht University, positioning these questions surrounding expression and agency within a literature review of robotics studies. Contrary to the behaviour exhibited by Stein, he argued that the current understanding of prompts within robotics is inherited from their use within LLMs, and that this came with several dominating associations. Namely, he contend that prompts are seen as intentional, prescriptive, optimisable, and commanding. All of this served to locate the power of prompting squarely within the human prompter, with the robot simply being the passive recipient of instructions. But Aengus believed that this obscured the power that robots can exercise in how they respond to a prompt. To illustrate this he drew on two concepts from improvisation studies: norms, which are simultaneously formed through the ways in which two performers respond to one another and influence the conditions of these interactions; and orientation, or the observation that the goal of an improvisation can shift throughout its performance. These concepts showed that prompting is not a matter of strictly following commands, but rather playfully engaging with them in open ways.
Developing upon the theme of the general understanding of prompts, Mike Ligthart, Assistant Professor of Social AI at the Vrije Universiteit in Amsterdam, then offered an interdisciplinary overview of their various meanings. Drawing from the fields of creating writing, theatre, psychology, and LLMs, he showed that although specific understandings of prompts and prompting differed across each of these, they all conveyed the general notion of a signal—whether that be an action, image, utterance, or something else—that leads to a certain outcome. He then broke down the exact process by which this works within ChatGPT, explaining that its responses is not just based upon a single prompt provided by the user, but by the conversation generated by the user’s previous prompts as well. This shows that ‘prompts’ specific behaviours is not simply the signal, but the context in which it is communicated and received as well. Hence, perspectives from theatre studies can prove incredibly useful in identifying the ‘scene’ within which a human and robot are performing, and how it influences their behaviours. To illustrate this he encouraged the cohort to play around with various LLMs, showing how their internal programming influenced the ways in which they responded to written prompts. This very clearly showed that holding a conversation with an LLM is by no means ‘natural,’ but rather determined by the values and priorities that went into its creation.
Moving away from robotics and towards choreography, the final portion of the day was headed by Efrosini Protopapa, a choreographer and researcher based at The Place in London. Rather than hosting a presentation or discussion, she engaged the cohort in a practical demonstration of collaborative prompting, in which she provided two volunteers with a series of prompts on how to move around the room. Examples included walking backwards until you hit something, yelling out the name of a member in the audience, and, in the event that you did not know somebody’s name, making one up for them instead. All of this was to display the creativity that actors can express when reacting to prompts, even while abiding by its overall ‘score’ or structure. Efrosini explained that once she has practised such a choreographic score, she begins to refine it, searching for opportunities to facilitate emergent behaviour whilst simultaneously maintaining a thematic coherency.
Closing off the segment on performing arts’ perspective on prompting, Nuno Atalaia, a PhD researcher at Radboud University in Nijmegen, presented his work on developing critical AI literacy through the form of theatre. He explained that as part of his PhD research he had been helping to produce a play that thematised the associations we have with artificial intelligence. The play featured a robot who would move around the stage in a frantic and sometimes intimidating manner. In the programme of the show it was clearly stated that the robot was piloted by a remote human operator, but the vast majority of the test audience ascribed its activity to an autonomous AI, and marvelled at the supposed degree of sophistication that the technology had reached. What this example demonstrated was the ways in which robots are also capable of prompting humans: through its appearance, behaviour, and role within the drama, the robot appeared to prompt the audience into imbuing it with agency and character, unconsciously altering how they perceived and engaged with it. This proved an illuminating closing point on the discussions that had preceded it.
And so the day came to a productive close with many different perspectives having been shared. Core questions that recurred time and time again were those of where the agency of prompting lay, the role that its situational context played, and how we can maintain a balance between individual expressive freedom and an overarching structure. These are questions that Maaike and Jorrit will address in their continuing research, which will continue to integrate perspectives from a broad spectrum of disciplines.