All posts filed under “Energy

Thinking About Things That Think About How We Think

Cross-posted from the Environments Studio IV blog, Carnegie Mellon School of Design

We often hear the phrase ‘intelligent environments’ used to describe spaces in which technology is embedded, in the form of sensors, displays, and computational ability. This might be related to Internet of Things, conversational interfaces or emerging forms of artificial intelligence.

But what does ‘intelligence’ mean? There is a long history of attempts to create artificial intelligence — and even to define what it might mean — but the definitions have evolved over the decades in parallel with different models of human intelligence. What was once a goal to produce ‘another human mind’ has perhaps evolved into trying to produce algorithms that claim to ‘know’ enough about how we think to be able to make decisions about us, and our lives. What we have now in ‘intelligent’ or ‘smart’ products and environments is one particular view of intelligence, but there are others, and from a design perspective, designing our interactions with those ‘intelligences’ as they evolve is likely to be a significant part of environments design in the years ahead. Is there an opportunity for designers to explore different kinds of interactions, different theories of mind, or to envisage new forms of intelligence in environments, beyond the dominant current narrative?

Building on the first two projects’ treatment of how humans use environments, and how invisible phenomena can be materialized, for this project the brief was to create an environment in which visitors can experience different forms of ‘intelligence’, through interacting with them (or otherwise experiencing them). The project was not so much about the technical challenges of creating AI, but about the design challenges of enabling people to interact with these systems in everyday contexts. So, quick prototyping and simulation methods such as bodystorming and Wizard of Oz techniques were entirely appropriate—the aim was to provide visitors to to the end-of-semester exhibition (May 4th, 2017) with an experience which would make them think, and provoke them to consider and question the role of design in working with ‘intelligence’.

More details, including background reading, in the syllabus.

We considered different forms of behaviour, conversation, and ways of thinking that we might consider ‘intelligent’ in everyday life, from being knowledgeable, to being able to learn, to solving problems, to knowing when not to appear knowledgeable, or not to try to solve problems. If one is thinking about how others are thinking, when is the most intelligent thing to do actually to do nothing? Much of what we considered intelligent in others seemed to be something around adaptability to situations, and perhaps even adaptability of one’s theory of mind, rather than behaving in a fixed way. We looked at Howard Gardner’s multiple intelligences, with the ideal of interpersonal, or social, intelligence being one which seemed especially interesting from a design and technological point of view — more of a challenge to abstract into a set of rules than simply demonstrating knowledge, a condition where the feedback necessary for learning may not itself be clear or immediate, and where the ability to adjust the model assumed of how other people think is pretty important. How could a user give social feedback to a machine? Should users have to do this at all?

Each of the three resulting projects considers a different aspect of ‘intelligence’ from the perspective of people’s everyday interaction with technologies in the emotionally- and socially-charged context of planning a party or social gathering, and some of the issues that go with it.

Gilly Johnson and Jasper Tom‘s SAM is an “intelligent friend to guide you through social situations”, planning social gatherings through analysing interaction on social networks, but which also has Amazon Echo-like ordering ability. It’s eager to learn—perhaps too eager.

Ji Tae Kim and Ty Van de Zande‘s Dear Me, / Miyorr takes the idea that sometimes intelligence can come from not saying anything — from listening, and enabling someone else to speak and articulate their thoughts, decisions, worries, and ideas (there are parallels with the idea of rubber-duck debugging, but also ELIZA). In this case, the system is a kind of magic mirror that listens, extracts key phrases or emphasised or repeated ideas, and (in conjunction with what else it knows about the user), composes a “letter to oneself” which is physically printed and mailed to the user. Ty and Ji Tae also created a proof-of-principle demo of annotated speech-detection that could be used by the mirror.

Chris Perry‘s Dialectic is an exploration of the potential of discourse as part of decision-making: rather than a single Amazon Echo or Google Home-type device making pronouncements or displaying its ‘intelligence’, what value could come from actual discussion between devices with different perspectives, agendas, or points of view? What happens if the human is in the loop too, providing input and helping direct the conversation? If we were making real-world decisions, we would often seek alternative points of view—why would we not want that from AI?

Chris’s process, as outlined in the demo, aims partly to mirror the internal dialogue that a person might have. Pre-recorded segments of speech from two devices (portrayed by paper models) are selected from (‘backstage’) by Chris, in response to (and in dialogue with) the user’s input. There are parallels with “devices talking to each other” demos, but most of all, the project reminds me of a particular Statler and Waldorf dialogue. In the demo, the devices are perhaps not seeking to “establish the truth through reasoned arguments” but rather to help someone order pizza for a party.

Environments Studio: Materializing the Invisible

Timelapse of studio

Timelapse of studio, by Jasper Tom

In Materializing the Invisible, we considered invisible and intangible phenomena—the systems, constructs, relationships, infrastructures, backends and other entities, physical and conceptual, which comprise or influence much of our experience of, and interaction with, environments both physical and digital. ‘The invisible’ here is potentially everything from how the building’s heating system works, to the algorithms behind targeted ads, to who’s friends with whom, to where corruption is occurring in government, to where your IoT fridge sends the data it collects, to people’s mental imagery of time, to the electricity use of devices, to networks of cameras and sensors, to how political decisions are made. It also potentially includes things that happen at scales or in dimensions we can’t directly comprehend, from planetary processes such as climate, to the interaction of electromagnetic fields, to the microscopic. And things that happen, that enable day-to-day functioning of our lives, but we don’t know much about. Where does our food come from? Where does our waste water go? What route did that package take to get to us?

The process of revealing the invisible can improve understanding, help people explore their own thinking and relationships with these complex concepts, highlight problems, power structures and inequalities, reveal hidden truths, connect people better to the world around them, and enable people to act. It is not necessarily about visualizing the invisible—it can be about making it audible, tangible, smellable, or otherwise experienceable: we explored techniques from fields including data visualization, sonification, data physicalization, ubiquitous computing, tangible interaction, analog computing, qualitative displays, and the study of synaesthesia to create ways to materialize these invisible phenomena.

More details, including background reading, in the syllabus.

As a starting exercise we examined some ‘invisible’ and unknown things within the building itself (Margaret Morrison Carnegie Hall), noting questions and ideas with Post-It notes in situ. These ranged from questions about who has access to certain rooms or controls, to what some of the controls are in the first place. There were also traces of action and use—patterns which might be invisible in the sense of not being paid attention to, but nevertheless present in the use of the building.

The class project was to choose a phenomenon which is ‘invisible’ within a physical, digital or hybrid environment, find a way of getting access to it, and design and build / make / create a way of materializing the phenomenon, making it accessible to people more widely. As a group we brainstormed different phenomena which might be investigable, and possible forms of representation.

Ji Tae Kim’s project Whitespace looked at the invisible aspects of communication in text messaging, following on from his previous project Fear of Missing Out. Whitespace explores ways to materialize and express “rich contextual and verbal cues” through “an intuitive extension to instant messaging”. Working prototypes used copper tracks, Bare Conductive ink and Touch Board, and Arduino.

Jasper Tom and Chris Perry‘s project Kairos examined “an invisible phenomenon ingrained in everyday life”: the passage of time in a space, specifically around working at a desk. The question “Where did the time go?” and the idea of desk legacy, the patterns of use left by a previous user of the desk in a shared workspace, informed by analysis of timelapse video of the studio, came together with inspirations such as Daniel Rozin’s Wooden Mirror, MIT Tangible Media Group projects such as Daniel Leithinger’s work, and Tempurpedic foam, to create a desk surface which could ‘play back’ the patterns of how it had been used, via an interface using wooden blocks. A working prototype of part of the surface used Arduino and servo motors to demonstrate the effect.

One interesting aspect discussed during Jasper and Chris’s presentation was how while evidence of physical work is often obvious in space, such as a painter’s palette, the evidence of digital work is often invisible—a slightly worn keyboard, perhaps, but little else.

Gilly Johnson and Ty Van de Zande worked together to explore aspects of human movement (dance and exercise), and the related issues of hydration and focus. Focus + Movement proposed a color-changing bodysuit which could work together as part of a system with a water bottle, both to make the invisible patterns visible, and to enable reflection. Gilly and Ty captured movement by dancers using a Kinect, connected to Max MSP, and then simulated the body suit via After Effects.

Environments Studio: Design, Behavior and Social Interaction

Studying Pittsburgh's Greyhound Bus Station: Jasper Tom
Jasper Tom investigated patterns of people’s behavior in Pittsburgh’s Greyhound Bus Station

In this short introductory unit, we looked at ways in which the design of environments, and features within them, affects people’s behavior and interaction with each other. Design influences what people do, but often the ‘links’ are invisible or only apparent by their effects. Or, we notice them in passing, but do not take time to reflect on them or draw parallels across situations.

Studying the fear of missing out with messaging: Ji Tae Kim
Ji Tae (Joseph) Kim examined how the design of messaging and social media leads to ‘fear of missing out’ through unplugging himself for a week

As designers pioneering new approaches to creating environments for human experience, cultivating a kind of ‘hypersensitivity’ to noticing—and learning from—the ways in which design and behavior interact can be part of developing the attention to detail which will serve you well professionally. Details of the unit in the syllabus.

Studying a pedestrian crossing: Chris Perry
Chris Perry observed the different ways in which people use a pedestrian crossing at the entrance to CMU, and how the design affects those actions

We started with quick observation exercises aimed at developing (or refreshing) a capacity for noticing, for paying attention to the ways in which people and environments affect each other. We looked around campus for instances of points of confusion, unintended uses, constraints, and disobedience in physical environment settings, and discussed how these effects manifest in different ways—what could we find? (Photos here by Chris Perry, Gilly Johnson, Jasper Tom, Ty Van de Zande and Dan Lockton.)

We examined ideas around how environments influence people, and are in turn influenced, both physically and digitally, from thigmotaxis to stigmergy, shearing layers and pace layers, fundamental attribution error and design for behavior change. We also thought about the practice of observation, noticing and deconstruction of people’s actions in different ways, and in different levels of detail. The project brief was around designing a way to do research in this field—designing a ‘probe’ rather than a solution to a problem:

  • Choose a situation where ‘design’ seems to be affecting people’s behavior in an environment (physical or digital)
  • Find a way of studying what’s going on—what patterns exist? In what different ways are people’s behavior affected?
  • Visualize (or otherwise communicate) what you find
  • (optional: suggest ways things could be different, if you feel they need to be)
  • Keep a blog of your process (photos, sketches, notes)

Here are the projects:

Comparing a coffee shop and a tea shop: Gilly Johnson

Gilly Johnson compared structural and systemic aspects of the atmosphere and experience in Coffee Tree Roasters in Shadyside, and Dobra Tea in Squirrel Hill, including the layout and spatial division, and emerging themes such as service and trust: full details of the project.

Fear of Missing Out: Ji Tae Kim

Ji Tae Kim: Fear of Missing Out

Ji Tae Kim examined how the design of messaging and social media leads to ‘fear of missing out’ through unplugging himself for a week: full details of the project.

Greyhound Station: Jasper Tom

Jasper Tom investigated how the design of Pittsburgh’s Greyhound Bus Station influences patterns of people’s behavior: full details of the project.

Managing information across environments: Ty Van de Zande

Ty Van de Zande looked at how people manage information such as to-do lists across physical and digital environments, and developed a framework for investigating this in a structured way: more details of the project.

How to Cross the Road: Chris Perry

Project 1
Chris Perry observed the different ways in which people use a pedestrian crossing at Morewood Avenue and Forbes Avenue, at the entrance to CMU, and how the design affects those actions: more details of the project.

What does energy look like? Drawing Energy book now available

Drawing Energy book

Some news from the SusLab project:

Last year, Flora Bowden blogged about our investigation of people’s perceptions of ‘energy’–how do people visualise, or think about, what is for the most part an abstract, invisible concept?

A book detailing our research, Drawing Energy, is now available to download or order:

Bowden, F., Lockton, D., Gheerawo, R. and Brass, C. (2015). Drawing Energy: Exploring perceptions of the invisible. London: Royal College of Art. ISBN 978-1-910642-10-8. Editor: Rama Gheerawo (PDF)

Drawing Energy describes a drawing-based research project undertaken by the Royal College of Art as part of SusLabNWE (2012-15). The project explored people’s perceptions of energy, by asking them to write, draw or illustrate their thoughts and reactions to the question ‘What does energy look like?’ Over 180 members of the public took part in the process.

The larger SuslabNWE study saw 11 partners from Germany, the Netherlands, Sweden and the UK come together to understand and investigate energy use in the home. At the Royal College of Art in the UK, we looked at bringing together two ideals and practices around inclusive design and sustainability. Both often have different starting points and deal with different scales. Inclusive design usually focuses on people’s needs and capabilities at the domestic scale, while sustainability embraces complexity and systems thinking, addressing systemic change.

Drawing Energy negotiates a space between the two, bringing together people’s aspirations and perspectives with the context of socio-political mandates and changing infrastructure or technologies. The study also moves beyond the idea of purely functional research (such as numerically measuring energy use) to depict the less tangible area of how people relate to energy in a visual, literal or metaphorical way — it takes us from data ‘performance’ through to human ‘perception’. The work represented in this collection builds on a history of using drawing as a tool for research and as a way to enable people to express their ideas and imagination fully.

We hope you appreciate this publication, whether you see it as a strategy within design research, or simply enjoy it for the rich and varied artwork that represent the public’s views of energy.

Drawing Energy: Exploring Perceptions of the Invisible was designed by Hannah Montague and edited by Rama Gheerawo.

Drawing Energy - gallery

Drawing Energy and Powerchord at the London Design Festival 2014

The latest Powerchord prototype in use..

The London Design Festival is a huge event taking place across London from today (13th) for the next couple of weeks, and we’re proud to say that two of our SusLab mini-projects, Drawing Energy and Powerchord, are featured, as part of two exhibitions.

drawing1  forblog20140910_191005

V_and_A_DiliffV&A Digital Design Weekend: 20 & 21 September

At the Victoria & Albert Museum, the 2014 V&A Digital Design Weekend, on Saturday 20th and Sunday 21st September, from 10.30am to 5pm, is a fantastic transformation of the V&A into “one big workshop… where visitors come together with artists and designers to discuss and think about objects, making and working collaboratively.” We’re honoured to be presenting our work in some very talented company, including James Bridle, Tine Bech and Bristol’s REACT Hub.

You can take part in Drawing Energy–please come along to see the collection, and create your vision of energy!–and play with Powerchord, and contribute to shaping the next stage of its development.

Dyson exterior_Helene BinetBreaking Through: New projects from the Helen Hamlyn Centre for Design: 15-25 September

Our projects are also featured in the Helen Hamlyn Centre’s own exhibition and symposium, taking place in the Dyson Building at RCA Battersea, each day from the 15th to 25th September, from 10am to 5.30pm. Breaking Through “demonstrates how emerging ideas can shape alternative futures in areas as diverse as energy use, office life and ageing populations–when ethnographic research and people-centred design are considered in tandem. From designs for a new London taxi to innovations in healthcare and developments for digital communities, there is an emphasis on user push rather than technology pull as the driving force to improve people’s lives through design.”

We’ll be showing the results of Drawing Energy so far, and you can also play with Powerchord by using appliances and hearing how it responds.



About the projects

Drawing Energy (What Does Energy Look Like?) is a drawing project led by the Royal College of Art to explore how people imagine and think about energy. It is part of the wider European SusLabNWE project that is exploring energy use in the home.

Over the past year we have asked over one hundred people – students, children, academics, energy experts, designers and members of the public – to draw for us what they think energy looks like.

The project is described in a paper presented by Flora Bowden at the International Association of Empirical Aesthetics 2014 Congress in New York:

  • Bowden, F., Lockton, D., Gheerawo, R. & Brass, C. (2014). ‘Drawing Energy: Exploring the Aesthetics of the Invisible’. IAEA Congress 2014: Congress of the International Association of Empirical Aesthetics, 22-24 August 2014, New York (paper PDF).

Powerchord is a prototype data sonification system, under development, which turns near-real time electricity monitoring, of multiple household appliances, into sound. The concept was developed from ideas suggested by householders during co-creation sessions as part of the SusLabNWE project.

The prototype uses the ‘guts’ of a CurrentCost energy monitor, connected to an Arduino which reads the XML data stream from the monitor and maps the power levels to particular tracks, played using a WAV Trigger. The current iteration uses birdsong, of different intensities, from recordings at

The project is described in a paper presented by Dan Lockton at the SoniHED Conference on Sonification of Health and Environmental Data, 12 September 2014, York:

  • Lockton, D., Bowden, F., Brass, C. & Gheerawo, R. (2014). ‘Bird-wattching: exploring sonification of home electricity use with birdsong’. SoniHED — Conference on Sonification of Health and Environmental Data, 12 September 2014, York (paper PDF).




Introducing The Story Machine: Part 1


Friction in integrating digital storytelling into community activities

For some community groups, the use of technology and digital media is built into the work they do–for example, the Wards Corner Community Coalition‘s very successful use of Stickyworld and social media to tell their story, integrated into the organisation of a whole range of community events.

For other groups, it’s something they would like to weave better into their activities, but which can be difficult to do strategically, for reasons including not just the diversity of skills within the group, but the time and organisational requirements for volunteers to coordinate everything alongside actually running the events. The challenges are rarely ones around motivation or engagement–volunteers are almost by definition engaged in what they are doing–nor indeed, in an era where many group members will regularly use mobile phones, social media and so on themselves, it is not necessarily solely a trite ‘digital divide’ problem.

Instead, in some cases it seems to come down to something like process friction: the route from someone taking photos or video or talking about what they’re doing at a group event, to those photos or videos or stories (in whatever form) being turned into an ‘output’ such as a blog post, an update to the group’s website, or being shared more widely beyond the group (perhaps even to promote it), can be strewn with gaps and bumps and contingencies which make it slow or fractured, from technology incompatibility to issues with user accounts to mental models of how systems work. The route probably involves multiple people, each with his or her own interests, skills and time commitments.

Read More