Getting real with AR – the magic, the mundane, and the mysterious

Remember AR? After years of incubation in innovation labs, it’s still pecking away at the inside of its shell. The recent Herzog & de Meuron exhibit at the RA illustrates both the potential and pitfalls of AR/MR design. Ironic, in a way, that an exhibit so wholeheartedly about design – and the importance of model-making, concept co-creation, and process in design – also serves as a powerful illustration of the challenges of successful AR design.

Herzon and de Meuron is a Swiss-headquartered global architecture firm. Londoners might know them as the designers of the Tate Modern, the 2012 Olympics site master plan, and new Chelsea F.C. stadium. Amongst other projects. You’ve seen their work even if you didn’t realise it at the time. I went to the exhibition as I’m interested in architecture, in general, and am a fan of their work, in particular. And I also love architectural models. That H&deM had developed an AR app specifically to support the exhibition was, for me, initially just an amuse-guele. But the ways in which the app succeeded and failed turned out to be very interesting: they suggest a few principles for the AR grimoire, in the way of Do’s and Don’ts.

Magic

For me, the most exciting experience in the exhibition app was the full scale overlay on a real room sized model. The 3D model surface was plain white – when viewed with your eyes only. Viewing the model surface via your phone (and via the app), you saw a set of decorative elements superimposed on the blank contours of the model, rendered with appropriate shadows.

H&dM RA exhibit 2023 app
Screen capture of AR view of room scale model Project 377
There is no wood trim – it’s AR

This model is taken from the design process of the (beautiful) Kinderspital (Children’s Hospital) in Zurich, a project that H&deM have already put over a decade’s worth of work into. Of course, full scale models aren’t always feasible in prototyping work. But when dealing with such a diversity of stakeholders – patients, researchers, staff, benefactors – I can see they could be useful in helping to reveal preferences and needs.

However, what struck me most about this work wasn’t that it was useful. Or even that it was beautiful. It was – simply – that it was magic. It was magic to see the normal world transformed, in real time, at real scale, and believably – by simply by viewing it through the lens of my phone. It’s typical that AR apps superimpose views of digital objects over your view of the real world. Dinosaurs roam football stadia. Butterflies flit in the auditorium. Pleasantly surreal, yes, but not magic.

What made this magic as an experience was the experience of a total integration of the 3d physical world and the AR overlay. I could move from side to side, and look up and down, and walk, and the overlay correctly preserved registration of the service, and the perspective I would see it from. It really seemed to be there – and part of my physical environment, in a normal way. It was the simple normality of it that I found to be magical.

Maybe not everyone finds fake wood trim so exciting. But the principle of perceived integration with the constraints of the physical environment is something that can be applied more broadly. The dinosaur in the stadium can bump its head on the stands. The butterfly can emerge from projection mapped 2D into 3D free space. Simple AR stickers that float context free simply don’t have the same persuasive power.

Can research create magic? After years (er, decades…) of doing an entertaining variety of different types of reseearch, I’d say probably not. It can certainly improve alignment between the realisiation of a service or product and the needs of its users and investors. Also, research can – in time – be used to generate, articulate and refine design heuristics that are useful in mapping out more and less promising areas of a design space, and in explaining – in hindsight – why some things work and others don’t.

The mundane: people do the darndest things

There’s also the more low level but very useful approach to design research, of just plain old seeing what people do when they are faced with a design they can interact with. A prototype design is fine – as H&deM well know, as they have based their core architectural design process on exploration. There is very little in design that doesn’t benefit from being wind tunnelled by actual ordinary people, and their actual ordinary confusions and behaviours. This holds for apps as well as architectural designs.

So, seeing people at the exhibit trying to interact with the AR app – and failing – was very instructive. It would have been great if H&dM had been able to really check out how people used their AR app before setting it loose on the world, as the issues I saw just from watching people for a half hour or so were pretty easy to fix. I’d batch all the snags into the same bin, as discovery problems, basically:

How do you make it go?
Is this all there is?

Generic discovery challenges with the H&deM AR app

How do you make it go?

AR-enabled exhibits were indicated by a non-functional icon on the exhibit card
H&deM RA Exhibit 2023

The symbol indicating the availability of an AR layer to an exhibit was an icon of a sphere in a box. I saw one gallery-goer aim his phone at the icon on the exhibit card, expecting it to activate the AR experience. Not unreasonable – in hindsight. However, the icon was just a signal that you needed to search elsewhere in the display for a QR code card. Not everyone discovered this. Separating the actionable cue (the QR code) from the informational cue (the icon) kept the information card for an exhibit looking lean, but introduced an extra step not everyone knew to take.

Is this all there is?

Once the QR code had been located, sometimes there was an extra challenge, which was to understand whether or not you were seeing what you were supposed to, and what it was. There was one exhibit where the overlay wasn’t visible until you backed up almost into the facing display case. From what I could see of how people were using the app, most people didn’t usually discover there was any there there.

There was another exhibit, where what you saw was just a label for one of the exhibits. But of course that might have meant I missed. the point – as it seemed quite pointless. There were a few more exhibits where AR “stuff” floated above the physical exhibit but its relationship to the physical layer was unclear.

What the AR app showed was sometimes quite mysterious
H&deM RA Exhibition 2023

The relationship of the AR overlay to the physical exhibit wasn’t always obvious
H&deM RA Exhibition 2023

Part of the fun of exploring an AR layer for a display of architectural models is that there aren’t any rules about what to expect to see in the layer or how to interact with the layer. Fundamentally it’s going to be a discovery exercise. But the sheer variety of options presented in the H&deM app – often without much or any cuing – made it difficult for me to know whether I was actually seeing something “as designed” – or whether I’d managed – somehow- to miss the point entirely.

These confusions about the use of the exhibit app aren’t complicated ones – they are easy to detect by observing how people interact with the app. They are also solvable in various ways. More explicit cuing off-app could be useful- and damn the torpedoes around keeping the real world graphics lean and mean. And a more limited (and therefore predictable) repertoire of options for what the AR overlays disclose would also help people onboard to the experience more easily.

C’est magnifique, mais… qu’est que c’est que ça?

There was another digitally enhanced exhibit I enjoyed just as much as the room scale fake wood trim. I’m not sure if you’d call it AR. It probably isn’t. But it’s lovely. It’s an exhibition-wall-scale diagram of a floor plan for the Kinderspital in Zurich. I’ve never seen a plan that big. For massive projects, on a plan this big you can see features that would be invisible at other scales. I’m intrigued to think of how this might help people understand how the design works, or how to build it efficiently. (I’m not actually sure what the answer might be. Needs a bit more thinking.)

Not only is it impressively massive, but it has projection mapping overlays that change and pick out and highlight different aspects of the design. I found it interesting even without the overlays. But the overlays are certainly a delightful icing.

Grand scale visualisation of a grand scale project
H&deM RA Exhibition 2023

Multiple views highlighting different aspects of the plan are delivered via projection mapping
H&deM RA Exhibition 2023

So it’s magnificent, oui oui, but is it AR? Yes, in a way, in that it’s a combination of physical – in this case, a giant scale plan of a giant building – and digital augmentation, delivered through the various projection mapping overlays that are possible. No, in a way, as it seems like pretty straightforward static data visualisation with multiple viewing options – albeit at a truly heroic scale. One element is physical and the other digital, so it is “digitally augmented”, but the same effect could be accomplished – less heroically – with an overhead projector. I think part of my feeling that it’s not “real” AR is that it doesn’t do anything. But architectural models typically don’t, on their own, do anything. Their usefulness comes from their ability to support thinking about them, and simulating experience. The interactions it supports are open ended, and non specific. However the potential now exists to do more, and have the model participate more actively in interactions with it. With advances in adoption and capabilities of BIM (building information modelling), I think we can look forward to more and even better interactive capabilities where this came from in future.

In summary

I was expecting to find the exhibition delightful. I like architectural models and I like H&deM’s work. And indeed so I did. I wasn’t expecting the exhibition to have an AR app – or for the AR app to provide such a lot of food for thought about what works and what doesn’t. So, for me, the fact the AR view of the exhibition contained both moments of delight and things that really needed fixing was a bonus.

Deep diving into design with Spelunky

I’ve been catching up a bit with what’s up in games.  Some day I will tell you about my knitted telephone @developconf.  But today I want to share what I most enjoyed from reading Derek Yu’s book on the making of Spelunky.

This is not what this article is about

Splelunky (the book) is the tale of how a small scale, free indie game (called Spelunky) ends up becoming a massive indie hit (also called Spelunky), distributed on Steam, PSA  and XBLA,  and selling over a million copies    I ran into the book via an en passant reference from Liz England on Twitter which pulled out a great quote to do with one of the challenges of games user research and user research in general – and life in general.

https://twitter.com/lizardengland/status/877884991610404865

Yup!   People are generally pretty great at telling you when they aren’t happy, especially when you specifically give them a heads up that’s something you’re interested in hearing about.    But people are sometimes not as great at being able to attribute their unhappiness to its cause.

I think this point is a real gem.    On the basis of this  I thought the book might have other treasures in it too.

And…  it did.   The book is both easy to read and insightful.  It’s also funny and humble. I’d recommend it to anyone interested in game design.

The stories in the book aren’t recipes for success or straightforward morality tales.   Although the ending is happy,  you get the sense that despite trying to make the right choices at each progression towards publication – as we all do –  the ending might well have turned out differently.   And there is a  deeper lesson, too, when looked at as a whole.   Just as you’d expect from the author of a game which – pretty much just for the sheer mischievous joy of it – hides a deeply context-dependent eggplant.

The gems

Here are my favourite gems – starting with the one Liz England picked out.

People may know what they feel but not why

Like patients, players are often most in tune with how they feel about the problem rather than what is causing it.  Doctors and game developers, with their experience and knowledge of their fields, need to ask questions, perform tests, and eliminate possibilities in order to zero in on the correct solution.    p. 57

Users’ practice reveals the nature of a design

The next gem I found is not opaque enough to be a koan – but it is still pretty wise:

It was only after the audience entered the picture that I started to understand the reasoning behind my own designs. p. 59

I love this one.  Personally, I can’t imagine designing something and not being utterly passionately interested in how people actually use it.

There is a culture gap here.   There are many design discovery practices which – not unreasonably – talk about  all the useful insight generating things you can do ahead of actually building something, that can make the thing you build that much better.  (The Ideo field design toolkit – which I only recently discovered – is a handy resource here, even though it isn’t designed for games – and there are interesting reasons why there is an awkward stretch to map it to games, which is a story for another day).

But the practice of using real audience behaviour to understand how a design is working in practice is not so well established.   As the authors of Designing with Data point out,  taking real advantage of data from use is not always part of designers’ training.   In games, Numbers are often used measurement of the sufficiency – or insufficiency –  of the current design, rather than as source of insight into how the design operates.   For sure that can take the fun out of it.

You can’t have it all – or at least not all at once

True for games as well as just about everywhere else, this one.

We can’t have everything that we want all at once… We can’t know what to expect and also be surprised.  We can’t be free from frustration and also be challenged.  We can’t go unchallenged and also feel satisfied with our accomplishments.  Mystery, surprise, tension, challenge and a real sense of accomplishment always come at the cost of feeling uncomfortable.   p. 86

I think this tension between frustration and satisfaction is what game ‘balance’ is really about.   It isn’t fun to shoot fish in a barrel.    There needs to be a rhythm of ebb and flow between striving and accomplishment, and between exploration and the joy of discovery.

You gotta trust yourself (and your players)

As Etta James says.

It is not my intention to quote the whole book to you but the next gem follows directly after the one I just picked up.    But it’s a bit like finding great stones on a beach: it’s difficult to stop at just one.

It’s difficult to pick just one gem

So, moving right along to the next sentence after the previous one I quoted:

The best games come out of a mutual respect between the creator and the player… the creator must trust in the curiousity and abilities of their players.  Continuously interrupting play to steer players with direct text message and other obvious hints not only infantalises them but it also reveals the creator’s insecurity in their ability to design games…The “joy of discovery” is one of the fundamental joys of play itself.  p. 86-87

Indifference can be a virtue – but don’t take it too far

Spelunky takes a kind of extreme stance on the question how much help a design should provide for onboarding:

Unfortunately, Spelunky’s core ideas defy explanation to new players, because the game’s indifference to them is so much of its appeal.   p.  102

As Sir Humphrey Appleby of Yes Minister  would say that – is a “courageous” decision. (And one that their Microsoft XBox producer tried to work with them to mitigate.)

It’s likely that this stance creates more casualties amongst the new user corps than a more gradual onramp.   But it’s also possible that those who persist end up being more committed.

I’d be interested to know the functions through which initial eventually successful effort is related to long term commitment.    As a psychologist, I could make up a number of different answers to this – but I’d prefer to ask the question and explain the answer after I know what it is.

Choose the committed and the committed will choose you

The way Yu explains his design choices,  it sounds as though he was pretty happy for  the game to select for the type of users who valued the type of experience he wanted to provide for them:

With Spelunky, I was only willing to go so far to offer new players a hand.  Beyond that point, I felt like it would hurt the experience for players who were more eager for  challenge.  And when you have to choose between those two groups, it’s not a hard choice at all. p. 108

There is for sure a tension between providing assistance and pleasing those players who want challenge.   But it seems to me there are few different escape routes:

• making the path to skill acquisition a real and organic part of the designed experience  – rather than something that’s jury rigged afterwards with some nasty annoying pop ups
  • Yu does give a big nod to the excellence of onboarding in Super Mario – the challenge is in how to generalise this to other games
• adjusting the challenge level dynamically, so that players are faced with a degree of challenge that is likely to be more satisfying for them
  • This moves the goal posts from finding a ‘good enough’ setting for everyone, to finding ‘good enough’ personalisation – which is always going to be not good enough for some
  • This is clearly tricky for PvP games, but can be useful in initial PvE stages
• making ‘failure’ more enjoyable, by providing a degree of reward – such as exciting visual effects – for behaviour that approximates the behaviour you want to encourage, even if it isn’t ‘winning’
  • as discussed, near misses can – under some conditions – deliver almost the same thrill as a wins
• changing the intent of the interaction to be less about winning and more about other aspects of the experience
  • this is partly about genre, but even within genres in which winning is important,  it is about making the game experience non-binary -the most important aspect is not necessarily about win or loss

First impressions matter

Yu realises – with hindsight – that they didn’t make it clear enough to prospective players what the game experience was supposed to be about, and they should have made the screen shot and demo promos more enticing and informative:

Spelunky’s screenshots, while packed with details, look mostly like a lot of rocks and dirt, while the interesting aspects of its randomization are hard to get across in even a few play sessions, let alone a non-interactive trailer.

Of course, beneath Spelunky’s unassuming surface are hidden big secrets and shiny treasures.  p. 158

Tough one, this.    How to communicate a mystery and keep it mysterious?   I think the strongest moral to be mined from this story is that creating draft marketing approaches should be a part of every product proposal.

Difficulty is important – but the best progression is non-linear and rhythmic

I usually wonder about why designs have the difficulty progressions that they do.    Yu gives his thoughts on it here:

The more I play and create games, the less convinced I am that the difficulty of games should be thought of in terms of a linear or exponential ramp upwards where, as the player gets stronger, you need to make the opposition increase proportionally in strength….[there is] something futile and perhaps nihilistic about endlessly cranking a single knob that goes from easy to hard.  Rather I believe it makes more sense to think about difficulty in terms of the game’s overall pacing.   Difficulty should ebb and flow, and make room for other aspects of play. p. 178

If you draw a comparison to pacing and tension in narrative, it is easy to see that establishing a rhythm – and, at times,  violating it-  is potentially a very strong design trope that resonates with how we experience meaning.    I haven’t heard anyone express this as clearly as Yu has.

The deeper lesson

What is the deeper lesson revealed by these gems?  The stories Yu tells are interesting because of the the motivations he reveals for making the design choices he makes.    He has a great eye for analysing and articulating what makes a design tick, and you get the sense that his heart is in the right place, always.   To me,  what is striking about Yu’s approach is his attention, at all times,  to viewing the design of the game through the lens of the player’s possible experience of it.    That is the real eggplant that is hidden in plain sight.

 

 

 

 

 

Not all zones are created equal: player pleasure and player anaesthesia

I’ve been chasing the idea that, as ‘pure play’ chance experiences,  machine slots might embody interesting design patterns about the use of chance, ones that are generalisable to other genres of game designs and other types of player experiences.  To this end,  I’ve been mining  Natasha Schüll’s prize winning ethnological study, Addiction by Design for what it says about the player experience of chance.

What’s the attraction for players?  According to Schüll, people play machine slots in order to change how they feel.

“gamblers… act upon themselves through gambling devices with a goal of regulating their own affective states”,  p.20

But what is the state people are playing in order to achieve?  Players sometimes refer to it as ‘the zone’:

“The speed is relaxing,” said Lola… “It’s not exactly excitement; it’s calm, like a tranquilizer.  It gets me into the zone.” p. 54

What is ‘the zone’?  The term is often used in a way that is closely related to Csikszentmihaly’s concept of ‘flow’ – an optimal state in which people feel a sense of total absorbtion in their activity.

But clearly there’s zone and there’s zone.    The zone reached by machine slot players seems to be a zone of being zoned out, rather than the zone of peak human experience.   The experience seems to produce a kind of anaesthesia.

“The solitary, uninterrupted process of machine play…tends to produce a steady, trancelike state that ‘distracts from internal and external issues’ such as anxiety, depression, and boredom.” p. 17, Fn 89

“it is not the chance of winning to which they become addicted; rather what addicts them is the world-dissolving state of subjective suspension and affective calm they derive from machine play.” p. 18

“…the “zone” – the elusive point of absorption, beyond contingency, that machine gamblers perpetually seek…  …[is] at once ‘safe’ and ‘precarious’ – a gentle seesaw of play credit that is mirrored in a gentle seesaw of player affect, both of which might at any instance lose momentum and come to a standstill.  The zone state is attainable only at the threshold where rhythm holds sway over risk, comfort over perturbation, habituation over surprise.”  p 135 Fn 93

This is quite different from the idea that gambling is in some way exciting.

Source: Wikimedia

Schüll’s discussions with industry insiders suggest that the most successful slot machines, successful in the sense of producing maximum revenue, are those with low volatility – a stream of small payouts and small wins – but which result in players who spend a long time on device.  Players do differ in risk preferences – but the most profitable ones are those who spend time in large amounts, spend money in small increments, and win steadily but not dramatically.     Large payouts can distract players and disrupt them so much that they fall out of their dreamlike state  – and perhaps even cash out, walking away with their winnings.

“As the journalist Marc Cooper remarked in 2005, “the new generation of gambling machines has, predictably, produced a new generation of gambling addicts: not players who thive on the adrenaline rush of a high-wager roll of the dice or turn of  a card but, rather, zone-out ‘escape’ players who yearn for the smooth numbness produced by the endlessly spinning reels.”  p. 128 Fn 79

The question of why playing on low-volatility machines seems to induce a trancelike state is not much analysed, although Schüll provides eloquent descriptions of “rhythm over risk”, and”a gentle seesaw of play credit that is mirrored in a gentle seesaw of player affect” (p.135).   This question is something I’ll turn to in a post or two, after I look at what Schüll has to say about the designs which produce the player experiences she catalogues.