Refik Anadol’s animations – meaningful moving parts recombine to generate art

Refik Anadol has made his name by producing super scale animations that rely on machine learning algorithms, a large cast of studio assistants, and meanings which are borrowed from the raw material consumed by the algorithms. His invited exhibition at the Serpentine in London, entitled Echoes of the Earth: Living Archive, takes over the whole of the Serpentine North Gallery, is totally eye-catching. The TL;DR is: go. Quickly. It closes April 7 2024.

The images in the exhibit range from literal-seeming to highly abstract, and the animations which sequence them flow from one mode to the other. Everything is synthetic – the images are derived from applying machine learning algorithms to very large datasets of other, real images. On which more later. The results sometimes seem literal – as with this synthetic coral:

Coral-like image
Refik Anadol Echoes of the Earth: Living Archive
Serpentine Gallery London 2024

Sometimes the images produced by the operation of the models are highly abstract, as in this candy coloured image, which seems fluid-like and tissue-like, but does not obviously represent any particular thing:

Abstract multi-coloured particle-based image
Refik Anadol Echoes of the Earth: Living Archive
Serpentine Gallery London 2024

The animations often looks like they are driven by a fluid dynamics simulation. They proceed at a comfortable pace. The colour palettes are harmonious, the surface lighting is gentle. Because of the palette and palette transitions chosen, the sequences have a tendency to look “nice”.

The visuals aren’t always explicitly representational, although the models which generate them use real image captures as inputs. But even when they aren’t interpretable as specific things, the outputs all have a biomorphic flavour to them, reflecting the characteristics of their inputs. Some sequences evoke associations of clouds of organic tissues in perpetual motion and transition. As indeed real weather and real tissues are.

Harmoniously coloured abstract image sequence – reminiscent of changes in weather and in living tissues
Refik Anadol Echoes of the Earth: Living Archive
Serpentine Gallery London 2024

Sometimes, the images flow from the purely abstract towards representational, then back off again. This gives the feeling of life-forms being created – and dissolving. The pace is slow enough that it doesn’t feel frenetic, rather graceful.

Synthetic creatures emerge from more abstract images – and dissolve again
Refik Anadol Echoes of the Earth: Living Archive
Serpentine Gallery London 2024

As is the way with the state of the art around synthetic ML-based images of beings, there are times when the synthesis creates “errors”, as seen in this bird with two beaks, and a foot whose claw doesn’t grip the surface:

Model outputs are driven by visual similarity not physical constraints – check the beak(s) and claws
Refik Anadol Echoes of the Earth: Living Archive
Serpentine Gallery London 2024

What are we to make of this? The images being synthesised and streamed together explore the latent space of the models – creating images of possible beings. The models are based on very very large datasets of actual images, but the model outputs are novel. Sometimes they create things which we instinctively understand would be unable to exist in reality.

In a way, these model “errors” serve to highlight the preciousness of actual life and beings. Not all corners of this very effortfully produced and highly dimensioned latent space are actually habitable. Visual representation is fundamentally a surface phenomenon, with only imperfect connections to mechanism.

Sometimes what is created is beautiful, as in this sequence of a lotus turning into a lotus:

The model dreams impossible dreams – of a lotus within a lotus
Refik Anadol Echoes of the Earth: Living Archive
Serpentine Gallery London 2024

There is treasure to be found in mining latent spaces, and Anadol and his team have certainly found some. Dreams, even deep ones, do sometimes turn to nightmares though. The potential for monstrosity, as well as beauty and impossibility, is equally latent in the technique.

The story of how the base data images are sourced, and how the resultant images are synthesised is very much a part of the work. Which is very much on trend. This display, which is part of the exhibit, dramatises the multitudes of base data images that went into creating the models:

The origin story is part of the work
Refik Anadol Echoes of the Earth: Living Archive
Serpentine Gallery London 2024

The works are made possible by the collaboration of an impressive number and range of institutional collaborators, such as the Natural History Museum, and the Smithsonian, who have collectively contributed millions of images from their own collections to be used as training data. Many of the works use what Anadol calls their Large Nature Model, a play on Large Language Models (LLMs) of ChatGTP fame. (This model is, apparently, open sourced – although, interestingly, I am not the only one who can’t find it on GitHub.) All the works seem to use NVidia StyleGan2 as the underlying model framework.

This is the first of Studio Anadol’s installations that I’ve seen in person. In concept and style it has strong links to his blockbuster 2022-23 installation, Unsupervised, at MOMA in NYC. Unsupervised was a never-ending generative machine learning-based animation using, as its raw material, 200 existing works in MOMA’s collection. This use of freely contributed material and generative ML-driven animations are common threads tying the the two exhibits together.

The MOMA installation was accompanied by righteous chat about it being “on the blockchain” – but times change. The Serpentine exhibition also surfs the zeitgeist, but doesn’t mention crypto. Now the force being harnessed is the sharp wind of eco-anxiety. This is somewhat ironic, given the compute power – and energy – which must have been involved in creating the exhibition’s works. Anadols’ explanation of the intent and process behind the highly collective work – of which the Serpentine exhibition is just the first public manifestation – was launched at Davos last year: you can check out the promo here.

So? It’s an eyeful, in a good way, and it represents boss levels of effort and collaboration. But the outputs – although they are “nice” – make me melancholy. In exploring the latent corners of a many thousand dimensioned model space built from a multitude of images of living things – what have we learned? That actual, working life is precious in its uniqueness? It’s a bold exploration, and it’s certainly uncovered some beauty, but I don’t feel the exploration has helped us, collectively, arrive anywhere. Yet.

Deep diving into design with Spelunky

I’ve been catching up a bit with what’s up in games.  Some day I will tell you about my knitted telephone @developconf.  But today I want to share what I most enjoyed from reading Derek Yu’s book on the making of Spelunky.

This is not what this article is about

Splelunky (the book) is the tale of how a small scale, free indie game (called Spelunky) ends up becoming a massive indie hit (also called Spelunky), distributed on Steam, PSA  and XBLA,  and selling over a million copies    I ran into the book via an en passant reference from Liz England on Twitter which pulled out a great quote to do with one of the challenges of games user research and user research in general – and life in general.

https://twitter.com/lizardengland/status/877884991610404865

Yup!   People are generally pretty great at telling you when they aren’t happy, especially when you specifically give them a heads up that’s something you’re interested in hearing about.    But people are sometimes not as great at being able to attribute their unhappiness to its cause.

I think this point is a real gem.    On the basis of this  I thought the book might have other treasures in it too.

And…  it did.   The book is both easy to read and insightful.  It’s also funny and humble. I’d recommend it to anyone interested in game design.

The stories in the book aren’t recipes for success or straightforward morality tales.   Although the ending is happy,  you get the sense that despite trying to make the right choices at each progression towards publication – as we all do –  the ending might well have turned out differently.   And there is a  deeper lesson, too, when looked at as a whole.   Just as you’d expect from the author of a game which – pretty much just for the sheer mischievous joy of it – hides a deeply context-dependent eggplant.

The gems

Here are my favourite gems – starting with the one Liz England picked out.

People may know what they feel but not why

Like patients, players are often most in tune with how they feel about the problem rather than what is causing it.  Doctors and game developers, with their experience and knowledge of their fields, need to ask questions, perform tests, and eliminate possibilities in order to zero in on the correct solution.    p. 57

Users’ practice reveals the nature of a design

The next gem I found is not opaque enough to be a koan – but it is still pretty wise:

It was only after the audience entered the picture that I started to understand the reasoning behind my own designs. p. 59

I love this one.  Personally, I can’t imagine designing something and not being utterly passionately interested in how people actually use it.

There is a culture gap here.   There are many design discovery practices which – not unreasonably – talk about  all the useful insight generating things you can do ahead of actually building something, that can make the thing you build that much better.  (The Ideo field design toolkit – which I only recently discovered – is a handy resource here, even though it isn’t designed for games – and there are interesting reasons why there is an awkward stretch to map it to games, which is a story for another day).

But the practice of using real audience behaviour to understand how a design is working in practice is not so well established.   As the authors of Designing with Data point out,  taking real advantage of data from use is not always part of designers’ training.   In games, Numbers are often used measurement of the sufficiency – or insufficiency –  of the current design, rather than as source of insight into how the design operates.   For sure that can take the fun out of it.

You can’t have it all – or at least not all at once

True for games as well as just about everywhere else, this one.

We can’t have everything that we want all at once… We can’t know what to expect and also be surprised.  We can’t be free from frustration and also be challenged.  We can’t go unchallenged and also feel satisfied with our accomplishments.  Mystery, surprise, tension, challenge and a real sense of accomplishment always come at the cost of feeling uncomfortable.   p. 86

I think this tension between frustration and satisfaction is what game ‘balance’ is really about.   It isn’t fun to shoot fish in a barrel.    There needs to be a rhythm of ebb and flow between striving and accomplishment, and between exploration and the joy of discovery.

You gotta trust yourself (and your players)

As Etta James says.

It is not my intention to quote the whole book to you but the next gem follows directly after the one I just picked up.    But it’s a bit like finding great stones on a beach: it’s difficult to stop at just one.

It’s difficult to pick just one gem

So, moving right along to the next sentence after the previous one I quoted:

The best games come out of a mutual respect between the creator and the player… the creator must trust in the curiousity and abilities of their players.  Continuously interrupting play to steer players with direct text message and other obvious hints not only infantalises them but it also reveals the creator’s insecurity in their ability to design games…The “joy of discovery” is one of the fundamental joys of play itself.  p. 86-87

Indifference can be a virtue – but don’t take it too far

Spelunky takes a kind of extreme stance on the question how much help a design should provide for onboarding:

Unfortunately, Spelunky’s core ideas defy explanation to new players, because the game’s indifference to them is so much of its appeal.   p.  102

As Sir Humphrey Appleby of Yes Minister  would say that – is a “courageous” decision. (And one that their Microsoft XBox producer tried to work with them to mitigate.)

It’s likely that this stance creates more casualties amongst the new user corps than a more gradual onramp.   But it’s also possible that those who persist end up being more committed.

I’d be interested to know the functions through which initial eventually successful effort is related to long term commitment.    As a psychologist, I could make up a number of different answers to this – but I’d prefer to ask the question and explain the answer after I know what it is.

Choose the committed and the committed will choose you

The way Yu explains his design choices,  it sounds as though he was pretty happy for  the game to select for the type of users who valued the type of experience he wanted to provide for them:

With Spelunky, I was only willing to go so far to offer new players a hand.  Beyond that point, I felt like it would hurt the experience for players who were more eager for  challenge.  And when you have to choose between those two groups, it’s not a hard choice at all. p. 108

There is for sure a tension between providing assistance and pleasing those players who want challenge.   But it seems to me there are few different escape routes:

• making the path to skill acquisition a real and organic part of the designed experience  – rather than something that’s jury rigged afterwards with some nasty annoying pop ups
  • Yu does give a big nod to the excellence of onboarding in Super Mario – the challenge is in how to generalise this to other games
• adjusting the challenge level dynamically, so that players are faced with a degree of challenge that is likely to be more satisfying for them
  • This moves the goal posts from finding a ‘good enough’ setting for everyone, to finding ‘good enough’ personalisation – which is always going to be not good enough for some
  • This is clearly tricky for PvP games, but can be useful in initial PvE stages
• making ‘failure’ more enjoyable, by providing a degree of reward – such as exciting visual effects – for behaviour that approximates the behaviour you want to encourage, even if it isn’t ‘winning’
  • as discussed, near misses can – under some conditions – deliver almost the same thrill as a wins
• changing the intent of the interaction to be less about winning and more about other aspects of the experience
  • this is partly about genre, but even within genres in which winning is important,  it is about making the game experience non-binary -the most important aspect is not necessarily about win or loss

First impressions matter

Yu realises – with hindsight – that they didn’t make it clear enough to prospective players what the game experience was supposed to be about, and they should have made the screen shot and demo promos more enticing and informative:

Spelunky’s screenshots, while packed with details, look mostly like a lot of rocks and dirt, while the interesting aspects of its randomization are hard to get across in even a few play sessions, let alone a non-interactive trailer.

Of course, beneath Spelunky’s unassuming surface are hidden big secrets and shiny treasures.  p. 158

Tough one, this.    How to communicate a mystery and keep it mysterious?   I think the strongest moral to be mined from this story is that creating draft marketing approaches should be a part of every product proposal.

Difficulty is important – but the best progression is non-linear and rhythmic

I usually wonder about why designs have the difficulty progressions that they do.    Yu gives his thoughts on it here:

The more I play and create games, the less convinced I am that the difficulty of games should be thought of in terms of a linear or exponential ramp upwards where, as the player gets stronger, you need to make the opposition increase proportionally in strength….[there is] something futile and perhaps nihilistic about endlessly cranking a single knob that goes from easy to hard.  Rather I believe it makes more sense to think about difficulty in terms of the game’s overall pacing.   Difficulty should ebb and flow, and make room for other aspects of play. p. 178

If you draw a comparison to pacing and tension in narrative, it is easy to see that establishing a rhythm – and, at times,  violating it-  is potentially a very strong design trope that resonates with how we experience meaning.    I haven’t heard anyone express this as clearly as Yu has.

The deeper lesson

What is the deeper lesson revealed by these gems?  The stories Yu tells are interesting because of the the motivations he reveals for making the design choices he makes.    He has a great eye for analysing and articulating what makes a design tick, and you get the sense that his heart is in the right place, always.   To me,  what is striking about Yu’s approach is his attention, at all times,  to viewing the design of the game through the lens of the player’s possible experience of it.    That is the real eggplant that is hidden in plain sight.

 

 

 

 

 

Making, combining, experimenting and inspiring at #playful13

I lucked into this conference last year when someone I know on twitter couldn’t attend.    It’s now inked into my diary for the forseeable future.  Playful describes itself as

A sticker-book of brilliant thinking designed to make you want to Make Stuff.

And it’s not wrong.

Full marks for diversity of topics.  We had mummified deer covered with beeswax, which inspired chefs into new ways of dealing with rotten plums,  a billion or so different snakes and ladders boards generated through genetic algorithms, eye popping mind-bending graphics making complexity out of simplicity, and back again, a poignant and poetic talk about boxes,  and a story about dropping out of the rat race in a virtual world.   We had real adventure playgrounds,  virtual performance art,  bananaphones, folding wheels, not terribly practical jokes on the postal service, people at Facebook dancing a marked up two step on polished concrete that based on the rhythms of a loved up couple’s Facebook posts.  We had movies of smiling engineers drilling into hardened steel with home made EMF machines.   We took a deep dive into the physical pleasures of response curves in button pressing, and the pleasurable physics of custard-punching.

What do mummified deer, dustard punching, and bananaphones have in common?  The announced theme was “playing with form”.  What I really picked up from the talks as a unifying theme was a strong feeling of joyous experimentation, and a passionate belief in the importance of experimenting as a way of opening up creativity.

All this in peaceful, beautiful Conway Hall, a London landmark of the humanist movement.  I love Conway Hall.   And I loved the event.  All the talks were interesting.   And everyone in the audience I spoke with was too.

Conway Hall

Two different talks used the same quote,  from Mike Chrisp –

Build what it is you want to build and learn as you go.

What good advice.

Many thanks to Mudlark for curating, and to the speakers*, for blowing a big, teasing, leaf-strewn wind through my mind.

* Duncan Fitzsimmon, Ann Holiday, George Buckenham, John V Willshire, Fran Edgerly, Pippin Barr, Dani Luri,  Marie Foulston, Ben Reade, Rev Dan Catt, Stephanie Posavec,  and Rob Lowe (aka Supermundane).