Worlds beyond

Teachable moments in virtual reality

THE BLUE WHALE is only a metre or two away from me, and its huge right eyeball is level with mine. I have never seen the largest creature on Earth from this angle, at this depth, in these dimensions. Its body mass fills my vision, and I have to turn my head 180-degrees to take it all in. I’m standing on the bow of a sunken ship, and I have watched, enthralled, as this dweller of the deep sea approached from the dark-blue distance on my left. The surrounding schools of fish and graceful manta rays take little notice of the giant: for them, its presence deserves the equivalent of a submerged shrug; it’s something they see every day. But for myself, standing here on the ship, rooted in place in wonder, it is an extraordinary encounter.

As it hovers before me, the beast blinks and emits a few curious groans before tiring of this underwater interloper. Just another human being. Boring. When it swims to my right in search something more interesting to look at, its tail flukes almost lash me on the way past. In sum, I’ve spent only a minute in its company, but I feel as though quite a few things have changed. This is a fork in the road: my life can now be categorised as ‘Before Whale’ and ‘After Whale’. But perhaps the most astounding part of this experience is that it is running entirely on computer code. Even though every aspect of this scene feels real, it is not.

After I watch the whale disappearing into the dark-blue distance, I turn around to see white text projected on the back of the sunken ship. It is a list of credits for the team of people who worked on this convincing simulation. It’s named ‘Whale Encounter’, and it’s part of a virtual-reality game called theBlu. In reality, you see, I’m standing in a tiled living room in New Farm, in inner-city Brisbane, wearing a headset that is attached to a powerful computer by a thick, black cord. The view from the balcony outside is filled by the Story Bridge. Beneath the steel structure runs the Brisbane River, where there are no blue whales, as far as I’m aware.

Next, I use one of the wireless controllers in my hands to point and select ‘Turtle Encounter’. This is just as impressive as the previous immersion – and several minutes longer, too. In startlingly clear water, with the sun shining through the surface above me, I stand at the edge of a coral reef. A loggerhead turtle cuts a lazy circle above me, just out of reach. On my right, I’m approached by hundreds of football-sized bright orange jellyfish. Before long, these small creatures are accompanied by several enormous, man-sized giants whose tentacles trail behind them like windblown dreadlocks. Using the controllers, I can prod the jellyfish to affect their trajectory. It is simply gorgeous, and like the whale, it inspires a sense of awe unlike anything I’ve experienced while playing a traditional video game.

When it comes to eliciting emotions, it appears that virtual reality is streets ahead of everything that’s come before. Moreover, it strikes me that this sort of experience could foster new understanding for children who struggle to process information that’s delivered verbally, or presented on a printed page. Any child can strap on a headset, fire up theBlu and take something away from the experience of feeling as though they are immersed underwater, inside a potentially dangerous and hard-to-reach part of the planet, while still engaging their minds in a way that a textbook, or even the most fervent educator, might fail to achieve.

In the classroom, it’s one thing to hear your teacher tell you that a blue whale is bigger than two buses, end-to-end – but words such as these tend to ricochet away from young ears like a basketball pinging off the outer edge of a hoop. Witnessing a swimming whale with your own eyes, though? That’ll stick, because there’s an emotion attached to it: wonder.

This sort of psychological response to a stimulus is essential for learning at any age, but perhaps more so in children, whose schooling begins with learning a little about a lot of topics. This bombardment of facts can easily become overwhelming: a student who feels a strong connection with language may be turned off by the hard numbers of mathematics. That early withdrawal from one subject area while preferring another may unwittingly determine the course of their life and career, which they may regret in years to come. But what if there was a way to use technology to make learning maths more emotionally engaging? What if something like theBlu could be re-coded as an educational tool, where children are encouraged to chart and measure the rise and fall of a sea creature they can see with their own eyes, through a VR headset, rather than learning from static lines drawn in their workbooks?

In her book Emotions, Learning, and The Brain (WW Norton, 2015), Dr Mary Helen Immordino-Yang writes: ‘Even in academic subjects that are traditionally considered unemotional, such as physics, engineering or math, deep understanding depends on making emotional connections between concepts.’ The associate professor of education, psychology and neuroscience at the University of Southern California told The New York Times that it’s ‘literally neurobiologically impossible’ to think deeply about things that you don’t care about, which might explain why maths classes can be such a slog for unengaged children. Through the use of a functional MRI scanner, Dr Immordino-Yang has observed the brains of students while they’re mid-learning. ‘When students are emotionally engaged,’ she said, ‘we see activations all around the cortex, in regions involved in cognition, memory and meaning-making, and even all the way down into the brain stem.’

During my primary schooling in the 1990s, the era’s most impressive multimedia learning tool was the Microsoft Encarta series of CD-ROM encyclopaedias, which combined audio and visual elements with a traditional, textbook-like presentation of information. Though it was mind-blowing at the time, the concept now seems rather quaint in the age of Google and Wikipedia, where you can find the answer to just about any question by typing a few keywords into a search engine.

But Encarta remained wedded to the central necessity of communicating via words. I’m lucky enough to be the sort of person who revels in the mode of learning that is based on reading – but how many of my peers were left behind because they were unable to process sentences and paragraphs? How might the children of today and tomorrow be better served by incorporating VR experiences into schooling practice in order to deepen understanding and stoke curiosity?

Gaining an appreciation for the size and beauty of marine mammals is just one of many educational opportunities that spring to mind. What about visiting a full-scale replica of the Colosseum as part of an ancient history class? Or gazing up at the enormous sandstone rock in the centre of Australia while learning about Uluru and its significance to the nation’s First Peoples? Or coming to terms with the atmospheric composition of each planet in the solar system by examining scientifically accurate VR models of Saturn and Venus? Or what about visiting Mars in order to better understand how humans might start a colony there in the near future?

That last example is no simple hypothetical: in April 2016, a group of elementary and middle school students in Washington DC boarded what they thought was an everyday school bus for what they assumed would be a regular field trip to the USA Science and Engineering Festival. Instead, soon after the wheels of the bus began turning, the glass windows transformed into high-definition LCD displays and, outside, the students were shocked to find they were now driving on what appeared to be the surface of Mars.

For this first ever group VR vehicle experience, a team of artists and programmers mapped two-hundred square miles of the Red Planet using the same software that’s used to construct the architecture of advanced video games. Rather than having the students each don headsets and sit still, however, the innovation here was to turn the bus itself into the equivalent of a giant headset, while also relying on an array of hidden technology such as accelerometers, GPS, magnetometers and a laser-surface velocimeter. The result was that when the bus moved, the surface moved; and when the bus turned, so did the simulation of Mars.

Video footage posted online captured the students’ reactions to the moment when Washington suddenly looked a whole lot like one of our nearest neighbours in the solar system. As they screamed with wonder and delight, it’s clear that some unforgettable memories were being etched into their minds. Over the course of a few hours, while the driver guided the vehicle around some fairly mundane inner-city streets, the kids were wide-eyed and open-mouthed as they took in the planet’s geological features, the car-sized robotic rover aptly named Curiosity, and a realistic base camp. Because of the extensive mapping, they weren’t seeing the same thing over and over: at one point, the bus encountered a massive Martian dust storm, which relied on immersive audio engineering to ensure that the experience didn’t just look real, but sounded real too.

Six months in the making and funded by aerospace and defence company Lockheed Martin, the bus was designed by Oscar-winning creative studio Framestore VR. Handily, its team knew a bit about coding the Red Planet, having worked on more than three hundred special effects shots for the 2015 science fiction film The Martian. On location at the USA Science and Engineering Festival, over 2,500 attendees boarded the bus, which is part of Lockheed’s ‘Generation Beyond’, a multi-year national educational program designed to inspire children to pursue careers in STEM – science, technology, engineering and mathematics. At the core of the thinking behind the field trip to Mars was a simple maxim similar to theBlu’s underwater simulations: show, don’t just tell.

By Lockheed’s estimation, the first manned mission to Mars will occur within the next two decades, and the company is seeking to inspire the next generation’s sense of possibility in order to ignite their passion for developing the skills required to get humans to deep space. ‘We knew that just telling students they could be the first person on Mars wasn’t enough. We needed them to believe it,’ Framestore VR’s website states. ‘So we reached out to those future scientists, explorers, and pioneers – elementary and middle school students – and showed them, their parents and teachers, and the world just how close our future on Mars really is.’ In due course, that one small VR trip for children might end up being part of one giant leap for mankind.


BOTH THE UNDERWATER and Martian experiences were designed by educated adults working in a nascent industry built on incremental advances in processing power, graphic rendering and interface design. What happens when control of these novel environments is given over to children, though? How might young eyes manipulate the existing technology and, in time, see ways to improve and innovate beyond current limitations?

Through a simple flick of the wrist, it’s possible to shift the location of the sun in the sky. In one moment, the brightness of high-noon daylight dominates the scene; then, in a flash, the deep shades of twilight snap into view. The godlike controller of this impossible task is a young girl who sits before a computer screen in the air-conditioned cool of the Redcliffe Library on a Wednesday in January. Outside is the incessant glare of the Queensland summer, yet in here, she is using a software application named Unity to build an unreal world. Soon, she will be able to step inside it and see things she has never seen before.

With each click of the mouse, this girl authors a story unlike any other, for Unity allows its users to sculpt terrain, plant trees and summon bodies of water – among many other tasks – in a totally unique and individualised manner. Setting the height and direction of a light source – an in-game sun – is a crucial step towards making this scene feel realistic. Wearing denim shorts, joggers, a black shirt and her blonde locks tied loosely behind her head, this girl in her early teens has opted to spend two hours of her summer holidays being instructed by two young men whose interest in computer-generated worlds stretches far beyond the superficial. Instead, their gaze is fixated on the long-term implications of virtual reality technology, which has long hovered at the edges of popular culture and mainstream consciousness – yet that may soon change. This young girl and five other school-aged children may be getting a glimpse into the future, as well as a potential headstart on her peers.

The leader of this workshop is a severe-looking young man with a shaven head and a spotless white T-shirt named Alexander Van Cooten – ‘Lex’ for short. His co-tutor for this midday session is David Chaseling, a strongly built dude in a black T-shirt who comes across as less intense than his colleague simply because he smiles more often. The two enthusiasts have brought four powerful PCs to the library for these six children to use, as well as an impressive device at the front of the room which some of the attendees have seen on YouTube, but never in the flesh: it’s called a Vive, a product developed by Taiwanese smartphone manufacturer HTC in conjunction with American video-game developer Valve. It’s a VR headset that retails for about $1,400, and so remains out of the reach of all but the most dedicated or wealthy early adopters.

Other than the tutors’ step-by-step guide to using Unity for the first time, the classroom soundtrack is one of mouse-clicking and occasional whispered comments. Blown up on the projector screen is a bigger version of the view on Chaseling’s monitor – a square of terrain hovers in mid-air, free of gravity, as he slowly adds features, definition and landmarks. The kids quickly get the hang of painting hills onto the terrain with a brush tool: at the maximum setting, a click of the mouse will see 600-metre-tall towers jutting out of the landscape like exclamation points.

The tutor reminds the children that at the end of the class, they’ll put on a headset and see this world with their own eyes. ‘Some students like to build a big mountain to get a fright when they first jump in,’ he suggests. One boy in a tie-dyed T-shirt asks whether he could jump off a cliff in-game and fall to his virtual death. ‘You could,’ replies Chaseling thoughtfully. ‘But it probably wouldn’t do anything unless you coded it to.’ Circling the room to check their monitors, he notes that sure enough, everyone has opted for the ‘massive tower look – which is cool’. With a shake of the head, he says that someone in the last class opted to build an imposing wall, and named it after a certain American president.

There is a simple joy in using this application to do things that are otherwise difficult, if not impossible, for the average Redcliffe adolescent to achieve in their ordinary life. While they happily toy with Unity’s intuitive user interface, indulging their creativity in whichever way makes the most sense to their young minds, there is a bigger story at play here. For decades, children have gathered around screens and monitors to control avatars that are faced with decisions and choices that exceed their wildest dreams. Whether that’s embodying James Bond in GoldenEye 007 (released on the Nintendo 64 console in 1997); becoming a blocky world-builder in Minecraft (released on PC in 2011), or attempting to shoot down incoming alien ships in the classic arcade game Space Invaders way back in 1978, this sort of entertainment has long rewarded its players with rare agency.

Up until now, though, the divide between player and on-screen action has been stark. Gamers were constantly aware that they were holding a hard plastic controller, or using a keyboard and mouse, or standing in front of an arcade machine while employing their hand-eye co-ordination to manoeuvre avatars displayed on a flat surface before their eyes. Depth of field and three-dimensionality were innovative leaps that drew players further into these worlds – Super Mario 64, released in 1996, was a generational landmark which offered a fully realised 3D game world unlike any before it, which is part of the reason why it sold more than 11 million copies worldwide. But true immersion remained out of reach, because there was simply no way to enter the screen, no matter how close to the TV you sat. By bringing players inside these worlds for the first time, however, VR games offer the tantalising prospect of a deeper, richer experience.

Today in Redcliffe, Van Cooten has booked three sessions back-to-back, and the small classes are all full. Advertised to Moreton Bay Region Library members as part of its Teen Holiday Workshops, the idea here is to provide a central location for early access to cutting edge technology at zero cost. As co-founder of a monthly VR Club meet-up, business owner of a software development company and operator of a Queensland-first VR arcade in Brisbane’s city centre, Van Cooten is a 24-year-old betting big on virtual reality as something that’s here to stay. These beginners’ workshops are child’s play to him, yet he sees them as an important awareness-raising initiative for the next generation of gamers, software developers and entrepreneurs. Part of the excitement of getting in on the ground floor is to see what younger operators come up with, as they use the industry’s existing tools in original and unexpected ways.

As the two-hour limit approaches, the tech-savvy children each use their mouse to choose the location of a ‘camera rig’ where their in-game vision will begin. The instructors use a portable hard drive to download copies of the Unity files from each PC. And then, one by one, the six students take turns putting on the HTC Vive headset and entering their virtual worlds for the first time. On a flat screen at the front, their classmates can see what they are seeing, but it appears to be a poor facsimile of the actual experience. Once the sleek goggles are fixed to their vision, a couple of the kids are struck dumb and can say little more than ‘Wow!’ while looking around the room. Or at least that’s what it looks like to the rest of us; their own eyes are processing the textured towers, trees, wooden docks and treasure chests that are all exactly where they chose to place them.

The only girl in the class is last: hands by her side, she silently savours the experience, and takes some tentative steps toward the outer limits of the small space at the front of the room. She and her peers are restricted by the light-sensing technology that accurately tracks their location in real-time, as well as by the thick, black cord attached to the PC that runs the Vive hardware. ‘In a couple of years, you’ll probably laugh that there was ever a cable for this,’ says Chaseling, hinting at the likely innovation of wireless headsets to allow greater movement, and thus greater immersion.

Because of the time limitation, each child can only spend a couple of minutes looking around and remarking on what they’re seeing, but it is still a potent glimpse of what lies ahead in their young lives. A gangly teenager in thongs and a white singlet considers the world he and his younger brother created and, after removing the headset, looks dismayed to be back inside the library. With a trace of disappointment in his voice, he says, ‘Oh, I like that reality way better.’

He’s making a joke, and he draws the appropriate response from his classmates. But part of the reason that thought would even occur to him is because of the sheer novelty of the experience. Before this class, none of them had ever seen anything like it before – but now they have. It’s only when they get past those superficial responses to the new stimulus that the real emotional engagement and teachable moments will begin. Perhaps this technology will soon find its way into their classrooms as a teaching tool to improve their understanding of the world around them, as well as creatures beneath the waves and planets beyond Earth.

Get the latest essay, memoir, reportage, fiction, poetry and more.

Subscribe to Griffith Review or purchase single editions here.

Griffith Review