It’s not every day that you stumble across a fundamentally new form of storytelling, but that’s exactly what’s happened in the last few months thanks to a confluence of exciting new technologies and some smart, motivated people working in the so-called field of “virtual reality.”

The arch from the Holodeck of the USS Enterprise (NCC-1701-D). Source: Wikipedia. http://en.wikipedia.org/wiki/Holodeck

But before I continue, I just want to warn you that everything you’re about to hear and see will sound absolutely ridiculous, almost as if it was dreamed up for a science fiction story or lifted from Star Trek. And in a sense, as with most new technologies, it was  — but science fiction has now become science fact.

What I’m about to show you is how the “Holodeck” experience first popularized by Star Trek the Next Generation, and also explored by novelists William Gibson and Neil Stephenson, is starting to become real.

[vimeo]http://vimeo.com/95873596[/vimeo]

It’s set up in my office right now, and also about 10 miles west of me in the empty barn, affectionately referred to as the “Holobarn,” of my creative hacker friend Lorne Covington who operates under the banner of NoirFlux.

This video from my class at the S.I. Newhouse School, New Tech for New Media, shows a little of what’s possible. [youtube]http://www.youtube.com/watch?v=vOTM7AWNmf0[/youtube]

What you will see in the video is three distinct technologies: 1) An Oculus Rift VR headset that generates a 3D image, 2) A Kinect depth camera that monitors your movement in 3-dimensional space, and 3) a graphics PC that synthesizes all of these inputs and outputs the experience wirelessly to the Rift. There’s also a second depth camera mounted on top of the Rift that allows the tester to see her hands as she moves them around.

In this case, the VVVV environment that Covington has programmed is creating the illusion of a screen floating in the middle of the air inside a house based on an Escher painting. The screen has Google Maps projected onto it, and the student is navigating the map by using iPad-like swipe gestures. (I warned you that this would start to sound ridiculous!)

Turning the classroom into a holodeck

All 16 students in the NTNM class went through this experience, and a few conducted extended labs. Together we have have been thinking about what it means for storytelling in the future — if you can even call it that. It’s more like story-experiencing, or what VR pioneer Nonny de la Peña refers to as Immersive Journalism. When you work with VR you quickly start to realize that there’s no vocabulary to describe some of these concepts.

Using the Unity3D gaming engine, we’re now experimenting with creating virtual worlds that people can walk through — like this city in a canyon that I arranged myself in a couple hours.

[youtube]https://www.youtube.com/watch?v=lZcfqfNBut4[/youtube]

We have found that once you get over a small learning curve, it’s surprisingly easy to create rich environments using 3D models in the Unity Asset store, then let people explore them. Some of my students are taking the extra steps to do simple usability testing to get a sense of how people respond to these types of immersive experiences.

One student recently sent people through a virtual world and asked for feedback on what it was like, which you can read about in the lab report. What I find most interesting is how universally positive and excited people are after entering these environments — not the usual response to new technologies. Some remark that they could stay in it forever, and they only come out after they feel nauseus — an effect that is more likely when using the Rift from a chair. When you walk around in the “Holodeck” configuration which uses your physical location in the room for navigation, motion sickness is less common. Your mind receives the signals of movement that it expects based on the feedback from your eyes.

What will you put in your Holodeck?

I predict that we will soon start seeing similar Holodecks all over the place. That’s because it’s not a single product you buy off the shelf. It’s made up of a bunch of different technologies, most under $400 and many open-source, that are combined to create immersive experiences. With enough tinkering and patience anyone can do this for $2,000 or less depending on what other computing equipment they have lying around.

While there are many implications for this technology, as the Newhouse School’s Horvitz Chair of Journalism Innovation my focus is on journalism and storytelling. What kinds of stories and experiences can we “tell” — or “transfer” — to help people understand complex ideas by letting them walkin through and manipulate them? The possibilities are endless. Here are just a few ideas I’ve been contemplating over the last few months:

• Experience the effects of climate change: It’s one thing to read about what rising ocean levels mean for coastal areas, and quite another to be standing in New York City, London or New Orleans, dial in a date and climate study, and see if you’re underwater.
• Science and medicine: When the next big cancer drug comes out, you could walk inside a cell and look around as the drug does its work on cellular DNA. Heard about that new planet NASA found? Plug in your goggles and go there.
• Visualizing data: Data visualization, much driven by the data-driven documents codebase, has been a big trend in journalism lately. A next logical step is to create 3-dimensional, immersive visualizations that you can fly through and manipulate using your hands and entire body.

Over the next year I plan to spend time with media companies exploring some of these ideas through my new FutureForecast Consulting business. I also hope to host a panel and workshop about it at the Online News Association conference in September along with Noirflux, Nonny de la Peña and some others.

And speaking of Nonny, she is already working on her third immersive documentary using one of the prototypes used by Palmer Luckey before launching the Oculus Rift. Here’s a video of her second piece, Project Syria.

[youtube]https://www.youtube.com/watch?v=jN_nbHnHDi4[/youtube]

Going beyond virtual reality

Perhaps you’re thinking, OK Pacheco, I get it. Sounds like a gaming platform. How does that relate to journalism? And this is where things start to get really trippy. Are you ready?

In the not too distant future, I’m convinced that it will be possible — maybe even commonplace — to inject 3D information that is either scanned or inferred from the physical world into a Holodeck environment. At this point, the term “virtual” doesn’t really do it justice. It may feel more like transferred reality, or what Fast Company writer Rebecca Greenfield calls 3D Transportation.

The pieces to do this are all available now, such as:

• 3D scanning drones like SenseFly are scanning mountains and mining sites as we speak. What happens when this technology costs not $40,000, but $400? It will happen.
• A company called Replay Technologies can create Matrix-like instant replays using data captured from 4K cameras all around the stadium. Here’s a video of the replay, and here’s more about how it works.  What happens when such systems can be placed anywhere, perhaps even carried by drones? Then the 4K arena can be taken anywhere — such as the site of a recent earthquake or a large outdoor event — and people can virtually teleport in from home.
• Small handheld scanners like Google’s Project Tango or Structure.io will make it possible to scan places as you visit them, then export what you scanned as 3D object files.
• And finally, craziest of all, data from 2-dimensional photos can now be combined to extrapolate realistic 3D objects. Cornell Professor Noah Snavely recently used randomly posted photos from Flickr to create a 3D point-cloud reconstruction of Dubrovnik. And another creative hacker has created a way to see 3D point clouds using data from Google’s Street View.

Based on this, what’s to stop someone (maybe me!) from going to a place like Machu Picchu, scanning the inside and outside of every building, scanning it from a drone above, then sending you a link so you can virtually go to Machu Picchu from your Oculus Rift?

This is the point at which you Trekkies out there can confidently say, “Computer, Arch. Take me to Machu Picchu circa early 21st century earth.” It sounds like science fiction or magic, but the technology is available to do all of this now. As Arthur C. Clarke so eloquently put it, “Any sufficiently advanced technology is indistinguishable from magic.”

Collective community visioning

On a final note, I have been thinking about how advances in 3D scanning, visualization and printing can be combined to better engage and inform communities around building projects. In the spirit of throwing spaghetti on the wall to see if it sticks, I’ve thrown this idea out there under the banner of Visualize Cities. I’m already talking logistics with stakeholders across Syracuse University, including at the School of Architecture and College of Visual and Performing Arts. The next step is to find some community partners. If you’re also interested, fill out the form on the page and I’ll get in touch.

– Prof. Dan Pacheco

Professor Pacheco in the Press