The “Holodeck” is here. How will it influence storytelling?

It’s not every day that you stumble across a fundamentally new form of storytelling, but that’s exactly what’s happened in the last few months thanks to a confluence of exciting new technologies and some smart, motivated people working in the so-called field of “virtual reality.”

The arch from the Holodeck of the USS Enterprise (NCC-1701-D). Source: Wikipedia. http://en.wikipedia.org/wiki/Holodeck

But before I continue, I just want to warn you that everything you’re about to hear and see will sound absolutely ridiculous, almost as if it was dreamed up for a science fiction story or lifted from Star Trek. And in a sense, as with most new technologies, it was  — but science fiction has now become science fact.

What I’m about to show you is how the “Holodeck” experience first popularized by Star Trek the Next Generation, and also explored by novelists William Gibson and Neil Stephenson, is starting to become real.

It’s set up in my office right now, and also about 10 miles west of me in the empty barn, affectionately referred to as the “Holobarn,” of my creative hacker friend Lorne Covington who operates under the banner of NoirFlux.

This video from my class at the S.I. Newhouse School, New Tech for New Media, shows a little of what’s possible. YouTube Preview Image

What you will see in the video is three distinct technologies: 1) An Oculus Rift VR headset that generates a 3D image, 2) A Kinect depth camera that monitors your movement in 3-dimensional space, and 3) a graphics PC that synthesizes all of these inputs and outputs the experience wirelessly to the Rift. There’s also a second depth camera mounted on top of the Rift that allows the tester to see her hands as she moves them around.

In this case, the VVVV environment that Covington has programmed is creating the illusion of a screen floating in the middle of the air inside a house based on an Escher painting. The screen has Google Maps projected onto it, and the student is navigating the map by using iPad-like swipe gestures. (I warned you that this would start to sound ridiculous!)

Turning the classroom into a holodeck

All 16 students in the NTNM class went through this experience, and a few conducted extended labs. Together we have have been thinking about what it means for storytelling in the future — if you can even call it that. It’s more like story-experiencing, or what VR pioneer Nonny de la Peña refers to as Immersive Journalism. When you work with VR you quickly start to realize that there’s no vocabulary to describe some of these concepts.

Using the Unity3D gaming engine, we’re now experimenting with creating virtual worlds that people can walk through — like this city in a canyon that I arranged myself in a couple hours.

YouTube Preview Image

We have found that once you get over a small learning curve, it’s surprisingly easy to create rich environments using 3D models in the Unity Asset store, then let people explore them. Some of my students are taking the extra steps to do simple usability testing to get a sense of how people respond to these types of immersive experiences.

One student recently sent people through a virtual world and asked for feedback on what it was like, which you can read about in the lab report. What I find most interesting is how universally positive and excited people are after entering these environments — not the usual response to new technologies. Some remark that they could stay in it forever, and they only come out after they feel nauseus — an effect that is more likely when using the Rift from a chair. When you walk around in the “Holodeck” configuration which uses your physical location in the room for navigation, motion sickness is less common. Your mind receives the signals of movement that it expects based on the feedback from your eyes.

What will you put in your Holodeck?

I predict that we will soon start seeing similar Holodecks all over the place. That’s because it’s not a single product you buy off the shelf. It’s made up of a bunch of different technologies, most under $400 and many open-source, that are combined to create immersive experiences. With enough tinkering and patience anyone can do this for $2,000 or less depending on what other computing equipment they have lying around.

While there are many implications for this technology, as the Newhouse School’s Horvitz Chair of Journalism Innovation my focus is on journalism and storytelling. What kinds of stories and experiences can we “tell” — or “transfer” — to help people understand complex ideas by letting them walkin through and manipulate them? The possibilities are endless. Here are just a few ideas I’ve been contemplating over the last few months:

  • Experience the effects of climate change: It’s one thing to read about what rising ocean levels mean for coastal areas, and quite another to be standing in New York City, London or New Orleans, dial in a date and climate study, and see if you’re underwater.
  • Science and medicine: When the next big cancer drug comes out, you could walk inside a cell and look around as the drug does its work on cellular DNA. Heard about that new planet NASA found? Plug in your goggles and go there.
  • Visualizing data: Data visualization, much driven by the data-driven documents codebase, has been a big trend in journalism lately. A next logical step is to create 3-dimensional, immersive visualizations that you can fly through and manipulate using your hands and entire body.

Over the next year I plan to spend time with media companies exploring some of these ideas through my new FutureForecast Consulting business. I also hope to host a panel and workshop about it at the Online News Association conference in September along with Noirflux, Nonny de la Peña and some others.

And speaking of Nonny, she is already working on her third immersive documentary using one of the prototypes used by Palmer Luckey before launching the Oculus Rift. Here’s a video of her second piece, Project Syria.

YouTube Preview Image

Going beyond virtual reality

Perhaps you’re thinking, OK Pacheco, I get it. Sounds like a gaming platform. How does that relate to journalism? And this is where things start to get really trippy. Are you ready?

In the not too distant future, I’m convinced that it will be possible — maybe even commonplace — to inject 3D information that is either scanned or inferred from the physical world into a Holodeck environment. At this point, the term “virtual” doesn’t really do it justice. It may feel more like transferred reality, or what Fast Company writer Rebecca Greenfield calls 3D Transportation.

The pieces to do this are all available now, such as:

  • 3D scanning drones like SenseFly are scanning mountains and mining sites as we speak. What happens when this technology costs not $40,000, but $400? It will happen.
  • A company called Replay Technologies can create Matrix-like instant replays using data captured from 4K cameras all around the stadium. Here’s a video of the replay, and here’s more about how it works.  What happens when such systems can be placed anywhere, perhaps even carried by drones? Then the 4K arena can be taken anywhere — such as the site of a recent earthquake or a large outdoor event — and people can virtually teleport in from home.
  • Small handheld scanners like Google’s Project Tango or Structure.io will make it possible to scan places as you visit them, then export what you scanned as 3D object files.
  • And finally, craziest of all, data from 2-dimensional photos can now be combined to extrapolate realistic 3D objects. Cornell Professor Noah Snavely recently used randomly posted photos from Flickr to create a 3D point-cloud reconstruction of Dubrovnik. And another creative hacker has created a way to see 3D point clouds using data from Google’s Street View.

Based on this, what’s to stop someone (maybe me!) from going to a place like Machu Picchu, scanning the inside and outside of every building, scanning it from a drone above, then sending you a link so you can virtually go to Machu Picchu from your Oculus Rift?

This is the point at which you Trekkies out there can confidently say, “Computer, Arch. Take me to Machu Picchu circa early 21st century earth.” It sounds like science fiction or magic, but the technology is available to do all of this now. As Arthur C. Clarke so eloquently put it, “Any sufficiently advanced technology is indistinguishable from magic.”

Collective community visioning

On a final note, I have been thinking about how advances in 3D scanning, visualization and printing can be combined to better engage and inform communities around building projects. In the spirit of throwing spaghetti on the wall to see if it sticks, I’ve thrown this idea out there under the banner of Visualize Cities. I’m already talking logistics with stakeholders across Syracuse University, including at the School of Architecture and College of Visual and Performing Arts. The next step is to find some community partners. If you’re also interested, fill out the form on the page and I’ll get in touch.

– Prof. Dan Pacheco

 

 

Professor Pacheco in the Press

2016 Mirror Awards Ceremony - YouTube

Comments:

  • Nonny de la Peña had this to say about Professor Pacheco after her acceptance of an I-3 Mirror Award.

    “Once the [Oculus] Facebook sale happened, $2 billion dollars, you’re not so nuts. But before that happened, a lot of people thought trying to do journalism in virtual reality was crazy...

Students, Faculty to Create Content for HoloLens Augmented Reality Headset

Comments:

  • Prof. Dan Pacheco has been accepted into the HoloLens developer program and will work with Newhouse students and faculty to create content for the device. - Dan Pacheco

Tags:

Immersive journalism: What virtual reality means for the future of storytelling and empathy-casting - TechRepublic

Comments:

  • Prof. Pacheco was interviewed by CBS Interactive's TechRepublic about using virtuality for journalism in the story: "Immersive journalism: What virtual reality means for the future of storytelling and empathy-casting." The piece also referenced Pacheco's Virtual Reality Storytelling course and included a link to a 360 video of the SU football team by class member and football player Eric Jackson...

http://storynext.gannett.com/state-of-vr.pdf

Comments:

  • Prof. Pacheco co-authored the primer on virtual reality in this report on The State of Virtual Reality in Journalism for Gannett and the Knight Foundation. - Dan Pacheco

Tags:

Daily Orange: Universities should embrace experiential media courses

Comments:

  • "There’s a virtual reality course being taught this semester in the S.I. Newhouse School of Public Communications called Virtual Reality Storytelling. (Full disclosure: I am taking the course right now. It’s awesome.) The course’s professor, Dan Pacheco, says demand for the course is high, and for good reason in that it may set you apart in a job interview...

StoryNEXT conference features virtual reality – Hypergrid Business

Comments:

  • Professor Pacheco has been added to the advisory board for StoryNEXT, a new annual conference focused on innovators in virtual reality storytelling. - Dan Pacheco

Tags:

Recent Breakthroughs In Virtual Reality Go Beyond Simply Playing Video Games | ThinkProgress

Comments:

  • "“The more difficult and complicated the subject, the more we seem to want to close the news app and open up Candy Crush,” said Daniel Pacheco, a journalism professor at Syracuse University who specializes in VR storytelling. “VR can counteract this tendency because of its ability make you feel physically present somewhere and to comprehend information by exploring it in an almost physical way...

The Des Moines Register wins Murrow award for Harvest of Change VR Project

Comments:

  • The Des Moines Register's Harvest of Change, which was the first project to incorporate virtual reality in a commercial news setting, has won a prestigious Edward R. Murrow award. This emerged from Professor Dan Pacheco working as an innovator in residence at Gannett Digital, where he brought an Oculus Rift and knowledge of how to create VR experiences in Unity...

HoloLens Brings 3D Media Into the Physical World

Comments:

  • Professor Pacheco reports on his experience with an extended HoloLens demo at the E3 conference and provides ideas on how it could be used in journalism and media. - Dan Pacheco

Tags:

"All the President's Avatars"

Comments:

  • Professor Pacheco is interviewed by a Swedish tech magazine about virtual reality journalism. - Dan Pacheco

Tags:

Adobe Adds Mixamo 3D Animation Service To Creative Cloud Mix | TechCrunch

Comments:

  • "Students in Pacheco’s VR Storytelling course (http://vrstorytelling.org), most of whom have no 3D modeling experience, use Mixamo to easily and inexpensively add interactive characters to scenes that are viewed in Oculus Rift virtual reality headsets" -

    Welcome to the Washing Machine

    Comments:

    • Prof. Pacheco predicts the future media-disrupting technologies for CommPRObiz: "Part of my job as the Peter Horvitz Chair in Journalism Innovation is to look on the horizon and identify trends that might lead the next waves of disruption. I also challenge my students to do field tests of disruptive technologies, then use what they learn to predict where things will be in another 5 years...

Our list of the best 11 journalism experiments

Comments:

  • Columbia Journalism Review just listed the "Harvest of Change” virtual reality journalism project that Professor Pacheco co-produced at Gannett as one of 11 noteworthy journalism experiments today. BDJ senior, now graduated, also contributed as a developer. -

    Virtual Reality Storytelling - From Screen to Beam : Documentary Campus e.V.

    Comments:

    • Creating Your First VR Experience
      Practical tips for getting started in VR using freely available software and 3D assets. Prof. Dan Pacheco will draw upon material from his Virtual Reality Storytelling class at the S.I. Newhouse School to provide practical steps for creating interactive scenes in Unity3D that can be exported for web and the Oculus Rift...

One thought on “The “Holodeck” is here. How will it influence storytelling?

  1. Pingback: The ‘Holodeck’ Arrives in Newsrooms. How Will VR Influence Storytelling? | Idea Lab | PBS

Leave a Reply

Your email address will not be published. Required fields are marked *