The “Holodeck” is here. How will it influence storytelling?
April 30, 2014
It’s not every day that you stumble across a fundamentally new form of storytelling, but that’s exactly what’s happened in the last few months thanks to a confluence of exciting new technologies and some smart, motivated people working in the so-called field of “virtual reality.”
But before I continue, I just want to warn you that everything you’re about to hear and see will sound absolutely ridiculous, almost as if it was dreamed up for a science fiction story or lifted from Star Trek. And in a sense, as with most new technologies, it was — but science fiction has now become science fact.
What I’m about to show you is how the “Holodeck” experience first popularized by Star Trek the Next Generation, and also explored by novelists William Gibson and Neil Stephenson, is starting to become real.
It’s set up in my office right now, and also about 10 miles west of me in the empty barn, affectionately referred to as the “Holobarn,” of my creative hacker friend Lorne Covington who operates under the banner of NoirFlux.
This video from my class at the S.I. Newhouse School, New Tech for New Media, shows a little of what’s possible.
What you will see in the video is three distinct technologies: 1) An Oculus Rift VR headset that generates a 3D image, 2) A Kinect depth camera that monitors your movement in 3-dimensional space, and 3) a graphics PC that synthesizes all of these inputs and outputs the experience wirelessly to the Rift. There’s also a second depth camera mounted on top of the Rift that allows the tester to see her hands as she moves them around.
In this case, the VVVV environment that Covington has programmed is creating the illusion of a screen floating in the middle of the air inside a house based on an Escher painting. The screen has Google Maps projected onto it, and the student is navigating the map by using iPad-like swipe gestures. (I warned you that this would start to sound ridiculous!)
Turning the classroom into a holodeck
All 16 students in the NTNM class went through this experience, and a few conducted extended labs. Together we have have been thinking about what it means for storytelling in the future — if you can even call it that. It’s more like story-experiencing, or what VR pioneer Nonny de la Peña refers to as Immersive Journalism. When you work with VR you quickly start to realize that there’s no vocabulary to describe some of these concepts.
Using the Unity3D gaming engine, we’re now experimenting with creating virtual worlds that people can walk through — like this city in a canyon that I arranged myself in a couple hours.
We have found that once you get over a small learning curve, it’s surprisingly easy to create rich environments using 3D models in the Unity Asset store, then let people explore them. Some of my students are taking the extra steps to do simple usability testing to get a sense of how people respond to these types of immersive experiences.
One student recently sent people through a virtual world and asked for feedback on what it was like, which you can read about in the lab report. What I find most interesting is how universally positive and excited people are after entering these environments — not the usual response to new technologies. Some remark that they could stay in it forever, and they only come out after they feel nauseus — an effect that is more likely when using the Rift from a chair. When you walk around in the “Holodeck” configuration which uses your physical location in the room for navigation, motion sickness is less common. Your mind receives the signals of movement that it expects based on the feedback from your eyes.
What will you put in your Holodeck?
I predict that we will soon start seeing similar Holodecks all over the place. That’s because it’s not a single product you buy off the shelf. It’s made up of a bunch of different technologies, most under $400 and many open-source, that are combined to create immersive experiences. With enough tinkering and patience anyone can do this for $2,000 or less depending on what other computing equipment they have lying around.
While there are many implications for this technology, as the Newhouse School’s Horvitz Chair of Journalism Innovation my focus is on journalism and storytelling. What kinds of stories and experiences can we “tell” — or “transfer” — to help people understand complex ideas by letting them walkin through and manipulate them? The possibilities are endless. Here are just a few ideas I’ve been contemplating over the last few months:
- Experience the effects of climate change: It’s one thing to read about what rising ocean levels mean for coastal areas, and quite another to be standing in New York City, London or New Orleans, dial in a date and climate study, and see if you’re underwater.
- Science and medicine: When the next big cancer drug comes out, you could walk inside a cell and look around as the drug does its work on cellular DNA. Heard about that new planet NASA found? Plug in your goggles and go there.
- Visualizing data: Data visualization, much driven by the data-driven documents codebase, has been a big trend in journalism lately. A next logical step is to create 3-dimensional, immersive visualizations that you can fly through and manipulate using your hands and entire body.
Over the next year I plan to spend time with media companies exploring some of these ideas through my new FutureForecast Consulting business. I also hope to host a panel and workshop about it at the Online News Association conference in September along with Noirflux, Nonny de la Peña and some others.
And speaking of Nonny, she is already working on her third immersive documentary using one of the prototypes used by Palmer Luckey before launching the Oculus Rift. Here’s a video of her second piece, Project Syria.
Going beyond virtual reality
Perhaps you’re thinking, OK Pacheco, I get it. Sounds like a gaming platform. How does that relate to journalism? And this is where things start to get really trippy. Are you ready?
In the not too distant future, I’m convinced that it will be possible — maybe even commonplace — to inject 3D information that is either scanned or inferred from the physical world into a Holodeck environment. At this point, the term “virtual” doesn’t really do it justice. It may feel more like transferred reality, or what Fast Company writer Rebecca Greenfield calls 3D Transportation.
The pieces to do this are all available now, such as:
- 3D scanning drones like SenseFly are scanning mountains and mining sites as we speak. What happens when this technology costs not $40,000, but $400? It will happen.
- A company called Replay Technologies can create Matrix-like instant replays using data captured from 4K cameras all around the stadium. Here’s a video of the replay, and here’s more about how it works. What happens when such systems can be placed anywhere, perhaps even carried by drones? Then the 4K arena can be taken anywhere — such as the site of a recent earthquake or a large outdoor event — and people can virtually teleport in from home.
- Small handheld scanners like Google’s Project Tango or Structure.io will make it possible to scan places as you visit them, then export what you scanned as 3D object files.
- And finally, craziest of all, data from 2-dimensional photos can now be combined to extrapolate realistic 3D objects. Cornell Professor Noah Snavely recently used randomly posted photos from Flickr to create a 3D point-cloud reconstruction of Dubrovnik. And another creative hacker has created a way to see 3D point clouds using data from Google’s Street View.
Based on this, what’s to stop someone (maybe me!) from going to a place like Machu Picchu, scanning the inside and outside of every building, scanning it from a drone above, then sending you a link so you can virtually go to Machu Picchu from your Oculus Rift?
This is the point at which you Trekkies out there can confidently say, “Computer, Arch. Take me to Machu Picchu circa early 21st century earth.” It sounds like science fiction or magic, but the technology is available to do all of this now. As Arthur C. Clarke so eloquently put it, “Any sufficiently advanced technology is indistinguishable from magic.”
Collective community visioning
On a final note, I have been thinking about how advances in 3D scanning, visualization and printing can be combined to better engage and inform communities around building projects. In the spirit of throwing spaghetti on the wall to see if it sticks, I’ve thrown this idea out there under the banner of Visualize Cities. I’m already talking logistics with stakeholders across Syracuse University, including at the School of Architecture and College of Visual and Performing Arts. The next step is to find some community partners. If you’re also interested, fill out the form on the page and I’ll get in touch.
– Prof. Dan Pacheco
Notes from Professor Dan Pacheco’s Participation in Chancellor Syverud’s Inauguration Panel
April 11, 2014
Professor Dan Pacheco recently had the honor of speaking on a panel at Chancellor Kent Syverud’s inauguration on “Great Universities in the Next 25 Years.” This was a distinguished and diverse panel of 8 SU experts on a range of topics of interest to universities now and in the future. You view a video of highlights here:
Pacheco addressed this question: “What can we abstract from successful innovators in business and innovative organizations to change productively, and preserve what is most important?”
What Makes Universities Great, panel at @SyracuseU. (w/ George Saunders, Jim Boeheim, @pachecod & more) pic.twitter.com/R4Aya9Kofu
— Tracy Tillapaugh (@tracytilly) April 11, 2014
Each panelist had only 7 minutes to respond, so . Following are the notes Professor Pacheco prepared before the panel.
1. What makes a university great? The people and talent.
I was recruited out of a startup. Why did I come here? Some people thought that was weird. The answer: because a university is an amazing collection of talent that is truly unmatched anywhere else.
My goal was to help students become more innovative and entrepreneurial in journalism and media by connecting them with each other.
— Students in design, engineering, business
— Together they are the ones who will work together to create the future, and we can start that here.
What makes a university amazing is that there are smart, young, energetic individuals. Need to help make sure they don’t just burrow into rabbit holes for their individual talents. Need to mix them up, because it’s in the mixing that they find new innovations that change the world.
2. Interdisciplinary collaboration isn’t just a “nice to have” in the business world. It’s required in order to make new things happen.
We need to provide those opportunities here on this campus
— Between departments and schools
— Between complementary classes
— Software engineers collaborating with designers
— Designers with content producers
— Content producers with marketing and business
Many different skills are needed to create an amazing new product like Facebook or Twitter, or Google Glass, or the Oculus Rift. The idea that one person or one group or one department can do everything necessary to make things like this happen is false.
The idea that Steve Jobs invented the iPhone — if he were alive today, he would say that’s false. The iPhone was created by a team of many people with different types of skills, and the most important members of those teams are always the customers.
3. Embrace new technologies, know the basics of how they work.
— Understand new technologies (the puropose of the Digital Petting Zoo.)
— Learn just enough to be dangerous.
— If you can get past that danger zone, learn more so that you become an expert. But if like most you can’t do that, at least understand how it works at a fundamental level so that you can have an effective conversation with an expert, and together you can build the future.
4. Be comfortable with disrupting yourself so you have a future. If you don’t do it, there are many others who will.
— There are plenty of companies that are circling the wagons of academia who are happy to disrupt what we have. We cannot prevent that, but we can get ahead of the disruption so that we have a role in it.
— Companies like Apple, Google, Facebook and startups understand this all too well.
— The only ones that survive a paradigm shift and replace what is the next big thing are the ones who disrupt themselves.
— Consider a company like Facebook that has a billion registered users. A typical company would be comfortable with sitting on top of that pile of money and be content not to change anything. But in fact, that’s not what Facebook does. They just bought Oculus Rift for $2 Billion, WhatsApp for much more and Instagram before that. These companies could be seen as potential future competitors, but now they’re part of Facebook’s future.
— Nobody wants to be on the receiving end of change. It’s a lot easier, and more fun, to be the one who is making change. The only way to do that is to intentionally tear apart some of the very things that make you successful today so that there’s room for what will make you successful tomorrow.
5. Innovation is a messy process. Embrace the mess.
Embrace agile development , the startup approach.
— Spend 90 days in a “sprint” to launch a minimum viable product
— Small group of people work on it. Make it “just good enough.”
— Lean startup: continue to iterate based on how customers actually use the product. Test assumptions, iterate, rinse, repeat.
This is the opposite of the legacy organization approach known as “waterfall,” which consists of:
— People sitting around a table for months conceptualizing a new product or service. Getting buy-in. Planning out everything.
— By the time they launch that service, the world has changed around them and it’s already out of date.
6. Encourage risk-taking even if it leads to failure.
— “Learning is about the journey, not the destination.”
— An example is this coding class I’m doing with Dan Schultz: How to Make Almost Anything on the Web.
— Nobody is penalized for trying and failing as long as they publicly document what they tried, what worked, where they got stuck.
— Email list: students help each other out, like sherpas helping each other get up the mountain.
— The key lesson to learn is how to continually teach yourself how to code, because the field keeps changing.
— Very different from teaching timeless art or science that never changes.
7. Finally, we need to get out of our students’ way.
I crowd-sourced this question to students in my startup class, and this is what they told me:
— “Break down silos so we can take more classes from more innovative professors.”
— “Schools and programs in the university need to stop competing with each other. It’s not helpful.”
— “Give us more room for electives for those who want to focus on emerging media and technology. Remove some less useful requirements.”
— “Facilitate collaboration with students who have different skills, like developers and designers, so we can create new products and businesses.”
NEW 300/600: Professor Pacheco’s Digital Media Innovation 3-Pack Courses
April 7, 2014
Journalism and communications fields are undergoing constant disruption from new digital technologies. Stay ahead of the change through these three 1-credit courses from Journalism Innovation Chair Professor Dan Pacheco. Take 1, 2 or all 3 credits — it’s your choice!
All are taught by Newhouse’s Journalism Innovation Chair Dan Pacheco with assistance from industry guests and experts. They’re open to students from all majors at Syracuse University.
DAYS / TIMES: Spring 2016 courses will meet Tuesdays and Thursdays from 12:30 – 1:50 p.m.
COURSE 1: Interactive Data Visualization
Meets Jan. 19 – Feb. 18.
Class numbers: 44138 undergrad, 44240 grad.
Go to the class web site.
Journalists who understand data and analytics are in high demand at top news organizations. Learn where to find good data sources, “interview the data” to uncover stories, and create visualizations that you can publish online.
COURSE 2: New Product Development and Product Management
Meets Feb. 23 – Mar. 31.
Class numbers: 44236 undergrad, 44242 grad.
Tools like WordPress make it easy to publish text and photos, but if you want to do more you’re stuck in the mud. Learn how to conceptualize, define and in some cases even code your own digital products for web sites, mobile sites, mobile apps and wearable devices.
COURSE 3: New Tech for New Media
Meets April 5 – Final Exam Week.
Class numbers: 44238 undergrad, 44244 grad.
Get hands-on experience with cutting-edge technologies that will change the face of media in the future — including the Oculus Rift, Google Glass, and flying camera drones. Learn how to tell stories in a virtual-reality environment using a real, working Holodeck that uses the Rift and Unity3D gaming engine.
Please contact Professor Pacheco at drpachec at syr dot edu, or fill out the contact form.