Author Archive: Daniel Pacheco

Newhouse students experience the future of media technologies

Read this story about what students do in Emerging Media Platforms by Michah Castelo on the Newhouse Web site: https://newhouse.syr.edu/news-events/news/newhouse-students-experience-future-media-technologies-0

Prof. Pacheco’s Interview with GEN Summit with the Global Editors Network

Read Professor Pacheco’s interview, “Will smart glasses replace screens?“, published in Global Editors Network before his GEN Summit keynote in Athens.

MND 413/613: Emerging Media Platforms

Meets: Tues/Thurs 12:30-1:50 p.m.

This class gives you exposure to a variety of emerging technologies that can impact how media is produced or consumed. Assignments are meant to give you a basic understanding of how to create content using a variety of digital tools, rather than mastery of them. The final project is to either run a field test for a media hypothesis using an emerging tech of your choice, or write a research paper about an emerging technology and how it may influence media. Topics covered include:

  •  XR (VR, AR, MR)
  • Aerial video with drones
  • Environmental sensing
  • Chat bots and AI
  • Smart speakers
  • Data visualization
  • Coding for emerging platforms

For a better idea about the course, here’s a story on the Newhouse web site about the class, including testimonials from students who took it.

Prof. Pacheco’s Fall 2020 Classes – Now Open for Registration

My two fall 2020 classes — Virtual Reality Storytelling and Emerging Media Platforms — are now open for registration. Here’s details on both. Questions? Contact me!

MND 545: Virtual Reality Storytelling

Meets: Mon/Wed 3:45-5:05.

This course gives you exposure to virtual and augmented reality technologies, including many that are completely cloud-based and accessible with or without a virtual reality headset. You will learn how to create the types of immersive VR and AR experiences popular at the Sundance and Tribeca film festivals, as well as an entirely new set of cloud-based immersive platforms (Amazon Sumerian and Mozilla Hubs) where stories can be experienced socially with others, with or without VR headsets. Imagine virtual experiences like concerts, conferences, artist gatherings and more, but in VR. At a time when everyone is socially distant due to COVID-19, this type of “Social VR” presents a powerful opportunity to bring back some of the types of events that have been lost due to the pandemic.

  • Social VR
  • 360 video production
  • 3D scanning
  • Creating interactive CGI scenes
  • Light coding for VR using A-Frame
  • Motion capture and character animation
  • Immersive storytelling techniques

MND 413/613: Emerging Media Platforms

Meets: Tues/Thurs 12:30-1:50 p.m.

This class gives you exposure to a variety of emerging technologies that can impact how media is produced or consumed. Assignments are meant to give you a basic understanding of how to create content using a variety of digital tools, rather than mastery of them. The final project is to either run a field test for a media hypothesis using an emerging tech of your choice, or write a research paper about an emerging technology and how it may influence media. Topics covered include:

  • XR (VR, AR)
  • Aerial video with drones
  • Environmental sensing
  • Chat bots and AI
  • Smart speakers
  • Data visualization
  • Coding for emerging platforms
  • Wearable cameras (such as Snap Spectacles)

For a better idea about the course, here’s a story on the Newhouse web site about the class, including testimonials from students who took it.

Dan Pacheco in 3D

Witness the advent of the 3D selfie.This scan of Professor Dan Pacheco was made using the Structure sensor. The model was imported into Sketchfab, a site that makes it easy to share 3D models on the web. Click the Play button on the image, then click and drag your mouse around to see the model from different angles.

Dan Pacheco
by Dan Pacheco
on Sketchfab

You can also embed 3D scans into a game environment using the free version of Unity 4. To learn the basics of how to use Unity, check out Professor Pacheco’s tutorial from the ONA 2014 conference or any of the lecture notes from his Virtual Reality Storyteling class. It includes a Unity template to help you hit the ground running with creating 3D environments.

Click in the image below, then use your arrow keys and mouse to move around. If you haven’t used the Unity Web Player before you may need to download it first. Eventually this extra download won’t be necessary, as future versions of Unity will export environments as WebGL and won’t require a plugin.

The Future of Digital Media Panel

On Sept. 29, 2014, in celebration of dedication day for the Newhouse School’s Dick Clark Studios and the Alan Gerry Media Innovation Center, Horvitz Chair Professor Dan Pacheco moderated a panel about the future of digital media. It included an accomplished group of digital media experts including:

  • Kristina Hahn, Head of Consumer Packaged Goods at Google.
  • Ed Wise, SVP of Branded Entertainment at Turner.
  • Mitch Gelman, VP of Product at Gannett Digital.
  • Larry Hryb, Director of XBox Live at Microsoft.

The panel focused on current and emerging trends in digital media, includingthe rise of YouTube as a competitor to television, branded content, the use of virtual reality and gaming to interest younger audiences in journalism, and creating story experiences in the Oculus Rift. You can read press coverage of the panel from the Newhouse web site, The Daily Orange and Syracuse.com. You can also watch the full 90-minute live recording on UStream here:

Broadcast live streaming video on Ustream

The panel was also covered widely through social media. See what resonated with people in the Twittersphere through this curated Storify by Newhouse graduate student Lauren Cavalli.

Notes from Prof. Pacheco’s presentation at CUNY Entrepreneurial Journalism Summit on Oculus Rift

Here are some notes from Professor Dan Pacheco’s short presentation about the Oculus Rift for journalism at the CUNY’s Entrepreneurial Journalism Summit on July 10, 2014.

I’m Dan Pacheco, chair of journalism innovation at the S.I. Newhouse school. I’m sorry I can’t be there in person.  You can also get some background on this post at my site: Journovation.syr.edu.

Everything I talk about today will also be covered in detail on an ONA 14 panel called The Holodeck is Real in September about gesture and body-responsive interfaces.

One of those is the Oculus Rift, a VR headset that for the first time in history can make you feel like you’re somewhere else. Unlike Google Glass, which augments your reality, VR is more like transferred reality or 3D transportation. I like to say that for journalism, this technology makes it possible to almost literally walk a mile in someone else’s shoes.

The effect is so real that the common response from people who use the Rift for the first time are usually along these lines:

“Oh my God, I feel like I’m somewhere else.”
“This is crazy!”
“This is amazing!”
“That’s weird. Where are my hands?”

And that last part is important because what it says is that the feeling of being physically present somewhere is is so strong that your brain is disturbed by the fact that it doesn’t see hands and legs.

Two Types of Immersive Experiences

There are currently two methods for using this for journalism: 1) 3D modeling in a gaming environment, and 2) 360-degree video.

3D Modeling and Gaming Environments

Let’s talk about 3D modeling. Imagine the most realistic scene you have seen in an Xbox or Playstation game — whether it’s Call of Duty or Grand Theft Auto. Then imagine that the models for the buildings are based on buildings from the real world, from a real street. The script of the game isn’t fictional, but reflects either something that really happened, or is a recreation of something that happened or would typically happen.

Using readily available software like Unity3D, 3ds Max and Photoshop you can create a scene with high fidelity to the actual place in a matter of weeks. You can then create and animate avatars using a site called Mixamo to create somewhat realistic people and map audio recordings to them. This is something that USC’s Nonny de la Peña pioneered a couple years ago. Back then it would take her months to build a single scene, but recent advances make it possible to create such a scene in just a single week. I owe a lot to Nonny for both inspiring me, and sharing her knowledge, around VR and journalism.

I have recently been teaching some of my students how to create VR stories, and one is getting quite good. We hope to debut our first immersive story at the Online News Association conference in Chicago in September. At a workshop we will show journalists how they can get started even without any 3D modeling expertise, as the Unity Asset Store has thousands of ready-made buildings, objects and avatars that cost $5-$40, and even some free objects. But if you want to delve into 3D modeling, be ready to hire an expert — ideally someone who has worked in the gaming industry. And get familiar with finding and applying 3D textures.

360 Degree Video

The second method, 360 degree video, is relatively new and thus crude. But I believe that when the Rift and similar devices have high enough definition and frame rate, 360 video will become its own industry. Digitally downloaded experiences will quickly become a popular and almost natural new form of media.

What makes 360 video so different from 360 photos is, again, this sense of presence and being taken somewhere else.  A Silicon Valley startup called Jaunt (http://www.jauntvr.com/) is already working with companies like Dreamworks to capture 3D video that can be experienced in VR headsets, with video shoots running into the hundreds of thousands. The pricing is not so helpful for journalists, but at the other end of the spectrum there’s a new startup called Giroptic (https://www.kickstarter.com/projects/giroptic/the-worlds-first-full-hd-360-camera) that’s shipping a $350 360 HD waterproof camera. I’m happy to say that I will be getting one of these in November thanks to backing the project.

On the Horizon: 3D Scanning

A few years from now you should expect to see the output from a third category of experience seamlessly appear inside these VR environments. The same technology used to scan objects for 3D printers can be used to create objects with accurate texture-mapping into a VR environment. Matterport (http://matterport.com) and Structure.io (http://structure.io) are two startups to watch in this space.

OK, So What are the Applications for Journalism?

There are so many journalistic applications for this that I almost hasten to list ideas out of fear that it will limit peoples’ imaginations. But here are a few projects I and some collaborators are working on.

1) Recreating events that actually happened with full audio and a cast of avatars representing real people (Nonny de la Peña’s speciality). This is very labor intensive, but also very powerful. It creates a sense of empathy that is hard to get without actually being there. This is the next best thing.

2) Science and nature visualizations. Example: showing the effects of flooding and sea level rise in different coastal areas. Through a Unity 3D plugin, you can create the terrain for any place on earth in 5 minutes, then flood it with a plain of water that animates upward. When you put the first-person camera 6 feet in the air you can even simulate what it’s like to be caught in a flood.

3) “Gamified” explanatory pieces, where the user walks around and interacts with different objects and avatars and, through the game experience, learns the different elements of a story. No two person’s path will be the same, but if you use the right game incentives everyone will end up with the same information — and likely retain it better due to the experiential learning involved.

4) Travel journalism. Going to Machu Picchu, or some other exotic location? Take your waterproof 360 degree camera so that others can go along with you!

5) Sports journalism. I predict that at the next World Cup, you will be able to purchase a “V Ticket” to attend virtually. 360 cameras will be all over the stadium, at the sidelines and even at the goalposts or on the field. You’ll use a game controller to switch camera angles — and you will pay dearly for the best vantage points.

What’s the Revenue Potential?

3D game grids and immersive video have obvious opportunities for revenue, including:

  • New types of ad placements. In 360 video there’s a circular area directly above and below you, and some videographers are placing sponsorships there.
  • Natural product placement in VR gaming environments. For example: if a McDonald’s happened to be in a scene you’re recreating, McDonalds could be urged to pay a little bit for a higher-quality sign, or to render the inside of the restaurant if people want to go inside. This is tricky because in journalism we can’t allow money to affect what’s covered, but with the right ethical guidelines in place I think it could affect how many resources are put into recreating it.
  • Explanatory historic pieces that are tied to companies. For example: a visual 3D history of the personal computer, sponsored by Microsoft.

If you’re interesting in learning more about VR journalism, or are working in this space and want to share ideas and practices, please contact me at drpachec at syr dot edu or pachecod at gmail dot com. Thanks!

Dan Pacheco

The “Holodeck” is here. How will it influence storytelling?

It’s not every day that you stumble across a fundamentally new form of storytelling, but that’s exactly what’s happened in the last few months thanks to a confluence of exciting new technologies and some smart, motivated people working in the so-called field of “virtual reality.”

The arch from the Holodeck of the USS Enterprise (NCC-1701-D). Source: Wikipedia. http://en.wikipedia.org/wiki/Holodeck

But before I continue, I just want to warn you that everything you’re about to hear and see will sound absolutely ridiculous, almost as if it was dreamed up for a science fiction story or lifted from Star Trek. And in a sense, as with most new technologies, it was  — but science fiction has now become science fact.

What I’m about to show you is how the “Holodeck” experience first popularized by Star Trek the Next Generation, and also explored by novelists William Gibson and Neil Stephenson, is starting to become real.

It’s set up in my office right now, and also about 10 miles west of me in the empty barn, affectionately referred to as the “Holobarn,” of my creative hacker friend Lorne Covington who operates under the banner of NoirFlux.

This video from my class at the S.I. Newhouse School, New Tech for New Media, shows a little of what’s possible. YouTube Preview Image

What you will see in the video is three distinct technologies: 1) An Oculus Rift VR headset that generates a 3D image, 2) A Kinect depth camera that monitors your movement in 3-dimensional space, and 3) a graphics PC that synthesizes all of these inputs and outputs the experience wirelessly to the Rift. There’s also a second depth camera mounted on top of the Rift that allows the tester to see her hands as she moves them around.

In this case, the VVVV environment that Covington has programmed is creating the illusion of a screen floating in the middle of the air inside a house based on an Escher painting. The screen has Google Maps projected onto it, and the student is navigating the map by using iPad-like swipe gestures. (I warned you that this would start to sound ridiculous!)

Turning the classroom into a holodeck

All 16 students in the NTNM class went through this experience, and a few conducted extended labs. Together we have have been thinking about what it means for storytelling in the future — if you can even call it that. It’s more like story-experiencing, or what VR pioneer Nonny de la Peña refers to as Immersive Journalism. When you work with VR you quickly start to realize that there’s no vocabulary to describe some of these concepts.

Using the Unity3D gaming engine, we’re now experimenting with creating virtual worlds that people can walk through — like this city in a canyon that I arranged myself in a couple hours.

YouTube Preview Image

We have found that once you get over a small learning curve, it’s surprisingly easy to create rich environments using 3D models in the Unity Asset store, then let people explore them. Some of my students are taking the extra steps to do simple usability testing to get a sense of how people respond to these types of immersive experiences.

One student recently sent people through a virtual world and asked for feedback on what it was like, which you can read about in the lab report. What I find most interesting is how universally positive and excited people are after entering these environments — not the usual response to new technologies. Some remark that they could stay in it forever, and they only come out after they feel nauseus — an effect that is more likely when using the Rift from a chair. When you walk around in the “Holodeck” configuration which uses your physical location in the room for navigation, motion sickness is less common. Your mind receives the signals of movement that it expects based on the feedback from your eyes.

What will you put in your Holodeck?

I predict that we will soon start seeing similar Holodecks all over the place. That’s because it’s not a single product you buy off the shelf. It’s made up of a bunch of different technologies, most under $400 and many open-source, that are combined to create immersive experiences. With enough tinkering and patience anyone can do this for $2,000 or less depending on what other computing equipment they have lying around.

While there are many implications for this technology, as the Newhouse School’s Horvitz Chair of Journalism Innovation my focus is on journalism and storytelling. What kinds of stories and experiences can we “tell” — or “transfer” — to help people understand complex ideas by letting them walkin through and manipulate them? The possibilities are endless. Here are just a few ideas I’ve been contemplating over the last few months:

  • Experience the effects of climate change: It’s one thing to read about what rising ocean levels mean for coastal areas, and quite another to be standing in New York City, London or New Orleans, dial in a date and climate study, and see if you’re underwater.
  • Science and medicine: When the next big cancer drug comes out, you could walk inside a cell and look around as the drug does its work on cellular DNA. Heard about that new planet NASA found? Plug in your goggles and go there.
  • Visualizing data: Data visualization, much driven by the data-driven documents codebase, has been a big trend in journalism lately. A next logical step is to create 3-dimensional, immersive visualizations that you can fly through and manipulate using your hands and entire body.

Over the next year I plan to spend time with media companies exploring some of these ideas through my new FutureForecast Consulting business. I also hope to host a panel and workshop about it at the Online News Association conference in September along with Noirflux, Nonny de la Peña and some others.

And speaking of Nonny, she is already working on her third immersive documentary using one of the prototypes used by Palmer Luckey before launching the Oculus Rift. Here’s a video of her second piece, Project Syria.

YouTube Preview Image

Going beyond virtual reality

Perhaps you’re thinking, OK Pacheco, I get it. Sounds like a gaming platform. How does that relate to journalism? And this is where things start to get really trippy. Are you ready?

In the not too distant future, I’m convinced that it will be possible — maybe even commonplace — to inject 3D information that is either scanned or inferred from the physical world into a Holodeck environment. At this point, the term “virtual” doesn’t really do it justice. It may feel more like transferred reality, or what Fast Company writer Rebecca Greenfield calls 3D Transportation.

The pieces to do this are all available now, such as:

  • 3D scanning drones like SenseFly are scanning mountains and mining sites as we speak. What happens when this technology costs not $40,000, but $400? It will happen.
  • A company called Replay Technologies can create Matrix-like instant replays using data captured from 4K cameras all around the stadium. Here’s a video of the replay, and here’s more about how it works.  What happens when such systems can be placed anywhere, perhaps even carried by drones? Then the 4K arena can be taken anywhere — such as the site of a recent earthquake or a large outdoor event — and people can virtually teleport in from home.
  • Small handheld scanners like Google’s Project Tango or Structure.io will make it possible to scan places as you visit them, then export what you scanned as 3D object files.
  • And finally, craziest of all, data from 2-dimensional photos can now be combined to extrapolate realistic 3D objects. Cornell Professor Noah Snavely recently used randomly posted photos from Flickr to create a 3D point-cloud reconstruction of Dubrovnik. And another creative hacker has created a way to see 3D point clouds using data from Google’s Street View.

Based on this, what’s to stop someone (maybe me!) from going to a place like Machu Picchu, scanning the inside and outside of every building, scanning it from a drone above, then sending you a link so you can virtually go to Machu Picchu from your Oculus Rift?

This is the point at which you Trekkies out there can confidently say, “Computer, Arch. Take me to Machu Picchu circa early 21st century earth.” It sounds like science fiction or magic, but the technology is available to do all of this now. As Arthur C. Clarke so eloquently put it, “Any sufficiently advanced technology is indistinguishable from magic.”

Collective community visioning

On a final note, I have been thinking about how advances in 3D scanning, visualization and printing can be combined to better engage and inform communities around building projects. In the spirit of throwing spaghetti on the wall to see if it sticks, I’ve thrown this idea out there under the banner of Visualize Cities. I’m already talking logistics with stakeholders across Syracuse University, including at the School of Architecture and College of Visual and Performing Arts. The next step is to find some community partners. If you’re also interested, fill out the form on the page and I’ll get in touch.

– Prof. Dan Pacheco

 

 

Notes from Professor Dan Pacheco’s Participation in Chancellor Syverud’s Inauguration Panel

Professor Dan Pacheco recently had the honor of speaking on a panel at Chancellor Kent Syverud’s inauguration on “Great Universities in the Next 25 Years.”  This was a distinguished and diverse panel of 8 SU experts on a range of topics of interest to universities now and in the future. You view a video of highlights here:

YouTube Preview Image

Pacheco addressed this question: “What can we abstract from successful innovators in business and innovative organizations to change productively, and preserve what is most important?”

 

Each panelist had only 7 minutes to respond, so . Following are the notes Professor Pacheco prepared before the panel.

1. What makes a university great? The people and talent.

I was recruited out of a startup. Why did I come here? Some people thought that was weird. The answer: because a university is an amazing collection of talent that is truly unmatched anywhere else.

My goal was to help students become more innovative and entrepreneurial in journalism and media by connecting them with each other.
— Students in design, engineering, business
— Together they are the ones who will work together to create the future, and we can start that here.

What makes a university amazing is that there are smart, young, energetic individuals. Need to help make sure they don’t just burrow into rabbit holes for their individual talents. Need to mix them up, because it’s in the mixing that they find new innovations that change the world.
2. Interdisciplinary collaboration isn’t just a “nice to have” in the business world. It’s required in order to make new things happen. 
We need to provide those opportunities here on this campus
— Between departments and schools
— Between complementary classes
— Software engineers collaborating with designers
— Designers with content producers
— Content producers with marketing and business

Many different skills are needed to create an amazing new product like Facebook or Twitter, or Google Glass, or the Oculus Rift. The idea that one person or one group or one department can do everything necessary to make things like this happen is false.
The idea that Steve Jobs invented the iPhone — if he were alive today, he would say that’s false. The iPhone was created by a team of many people with different types of skills, and the most important members of those teams are always the customers.

3. Embrace new technologies, know the basics of how they work.
— Understand new technologies (the puropose of the Digital Petting Zoo.)
— Learn just enough to be dangerous.
— If you can get past that danger zone, learn more so that you become an expert. But if like most you can’t do that, at least understand how it works at a fundamental level so that you can have an effective conversation with an expert, and together you can build the future.

4. Be comfortable with disrupting yourself so you have a future. If you don’t do it, there are many others who will.
— There are plenty of companies that are circling the wagons of academia who are happy to disrupt what we have. We cannot prevent that, but we can get ahead of the disruption so that we have a role in it.
— Companies like Apple, Google, Facebook and startups understand this all too well.
— The only ones that survive a paradigm shift and replace what is the next big thing are the ones who disrupt themselves.
— Consider a company like Facebook that has a billion registered users. A typical company would be comfortable with sitting on top of that pile of money and be content not to change anything. But in fact, that’s not what Facebook does.  They just bought Oculus Rift for $2 Billion, WhatsApp for much more and Instagram before that. These companies could be seen as potential future competitors, but now they’re part of Facebook’s future.
— Nobody wants to be on the receiving end of change. It’s a lot easier, and more fun, to be the one who is making change. The only way to do that is to intentionally tear apart some of the very things that make you successful today so that there’s room for what will make you successful tomorrow.

5. Innovation is a messy process. Embrace the mess.

Embrace agile development , the startup approach.
— Spend 90 days in a “sprint” to launch a minimum viable product
— Small group of people work on it. Make it “just good enough.”
— Lean startup: continue to iterate based on how customers actually use the product. Test assumptions, iterate, rinse, repeat.

This is the opposite of the legacy organization approach known as “waterfall,” which consists of:
— People sitting around a table for months conceptualizing a new product or service. Getting buy-in. Planning out everything.
— By the time they launch that service, the world has changed around them and it’s already out of date.

6. Encourage risk-taking even if it leads to failure. 
— “Learning is about the journey, not the destination.”
— An example is this coding class I’m doing with Dan Schultz: How to Make Almost Anything on the Web.
— Nobody is penalized for trying and failing as long as they publicly document what they tried, what worked, where they got stuck.
— Email list: students help each other out, like sherpas helping each other get up the mountain.
— The key lesson to learn is how to continually teach yourself how to code, because the field keeps changing.
— Very different from teaching timeless art or science that never changes.

7. Finally, we need to get out of our students’ way.

I crowd-sourced this question to students in my startup class, and this is what they told me:

— “Break down silos so we can take more classes from more innovative professors.”
— “Schools and programs in the university need to stop competing with each other. It’s not helpful.”
— “Give us more room for electives for those who want to focus on emerging media and technology. Remove some less useful requirements.”
— “Facilitate collaboration with students who have different skills, like developers and designers, so we can create new products and businesses.”

Next News Startup Survey