In the first part of our interview with the Sundance Kids, we looked at their creative process and got a feel for what it's been like working on a Sundance installation. In this part, Charles, Garrett, and Paulo share their tips and recommended resources for those looking to make holograms interactive between several people, how designing for AR is like permission marketing, and why you can't forget the smallest details.
Key Takeaways from the Engineers Who Built Journey
→ Imitate the real-world as much as possible (physics, people's points of view)
→ Always be thinking of how multiple users would be seeing and interacting with the holograms
→ AR is a new medium for storytelling: understand the technology as well as you can, learn its limitations and strengths, and build your experiences and content accordingly
Tips, Learnings & Resources from the Sundance Kids
Victor: Man, I envy the fact that you guys have a lot of patience for programming (laughs). Now without giving away any secret sauces, do you guys have any tips for other devs looking to create collaborative apps in AR?
Charles: Calibration, calibration, calibration. It’s the key. Collaborative experiences in AR are all about making users feel like they are interacting in the same world space, even though what’s really happening behind the curtain [or should I say circuit board] is a lot of mind-trickery.
Mind-trickery?! How did the Jedi get their hands on this? The last time they got involved, a huge civil war broke out in a galaxy far, far away.
Charles: (laughs) Our testers often pointed out that poorly calibrated experiences instantly took them out of the experience. Paulo especially dedicated a lot of his time trying to minimize collaboration errors as much as possible, using a lot of really creative [and math intensive!] methods. And while we were close to getting things right, I have to say there’s still a lot of room left for improvement here.
Garrett: One tip I have is to make it clear when an interaction is intended for both people, or for one person at a time. When dealing with multiple people in the same space, there’s a lot to consider: Can both people see the same things? Do you tweak graphics and animation to look great for both parties, or keep things consistent? Do you grey out or fade objects only intended for another user, or simply allow location and distance to imply this? We’ll be working on these and other questions as we continue to explore multi-user AR in our projects.
At Sundance New Frontier, two participants look at the same holograms with the Meta 2 DK.
Paulo: The concept underlying collaborative AR apps is simple and straightforward: you need to make objects look like their real-world counterparts, both users should see the same object in the same place, interact with the object regardless of their position, and see the object from their [respective] points of view.
But when it comes to actually coding a collaborative app in AR, the technical challenges are actually huge: you have to make sure there is low latency while people are handling the object, that object movement is as smooth as possible, and calibration to align the object from both users’ perspective. In order to achieve those things, we can use a variety of techniques and design considerations. One of those techniques is using common origin in the space between users. Another consideration is to design interactions [with holograms] to be as flexible as possible. For instance, people could tap on the brain pieces in Journey with one finger, with their entire hand, or even by grabbing the pieces, and we’ve seen people interacting using all the possible ways to do it.
Watch Journey to see how the programming and UX & UI design considerations came together
In the video, you see the entire experience the audience at Sundance saw. And more importantly, you can see how the Sundance Kids implemented the design considerations they discussed – not too shabby work!
As someone who doesn't know a line of code to save his life, your tips actually made a lot of sense. Off the back of the previous question, what were some of the resources and or shortcuts that helped you guys?
Charles: We made use of existing APIs compatible with the Unity engine to help our process. One of the shortcuts we used for creating content was taking video assets from the creative team and projecting them directly onto a 2D transparent plane. We could then fold / bend the plane in 3D space, giving it the impression of a fully 3D object. Looking back on the project, we probably could have taken more shortcuts like this to decrease development time.
Garrett: From the technical art and illustration side, many of the challenges I faced were related to rendering. I spent a lot of time on this project learning about and troubleshooting transparency problems in Unity. Random forums across the Web including the Unity forums, as well as Unity answers and [support] docs were useful here, but also looking at real examples from the Asset Store. Another hugely useful resource were the built-in shaders, which need to be grabbed separately from Unity’s website, the included Unity Cg files, which can be found in the install path, and Alan Zucconi's blog. He also has an excellent book on Unity shader writing called Unity 5.x Shaders and Effects Cookbook.
Paulo: One of the things that helped make the experience look unique and more compelling was the collaborative interactions, built upon UNet, which is [a portmanteau] of Unity Networking and allows people to add a multiplayer aspect to the things they're making with Unity. We also worked with shaders and animations a lot.
Now taking that question and applying it broadly to everyone out there who is thinking about telling compelling stories in AR, what are the most important coding / creative learnings would you want people to know?
Charles: I think it’s really important to understand scope when building any large project. Lacking a full grasp of the scope on my part contributed to harsher development timelines. It’s always smarter to overestimate how much time it takes to finish a task, especially when it comes to new technology.
Ah, yes. That reminds me of that famous cartoon where a simple tire swing gets misinterpreted by every department in a company:
Charles: Yeah, it’s also important to think about AR as a completely new medium, and to constantly ask why the story you’re telling necessitates the use of AR technology. While our project made a lot of breakthroughs in AR collaboration and storytelling, I think with more time we could have created even more exciting and experimental content that would have further pushed the limits of AR interaction. Certainly that’s the next step for us, so look out for that soon!
Garrett: Hmm...One tip I have is learn from the massive amount of work that’s already being done in games, film, and VR, but also challenge anything that doesn’t seem to fit. The vast pool of rendering, design, and interaction knowledge is a huge resource. The pieces you need to construct an answer may be scattered around the Internet. Learn from the work going on around you, but try new things as well. Understand the technology as well as you can, learn its limitations and strengths, and build your experiences and content accordingly. Test everything, constantly. Check both your visuals and your code constantly, on the device. And watch lots of good anime for inspiration.
Paulo: My advice would be in order to make things look real, you need to follow the real world's rules. I know I'm stating the obvious here, but it's key that you keep real-world physics in mind because the smallest details [of object interaction] are important for what's seen in the final cut. For example, real-world objects don’t move linearly because of physics and instead move according to an acceleration curve.
And especially from the users' perspective, you don't want to make any sudden surprises that will startle them, unless you're deliberately trying to shock or scare users. For example, towards the end of Journey, we have the holographic brain slowly fill users' fields of view as it gets larger and fades into the universe, as opposed to having the brain suddenly and instantly appear in people's faces. Delightful surprises in AR are good if its gradual / predicted. Surprise the user because in the end, we blow up the glass brain shader, we made it gradually bigger. Surprise for the user is good if its gradual / predicted.
Delightful surprises in AR are good if they gradually appear / are anticipated by users.
Ahhh...that is an interesting AR design concept. It's almost like how marketers have to keep in mind that marketing is permission-based, and that consent must be given by the audience, otherwise you're going to create a terrible, unwanted marketing experience for people.
Paulo, Charles, and Garrett: Exactly!
I reckon you guys seem to know this marketing stuff! Have you ever considered changing careers? (laughs) Thank you again for sharing your learnings, gents!