In the following article I share my experiences from teaching and give you some recommendations on how to dive into the world of Extended Reality (XR). In my case, with a focus on AR and MR experiences. I start with a brief theory introduction, to at least cover some essential basics.
AR, MR, VR, XR or spatial computing?
So first, let's quickly explain some acronyms we love in our tech world. XR refers to extended realities, which is a broad term that encompasses Virtual Reality (VR), a completely virtual world, and Augmented Reality (AR), the real world with virtual objects.
Mixed Reality (MR) is a blend between Augmented Reality (AR) and Virtual Reality (VR). In MR experiences, virtual elements are added to the real world, allowing us to interact with them. This means we can influence both the real and virtual worlds. In contrast, with VR, we solely manipulate the virtual world.
Reality has always been too small for the human imagination. We’re always trying to transcend.
Brenda Laurel - author of "Computers as theater" (1980)
Extended realities are exciting as the only limitation to our experience is our imagination. We love diving into fiction and discovering new realities or escaping our physical realm around us. But it's also diving into the unknown, which might cause fear or makes us cautious, especially in the process of making.
The term Spatial Computing was defined by Simon Greenwold in 2003. He described Spatial Computing as the human interaction with a machine in which the machine retains and manipulates referents to real objects and spaces.
Here are some examples to get an understanding about each discipline:
Anandale (VR)
Anandale is a virtual reality art experience by Kevin Mack. Explore an abstract three-dimensional labyrinth flying through evolving shapes and colors populated with intelligent artificial life called Blorts.
Magic Bench by Disney research
An older example on a mixture of applied AR and MR to interact with virtual characters in a fun way. Please watch the video and explore the disney research site for further details.
Pokemon Go
I guess the most famous AR application out there. No big explanation needed. Maybe it's time to re-look at the Pokemon go app to see where it developed over the years.
Also a simple way to learn simple AR interactions and to get a feeling for the spatial in general.
Others
Microsoft Hololens and it's applications
Start designing in AR or MR – sketch ideas.
Most undergraduate students might feel intimidated by the examples because they only have a basic understanding of 3D and interaction. Their main skill is arranging objects, which they learned in graphic design. The next important question that arises is which program to use. However, instead of finding excuses, we should embrace what sets designers apart and simply sketch our ideas.
Before discussing concepts like presence, agency, the story, or different types of input interactions, always take the time to think about and outline your idea. For all types of designing for new media, there's a big hurdle to learn or dive into some technology one never used before.
Designing for AR – exploring solutions to refine idea
In AR there are many applications to test an idea. Once you have an idea or sketch, you can explore different solutions and understand how they were achieved. This will help you refine your idea and simplify it to get started. Most likely you end up looking at many tools, such as lens studio, AR Core, Spark AR, AR Kit or maybe unity3D, InstaVR or roundme. There are many apps and most of them are for free to get started. Unfortunately the entry barrier for many of the mentioned tools isn't the lowest.
The key here is simplifying. New programs and software can be overwhelming if you're not experienced. Unfortunately, when it comes to new media practices, you have to explore a program or technology to see if your idea is feasible. Instead of being overwhelmed, it's best to simplify an idea to get into the stage of testing and making. So to give you a feeling of easier entry points, I show you three simpler ways to get started:
Artivive
There's Artivive, for example, a platform to create simple AR art by linking classical paintings or posters with a digital layer. Artivive is working as long as no further interaction is needed, and the development process is very simple. Create a video that gets played when a user hovers his capture device (mostly your phone) over the painting or artwork. The object itself could be transformed to a QR code, so there's no additional activation needed.
Similar to artivive there's the magic photos app, that is even simpler. To put it simply, imagine having a still photograph that comes to life as a short video with just a hover of your camera.
To get started with Artivive, here are some tutorials to follow.
Adobe Aero
I was unsure if I should include Adobe Aero on the list, but it's the next logical step after Artivive. It allows you to not only place videos, but also other objects, and easily interact with them in an AR environment.
Adobe Aero is great for exploring and experiencing interaction in virtual spaces. It's the best way to understand how ideas can be brought to life and to learn about spatial environments and how to interact with them.
In terms of the limitations of the software I have to admit that Adobe Aero is a beta software (for years now). Adobe is definitely prioritizing the incorporation of an increasing number of AI features into their software suites in order to stay ahead of the curve in the AI landscape, alongside midjourney and stability AI. Hopefully Apples Visio Pro will push the development of Adobe Aero next as designers need to learn how to design for XR.
Also, for one of my last years classes, most students used Adobe Aero to create their interactive AR posters. By building upon their existing strength in graphic design, students can take their skills to new heights in the virtual realm. Some examples of the students work will follow.
Here you find some good starting tutorials showing a good range of the capabilities of Adobe Aero. You will notice, that Adobe Aero is mainly the app to put together the assets you create in other apps:
Example Projects from Students
As being said, we used Adobe AERO for a course around new media in general, besides looking at other contemporary technology, such as generative design with the help of p5.js or the use of AI image generators.
For their submission, the students were asked to select a topic that suits under the overall theme of "change". A fictional exhibition at a museum. Their task was to create an Augmented poster to promote the exhibition.
Here you find two examples of what the students came up with. One is about the topic dementia and the changing state of mind (left side). The other one is about the topic of speed, the change in speed and the double meaning (right side). What you see are only simple videos. In case of the right example, the Augmented experience is much more interesting, as it is interactive and you are able to "take the pill".
The left example is working better as a simple video. Both great works of the students.
Poster by Mona Kerntke
Poster by Sofia Mari Surkau
Other resources (more advanced and more contemporary):
Get started with Geospatial Creator in Adobe Aero
Create world-scale augmented reality experiences in minutes with Google’s Geospatial Creator
Adobe After Effects
To bring across an idea, there's no need to build AR in "real". After effects is a great tool to simulate a fictive world. Its tracking abilities allow users to simulate real-world objects and envision a fictional future world, all without the need for actual VR or MR glasses. Undergraduate students often use After Effects to learn motion design and interface design. This knowledge can be used to imagine AR or MR worlds and the best part is, there's no need to really build the app, but it's an advanced instrument to sketch an idea even further.
After Effects is perfect for prototyping UI concepts. The only missing component is the lack of a real interaction experience. But the resources required to bring the vision to life must be generated. This process primarily involves utilizing gestures and movement to gain a comprehensive understanding of the interactions involved and therefore also deepens the interaction knowledge.
Check out Volodymyr Kurbatov's YouTube videos for awesome examples of how After Effects can be used. They are a great source of inspiration! Here are some shots from his channel:
Here's a tutorial on how to build the card effect from Volodymyr Kurbatov with after effects:
Iteration for refinement of AR and MR ideas.
When you start learning, it can be challenging. To understand how your design works in a new medium, try different approaches and think about how you've implemented it. Remember the rules you've learned from interaction design. Usability in space behaves quite differently.
There's no "simple" touch, as it's hard to identify if one interacted with an object by purpose or by accident. An early prototype with the help of After Effects or Aero helps to refine the rough concept.
For implementation other, more complex programs are needed, but the true question would be, is this still the space of the designer? Depends on your role. To better understand and explore new media, I recommend exploring new spaces and programs. This will enhance your contextual understanding.
My understanding of generative design, new media design in general is that playing with the medium itself, whether it is the data, the visuality or the code, leads to a better understanding of the medium and therefore a better result.
The trickiest part and most annoying parts are the intersections with other programs or limitations of software. Especially when it comes to importing and exporting data formats from one source to another.
One of the most challenging aspects of teaching is explaining the importance of learning to overcome obstacles, to feel frustrated with a project or software, and ultimately, to redirect and improve the approach towards finding a solution.
Working with AR or MR designs in the current working environment provides a unique opportunity to gain a deeper understanding of the inherent limitations of new media technologies as they emerge. Designing for Extended Realities (XR) is more challenging compared to using AI generators, which have significantly improved in just one year and are now almost ready for the mass consumer market. AR, MR, and VR have been around for a long time, but they haven't gained popularity among the general public. This is because there are several challenges such as hardware and software limitations, concerns about data privacy (like the case of Google Glass in 2012), and changes in how we interact with the technology, which make it quite complex.
Edge cases of developing solutions in Adobe Aero or After Effects
Adobe Aero is a prototyping tool that falls somewhere in the middle fidelity range. It allows for some interactions and also provides a simulated visual experience. The problem with Adobe Aero is not just the beta issues, which cause strange errors like problems with file naming or importing/exporting. There are also strange issues with the file import functionalities, especially when it comes to 3D assets.
So one major issue is placing a real interactive 3D Object on the stage that is animated. For Adobe business subscribers, there are a lot of FBX files on Mixamo to be downloaded for free. For educational subscribers, the option isn't there.
So if you want to have a look at this part, there are easy tutorials, such as this one. To make your own FBX in Blender or Cinema4D, you should carefully adjust the export settings. It would be helpful to follow specific tutorials like the one for Blender or the one for Cinema4D to ensure it meets your requirements. Also a better tutorial on the intersection of Cinema4D and Adobe Aero is this one from schoolofmotion.com.
Working with 3D in various fields like web design, movie production, and game development often involves dealing with file export problems. And I assume we all hate exporting issues from the one source to the other... hopefully our magic AI developments also help to overcome this barrier.
Adobe After Effects is mainly considered a tool for low-fidelity prototyping. It lacks real object interactions, so we simulate interactions to create additional assets. After Effects is better than Adobe Aero in terms of conceptualizing because it allows you to focus on your creative goals without having to deal with technical problems during idea development.
Further Resources
- Road to VR – news site on VR topics
- UploadVR – VR and AR news
- VRScout – News site
- AR/VR journey – News site
- Virtual Reality Pop – Articles around XR
- Publications from Stanford on VR
- Free book on virtual reality by Steven M. LaValle