Designing Immersive Experiences

A two week course combining interactive spaces, prototyping, and mixed reality.

Combining interactive spaces, prototyping, and mixed reality, student will explore how our spaces will fuse the digital and physical. In this course we will explore how to craft meaningful interactive spaces through practical, hands-on prototyping and scenario building. We will introduce and explore concepts from the design of internet connected objects to the worlds of virtual and mixed reality. An emphasis will be placed on rapidly prototyping and exploring these designs.

Course Introduction & Process

Each day we see and interact with a variety of screens. From handheld mobile devices to large-scale digital displays, our current technologies utilize screens to provide flat views and interfaces into a vast world of digital materials. However, there is a change happening as these digital technologies transition into the objects and environments around us. Interfaces are moving from a world of ‘magic paper’ to a world of objects and environments.

“Computing is not about computers any more. It is about living”

— Nicholas Negroponte

We live in an amazing time where digital technology is beginning to take many different physical forms, with a rise in new interactions and sensorial experiences. But how will these new connected physical-digital spaces impact our lives? What do we expect out of space as we do with the rest of our digital artifacts? As these technologies become ubiquitous how do we shape them to be meaningful parts of our lives?

From this perspective we see our physical world becoming infused with the digital, but the opposite is also happening. Virtual and Mixed Reality offers a new take on these questions by letting us embed digital objects with properties from the physical world such as location, scale, and persistence in the world. How do these technologies work to create and augment environments?

In this course, we will explore how to craft meaningful connected spaces through practical, hands-on prototyping and scenario building. We will introduce and explore concepts from the design of internet connected objects to the worlds of virtual and mixed reality. An emphasis will be placed on rapid prototyping and exploring these designs. Together we will explore a number of quick exercises to better understand these technologies.

With the learned knowledge of prototyping techniques, affordances, physical materials and digital tools, participants will build a working physical prototype to communicate their final idea. They will learn how to use immersive storytelling rather than screen-based presentations to demonstrate the experiences.

Week 1: Day 1

Day one covers the logistics of the course and begins the plan for what will happen at the end of the week in terms of showing the prototype projects to other courses or visiting guests.

Lecture: Intro to the Course (link to Google Slides Presentation)

Intro to the Course Slides

Kick-off Session for the Course:

Tell us a bit about yourself... who are you?, where you are from?, what interests you / what you do?
What would you like to get from the course?
Describe an environment you have experienced that you love or made a meaningful impact on you.

Exercise: How to Brainstorm / Intro to Bodystorming

Over the last decade brainstorming has come under fire as a methodology for some good reasons. It can produce less ideas with less variation. Read this Harvard Business Review article which sites some of the research for more information about the controversy. Many of us have experienced a brainstorm that fell flat, here are some guidelines we use to frame more effective brainstorming.

Explore the Terrain of the Idea

Don't be afraid to get weird, take tangents, and express the ideas you are worried will make you look foolish or unknowledgable.

Start with the agreement that the goal of the Brainstorm is to map the terrain of the idea together. One of the strengths of a group brainstorm can be building on each others ideas to find new areas to explore. Everyone should play a role in facilitating and advocating for others in the group.

“We do not like to appear “ignorant” or “stupid” or just plain wrong. Of course, we all know that our own minds are full of messy confusion, and that many others are in the same plight, but we imagine that some minds are tidy and neat and sharp. We see no reason to advertise not being in this class, especially in the presence of people such as bosses and teachers who have power over us. So voices within caution us to be careful of what we say: Talking too much might reveal what kind of mind we have and make us vulnerable. Eventually this caution becomes a habit.”

— Seymour Papert, Constructionism in Practice

Constraints in Design (and life!) can be seen as empowering (read more about the value of constraints) the process of problem generation and problem solving. Let's walk through an example that might help elucidate the way constraints can help the process. Imagine for our idea we have been struck by the amount of violence in the world and we would like to reduce that violence. We decide to start brainstorming with the question...

How might we create world peace?

It's a great question, but I suspect your brainstorm will be ambling and unlikely to produce many ideas that are solid enough to build on. The terrain is so vast that it's unlikely you will cover it in any meaningful detail. How could we improve the brainstorm? Constraints.

How might we increase peace with a daily ritual?

This will be a much more productive brainstorm because the specificity will help you cover a more manageable amount of terrain. This is the altitude that will yield the best brainstorms results. Can there by too many constraints? Absolutely...

How might we triple peace with a evening ritual using Android phones in Chicago with a cubist artistic approach?

You can probably come up with some ideas, but you will get diminishing results quickly. The problem is that there are a lot of constraints to keep in mind at the same time and the area of coverage has become too constrained. This is sometimes referred to as 'overfittting' the problem space because the ideas are too close to one or a few specific solutions.

So before brainstorming come up with some different 'How might we' style questions to ask the group. If you are having trouble forming the questions, this might help to create a variety.

How might we use the ____ to ____ with _____ ?

How might we create ____with___to____ ?

How might we showcase ____ with ____ in order to _____ ?

The blanks could be goals, technologies, places, methods, audiences, or any else you find useful. Now that you have your questions for the brainstorm, follow these rules for the brainstorm session itself....

1. It’s blue sky time, defer judgment
2. Encourage wild ideas
3. Build on the ideas of others
4. Stay focused on the topic
5. One conversation at the time
6. Be visual.
7. Go for quantity
8. It’s about ideas generation, not idea evaluation.
9. Respect everyone and let everyone speak
10. Have fun!

This presentation covers the thoughts behind the course, our history in this area, and thoughts about UX for the future, when we are unable to distinguish the physical and digital.

UX for Created Realities

Demo: Connecting Physical and Digital Worlds

Makey Makey

https://makeymakey.com/apps/

Spacebrew Jam Session-7.jpg — Introduction to Spacebrew

We use Spacebrew.cc to help construct interactive spaces for the course.
Additional Resources:
Spacebrew Admin Interface
Spacebrew Javascript Slider Example
Spacebrew Processing Library and Example

Week One: Day 2

Day 2 kicked off with and introduction to 3D Prototyping methods, both physical and digital.

3D Prototyping Presentation

We then introduced YUXI: The Mixed Reality Hardware Toolkit. It started with installing drivers and using the Serial Console cable to connect to the Raspberry Pi. We learned that installing both drivers is a better methodology for getting started.

Connecting to the Raspberry Pi

After installing the drivers, to connect via the serial cable you must first look for the name used to connect to the Pi

ls /dev/cu.*

And then you can use the screen command to open a terminal session to the Pi, for example:

screen /dev/cu.SLAB_USBtoUART 115200

You can use raspi-config to change the wifi settings and enable SSH. Once enabled you can login to the device via ssh like this..

ifconfig
ssh pi@192.168.1.1

The address should be the address that your pi has received from the wifi router. You can use ifconfig to determine the address of your pi on the network. Once connected we began to run through 2 demo projects.

Getting Started with YUXI: The Mixed Reality Hardware Toolkit

Hello World YUXI Example

Lights and Buttons YUXI Example

We learned some basic Terminal Commands

ls
screen
pwd
cd
cd ../
clear
mkdir
sudo raspi-config
ping google.com
control-c (stop a command line action, sometimes control x, z, or d are useful too)
nano/pico (text editor)
ifconfig
ssh pi@ [192.168.1.xx]
You can learn more at https://opus.nci.org.au/display/Help/Linux+Command+Quick+Reference+Guide

Some Yuxi Related Tools

Raspberry Pi
Python
AFP->Mount RPI - run ‘sudo apt-get install netatalk’, then use the afp://YOURIPADDRESS to mount the raspberry pi as a drive on your mac
VS Code
Unity
Spacebrew
Git/Github

Week One: Day 3

In the morning we went through a presentation about AR/VR/MR/IRL…

All the Realities: AR/VR/AV/MR/IRL

We went through Unity and Unity scripting. Code snippets here.. https://github.com/Sensorycc/YUXI-Mixed-Reality-Hardware-Toolkit/wiki/Useful-Code-Snippets

Example Script

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class CapsuleInteractions : MonoBehaviour {

    public Renderer rend; 
    public float speed = 10.0f;
    public Color altColor = Color.red;


    // Use this for initialization
    void Start () {
        rend = GetComponent<Renderer>();
    }
    
    // Update is called once per frame
    void Update () {
        if (Input.GetKeyDown ("s")) {
            toggleCapsule();
        }

        // change rotation
        transform.Rotate(Vector3.left, speed * Time.deltaTime); 

        // change the color of the object
        if (Input.GetKeyDown (KeyCode.R)){
            altColor.r -= 0.1f;
            rend.material.color = altColor;
        }
        if (Input.GetKeyDown (KeyCode.G)){
            altColor.g += 0.1f;
            rend.material.color = altColor;
        }
    }

    public void toggleCapsule() {
        rend.enabled = !rend.enabled;
    }
}

It might help to look at a few of these resources:

The tip and tricks page for YUXI - https://github.com/Sensorycc/YUXI-Mixed-Reality-Hardware-Toolkit/wiki/Tips-and-Tricks

The PySpacebrew page for Python Spacebrew commands - https://github.com/Spacebrew/pySpacebrew

The Unity Spacebrew documentation - https://github.com/Spacebrew/spacebrewUnity

Introduced the hardware challenge where teams were given sensor or actuator to create a tutorial with as specified here… https://github.com/Sensorycc/YUXI-Mixed-Reality-Hardware-Toolkit/wiki/What-your-tutorial-should-be

SenseHats - https://www.adafruit.com/product/2738
Neopixel - https://www.adafruit.com/product/1376 /
- Fade Candy https://www.adafruit.com/product/1689
Cap-touch - https://www.adafruit.com/product/2340
Power Relay - https://www.adafruit.com/product/2935
Solenoid - https://www.adafruit.com/product/2776
Servos - https://www.adafruit.com/product/1449

We dedicated morning time to trying new experiences in HoloLens and Oculus Quest.

Week One: Day 4

In the morning we went through a tutorial on Unity and Physics which can be found here… https://github.com/Sensorycc/YUXI-Mixed-Reality-Hardware-Toolkit/wiki/Tutorial-on-Physics-Events-in-Unity.

The rest of the day was spent working on projects, documenting, and gathering insights to be used the following week.

Week One: Day 5

Each group presented was asked to present a demo and a tutorial for their specific sensor or actuator. These were the teams and the links to their tutorials…

SenseHats

Team 1: Amit, Shailee
Team 2: Siddharth, Zoe, Mitsu
https://www.adafruit.com/product/2738

Neopixel

Team 3: Josheph, Aditi
Team 4 :Fei, Jacob - https://github.com/Sensorycc/YUXI-Mixed-Reality-Hardware-Toolkit/wiki/World-On-My-Shoulders
Neopixel https://www.adafruit.com/product/1376
Fade Candy https://www.adafruit.com/product/1689

Cap-touch

Team 5: Aakash, Suniti
Team 6: Laurin, Hiroki
Cap touch hat https://www.adafruit.com/product/2340

Location:

Copenhagen Institute of Interaction Design (CIID)

Toldbodgade 37b, 1253
København K
Denmark

Learning expectations:

Using your hands – learning to make physical prototypes of all scales, starting at the analogue level, and working up to integrating digital components
How to use analogue and digital tools to create interactive experiences
Deepen prototyping skills and develop an understanding of how and when to use each technique
Learning basic methodology to create an experience through user journeys, research, prototyping, user testing, body/brainstorming, workshopping, etc
Learn practical idea generation and critical thinking methods
How to create & use physical affordances, and how to design for action
Experience and develop a vocabulary for Virtual and Mixed Reality interaction design
Develop an ability to find and expand insights related to interactive environments
Prototyping communication skills – learning how to create and use prototypes, and when (fidelity vs. content, contextual use)

Downloads and Links:

Unity Resources:

Other Resources:

Game to Learn the Terminal

Week Two: Day 1

We shared our insights from the previous week (https://github.com/Sensorycc/YUXI-Mixed-Reality-Hardware-Toolkit/wiki/CIID-2019---Insights-week-1) and clustered them into groups. We rearranged the teams to pursue one of these areas of inquiry.

Group 1 Insight : Vulnerability
Students : Hiroki, Zuyi, PJ, Suniti

Group 2 Insight : Virtual Agency
Students : Joseph, Zoe, Silvia

Group 3 Insight : Multiuser Social
Students : Byron, Aakash, Fei, Jacob

Group 4 Insight : Surprise
Students : Laurin, Aditi, Minatsu

Group 5 Insight : Physical Framing
Students : Marc, Sophia, Lea, Siddarth

Group 6 Insight : Microinteractions
Students : Mitsu, Pooja, Gaana

Group 7 Insight : Physical I/O
Students : Davi, Amit, Yang, Shailee

We presented additional code and animation tutorials, which can be found here..

Animation: https://github.com/Sensorycc/YUXI-Mixed-Reality-Hardware-Toolkit/wiki/Animation-Tutorial

Useful Code Snippets: https://github.com/Sensorycc/YUXI-Mixed-Reality-Hardware-Toolkit/wiki/Useful-Code-Snippets

Week Two: Day 2-5

We continued the design development of our projects heading into the demo day on Friday.

Blurred lines AR CR: Watch out for your snacks

https://github.com/Sensorycc/YUXI-Mixed-Reality-Hardware-Toolkit/wiki/Blurred-lines-AR-CR:-Watch-out-for-your-snacks

CIID IDP 2019 - Designing Immersive Experiences