Designing XR exhibition experience, creating augmented storytelling for Gen Z with embodied interaction.
Unity
Blender & Rhino
After Effects
Premiere Pro
Figma
3D Design
XR UX Interaction Design
Concept Development
Physical Prototyping
Yawen Yang
Yanqing Yang
XIaoyu Huang
OVERVIEW
Dimono, designed for CMNH Dinosaur in their Times exhibition, offers a multi-sensory experience where patrons embody their chosen dinosaur, partaking in an evolutionary narrative across eras. Through interactive projection, users engage in real-time conversations, exploring the Mesozoic Era with mixed reality devices. This immersive experience bridges the gap between Gen Z visitors and museum exhibition stories, allowing patrons to appreciate authentic dinosaurs and spatially connect with narratives. Afterward, patrons can review their journey through phone-based AR, transforming passive learning into a personalized exploration of their own lives.
OBJECTIVES
How can we engage and educate Generation Z visitors, who have short attention spans and high digital literacy, in a way that connects them to exhibition narratives?
OVERVIEW
Dimono, designed for CMNH Dinosaur in their Times exhibition, offers a multi-sensory experience where patrons embody their chosen dinosaur, partaking in an evolutionary narrative across eras. Through interactive projection, users engage in real-time conversations, exploring the Mesozoic Era with mixed reality devices. This immersive experience bridges the gap between Gen Z visitors and museum exhibition stories, allowing patrons to appreciate authentic dinosaurs and spatially connect with narratives. Afterward, patrons can review their journey through phone-based AR, transforming passive learning into a personalized exploration of their own lives.
OBJECTIVES
How can we engage and educate Generation Z visitors, who have short attention spans and high digital literacy, in a way that connects them to exhibition narratives?
We have designed for an experience that prioritizes task optimization, customer-helper synergy, and immediate information access. Sammy epitomizes an intelligent agent, facilitating multitasking through hands-free dialogue for movers.
Scenario 1
Book the right service effortlessly
Tedious booking steps
Scenario 2
Make alteration easily
Repetitive steps to rebook
Scenario 3
Smooth connection between customer & third-party helpers
Inefficient communication & collaboration
Scenario 1
Time management for organized
multi-tasking
Stressful multi-tasking
Scenario 2
Make driving safe
Unfamiliar with vehicle operation
Scenario 3
Accompany driving
Stressful driving experience
The roles and value of Sammy is designed as personable for customized needs, reliable for supporting possible chaotic multi-tasking scenarios, and energized for providing mental support.
Different motion states for Sammy are designed communicate the different states of Sammy to an audience and to visualize responses to a user's input.
Dimono has two parts - a "participatory museum experience" to transform distant historical facts into engaging and relatable narratives for Gen Z, and "path collector" as a tangible reminder of their journey to encourage reflection and revisit to reinforce the delivery of museum's message to their audience.
01 Embodiment
The visitors can choose a dinosaur active in Triassic period they are most resonated to and mimic their posture to start their journey. We let the visitors to become dinosaur to know about dinosaur.
02 Onboarding
The visitors receive their headsets and path collector, going through onboarding to understand the basic interaction.
03 Interact with Paintings
Approaching and lingering at a painting for over 2.5 seconds activates an informative display. Information appears in the head-up display.
04 Transition
When entering new period of time, visitors will be stopped by virtual boundary. It will introduce how the previous period end and what dinosaur they evolved into to acknowledge their progress.
05 Interact with Fossils
When interacting with the fossils, visitors can appreciate the authentic dinosaur and the information and stories spatially together by using guided hand gestures. This is the fundamental interaction for all the fossils in the exhibition.
06 Path Collector
Each fossil display is paired with an incubator. Visitors use the collector to trigger the interaction and collect unique features of their dinosaur fellows. This creates a personalized path through the exhibit.
07 Talk with Iconic Dinosaurs
In addition to the basic interaction, the iconic dinosaurs like t-rex and Dippy will be brought to life and talk to the visitor proactively. Their mood depends on the relationship between them and the dinosaur the visitor is embodying. The storytelling is designed to inspire the resilience of life with museum’s core value.
08 Talk with Iconic Dinosaurs
when visitors leave the exhibit, they can retrieve their experience by scanning unique marks on their collector.These marks represent the diverse features of the dinosaurs they encountered, creating a lasting memory of their journey.
We used laser cutter and 3D printer to create the physical path collector and incubator. We also used Unity and Meta Quest 3 to create workable MR prototypes for testing and iterating.
The new multi-control system streamlines complex procedures, enabling faster and tailored operations that significantly boost clinical efficiency.
Old Injectors
Inconsistent settings with analog and digital controls across units
Our New Injector
Standardize setting process across touchscreens for ease of use
Old Injectors
Fully rely on control room to remotely set up protocols
Our New Injector
Tailored access permissions to support Injector-led workflow possibility
Old Injectors
Transfer patient data manually, causing delays and errors.
Our New Injector
Synchronize patient data across all displays for accessibility
High standard visual design boosted brand recognition, establishing our client as a leader in medical device aesthetics and user engagement.
The comprehensive design system ensures visual consistency and strengthens brand identity across all control units to facilitate scalability and future enhancements.
Attention to ID + UIUX enhances the product’s aesthetic appeal and functionality, leading to the successful product launch in 2024.
The system utilizes digital simulation to enhance comprehension of the injection process, helping to reduce errors and increase confidence.
The graphics provide a clear representation of the physical operational structure of the injector, allowing users quickly understand internal mechanisms.
The animations demonstrate the fluid dynamics and injection progress, enhancing the comprehension of the injection procedure.
Our client, CMNH, is committed to deepening the understanding of our natural world, to foster a sense of collective responsibility for our future. We’re focusing on their Dinosaur in Their Times exhibition, known for its famous holotypes specimens & reconstruction of the dinosaur habitats.
Our target audience, Gen Z visitors, whom the museum is currently struggling to retain. They were observed to often just pass through the exhibit and marvel at the wonders. We also conducted one-on-one interviews to understand their needs and values.
Gen Z visitors are characterized by shorter attention span and high digital literacy supported.
It may be more challenging for them to stay focused on traditional, less interactive museum, potentially impacting their ability to retain information presented in these formats.
The Gen Z are actually very interested in dinosaurs. They grew up with Jurassic parks. The dinosaur stories often impress them a lot once explained or revealed.
However, the rich stories within the exhibition are not reaching Gen Z visitors. The static displays don’t quite capture their fleeting interests, leading them miss the chance for deeper engagement in the exhibition narratives.
We discovered that the designed route is chronological, the earliest dinosaurs are shown first and follow the path of evolution. However, the curations of the spatial narratives of evolution are often neglected due to the scattered and overwhelming information.
By incorporating XR interactions and augmented reality, we aim to spark curiosity and enhance learning through immersive storytelling. Our goal is to convert passing interests into sustained engagement ultimately envisioning CMNH as an interactive, participatory museum.
Our client, CMNH, is committed to deepening the understanding of our natural world, to foster a sense of collective responsibility for our future. We’re focusing on their Dinosaur in Their Times exhibition, known for its famous holotypes specimens & reconstruction of the dinosaur habitats.
Our target audience, Gen Z visitors, whom the museum is currently struggling to retain. They were observed to often just pass through the exhibit and marvel at the wonders. We also conducted one-on-one interviews to understand their needs and values.
Gen Z visitors are characterized by shorter attention span and high digital literacy supported.
It may be more challenging for them to stay focused on traditional, less interactive museum, potentially impacting their ability to retain information presented in these formats.
The Gen Z are actually very interested in dinosaurs. They grew up with Jurassic parks. The dinosaur stories often impress them a lot once explained or revealed.
However, the rich stories within the exhibition are not reaching Gen Z visitors. The static displays don’t quite capture their fleeting interests, leading them miss the chance for deeper engagement in the exhibition narratives.
We discovered that the designed route is chronological, the earliest dinosaurs are shown first and follow the path of evolution. However, the curations of the spatial narratives of evolution are often neglected due to the scattered and overwhelming information.
By incorporating XR interactions and augmented reality, we aim to spark curiosity and enhance learning through immersive storytelling. Our goal is to convert passing interests into sustained engagement ultimately envisioning CMNH as an interactive, participatory museum.
Is moving a headache for you? We feel overwhelmed by the planning process. We struggle with loading and packing. We have trouble with moving logistics and coordination. What's more, it’s something so many people are doing all the time.
33%
of people in their 20s move in a given year
17-20 Million
people may move this summer
11
average number of times a person moves in a lifetime
1 out of 5
people move every year
U-Haul is one of the biggest transportation and storage solution company in North America. Their movers, mostly do-it-yourselfers are those embarking on this moving journey, who is bound to encounter moments of stress.
75%
of movers are “do-it-yourselfers”
costs 50% less
than full service moving solutions
127,000
number of trailers in U-Haul’s fleet
176,000
number of trucks in U-Haul’s fleet
We conducted primary research on U-Haul's exiting service. Their service is comprehensive but overwhelming. Therefore, we did an interface audit on their U-Haul website & app and ecosystem analysis to learn and find insights into the current user flow. We discovered that:
The navigation is rather overwhelming. We observed chaotic information architecture, wordy & confusing description and repetitive but hard-to-find access towards instructions.
U-Haul's service is very comprehensive. However, it also makes the customers lost in the searching. What's more, the connection between customers with adding-on service like Moving Helper is very loose and unguided.
Through 8 + one-on-one user interviews and online research, we found common themes in grievances and complaints.
Qualified Interviewees
21+ Years Old Do-It-Yourself Movers
"I'm afraid of driving a big truck. There is no rear mirror!"
"We were all not familiar with trucks and the interior and accidentally broke a handle."
Anxiety about driving trucks & misuse of equipment
Design for anxiety relief and instruction
"None of the records reflect the movers I ordered, no emails or text nor the website. Does this mean they are not booked and I need to book someone else?"
Challenges in accessing information & alternation
Design for accessible information in real-time
"The way U-haul helps me choose the truck isn't effective. The customer service makes me wait for hours."
"I had to cancel my reservation and rebook just to change the moving date."
Challenges in choosing the right service
Design for an agent could assist in choosing and altering order without redundant steps
"I tried to plan everything ahead but the moving day is still very stressful. Things always go wrong."
Reveal mental stress in multi-tasking
Design for schedule alerts and reminders for users
CUI can improve user experience, streamlined interactions, accessibility, and task automation for enhanced efficiency.
Speculating how CUI assistant would improve the situations
To address our research question in our design, we followed these four design principles:
This system blends various technologies such as digital projection mapping, Mixed Reality (MR) interactions, and natural language processing.
Digital Projections Mapping
XR Interactions
Nature Language Processing(NLP)
To seamlessly guide our visitors, we created an integrated visual system.
01 Information Panel
Visitors initiate interaction with gaze control which will trigger the info to appear within 2.5 seconds.
Eye Control
To interact with the fossil, hover over to reveal the info panel, pinch for more information, and grab to examine fossils closely. We designed the hierarchy to avoid info overload.
Hover
Pinch
Grab
02 Navigation Bar
Navigation bar has three distinct stages. We designed it appear as eggs to relate to the ‘life’ topic. Pat to switch among three stages—default, unselected, and selected—to activate information panels and secondary selections.
Pat
Pinch
03 Conversational UI
Distinct dialogue bubbles for iconic dinosaurs and visitors for visual clarity. And pinch the hint bubbles for suggested conversation starter.
Speak to Text
Pinch
When design the sound space in MR experience, we used the following principles:
Bring the users back to the dinosaurs’ time and environment with 3D spatial audio; Aural feedback to user’s interactions; Enhance the emotional and non-intrusive narrative events; Maintain immersion and consistency across platform.
In a market dominated by CT injectors with control room displays and limited on-machine controls, our challenge was to innovate a user experience with advanced on-machine touchscreens to meet and exceed current clinical procedures.
Even with few direct examples, market research showed growth potential for syringeless injectors. It also showed the need for intuitive on-machine controls to streamline workflows for better user interaction and efficiency.
Insight 1: Look into the future trend
We should focus on design UX for the new syringeless injectors, which are poised for growth due to their efficiency, safety, and waste reduction.
Insight 2: User habits matter
Preserving conventions, such as color coding (blue for saline, green for contrast medium), enhances usability and eases transitions.
Insight 3: Explore workflow integration
Recent injector systems allow customized patient protocols and usage tracking to improve operational efficiency. Visibility on the machine can further enhance accessibility.
Insight 4: Screen-Driven Clarity
On-machine touch screens should be designed to offer intuitive controls and simulation capabilities, simplifying training and reduce dependency on remote systems.
I led discussions across teams to improve the input information architecture and wireframes by asking straightforward and critical questions about feedback and requests. We also utilized paper prototypes for quick, remote cross-team evaluations, enabling efficient design enhancements.
Collaboration Adjustment
Fast Prototyping
Refined Information Architecture & Wireframes
We decided to focus the design to create an experience that prioritizes task optimization, customer-helper synergy, and immediate information access, ensuring a seamless and stress-free experience for all users.
We identify our core customer as the do-it-yourselfers who feel anxious and overwhelmed by the moving process. The Moving Helpers as the U-Haul contractors are our secondary users, providing professional accessibility service to those in need and getting extra income.
Based on our immersive research, we identified 3 main roles that Sammy acts as, which are guide, reminder, and cheerleader. The percentage of the 3 roles and personalities that Sammy exhibits when interacting with customers and moving helpers varies depending on the current scenario.
More detailed storyboard & user journey map are created to refine the narrative, incorporating feedback, and enhancing visual storytelling to align better with the design focus.
Overall
Detailed - Reservation
Detailed - Contacting Moving Helper
As we learnt that incorporating digital simulations into the CT injector enhances the intuitiveness of the injection process. The challenge was to achieve this without overloading system resources, ensuring high reliability crucial in a medical context.
I collaborated closely with clients and engineers to determine the optimal amount of essential animated simulation through technical testing and iterative feedback.
After determining the key animated simulations, I started to explore desirable ways to both communicate impactful message and efficiently utilize resources, especially for injection and fluid simulation.
✅Pros
Memory-efficient, simple & best for low-memory systems
❎Cons
Less dynamic
✅Pros
Flexible, interactive & great for dynamic content
❎Cons
Requires careful memory management
✅Pros
Simple & requires minimal coding effort
❎Cons
Low resolution, memory-heavy & less flexible or interactive for real-time updates
We tested UI components using paper prototypes to fast iterate the positions and angles of UI components. And then we built a digital demo with Unity. The design of path collector and incubator keep iterating when we updated spatial UI component.
Paper Prototypes
The rapid prototyping helped us test the spatial postion and angle of the UI components.
Unity Prototypes
Unity demo helped us test the scale and position of Egg UI in spatial space; Test the “pat” and “distance grab” gesture usability; Test the usability of interaction with skeleton and user flow.
Physical Prototypes
The path collector and the incubator are the key to enter the AR interactions. They should not only act as a working prop but also an attractive piece of designed product.
The arrangement of UX components aims to minimize visual disruption and easy and ergonomic reaching.
‘Sammy’ is the intelligent conversational assistant we design to integrate in the U-Haul app. The mechanism and system of Sammy is designed according to the design focus we generated based on our research.
U-Haul introduced it's first version of its "Sammy U" character in 1953, named after the company's founder.
Style Identification
The visual design is informed by U-Haul’s brand. It takes cues from U-Haul’s boxy, yet rounded equipment and incorporates core values like reliability and stability into its tone.
Final Form
Our final visual design began with a more abstract representation of a box and landed at a more literal design that felt more on-brand with U-Haul. The idea of an active opened box speaks to the core user experience for U-Haul service.
Motion Design
Curated visual cues and motion states, fostering more natural and engaging tech interactions. Sammy's varied motions add personality and clarify messages to users.
Our emphasis was on ensuring optimal reading legibility specifically tailored for mobile users, maintaining alignment with U-Haul's brand identity throughout the design system.
We developed a UI design system that incorporates U-Haul's branding elements, encompassing a cohesive color palette, information cards, icons, and buttons.
The app's design features two distinct user paths: one for users with upcoming orders and one for starting brand new orders.
When a user starts a conversation, a compact pop-up screen captures and exhibits the inquiry, preventing abrupt transitions to new pages. This pop-up then expands to a mid-height page, showcasing all dialogues, arranged from the bottom upward.
Here are the final interfaces for the redesigned U-Haul app. These screens cover the customers' pre-moving experience, Moving Helper collaboration & moving day (driving) experience.
Designing a unique visual interface involved navigating the client’s cautious decision-making process. The client often leaned towards familiar, traditional styles, seeking safety in what was known. Our challenge was to find the sweet spot to elevate the product and maintain competitiveness.
Conducted iterative design adjustments based on client feedback, maintaining and learning from high visual standards while introducing innovative elements.
The development team expanded the interface using initial components, but inconsistencies in screens revealed the need for standardized guidelines. I proposed a design system across all device to ensure consistency and scalability in future designs.
Engineering Team Inquiry
Design System Templates
To transform the natural history museum journey for Gen Z visitors, empowering them with knowledge that bridges the past and present, inspiring stewardship for our collective future.
As one of the biggest projects I was in charge of, the CT injector project strengthened both my technical and interpersonal skills, making me a strong team player.
Working in medical design requires close cooperation with industrial design, human factors, and engineering teams. Maintaining tight collaboration across disciplines and open for iteration ensures that the design meets both high standard market and functional regulatory requirements.
Educating clients requires skillful communication, as they may not always realize that their requests might not be feasible. I discovered that using their language and visualizing ideas effectively can bridge understanding gaps, leading to better decision-making and outcomes.
Leading this project gave me the opportunity to work closely with developers, project managers, and top decision-makers. I learned that asking questions clearly stating requests and expectations to clarify uncertainties are essential for team efficiency and ensures everyone is aligned.
My piles of paper prototypes
Our new design makes U-Haul’s services not only more efficient, but also more inclusive and accessible, which in turn supports more diverse users and aligns with the principle of social responsibility. An expanded customer base also supports the growth of the shared economy. What’s more, the feasibility of Sammy is high, leveraging existing new technology like generative AI, and GPT-4.
Child PPE customization App, which bridges the function & the heritage, forming a new way to both care about the children & our tradition.
An anti-bullying guardian system utilizes the concept of non-directive play therapy to evoke students' empathy & self-worth through workshops & play.
