Nina Serpiello, PhD

Ergonomic Design

Teague & Boeing

Research in VR for aircraft architecture & interiors

Role

Research & Strategy Manager

Year

2016–2018

Impact

Prototyped tools in VR for spatial accuracy relative to human perception

Business Challenge: create research tools in VR to design for perceptions and experiences in airplanes.

Result: worked with VR developers to prototype tools to test the accuracy of spatial perceptions in a human-scale mockup. Perception was close to perfect; tools need iteration.

While at Teague as the Research & Strategy Manager, I initiated exploration studies to use virtual reality with accurate-scale physical mockups to capture reliable user experience data. This information would be useful as design input and verification of aircraft architecture and interior design processes, as well as a compelling, potential sales tool for Boeing to airline clients.

Teague had two expert disciplines: building mockups for architecture and lighting design, and digital visualization for interior design. My research team worked with the VR developers to prototype a mixed-technology system to increase the accuracy of visual space perception. We pushed further to create an avatar that tracked a person’s actual movements within a physical, interactive mockup.

We conducted two prototype studies to prove our hypotheses and examine the human factors data that could be extracted from the 3D models and movements. The technique had a promising future to create reliable user experience data in airplane development for Boeing’s business, and service design for Teague’s business.

“The VR avatar is accurately capturing the moment I make contact with a surface. I felt confident moving through space with the headset on.” — Research participant
3D render of an airplane cabin interior with a tracked avatar, used for VR spatial perception research
Prototyping a precision measuring setup in VR.
First-person VR view of a hand reaching toward an airplane cabin seat
Tracking accuracy of a user touch point in VR.
Overhead view of a dark physical aircraft cabin mockup with reflective motion-tracking markers
Prototype of a testing apparatus for precision measuring.

Detailed Case Study

Two exploratory studies asked whether virtual reality could give Teague and Boeing reliable human factors data: first a field trip to an immersive VR environment, then a hands-on study combining VR with a physical cabin mockup (VR+MU).

Study 1: Immersive VR at Vulcan’s Holodeck

Research questions

  1. Useful technology? Is an immersive VR environment useful for our projects in justifiable ways?
  2. Partnership options? What might be the options to partner with Vulcan to use their Holodeck?
  3. Inspiration for design? What could make VR experiences more compelling for clients?

Under a mutual NDA, our team visited Vulcan Innovation Labs for a field trip and technical conversation. It inspired us and informed them. Their prototype was an immersive bubble big enough for six people to move around in without headsets. Cameras hung at the apex, the floor was suspended above vibration speakers, and 360° content was mapped to the fabric seams or kept dark to hide them.

Inside a white dome-shaped immersive VR bubble, a woman stands on the suspended floor Two men stand inside a 360-degree projection of a turquoise sea and cliffs A glowing spherical tent lit from inside at dusk among trees
Vulcan’s prototype Holodeck: 360° content with no headsets.

Takeaways

  • 360° environments may help with research in autonomous driving. Without headsets, people can talk naturally.
  • If a second round of tests piqued interest, we could negotiate time in the Holodeck for a pilot.
  • Sound vibration used to simulate motion had a fascinating effect.

Study 2: Human factors in VR + a physical mockup

The second study paired a VR model with an accurate-scale physical mockup of a three-seat row, overhead bins and an aisle, and tracked participants as they moved through both.

Research questions

  1. Visual scale: Is the visual perception of scale between VR and physical form equivalent?
  2. Spatial perception: What are the sensory aspects of evaluating a combined VR+MU model?
  3. Research protocol: How will we get reliable, measurable results and good experiences in studies?

Participants & protocol

We recruited people across a range of human scale, and collaborated on the protocol with VR developers, designers and the model shop. Participants first saw the physical mockup and learned how to hold the tracking wand.

Hand holding a motion-tracking wand with reflective marker balls beside white mockup seats Researcher wearing a full motion-capture suit and VR headset beside a minimal foam mockup
Learning to hold the tracking wand beside the physical seats, and a minimal foam mockup for motion capture and measurements.

The 19-step protocol

  1. Measure arm
  2. Instruct on wand
  3. Adjust headset
  4. Orient to target
  5. Estimate space
  6. Navigate aisle
  7. Touch VR target
  8. Navigate aisle
  9. Touch MU target
  10. Get hands
  11. Sit in seat
  12. Reflect
  13. Estimate space
  14. Move to bin
  15. Operate bin
  16. Reflect
  17. Remove headset
  18. Estimate space
  19. Reflect

Touching targets on the physical seats gave tactile feedback that oriented people to VR. They then walked the virtual aisle and touched targets on the bins in VR only, with no tactility, before touching targets on the physical model.

Participant in a VR headset reaching up toward an overhead bin in the physical mockup Participant in a VR headset pressing both hands against the overhead bin
Participant in a VR headset operating the physical overhead bin Participant in a VR headset sitting in a mockup seat holding controllers

Prototypes & technical set up

In pursuit of measurable research outputs, the VR team built tools quickly to estimate distances and align precisely. VR targets were sited to placement dots on the mockup. The wand and pointer were accurate to 1.5″, but the hand position the wand required forced an awkward movement.

Tracking wand pointed at a blue tape target on the mockup The same seat row seen in VR, with the rendered wand in the foreground
Aligning a physical target with its virtual twin.

Motion tracking experiments compared hand renderings. A skeleton hand helped participants understand where they were in space. Human hands felt more natural and magical, but weren’t accurate to each person’s physiology.

Rendered human hands reaching toward a white seatback in VR Rendered hands raised in an empty white VR space
Rendered hands in VR.
Motion-capture cameras on stands around the seat row against a black backdrop Reflective tracking markers mounted on the physical overhead bin
Motion-capture cameras and markers revealed the physical mockup in VR.

Results

Visual scale

  • People said VR-only scale was equal, but the combined model showed perception of scale was 0.5 to 1.5 inches off in each 3D axis.
  • VR appeared slightly farther away. When people hit a surface or bumped an aisle seat, they expressed surprise.
Yellow and red dot targets on blue tape marking touch points on the mockup A labeled sticky note and red target on the mockup wall
Placement dots recorded where people expected surfaces to be.

Spaciousness

  • When standing, looking ahead or reaching above, people said the aircraft seemed spacious.
  • When seated, people felt cramped. They could feel the armrest, but couldn’t see or feel where their knees were relative to the seat ahead.

Other senses

  • People could navigate a VR aisle, but felt uncertain when touching real objects, which may leave them feeling unbalanced.
  • People moved more naturally with tactile feedback, and needed their hands near objects to judge distance. Inaccuracy seemed minimized when they had a visual reference to self.

This matches published research: placing the hands near an object can improve visual sensitivity and precision (Schendel & Robertson, 2004; Vishton et al., 2007).

Accurate data

We need better tools to compare physical measurements with VR data.

Researchers measure a participant’s arm with a tape measure while holding VR controllers
Physical measurements taken to check against VR data.

Challenges

  • Accurate set-up and calibration require precision and vigilance.
  • VR tools had distracting glitches, such as jumping and disappearing, and inaccuracies like hands passing through walls or twisting fingers, which disturbed some participants.
  • High resolution implied accuracy, so it became frustrating when precise movements didn’t align to targets.
  • With peripheral vision and sound compromised, and no color, shadows or other people in the model, cognitive load was high.
  • Errors can accumulate over tools, time and individuals. Improvements will be a continual process of development and testing.
VR view of a rendered hand distorted as it reaches toward an overhead bin
A tracking glitch distorts the rendered hand near the bin.

Lessons learned

  • Empty, white models make distance difficult to judge.
  • Verbal facilitation may add to cognitive load. Tactile guidance is immediate.
  • Repeatedly switching between VR-only and VR+MU increases cognitive load, causing mistrustful actions and worry while people mentally recalibrate.

Design guidance for future projects

  • Tool set-up, calibration and research protocols: 2 weeks, with a team of 2 or 3 using familiar tools.
  • Recruiting, research with VR+MU and reporting: 2 weeks, with a team of 2 or 3 and fewer than 20 participants.