Designing mixed reality to support field service technicians
Timeline
September 2020 - June 2021
Tools
Figma, Photoshop
Role
Ux designer
Tech Assist, a Mixed Reality experience concept, aims to provide self-serving assistance on-site by enhancing field service technicians' productivity when dealing with complex inspection workflow processes and minimize downtime when handling time-sensitive situations in reactive services scenarios.
Problem
Mixed Reality is impacting the way people do business and engage with customers. One of the main industries that is currently incorporating this technology at a fast rate is the Field Service Management Industry (the practice of managing, supporting and optimizing field technician workflows). A key component of this industry revolves around reactive services in time-sensitive situations for complex equipment and machinery such as fire protection systems. The providers of these services employ highly skilled and certified technicians that are familiar with strict regulatory protocols, guidelines, and checklist procedures under which they can provide their services. However, senior field service technicians have been retiring in great numbers and with the increase of the equipment complexity, new or less experienced technicians often find themselves needing assistance to resolve the issues they encounter.
How might we leverage Mixed Reality to support field service technicians’ on-site visits, and facilitate self-service assistance to successfully prepare and execute complex inspection processes?
Demographic
Research methods
We used a mixed methods approach to capture key insights from users that were a close match to our target demographic. The first step involved using Microsoft's internal data to understand the specific needs and pain points of the stakeholders. We then added the quantitative insights we gathered from conducting 11 semi-structured interviews using Microsoft's usability testing platform. For the second phase, we resorted to contact three local field service technicians and did snowball sampling to expand on our qualitative findings. On the third and final step, we identified areas of opportunity by thoroughly analyzing the Microsoft Dynamics 365 website to learn about their current suite of products and case studies that applied to the field service industry. Our combined research findings revealed key insights that defined our user's persona and guided design decisions throughout the development of the project.
Key research insights
1
The execution of an on-site inspection often deviates from the initial workflow inspection plan. Making adjustments once on-site is a persistent issue that can be tedious and time consuming.
2
Having to use several paper-based protocols and procedures while manually checking for anomalies, can make it difficult to accurately follow workflow steps, resulting in increased risk of human error and potentially costly downtime.
3
Capturing on-site information with the appropriate level of detail for each individual step, is challenging and ultimately complicates the task of effectively and efficiently identifying discrepancies.
Persona
User journey map
Design goals
1.
Facilitate self-serving support assistance during complex inspection workflow processes.
2.
Reduce cognitive load and minimize downtime through precise step by step instructions.
3.
Provide a precise, hands-free inspection workflow experience by enhancing productivity and minimizing human errors.
4.
Assist in the creation and management of asset content, making it easier to collect data, add annotations, and attach media files.
5.
Ensure traceability and monitoring of steps by recording insightful telemetry.
Design process
We kicked off our design process with a brainstorm session in collaboration with Microsoft, our sponsor, using Crazy 8's design technique. We then went through multiple prototype and test iterations that allowed us to define the best fitting user flows. The early iterations involved a simple draft prototype that we tested with Bodystorming sessions among personal contacts, professionals working on Mixed Reality, and local fire inspector technicians. Their feedback was used to improve the user flows and to sketch out a low-fi storyboard that underwent an extra testing iteration. The end result was a series of high-fi renders that show highlights of core interactions with our Mixed Reality concept.
Tools
Our input and output Mixed Reality features are based on Microsoft's MRTK toolkit's design system. A water-foam deluge fire suppression system, commonly installed in small aircraft hangar spaces, was selected as the model for user flows. Viking Corp provided us a 3D rendering of one, that we incorporated into our prototypes.
Our design proposal
Our Mixed Reality experience concept, Tech Assist, centers on using means of 3D holographic visualizations with life-size overlay features to effectively guide users through detailed step by step instructions during a complex workflow inspection process.
User flows and low fi storyboards
.png)
.png)
Design opportunities
Prior to beginning the inspection workflow process, Tech Assist will allow users to access, prepare, and manage guidelines and regulatory protocols in an efficient manner using web-based management features. It will also set the checkpoints that will be inspected by layering a 3D preloaded reference onto the real equipment.
Key user insights
One of the participants thought the 3D life-size overlay feature was a cool way to engage users since it made him feel like being in a video game.
​
Participants suggested using simple interactions like hand menus.
Design opportunities
During the workflow inspection process, users would be able to request various inputs using web-based portals (e.g. 3D model references, documents, pictures, audio, and video files), and attach supporting materials using augmented reality tools.
Key user insights
Participants were excited about the flexibility to use voice-command features as it allowed for simplistic, hand-free interactions. However, one key suggestion was adding more voice feedback to alert users of the success or status of a voice-command once issued.
Design opportunities
At the end of the workflow process, Tech Assist will present a detailed report that allows traceability and monitoring of the steps using automated cloud-connection services.
Key user insights
Participants liked the idea of automated reporting as this would take time off from fulfilling the report by hand or through mobile/table input.
​
One participant believed Tech Assist would be less practical to use in the long-run as technicians tend to rely more on personal experience and personal contacts to solve issues.
Prototype core interaction highlights: variance detection
We proposed a Mixed Reality experience concept that centers on using means of 3D holographic visualizations with life-size features to effectively guide users through detailed step by step instructions during a complex workflow inspection process.
​
Our design concept would allow users to request various inputs using web-based portals (e.g. 3D model references, documents, pictures, media files), and attach supporting materials using Augmented reality tools.
.png){kind=small}
The user "Air-taps" on first checkpoint. Tech assist displays a gem-like hologram that enlarges in size to guide users through each inspection checkpoint of the process. It automatically displays the 2D slates with every step of the inspection and specifications of each piece from every component, supporting the user in reducing cognitive load while enhancing productivity.
The user can invoke "Palm menu" to request video tutorials and/or other multimedia files filtered by date from previous inspections, facilitating self-service assistance and minimizing downtime during the inspection process.
The user can confirm with a "Voice command" the status of the inspected piece from every checkpoint, if it passes the inspection, the user will Voice command "Pass", if it fails, the user will command "Fail", Tech Assist will confirm with "Audio feedback" the status.
The user can select from an array of 3D holograms of different pieces using "Air-tap" or "Voice command". Tech Assist will layer a 3D model reference onto the physical space.
The user benefits from contrasting the 3D hologram against the physical piece, executing a precise hands-free inspection workflow and making it easier to identify discrepancies and lessen human error.
The user can select the "Annotations" button from the palm menu. Tech Assist will display different toolbar menus to make multimedia artifact annotations, like taking photos, videos or audio recordings. It also displays utility tools for hand annotations as if "writing in the air". These annotations can layer onto the physical piece, assisting to better create asset content and attach media artifacts.
Throughout the inspection workflow process, Tech Assist will automatically be reporting and recording telemetry on every step, ensuring a detailed traceability and monitoring of the overall workflow process.
 |  |  |  |
|---|---|---|---|
 |  |  |  |
 |  |  |  |
 |  |