This post shall remain the latest until for a long time, it outlines the latest development done on CompuRay so far. We call this version as V3.

Project Objective:

Just to remind, our project objective in brief, is to augment the experience involved during troubleshooting or upgrading of a traditional desktop computer. This project makes use of Microsoft HoloLens, the AR device recognizes various components of the computer and the results are spatial mapped to the real-world through HoloLens. The HoloLens was trained to recognize the following components of a computer cabinet: CPU, Drive Bay, SMPS, GPU, RAM and PCI Slots. The following equipment and software were used: Microsoft HoloLens, Unity and Microsoft HoloToolKit. The interface provided for a user to interact with HoloLens is through voice commands, except during the phase of analysis. The advantage of voice commands over traditional HoloLens recognizable gestures includes handsfree control along with a more natural UI and almost a zero-learning curve in order to operate the device.

Success Metric:

1. Object Detection: The project successfully identifies different components within a specified boundary and the tags are stably anchored in real world. The predictions seen on Azure Dashboard are accurate.
2. Natural UI: The project provides the user with a very simple UI that is through voice commands which has almost zero learning curve along with an only one gesture-based interaction step. The advantage of having voice-based commands allows the user to be totally involved in the work, with hands being free for other tasks.
3. Spatial Mapping:The project successfully manager to anchor tags to right objects in almost all cases, a few exception cases might exist based on the lighting conditions.

Future Work:

This project makes use of Universal Windows Platform, provided by Microsoft. During various tests that were conducted, HoloLens at times was unable to detect the right depth and thus tags were anchored slightly offset from the actual component’s position. This provides an opportunity to dive deep into depth sensing implementation in order to deconstruct and improve the depth sensing of HoloLens for it to scan better inside a desktop computer. As this project is using the default SpatialMapping script provided by Microsoft, modifications were required in order to have gesture-based cursor interaction with a hologram menu, however, it was hard to make modifications to it and any modification led to breaking of the entire app, if project had some more time that would allow us to make right modifications allowing the project to have an even more user-friendly UI with multiple options to customize the app such as to reset the app, in order to reset the scene and clear all the tags anchored previously. As we faced difficulty in adding 3D objects to the scene due SpatialMapping script configuration, a modification to it would also help us add interactable objects to the scene based on the tags found.

Screenshots:

Video Demo: CompuRay Demo

Please feel free to leave a feedback, thank you!

Introduction:

A typical desktop computer consists of various parts, one that is user accessible is the Tower or also called a Cabinet, which houses the motherboard and associated components along with various drives and power supply. However, in order to troubleshoot or swap or upgrade any component, one needs to assume that the user has a sound knowledge of various interfaces and connectors involved in communication between parts including how and where they belong, which is not true in most cases.

A similar case may also be encountered in large server farms where there are several numbers of servers stacked and troubleshooting the right server with right part may be a difficult job. This project, with the help of Microsoft HoloLens and Computer Vision will augment a user in scenarios described previously by providing information in real-time on the go at their job site without having to deviate from their workflow. HoloLens will identify various components in the scene and load appropriate 3D models and data related to them, thus helping the user make informed decisions and shall also guide during part/component replacement procedure using 3D animations.

 

Design Document:

 The design of this project requires the following minimum components, a brief description about their capabilities is also discussed along with how we intend to use them:

• Microsoft HoloLens– A pair of mixed reality smart glasses developed and manufactured by Microsoft. It features an IMU, an energy-efficient depth camera with a 120°x 120°angle of view, a 2.4-megapixel photographic video camera which will be mainly used for this project. As HoloLens is a full-fledged computer mounted on our head, computer vision algorithms shall run on device and recognize various components and then display appropriate animations and information on its 2.3-megapixel widescreen stereoscopic head-mounted display. As of now, planned API to be used for image recognition is Microsoft Azure Computer Vision API, a set of APIs designed to provide developers with image processing and analysis. Based on information processed by the APIs, a scene designed using Unity will be displayed.
• A development computer– A Windows 10 device, with Visual Studio and Unity installed that supports development of Mixed Reality Apps.
• A sample cabinet – A sample cabinet that consists of varying components along with a few extra components that can be used to demonstrate a procedure.

• A sample image database– A sample set of varying cabinet configurations along with various components and their connectors, cables will be required to train Computer Vision algorithms in order to identify and display information accordingly. The larger the database, the more accurate results will be.

 

Necessary Devices and Assumption

Microsoft HoloLens, a Windows 10 device to develop and an assumption of having the basic intuitive sense to use Microsoft HoloLens, along with tools to open a computer cabinet and make appropriate amendments.

Platform(s)

Microsoft HoloLens running Windows Mixed Reality.

User Interface and Controller Mechanism

The user shall be interacting in real world and HoloLens shall detect various gestures based on what the user wants to achieve, for example, once the configuration of the cabinet is detected and HoloLens has augmented the scene with appropriate information about parts in the scene, user may then select a part or component that needs to be replaced or requires information about by tapping an AR based anchor next to that particular part in real world as seen through the display of HoloLens. This shall provide hopefully a plethora of options for the user to select from based on requirement. The user shall mainly interact through various gestures, that are detected by HoloLens.

Who’s Doing What?

Kushal Kusram shall be involved in development of required image recognition and communication mechanisms on Windows Mixed Reality for Azure Computer Vision to recognize, track 3D unity objects in AR and make inferences based on components present in the scene, shall provide with one single result that shall be provided to Unity.

Shweta Yadav will work with Unity and C# to process the recognized cabinet parts and add information to them such as 3D models and other relevant data in the scene.

Assessment:

 HoloLens shall allow user to interact with various components of cabinet in augmented reality and help recognize different parts in the cabinet such as processor, motherboard, different drives, etc. In case user needs information about some specific part, they may tap on that part to know more about it. Also, if some component is missing or needs to be replaced, the system will help recognizing it and provide instructions for replacing it easily. Considering this scenario, the main strength of this project is helping users navigate their way themselves in not just DIY projects for computer configurations, but this same concept may be further expanded to various setups such as airplane engines, car engines or any such setup where a hard copy of instruction manual may be mapped into augmented reality making it more interactive and useful.

The criteria to judge the success of this project will be the smooth navigation between different parts within specified boundaries, variabilities with distance of the user from cabinet and relevant information about each part.

Sketches

Please leave your feedback about our project idea.

Thank you.