Introduction:

A typical desktop computer consists of various parts, one that is user accessible is the Tower or also called a Cabinet, which houses the motherboard and associated components along with various drives and power supply. However, in order to troubleshoot or swap or upgrade any component, one needs to assume that the user has a sound knowledge of various interfaces and connectors involved in communication between parts including how and where they belong, which is not true in most cases.

A similar case may also be encountered in large server farms where there are several numbers of servers stacked and troubleshooting the right server with right part may be a difficult job. This project, with the help of Microsoft HoloLens and Computer Vision will augment a user in scenarios described previously by providing information in real-time on the go at their job site without having to deviate from their workflow. HoloLens will identify various components in the scene and load appropriate 3D models and data related to them, thus helping the user make informed decisions and shall also guide during part/component replacement procedure using 3D animations.

 

Design Document:

 The design of this project requires the following minimum components, a brief description about their capabilities is also discussed along with how we intend to use them:

• Microsoft HoloLens– A pair of mixed reality smart glasses developed and manufactured by Microsoft. It features an IMU, an energy-efficient depth camera with a 120°x 120°angle of view, a 2.4-megapixel photographic video camera which will be mainly used for this project. As HoloLens is a full-fledged computer mounted on our head, computer vision algorithms shall run on device and recognize various components and then display appropriate animations and information on its 2.3-megapixel widescreen stereoscopic head-mounted display. As of now, planned API to be used for image recognition is Microsoft Azure Computer Vision API, a set of APIs designed to provide developers with image processing and analysis. Based on information processed by the APIs, a scene designed using Unity will be displayed.
• A development computer– A Windows 10 device, with Visual Studio and Unity installed that supports development of Mixed Reality Apps.
• A sample cabinet – A sample cabinet that consists of varying components along with a few extra components that can be used to demonstrate a procedure.

• A sample image database– A sample set of varying cabinet configurations along with various components and their connectors, cables will be required to train Computer Vision algorithms in order to identify and display information accordingly. The larger the database, the more accurate results will be.

 

Necessary Devices and Assumption

Microsoft HoloLens, a Windows 10 device to develop and an assumption of having the basic intuitive sense to use Microsoft HoloLens, along with tools to open a computer cabinet and make appropriate amendments.

Platform(s)

Microsoft HoloLens running Windows Mixed Reality.

User Interface and Controller Mechanism

The user shall be interacting in real world and HoloLens shall detect various gestures based on what the user wants to achieve, for example, once the configuration of the cabinet is detected and HoloLens has augmented the scene with appropriate information about parts in the scene, user may then select a part or component that needs to be replaced or requires information about by tapping an AR based anchor next to that particular part in real world as seen through the display of HoloLens. This shall provide hopefully a plethora of options for the user to select from based on requirement. The user shall mainly interact through various gestures, that are detected by HoloLens.

Who’s Doing What?

Kushal Kusram shall be involved in development of required image recognition and communication mechanisms on Windows Mixed Reality for Azure Computer Vision to recognize, track 3D unity objects in AR and make inferences based on components present in the scene, shall provide with one single result that shall be provided to Unity.

Shweta Yadav will work with Unity and C# to process the recognized cabinet parts and add information to them such as 3D models and other relevant data in the scene.

Assessment:

 HoloLens shall allow user to interact with various components of cabinet in augmented reality and help recognize different parts in the cabinet such as processor, motherboard, different drives, etc. In case user needs information about some specific part, they may tap on that part to know more about it. Also, if some component is missing or needs to be replaced, the system will help recognizing it and provide instructions for replacing it easily. Considering this scenario, the main strength of this project is helping users navigate their way themselves in not just DIY projects for computer configurations, but this same concept may be further expanded to various setups such as airplane engines, car engines or any such setup where a hard copy of instruction manual may be mapped into augmented reality making it more interactive and useful.

The criteria to judge the success of this project will be the smooth navigation between different parts within specified boundaries, variabilities with distance of the user from cabinet and relevant information about each part.

Sketches

Please leave your feedback about our project idea.

Thank you.