We did it – a new HoloLens application!
Zühlke office in Belgrade recently implemented the first version of a Shared Whiteboard application. This application was developed specifically for Microsoft HoloLens, as part of the Augmented and Mixed Reality focus group in Belgrade. The main purpose of the application is to give two users at different locations the impression of having the same, shared whiteboard in front of them while actually every user has their own whiteboard.
Why did we do it?
The Zühlke office in Belgrade is by definition a “distributed development center”. This means that most of our projects are partially built in Belgrade and partially in some other Zühlke location. Working at Zühlke Belgrade for two years already, I was also part of a project like that. What I considered the biggest demerit of distributed development is the fact that there is no way to use a whiteboard in a meeting, because it is not possible to write and draw on the same whiteboard at the same time. Hence, my idea was to build an application that would overcome this problem and make distributed development more pleasant than it is now, so our focus group started working on this project about two months ago.
How did we do it?
At first, we could not make solid decisions regarding the architecture of the application, so we just identified smaller parts that needed to be implemented.
First, we started working on the application that would run on HoloLens. We used Unity with HoloToolkit for this purpose.
The first task was to enable the user to place a virtual whiteboard onto the same position as the actual whiteboard. The initial idea was to use Vuforia, which would recognize some marker that we placed on a whiteboard. However, we had problems with aligning whiteboards with this approach. After that, we tried using the Hand Draggable component from HoloToolkit. Yet again, the same problem appeared, as dragging with your hand is also not precise enough. Therefore, we decided to use TapToPlace from HoloToolkit, which can place the virtual whiteboard right on the actual whiteboard and allow the user to fine-tune the placement.
After we had managed to place the whiteboard properly, we introduced voice commands for sending images to remote participants and receiving images from them. This went smoothly, as HoloToolkit has great components for adding voice commands, and the Unity engine has great support for creating http requests.
The initial two parts mentioned above were the core of the HoloLens application, so we decided to start working on a server application that would be used as a mediator between the two participants, as shown in the image below. Azure came to our mind first and it showed to be good enough for this purpose.
This server app is a simple Web API application through which we provided endpoints for uploading the image for the other participants and getting the last image from the other participants.
Whenever an image gets uploaded, a whiteboard extraction algorithm is triggered and the image containing only the extracted whiteboard is saved to the file system. This algorithm is actually the core of the whole Shared Whiteboard application and it was the most complicated part to implement. Since it is an image-processing algorithm, our first idea was to use OpenCV, a currently very popular C++ library for image processing. However, we found it difficult to include it in the Azure Web App, so we switched to a not so popular .NET library called AForge.NET, which proved to be even better than OpenCV. The algorithm finds all rectangles in the image and then for each rectangle, it checks whether it matches any of the four Zühlke logos that we put in the corners of the actual whiteboard. When it finds all four logos, it rotates the image if needed and crops out the rectangle that now contains only the actual whiteboard. When the other participants requests the last image, they will get this image containing only the extracted whiteboard.
As soon as we had implemented this algorithm, we had a fully functioning application. Eventually, we added some functionalities for creating a session, which will be useful when scaling the application, and for resizing the virtual whiteboard, so that different sizes of actual whiteboards are supported.
The application we developed was quite a challenge. We had the opportunity to work in areas which are not that common in our regular projects and we had a lot of fun. Eventually, the application proved to be very successful. Soon, we will have the opportunity to test it on actual, commercial projects, where the application will show its true value.