[The following is an interview by Chinese VR media VRtuoluo about the tech and thoughts behind Noitom’s massive multiplayer VR experience]
It might not be difficult to create VR experience for 4 people at the same time, but how about for 100 people? What are the difficulties in creating a VR experience for 100 people? Professor Dai Ruoli from Noitom would like to elaborate on this topic.
On November 26th 2016, more than 100 Media guests and 4S store customers were invited to the new generation Tiguan import car release at Shenzhen OCT Creative Show Center. Given the theme of the release is a touch of science, VR experience became the spotlight.
With the assistance of onsite staffs, the guests put on their headsets, held up their interactive controllers and entered the virtual space at the same time to have 10 minute interaction with the hero of the release. The Tiguan 3D model was gradually disassembled according to the introduction and the guests can change its color setting or functions inside/outside of the car with their controllers. Apart from that, when the guests walked closer in the virtual space, they can actually see the other people’s image authentically reflected according to their real life position.
After the release, perhaps the guests have got to know the new car and VR experience. But what they don’t know is the world famous action capture company Noitom that brought them this VR experience.
The boldest thing ever
“This is possibly the boldest thing Noitom had ever tried.” Professor Dai Ruoli, CTO of Noitom, said so during the interview in his Beijing headquarter office.
It all started 4 months ago. “Volkswagen import department contacted us and hope to add VR presentation into the brand new import car Tiguan’s release. And in the past 4 months, the senior executives of our partner came to visit successively and experienced Project Alice, a solution which they all think very suitable for new car release and 4S store presentation.”
At the beginning of 2016, Noitom announced its VR solution Project Alice. After a year’s iteration, Project Alice is reaching maturity and the Volkswagen new car release is its latest successful implementation. Still we wonder what exactly is the bold thing that Professor Dai Ruoli is referring to.
It is not the first time that VR is utilized in car release. For a major release event, a mobile VR headset and an Android phone seems to be a nice combination now as they can bring VR experience at a manageably low cost. Formerly, the Gear VR presentation at Samsung release has proven to be a successful case; however, that is not exactly what Noitom want to do.
For Professor Dai Ruoli, VR experience has to be social related. “Social factor is a crucial aspect of VR interaction. Because human are social animal and we like to stay together; and it fits the characteristic of a release as well.” Therefore Professor Dai Ruoli and his colleagues made a bold plan: they want to have 100 people enter the virtual world at the same time and make sure they can feel each other’s existence as well.
Noitom is a company that provides large-space multi-person business solution, and they want to include Project Alice into the solution they provided to their crucial partner in automobile industry. Professor Dai Ruoli and his colleagues set up over 100 individual PCs and headsets at the event as well as their unique optical inertial mixed space action tracking system. They hope to add a touch of science for the experience, which will better fit the brand signature of the Volkswagen import car release.
However, locating 100 people at the same time is no easy task.
About two months ago, Noitom held a VR open event in Beijing Dashila’r. During that event, the “Beijing Workshop: Poem of light and shadow” Noitom presented can support 8 people simultaneously. However, locating 8 people is much more easier than locating 100. First of all, in order to make sure the system can differentiate the 100 individuals, they need to create a spatial tracing that can simultaneously detect 100 headsets; if counting the controllers, then the tracing capacity need to reach 200. This will be a huge challenge for the stability of the system, the calculation capacity of the computer and the design of identification marks
Furthermore, when the individual points to be trace reach a certain amount, it will naturally multiplied the data throughput of the system. “Imagine one server only have one client while the other have 100 client; the calculation efficiency and stability must be different and if any problem emerges, it might led to lagging VR experience and very possibly physical discomfort as well, which is the most fatal result.
In the end, Volkswagen Import Car held three release events in total with one of them hosting 105 people, all seats were taken, including those Noitom prepared just in case. Facing these picky guests, the consequence is unbearable if the presentation fails. “If in the middle of the presentation somebody stands up and says he is dizzy, then we are screwed.” Professor Dai Ruoli said with a smile.
Fortunately, nobody stands up and they succeed.
From optical rigid body to wheel rebuilding
In the office, Professor Dai Ruoli showed the reporter an equipment attached to the headset: it was made by 3D printer and looks like antlers. At the tip of each horn there is a small ball which Professor Dai Ruoli called as all-purpose rigid body. It is the key why the computer camera can recognize each individual in the pace.
As the branches these rigid bodies are located have special shapes, therefore the small balls in the space, aka the mark points, have different combination as well, which makes these rigid body unique in the space and can paired with each user’s headset and controllers. However, this is the first difficulties in the 100 people presentation as well: it is easy to produce one rigid body or two, but Noitom need to produce 200 unique and beautiful rigid bodies, and there are restrictions on size and shape as well. “You cannot ask the guests to wear the antlers right?”
“We spend 17 months to research how to recognize up to 8 people, and another 4 months to test the algorithm and grow that number from 8 to 100.” Professor Dai Ruoli said.
After they have 200 unique rigid bodies, the engineers in Noitom is ready to conquer the next challenge: how to transmit the data of 100 people without lagging. At the release, 100 people will simultaneously transmit two set of data via a limited network bandwidth to the server software for recognition and locating. If anything happens to this two set of data, many people might have to face a blank screen or experience severe lag, which might affect the experience or even cause physical discomfort. Noitom’s solution is quite ingenious though: they set a priority level for every guests’ two equipment.
“What do people care most about in an experience?” Professor Dai Ruoli asked the reporter in his office. “The one thing that affects an experience most is the lag in head movement, normally if the lag is longer than 20ms, it will lead to motion sickness. While people are less sensitive to the lag in their hand movement or the other people’s movement, even if the hand movement is 30ms behind, it still won’t raise any attention. Based on this, we set our priority.”
When every guest are in the position to experience, the data stream begin to arrange and group according to the algorithm Noitom has developed to determine whose data will be upload and download. This is how they ensure the experience can be smooth and unaffected in a network with limited bandwidth.
The last difficulty lies in the management aspect. “Imagine supervising 100 machines at the same time, how can I ensure they are all functioning? If I saw somebody acting weirdly, can I just walk over and said ’Sir, could you please kindly take off your headset and show it to me?’ Of course I cannot, even if I can, it is highly inefficient. Therefore we need to remote control.”
The reporter follows Professor Dai Ruoli to see a set of manage system developed by Noitom. Through the system, they can easily switch the presentation content and monitor the view point and performance of different headsets. At the same time, it is the server of Noitom’s inertia action capture sensor as well. Though such system is no rarity in computer room monitor field, but it was never applied in multi-person VR experience before.
Professor Dai Ruoli described this project as building a wheel as they are developing tools and content that are new to VR for management purpose, just like re-building a wheel.
Given the three major problems are solved, the presentation seems to be all prepared. But the real situation is much more complicated.
In war time, preparation is never well enough
“In war time, preparation is never well enough.” Professor Dai Ruoli laughed and nodded at the reporter’s comment. In Shenzhen, Noitom had encountered many uncontrollable factors, some seems to be minor interference while some are frustrating.
“For large event like this one, we have to build a set up for simulation. We successfully completed the set up in Tongzhou, Beijing and all the tests turned out well. But when we came to Shenzhen, problems began to emerge.” Professor Dai Ruoli said.
So what is the problem then? Well, it is the temperature. After Noitom moved the whole system and set up from 7 degree Beijing to 25 degree Shenzhen, the heat of 100 machines affected the system performance. Therefore they have to add an extra fan for every motherboard to solve the problem.
And there were more tricky problems waiting ahead: The host arranged an opening dance for the release event and another one before the VR experience, and both need to use the audio system. However once the audio system is turned on, the optical cameras’ position will shift away.
“On the big day, we adjusted the angel and accuracy of every camera and all the marks are fine. Then came the opening show. And with that vibration brought by the noisy music, all the tracing are not accurate any more.” Professor Dai Ruoli said with a frustrating smile.
In the end, Professor Dai Ruoli came up with a solution: he turned on the subwoofer for 40 minutes before the release so that all cameras onsite can find a most stable position in the vibrating environment, then they can adjust and test. In his words, all cameras are “accustomed to the vibration”.
Other problems continued to pop up as well: there are light interferences onsite; some guest’s cloth can reflect light; some guest accidentally touched the camera and affect the accuracy of tracing. “The guests arrived at the event and saw a beautiful umbrella shape column in the middle of the room so they cannot help to push it, which lead to a total mess.”
Facing these emergencies, Professor Dai Ruoli and his colleagues have to solve them one by one. “We asked the photographer to turn off the flash and only use large aperture opening to film. For the guest with light-reflecting cloth, we asked him to take it off for a while. And for the column, we asked Volkswagen to arrange some hostess to stand in front of it and prevent anybody from touching it.”
Though Professor Dai Ruoli said these solution with ease, but he and his colleagues must had taken great pressure in implementing them. Not only do they need to nail the technique, they also have to coordinate and negotiate with all parties. “We asked them to stop using flash light onsite and they felt like we are nuts and know nothing about the industry. But we have to forbid that to ensure the quality of the experience. You just need that resolution.”
In the end, Noitom solved every problem of this large event and raised the 70% successful rate up to 99%. Everybody is highly focused to solve every challenge, “It’s like everybody is given a strong dose of epinephrine and their potential soars.”
“We might fail next time, but we made it this time.”
The future of VR in Automobile Industry
When people are still focusing on VR’s application in entertainment field, it has already penetrated into traditional industries like automobile. At the beginning of this year, it is said that many renowned automobile companies are introducing VR technology into manufacture field, assisting the engineers to design automobile or simulating the production process. There are also many successful VR automobile presentation cases. In the recent Guangzhou Automobile Exhibition, VR content from many booths attracted great attention. Noitom also provided another automobile company with multi-person VR automobile presentation experience on that exhibition as well.
“Why automobile industry? The answer is very simple, it is because the industry have enormous volume.” Professor Dai Ruoli explained. “The automobile industry is in great need of new technology and industry with such massive volume are usually very generous to accept VR. They have the capital and the interest to try VR.”
With the development of VR industry marching towards maturity, VR begins to return to its basic function as a tool. This is an evolution path with people’s attention shifting from novelty to functions, and people’s fake need fades away while the real need begin to emerge.Then for automobile industry, what is “the real need”?
Professor Dai Ruoli told the reporter that, in his point of view, there are many key points in VR’s corporation with automobile industry. The design and manufacture formerly mentioned are for industry production; while for offline sales, VR can help automobile companies to attract potential customers to the store, prolong their stay at the store and quantization sales KPI.
In automobile industry, Noitom is promoting their successful project. At the same time, many Volkswagen 4S store across China have introduced Noitom’s virtual vehicle presentation system so that customers can closely observe the new car in 4S store as if they are at the release event.
For Noitom, it is a prosperous direction. The mature system of Project Alice combined with top partners of the industry can brought comprehensive experience for the customers. Just as Professor Dai Ruoli has mentioned, their advantages lie in their system: when Noitom enters a certain industry, they would like to corporate with content vendor through system rather than take control of everything. No matter it is automobile or education, or VR real estate and VR presentation, Noitom has always stick to this way.
For Noitom, there is still a long way to go. “After finishing this boldest thing ever, we have new learnings as well. At least we are no longer working blindly, we know what exactly the market wants and what technical points the client would like us to tackle.”
“I think to corporate with any industry with capital and industry would be helpful for the whole VR industry.” Professor Dai Ruoli said so at the end of the interview.