KEY TAKEAWAYS
  • The ultra-wide image-synthesis technology of Kirari!, which involved installing multiple 4K cameras to capture a multitude of different angles, was used to create a full viewing angle creating a new, ultra-realistic sports viewing experience, which was used during the 2020 Olympics.
  • During the pandemic, further applications of this technology became apparent, such as sporting and cultural events that people couldn’t attend in person due to Covid restrictions.
  • The technology develops a sense of presence: the physical venues themselves were recreated for the purposes of remote viewing, with the courts, nets and spectators’ seating all set up to be just like the real thing.
  • A sense of unity was also created during the marathon at Tokyo 2020, where athletes were able to hear a cheering crowd in Sapporo, 516 miles away, who were watching the marathon runners in Tokyo in real time via ultra-low latency tech.

Originally published in NTT Technical Review, Vol. 19, No. 12, December 2021.

 

NTT R&D aims to create a new way to watch sports

Although device and video-relay technology has evolved substantially over the past 80 years, the fundamental experience of watching sports has not changed that dramatically. In preparation for the Tokyo Olympic and Paralympic Games in 2020, NTT began research and development (R&D) of an ultra-realistic communication technology called Kirari!, with the aim of creating a new, ultra-realistic sports viewing experience. We wanted to give the feeling of being at the venue to those who could not attend, by delivering the entire spectacle in real time. 

Kirari! removes the frames and borders of conventional TV and public viewing, and transmits the event space itself. Prior to the Olympics, NTT experimented with Kirari! in a variety of sporting and non-sporting events, such as the Japanese professional soccer league, Major League Baseball in the US, a synchronised performance by a three piece EDM group who were situated in three different cities around the world, as well as the opening event at the new National Stadium in Tokyo, among others.

Initially, Kirari! was intended to provide viewing for people who lived far away from the event venue, those who could not afford tickets, and those unable to go out due to illness. However, with the spread of Covid in 2020, people’s mobility became ever more restricted, and the scope of Kirari!’s possibilities grew even wider. 

Of course, people could still stream events as they always had done, but this was never going to be a sufficient substitute for in-person attendance. This was corroborated by the results of a survey titled  “Awareness Survey on Live Music Distribution” carried out in September 2020 by SKIYAKI Inc., which found people craved the presence, unity and occasion of in-person events. Indeed, 77% of the survey’s respondents preferred real life events because of the “sense of presence” (92.9%), “sense of unity” (93.6%), and “sense of occasion” (69.6%). This only furthered our desire to recreate these sensations, so we decided to take on that challenge at the Olympic and Paralympic Games Tokyo 2020.

Presence without being present

The sense of presence at a sporting event is ultimately what gives it the air of occasion and spectacle, and is something that sporting fans hold so close to their hearts. It’s the “I was there” moments that fans will treasure for life. Of course, in reality, you may see no more than just a speck in the distance, and although it is easier to see the action via traditional streams, the feeling of being there is what makes the real thing so special. 

The small frames of smartphones and TVs lend themselves to us perceiving the event as a separate space, and does not provide the same sense of presence one might get from being inside the venue; in other words, we do not feel that we are sharing this occasion with the athletes we have come to see and the other fans enjoying the spectacle.

To create the feeling that one is there, as if you were sitting in the stadium, we attempted to reproduce an expanse of space that covers the full viewing angle. To achieve this, we used the ultra-wide image-synthesis technology of Kirari!, which involved installing multiple 4K cameras to capture a multitude of different angles. The footage captured was then seamlessly stitched together in real time, to produce a single ultra-high-resolution video image. The resulting image had a resolution of over 20k, which meant that it could be blown up to a display several tens of metres wide without losing significant quality.

“We recreated three-dimensional images of the athletes for an audience at a remote location over 500 miles from the action.”

This technology was put in place on a few different occasions during the Olympics of 2020, including the sailing, badminton and table tennis events. In the sailing, spectators were naturally seated far from the action, but by using Kirari!, we were able to transmit the sailing course directly to an offshore display much closer to where the spectators were seated, reproducing the sensation as if the races were being held right in front of them. During the badminton and table tennis events, we recreated three-dimensional images of the athletes for an audience at a remote location over 500 miles from the action. 

To achieve this, we used a component technology of Kirari! which extracts the athlete’s image in real time and displays it in a holographic manner to a remote audience. To further enhance the sense of reality, the physical venues themselves were recreated for the purposes of remote viewing, with the courts, nets and spectator seating all set up to be just like the real thing. The holographic images of the athletes were then displayed within the dimensions of the remote venue to create a sense of the athletes’ presence just like that at the main event.

Together, whilst apart

Finally, we aimed to create a sense of unity; a connection between the athletes and spectators as well as among the spectators themselves. In all sports, the ruckus made by a cheering crowd is the most powerful tool for creating a sense of atmosphere and occasion, and it gives the spectators a closer connection to the sports stars as well as one another, something which cannot be recreated with online streaming. 

This sense of unity was exemplified once again during the marathon at Tokyo 2020 where a cheering crowd in Tokyo were able to watch the marathon runners in Sapporo, 516 miles away, in real time via our ultra low-latency tech. Not only were they able to watch 3D images of the marathon, but their cheers in turn were delivered to the athletes to create a sense of unity between the two parties that transcended distance.

 

All issues: NTT Technical Review.