'Glimpse into the future': MLSE, Amazon bringing AR, VR to sports viewing

If you simply walked into Scotiabank Arena’s practice gym on Monday evening, you’d see a room full of reporters, developers, as well as representatives of Amazon Web Services (AWS) and Maple Leafs Sports and Entertainment (MLSE).

What you wouldn’t see just yet is a 3D rendering of mannequins, representing players from the Toronto Raptors and Milwaukee Bucks, recreating their most recent NBA matchup. That is until you put on an augmented reality (AR) or virtual reality (VR) headset.

With the headset, you’re able to immerse yourself directly into the action virtually. The players come to life in front of you, providing the chance to follow them from baseline to baseline, or simply sit courtside — whatever you feel gives you the best view of defensive breakdowns and offensive explosions.

“It's a glimpse into the future,” said Humza Teherany, MLSE’s Chief Technology and Digital Officer.

MLSE Digital Labs and AWS have partnered to launch SportsX, a new research and development program. Its first initiative is introducing AR and VR to change the way we’re able to consume professional sports. That’s both for the fan experience and to help enhance team performance, as players and coaches can relive moments to help with their training.

As part of the Jan. 24 launch, MLSE and AWS demonstrated multiple variations of how AR and VR headsets can be used to display the data they've collected. It allows those wearing the headsets to either immerse themselves directly into the action, or to even receive real-time stats as part of their viewpoint while watching a game — such as during the Toronto Maple Leafs’ matchup versus the New York Islanders on Monday night.

“Frankly, this hasn’t been done anywhere else in the world,” said Teherany.

“The big thing about this program is building the future of sports from Toronto, to the rest of the world and for us to be able to use these great things inside MLSE. Hopefully it becomes examples for the rest of the sports world.”

How the technology works

The “Immersive Basketball Experience” uses a mix of optical tracking body pose data — such as the position of each joint and limb — 3D models and a video game development engine. The result is a life-sized way to relive the game, as the players are rendered into 3D around you.

For the NBA experience, MLSE and AWS are utilizing the in-game cameras that are present across all arenas, to collect the necessary data. The same experience can be used for NHL games, while there are advancements in other professional sports such as Formula 1 to get the biomechanical data, as we can expect this experience to come to all sports.

For the NHL, the experience can be even more realistic, since the league uses “NHL Edge.” To collect data, the NHL uses infrared technology that’s embedded in the pucks and in players’ jerseys — information that's relayed to — in this case — a AWS outpost in Scotiabank Arena. This technology allows for the tracking of each movement part of the game, making way for the “Extended Reality Stats Overlay” feature on an AR or VR headset.

While watching an NHL game, viewers can access an overlay that shows statistics for either team, such as puck possession, speed and distance travelled. Using a laser feature as part of an accompanying remote with the headset, viewers can even individually select a player, such as Auston Matthews, to focus on to get specific stats.

“You can enjoy the game, while seeing the analytics and data in real time all in one place,” said Teherany, noting that this can be used by an avid hockey fan or someone that’s learning the game.

Future uses: Arenas to everyday life

Currently, the cost of a couple thousand dollars, and size of the headsets, remains a challenge to get it to a mass group of people. Teherany says that when innovations like Apple’s augmented reality glasses do come out, it’ll change the market, and we can expect these features to be accessible in the size of your everyday glasses.

Eric Gales, the country manager for AWS Canada, says what’s changed significantly about the market in recent years is that before you needed a lot of gear to render an AR or VR experience. Now, thanks to all the collected information they have on the cloud, they’re ready to doll out this experience when there’s an accessible option to make it mainstream.

The “Extended Reality Stats Overlay” is an experience MLSE wants to give its fans inside arenas. It would help maximize the viewing experience, especially for those who are sitting high up in the crowd. A similar statistical experience — without the headset element— was also displayed on a table-sized panel in Scotiabank Arena — which could become a fixture inside its luxury suits.

Outside of an arena, there's the problem of figuring out how to license these games for AR and VR use — since broadcasting rights are owned by Rogers and Bell.

For the Immersive Basketball Experience, members of MLSE and AWS want to stretch out its capabilities as far as into people’s homes to use on a daily basis. Having spoken to multiple developers, a realistic timeline is 5-10 years, given the aforementioned cost and size issues of the headsets.

Christian Magsisi, MLSE’s vice-president of venue and digital technology, explains that for a life-size version of the players, you’ll need to have a bigger, more accommodating space. For example, if you wanted to be right next to a life-size Kawhi Leonard during his infamous 2019 game-winning shot, you’d need a large space like a gym or a field.

But with the capabilities of AR and VR, users will be able to lay out the experience on a smaller surface such as even a tabletop, helping them see the action unfold right in front of them.

Creating a competitive advantage

One of the benefits of the VR and AR experience is for the professional sports organizations part of MLSE. Magsisi says it allows players and coaches to recreate pivotal moments from their games and learn from them.

“This allows us to create a competitive advantage,” Magsisi said, noting that members of the Raptors and Leafs have already tried out the tech.

Instead of needing 10 real-life players to recreate a specific play on an NBA court, you’re able to use AR or VR to make that moment come to life virtually. Players and coaches are also able to watch or rewatch games with the headset technology that shows them crucial stats as the action unfolds, giving them an opportunity to see what exactly is contributing to their team’s play.

Gales also notes the impact this can have on enhancing their understanding of injuries, and how to avoid them to enhance player safety. This type of data and innovative research has been a focus for the NFL in partnership with AWS since 2019.

Getting people involved

Along with enhancing viewing experiences and helping teams get a competitive edge, SportsX is basing itself on the concept of getting communities involved. They want fans to help lead them toward future ideas through submissions, but also to try out the AR and VR experiences to gather feedback and improve the product.

"We've been watching sports the same way for so long," Magsisi said. "To give people virtual reality goggles is a huge change. So we need to ease people into it. So even though it may be available sooner, I think it'll take some time for people to really adopt.”

Those who are interested can immediately start to sign up on sportsx.io to test out the technology themselves.

More from Yahoo Sports