Advertisement

Toronto assistant GM Kyle Dubas talks Leafs, luck & analytics: 'It's not magic'

Kyle Dubas Brendan Shanahan

BOSTON — The summer of analytics has turned into the winter of discontent. Several NHL teams hired data-oriented people in the off-season and some have struggled so far. The list includes the Carolina Hurricanes, Edmonton Oilers, New Jersey Devils and Toronto Maple Leafs – four of the bottom seven teams in the league standings.

“Those teams haven’t magically transformed overnight into good teams, and people will jump on and say, ‘I told you that analytics [stuff] didn’t work. Those teams are no good,’ ” said Toronto assistant general manager Kyle Dubas, the summer’s highest-profile hire. “You go on my Twitter timeline after any game the Maple Leafs lose and see that firsthand.”

There is a simple explanation, though.

“It’s not magic,” Dubas said. “It’s not really magic. It’s a process, and it’s hard work, and it’s difficult, and you have to push your way through it.”

To have the best chance of success, no matter the philosophy, you need buy-in throughout the organization and time for the parts to align. You need management to draft, develop and acquire players that fit. You need the coaches to use the players in a way that fits.

The Leafs hired Dubas and created an analytics department in July, trying to catch up with other NHL teams.
The Leafs hired Dubas and created an analytics department in July, trying to catch up with other NHL teams.

Some teams have been using analytics for years, like the Chicago Blackhawks and Los Angeles Kings – winners of four of the last five Stanley Cups. Other teams are just getting started. There are different levels of buy-in from team to team.

The Leafs are a fascinating case study. They went from thumbing their noses at analytics to embracing analytics. They went from leaving their analytics budget unspent – choosing not to use their considerable financial advantage as the NHL’s richest team – to investing in an analytics department. It hasn’t paid off. Yet.

“There’s teams that are so far even ahead of us in our own league, and we’re trying to play catch-up as fast as we can,” Dubas said. “You’ve got to watch it, because if you try to rush it, you stumble and fall.”

Dubas spoke Saturday at the MIT Sloan Sports Analytics Conference. The title of his session: “How analytics has limited the impact of cognitive bias on personnel decisions.” In other words, he explained how analytics had helped him avoid bad influences and illogical decisions as the GM of the OHL’s Soo Greyhounds, shedding light on how analytics might help the Leafs.

At the end, he showed a graph. A red line showed the Greyhounds’ shot attempt differential from 2011-12 to present. It trended down, bottomed out for a while and finally shot up like a hot stock. A blue line showed the Leafs’ shot attempt differential this season. It traced the red line almost exactly at the start.

Where will it go from here? Well, the Greyhounds finished ninth in their conference in 2011-12, then sixth in 2012-13, then second in 2013-14. They’re first this season. The Leafs are 14th in their conference right now.

Dubas said he thought it would take longer with the Leafs. In the OHL, players are drafted and step right into the lineup. Their eligibility expires when they’re 20 years old, and the roster turns over. In the NHL, players are drafted and take years to develop. They can sign long-term contracts. There is a salary cap.

“It just takes time,” Dubas said.

* * * * *

The Leafs were a referendum on analytics last season. They were one of the worst possession teams in the NHL and allowed a league-high 35.9 shots per game, and as many predicted in the analytics community, it caught up with them. They missed the playoffs.

They introduced Brendan Shanahan as their new team president on April 20. Asked about analytics that day, he said: “Generally speaking, I think that if there is information out there for you and you choose to ignore it, then that’s a mistake by you.”

Shanahan had a lot to learn. So he did his homework.

Dubas got married in Mexico on July 6. While there, he received an email from Shanahan asking for a meeting. He had never talked to him before. All he knew was what Shanahan had said in that press conference.

Dubas says the Leafs' front office continues to ask questions but everyone's in line with analytics. (AP)
Dubas says the Leafs' front office continues to ask questions but everyone's in line with analytics. (AP)

“What do you think it’s about?” Dubas asked his wife.

“I don’t know,” she said. “But you better go.”

Dubas returned from Mexico and met Shanahan on July 10. They were supposed to talk for an hour. They ended up talking for seven hours. After that, Dubas met with Leafs GM Dave Nonis. He met with both of them. He met with both of them again.

The Leafs announced the hiring of Dubas on July 22, the same day they announced the firings of veteran executives Claude Loiselle and Dave Poulin.

Dubas was only 28 years old.

He went on to assemble an analytics department of bloggers and web designers: Cam Charron, Darryl Metcalf and Rob Pettapiece. Metcalf’s website, extraskater.com, had become popular for how it had made advanced stats easy to use and understand. Now it belonged to the Leafs and only the Leafs and was taken offline.

“We had the data,” Dubas said. “We had all that information. But the interface for us was so important.”

The Leafs also started using SAS, an analytics software suite. They won’t reveal the specifics of what they’re doing for obvious competitive reasons, but they will say it’s more than what we talk about in the mainstream media and the blogosphere.

“Yes,” Dubas said. “I think you have to. If you only go up to the line where something is publicly accepted, everyone is going to be at that same line. You have to try to advance it.”

At first, Dubas had to explain the basic principles to people from ownership on down – the importance of possession, the role of luck. In a meeting with Larry Tanenbaum and Dale Lastman – the chairman and a member of the board of Maple Leafs Sports & Entertainment – he talked about PDO.

“Well, what does it mean?” they asked. “What does it represent?”

Dubas explained that PDO wasn’t an acronym, that it was named for a commenter on the Internet. It was a measure of luck, or variance. It was combined shooting and save percentage that tended to revert to 100. If a team is above 100, as the Leafs have been when hot, then that success is likely unsustainable. And vice-versa.

Shanahan challenged Dubas to come up with a better explanation.

“You’re paying your staff millions of dollars,” Dubas said. “You don’t want to hear it’s more influenced by luck. These coaches and these staffers are supposed to have the answers. Luck isn’t a suitable answer to very successful business people. We’re owned by Bell and Rogers. It’s not a real … When you say luck, aren’t you supposed to be able to do the things that take luck out of it?”

Dubas went back to his staff. Pettapiece turned PDO into an acronym after all: “Percentage Driven Outcome.” Now when Dubas reports to the board members, as he did as recently as Friday, he is speaking their language.

“Once they understood it,” Dubas said, “they were fascinated by the fact that we could judge how much of our outcome was being fueled by percentages or variance.”

Dubas said the Leafs’ front office is speaking the same language now, too. At first, Shanahan knew Corsi and Fenwick, and he had questions about the rest of it. Now he knows far more and asks even more questions. Nonis has had an open mind, despite anti-analytics comments he had made in the past.

“That’s been the greatest thing with Brendan and Dave both,” Dubas said. “Dave challenged me to try to explain things that maybe I wasn’t good at explaining, so you can’t B.S. your way with inquisitive people like that. You have to give definitive answers and work on our own way to explain them clearly, and that’s been a big help, and they both have shown a huge passion to learn. Our whole staff has.”

* * * * *

Dubas comes off like he knows a lot but isn’t a know-it-all. He isn’t afraid to say he’s made mistakes. He isn’t afraid to say he doesn’t have all the answers. He isn’t afraid to say analytics – and individual analysts – have their strengths and weaknesses.

“I think in analytics in hockey, we’re so anxious to prove to people that we’re right that we oftentimes will totally dismiss something that may be there that we just can’t measure yet,” Dubas said. “I think that’s always the greatest fear we have. … I think just because we can’t necessarily measure it doesn’t necessarily mean it doesn’t exist. …

It's Year 1 in Toronto's analytics era, and there's a long way to go -- in the numbers and on the ice. (USA Today)
It's Year 1 in Toronto's analytics era, and there's a long way to go -- in the numbers and on the ice. (USA Today)

“I think to ignore that there’s a human element to it is not smart.”

But it’s also not smart to ignore that the human element can lead you astray. Dubas spoke about several kinds of biases. People tend to look for data that supports what they already think. They tend to weigh recent information too heavily. They tend to read too much into small sample sizes. They tend to gravitate toward simplicity because it’s easy to explain.

“What analytics taught me is, your eyes and your mind are lying sons of [guns] in the worst absolute way,” Dubas said. “Your eyes and your mind can fool you drastically if you allow them to.”

Dubas put up a slide titled “The Big Lesson.” You have to “accept randomness, uncertainty, chance and luck as major contributors to success or failure in the hockey business.” There is a “difference in knowing and valuing.”

“There is a massive difference between just collecting data and information and actually valuing it and using it to build it into your process about how you’re going to guide your team,” Dubas said. “Basically, there is a big difference between raw statistics and data gathering and using that data to incorporate into your process as a team.”

The Leafs have incorporated data into their process more and more – from moving away from enforcers, to firing coach Randy Carlyle. They want to exploit market inefficiencies, and they want to emphasize skill and puck possession. They don’t want to inundate players with numbers, but they want to educate them and encourage them.

They want to tolerate mistakes in the short term with an eye on the long term. Dubas even used the term “acceptable turnover.” If a player takes a proper calculated risk and it doesn’t work out – like, say, trying to carry the puck over the offensive blue line instead of dumping it in – he should not be berated when he returns to the bench.

“There’s a trust that’s built, and that helps the team and the player grow,” Dubas said. “That’s a massive issue as well. I hate when I read in a quote from a player, ‘We just need to keep it simple.’ I don’t know what that means, ‘Keep it simple.’ It really bothers me.”

This is not simple. This is not magic. The Leafs are about to miss the playoffs for the ninth time in 10 seasons, and patience is running thin in the Centre of the Hockey Universe. But this is still only Year 1 of the analytics era in Toronto, and there is a long way to go – on the graph, on the ice.

MORE NHL COVERAGE ON YAHOO SPORTS: