Tampa Bay went ahead to stay when Kucherov scored from the low right circle with 2:30 remaining off a cross-ice pass by Mikhail Sergachev.
Kucherov has a four-game goal streak and an eight-game point streak overall.
Tampa Bay scored three of the four goals between the teams over the final 8:56 of the game.
“It was a wild one, no doubt about it,” Tampa Bay coach Jon Cooper said.
Ottawa's Brady Tkachuk had his shot during a breakaway with 1 second to play go just wide. The chance came after Tampa Bay failed to score on an empty-net rush.
“Shows don't leave the game until its over,” Cooper said.
“It was just a weird one ... thankfully it didn't hurt us,” Lightning captain Steven Stamkos said.
Tampa Bay played without star defenseman Victor Hedman, who is considered day to day after he was scratched an upper-body injury.
“I don’t know if we should have won this game, that’s a good team and they don’t panic out there,” Joseph said. "I think we were just a little bit immature in the way we finished the game here.”
Point tied it at 3 with 7:05 to play in the third.
After Killorn scored at 11:04 of the third, Ottawa regained the lead at 3-2. Tkachuk stole the puck from Vasilevskiy behind the net and set up Giroux in the slot.
Forsberg made a strong post-to-post save during the opening minute of the second on Stamkos, who raised his stick after taking his left circle power-play shot thinking it would end up in the net.
Joseph, playing his first game against his old team after being traded in March, put the Senators up 2-1 on a breakaway with 7:39 left in the second. Joseph got his first goal of the season with Tampa Bay on a 5-on-3 power play.
The Lightning's two-man advantage was part of a five-minute penalty for a hit on Erik Cernak by Gambrell at 7:27. Cernak went to the locker room but returned later in the period.
Tampa Bay had six shots, and Stamkos hit the post, during the extended power play as the Lightning outshot Ottawa 26-8 through two periods.
“Strange game,” Point said. “Execution wasn't there for a lot of it. I thought our 5-on-5 was actually not too bad. The power plays in the second, we got so many and did nothing with them, It's tough.”
Stamkos had seven shots in the opening 40 minutes. He finished with eight shots and two assists. He has gone five games without a goal after getting seven in the first five of the season.
Sergachev tied it at 1 on the power play at 3:17 of the second.
Gambrell, playing in his first game of the season, made it 1-0 on the Senators’ first shot 5:44 into the game.
Vasilevskiy stopped Drake Batherson's shot during a 2-on-none during a second-period power play.
Senators: G Cam Talbot (ribs) went through an extensive on-ice session after the morning skate and is expected to return within a week. “We’re hopeful that over the next two, three games that he can get back,” Ottawa coach D.J. Smith said.
Senators: Start a three-game homestand Thursday night against Vegas.
Lightning: Host Carolina on Thursday night.
AP NHL: https://apnews.com/hub/nhl and https://twitter.com/AP_Sports
Mark Didtler, The Associated Press