The Ravens were down by eight with two timeouts when they got the ball back with 2:25 on the clock in regulation.
It took them 1:51 to score a touchdown and a two-point conversion to tie the game.
Then they scored again with a Tyler Huntley 1-yard run in overtime, and Kyle Hamilton grabbed an interception to seal a 30-23 victory.
With Lamar Jackson out due to an ankle injury, the Ravens were nevertheless effective on offense, with Tyler Huntley completing 24-of-32 passes for 201 yards with two touchdowns. He also had six carries for 20 yards with the game-winning TD in overtime.
Derrick Henry rushed for 102 yards and caught an 18-yard touchdown to help pace the offense. Justice Hill also had 73 yards on the ground, including a 47-yard run that set up a score.
The Falcons led for much of the game and even had a shot to win at the end of regulation with 40 seconds on the clock and two timeouts. But starting quarterback Michael Penix Jr. took a hard hit to his side as he was sacked with 27 seconds left in regulation and ended up suffering a rib injury.
Tua Tagovailoa played one snap in the fourth quarter after Penix was injured. While Penix tried to come back in, he was clearly holding his side and looked to be in pain.
After Huntley scored in overtime, Tagovailoa came back in to play Atlanta’s final drive. It actually started well with a 20-yard completion to Drake London, who finished the game with seven catches for 118 yards.
But then it all fell apart, as Tagovailoa’s pass on the next snap went right to Baltimore’s Kyle Hamilton for a game-ending interception.
Penix was announced as questionable to return with a rib injury. But with under three minutes left in overtime, he was, for all intents and purposes, out.
Penix finished 19-of-32 for 240 with one touchdown.
Bijan Robinson had 17 carries for 86 yards and caught four passes for 32 yards.
Now at 4-1, the Ravens will be on the road again in Week 6 to play division rival Cleveland.
The 2-3 Falcons will stay at home to play the Bears in Week 6. But they may have to play a backup at quarterback.