The Jambos midfielder broke the deadlock in the 58th minute with an unstoppable drive from the edge of the box before Kris Commons, who had just returned to the Hoops starting line-up, was shown a straight card by referee Craig Thomson in the 65th minute for a reckless challenge on Adrian Mrowiec.
And as the 10-man visitors piled forward in search of an equaliser, Stevenson scored a breakaway second in the 80th minute to leave Celtic in third place behind Motherwell and 10 points behind leaders Rangers, albeit with one game in hand.
The Parkhead club will now have the international break to regroup but that is of little concern to the home supporters who cheered their side off the pitch at the end of an exciting afternoon.
In the fourth minute Celtic keeper Fraser Forster made a good save from Jambos' lone-striker Stevenson who had escaped his marker to get in a header. Moments later, Hearts skipper Marius Zaliukas struck the post with a left-footed shot from the edge of the box.
Celtic roared back and in the 20th minute they should have taken the lead when James Forrest's cross from the right found Mohamed Bangura but his close-range header was brilliantly saved by Jamie MacDonald. Seconds later, following the corner, the Hearts keeper made another good save at his near post from Commons' drive.
On the half-hour mark, after Hoops midfielder Victor Wanyama was penalised for his challenge on Ian Black at the edge of the box, with Forster diving to prevent Jamie Hamill`s free-kick from going in. Celtic responded again and MacDonald made a fine block from Commons' powerful drive from just inside the box.
The interval allowed everyone to get their breath back but when play commenced it was the visitors who took the initiative and Commons had another long-distance effort which was dealt with comfortably by MacDonald. Celtic pressed with purpose but in the 58th minute Hearts took the lead with a fine strike by Skacel.
Hamill's high ball from the right fell out of the air and, when it broke from David Templeton, the Czech midfielder arrowed in a unstoppable left-footed drive from the edge of the box to give Forster no chance.With the home fans still celebrating, Lennon immediately replaced Mohamed Bangura with Anthony Stokes.
In the 65th minutes there was more trouble for the Parkhead men when Commons was sent off for a reckless challenge on Mrowiec in the middle of the park. Hearts had to defend desperately at times but in the 80th minute Stevenson made it 2-0 when, with the Hoops defence stretched and struggling with a long Danny Grainger punt, he ran on to clever back-heel from Templeton and drove high past Forster from 10 yards.