In an evenly balanced first half it was Caley's Russell Duncan who burst through to score his second goal of the season firing low into the corner of the net and giving the goalkeeper no chance.
Hearts were forced into their first change after just half an hour after Andrius Ksanavicius picked up an injury. He was replaced by youngster Calum Elliot.
Andrew Driver and Elliot linked up well with the end of the half approaching with Elliot stepping over the ball to leave it for Driver, but his effort lacked power and went straight to Michael Fraser.
The Gorgie men were booed off the pitch by the crowd at half-time.
Hearts looked intent on levelling the match as the second half began, but Andrius Velicka saw his effort well cleared by Fraser and Elliot's attempt on the rebound was also saved.
Disaster struck for Hearts on 53 minutes as Eduardas Kurskis was adjudged by referee Willie Collum to have brought down Marius Niculae in the area. He pointed straight to the spot and John Rankin dispatched a perfectly-taken penalty to beat the diving keeper.
Christophe Berra came close to pulling one back for Hearts minute later, but narrowly missed the target then, Ruben Palazuelos earned a corner which got his side back into the match after a good run down the left.
Driver delivered a perfectly placed corner which was headed in by Berra to make it 2-1.
Hearts brought on Saulius Mikoliunas and Michal Pospisil on 65 minutes and they had an immediate effect on the game. Only a fantastic save from Fraser stopped Mikoliunas from levelling.
But, for a second time in the match, Hearts' hopes looked to have been scuppered when Kurskis was sent off after squaring up to Duncan off the ball and receiving his second yellow card of the afternoon.
With all of their substitutes used up and no chance of bringing on the reserve goalkeeper things looked bad for Hearts, but the gods must have been smiling on them as Velicka was brought down in the area and won a penalty which he duly dispatched to equalise.
Inverness though were not to be thwarted and Marius Zaliukas let the ball slip to Bayne and unfortunate stand-in goalkeeper, Lee Wallace, stood no chance as the substitute struck to give Inverness victory.