Dougie Imrie thought he had earned Caley a point with a stunning free-kick just a minute earlier but Hearts had the final say.
Caley Thistle also had Roy McBain sent off in the second half for dissent after he had given away a penalty which Ryan Esson saved from Michael Stewart.
Hearts had the first attempt at goal when a Bruno Aguiar free-kick was not cleared properly but Stewart's 20-yard shot flew high and wide.
A superb throughball from the outside of Aguiar's boot almost put Stewart in but Esson was off his line quickly to grab the ball.
Caley Thistle were forced into a change after only 18 minutes when Thierry Gathuessi went off injured to be replaced by David Proctor.
Esson had to be alert midway through the half when Grant Munro slid in to prevent an Andrew Driver cross reaching Christian Nade and he produced an excellent reaction save.
David Obua managed to get his head to an Aguair corner but there was no power in his effort and Esson gathered easily.
But Caley Thistle wasted a great chance to go ahead after 27 minutes. Don Cowie's free-kick found Imrie unmarked at the back post and he had time to pick his spot - but elected to head the ball into the side-netting.
It gave his side confidence and Ross Tokely tried his luck with an over-ambitious effort shortly afterwards which curled yards too high.
Tokely was fortunate to escape a booking after catching Aguiar as the Hearts midfielder was looking to create an opening.
Garry Wood then should have done better in what looked like an offside position but he could not get the ball out from under his feet after good work from Iain Vigurs.
But Hearts snatched the lead six minutes before half-time when Esson was unconvincing as he went to meet Driver's cross and Obua outjumped him to score with a tame header.
Esson made a fine save to keep out a Driver shot early in the second half before Hristos Karipidis headed over as Hearts pressed for a second.
But Caley Thistle equalised after 56 minutes when Pavels Mihadjuks headed powerfully past Jamie MacDonald from a Don Cowie free-kick.
Stewart then had his penalty saved by Esson after Driver had been brought down by McBain. McBain was booked for his protests and then moments later shown a straight red card as he refused to let it lie.
Obua headed Hearts in front after 79 minutes from a fine Driver cross but Caley thought they had salvaged it with Imrie's late free-kick.
But it was not to be as Driver set up Kingston for the winner with just a minute left to end a breathtaking match.