Lee McCulloch threw Rangers a lifeline when he equalised after the break before Boyd came off the bench to secure the win from the penalty spot in the final minute.
Boyd was named as a substitute after suffering from a sickness bug earlier in the week, while Allan McGregor retained the gloves ahead of Neil Alexander. However, Hearts did swap goalkeepers, with Janos Balogh starting ahead of Marian Kello following the 4-0 defeat to Dinamo Zagreb in the Europa League on Thursday night.
Michael Stewart and Marius Zaliukas were both suspended for the visit of the Scottish champions, paving the way for the introduction of Ian Black and Witteveen.
The match got off to an explosive start when Kenny Miller and Suso Santana clashed on the touchline and both players picked up a caution with just three minutes on the clock. McGregor was then called into action to smother a weak effort from Black, seconds after John Fleck claimed he had been fouled at the other end of the park.
The Hearts were determined to make amends for their dismal European efforts and were threatening again when Lee Wallace flighted a cross to Suso at the back post but Papac coolly snuffed out the threat before he could pull the trigger. Rangers were dealt a major blow when they were reduced to 10 men with just 14 minutes gone when Thomson was shown a straight red card by referee Craig Thomson for a challenge on Black. Only the woodwork prevented Hearts from taking the lead when a lovely swerving effort from Suso smacked off McGregor's right post. But the Rangers goalkeeper was plucking the ball out of the back of the net with 31 minutes gone following a blunder. Witteveen exchanged a one-two with Christian Nade before attempting to sidefoot home. McGregor should have held the effort but spilled the ball over the line to hand the home side the advantage.
Hearts could have doubled their lead shortly after the restart when Witteveen bore down on goal after racing on to a long ball but was denied his second of the day by a well-timed interception from Madjid Bougherra. McGregor was then called into action at the resultant corner when he did well to swipe a Goncalves header on to the crossbar before the danger was eventually cleared by the Rangers defence.
Rangers were level after 63 minutes. David Obua picked up a yellow card for a challenge on Miller and the foul proved costly for the home side. Steven Smith, who had replaced Fleck at half-time, took the free-kick and delivered into the packed goalmouth where Lee McCulloch was waiting to send a glancing header past Balogh in the Hearts goal.
Hearts were dealt a further blow when Calum Elliot left the pitch on a stretcher just seven minutes after coming off the bench to replace goalscorer Witteveen. Rangers were the team on top and could have taken the lead when Miller whipped a cross from the right to McCulloch but, this time, he connected awkwardly and blasted over the crossbar.
With 88 minutes on the clock, Rangers threw Boyd into the action in place of Miller and he fired Rangers into the lead from the penalty spot a minute later. Steven Naismith was tripped in the box by Ismael Bouzid and Boyd stepped up to convert to claim his first goal at Tynecastle in seven years and snatch all three points for Rangers.