The red-booted striker punished Celtic for some slack defending to suggest Hearts will again mount a title challenge this season.
The game looked set for a draw when Stilian Petrov cancelled out Bednar's opener but Neil Lennon was short with a pass-back thee minutes from time and Bednar struck the winner.
Celtic manager Gordon Strachan and Hearts coach John McGlynn were both sent to the stands late in the first half after an angry exchange between the two dug-outs.
Hearts dominated the first half-hour as they took the game to Celtic.
The home side started at a breathtaking pace and Celtic were over-run in midfield in the early stages.
Bednar shot well over the top in the fourth minute as Hearts threatened and, 15 minutes later, the Czech player was not far away from connecting with a Bruno Aguiar free-kick.
For all Hearts' dominance, Artur Boruc did not have too much to trouble him in the first half.
Bednar should have hit the target in the 26th minute when he met another Aguiar free-kick but the ball struck his shoulder and drifted over.
Ibrahim Tall went off with an ankle injury to be replaced by Robbie Neilson as Hearts had to reshuffle their ranks.
Celtic came more into it in the final 15 minutes of the first half. Shunsuke Nakamura pulled a shot wide before Jiri Jarosik lashed another effort wide.
Craig Gordon was finally called into action four minutes from the break when he saved well from Kenny Miller.
The game erupted shortly afterwards when play was stopped after an injury to Julien Brellier with Celtic threatening.
There followed an angry exchange between the two benches and referee Stuart Dougal sent both McGlynn and Strachan to the stand.
Hearts took the lead four minutes into the second half with a well-worked goal. Michal Pospisil freed Bednar and he produced a cool finish to put his side ahead.
Boruc then made a fine save to deny Brellier as Hearts continued to look dangerous.
However, Celtic scored against the run of play after 65 minutes when they broke quickly from a Hearts corner and Aiden McGeady fed Petrov who produced a terrific left-foot finish.
Then, with just three minutes left, a misplaced pass from Lennon let in Bednar and he rounded Boruc to nab the winning goal.
Aguiar hit the crossbar in injury-time to underline Hearts' dominance.