Substitute Jamie McAllister's 74th-minute goal proved decisive as Hearts reached the Scottish Cup fourth round.
Hearts led at the interval thanks to captain Steven Pressley, who was allowed to head home unchallenged in the 24th minute.
Kilmarnock threatened to equalise in the second half, and Paul Di Giacomo and Steven Naismith will feel they should have done better when well placed just after the restart.
McAllister made them pay five minutes after coming on when he slid in to beat Alan Combe from eight yards.
Colin Nish pulled one back for Kilmarnock five minutes from time, but Hearts held on.