City drew 1-1 in the 120 minutes, with Marcus Tudgay netting a late equaliser in the 90 minutes. Ruben Lameiras was the unfortunate City player to miss in the lottery of the penalty shoot-out as Dale netted all five of their penalties.
Bryn Morris and Reice Charles-Cook made their first full starts for the club as City were taken past 90 minutes at Spotland.
Subs: Burge (GK), Johnson, Haynes, C.Thomas, Swanson.
The Sky Blues started on the back-foot and both Sam Ricketts and Aaron Martin were kept busy with a number of blocks in the opening exchanges.
The hosts almost had the opener when McDermott had a pop-shot which was deflected, but debutant Charles-Cook made a fantastic stop to claw the ball away from the top corner.
Charles-Cook could do nothing about the opening goal though, as Dale hit City on the break and McDermott hit a majestic strike past the keeper outside the area into the top left-hand corner.
The second-half started at a better pace for the Sky Blues and it wasn't long before James Maddison came on for Romain Vincelot.
Tony Mowbray's men looked revitalised and pressed the issue, but almost went 2-0 down with 20 minutes to go, only for Aaron Martin to make a fantastic block to deny Calvin Andrew in the area to keep it at 1-0.
George Thomas came on for Jim O'Brien for the final 15 minutes of normal-time as the Sky Blues hunted down the goal that would take the game to extra-time.
And they almost had in when Ricketts picked out Maddison in the area and he got half a yard to smash a shot which was well saved by Lillis.
But Tudgay was there to get the crucial goal when Maddison and Thomas combined superbly and the latter's cross was met by the striker who poked a brilliant finish in from six-yards.
City almost had the winner in normal time but Tudgay had a goal dramatically disallowed, when he hit a wonderful volley in the area but the linesman flagged late to deny.
Extra-time was scrappy as both teams began to tire but Dale should have had the lead on two occasions, first Charles-Cook made a super save from Jamie Allen before Ian Henderson should have had a goal when he headed a glaring opportunity straight at Charles-Cook.
Stokes then should have netted a clinching goal of his own with eight minutes to go when he robbed Jimmy McNulty in the area but he hit the side netting on his left-foot from five-yards.
Allen then hit the post with minutes to spare with a volley from 20-yards before the entertaining cup tie would be decided by penalties.
Michael Rose netted confidently to open the shoot-out before Tudgay smashed into the left corner to make it 1-1. Henderson netted before Morris smashed in a great penalty. Kennedy and Maddison then exchanged scoring spot-kicks before Joe Rafferty gave Dale the lead. Lameiras was unfortunate as he sent the keeper the wrong way but rattled the bar with his spot-kick. Calvin Andrew then scored despite Charles-Cook getting close to the penalty as the Sky Blues were eliminated from the Capital One Cup in the first round.
TEAMS
Coventry City: Charles-Cook (GK), Phillips, Ricketts, Martin, Stokes, O'Brien (G.Thomas 76), Vincelot (Maddison 53), Morris, Fleck, Lameiras, Tudgay.
Rochdale: Lillis, Rafferty, McNulty, Lancashire, Vincenti, McDermott, Allen, Camps (Rose 99), Kennedy, Bunney (Andrew 45), Henderson.
Subs: Eastham, Allesandra, Noble-Lazarus, Tanser, Diba Musangu.
Subs: Eastham, Allesandra, Noble-Lazarus, Tanser, Diba Musangu.
Penalties scored: CCFC - Tudgay, Morris, Maddison. ROCHDALE - Rose, Henderson, Kennedy, Rafferty, Andrew.
Attendance: 1986
Referee: E Ilderton
Cards: Tudgay, Morris, Rafferty.