Latics were pushed all the way by the Forest youngsters – managed by former Latics chief Warren Joyce – with both sides knowing victory would keep them on the Wembley trail.
And after the game finished goalless after 90 minutes, three saves out of three by Tom Watson in the shoot-out – in addition to successful spot-kicks from Silko Thomas, Matt Smith and Will Aimson – saw Maloney’s men through.
In reality, all of the excitement was saved for the end of the game, with most of the game being a tough watch as Latics dominated possession but once again found it difficult to translate into chances in the final third.
“We’ve been involved in so many games like that…it’s job done, but the second half certainly felt like an Under-21 game,” said the Latics boss.
“I actually thought we got better as the game went on, the young ones who came on, I was pleased with what they did.
“The way they defended, they went man for man, meant our movement and rotation had to be quick and tense, and for sure we didn’t get that in the first half.
“We had a big chance with Joe (Hugill) – it was probably the biggest chance we had in the game – and that unfortunately didn’t go in.
“In fairness, we changed things for the second half, we really tried to emphasise how much movement we needed. And probably the longer the game went on, the better it went for us.
“It was great to get Paul Dummett on the field for his full debut, and K’Marni (Miller) showed when he came on what he brings.
“He’s still got so much to work on but you can see the potential he has.
“And I’m really happy for Maleace (Asamoah)…for him to get his first 90 minutes will really stand him in good stead.”
Forest had an early sight of goal, with Adam Berry – wearing the No.99 shirt – seeing a stinging strike tipped over the top by Watson.
At the other end, a defensive error allowed Hugill to race in on goal, one on one with goalkeeper Aaron Bott.
The on-loan Manchester United man slipped the ball past Bott, but with nowhere near enough power and a defender got back to easily clear.
But that was about it in terms of entertainment, save a late free-kick from Chris Sze – back in the fold after an absence of a couple of months – that cleared the bar by a few inches.
And with Watson following up successful spot-kicks from Thomas, Smith and Aimson – to deny Jack Nadin, James Sinclair and Kyle McAdam – Latics booked their place in the knockout stages of the competition, with the next round scheduled for December.