My threshold of annoyance was finally crossed this evening and I decided I'd better figure out why this was occurring. Most resources I found online seemed to point in the direction of ensuring "UseDNS no" has been set in /etc/ssh/sshd_config, unfortunately this was not the problem in my case.
After digging around a bit longer, I bumped into a helpful thread on the Ubuntu forums that pointed out the actual problem was likely with PAM.
Making the recommended change by commenting out the following PAM connector in /etc/pam.d/common-session did the trick.
# session optional pam_ck_connector.so nox11