After eliminating all possible causes – NLB, SharePoint site configuration, IIS security and settings – it turned out that it wasn’t even IIS- or SharePoint-related issue at all. Starting with Windows Server 2003 SP1 and higher (Windows Server 2008 and R2 editions in that list as well), as a security measure Microsoft introduced a loopback check to prevent man-in-the-middle (MITM) attack, when a malicious application (such as spyware) can try to eavesdrop communication with a remote server by introducing itself locally as a remote host. Please note: loopback check happens only when host headers do not match local computer name.

The symptoms and solutions are described in Microsoft KB article: http://support.microsoft.com/kb/896861
Additionally a few other related issues (accessing network shares, etc) are outlined in two more KB articles: http://support.microsoft.com/kb/887993 and http://support.microsoft.com/kb/926642.

To deal with this issue you have two options: either explicitly specify all host headers in the registry (the most secure, but also the most cumbersome solution), or disable loopback check entirely.

If you decide to opt for completely disabling loopback check (on a development or test server), here is one command line you can achieve it through. Please remember to restart your server after changing the registry!

REG ADD HKLM\SYSTEM\CurrentControlSet\Control\Lsa /v DisableLoopbackCheck /t REG_DWORD /d 1