I experienced the problem recording the speaker output (Stereo Mix) with Vista 64 and my Gigabyte GA-EP45-UD3P Motherboard. It has the Realtek ALC889A onboard audio. What happened is that the standard Microsoft drivers for your audio have the Stereo Mix disabled (because, of course, we must make sure that you cannot infringe copyright even if that renders your PC inoperable). Nothing you do will get them back unless you load the Realtek drivers for your audio.
I found the drivers for mine at:
Go to the Gigabyte website and put your motherboard number in the search box and then Search For -> Driver.
Install the driver and you should get another speaker icon in the tray for the Realtek HD Audio Manager. Right click then select Audio Devices. In the Recording Tab you should now have CD Audio and Stereo Mix. Enable those.
In Audacity, Edit -> Preferences, Audio I/O Tab, Recording Device dropdown, select Stereo Mix. Play other tracks while recording new one can be checked but make sure that Software Playthrough is unchecked. You will now be able to record the audio output of your PC in Audacity.
IFAIK, this will work for any audio chipset. Just determine your exact soundcard or motherboard model, go to the manufacturer's website and download the most current AUDIO drivers for your Vista flavor.