Are you using the TV for audio? Or using a home theater receiver? I hope it's the latter, as you're missing out on a lot of good sound otherwise.
The answer is that you also need to have a switch for the audio if your TV has only one input. Most people use a home theater receiver with multiple audio inputs. Many also use a receiver with multiple video inputs, which gives you the switching too. Many newer receivers also feature HDMI switching, and even up-conversion of analog video to HDMI.