Even if did spy, in theory it could only spy on what is watched through the Android device. For example, it cannot tell what channel you're watching, what buttons you push on the TV remote, whether you mute or change channel during ad-breaks, etc.
Although our Sony TV has smart features, we probably only watched a single programme since the TV was purchased around 4 years ago. Pretty much everything we watch on demand is on a laptop connected by Chromecast. I'm not sure what Chromecast's privacy is, such as whether it captures the URLs we visit or what programmes are natively cast through it, but I'm sure it doesn't have access to how we use the TV and remote.