The biggest problem with Internet connectivity is that TV manufacturers keep creating a new system every year that is not backwards compatible with older TVs, so I'm not surprised that customers are noticing this and choosing to buy basic TV models without all the Internet features.
An Internet based set-top box can provide a lot more features than what would come built-in to a high-end TV and tend to be supported for several years. If the set-top box does become obsolete, it's just a matter of getting a new box rather than replacing the entire TV.
As for smaller TVs, many people are now using Tablets and laptops as a second TV, so this would explain the drop in sales of smaller size TVs.
Finally, I don't think 4K is going to do as much of an impact as going from standard definition to HD. The difference between HD and 4K would only be noticeable when standing right in front of the TV. The main problem at present is that many HD channels have reduced the bit-rate to the point where the picture quality is barely better than DVD quality.