Due to the huge number of irrelevent replies to this thread recently, these have been moved to a separate thread in the CD Freaks Living Room.
To help answer this question, from what I would imagine, the only way to tell if a DVD has been copied is if the rental DVD has software that reports where it is running from back to a central server (such as the rental company). For example, some DVDs have included DVD playback software such as "PC Friendly" discs, etc. If this is the case and the included software reports details about the disc, then if someone tries to copy the disc contents to a recordable disc and plays it back, the software could report that disc type back to the central server (while playing from a recordable). As a result they would know the IP address that is using a copied disc.
However, if anyone plays a disc using their own playback software (PowerDVD, WinDVD, etc.), does not allow the disc to launch any of its bundled software and copies it without the bundled software, then there is no easy way that the rental company can tell if the disc is copied.