Welcome to CD Freaks
Most pen drives are very slow at writing small files due to the nature of Flash storage they use. For example, my Dan Elec 16GB pen drive is rated at 14MB/s max sequential write, but when I timed how long it took to copy 428 documents (12MB) to the drive, it took 54 seconds, i.e. 8 files per second.
With 20,000 files and assuming your flash drive has similar performance to this one I tried, this would likely take ~20 minutes for the transfers (based on 14MB/s) plus a further ~40 minutes for creating the individual files (based on 8 files per second). So if the copy process appears to be taking less than an hour, it's quite possible that there's still a substantial cache in memory of files waiting to be written to the pen drive.
I'm not sure if the Mac caches write operations to USB drives, but you can easily check by looking at the LED on the pen drive. On most drives, this is blinking or lit while activity is taking place, so if it's still lit or flashing after the copy process appears to have completed, wait until it goes out and remains off before unplugging it.
If you have a disk surface scanning utility or have access to a PC to run HD Tune on, another thing worth trying is a full surface scan of the pen drive to make sure there are no bad blocks on it.