Plextor drives are prone to dropping to 8x speed when extracting audio. My Premium-U, PX-755UF & 3x PX-716s have always behaved this way. It isn't a fault, just that Plextors aren't actually particularly good CD readers and very intolerant of less than perfect/pristine discs. They seem to be programmed to drop the speed at the first sign of difficulty and are very reluctant to increase it again.
Also, many drives are significantly slower when extracting audio securely rather than normally. Which audio extraction software & settings are you using?
Regarding the access times, I haven't tested audio CDs vs data CDs myself. But audio CDs (CD-DA) use a completely different file system/disc format to data CDs. Data CDs have much better error correction, but this requires much more disc space.
Famously, Norio Ohga (vice-president of Sony) demanded that the discs should be able to hold his favourite piece of music - Beethoven's Symphony No. 9 (specifically the 1951 Bayreuth Festival performance) - so its capacity had to be increased from 60 minutes to 74.
To achieve this the physical disc size was increased by 5mm, but this wasn't enough and more capacity had to be found somewhere. So unlike data CDs, audio CDs don't have a checksum to confirm that the data has been read correctly - hence the need for secure ripping. (In any case you wouldn't want a CD player which kept pausing while it went back and re-read the disc, much better to carry on and try to hide any audible errors.) And it is much more difficult for the drive to find the correct location on the disc - which is why drives have different read offsets.
I expect this explains the high access time you are seeing when testing with an audio CD.
In normal use the access time is not important for audio CDs, as they are read linearly with random seeks are only required when skipping tracks.
What does matter is that the drive is able to read the disc consistently. Audio CDs are actually read in small bursts - not in one continuous stream as one would imagine, like the stylus following the groove in a record. Many older drives (including extremely expensive hi-fi equipment) didn't have a consistent read offset, so when the drive started reading the next burst it wasn't actually reading from the intended location on the disc. The result was a small step in the waveform which could be heard as a click. (You can replicate this by deleting a small section of audio using your favourite audio editor.)
So it would make sense for a drive to deliberately opt for slower, but more accurate movements of the optical pickup when an audio CD is detected.