Offsets handling (syncing of audio data vs. Q channel)

Seems that Andre Wiethoff (EAC) found a way to determine the exact quantity of samples that his Plextor Ultraplex 23TS reading was deviating from the position where 1st audio sample "should" be.

It's said that his experiment was based on reading after the program area, inside lead-out (both areas as seen by his drive, not the real ones, BTW, not every drive can do that) and looking for the end of back noise of the last track of a CD with analog recordings. That is the beginning of the AccurateRip project, that uses reference CDs to help users to find the offsets of their particular drives. "Stream is Accurate" feature in the drive is a must.

http://users.pandora.be/satcp/eacoffsets02.htm#-
http://accuraterip.com

I would like to know what you think.

Yes, but EAC doesn't publish its offset database, as far as I am aware, or its definition of what 'where 1st audio sample "should" be' means, according to them. Perhaps they use this particular Plextor Ultraplex 23TS as the "one and only" reference drive, with offset 0?

At about the same time I started this thread here raising the subchannel/audiochannel synchronisation problem, I did the same at the EAC forum. I didn't get an answer there, unfortunately.

Sounds reasonable. I would appreciate a description of his exact method so it could be held up to scrutiny. "It's said" and "...was based on..." doesn't cut it for me, I'm afraid - I have a job is in the high-tech software engineering field, and I've learned the hard way that specification and documentation of techniques and methodology are not 'nice to have', but essential when it comes to assessing technical issues.

It is an interesting project; it can surely help to assure full audio extraction accuracy (relative to the offset problem). One small nitpick is that they seem to make no attempt to also do perfect extraction of subchannel information. This will not interest most people of course, but then I'm not most people in this respect :-)

However, for all the interestingness, I only see windows executables there (I don't do windows). I need access to the source code to assess its technical merits. Furthermore, they should open up the description of their database format; it seems like you need their particular software to access it.

Best regards,

Sidney

Drive offsets are in http://www.accuraterip.com/driveoffsets.htm. I don't know if Andre determined a quantity of physical bytes between 1st valid audio byte and control byte (I think not, at least not very directly and must say I'm not sure there's a unique standard count), but he discovered how to make any accurate stream (and lead in/out overread) drive to find such 1st audio byte and that's at least good enough to extract precisely what TOC is supposed to point at. There's no reference drive but reference CD pressings, Plextor Ultraplex 32TS has a sample offset of +679.


Actually, they use verifying calculations, at least to compare extractions too.


After all, their main objective are audio tracks, not accessory information, anyway EAC reads ISRC, CD-TEXT and MCN (not sure of this last) too. It's good enough for non CD+G or CD+MIDI Audio CDs.


The ofsset is given in samples (L+R, 4 bytes, 32 bits).

It is my understanding that it is not exactly clear where the TOC is pointing; the TOC can be related to the subchannel frame sectors (98 frames) easily enough, but subchannels and audio are not so straightforward to match up. I think I'd like to compare notes with the EAC author, is he easily reached (e.g. on the EAC forum)?
Ok. From the list, I see both my plextors are at +30, which equals 5 frames of audio (there are 6 samples in a frame). This then means (from a hardware point of view) that I should delay the audio relative to the subchannel in such a way, that the first sample, left channel, most significant byte of audio data corresponding to a given sector (as determined by the subchannel) is found in the first audio byte contained in frame 5 of that sector (where the first frame of the sector, the one having the SYNC0 EFM pattern, is numbered as 'frame 0').

Unless, of course, I shift the data in the wrong direction
Well, that's the drive database. It's good this is in the open (although a definition of what this number means would be helpful).

What I am thinking about is the song database, i.e., the checksums of perfect extractions. That's quite an asset; in fact, one of the long-term goals of my hardware cd hacking project is to produce something similar. I would like to be able to access the data they collected.

Best regards, Sidney

Hi,

Maybe this is a bit offtopic, but, does anyone know why PlexTools agrees with EAC as to what is the "reference" offset for extraction?

Did they choose the same offset just to follow EAC's user base and avoid confusion? Or did they have other reasons?

Just curious.

Regards.

These are the words of Andre (EAC):
Words from Olli (CCD):

For what it's worth...

I am involved with the mastering process at huge CD/DVD factories all over the world. Our software allows the mastering engineer to set the skew (offset between main and subchannel) to whatever they like (including negative!). And, the Red Book does allow +/- 75 sectors.

Thanks a lot RichMan for that field based confirmation. So, we're close to a concensus.

What we would just love to know is...

What is in the perfect case (skew=0), the exact physical byte quantity between the 1st byte of a subcode block and the related 1st audio sample, and if there's a congruence with EAC, AR and Plextools offset correction.

When feeding the 2352 bytes of a sector to the CIRC encoder, it will interleave them with another bytes in such a way, that will use space from 3 sections of 98 channel frames when being held on a CD.

Our CD will have a sequence of such sections, addressed according program duration, adjacent to 2 extra sections that still hold user bytes.

What are the addresses of those 2 sections (PG and/or LO)?

@reddish

To answer the first post in this thread (I think)...

First, the subcode is NOT interleaved like the audio data. One subcode byte is added to the mix after the CIRC interleaving of audio data.

If you are capturing the raw EFM (HF) signal and your software can find the SYNC and de-interleave the data, then you have what you are looking for. Once you find the subcode SYNC pattern, the audio data that lies beneath will be the audio data that goes with that decoded subcode frame.

What I mean to say is this:

If you de-interleave the raw EFM and then find the first subcode SYNC and then decode the next 97 EFM frames to find that the Q channel indicates 00:02:00...then the first byte of audio that was coupled with that first EFM frame is the first byte of audio for 00:02:00.

The de-interleaving process is not 'drive' dependent. There is only one way that the audio is interleaved and there is only one way for it to be de-interleaved (refer to Red Book). If your software de-interleaves correctly, then whatever A-Time that the Q channel says you are reading is indeed the correct audio sample that goes with that A-time.

Where drives come into play is in the delay of their circuitry for the subcode stream and the audio stream to the final output. You have bypassed this if you are checking the raw EFM data from the laser pickup so there is no reason at all to consider any particular drive in this issue. Drive 'skew' is a totally different subject.

When you decode the Q bit of subcode and find it to be 00:02:00, then rest assured that the first sample of audio that goes with that starting frame (that forms 00:02:00) is the first audio sample for that particular A-Time.

There is no possible way to be more precise. What is on the CD is the 'bible' if you will, right or wrong. None of us can decide if this is what was intended or not. Only the recording studio can set us straight.

I hope this makes sense. If not, feel free to argue/ask...

Rich

Hi. Sorry about the slow reply (my laptop broke down).

Correct.

After EFM decoding, I have 1 byte of subchannel data, 24 bytes of audio data, 4 bytes of C1 data, and 4 bytes of C2 data (per 588 channel-bit frame). I can also de-interleave the audio data just fine, but for one thing: it is not clear how the audio data relates to the subcode channel; since the 24 bytes of audio data (as found in one 588 channel-bit frame) spreads out over an extent of 106 frames due to de-interleaving, there is no clear connection between which data audio corresponds to which frame address (as can be found by considering the Q channel).

Please define "lies beneath". This definition is exactly what is lacking.

It is the elusive "coupling" that you speak of, that I am looking for. Considering the fact that the 24 audio bytes as found in the SYNC0 frame are spread out over 106 frames, this begs the question of how to join them, in an absolute sense.

Yes. But de-interleaving concerns only the audio data. What is drive-dependent is the matching of sector (or frame) addresses (as determined by the Q channel) and audio data.

My software de-interleaves correctly. It does so on the audio data only. I am thus able to extract the audio stream perfectly. It is the next step (determining the correspondence between the audio data and the Q-channel addresses) that is the problem.

You are quite wrong here - it is precisely the same subject. In effect, I have engineered my own digital CD readout system, and I have to make the exact same choice that any drive maker faces: how do I match the audio-data to the subchannel data? Drive manufacturers make different choices here, and I am wondering what is the "canonically right" choice.

There are 24 bytes of audio data in the first frame of the 00:02:00 sector, They belong to 24 different frames in the time-domain, and actually transgress sector boundaries. Allow me to give a small exposition to once again explain the problem.

Let's call the sector you refer to (00:02:00 abs.time) "sector zero". It can be recognized by looking at the Q subchannel, decoding 98 consecutive frames starting with the special SYNC0 and SYNC1 EFM patterns; this gives 16 bytes of Q channel data, and thus an A-time stamp (the first two frames contain the SYNC1 and SYNC2 EFM patterns and contribute no subchannel data). The Q subchannel data thus identifies the first sector, and the first frame.

Let's call the first (SYNC0) frame of this sector zero "frame zero".

"frame zero" looks like this:

24 bit sync, 3 bits merge, 14 bits EFM-CODED SUBCHANNEL DATA, 3 bits merge, 32* (14 bits EFM-CODED AUDIO/CRC, 3 bits merge).

After throwing away the 24-bit SYNC and merge bits, and doing EFM decoding we are left with the following:

1 byte SUBCHANNEL DATA (in this case, the EFM pattern is 00100000000001 or 'SYNC0' so we don't encode any actual subchannel information)
24 bytes worth of AUDIO data
8 bytes worth of ERROR CORRECTION data (for the audio data).

When decoding, the AUDIO data found in the 24 bytes of frame zero spread out over 106 frames. This means that the audio data corresponding to the time frame covered by the first frame (which is the first 1/75/98 = 1/7350 seconds of audio data of the CD - 6 samples worth of data) comes from other frames surrounding (or preceding, or following) our "frame zero". Now my question is precisely this: relative to "frame zero", where do the 24 bytes comprising the first 6 samples of the CD come from, exactly?

I know that it must look like this:

L0 = to_signed(frame[OFFSET-108][ 0]*256 + frame[OFFSET-105][ 1])
L2 = to_signed(frame[OFFSET-100][ 2]*256 + frame[OFFSET- 97][ 3])
L4 = to_signed(frame[OFFSET- 92][ 4]*256 + frame[OFFSET- 89][ 5])

R0 = to_signed(frame[OFFSET- 84][ 6]*256 + frame[OFFSET- 81][ 7])
R2 = to_signed(frame[OFFSET- 76][ 8]*256 + frame[OFFSET- 73][ 9])
R4 = to_signed(frame[OFFSET- 68][10]*256 + frame[OFFSET- 65][11])

L1 = to_signed(frame[OFFSET- 46][16]*256 + frame[OFFSET- 43][17])
L3 = to_signed(frame[OFFSET- 38][18]*256 + frame[OFFSET- 35][19])
L5 = to_signed(frame[OFFSET- 30][20]*256 + frame[OFFSET- 27][21])

R1 = to_signed(frame[OFFSET- 22][22]*256 + frame[OFFSET- 19][23])
R3 = to_signed(frame[OFFSET- 14][24]*256 + frame[OFFSET- 11][25])
R5 = to_signed(frame[OFFSET- 6][26]*256 + frame[OFFSET- 3][27])

Here,

* Lx and Rx denote the left/right samples corresponding to the first 1/7350'th second of the CD;
* frame[i][j] is the j'th byte of audio data (0<=j<=23) in the i'th frame (i=0 --> frame zero)
* OFFSET is a rather arbitrary integer that I am looking for.

Not true. The Red Book should specify the number OFFSET, but it doesn't seem to do that.

I see what's on the CD. I need to know how to interpret it. For this I need the number OFFSET.

Thanks for the feedback. I feel that you know your stuff but perhaps you missed my point about the one degree of freedom (the OFFSET number) that is seemingly not specified in the standard, and that gives rise to the woes of drive skew (which is nothing more or less than an inability of drive manufacturers to agree on the correspondence between the Q subchannel data and the audio data). I hope that my new attempt at explaining the issue is clearer.

Best regards,

Sidney

Oops, I mean (0<=j<=11) or (16<=j<=27), here.

(And where I wrote 'CRC', I meant 'CIRC').

Regards, Sidney

I managed to miss this post, this is very interesting...! Can the mastering engineer set the offset with a granularity of 'sectors' or 'frames' ? I would suspect the latter?

Then, could you quote the relevant section from the RedBook please? Call me pedantic (you wouldn't be the first ;-)) - if it allows +/- 75 sectors, it should also say +/- 75 sectors relative to what.

Regards, Sidney

Are you talking about a data CD here? I don't understand what you're saying.

Regards, Sidney

The skew setting during mastering is in sectors (not sure why).

After skimming through the Red Book again, it seems the +/- 75 sectors is in reference to the skew between TOC track start times and actual track start times in the program area (Q-channel change points). Sorry for the mistake here.

reddish, I have lots of resources where I work and I will seek the answers to your questions.

I would have to say the first samples on the CD are generated by the encoder. But these samples are at 00:00:00 so it should not matter. Our encoder will normally generate the first 2 seconds (00:00:00 to 00:02:00) since Red Book says these sectors must not contain user data. Since they should be empty anyway, we generate them. I have never seen where authoring studios place audio before 00:02:00 since it would be a violation of spec.

If you already have the de-interleave function, then once you de-interleave the data, you DO know where each byte of audio came from.

If you decode the raw stream and find the starting point (sync0) of frame 00:02:00 and then you de-interleave enough times to get a complete set of 24 audio bytes that go together, you will then have the audio samples that go with 00:02:00.

Sorry if I'm not explaining it clearly. It's really a tough topic and I am in no way an expert on this. If you find flaws in my replies, just point them out and I'll check into it (I'm still learning too).

Rich

I'm not sure if I'm correct here, but I'd like to point out that on many of my audio CDs, when reading sector 0 (00:02:00) on my LiteOn drive and many others they miss out a small portion of audio, which i suspect that those parts are before 00:02:00 or that the drive manufacturers are making a skew offset reading decision. I maybe wrong here, because I don't have a Plextor Premium or later to test out on with it's so called 0 offset skew as reported by many people on the net. The reason why I know that there is missing audio is because I can reading into the pregap (before 00:02:00) and there exists the missing audio.

You patiently shown how 24 bytes fed to CIRC are spread along 106 channel frames.

A consecuence is that a whole sector will end up using 203 channel frames, 196 of them will form two sections, while the another 7 will be part of another section (before or after those 2), or be divided in 2 channel frame sequences that will belong to another 2 sections (one before the first, and the other after the second). Trascendence is spiral must have at least N + 2 to 3 channel frame sections to hold N sectors (it will hold 3 to 4 incomplete sectors, have at least 2 to 3 pointless addresses at spiral's end, if more, even have sectors without an existant address).

If we refer to sectors at both ends of program, it's easy to see at least 105 channel frames outside program area will have to hold user data. Philips must allow PG or LO or both to hold such data, if there's a restriction for one of them, degree of freedom for main to sub relationship shortens, so we are in a smaller range to "canonical reference".

I would have to agree the OFFSET is 108 from all the data that has been presented.
It certainly is the most logical choice and richman says " Our encoder will normally generate the first 2 seconds (00:00:00 to 00:02:00)".

So that would mean to me when you get to first frame at 00:02:00 then "L0" (left channel 0th sample MSB) should be the first byte in the payload of that frame since you dont want the audio data to spill over into frames before 00:02:00. It would also mean that the first few frames starting at 00:02:00 are going to have to have dummy audio bytes to fill in the rest of there 24 byte payload. Maybe this is the reason for having the pregap area is to 'clear out' the hardware so that when you reach the start of audio data the decoding buffers will be in a determinate state.

Assuming that OFFSET is 108 would also mean that the final sector would have to have some trailing sectors (lead-out?) to be able to reconstruct all the audio in the last frame. I guess this is implied since the first interval of pregaps(that follow leadin) and postgaps/leadouts that follow data and audio tracks are supposed to have the same format for the Q subchannel in regards to the control field and Q-mode(as mentioned in ecma130) to the previous track ,i.e. when you reach the end of a audio ,data track or leadin you can read into the gap area to reconstruct whats in the final sector for that track.

Since the CDrom format is built on top of audio, could you try reading a cdrom and
triggering your reading setup to a certain known sector and comparing the 'raw' 2352
data that you read from your computer to what you read with your hardware? You said
you trigger to a paticular sound on the CD so you should be able to do the same
with the 'raw' 2352 sector data which is basically audio samples. If you can reconstruct
it with OFFSET=108 then you have the answer. It boils down to where are the audio samples
scattered about relative to the frames inside one sector.



And regarding the topic of 'track offsets', since an audio sector timeframe is determined
by the Q subchannel and that the subchannel data is not subjected to massive interleaving and error correction like the 24 byte payload then you dont have 100% assurance that it will be correct all the time. Maybe this accounts for the 'offsets' people are seeing.
Assuming that a cd drive waits till Absolute time 00:02:00 before decoding audio samples,
I could see that if the Q channel was corrupted in say the first few sectors starting at
or before 00:02:00 and told the drive the absolute time was 00:01:xx then all of sudden the
time is 00:02:0x and consistent, then the drive will have to make some choices about what
audio data to send.

reddish:
I have to admit this is one of the most interesting threads brought up in a forum and
has presented a lot of interesting topics. I hope you spend quite a bit of time to present
your project to the world in the same precise and informative manner that you present information and questions in your posts.

Hmm. I think not quite - this can't be for all CDs, also tests were made on clean brand new audio CDs and reading at 1x speed! Some how I don't think it's due to corruption.

Maybe this sheds some light to the track offset problem:
/Quote/<richman>
The skew setting during mastering is in sectors (not sure why)... After skimming through the Red Book again, it seems the +/- 75 sectors is in reference to the skew between TOC track start times and actual track start times in the program area (Q-channel change points). Sorry for the mistake here. /End Quote/

in ecma103 it talks about the Q channel in the leadin containing information on the contents of the disk, refering to the track time data as 'items':
"An item indicates the address of the beginning of the user data of an Information Track. It is expressed in absolute time with an accuracy of ± 1 s."
and again: "Thus, the table of contents points to the start of an Information Track
on the disk in terms of the absolute time in the Control bytes with an accuracy of ± 1 s."

That would explain the +/- 75 sectors. I just want more clarification on what exactly
is modified with this skew setting. Does it mean that audio sectors will remain in there
normal starting position after pregap but the Q channel time stamps will be 'skewed' relative to all the sectors? Would be nice to see some example or diagram to explain this.

And why is this large skew setting in the specification? Was it to compensate for the hardware (when CD was being developed) or to allow some last minute adjustment to a recording during mastering to move the silence to the front of the tracks or the end?

What about this?
|
|
v

Where is this picture from - and what value for the OFFSET (as per the previously stated, precise definition) can be derived from it, you think?

Regards, Sidney

It's Red Book alternative, 1st sector byte MUST be 4th frame 1st byte.

Does the "total mess" thing just mean that not every drive can return C2 pointers, or that we have "offset like" issues on C2 pointers too?

Thanks.