How?: Skip damaged sector instead of getting hooked to it


#1

There is a horrible issue regarding data storage:


This also happens to other kinds of storage.
Nobody so far was able to explain the cause of the SCSI deadlock to me.

How can I, at least, reduce the time a disc drive spends on attempting to read a sector in the case of damage, and make it skip to the next sector?
If unreadable, do not stay hooked! Just skip and deal with it later and unfreeze the program!
For example: LiteOn waits 10 seconds per damaged sector. For 100000 sectors, that would take more than a week of time.
But if this gets reduced to 0 2s, it can save a lot of serious time.


Universal open-source optical disc drive firmwares?
Why exactly does a disc drive freeze the PC when encountering unreadable sectors?
#2

There are several methods of dealing with lots of bad sectors:

  1. The most obvious one is setting the drive’s ‘hardware read retry count’ to 0. Most imaging software is able to do this, and it can also be done manually by issuing the appropriate ‘Mode Select’ SCSI command. There are actually many interesting options that can be set with this command (such as disabling hardware error correction or forcing the drive to return partially corrupted sectors if they couldn’t be read perfectly. You can read the Mt. Fuji standard for all the details about this and more), but very few drives actually support these. Most drives will support setting hardware read retry count though, and it does cause a noticeable improvement, but often not enough. This brings us to:

  2. Read around the bad sectors. The Linux ‘ddrescue’ command is quite good at this. The idea is to keep reading until you run into a bad sector. Once you run into one, skip many sectors ahead (like >10000) and start again. When you reach the end, do the same thing but this time seeking backwards and decrease the skip amount on each iteration. This way you can quickly ‘carve out’ the bad sector regions and get most of the readable data quickly. (I’m actually developing a Linux utility that combines methods 1 and 2 at the moment).

  3. Finally the hackiest method (which only works for CDs), is to fool the drive into reading your data CD as audio. This potentially disables all attempts at ECC correction (depending on the drive and read command used) resulting in zero delay on corrupt sectors. This can be done on many older drives by first inserting an audio CD for the drive’s logical unit to recognize, then pulling the drive’s top cover off and physically swapping the audio CD with the data CD you want to read. Because the CD was not ejected and loaded, the logical unit will believe it’s still reading the audio CD. Note that the ‘sectors’ you get this way will be scrambled and possibly have some drift, but it’s not too difficult to extract the actual data from them (I’ve developed a utility that does this as well). This method is most useful to get a rough idea of what data is present on very bad CDs - you will miss out on many perfectly readable (with ECC correction) sectors.

As for the technical reason for the delay on bad sectors - it’s actually because the drive’s logical unit is trying very hard to reassemble the corrupted sector into the perfect state it once was using several levels of ECC (error correction code). After several iterations of this, the logical unit gives up and returns an error. Now as for why the SCSI standard didn’t provision for some kind of async cancellation command or why the entire data bus must remain in a busy state until the drive either returns the data or an error - I’d love to know that myself. It may have something to do with hardware limitations back in the day.


#3

VERY INTERESTING CLUES, thank you!
I have read all of it, and you seem to be a real expert.
Seriously, this is highly interesting! I really appreciate the time and work you put into your text!.


#4

Just a few questions:
*What software are you developing?
*Can MtFuji read all 7203 bytes from sectors? (the layers above 3234b and 2352b)
*Can MtFuji manually read (directly from rotation engine) and set rotation speeds? (maybe overclocking slimdrive to 32×CAV for CD).
*Can MtFuji enable reading discs DURING acceleration instead of waiting?
*Can MT Fuji enable MtRainer?
*Can MtFuji instruct the disc drive not to hold the disc but let it spin off, if inactive for too long?
*Can MtFuji bypass 05/21/00 LOGiCAL BLOCK ADDRESS OUT OF RANGE? (e.g. recover quick formatted CD-RW with missing TOC)?
*Can MtFuji enable DPM (data position measurement)?
*How can Mount Fuji be implemented by custom firmware?

It would really be good, if I see the status of the disc drive (drive reports rotation speed, current sector address, number of done/left retries/time to read sector, ability to accept SCSi commands from PC such as sector jumping, skipping, etc.) during a struggle with damaged discs instead of waiting while my PC freezes.

But your post was really interesting! Manufacturers must respect power users!


#5

Thanks!

The Mt.Fuji standard is basically the subset of SCSI commands relevant to optical media. It’s a manual for how to talk to a generic CD/DVD/BD logical unit, not how to bypass it. Most of the things you’re looking for would require hardware/firmware modification or knowledge of vendor-specific protocols.

*Can MtFuji read all 7203 bytes from sectors? (the layers above 3234b and 2352b)
No, the most you can get is 2774 bytes (2352 + subchannels (96) + C2 error data (296)) from the READ CD command. For anything deeper than that you’d need custom firmware or a special drive, and to see the raw EFM bits (the ‘7203 bytes’) you’d have to be directly measuring the laser signal.

*Can MtFuji manually read (directly from rotation engine) and set rotation speeds? (maybe overclocking slimdrive to 32×CAV for CD).
No, you can set rotation speeds, but only through the logical unit - not directly.

*Can MtFuji enable reading discs DURING acceleration instead of waiting?
No, this behavior is controlled by the logical unit.

*Can MT Fuji enable MtRainer?
This depends on whether the device supports packet writing.

*Can MtFuji instruct the disc drive not to hold the disc but let it spin off, if inactive for too long?
Yes, most drives support the “Power Condition” mode page, which allows to set spindown timing, with the MODE SELECT command.

Can MtFuji bypass *05/21/00 LOGiCAL BLOCK ADDRESS OUT OF RANGE? (e.g. recover quick formatted CD-RW with missing TOC)?
No, this behavior is controlled by the logical unit - it determines valid LBA ranges when the disc is first inserted. Custom firmware or vendor-specific commands can bypass this, but you can also try the disc swap trick I wrote about earlier to fool the logical unit into believing your wanted LBA range is valid.

*Can MtFuji enable DPM (data position measurement)?
No idea, that’s beyond the scope of this standard.

*How can Mount Fuji be implemented by custom firmware?
I think it’s the other way around - if you’re writing custom firmware it’s up to you to adhere (or not) to this or any standard.

*What software are you developing?
I’m working on CD/DVD recovery/archival software. There’s 2 separate parts. The archival program runs on Windows and it’s main purpose is to separate file data from the metadata (filesystem, sector modes, subchannel, etc) of a CD image. This allows me to de-dupe and store the actual files in various proper locations, and later be able to reassemble the original disc image bit-for-bit as it once was. The program can determine data sectors’ corruption state based on ECC data and merge multiple separate images, keeping the best sectors from each image.
The recovery program is a later addition that runs on Linux (since device control is much easier there). It uses the methods I wrote in my previous post to read discs, and basically feeds data into the archival program.
I’ll eventually publish these as freeware, once they’re stable.


H653B hates SATA adapter? Read request frozen forever
#6

You’re fantastic, ceym6464.
Thank you for putting so much work into your replies. I really appreciate it.
I did not expect that you put so much work into it.


#7

Another question:
Can Mount Fuji force higher quality DAE/DVE?

CDDA, CD-i, VCD and SVCD have inferior error correction.
The disc drive does of course attempt to extract with high quality, but not 100% integrity.

CD-ROM and DVD require always 100% of data integrity.
One faulty byte can likely cause the malfunction of a program/software stores on it.

Can Mount Fuji or any firmware hack enforce a higher data integrity on CDDA/etc.?

I know about cdparanoia, but that is just the PC’s software side.
How can the drive itself try harder for a perfect DAE?


#8

The problem with CDDA/etc is that it’s lacking a whole level of ECC and EDC codes typically present in data CDs. In CDDA, all 2352 data bytes of the sector are userdata, while in Mode 1 and CD-XA Form 1 (typical data CD formats), only 2048 bytes are userdata and the remaining bytes provide an additional level of sync, tracking, EDC, and ECC codes. So for CDDA, the only error correction the drive can perform is by using the ECC bits in the sub-frames making up a sector. I’m not an expert on this, but I think it’s just not possible for the drive to tell whether a sub-frame was reassembled with 100% integrity from this limited ECC information. You can’t get raw access to these sub-frames without custom firmware / vendor-specific tricks but you can request C2 error data to evaluate the quality of the sub-frames, and you can re-read the sector multiple times and compare both the data stream and the C2 error stream for discrepancies. This is probably one of the things that CDDA ripping software does.

Some drives support some error recovery flags which you can set using the MODE SELECT command. They tend to be set to maximum possible error recovery by default though. Look at page 752 of Mt.Fuji v9 here: ftp://ftp.avc-pioneer.com/Mtfuji_9/Spec/FUJI9R100.pdf


#9

Page 752? Has anybody ever read all pages?
That must really be massive!.

Thanks.
You seem as experienced as Peter Van Hove (sole IsoBuster developer).


#10

Just one more question:
Is there a PowerShell/CMD command for Windows to issue that SCSi command?
I have not tried it yet with Linux, but I just don’t know the Windows PowerShell command. Searching for “hardware read retry count” on the internet brought no useable results. An Internet search for “Mode Select SCSi” didn’t either.

I would appreciate your clues.


#11

I haven’t done any research about how to do this in Windows. From what I know by observing other software, there are multiple interfaces available for interacting with optical drives (look in the I/O settings tab of ImgBurn, for example). What you’re looking for is the ability to send raw “packet commands” to the drive. This will let you directly interface with the drive’s logical unit and use all the packet commands listed in the Mt.Fuji standard (“MODE SELECT” is one of those commands). And if you can call DLL functions from PowerShell, it should certainly be possible to send packet commands to the drive (provided you find an interface that supports this).

In Linux, this is rather simple - it only takes 2 lines of code. Just open the drive with open() and you can send packet commands using ioctl (example: ioctl(fd, CDROM_SEND_PACKET, &cgc); where fd is the handle you got from open() and cgc is a properly filled CDROM_GENERIC_COMMAND structure). Reference: https://elixir.free-electrons.com/linux/v3.3/source/include/linux/cdrom.h#L281

By the way, if you haven’t read this yet, http://www.chipchapin.com/CDMedia/ is something you should read if you’re interested in the internals of CD design. After this, you can read the various optical media ECMA standards (http://www.ecma-international.org/publications/standards/Standard.htm), to find out just about everything about the physical design of CDs/DVDs and a lot about the logical design as well. Some questions you posted earlier can be answered by reading these. Unfortunately, plenty of logical design info remains behind paywalls, particularly for DVD and BD).


LaserLock
#12

Excellent and detailed explanation.

Too bad. But I guess, that the links you provided will satisfy my knowledge hunger for a while.


#13

If you want to know what kind of crap is being read when a dvd drive is encountering a “read error” on a dvd, one way is to dump the data before the error correction is done. One method which this might do doable, is what programs like friidump do.

Unfortunately it is very hardware dependent.

I looked at what friidump did and wrote my own code to see whether it could be also done on a 2009-2010 era LiteOn rebadged drive. Surprisingly, it actually worked. (I haven’t looked at it in over 5-6 years).

It turns out for that particular LiteOn drive (probably an ihas124 A rebadge), the random “bad sectors” looked like the drive wasn’t able to “lock on” to the EFMPlus block for a particular sector. The raw data for that “bad sector” looked like pure junk, where it didn’t even have the a proper ID “header” as the first 4 bytes.


#14

Another source for this type of programming, is to see the dvd + css specific code sections in the open source VLC player. Most of it is written using the scsi interface.


#15

IIRC, the way I was able to get the LiteOn drive to dump the raw data after the EFMPlus demodulation but before any error correction was performed, was to issue a scsi command to first read 16 consecutive sectors (forming an ECC block) but the temporary read buffer sent to the generic read command being deliberately too small. Technically this will send back an error.

Somehow the raw uncorrected data for the 16 consectutive sectors is still in the dvdr drive’s cache. So a command which reads the LiteOn drive’s cache is issued, which retrieves this raw uncorrected data. I’ll have to look up exact scsi command which does this.


#16

read buffer 0x3c command

http://wiki.osdev.org/ATAPI

IIRC, one of the friidump methods uses this 0x3c read buffer atapi/scsi command to do that same raw sector dumping.


#17

Awhile back I tried using friidump to do a raw data dump of a single-layer dvd disc. IIRC, it took around 20-30 minutes.

Just for fun, I dragged and dropped that raw data dump file into VLC and surprisingly it actually played the video/audio content (albeit the video was scrambled due to the css encryption which wasn’t removed). This raw data dump file wasn’t mountable like a generic iso filesystem.


#18

If you want to go to an even deeper level before the EFMPlus demodulation is done, I don’t know if that is doable with off-the-shelf dvdr drives via software. I suppose if one knows how to do embedded programming (ie. designing firmware) for Mediatek chips used in current LG and LiteOn dvdr drives, then one will know whether this could be done.

If this can’t be done at all via rewriting the firmware, then the only other way would be to read some voltage levels directly after the lasers reads it as a disc is spinning (such as with an oscillioscope or something which can read+dump these voltage levels in real time).

Somebody actually did something like this at a very basic level using an old junk dvdr player, and wrote a blog post about it.

http://debugmo.de/2007/07/read-your-dvds-the-raw-way/


#19

I have three more links, @Ceym6464:


#20

@Ceym6464 I really liked your contributions.
You even won a rare and limited badge..

Where are you? Hopefully, you will return.