I've got a particular brand of CD wallets which leave imprints on the recording side of the discs. The imprints come from the plastic padding material on the inside of each pocket. These can be removed but only if cleaned with lighter fluid (e.g. zippo or ronsonol - liquid not the can sort). Nothing else works. You might have to press a little bit more firmly when cleaning the disc, but it does work.
You might want to give this a try and see if it does anything. And if it does remove those spots, turn a hard, accusing eye at those disc wallets of yours and never buy that brand again. Not all disc wallets are like that. I've got some case logic cases which have been holding discs in them for a year plus. Zero degredation issues, spots or marking from the sleeves.
And in case you were worried, lighter fuel is safe on recording surfaces (provided you don't smoke while cleaning your discs - I had a mechanic who used to fuel up my car while talking with a lighted cigarette at the corner of his mouth). I have cleaned many discs that way with no side effects at all.