NO. In fact the reason you don't notice it until the car is warm is because you'll only see it when the revs are low. When the revs are low, so little fuel is being delivered that even a slight change in fuel will alter the idle drastically (if regular idle consumes 1mL/minute, and you drop that to 0.5mL/min due to a clogged filter, that's a huge % change). In contrast, when you're at WOT, you're delivering a pile of fuel, so a small change in the amount delivered isn't going to appear noticeable (if you drop from a regular 20mL/min at WOT to 19.5mL/min, you ain't gonna notice ). When you first start the car, the revs will be up higher. Maybe 1200-1600, depending on how cold it is where you live. You won't see it then. When the idle drops down to 800 (where it's supposed to sit when warm), you'll start to notice it dropping off a bit.

If you're REALLY in touch with your car, you might be able to feel it at WOT or higher revs, but I couldn't feel it when I had the same problem.

You really have nothing to lose by checking it out. Even if it's not badly clogged, a new fuel filter is cheap as heck, and it'll improve your gas mileage a bit to boot.

(BTW, those numbers for fuel consumption may or may not be accurate, just used them for example. Too lazy to actually calculate real fuel consumption. It's easy to do, though.)