The filler should be dry and will be able to be picked off. Its not going to hur the paint at all just like dried mud.