The answer depends entirely upon what you're trying to do with those filters.

For example, I've only used colored filters when shooting black and white film in order to, say, enhance clouds.