You will be committing odometer fraud. Your car will be flagged for it the next time it's registered or inspected depending on the state where you live.
The odometer is built into the speedometer. If you can swap the speedometers out, and all the parts that you want to keep on the manual cluster (or buy a manual face kit with a needle puller and transfer everything else), then your car's mileage will stay correct.
Removing the odometer cluster from the speedometer will break it. It's designed to break if tampered with.
There's a
DIY on it, but since you've done it before, you probably don't need it.