No, to the best of my knoweldge you don't have to take out the arm rest, but I think the bolsters from the recaros will rub the armrest, and make it no fun to put the seat belts in. Not sure on that one, but the armrest replacement for the JDM ITR is pretty cheap any way. As far as the five point harnesses go, you have a couple options, either use the rear mounting points for the the back seat belts under the seats, which is probably your best and cheapest bet, or buy a harness guide bar, or lastly, invest in a roll cage.

Good luck,
Aj