I would answer the phone and simply ask him why he is calling your wife (pretty much in those words). Then I would tell him that if he continues to call you will report him to the police as a crank caller and at that point you can also see about stalking charges since it is a repetitive situation.

There is no need of physical violence in this matter but lay it out there that his calls are not appreciated by either you or your wife and are an intrusion in your life...sleep...whatever.