Left and right should be dead nuts, sweep it around with a .0005" or better indicator and adjust accordingly for left and right. For the tail stock height you can try by tightening down the tail stock more. This will generally pull it down closer.
When adjusting for left and right (if yours has the set screws on each side) make sure the tail stock is not locked down. If it's locked down this will impede correct adjusting.
Mine sits .003" high unlocked, when I crank it down good and tight, it's only .001" high, which I can live with using a floating pusher. Left and right is dead on.
Are you drilling and then pre-boring, proper fitting bushing?, chambering all the way with the finish reamer??
Are you cutting the tennon and threading before or after chambering? My take is, the easiest place to knock your set-up off is when cutting the tennon and threading because of tool pressure. Are you dial checking your work as you go??
In order to better help you find out "why"? We need to know exactly how you went about chambering said barrel from beginning to end.
It's all about rigidity and checking your work as you go.
If you can chamber by the numbers and are not indexing the muzzle high or low, I would cut the tennon to final dimension and thread last.