With my 6 PPC Farley with a Hart barrel and my 30 BR Shilen DGA with a 3-groove Lilja, the first shot is pretty much right on. Cleaning and Lock-Ease don't seem to make much difference. I clean between targets only for something to do and to make cleanup at home easier.
Hunting rifles, of which I have many, vary widely. Most are close enough to POA, clean or fouled, hot or cold, that I don't worry about it. One Weatherby Vanguard .30-06 in particular seems to throw the first shot left. It certainly has not cost me any animals but someday I'll get around to thoroughly testing it to look for a pattern.
I think fouling is one factor and heat is another and each has its own effect. Is point of impact shift the result of fouling build up or is it due to heat warping the barrel a bit? You could sort that out by letting the barrel go totally cold between strings while leaving it fouled.
I read once, in this forum I think, of an old hand telling a guy with a finicky Weatherby magnum to run a box of shells through it as fast as he could. After that, it shot consistently well. It had something to do with the heat aligning the molecules in the steel. Sounds extreme, but if you've tried everything else, why not?