As I look at your photos, I see that your forearm stop is out quite a ways. Do you have enough weight in the butt so the rifle isn't "light" on the rear bag?
* * *
On to opinion: I think you shoot groups, and not score, right? I've always found that a bag setup for score where you have to move the rife (more than one bull) to be much fussier. Several posts with Al Nyhus on this topic you could look at.
But if it's group, I'd look elsewhere. As Wilbur says so often, a *good* rifle ignores this sort of issue. If you have another barrel for the rifle, and if it too shows ho-hum performance, I'd first suspect the striker assembly, which includes everything -- shroud, sear fore & aft, etc., etc. But as Wayne Campbell has been through that, I'd rule that out & things get pretty subtle. You found issues with the bolt & stock. Grasping at straws, how about the trigger & stock? Is the barrel tight enough?
Try some really different things. If you're shooting a high-ogive bullet, try a Fowler. Try jumping each kind of bullet .020 or so.
Instead of N-133, try something like 24-25 grains of H-4198. With both powders, try a hot primer, like a CCI mag, and a very mild primer, like a Wolf. If, with all these changes you're still 2&1 (or just ho-hum performance), I'd think you could rule out ammunition.
At which point (different barrel, ammunition), and assuming you've tried another rest set up, & different scope, I'd sell the rifle to a varmint hunter, because the odds are you'll never find the problem.
FWIW -- Guessing at what might be the problem is a whole lot less likely to find it than just going out & doing the work.