There are a lot of little things to shooting off a bench that might benefit.
Making sure that the forearm doesn't have slop in the front bag. Ditto for the butt and the rear. Your rear bag should be super stable. If it is rounded on the bottom and is rocking around, weird things can happen.
While I'm thinking about it, a 3" wide, flat forearm "rider" in the matching 3" front bag can be very helpful. Something like in the following link. That one is for ARs.
http://www.midwayusa.com/product/645901/egw-bag-rider-ar-15-fits-free-float-tubes
You have removed the sling swivels, or made sure that they aren't in the party, right?
A 308 has a lot of recoil compared to what we normally think of as a benchrest rifle. I assume that you're ripping the pistol grip and shouldering the rifle. Getting those to be consistent is a big deal.
What kind of optics are you using? A little slop in aiming can grow groups.
How are you cleaning the barrel? You might look up how benchrest folks clean. Bore guide and a one-piece rod are essentials.
If you do get to the point that you want to invest more, you might get a benchrest gunsmith to install a custom barrel. That will be somewhat pricey, but it will be a huge bang for the buck, nonetheless. Get it cut with a custom chamber. You might look into a heavier barrel profile to tame some of that recoil. You might even look into a different cartridge without as much recoil. If you do, I humbly suggest the 6 BR or, perhaps, a 30 BR. They each have advantages.
If you get that barrel cut with a custom chamber, I'd suggest switching to a Harrell's, Jones, Whidden, et al custom *Full Length* die that closely matches your chamber. You'll reap all the benefits of the neck sizing that you're doing now in your factory chamber, without the disadvantages. There is few things as pleasant as reloading with "perfectly" matched chamber and dies.
OK, that's enough for now. Have fun. That's a good shooting 308 you have there.