Three possibilities I can think of. The first, and most likely, is heat. Something could be off in terms of the fit of the barrel, action, or stock. As the "system" warms, things don't move evenly. It could be the fit is fine and it is just one of those barrels, but the 5-inch, three shot groups suggest more than that. Checking/fixing this sort of thing is a matter for a good gunsmith.
The second most likely possibility is something is just wrong with the scope or scope mounting. Are things tight? Can you swap out to another scope & mounts? You say you had another scope on it. Did that show the same pattern? Have you checked the mounts -- bases & rings?
The third possibility would be the twist rate, which I believe is 1-10 for your Savage. I have a factory-rebarreled Savage in .25/06 I used for Factory Class in 1,000 yard benchrest. I used 110 grain Fowlers (a flat base bullet, no longer available). It shot quite well. A Berger VLD is probably a longer bullet, and may not be stable in the 1-10 at the temperature/elevation you're shooting at.
A fourth, slim possibility is that your rifle doesn't like the particular bullet/powder combination, but you're talking about 5-inch groups at 100 yards. That is larger than the usual out-of-tune, or "doesn't like" scenario.
If you have any shorter bullets to hand -- lighter is the usual claim, but gyroscopic stability is more a factor of bullet length than weight -- give them a try. I doubt that will work, and most likely you'll need a good gunsmith.
If, by process of elimination, it does turn out to be just the barrel, I'd rebarrel it with a custom barrel from any of the advertisers on this forum -- Hart, Lilja, Pac-Nor, Shilen, etc. When I had my Savage rebarreled by the factory, it cost more than using a custom barrel. I had to do this to stay legal for Factory Class, but that's not one of your goals, I assume.
The .25/06 is a great chambering for long-range shooting. Way under appreciated, I'd say. And now that Randy Robinett is making benchrest-quality .25 bullets, (as well as the Bergers), it is fully benchrest-capable for 1,000 yard use. Of course, it's capabilities for hunting have long been appreciated.
Good luck to you