Just sayin'.....
It sounds as if your barrel is getting fouled more quickly than your other barrels causing a decrease in accuracy. Based on your explanation, that's about all it could be. I've had maybe two barrels that wouldn't make it through a single 5 shot group (about 10 shots with sighters). It seemed that those barrels shot extremely well until they quickly gave up the ghost and threw a shot. The barrel will give you a good target now and then but consistency is what you want...what we all want.
My other barrels just wouldn't shoot well.....entirely!
Save up some money and have another barrel installed. The new barrel may be worse than the one you have but it's worth trying. Alternatively, you can fix the whole deal by selling the rifle and trying another....maybe....