Don,
I haven't done this specific job, but have worked with a lot of drillings in the past. I have cut the shotgun barrels off, bored out the action stubs and sleeved new barrels in place in drillings and have also built insert barrels for them. Lots of work.
You should be able to do this, but the safest way is to use a barrel insert which contains all the pressure of the cartridge on its own. You can open up the inside of the shotgun barrel to make more room for barrel thickness if you wish, but you might have enough without. I used to run hoop stress calculations before I did this kind of thing and you will find that barrels don't have to be too thick to be safe at normal pressures - most are highly overbuilt to prevent idiots from blowing up guns. I believe (from fading memory) that .030" wall thickness or so at 12 ga diameter will hold 20,000 psi or so, making a barrel with walls of .200 thick should be able to hold most high power rifle cartridges with a reasonable margin of error, but run your own calculations and talk to some other engineers before you take the plunge.
Couple of things - you might be able to get an insert barrel already made by folks in Europe. You might not be able to get 260 Rem, but they may have something close enough (6.5 x 55). The Europeans are way ahead of us on using such devices and they have already worked out the technology as to how to fit these to regular shotgun chamber/barrels so you can remove or install the barrel as suits your fancy. Also, don't do this with a cheap shotgun. My experience with the low price point guns is that they are made of rather soft steel and they will stretch and peen when stressed. A 260 Remington cartridge will stress the action some, as it operates at 4 times the pressure of a 12 ga shell, but it has less base area, so back thrust is not 4 times as much. Still, hardened hinge pins/trunnions and a Greener cross bolt or other action bolting system of merit is advisable. Browning, Beretta, Ruger, yes - Most guns built in Brazil, Turkey, Russia or China - No. You may have to reduce firing pin diameter.
Anyway, these kinds of projects are a lot of fun. Just make sure you are doing things safely. I would be happy to hear what you decide to do.
Scott