What he said...
I have polished many of my chambers to a mirror finish with 1,000 grit "Wet or Dry" and oil, and there has never been a problem.
Funny about how this wives tale goes - I have heard it both ways, "Never polish a chamber, it'll cause the brass to stick to the walls", and "Never...