Is it Mind Over Matter?
Jennifer, On getting rid of your old clothes, a friend of mine who had RNY 7 yrs a go told me, "its like a weight hanging over you to keep them. You feel like you are wasting that money by not using the clothes and it will be a huge weight off your mind when you get rid of them."
I took it with a grain of salt, and started to go through my closet, intending to only get rid of the stuff I didn't like, and by the end Id cleared out over 2/3rd of my closet. It was a hugely freeing experience!!! Go on, give it a try, start small (or big as in donate some of those largest of your sizes, and see where you end up!