Stress and doing the wrong thing..
Take a deep breath and count to ten. I also have anxiety disorders. I am one for massive panic attacks. Like Necee said, your fear is getting in your way. Don't let your fear over come you. I know one thing that helps me is to STOP thinking about it. You can worry yourself into a panic attack. It's hard to do, but try your hardest to think happy calming thoughts. While your getting remember how far you've got, and think positive thoughts ONLY! I'm like you, I got off my meds and working through it all. Don't let it win, you have control over yourself, you've proved that.
You sound like me being a hermit. I also don't have visitors because of what my house looks like. If your house is clean, don't worry about it. He's a person just like you. He doesn't have any expectations of you. Remember he's just there for a visit, not to move in.