I don't like the rubbery feeling of my skin; I don't know why it should feel different under my fingers when it has no sensation, but it does.

Wow - so it does! My guess is when you touch your mouth with your fingers, both fingers and mouth send the sensation back to your brain. When only the fingers can feel, it's interpreted differently.