I think that you have a good point.. but if you explain why you distance yourself, then they are more likely not to feel abandoned or betrayed, they are more likely to understand in the long run. If you don't explain first then I agree they may feel betrayed.