The only thing you can do then is listen to your gut and follow your instincts.
If you cannot cut them off from your life, you must act smart. Setting boundaries is a must with people like them.
If someone is coming in the way of your happiness and wellbeing then it is OK to limit those contacts.