Usually, out of everyone's followers, only a small number are actually engaged followers. That's the norm. Appreciate the ones who appreciate you, and enjoy the larger follower numbers the ghost ones give you. :)