You are not overreacting and we are working with Dexter to break him out of that habit. We adopted him at the shelter, and we know one of his previous owners taught him this, whether it was on purpose or not is beside the point. So positive reinforcement, working with the dog, and we are slowly breaking him out of it.
We noticed this early in our relationship with our adopted dog, and it was not just black people. Now he even allows some of my neighbors to pet him. I call that PROGRESS.
But in the defense of the dog, chihuahuas have a napoleon complex and are very territorial. So at times, he growls just to do that. He is a shelter dog, and four months on he has gotten much, much better.