If you look at the science of selective breeding, it is indeed possible to breed humans with certain goals in mind. It's like breeding dogs for specific outcomes. As you said, it doesn't always produce the desired outcome, but it does often enough to be repeatable.
If I'm not mistaken, selective breeding was sometimes practiced with slaves in the southern U.S. states during the period of slavery. Of course, research on that subject these days would likely be greatly frowned upon, if the information was even still available.
Selective breeding of humans would be a lot more predictable with highly developed genetic mapping as a starting point. Actually, there are certain genetic diseases that could be eliminated by looking at the genetic predispositions and mutation markers before creating children, but that is perhaps a different subject.
The question of right or wrong doesn't have all that much to do with the science, but much to do with the question of if it should be done.