In all seriousness, part of the reason is that European surnames were gradually invented as descriptives of the person or family they named. People called Brown were tan, brown-haired, etc I.e John Brown literally was "That brown guy John". White also meant hair color or complexion, while Green usually meant a location where the person lived, like Joe of the Green Hill would become eventually Joe Green, or Joe Hill etc. White also could mean someone from Wight, in England.
Gray and Black, however, weren't colors at all despite the coincidence. Most evidence points to Black being an altered form of Blake, which itself came from a version of someone who was "by the lake" (an extremely abbreviated timeline here) and Gray was a bastardized version of someone "Of the Cray".
Finally, a lot of name origins were confused by the habitual "americanization" of ethnic names on immigrants to the US. frequently names were arbitrarily assigned which only barely resembled the original version. Black is one like this, as there aren't many instances of it before the 1890's when immigration from overseas started to really jump.