The reason this has implication in clothing is that the shirt you are wearing has an "anti colour" which is influencing the way your skin looks. If your shirt color is throwing off a flattering halo, you will look good. If it is throwing off an unflattering halo, your dark circles and blemishes will actually be enhanced!
When I worked at Dairy Queen in the 70's, I had to wear an orange-colored uniform. The supervisor wouldn't stop going on and on about how sick I looked. I think she might have had other problems than acute observation! Conversely, in my closet I have a favourite blue shirt that I wear and I invariably get compliments on, even if I'm not wearing any makeup. There is definitely something to the colours we wear and how we are perceived.
[I love how many fields such as philosophy, science, and business are being given the graphic novel treatment. So I did a preliminary sketch of a panel illustrating this topic.]