No, most of the places I notice my aversion to yellow light have pretty neutral colours; off-whites, greys, the occasional green. I haven't looked into it, but I would say the yellow lights (at least the ones that bother me) distort the colours more. I have no idea when the Colour Rendering Index of them are, just that they seem much hasher. Maybe yellow lights are more tolerable to most people, so they install more? Yellow lights aren't rated for higher power though. Maybe it's purely the colour temperature, maybe it has to do with sun exposure and cloudy days?
Just the CRI wouldn't explain why some people find the addition of artificial light unnecessary at best while others seem to need it to see. There's probably some amount learned perception going on, but I don't think you can learn to be blind to certain light, can you?