Transformers are not built with our knowledge of language. That's a gross approximation -- it would honestly be more accurate to say they're modelled after the human brain than that they're built with our understanding of language. A big problem is that the connection between AI and language is poorly understood -- we can't even understand what the word2vec axes are.