TIL about wezterm's "adjust_window_size_when_changing_font_size" option you can set to false and get the behavior I've wanted all terminals to have forever:
@fasterthanlime Kitty does that as well, I don't think I had to set any configuration options for it to do so. (Agreed it's the sensible behavior)
iTerm2 has the "general -> window -> adjust window when changing font size" check box, checked by default but if you uncheck it you get the sane behavior.