Exactly and working against that fear is the important part, throwing out people that are problematic is a good step, you can simultaneously work on better integration, like Germany did with the overhaul of the citizenship program.
Efficient integration also means reducing the number of immigrants in certain areas that have too many, moving them somewhere else is better as they don't have the ability to build subsocieties
And of course its not about the numbers itself, its about the number of immigrants that shouldn't be here due to crimes, there was almost a decade of the issue being effectively ignored and the voters show that, every other day is a headline about a immigrant doing crimes and being known beforehand to the authoritys, its a actual issue, this issue causes xenophobia.
And im not in favor of a "upper limit" or throwing out people that integrate. Its idiotic to do that and doesn't help at all, its even counterproductive.
And of course, fighting the root of the problem would help a lot but its often not possible or too complex, or in some cases counterproductive "look how much money we spend on those [slurs] while people here have it so bad" -right people fishing for voters (efficiently due to our current economic situation)