If we evidence a criminal damage then we would act. But can’t usually. You say upsetting to endure, consider for a second, how upsetting it would be, being uprooted every few days. That’s why we always act in the best interests of all parties with proportionality and legality.
You can’t say both these things:
(1) It’s the travellers’ lifestyle choice to constantly move about and we should respect that that’s how they prefer to live.
(2) It’s “upsetting” for travellers to constantly have to move about from one place to another.