Why sadly? This is basic consumer protection stuff. It seems like currently this fee ends up being paid by the vendor or some middleman, but that seems like BS too. Visa/whoever should be returning the fee and folding it into their business model as the cost of doing business.
Even if you do everything you can to prevent it, you still incur refunds. It is simply impossible to get everything right 100% of the time - not in the least because people keep buying the wrong items. Things like cancelled or modified orders happen all the time, and you can't really do anything to prevent those.
And that sounds like the cost of doing business to me. For especially egregious cases the companies should reserve the right to charge customers for being stupid and wasteful.
Where I live, that expectation is enshirined in a law created in 2001. The reasoning is, with remote buying you can't physically inspect the item. Hence you can decide to return an item at no cost, provided you have only handled it the way you would have been allowed in a physical store.
I believe the law was originally created due to massive over-promising of results by tele-marketers.
(I live in the Netherlands, but similar laws are common in Europe I believe)