I like to compare it to wearing a seatbelt in that it's not so much a case of "how many years I've been driving and not needed one" as it is "being prepared should the worst ever happen". For me anyway, a no-brainer in that it has the upside of protecting my lenses to a degree, and ZERO downside. For sure there are occasions where it makes ghosting or flare worse, but I see those situations a mile off and simply remove the filter on those occasions - so it's not like I have to choose between image quality or front element protection -- I just run with both.
In practice, removing a filter in extreme contrast situations doesn't eliminate ghosting and flare - it just reduces it - so it usually still needs to be dealt to in PP anyway. Plus, after shooting in salty air, it's a darn site easier to run a filter under the tap than it is a lens
In my opinion people get far far far far far too hung up over this theoretical "loss of image quality" theorem; I know that for many photography goes hand in hand with perfectionism (borderline excessive/compulsive in some cases) (and I'm serious about that - and yes, I consider myself one of them) - but I've had to learn that in the "real world" we just end up making life difficult for ourselves if we don't learn to weigh the factors that do make a difference -v- those that don't.