I kind of thought that's what you did. Thank you for confirming this.
Ultimately, the image still has to look "natural", otherwise our brain will suggest there is something wrong with the shot, Blur is one of those things that has to be done extremely subtlety or it will look wrong (something I learned from my own mistakes).
I suspect that you might recognize the same issue in this shot I took quite some time ago... The blur is all in the same plane, so I incorrectly thought I might get away with it.