Of course. They'll simply feel foolish for trying-- like speaking to a wall, I believe the saying goes.

[Then he shrugs with both arms.]

It's the best way to move about unbothered.