It's bad enough when you can see the phone in someone's hand. The loud 'private' conversations in public places (are we going back to the public transit again?).
This happened in my vicinity this past week as I waited for the bus. A woman also at the bus stop was having a heated argument with someone on the phone. All we could hear was her very rude side of the conversation; no one wanted to look her in the eyes either for fear of being similarly attacked or because we were embarrassed on her behalf for the rude public display.
When the phone is invisible, it can be downright unnerving. If you're on the wrong side of the person with a Bluetooth device or on any side of someone with one of those 'phones' that looks like they're just listening to a music device, their audible utterances seem like they might be directed to you. I, for one, have responded on many an occasion and gotten a glare back at me.
If you don't want me to listen to your conversation, maybe even insert myself unwittingly into it, then don't have your conversation in public.