Actually, that is not anywhere near as safe as you think it is. Studies have been done (I'll try to find one later - this has been discussed on this forum before) that show that hands free phone use is just not the same as talking to a passenger.
Some of it is a bit esoteric, but it is a documented thing. When you talk to a passenger, you are still "in that space". When you hands-free with someone, you are mentally "in their space". Your brain puts you in this conversation with a person that you picture being far away, it distracts you, even if you don't think it does.
Secondly, the passenger is there with you, and they will also be seeing things that you see, and you, and get feedback, so when a questionable situation arises, when your face suddenly gets focused on something outside the norm, they automatically, subconsciously, pause their speech until the coast is clear. But with someone on the other end, the driver is also (probably subconsciously), trying to keep the conversation going, as they know the other person is still listening unaware of what's going on on the other end. It distracts the driver.
I've seen it with my wife using the phone in the house. She's deep in conversation with her sister, and she is just far less aware of her surroundings as she moves from room to room. You can tell she is "in that space" conversing with her sister, rather than "in her space" noticing the rug is crumpled and she might trip on it.