You are certainly not being any safer. The act of engaging in a phone call is what is distracting, not holding a device. Don't kid yourself.
http://www.usnews.com/news/articles/2013/06/12/aaa-study-usi...
http://mentalhealth.about.com/library/sci/0701/blcellphone70...
http://www.alertdriving.com/home/fleet-alert-magazine/north-...
http://www.mindthesciencegap.org/2013/03/14/look-no-hands-is...
Further, nobody is talking about people who use these technologies in private, they are talking about people who are using them in public.