Research has indeed shown that one becomes a much worse driver when trying to do a telephone conversation, a but the same effect is much less pronounced when having a conversation with a passenger. The passenger is able to read the traffic situation and won't demand immediate answers to tricky questions while there traffic requires full attention. I'm not aware of any research on driving and audio books.

Does it actually matters weather one is a more clever driver when not diverting attention to an audio book? I think not, for simple driving 100% concentration is simply not needed - and if the audio book keeps one from falling asleep due to boredom, it may even be safer. When arriving in some unknown city and full concentration is needed, it's always possible to press the pause button.