I live in Michigan, historical home of the United States auto industry and so NOT a place that has very good public transportation anywhere, even in the bigger cities. So driving is really not something you can easily opt out of here.
I had to take some extra drivers education classes after I accidentally side-swiped a parallel parked car during a lesson. it was at the end of a zig-zag in the road that I misjudged.
Besides that, I have mostly been able to avoid getting in accidents, except for a couple of times when it was snowing heavily as it sometimes tends to in Michigan winter.
I do have a hard time dealing with driving when conditions are not the best. One example is bad weather such as mentioned above-snow, ice, heavy rain and so on. Construction zones can be very stressful for me, and just last weekend caused me to miss out on an event I wanted to attend in a city about an hour from where I live and work.
Roundabouts are horrible. One time, I pulled up to one and the crossing traffic was not even slowing down. Then some macho jackass pulls up behind me and starts honking and swearing. What was I supposed to do??
In general, other drivers and their unpredictability can stress me out. Either they are going to slow and I get restless, or they start tailgating me and I feel bullied. On American-style expressways, I have a hard time judging the timing when someone is merging onto the expressway from an on-ramp. Sometimes I try to slow down to let them in, and they do not speed up enough and so I have to either speed up and try to pass them, or keep slowing down which then can result in drivers behind me having to slow down and possibly get upset with me.
Making a left turn out of a parking lot, if there is not a traffic light, is something I always try to avoid, as it is easy to get stuck waiting for both directions of traffic to be clear and I am not always good at judging if I can make it or not.
Also, any kind of obscure situations stress me out. For example, not too long ago I was on a non-expressway divided road. I needed to turn left onto a crossing street, and pulled up to a red light. I am pretty sure that it would have been illegal for me to go straight ahead through the red, so I waited for the light to turn green. Before it turned green, people behind me who I assume wanted to turn left onto the opposite direction of the road I was trying to turn off of started honking. It may be that I could have gone straight ahead through the red, but I did not want to take a chance. And then I was angry and felt insulted and bullied by the other drivers honking at me.
A good deal of the time, I can drive where I need to with no major problems, but there are just enough of the above examples and others that I also do not always look forward to having to drive.