I was going to suggest considering a 205 as well. It's a pretty good airplane for what you seem to want.
The biggest downside of a 182 is the somewhat fragile nose gear. That heavy engine out front makes nose gear landings relatively common, especially if too much airspeed is carried into the landing. Just a little crow-hopping can create thousands of dollars of damage, not only to the firewall but also to the tunnel behind it. Keeping the airspeed to 1.3 Vso or less can make a huge difference.
I suggest that you should fly the airplanes you're considering. The difference in feel is pretty dramatic, compared to what you're accustomed to.
A piece of advice I've always given, though: buy the airplane that fits your actual mission 90% of the time (or some other high percentage--90% is my target). The vast majority of GA airplanes aren't flown full, but actually with only the pilot or the pilot and one passenger. For the other 10% of the time, consider renting one that fits. That's why I'm super happy with my 63 P172D, my steed for the past almost 16 years. It fits my mission most of the time--me and dog, or me and dog and camping equipment is the norm. When I had a family when the kids were young, the 182 fit the bill most of the time, as we did a lot of family traveling. So analyze your actual mission most of the time, and go from there.
Cary