I've lost friends and an instructor, and have struggled with this too. The answer for me has been in the specifics, in the continuous reevaluation of my personal minimums. Mine increased after an engine failure with my wife and 7 year old on board. That ended with a successful landing at an airport, thanks in part to superior piloting skills

and dumb luck. Here are some of the things I added to my minimums after that:
I gave up night flight. I still keep night current, just so I can do it if I need to, but I no longer plan trips that will begin or end in total darkness.
I follow roads through the mountains whenever I can unless there’s a better option for handling an engine out situation (e.g, significantly less time over hostile terrain). That’s not possible for much of the best flying we’re talking about here, but it’s something you can do to add to your options on long cross countries.
I fly higher than I used to. Altitude = options. This is different than the way I was taught to fly through the passes and down in the drainages, and not nearly as fun, but, as long as the weather allows it, being higher when transitioning over rugged terrain adds to the safety margin. I still get to do the canyon part on approach and departure.
I will do VFR on top only if the layer is thin and there’s at least 1,000 of clear air and hospitable terrain below. Otherwise, I fly around the area of coverage.
When departing an airport surrounded by hostile terrain, I circle for altitude before turning on course. One extra climbing 360 gets me another thousand or so feet of altitude. If something goes wrong, I have more time and options and a better chance of returning to the airport.
When departing my home airport, I don’t take the standard departure that immediately goes over a large body of water at low altitude. Instead, I fly down the shore and then follow a bridge with nearby levees across the water.
I bought a pair of nomex pants and a nomex shirt. When the new engine was finally installed, I flew 15 hours on it by myself, in VFR conditions around airports before I took anyone else in the plane, wearing the nomex the whole time. I don’t wear the nomex now unless it’s for a flight review, when I know we’ll be pushing things. I would like to have my family similarly outfitted for backcountry adventures, but the message that sends to them isn’t exactly positive. So I settle for telling them not to wear any synthetics and to go for long pants and sleeves unless it going to be hot. When my son and I fly to Alaska (hopefully next year), we’ll be wearing the Nomex.
Finally, out of consideration for my wife, I plan our trips for early morning hours in order to minimize time spent in turbulence. She was terrified by the engine out episode, and I’m lucky that she still flies at all, so that seems like a reasonable concession. I don’t like getting knocked around that much myself.
These are in addition to a whole host of other minimums, like not pushing the envelope on high DA operations when I can wait out the heat and wind, minimum risk flights immediately after maintenance, checking the engine at the beginning of the summer in addition to the annual, avoiding high traffic areas and events etc.
Some of the new self-imposed restrictions make it harder to get the most out of limited weekends and vacation time. Giving up night flight is the biggest hit in terms of utility. Following roads on long cross country trips takes more time. Overall, though, I am happy with the new balance. We still go places and I still love to fly. We’ve been all around the West and had a great trip down to Baja back in March, so it’s not a total straight jacket.
YMMV. My minimums are my own. I'm not offering them here as any kind of gospel, or suggesting that anyone who doesn't agree with them isn't smart or safe. If there's something useful in them for you, then good. I expect that some of the 135 guys will laugh or shake their heads at my minimums. That's ok. I have the luxury of being an amateur and see no reason not to take advantage of the flexibility that I'm fortunate enough to have in order to tip the odds in my favor a bit.
There's no shame in admitting to experiencing some fear. Lord knows I've had my moments. The answer for me has been in the specifics. So, what are your minimums? Are there things you do or have done in the past that you can modify or skip entirely without giving up on this amazing activity? There are so many great resources for learning this stuff-instructors, NTSB reports, personal misadventures, hangar flying etc. I had no idea when I took my first lesson that it would be so important, and so much fun. Good luck with it all.
CAVU