You're right that there are lots of daily activities where risks are unavoidable, things like driving, contractors, family gatherings, etc. I don't worry about them because they are part of everyday life, and as long as I do my due diligence (e.g. having car insurance, making sure contractors are bonded, etc.), I am fine living with these risks.
My concerns are with "discretionary" things where I have more control over managing/eliminating unnecessary risks. These risks can be easily eliminated without impact to my quality of life. So for example, lending bikes or power tools, refereeing a sparring session, etc, are "things" I can easily do without.
Thankfully I don't sit around worry about these thing too much (although I did a lot of risk management during my corporate career so I have a fairly active imagination)
, but I figure if there are specific actions I can take to eliminate these risks without impact to my quality of life, I'll happily do them.