I worry about dust with hand tools as well though I have had no medical issues form woodworking so far. I can offer some personal observations.
1. I bought one of those Dylos particulate laser devices and had it on prior to hand tool woodworking and then kept it on and did different types of hand tool woodworking operations. The particulates went up a little bit for chisel and hand plane work, more so for sawing, and much more so for using some 220 grit sandpaper by hand. My persona l conclusion was I was ok with common hand tool woodworking with maybe and exception for using sand paper. Mostly I don’t use sand paper. During nice months outside, I also keep the garage doors and side door open for a cross breeze. Mostly because I like the breeze in the shop but I figure it can’t hurt.
2. After a year of having a shop dedicated heating and air conditioning unit, I had it serviced. There was no dust to speak of on the filter for the unit.
3. I have a shop vac with a cyclone unit that I use daily in addition to a broom and dust pan. In the unit, I have a bad inside the shop vac to collect the shavings as well as a “HEPA” filter. After two years of use, I decided to change out the bag and filter (the 5 gallon bucket had been emptied frequently). As best I could tell, the bag had less than 5% of its volume taken up by shavings and the “HEPA” filter had no observable dust on it (I used sunlight like described in the original post) to look for it.
4. I’ve done deep cleans of the garage and really don’t see any fine dust strewn about anywhere.

As such, I feel fairly comfortable about the relative risk factor of fine dust coming from hand tools. It’s not zero but I think it is very low.

I am in the process of saving for a machine tool. I will spend a significant sum of money to get a dust collection system that might seem like overkill but I want to protect my lungs. In the 1980s, I refused to change the brake pads on our family cars and my dad called me names for worrying about the asbestos dust. He didn’t use dust collection in has garage workshop. Now in his 80s, he wheezes a fair bit and is part of a class action law suit for asbestos. I am sure I am doing something unsafe in my life but I am trying to avoid know risks where I can (e.g., don’t drink, if I use sand paper I wear a dust mask and really air out the garage afterwards, etc,).