You are right the clearance is tight and keep in mind it changes as the install progresses. I cut down fasteners to go the threaded stud route but gave it a try with the stock fasteners when I noticed they seemed happy to remain in their positions on the thruster body while positioning it. My neighbor helped me on the thruster side as the wife wiggled the propeller to help the shear pin enter the yoke. Overall, I'd say we got lucky at several spots; realize folks have posted regarding boat yards flat refusing this task!
If you go the stud route, keep in mind if the stud is too long it will contact the body of the thruster prior to the nut tightening the thruster to the other piece. I also kept in mind the fasteners are not a snug fit in the thruster, the point being the thruster could have a tendency to wobble on the studs losing the yoke to sear pin interface; perhaps folks that have gone the stud way can weigh in on this aspect.
In setting up for the stud method there were some bedeviling subtitles; I could tell how far the original fasteners had threaded into the hole but couldn't figure out what the stud would do when I threaded it in. Is the tapped hole a bottoming tapped hole? A through tapped hole? Would the stud resume turning when I screwed a nut down on it? I like using nylock nuts and felt pretty sure if it could turn it would turn when the nylock part of the nut hit.
So a poster put up about using LockTight and I decided I'd do that and cut the stud length to having 3 to 3.5 threads in the transom part which I could tell was the thread depth of the original fasteners and made sure it had the same amount of thread sticking out the end the nut would fasten to. While it's acknowledged a nut half threaded is rated at full strength, my personal preference is to end up with a bit of the stud sticking out so that's the length I shot for.
Also, keep in mind when you use LockTight, even LockTight blue instead of LockTight red you're pretty much through, you're not going to get back apart what you've used it on without heat and you won't find me waving a blue flame around in a fiberglass enclosed area that I cant even see into when my arm is down there!
In hindsight, I'd build a nice little cradle for the thruster so it might stay in place after it's shoved against the transom and all you have to do is rotate it to get the holes to line up with the fasteners. I also very much agree with your thought of using hex head bolts as that solves not only the clearance problem but also a problem I ran into; that of the initial finger turning of the bolt, much easier with hex head than the smooth round outside of an Allen head bolt!
Also, I had to finger tighten the Allen head bolt sufficiently so the wrench would "ratchet" on the back stroke and not just turn the bolt both ways. And of the two wrenches I ended up with, one was much much more resistant to ratcheting than the other one. Hex bolts eliminate this worry completely, of course.
So I take a little pride in doing something boat yards won't do and I'm a little sad because now that I"m good at it I'll never do it again! Whew, just the telling of the tale gets me going, I gotta go lie down. Hope this helps, rich