I'm sorry for your loss. It sounds like to me that the handicap is more prevalent on the male side of your family and not the female side. If he had fragile-x syndrome, that's something a person is born with; it doesn't develop--you can't catch it obviously. However, you may have the gene in you to pass one with someone else so I would look into genetic testing. According to the National Fragile-X Foundation, it is an inherited process so someone could be a carrier & not have any symptoms. Females can also be affected by the Fragile-X gene but it's more prevalent in males.
Again, I am very sorry for your loss--God Bless you and your family!