You are absolutely right. Consider what you have accomplished in the sport.
I have a number of jumps and a few skydiving credentials to go behind it. So if I were to add my skydiving experience, licensing and credentials - no matter how unrelated to the position I am seeking, it shows diversification, individualism, goal oriented, the list can go on. Positive attributes.
However, if I just started jumping, limited experience, perhaps that stereo-typing of "immature, death-wish" thing you referred to may pop up. Or the employer may think I will be too preoccupied with my newly found skydiving hobby and simply be too distracting while at work to either myself or my work mates.
Something to consider.