Well, the risk aspect has already been touched on. Insurance, etc.
Then there's money, but if you can control yourself and are already living within your means it's not that big of a deal.
My opinion, and my situation - time is the biggest factor. For me, the DZ is 1.5 hours away. That may not seem like much but that turns into 3 hours of only DRIVING for that day. Then, it's usually an hour wait just to get on a load at my dz. So there's 4 hours of my day gone right there for just one jump. And I don't want to do 1 jump if I go out there, I want to do 10.
But it all depends on you and your family situation. I want to get out there and swoop my comp velo. But I have no business flying that canopy if I can only get in a couple jumps a month, which isn't worth it to me.
If your family's cool with you spending each weekend or a day a week at the dz then go for it. But there are much more important things in life than skydiving. Now that I have a young family I don't know how some people spent every weekend with kids