I think it's a combination of genetics and diet/exercise/eating habits. My dad has struggled with weight issues on and off his whole life. When he really puts his mind to it, he can lose 20 or 30 pounds just like that. As soon as he gets it off, he starts snacking again, and he puts it all back on. He's a grazer. He is constantly in the fridge, eating little bits here and there. If's there's chips in the cupboard or peanuts, he's snacking on them. He also eats really fast - never takes his time. He tries so hard to lose the weight, but it's always a struggle, and he feels like he's starving whenever he's on a diet.
I am one of those naturally thin people. My mom is skinny too, and so are my sisters. As a little kid, I was so skinny and lanky, it embarrassed me. I used to pull my socks up to my knees thinking it would make me look like I had a little meat on my bones, when it really made me look even skinnier!
Now that I am older, I guess you could say I look "normal", and if I gain a few pounds, I really don't have too much trouble getting rid of them within a few days....just cut back on the fatty foods, go for a few hikes, do some yard work, drink a few less beers and wahlah - gone.
I don't think you can blame it all on genetics, but there's no doubt in my mind that genetics play a huge role.