My wife and I don't have kids and don't plan on having any. My 15 year old niece is spending the summer with us (not my idea!) and after this experience I am more convinced than ever that I do not want kids. Goodness, I have a new found respect for her parents and their ability not to lose it. Maybe it's her "goth" style, maybe it's the loud "metal" music all day, maybe it's her having an opinion on everything (especially if she knows she can pi$$ me off), I don't know, it makes me want to get a vasectomy.
But, although our decision not to have kids was not based on financial considerations, we are fully aware that it will have a financial impact on our ability to retire early (in our 40's most likely).
So how many kids should you have? Well, only you and your wife can answer that question.