We did what JG didn't: built a Texas "ranchette". Didn't find it particularly messy, burdensome or annoying, but it wasn't inexpensive.
We are probably not typical in that we built our dream/retirement house 20 miles away and lived there for my last 7 years in the working world, until the mortgage was paid off. Yes, it did take two months longer than we thought it would to complete construction (8 mos. vs. a planned 6 mos.).
Ten years ago we bought 5 acres of raw land, thinking that we would build our retirement home just before I retired, planned for 9 years in the future. The more we planned and dreamed about moving to our dream house in the country, the more impatent we got to just do it. The local RE market improved and, two years after we purchased the property, we sold the house, signed a one year lease, and started building.
Building your dream house via long distance is a challenge I've not experienced. We were close enough to visit the building site almost daily and keep an eye on every aspect of construction. (Yes, the builder and I got to know each other very, very well.
) But all said, it wasn't a bad experience and I wouldn't hesitate to do it again if I needed to.