For heating your home, heating with fuel oil or natural gas is almost without exception, cheaper than heating with base board electric. Why? Because you're converting the fuel oil or natural gas, to heat, right at the source where you need it. The infrastructure is already in place to fill the oil tank or pipe the natural gas right into your home. With the electric heat, the fossil fuel is turned into electricity at a generating station, run through wires for many miles where considerable loss takes place, then is turned into heat by heating high resistance wire or coils using high voltage. No wonder it's less efficient and more expensive.
That's not a given. It's highly dependent on where you live. My old house had home heating oil, and one of the best things I did was ditch it for an all electric heat pump. Overall, I figured it saved me about $1,000 per year, plus no need for a maintenance contract anymore, which had been around $300/yr when I converted in 2008, and probably would have been over $400 by the time I sold. Plus, I got the added benefit of having central air conditioning, although admittedly it was a bit weak upstairs, so I still ran one window unit. But, that's the risk you take when you retrofit an old house that never had it, compared to building it new with the system integrated.
There was also no natural gas available in that neighborhood. Distribution of home heating oil is also dependent on trucks, which bring it out to your house. If you live in an area where it's common, that's not an issue, but I have the feeling that the further south you go, the less common it is.
My current house has a hybrid system. It's dual zone. The upstairs unit is pure electric heat pump. The downstairs unit is heat pump, but it has an oil backup that kicks in when temps get below 40.
Heat pumps get less and less efficient, the colder it gets, and I think at 32 degrees they're totally useless, and have to go to some kind of backup, whether it's pure electric, or oil/natural gas/propane/etc. Right around 32 degrees, the electric probably isn't THAT inefficient, but the further you dip below freezing, the less efficient it becomes. So if you're in a climate where it regularly gets brutally cold, electric backup doesn't make sense.
Where I live, it might get down to single digits every once in awhile at night and before sunrise. I think it might have gotten down to below zero maybe once, in the past 20 years.
Things may have changed in recent years, but all-electric heat pumps used to be pretty cheap to install. So even if they weren't so efficient, you saw the savings up front, as natural gas and oil furnaces/boilers were traditionally pricier, from the get go.
So coincidentally with a house, just like an automobile, often a hybrid is the way to go, vs all electric, or all fossil fuel. It just depends on your own situation.