You can take a bus from Victoria to various locations on the mainland, including YVR (Vancouver International Airport). The bus drives on the ferry. At YVR, you can get on the Canada Line and travel into the city. So you could explore both cities without renting a car at all. Of course, with a car you will have more freedom to explore.
I have never stayed at a B&B in Vancouver but there are many charming ones. There are many hotels to choose from and the best choice depends on what you like to do and your budget. My personal favourite is the Westin Bayshore, which is on the shoreline at Coal Harbour, right next to Stanley Park. A different, more historic choice would be the Sylvia Hotel on English Bay. If I am catching an early morning flight, I like the Fairmont at the airport. Good areas to stay would be Coal Harbour, the West End, Kitsilano or downtown. Keep away from the Downtown East Side.
I hope kcowan will chime in as he lives in Vancouver half the time.