We'll be doing a similar trip, but starting the end of May. A few days in Port Angeles, almost a week in Sidney, and a week on Lopez Island in the San Juans. We'll probably take a day trip to Vancouver while we're in Sidney. From Lopez we'll then take the ferry to Anacortes, and visit Whidbey Island enroute to Seattle. After that we'll play it by ear, maybe stop off in Portland on our way back home.
I appreciate the suggestions posted here, you're giving me ideas!
If you're looking for afternoon tea in Victoria, the White Heather Tea Room is very nice. It's away from the city center in a more residential area. Thanks for posting the yelp list -- we might try the Abkhazi Garden teahouse this time around, it was closed the last time we were up there.
We'll definitely spend a day at Butchart Gardens, it's one of our favoritest places.
(I took this the last time we were there, in early June 2014):