Sydney, which is spittin' distance away and actually closer to the Victoria airport than Victoria, is a charming little town with a great outdoor market on (I think) Thursdays.
A couple of summers ago, I took a lesson in a seaplane (a 172 on straight floats) with Pat Bay Air--terrific fun, most I've had in an airplane in a long time. Because we were staying on Marilyn's brother's boat at Sydney, the instructor picked me up at the marina and then dropped me at the airport, as our flight out was later that afternoon.
My lesson included normal docking, plow taxiing, step taxiing, and ramping, and half a dozen landings and take-offs--instructor never touched the controls through any of that, just talked me through things--I'd never even been in a seaplane before, though I have gobs of boating experience to go with my flying background. One of the landings and subsequent take-offs was under power lines at Montague Bay--very new experience for me--"Come in at 100', stay at 100' after take-off--no worries". Yeah, right. Maximum altitude was 500' MSL/AGL (same thing)--and a real blast. I knew the area really well after boating there several times, but as always, perspective from the air is different. Since it was in a 172, the only significant flying difference was that it was slow due to all that drag from the floats, and chopping power brought it down fast. But what a blast!
The marinas at both Victoria and Sydney are interesting, if you like boats (I do). There's a neat little water taxi at Victoria that putts you around the marina and is worth the ride.
Personally Burchart Gardens doesn't appeal to me, but many people like it. There are nice pubs and restaurants near both the Victoria marina and the Sydney marina.
You'll have a good time, without much effort.
Cary