Try some T.S.P., Trisodium Phosphate. It's available at most hardware and/or home improvement stores. It's a powdered cleaner that you mix with water, sort of similar to 'Spic-n-Span'. Just use it with a broom or scrub brush, let it sit for a few minutes, then hose it off. That should take care of your problem.
I've used it on driveways, garage floors, and a whole bunch of other odds and ends....especially before painting stuff.
The name makes it sound kind of nasty, but it's not.