I have spent some time on this specific issue, and I do not think the evidence is clear. I could send you a fair amount of material on this (and would be happy to do so if you would like), but for now this may be sufficient to demonstrate the uncertainty around this issue:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4677810/ If you are interested more generally on dietary issues and prostate cancer, this is an interesting overview of some of the research, and also illustrates the uncertainty that exists:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4286914/
One of the things you have to be careful of with these sorts of issues in the extent to which the studies are retrospective and epidemiological. (Not all of them, but many of them). I am certainly not saying such studies are worthless, but in the absence of controlled trials getting at the specific issue, it is hard to reach definitive conclusions.
One further thing to consider, if your goal is to lower LDL: Statins tend to be pretty useful, and there is evidence showing a risk reduction for prostate cancer as well (though again the evidence is not perfectly clear).
* Nothing I write should be relied upon as medical advice*