Help me understand statistics in a medical study, please! :-)

[simple girl]

Hello all!

I have a question for those of you with a statistical background. I'm looking at a medical study and need to know how much I can trust the results to help me make a medical decision. (I'm not looking at this study only, but the research in this area is scarce, so I don't have much to go on.) The study involves a small number of patients, so I'm worried about how well it generalizes to the rest of the population with those same characteristics.

Details:

• 
  5 of 31 patients with specific characteristics (16%) had a cancer recurrence
  The researchers applied a Poisson regression analysis and found this was statistically significant (p = 0.0356) (significance was evaluated using a two-sided p < 0.05).

Someone on another discussion board I am on stated this about the study:

With study sizes this small, the margin of error is very large. For this study, it's nearly 13% with a 95% confidence interval (i.e., we're 95% confident that that risk of recurrence is 16% +/- 13%).

Is her analysis correct? Something seems wrong with it - but I have not studied statistics in YEARS. I KNOW there are people on here who will be able to help!

Let me know if you need any other details to make sense of this.

 

[Rich_by_the_Bay]

Hello all!

I have a question for those of you with a statistical background. I'm looking at a medical study and need to know how much I can trust the results to help me make a medical decision. (I'm not looking at this study only, but the research in this area is scarce, so I don't have much to go on.) The study involves a small number of patients, so I'm worried about how well it generalizes to the rest of the population with those same characteristics.

A somewhat useful interpretation: the 95% confidence interval is the range within which we can be 95% sure that the true value for the whole population lies. A large CI means might imply too few subjects, too modest a respone to the treatment, too brief a study, etc. A small value implies a tight clustering of values, and has higher a likelihood of reflecting true effect.

Hope that helps.

 

[Meadbh]

Agree with Rich.

The confidence interval consists of the upper and lower limits between which we are confident that the result lies, 95% of the time. (That's where the p value comes in).

In this case, with only 31 patients, risk of recurrence is 16% and the 95% confidence interval is 16-13 (3%) to 16+13 (29%). Therefore, based on the study, we can be confident that the rate of recurrence in the general population as defined is between 3-29%, 95% of the time.

If, instead, there were 3100 patients, with the same risk of recurrence (16%), one would expect the 95% confidence interval to be much narrower. For example, let's say it was plus or minus 3%, we could be confident that the recurrence rate in the general population was between 13%-19%, 95% of the time. Please note that this is just a hypothetical illustration.

Obviously, the more people there are in the study, the more clearly the outcome can be defined. Researchers do sample size calculations to estimate the number of people required, because you don't want to do a study that's too small and provides no information, but you don't want to do a study that's too big either, because you could be exposing people to risk.

Hope that helps.

 

[simple girl]

A somewhat useful interpretation: the 95% confidence interval is the range within which we can be 95% sure that the true value for the whole population lies. A large CI means might imply too few subjects, too modest a respone to the treatment, too brief a study, etc. A small value implies a tight clustering of values, and has higher a likelihood of reflecting true effect.

Hope that helps.

Agree with Rich.

The confidence interval consists of the upper and lower limits between which we are confident that the result lies, 95% of the time. (That's where the p value comes in).

In this case, with only 31 patients, risk of recurrence is 16% and the 95% confidence interval is 16-13 (3%) to 16+13 (29%). Therefore, based on the study, we can be confident that the rate of recurrence in the general population as defined is between 3-29%, 95% of the time.

If, instead, there were 3100 patients, with the same risk of recurrence (16%), one would expect the 95% confidence interval to be much narrower. For example, let's say it was plus or minus 3%, we could be confident that the recurrence rate in the general population was between 13%-19%, 95% of the time. Please note that this is just a hypothetical illustration.

Obviously, the more people there are in the study, the more clearly the outcome can be defined. Researchers do sample size calculations to estimate the number of people required, because you don't want to do a study that's too small and provides no information, but you don't want to do a study that's too big either, because you could be exposing people to risk.

Hope that helps.

That makes perfect sense. Ah, my statistics memory is coming back, LOL! Thank you both! This was a retrospective study, so they didn't have any control over how many patients ended up with a recurrence. Since so few recur, it makes doing studies on this topic really difficult to perform and interpret. Thanks again to you both!

 

[Nords]

That makes perfect sense. Ah, my statistics memory is coming back, LOL! Thank you both! This was a retrospective study, so they didn't have any control over how many patients ended up with a recurrence. Since so few recur, it makes doing studies on this topic really difficult to perform and interpret. Thanks again to you both!

I've only taken a couple of statistics courses, but I'm pretty confident that nobody can predict anything from 31 patients with any confidence.

They're saying "We think we've identified all the variables. We might not know exactly what's going on, but we know we're looking at the right things." Meanwhile if they took this to the FDA for a drug trial they'd be laughed out of the building.

This is like financial analysis assuming that the stock market is really some sort of bell curve. It might be symmetrical or log-normal or some other cool name, but the assumption is that the stock market can be equated to some sort of bell curve and then analyzed.

The problem is that the stock market is not a bell curve. And all the really really bad things happen in the .0001% of the stock market that is not (and never was) a bell curve.

31 patients is better than zero patients, but it's not a study. It's watching Vegas blackjack for an hour and then betting your entire life's savings. Or, in the case of the study's authors, it's an application for additional funding. And just like the stock market analysts or the Vegas blackjack dealers, they don't suffer if you lose all your money.

 

[Golden Mean]

As I understand it, a lot of retrospective studies are what you find the media picking up on. You know, the "people who drink a glass of wine a day live longer, so wine is good for you" stories. But correlation doesn't always inidicate causation, so I'd take any study of this type with a grain of salt.

GM

 

[Rich_by_the_Bay]

I've only taken a couple of statistics courses, but I'm pretty confident that nobody can predict anything from 31 patients with any confidence.

Not necessarily, though all other things being equal, sample size is an important factor in many cases.

But in a case where the the group outcomes are drastically different, where a dose-response effect is prominent or where the intensity of the response is strong it may well be valid with even small group size (assuming quality study design).

31 patients have gizzard cancer
- 15 patients receive gorillatoxin
- 16 patients do NOT receive gorillatoxin

- treated group all alive and well at one year
- Untreated group all die of gizzard cancer at 1 year
study double-blinded and well matched, etc.

Every study is different. Study size is important but sometimes mitigated by other factors.

 

[Nords]

Not necessarily, though all other things being equal, sample size is an important factor in many cases.
But in a case where the the group outcomes are drastically different, where a dose-response effect is prominent or where the intensity of the response is strong it may well be valid with even small group size (assuming quality study design).

It seems like the FDA has received a lot of bad press from situations where "smaller" trials of thousands of subjects were judged satisfactory, only to have unexpected problems pop up once millions were involved.

Does the FDA set a minimum on sample size or other trial aspects?

 

[Texas Proud]

Not necessarily, though all other things being equal, sample size is an important factor in many cases.

But in a case where the the group outcomes are drastically different, where a dose-response effect is prominent or where the intensity of the response is strong it may well be valid with even small group size (assuming quality study design).

31 patients have gizzard cancer
- 15 patients receive gorillatoxin
- 16 patients do NOT receive gorillatoxin

- treated group all alive and well at one year
- Untreated group all die of gizzard cancer at 1 year
study double-blinded and well matched, etc.

Every study is different. Study size is important but sometimes mitigated by other factors.

It seems like the FDA has received a lot of bad press from situations where "smaller" trials of thousands of subjects were judged satisfactory, only to have unexpected problems pop up once millions were involved.

Does the FDA set a minimum on sample size or other trial aspects?

First, I did not read the article....


But, the important statement was "this was statistically significant".... how significant might not be know with great accuracy, but it is not zero...


IOW, it is something that should be looked into further and see if something else can be learned...

I remember when our prof gave us a problem that screamed there was a problem, but when you did the math there was no 'statistically significant' difference... his point being that you should let the math determine if there is a difference that should be further researched instead of a seat of the pants look at the data....


PS.... I love it when the news say this candidate is leading with 20% of the poll and the other guy is in second with 19%... and then there is a big whoop-te-do when it switches the next day.... when the margin of error on these is 5% or even 7%.... heck, the guy with 15% could really be leading for all you know....

 

[simple girl]

Not necessarily, though all other things being equal, sample size is an important factor in many cases.

But in a case where the the group outcomes are drastically different, where a dose-response effect is prominent or where the intensity of the response is strong it may well be valid with even small group size (assuming quality study design).

31 patients have gizzard cancer
- 15 patients receive gorillatoxin
- 16 patients do NOT receive gorillatoxin

- treated group all alive and well at one year
- Untreated group all die of gizzard cancer at 1 year
study double-blinded and well matched, etc.

Every study is different. Study size is important but sometimes mitigated by other factors.

Very true. Wish I had gizzard cancer. Sounds like it'd be easier to treat. See, I haven't lost my sense of humor!

First, I did not read the article....


But, the important statement was "this was statistically significant".... how significant might not be know with great accuracy, but it is not zero...


IOW, it is something that should be looked into further and see if something else can be learned...

Also very true. The study authors of course indicated more research is needed. Unfortunately, I need to make a decision very soon based upon the limited data that I have combined with the conflicting medical opinions I have received. Maybe I'll toss a coin. LOL

 

[cbslc]

First, I'm sorry you have to make what sounds like a very tough decision.
I am a healthcare data analyst, so I'm going to de-humanize this.
Unfortunately the statistics are not meaningful without other pieces.
"5 of 31 patients with specific characteristics (16%) had a cancer recurrence" The missing piece of this statement is within XX years. If they had no cancer recurrence in 2 years, great. No recurrence in 1 year, 6mos, 3 mos ...
I've seen a lot of cancers staved off for 2 years, only to come back in a more deadly form at 25 mos.
A huge factor with cancer is age of onset. Do you know the median age of the 31 participants? Were they healthy otherwise ...
The stats are a important, but the study design and the years w/o recurrence are critical to making a decision.
I've seen a lot of people go through 1 year of agonizing chemo to gain 2 years of life. I've also seen terminally ill children that are medically "tortured" by their parents (and docs), trying to eek out more time.
Sorry to sound harsh, but I have to wonder if its all worth it in the end.
I wish you the best of luck.

 

[simple girl]

First, I'm sorry you have to make what sounds like a very tough decision.
I am a healthcare data analyst, so I'm going to de-humanize this.
Unfortunately the statistics are not meaningful without other pieces.
"5 of 31 patients with specific characteristics (16%) had a cancer recurrence" The missing piece of this statement is within XX years. If they had no cancer recurrence in 2 years, great. No recurrence in 1 year, 6mos, 3 mos ...
I've seen a lot of cancers staved off for 2 years, only to come back in a more deadly form at 25 mos.
A huge factor with cancer is age of onset. Do you know the median age of the 31 participants? Were they healthy otherwise ...
The stats are a important, but the study design and the years w/o recurrence are critical to making a decision.
I've seen a lot of people go through 1 year of agonizing chemo to gain 2 years of life. I've also seen terminally ill children that are medically "tortured" by their parents (and docs), trying to eek out more time.
Sorry to sound harsh, but I have to wonder if its all worth it in the end.
I wish you the best of luck.

For the 5 recurrences I am looking at -
Median time to recurrence was 42 months after surgery.
Median age was 41

This is not a case of trying to eek out more time with advanced cancer. This is early stage cancer that typically has an excellent prognosis, but I have a few particular features that are concerning. It is a case of trying to determine if I have the same risk of recurrence since I share many of the same features as those 5 that recurred. And then beyond that, it is the case of trying to figure out if the long-term risks of radiation are worse than the risks of going without it.

No one is going to be able to tell me what to do, sadly, b/c there is not much research for my particular situation, and the studies that do exist are small, like this one. I will research, research, research, but eventually it is going to come down to a gut decision on my part.

Thanks everyone for your feedback on the study. And I want to emphasize that I am not asking anyone to give me their advice on what to do - especially the docs on here - I would never ask that! Just was looking for some feedback on how to interpret the study!

 

[tightasadrum]

The "Poisson" function predicts the outcome of binary sampling, like yes/no; live/die; pass/fail. These are not variables. Think coin toss, and the probability it comes out heads. (.5) A variable function, which is a measurement of something like weight or height, is a lot more sensitive and can make more reliable predictions with a smaller population. (Think normal distribution. The more samples you have, the closer you can get to predicting the actual curve. But a few samples (30?) will get you in the ball park.) Poisson functions take a whole lot more sampling to get a valid prediction of the outcome probability. That is probably why medical studies usually take thousands of samples before a reliable prediction can be made about the population you're sampling. (What is the probability of being cured if you take the drug? Obviously more samples, more confidence.) Drug experiments must be very difficult. How do you isolate all the variables? Do the study with just men; or just teenage women with blond hair and fathers that retired early. Many of the variables are not even knowable.

 

[Bestwifeever]

...No one is going to be able to tell me what to do, sadly, b/c there is not much research for my particular situation, and the studies that do exist are small, like this one. I will research, research, research, but eventually it is going to come down to a gut decision on my part. ...

Darn, I wish you didn't have to be dealing with this on a real-life basis, SG, but instead had the luxury of just thinking theoretically about sample size and statistical outcomes.

Hang in there and as we've all said many times, we admire your going on the offense to seek out the best treatment for yourself!

 

[bbbamI]

Wish I had gizzard cancer. Sounds like it'd be easier to treat. See, I haven't lost my sense of humor!

Gizzard cancer you say....?

Don't know much 'bout them thar stistics...gimme some time would ya....?

 

[Nords]

First, I did not read the article....

Yet somehow you still felt qualified to share your opinion of it anyway.

Most especially in the situation that SG is confronting, if she's gonna take the time to post it then I'm gonna make the time to read it.

 

[simple girl]

Yet somehow you still felt qualified to share your opinion of it anyway.

Most especially in the situation that SG is confronting, if she's gonna take the time to post it then I'm gonna make the time to read it.

Awh, he couldn't read the article, b/c I didn't post it! I didn't want people to feel compelled to read it and give advice about what I should do. No one can make my decision for me. I was trying to keep emotion out of it and see what people thought of the statistical strength of this type of research. However, I guess someone might be interested enough to want to see it, so I've attached it.

Thank you everyone for being so caring and helpful!

 

[Meadbh]

Dear SG:

Thank you for posting the paper. The final sentence is telling:

"We do, nonetheless, realize the limitations of a single-institutional, retrospective study and believe additional research is necessary to confirm our findings."​

The authors are correct. This is a retrospective analysis, which can generate hypotheses for further research, but it doesn't prove anything. Therefore, you can make a decision based on your personal preference, and no matter what you choose, you will be right!

 

[simple girl]

Dear SG:

Thank you for posting the paper. The final sentence is telling:

"We do, nonetheless, realize the limitations of a single-institutional, retrospective study and believe additional research is necessary to confirm our findings."​

The authors are correct. This is a retrospective analysis, which can generate hypotheses for further research, but it doesn't prove anything. Therefore, you can make a decision based on your personal preference, and no matter what you choose, you will be right!

Thank you!!!

 

[Meadbh]

This might be a little technical, but take a look:

http://www.dartmouthatlas.org/downloads/reports/preference_sensitive.pdf

The bottom line is: this decision is about what's important to you, about your values.

 

[simple girl]

This might be a little technical, but take a look:

http://www.dartmouthatlas.org/downloads/reports/preference_sensitive.pdf

The bottom line is: this decision is about what's important to you, about your values.

Thank you - definitely good food for thought - this will help me look more closely at how I am going about making this decision. I definitely appreciate the insight!

 

[DeborahB]

For the 5 recurrences I am looking at -
Median time to recurrence was 42 months after surgery.
Median age was 41

This is not a case of trying to eek out more time with advanced cancer. This is early stage cancer that typically has an excellent prognosis, but I have a few particular features that are concerning. It is a case of trying to determine if I have the same risk of recurrence since I share many of the same features as those 5 that recurred. And then beyond that, it is the case of trying to figure out if the long-term risks of radiation are worse than the risks of going without it.

No one is going to be able to tell me what to do, sadly, b/c there is not much research for my particular situation, and the studies that do exist are small, like this one. I will research, research, research, but eventually it is going to come down to a gut decision on my part.

Thanks everyone for your feedback on the study. And I want to emphasize that I am not asking anyone to give me their advice on what to do - especially the docs on here - I would never ask that! Just was looking for some feedback on how to interpret the study!

It is really hard to make decisions about cancer treatment. The LT risks of radiation are not to be minimized. The quality of life issues are important as well. I had ovarian cancer 15 years ago and luckily needed no chemo/radiation and haven't recurred. As a result I became very involved nationally in cancer issues.

I am assuming you have DCIS since that's what the article is about - one resource I would strongly suggest is SHARE in NYC - they have a free phone number and website

SHARE Call toll-free (866) 891-2392

You can talk to survivors at SHARE who have a lot of experience. Good luck with such a difficult decision. If you want to contact me, please feel free to PM me.

 

[Texas Proud]

Yet somehow you still felt qualified to share your opinion of it anyway.

Most especially in the situation that SG is confronting, if she's gonna take the time to post it then I'm gonna make the time to read it.

And as pointed out in the next post, she did not even post the full article.... so how did you take the time to read it (if you are going to bust my chops for no reason, I am willing to bust yours for the same)

I commented on the stat question she asked, which was fully disclosed... I did not need to read anything else to answer that....

 

[simple girl]

It is really hard to make decisions about cancer treatment. The LT risks of radiation are not to be minimized. The quality of life issues are important as well. I had ovarian cancer 15 years ago and luckily needed no chemo/radiation and haven't recurred. As a result I became very involved nationally in cancer issues.

I am assuming you have DCIS since that's what the article is about - one resource I would strongly suggest is SHARE in NYC - they have a free phone number and website

SHARE Call toll-free (866) 891-2392

You can talk to survivors at SHARE who have a lot of experience. Good luck with such a difficult decision and/or had much difficulty making a decision. If you want to contact me, please feel free to PM me.

Thank you so much for the support and encouragement. And huge congratulations on no recurrence!!! I do belong to the breastcancer.org discussion boards and am a very active poster over there (hence, this is why I'm not on these boards as much lately. ) I have talked with several women in the exact same situation as me - we are ALL struggling to make a decision. We all are getting different opinions from various radiation oncologists and tumor boards. It's ridiculously confusing and soooooo draining trying to make a decision. I have two appts next week and will be pulling the trigger on my decision shortly afterwards.

 

[Texas Proud]

Let's hope whatever decision you make gets you past this terrible time and you live a long and healthy life.....