Help me understand statistics in a medical study, please! :-)

[Koolau]

Does the FDA set a minimum on sample size or other trial aspects?

This might help - (no need to read the article ) just skip down to "Drug Review Steps".

The FDA's Drug Review Process: Ensuring Drugs Are Safe and Effective

Keep in mind, this is talking about "new drugs" not "new or improved or existing" treatment methods as is probably the case here (i.e., just trying to answer your specific FDA question). By the way, the drug company is involved at every step with the FDA during the trials (at least they had better be). They negotiate sample size (and myriad other things) more or less on the fly in an attempt to end up with data that can convince FDA that the new drug is "good" and "safe". When you hear $1Bil to get a new drug to market, it's not BS. I think there have been trials with 10,000 subjects, but I could be wrong. I was once.

Hope this helps.

 

[DoingHomework]

But, the important statement was "this was statistically significant".... how significant might not be know with great accuracy, but it is not zero...

Actually we do know how significant. That's what the p-value quantifies. The researchers were testing a hypothesis. They likely assumed that the response(fate) of the patients with a certain characteristic was the same as the general population of all patients (people). This was their null hypothesis. They then performed a rigorous test of that hypothesis and rejected it to conclude that the people with a certain characteristic did have a different outcome. The p-value is the probability that that the null hypothesis (people with characteristics are the same as everyone else) is true. 1-p is the probability that the alternative is true (people with characteristics are different). The confidence interval stuff is a different, but related, type of analysis that estimates what the "average outcome" is for the population.

There is a lot of good interpretation going on here. I've had more than a few statistics classes and I wouldn't seriously criticize any of the statements I've read so far, though I do recognize both frequentist and Bayesian influences.

It's easy to dismiss such a study because it is small. But often you can't help but do a small study because there are simply very few cases (usually a good thing when speaking of disease). So it is more useful to get the most information out of the study as possible, in spite of the limitations.

In this case if I knew that 5 or 31 patients with certain characteristics survived after 10 years, I'd spend my time trying to figure out what those 5 had in common, or how to be in that group of 5 rather than getting lost in the statistics. Whether the chance of survival (or non-survival) is 16% or between 3% and 29% would be less meaningful to me than "How do I become one of those that survive?"

Incidentally, this whole small sample issue was addressed over 100 years ago in the interest of saving beer. Quality inspections at Guinness performed at high statistical confidence required wasting too much product so William Gosset developed the t-test (and with it the entire science of small sample statistics) to allow accurate estimation to be done using small samples. But in this case, with a sample size of 31, the t-distribution is essentially identical to the normal.

In this study the Poisson distribution was used according to the original statement. The Poisson is a reasonable approximation to the normal distribution for sample sizes of 10 or larger. It is appropriate to use the Poisson distribution since it is a discrete distribution where as the normal distribution is continuous.

When an estimate is given as 16%+/-13% you already have all the information about the "accuracy" of the study. Knowing whether it had 31 participants or 31 million adds no new information. It's true that the larger sample should make the confidence limits smaller. But if there is a lot of variability in the large study then the limits could also be large. Those limits are telling you how certain the estimate is. If knowing the sample size further influences your opinion of the "reliability" of the study then you probably don't understand the statistical interpretation as well as you think you do. Dismissing a study just because it is small is a pet peeve of mine. There is no mathematical basis for that. It is rooted in statistical ignorance of researchers. Some of the worst studies are the very large ones because they tend to pick up random events that confound results.

Basically, this study is telling you that it is about 97% certain than the group of people with those certain characteristics has a different outcome from the group of all people. When they say they used a two-tailed test they are telling you they can only detect a difference, not whether one if higher or lower.

The confidence interval gives you some additional information about what the average "success" rate would be 95% of the time if the study were repeated a large number of times.

 

[Meadbh]

Incidentally, this whole small sample issue was addressed over 100 years ago in the interest of saving beer. Quality inspections at Guinness performed at high statistical confidence required wasting too much product so William Gosset developed the t-test (and with it the entire science of small sample statistics) to allow accurate estimation to be done using small samples. But in this case, with a sample size of 31, the t-distribution is essentially identical to the normal.

Another Irish contribution to science!
Time for a pint!
Slainte!

 

[Nords]

And as pointed out in the next post, she did not even post the full article.... so how did you take the time to read it (if you are going to bust my chops for no reason, I am willing to bust yours for the same)

I commented on the stat question she asked, which was fully disclosed... I did not need to read anything else to answer that....

Maybe I was a tad too subtle back there.

I knew perfectly well that there was no article in SG's original post. But then, I actually read her post.

Your original comment was as I quoted-- not only did you not read the article, but you didn't even notice the absence of the article. You didn't say "Hey, where's the link?" or anything else to indicate that you were even attempting to read an article. From your commentary it's difficult to tell that you even bothered to read the post.

There are over 25,000 registered posters on Bogleheads.org, and many of the active ones spend their time talking past each other instead of engaging in thoughtful analysis & discussion. They don't have to read posts because they already know what they want to say and they're eager to get past the "reading" part to the "saying" part.

Let's not make this board like that.

 

[simple girl]

It's easy to dismiss such a study because it is small. But often you can't help but do a small study because there are simply very few cases (usually a good thing when speaking of disease). So it is more useful to get the most information out of the study as possible, in spite of the limitations.

In this case if I knew that 5 or 31 patients with certain characteristics survived after 10 years, I'd spend my time trying to figure out what those 5 had in common, or how to be in that group of 5 rather than getting lost in the statistics. Whether the chance of survival (or non-survival) is 16% or between 3% and 29% would be less meaningful to me than "How do I become one of those that survive?"

Thank you for your analysis. Yes, you are correct, they can't help but do a small study, because so few patients end up with a recurrence. However in this case, you don't want to be in the 5...they are the ones who recurred. Unfortunately, I share all but one of the same features those 5 shared. So my question was, is the study strong enough for me to believe my chances are pretty high to recur? The 3 to 29% chance is the potential range for risk of recurrence. That's a huge variation when you are trying to decide if you should have radiation or not to reduce that risk, especially when radiation does not come without its own set of risks.

 

[tightasadrum]

From what I've observed the last few years, risk for recurrence would have to be high to take on the radiation risk. DW elected to not have radiation based on what she felt was a low risk of recurrence and the serious organ damage her aunt sustained that was cause by radiation. This was reinforced again a few months ago. A close friend had experienced fainting spells. I picked her up and brought her to our house to keep an eye on her. (She didn't want to go to the hospital. ) That evening she had another episode. We took her to the emergency room, and right after they wired her up, she had another episode. Crash Cart time! Luckily everything was recorded. She flat lined for almost a minute! Prognosis: radiation damage to a part of her heart that regulated heartbeat. She now has a pacemaker and appears to be doing well. This doesn't answer your original risk question I know. But proceed with caution.

 

[simple girl]

From what I've observed the last few years, risk for recurrence would have to be high to take on the radiation risk. DW elected to not have radiation based on what she felt was a low risk of recurrence and the serious organ damage her aunt sustained that was cause by radiation. This was reinforced again a few months ago. A close friend had experienced fainting spells. I picked her up and brought her to our house to keep an eye on her. (She didn't want to go to the hospital. ) That evening she had another episode. We took her to the emergency room, and right after they wired her up, she had another episode. Crash Cart time! Luckily everything was recorded. She flat lined for almost a minute! Prognosis: radiation damage to a part of her heart that regulated heartbeat. She now has a pacemaker and appears to be doing well. This doesn't answer your original risk question I know. But proceed with caution.

That's exactly why I'm trying to figure out exactly what my risk of recurrence is. Unfortunately there are no clear answers in my situation, due to the small number of patients with my particular features. So I am left to guess what my estimated risk of recurrence is based on a tiny number of studies that have a small sample size.

Also, how long ago did your wife's aunt and your friend have rads? Rads techniques have improved considerably in the last few years. For example, they have respiratory-gated rads which times your zaps with your inspiration/expiration so that you are zapped only when your heart is in the more protected zone. No long term studies yet, though, that I could find, exist - and that is because heart damage usually does not show up until 10 or 15 years down the road, and the new techniques have not been around long enough to have these studies done yet. I think. Still researching! Rads really scares me. So does recurring. Coin flip anyone?

 

[tightasadrum]

Her aunt had radiation over 25 years ago, and our friend had hers over 15 years ago I think. So perhaps they are better at it now. It's a tough decision I know.

 

[Nords]

Her aunt had radiation over 25 years ago, and our friend had hers over 15 years ago I think. So perhaps they are better at it now. It's a tough decision I know.

Far, far better. It's the equivalent of going from point-blank range with a 12-ga shotgun... to a hypodermic needle.

 

[Texas Proud]

Maybe I was a tad too subtle back there.

I knew perfectly well that there was no article in SG's original post. But then, I actually read her post.

Your original comment was as I quoted-- not only did you not read the article, but you didn't even notice the absence of the article. You didn't say "Hey, where's the link?" or anything else to indicate that you were even attempting to read an article. From your commentary it's difficult to tell that you even bothered to read the post.

There are over 25,000 registered posters on Bogleheads.org, and many of the active ones spend their time talking past each other instead of engaging in thoughtful analysis & discussion. They don't have to read posts because they already know what they want to say and they're eager to get past the "reading" part to the "saying" part.

Let's not make this board like that.

I actually read her post... and it was a relatively simple question on statistics... I saw a piece of an article and what appeared to be a link. I did not bother to test if it was or was not... just commented that I did not read it... I only wanted to comment on what was 'there'.....

Your post only seemed to make reference to the one item of me not reading the article and not that I did answer the question (maybe not as well as others, but still addressed the subject)... I did not 'talk past' the OP at all, so I do not know why you took your time to make a snide comment...

 

[DoingHomework]

However in this case, you don't want to be in the 5...they are the ones who recurred. Unfortunately, I share all but one of the same features those 5 shared. So my question was, is the study strong enough for me to believe my chances are pretty high to recur? The 3 to 29% chance is the potential range for risk of recurrence. That's a huge variation when you are trying to decide if you should have radiation or not to reduce that risk, especially when radiation does not come without its own set of risks.

Yes, I realized after posting that I had stated it wrong regarding whether it is desirable or undesirable to be in the 5. But the point remains...given the information you have, what can you do? Are the characteristics you share with those with recurrence something you can change (e. g. weight) or are they "built-in" (e. g. skin color)? That's not a question I'm asking you. It's something to ask yourself when deciding how relevant that study is to you.

I can't even comment on what you should do regarding treatment. That's something you should discuss with your oncologist.

But the statistics are telling you that if you had to make one estimate of the probability of recurrence for the entire population from which the 31 came, it would be 16%. But the limitations of the study (size, uncontrolled variables, etc.) mean that the true value is 95% certain to be between 3% and 29%.

 

[ERD50]

Her aunt had radiation over 25 years ago, and our friend had hers over 15 years ago I think. So perhaps they are better at it now. It's a tough decision I know.

Far, far better. It's the equivalent of going from point-blank range with a 12-ga shotgun... to a hypodermic needle.

+1. I know a radiation therapist that just certified on some of the newest versions of equipment (Cyber-Knife). The pinpoint accuracy of the newer stuff is like night and day. There are something like 100 beams that all focus on a point around a mm in size. It can even track a moving target (like a spot on a lung that moves with each breath). She says the old equipment would hit people with so much extraneous radiation, they would really suffer with the side effects. These side effects are greatly diminished with the newer equipment.

-ERD50