Exploring Selection Bias
Wednesday, January 13, 2010
A new doctor comes to town, announcing that they are a family
doctor with an 'interest' in skin disease. The new doctor's office
employs an assistant who had previously worked with a
dermatologist. Their office advertises that they provide a variety
of cosmetic dermatology procedures. Within 6 months, you have seen
12 of their former patients. Every one of them was unhappy with the
care that they received. Some patients didn't get well, a couple of
patients had been given the wrong diagnosis, and some patients
thought that the doctor wasn't caring or thorough. What should you
- Ignore it, because it's none of your business;
- Get to know the doctor better and try to get a better sense of
them and their practice;
- Contact your dermatology society and its attorneys; or
- Contact the regional medical board to report the doctor.
Caring and committed dermatologists would probably do something
to make sure that patients weren't receiving bad medical treatment.
Perhaps very caring doctors would go out of their way to contact
the medical board or to petition for some form of government
intervention to put a stop to the perceived problem.
But consider how selection bias affects impressions of the new
doctor. For example, assume the new doctor sees 50 patients each
day. Over the course of 6 months, that's roughly 6,000 patients.
Perhaps 5988 of the 6,000 received great care and were totally
happy with the doctor and his or her practice. It is unlikely that
you will get to see any of those 5,988 happy patients, because they
will probably continue to visit the new doctor for their skin
problems. The only patients you are likely to see in your office
are the 0.2% who weren't happy with the other doctor. Although it
might seem, based on experience, that every one of the other
doctor's patients are extremely unhappy or poorly treated, you
wouldn't have seen a representative sample.
Consider what the other doctor's experience must be with
patients who saw you. Out of the 6,000 patients he or she saw
during those 6 months, probably a few score (if not a few hundred)
will have previously seen you or another dermatologist locally.
Few, if any, of those patients are likely to have been fully
satisfied, much less ecstatic, about the care they have received.
There might even have been a couple of missed diagnoses or possibly
even a missed melanoma diagnosis among them. Of course, those few
patients aren't at all representative of the thousands and
thousands of patients you've seen. You probably wouldn't want a
doctor to report you to the medical board based on such a sample of
Selection bias can have a profound effect on our observations.
Because we practice in a different compartment, the impressions
that dermatologists have of the dermatologic skills of primary care
doctors are based on seeing the primary care doctors' failures, and
never their successes. Likewise, a surgeon's impressions of a
dermatologist's surgical skills are affected by similar biased
sampling. Impressions of the effectiveness of medications and other
treatments used by patients before they visit the dermatologist are
also affected by selection bias. Selection bias can profoundly
influence studies of the prevalence, incidence and impact of
diseases such as melasma or psoriasis.
Selection bias can have an enormous effect on the apparent
frequency of a disease; much higher frequencies of skin disease and
higher impacts are seen in dermatology clinic populations than in
primary care or in the general population. Studies on the frequency
of arthritis in psoriasis patients have found that values can range
from 10% to nearly 100%, depending on whether the patients were
seen in a dermatology or rheumatology practice,
respectively.1 There might be 6 million or more
people in the United States with psoriasis and, although one third
of patients with psoriasis have severe disease, one third of the 6
million don't have severe psoriasis.2,3 The people who
see a dermatologist for psoriasis tend to be people with more
severe diseases, and while the greater majority of people with
psoriasis in the population (85-90%) have very limited
When clinical studies are carried out, the potential for
selection bias must be addressed. When information is sought for a
particular population, an enrollment strategy that obtains a
representative sample of that population should be used. The
population we see in our clinics is rarely, if ever, a
representative sample of anything other than the people who choose
to see a dermatologist. Thus, any assumptions we choose to make
about the patients who we don't see are made at our peril.
Consequently, we are likely to be led astray if we use our
experience to draw conclusions about the effectiveness of primary
care doctors in managing skin diseases or about the effectiveness
of over-the-counter treatments that patients might use before they
Selection bias is a very common phenomenon in dermatology and in
our daily lives. We shouldn't think that Columbine High School is
representative of secondary education in America or that the
terrorists we read about in the newspapers are in any way
representative of people of a particular religious faith. In fact,
what we see in the news is almost by definition not representative
of the norm, because the news reports what's new and different, not
what is ordinary.
There is a strong tendency among people to trust their
observations and experiences.4 We have to be very
careful not to put too much trust in observations that we should
know are strongly affected by selection bias. It is probably easier
to be misled into thinking that another doctor practises poorly
when in reality they take terrific care of their patients. Whenever
we are tempted to think ill of others based on our observations, we
really ought to consider the possibility that our observations
aren't representative. We would certainly hope that others would
extend us that same courtesy.
- Gladman DD, Shuckett R, Russell ML, Thorne JC, Schachter RK.
Psoriatic arthritis (PSA) - an analysis of 220 patients. Q J
- Stern RS, Nijsten T, Feldman SR, Margolis DJ, Rolstad T.
Psoriasis is common, carries a substantial burden even when not
extensive, and is associated with widespread treatment
dissatisfaction. J Investig Dermatol Symp Proc
- Fleischer AB Jr, Feldman SR, Rapp SR,et al. Disease
severity measures in a population of psoriasis patients: The
symptoms of psoriasis correlate with self-administered psoriasis
area severity scores. J Invest Dermatol
- Feldman SR. Compartments: How the Brightest, Best Trained, and
Most Caring People Can Make Judgments That are Completely and
Utterly Wrong. Xlibris: Philidelphia, PA, 2009.
For further reading on this topic, please see Feldman SR.
Compartments: How the Brightest, Best Trained, and Most Caring
People Can Make Judgments That are Completely and Utterly Wrong.
Xlibris: Philidelphia, PA, 2009.