”The importance of stratification in a statistician’s August kitchen”
August 2023. The teenagers have been home alone all week, I have enjoyed the peace and quiet in the summer house, and the sun has gassed and offered humid, warm days. It isn’t that they have misbehaved, but rather that it takes cutting edge competence to master the fruit flies of 2023 in my kitchen. Does the extent of this problem from last summer sound familiar? Dirty dishes waiting on the bench to get their turn in the dishwasher did not exactly help.
I enter the door, tired from a long drive, and meet a thick cloud of unwanted little pets who definitely cannot wait for tomorrow to get their treatment. While scrubbing the kitchen benches in the warm summer night an interesting study design slowly forms. Vinegar and a drop of fairy liquid, but was it honey as well? Or a bite of apple sauce? Concentrated or with water to make the little critters best fall into the trap?
I pour different combinations of the ingredients into four mugs. – Excellent, I will soon know exactly which one of the soups is best against fruit flies, as they get drawn to the fragrant witch’s brew and drown in the absence of surface tension.
Only an hour later I delightedly see the results roll in. The numbers of fallen prey are 19, 3, 69 and 14. Crystal clear: 69 fruit flies must be significantly more than 19 and the rest. The concentrated soup with all ingredients except water is the clear winner. I go to bed assured that my kitchen will also be clean from fruit flies in the morning. (This latter assumption was approximately right.)
How many are the examples of promising preliminary animal studies or early clinical trials, which disappeared without a trace at the next attempt of replication? What is actually happening? I wake up, and each mug contains probably 200 dead fruit flies, no less or more in any of the mugs, and I cannot draw any conclusion about what brew is most efficient.
There were of course a million aspects that I had not controlled for. In the evening, I had put one mug on the sink, one in the kitchen window and one in the living room window, and all of these places had different size clouds of critters in their vicinity at the time. The failed experiment highlights how difficult it is to take all aspects into account when planning a study. Only a minor part of all potentially disturbing elements are known in advance.
Had all possible so called effect modifiers been known, studies could have been stratified for them to adjust for the bias they create, at least if sample sizes were large enough. Stratification means grouping patients with similar properties and making sure the comparison between treatments is done within the groups, or after correction for these stratification properties. I have four treatments (four different soups) to be compared within each of the three locations – this requires many mugs, large research resources… The fruit flies were too soon over for the season.
I told a friend about my study. She listened politely, but then asked why I didn’t just go for a hunt with the vacuum cleaner a little now and then. Perhaps we should just let go of treatment regimens that prove difficult to confirm, and seek new ideas instead?
Naaa, everything is possible! Just give in to the fact that a whole year of study planning is needed (there too a similarity to the clinical trial). This autumn 2024 I can reveal that replicate, controlled experiments including a negative control speak clearly in favour of mixing equal proportions of apple cider vinegar, water and apple jam, plus the drop of fairy liquid. I hereby challenge you to test it yourself against your favourite mixture, in your best controlled study design, and we proceed this forefront of research together!
Artikeln är en del av vårt tema om News in English.