Bad experiment design

Here in New Zealand we tend to import big important current affairs shows such as 20/20 and 60 Minutes, of course we put our own host upfront and show a couple of local stories as well as the interesting ones, mostly from the US, that seem to come with the program.

Well on Monday, during 60 Minutes they had a discussion (and this was one of the local stories) about food coloring and children's behaviour (the video clip of the story at the link). They talked with the experts and afflicted parents about how food coloring is bad and is being phased out in places and why are we not doing it etc. This in and of itself is reasonable and studies have shown that coloring can lead to hyperactivity in (some) children

But the really bad part of this was when they set up an experiment to show just what effect that the colorings have. They got some parents to lend their children (the kids all looked to be around 6-10 maybe) to the demonstration and put them in two groups. One group would have a healthy color free afternoon tea and the other group would have an afternoon tea full of colorings. They tested the children by getting them to do a drawing and some writing both before and after the food, and the children with the color free food had very little change in their drawing/writing while those in the color group there was a marked decrease in competency. However the best illustration (as far as the producers and the anti-color people were concerned) was that the kids in the color group were just bouncing off the walls and in one case bouncing balls of the presenter and interviewee (a child psychologist I think).

On the face of it this sounds like a great demonstration that showed up exactly the concerns that exist about the colorings. The problem was in the controlling of the coloring/non-coloring foods. The coloring group got all the foods that you can give to kids with heaps of the bad colorings in them, things like candy, cordial drinks and coke and that sort of thing. The non-colorings group had lots of fresh fruit and water.

If you have not spotted why this does not show colorings in a bad light then maybe go back and compare those snacks again. The colorings group not only got colorings that the non-colorings group did not get but they also got lots of high sugar food (especially refined sugars) and caffeine that the control group did not get, for those of you at home these are known as confounding factors.

So what did the demonstration show, that a combination of lots of sugar, caffeine and coloring leads to kids bouncing off the walls. Last time I checked with my two little boys (and their friends) that amount of sugar alone will set kids off, as I witnessed at my elder boy's 4th birthday party last weekend.

How could they have done this better, well clearly the control group should have had the same amount and type of sugars, that way you would have been able to see the effect of the colorings, rather than what I suspect was mainly the effect of the sugar that these kids got to stuff themselves with. An example of a way that this could have been done was to use cordial drinks alone as the difference between the groups as many brands put out a color-free variety as well as the usual colored ones.

Unfortunately, my wife tells me that some of the other mothers at the playcentre my children go to, did not manage to see this fault in the demonstration and my wife was not able to convince them of why it did not show what they said it showed.

That all said, there does appear to be something to this hypothesis that food colorings can be bad for children's behavior - now if only we could get the TV people to realise how to design a demonstration to illustrate a point. Oh well I had better send them a copy of this.

Update: added link to clip from show.