Search This Blog

Tuesday, December 10, 2013

A More Rigorous Analysis: Does A Baby's Eye Gaze Really Predict Future Autism?

From Cracking The Enigma

By Jon Brock
December 6, 2013
That was the headline in the New York Times. The BBC declared that Autism signs present in first months of life. Turning the hype up to 11, a Canadian website boldly announced that Researchers prove that autism can be diagnosed right at the infant stage and that intervention is possible.

Nature, the journal that published the study, ran with Autism symptoms seen in babies, summarising the findings thus:

"Children with autism make less eye contact than others of the same age, an indicator that is used to diagnose the developmental disorder after the age of two years. But a paper published today in Nature reports that infants as young as two months can display signs of this condition, the earliest detection of autism symptoms yet."

Certainly, being able to identify infants who were likely to develop autism would be a ground breaking advance, opening up the possibility of very early diagnosis and intervention. It would also allow researchers to study the very earliest stages of autism development.

But, as with many studies that receive the full media treatment, there are caveats a-plenty. In fact, it could be argued that the results show the exact opposite of what the authors and the media coverage has suggested.

The study was conducted by Warren Jones and Ami Klin from Emory University. Back in 2002, Klin and colleagues published a study showing that adolescents with autism spent less time looking at the eyes of people in a movie clip than did typically developing adolescents. Although, like all things autism, this seems to be true of some but not all people with a diagnosis.

Since then, Klin and Jones have reported similar results in two-year-old children with autism. Their new study was an attempt to push that all the way back to the very earliest months of life.

Jones and Klin began with a sample of 64 infant boys, 38 of whom had an older sibling with autism, putting them at increased risk of having autism themselves [1]. They also tested 46 girls but later excluded all of them from the analyses [2].

At various time points between the ages of 2 months and 2 years, the infants were eye-tracked as they viewed short video clips of a female caregiver’s face and upper body.

Sample scanpaths for a baby later diagnosed with
autism (red) and a typically developing control (blue).

Then, at 3 years of age, the by-now toddlers were assessed for autism [3]. 11 boys (10 from the high risk group) were identified as having an autism spectrum disorder (ASD). They were then compared with the remaining 25 typically developing (TD) boys from the low-risk group.

The 64 boys were divided into 4 groups at their 3-year assessment:
ASD; low risk typically developing (TD); high-risk with some autism
symptoms (BAP); and high-risk with no autism symptoms (no-Dx).
Forty-six girls were tested but were not included in the reported analyses.

Jones and Klin began their analyses by building developmental trajectories for the two groups, similar to the growth charts that doctors use to tell, for example, whether a baby is putting on sufficient weight or not. Except here, what mattered was the percentage of time the babies were looking at the eyes in the video.

What they found was that the ASD boys showed a steady decline in eye gaze across time. However, there were no significant differences between the trajectories of the two groups until the final test session at 24 months [4].

The obvious interpretation of these data is that, in fact, eye gaze in infancy does not predict which infants will go on to develop autism (at least using Jones and Klin's set-up). Both from a practical and a theoretical point of view, that's an important finding. [5]

So how do we end up with "Baby's eye gaze signals autism"?

Having failed to find evidence for reduced eye gaze in ASD infants, Jones and Klin looked instead at the slopes of the developmental trajectories. In other words, not the amount of eye gaze at a particular time but the change in eye gaze relative to earlier and later time points. Here, they did find significant differences throughout the early months.

When the analyses were restricted to the data from 2 to 6 months, the boys who developed autism showed a negative slope (declining eye gaze) while the low-risk boys had a positive slope (increasing eye gaze).

This, in essence, is what all the excitement is about. The study suggests that if you measure a baby's eye gaze at multiple time points before the age of 6 months and notice a decline over time, then that baby is at heightened risk for developing autism. It doesn't matter how awkward the data collection process or convoluted the analysis, there's information about autism in a babies' eye gaze.

However, there was always something about this story that didn't quite add up for me - and it's taken a while to put my finger on it. But before I get to that, there are a few other more obvious points that need to be mentioned.

First of all, the final sample size is very small. This is understandable because, even with a high risk sample, you need to test a lot of babies just to get a dozen or so who develop ASD. But it doesn't change the fact that a study with only 11 participants in one group has to be considered preliminary at best. It wouldn't have taken much to get wildly different results and we really need a replication of this before we get excited.

Second, the most useful comparison would be between the high risk children who develop autism and the high risk children who don't. That's because, in practice, it's very unlikely that such an eye-tracking measure would be rolled out as a universal screening measure.

Not only would that be extremely expensive and time-consuming (especially if babies had to be tested on multiple occasions under highly controlled conditions to work out the slope of their trajectory), it would also throw up a huge number of false positives - babies who the eye-tracking test said were likely to develop autism but were never really at risk. So, in reality, only babies with a family history of autism would be tested. Unfortunately, Jones and Klin don't report this direct comparison [6].

But the biggest issue for me is this: The claim is that babies who develop ASD start off with typical eye gaze but it's the decline in eye gaze in the first 6 months that is the signal of impending autism. However, if the ASD babies start off at the same level as the TD babies then there should also be a significant difference in the amount of eye gaze at 6 months.

In fact, the two groups are almost identical in terms of their eye gaze at 6 months. And the only reason there's a difference in slope is that the ASD babies actually start off with greater eye gaze than the TD babies [7]. 

It would be incredibly interesting if this were true - if boys who go on to develop autism make more eye contact at 2 months than is typical. However, we have to be very careful because, despite the media coverage focusing on the fact that the babies were tested from 2 months, there is in fact very little data from 2-month-olds in either group.

How do we know this?

Figure 1d in the paper includes the data from a single ASD baby. Each dot corresponds to the amount of time spent gazing at the caregiver's eyes in a single video clip. Matching this up with Supplementary Figure 2b, which combines the data from all the ASD babies, we can see that this one boy provides the majority of the data points at 2 months. This in turn suggests that there are only two, perhaps three boys with data at 2 months. [8]

So, to sum up, it's far too early to be declaring that baby's eye gaze predicts future autism. Jones and Klin's study is an extremely impressive effort to answer an important question. It suggests that there is in fact very little difference between babies who do and don't develop autism in terms of their eye gaze (at least when the eyes are on a computer screen).

Future studies beginning even earlier and having even larger sample sizes may uncover evidence of atypical eye gaze - and there's a tantalising and intriguing suggestion that ASD babies may actually begin life with excessive eye gaze. [9]

However, not for the first time - and certainly not the last - the headlines obscure a far messier reality.


Jones W, & Klin A (2013). Attention to eyes is present but in decline in 2-6-month-old infants later diagnosed with autism. Nature PMID:24196715

Other Coverage


1. It's usually the case in these studies that "low-risk" infants have older non-autistic siblings but this isn't explicitly stated.

2. It's not clear why the authors went to the trouble of testing 46 girls for 3 years only to exclude them later. The rationale given was that only 2 developed ASD, but this should have been expected given that autism is much less common in girls. My suspicion is that reviewers made them take the girls out of the analyses - but this is another reason why having an open peer review process is important.

Update 09/12/13: Ami Klin has informed me that more girls are being tested and their data will be analysed when the sample size is sufficiently large.

3. The ADOS was used as a severity measure in one analysis so was presumably included as part of the assessment. Beyond that, I can't find anywhere in the paper that actually says how the the authors determined which kids had autism.

Update 09/12/13: My mistake here. Although in my defence Nature doesn't make things easy by having three variations on the Methods section, each of increasing level of detail. Full details of diagnosis are buried in the Supplementary Information (which is separate from the Methods supplement, which is where I was searching for diagnostic information). In short, diagnosis was a clinical best estimate based on all available information, including ADI-R and ADOS scores (and direct or recorded observation of the ADOS session), language and cognitive assessments, history, and "any other relevant information". Diagnoses were made by independently by two experienced clinicians and positive diagnoses were reviewed by a third clinician.

4. This is based on the lower panel of Figure 2e

5. It's certainly possible that eye gaze discriminates between ASD and TD infants earlier than the 24 months reported and that the lack of statistical significant is just a reflection of the small sample sizes. With more babies tested, perhaps there would be a significant reduction much earlier. But that is speculation based on data that we currently don't have.

6. The high risk kids who didn't meet the ASD criteria were divided further into two subgroups - those who had no diagnosis (noDx) and those were had the "broader autism phenotype" (ie some autism symptoms) but didn't meet the full ASD criteria (BAP). The noDx group appear to resemble the TD group (at least in terms of their trajectory between 2 and 6 months) while the BAP group are intermediate between the TD and ASD groups. Unfortunately, nowhere does the paper say how the BAP was defined or whether the cut-offs were decided upon before the study started.

Update 09/12/13: Again, there were in fact details provided in the Supplementary Information. BAP infants were those for whom there were clinical concerns documented at any one of the clinical assessments. They differed significantly from the noDx group in terms of their ADOS Total scores. It would still have been useful to include a direct comparison between the high-risk babies who did develop ASD and those who didn't.

7. Based on the lower panel of Figure 2e this increase in eye gaze is statistically significant.

8. It seems as though one of the reviewers spotted this issue because the supplement to the paper includes a re-analysis looking at just the data from 3 to 6 months. This suggests that there was still a difference in the slope between the two groups , but this was barely significant. This is based on the Receive Operating Curves (ROCs) in Extended Data Figure 9d, where the 95% confidence intervals include chance at low levels of sensitivity (high specificity) but not a high levels (low specificity).

Update 09/12/13: I hadn't noticed that Extended Data Figure 9a has the trajectories based on data from 3 to 24 months. With the data from 2 months excluded, the early trajectories look very similar. For the ASD group, there's a drop from about 47.5% eye gaze at 3 months to 45% eye gaze at 6 months. The TD group are steady at about 45% throughout. If there is a real difference, it's very subtle.

9. By this I mean "more than is typical".

About Jon Brock

Jon Brock is a research fellow at the ARC Centre for Cognition and its Disorders at Macquarie University in Sydney, Australia. His research focuses on cognitive and neural mechanisms involved in developmental disorders including autism, Williams syndrome, and Down syndrome. Publications can be downloaded here.
As well as this blog, he has also written for the Simons Foundation Autism Research Initiative, the Thinking Person's Guide to Autism, and The Conversation. Translations of some of my posts can be found on the Spanish-language website Autismo Diario.

No comments:

Post a Comment