On the difficulty of Autism diagnosis: Can we plot this better ?

[ New Update: see here on the Challenge, results and survey]

[Update: I have put some update in the text of this entry to reflect my better understanding of the graphs which doesn't change the graph itself]

When I met Catherine Lord a year ago, I was struck by this graph in her presentation (excerpted from "Autism From 2 to 9 Years of Age" [1]):

How can one understand it ? each circle denotes a set of people that have gone through one test designed to figure out whether or not kids were affected by Autism (PL-ADOS indicates Pre-Linguistic Autism Diagnostic Observation Schedule; ADI-R, Autism Diagnostic Interview–Revised and Clinician indicates that a clinician made an assessment based on an interview with the kid). Intersection between circles points to populations of kids that have gone through several tests [Update: and tested positive at these tests]. The intersection of all three circles indicate people that have gone through the three tests (clinician, ADI-R, PL-ADOS) [and tested positive at these three tests. A kid could conceivably be tested positive on the three tests and not appear in either graph A or B]. The number in the circle indicates the number of kids at age 2 that have been deemed Autistic (left circles) or in the Autistic Spectrum (right circles) testing associated with each circle (Clinician, ADI-R, PL-ADOS). The number in the parenthesis indicates the number of kids that have been diagnosed with the same diagnosis at age 9. Let's take an example: in the first three circles in the left, there is a label indicating: 16 (56%). This can be translated into: [Update: There 16 kids that were diagnosed with Autism at age 2 AND] they were also diagnosed with autism using ADI-R test, yet 56% remained in that diagnosis at age 9. [Update: One more thing: The category Autism [A] and ASD [B] denote kids that have been deemed Autistic or with ASD through a Best Estimate method at age 2, this method use several means to come to that conclusion. The percentages in parenthesis denote whether these kids are still Autistic or with ASD seven year later using a Best estimate method at age 9]. Some of these numbers are simply stunning because they show our current inability to do a good job a determining what constitute reliably autism at age 2. This is all the more important that an earlier diagnosis is really needed to probably change the outcome of this condition. Catherine and her co-workers eventually comment in the paper:

Diagnosis of autism in 2-year-olds was quite stable up through 9 years of age, with the majority of change associated with increasing certainty of classifications moving from ASD/PDD-NOS to autism. Only 1 of 84 children with best-estimate diagnoses of autism at age 2 years received a nonspectrum diagnosis at age 9 years, and more than half of children initially diagnosed with PDD-NOS later met autism criteria. Nevertheless, more than 10% of children with diagnoses of PDD-NOS at age 2 years received nonspectrum best-estimate diagnoses (ie, not autism or ASD) by age 9 years, and nearly 30% continued to receive diagnoses of PDD-NOS,indicating mild symptoms at age 9 years. A significant minority of children with milder difficulties within ASD at age 2 years showed only mild deficits in the clinical ASD range at age 9 years. Classifications changed substantially more often from ages 2 to 5 years than from ages 5 to 9 years. The bulk of change in diagnosis occurring in early years is consistent with another recent study. At age 2 years, diagnostic groups were more similar in functioning and IQ than the diagnostic groups identified at age 9 years, when the autistic group showed very poor adaptive functioning and the PDD-NOS group, much less abnormal verbal and nonverbal IQ. Among this specialized group of clinicians, clinical judgment of autism at age 2 years was a better predictor of later diagnosis than either standardized interview or observation. Contemporaneous agreement between clinical judgment and best-estimate judgment for 2-year olds was equal to that found between experienced raters in the DSM-IV field trials for older children and adults. Though the clinical diagnoses at age 2 years were made without knowledge of the ADI-R and ADOS algorithm scores, each clinician had administered either the PL-ADOS or the ADI-R and had the opportunity to discuss his or her impressions with the experienced clinician who had administered the other instrument. Thus, the information available to them was very different from the information obtained during a typical single office visit to a clinical psychologist or developmental pediatrician. The use of standardized measures seems likely to have improved the stability of diagnosis both directly through straightforward use of algorithms for autism and ASD and also indirectly through structuring clinical judgment. Of cases in which the classifications yielded by both instruments were not supported by the clinicians at age 2 years, 40% were children with severe mental retardation (and not autism) or children with very difficult behavior (and not autism), while the remainder were mild cases of autism characterized as uncertain. On the other hand, clinical judgments were consistently underinclusive at age 2 years, both for narrow diagnoses of autism and for broader classifications of ASD at age 9 years. Thus, scores from standardized instruments also made real contributions beyond their influence on informing and structuring clinical judgment. Overall, while standardized research instruments at age 2 years did not fully capture the insight in the form of certainty ratings made by experienced, well-trained clinicians, this insight was not by itself sufficient.

My main problem with this graph is that it does not make sense right away. Even though I have been thinking about it for a while, I still cannot get around a better way of displaying this data. This assessment over time is unique in the annals of Autism studies and hence a major milestone. I wish it were better designed to convey some of its underlying statistics or lack thereof. Examples shown by Andrew Gelman in his blog or Edward Tufte maybe a good starting point. In particular, I am wondering how one could adapt the graphical on cancer survival rate redesigned by Tufte to this study. Initially the Lancet study on cncer survival rate showed this graph:


Tufte redesigned it to a stunning comprehensible table


Can we do a better plot of the Autism study ?
[ Update 1: Andrew has posted the beginning of an answer here ]



[1] Autism From 2 to 9 Years of Age, Catherine Lord, Susan Risi, Pamela S. DiLavore, Cory Shulman, Audrey Thurm, Andrew Pickles. Arch Gen Psychiatry. 694-701, VOL 63, JUNE 2006