Expert Blind Spot and the Risks of Informal Design

[gusl]

from Ken Koedinger's statement on research goals:

Expert Blind Spot and the Risks of Informal Design

The contrast between educator predictions and the students’ actual difficulties is an instance of a more general phenomenon I call "expert blind spot". As we develop expertise, for instance with algebraic symbols, we lose the ability to accurately introspect on problem difficulty from a student’s perspective.
Consistent with this notion, we found that teachers with the most experience (high school math teachers) are the most likely to make the incorrect prediction that students would find the given word problems more difficult than the analogous equations. Middle school and elementary teachers are successively less likely to make this incorrect prediction, though they still do so in large numbers (Nathan, Koedinger, & Tabachneck, 2000, Nathan & Koedinger, 2000a).

I have often observed the expert blind spot phenomenon when educators have informally evaluated our intelligent tutoring systems. In some cases, they comment on how inflexible the tutor is, for instance, not allowing them to go on before fixing an error. Or, in the case of my ANGLE geometry tutor, some have been impressed with its flexibility. However, empirical studies have shown that what is inflexibility to an expert is support for the novice, for example, novice students learn more efficiently with immediate feedback while it feels constraining to experts. Conversely, what is flexibility for the expert can be confusing for the student, for example, students showed greater signs of floundering in ANGLE than in the earlier, less flexible, geometry proof tutor (Koedinger & Anderson, 1993a). More generally, multimedia, animations, and educational games have appeal because they often appear to make difficult ideas clear and lively. However, this clarity is much more apparent to an expert who already knows the difficult ideas being illustrated. Empirical studies have shown, for instance, that students do not always learn what we expect from games purported to be educational (e.g., Miller, Lehman, & Koedinger, 1999).

The expert blind spot phenomenon has important implications not only for educational technology and how it is evaluated (mostly it is not -- see the November, 1997 Consumer Reports for a pathetic example), but also for software evaluation more generally. Software designers who are expert programmers or experts in the end-user domain are susceptible to expert blind spot and are likely to create applications that are intuitive and easy for them to use, but unintuitive and hard for learners who are novices in the interface or the content domain. Because of expert blind spot, seat-of-the-pants interface design is not likely to work and empirical studies of users are critical.