Does 'Maintenance of Competence' Testing Make Us Smart?

In "Things That Make Us Smart," Don Norman, a well-known cognitive scientist, identifies two kinds of learning and two life situations in which learning may be applied. Learning to ride a bike, play the piano, fly an airplane, or perform surgery involves primarily training our reflexes by repetition. This is kinesthetic learning. Learning differential diagnosis, renal physiology, or higher mathematics are cognitive processes that require reflection to discern the subject's essential patterns and relationships and to internalize them.

Some activities obviously require one type of learning; for others it may not be so clear. Consider an airline pilot faced with an impending mid-air collision. The following anecdote is from a blog written by Gianni Reina.

"It was a quiet day at FL350 along the eastern seaboard. The first officer and captain were enjoying the early morning sunrise… [Suddenly] the TCAS warning shouted, 'TRAFFIC! TRAFFIC! CLIMB! CLIMB!' The first officer grabbed the yoke, disconnected the autopilot and prepared to climb the aircraft just when the captain yelled, 'My aircraft!' and pushed the aircraft nose over- in defiance of the TCAS warning...

"Now, everyone who has ever had TCAS training has been told you 'never disregard a TCAS warning!' You always follow the commands given by the unit. But this time the captain ignored the warning and did the exact opposite of what the TCAS was commanding. Why?

"...The answer is because the captain knew how this system of his aircraft worked, and he knew it was giving him bad information. The aircraft threat was above his, descending, the TCAS was giving him the command to 'CLIMB! CLIMB!' As we discussed, this alert should mean the threat aircraft is below his. This isn’t the case, though, so the TCAS warning is actually inaccurate. The correct warning should have been, 'CLIMB! CROSSING CLIMB!' The captain, knowing his system, being situationally aware and cognizant of the threat vector, correctly disregarded the TCAS warning and descended- thereby avoiding the collision."

Repeated practice in the flight simulator trains the reflexes to instantly respond to the alarm and not get paralyzed by indecision. The cognitive training of the pilot in this situation allowed him to take additional factors into account and override the automated instructions.

Now imagine that this situation had been a question on a recertification or "maintenance of competence" exam. How do you respond? Like the co-pilot with the "official" supposedly correct answer or like the pilot who knows or suspects that there is more to the question than meets the eye.

Judgment is essential for good patient care. There are many diagnostic and treatment options. Some are better than others, but who knows for sure? Many medical conditions are as yet understood poorly or not at all. How do you assess the risk/benefit of a test or treatment in a condition that remains poorly understood? Even common conditions like asthma are the subject of continual debate; definitive knowledge is simply lacking.

In some areas like molecular genetics, medical knowledge is changing at a prodigious rate. In other areas, a considerable portion of what passes for medical knowledge isn't, or will prove not to be. Keeping up with all this change in real-time does not often affect how one treats patients today.

The problem is that we would like to be able to definitively identify physicians who are unqualified but we don't really know how to quantitatively rate physicians' qualifications. But (it is said) something has to be done! What will it be? I would favor testing judgment, but that's harder than testing for "facts" - especially with a multiple choice test. The truly knowledgeable test-taker is often forced to play a "guess what I was thinking when I wrote this question" game; guess wrong and you lose. The more you know, the more likely you are to have trouble with the question. The successful test-taker, like the co-pilot, responds by reflex; taking time to reflect may save the plane, but it can cost you the test.

The hope is, I guess, that more unqualified than qualified physicians will flunk or simply give up on re-certification. It can then be proudly announced that the "bad ones" have been "weeded out" and that the public and regulatory agencies may rest easy. This looks like another example of our compulsion to regulate getting ahead of reason and science.