March 2024 board member spotlight
The following was published in the March 2024 AOBFP newsletter.
By Katherine E. Galluzzi, DO, CMD, FACOFPd.
Professor and Chair, Department of Geriatric and Palliative Medicine
Philadelphia College of Osteopathic Medicine
I graduated from medical school in Lewisburg, West Virginia (WVSOM ’84) at a time when computers were much more than an idea promoted by the 19th-century English mathematician Ada Lovelace, but still only available in the rarified air of research universities, governmental or business arenas.
As a high school senior in 1971, I was fortunate to see one of the first fully functioning research computers while on a nuclear research orientation week at the Argonne National Laboratory. I distinctly recall being led into a hermetically sealed space the size of an office conference room, and immediately feeling and hearing the hum of electricity coming from the massive metal box in the middle of it. The computer occupied most of the room! But it was capable of doing unimaginably complex computations and analyses that not even my new Texas Instruments hand-held SR-10 calculator could touch.
I relate these facts not to reveal my age (a lady never discloses that!) but, rather, to remind us of the simple fact that “we’ve come a long way, baby!” Yes, we have.
As a junior member of the AOBFP (we called ourselves “DITS” – directors-in-training) I argued that we should move beyond paper and pencils to compile and edit test questions. I felt that a swift transition to using computer programs to store and evaluate items was in order. I had seen the utility of these programs as an item writer and test construction committee member for COMLEX. It took a while for AOBFP to migrate to computerized question banks, but once it happened the logic was undeniable: computers could really help us get this job done!
AI enters the scene
Now we face a new challenge. A quick Google search enables us to find “information” on just about anything. Or better still – just ask Siri. A quick Google search also reveals numerous “ChatCPT detector” programs designed to identify works written by bots rather than humans which are also able to discern whether plagiarism has occurred.
Remember when you used to have to buy records to listen to the music you loved?
Artificial Intelligence is here to stay. Siri knows and “understands” my musical preferences and so curates the playlist for me. Or someone does . . . one wonders what significance the term “artificial” confers when, in fact, these search engines/programs can perform feats beyond the ability of most human minds. As you might infer from reading this, I am a writer. And I confess that I have experimented with AI writing programs to see what they can do. “Write it in the voice of Barack Obama” I might ask, and inevitably there will be a “Look” or “Listen” comment leading into an important statement. Write it in my voice? Lots of exclamation points, no doubt!
This doesn’t scare me per se, but it does give me pause. How do we ensure that the questions that we ask on high-stakes board examinations are valid, well-written, and – frankly, human? How do we make certain that we have created an examination that analyzes not just the test-taking ability of a given physician, but rather, their knowledge base, judgment, and perception?
LOL! I hope you weren’t expecting an answer to that question. Instead, I am asking you to consider your own strengths and specific areas of expertise and to consider becoming a Subject Matter Expert and/or Item Writer for the AOBFP certification exam. You already know that we are the ONLY board that requires a performance exam for certification of OMM skills. This highly personal, face-to-face clinical encounter allows us to not only assess one’s knowledge of anatomy, physiology and differential diagnosis. It tests how one actually performs a hands-on treatment, and as part of that encounter, the examiners also gain insight into your level of comfort, experience and professionalism. Eye contact!
Asking the tough questions
Can we ever hope to approach that level of insight on a 200-item written examination?
We can certainly try. This takes a dedicated group of accomplished clinicians who are experts in specific areas and who are willing to ask – in written form – the tough questions that we encounter in our practice of medicine.
Item writing, like all good writing, is an iterative process. You have to start with a problem, one which ideally has been researched (PubMed is a better place to start than Google since we need valid, recent references), and develop the case scenario with just enough (not too much!) information to allow the examinee to recognize the diagnosis, and then ask the question: which one of the following is the best next step, or appropriate medication, or plan of care. Next, the item writer must provide, in addition to the reference, a valid rationale for why the correct answer is right and why the distractors are all wrong.
It sounds easy, but it isn’t because, like I said, the process is iterative. The item must be vetted, edited, torn apart at times and at others, simply discarded. This is not, therefore, a process for those weak-at-heart or even mildly thin-skinned. We learn from doing, we learn from repetition, and we learn from each other. It just takes time and an open mind. Personally, I think this is one of the funnest things that we do: Learn from teaching and learn from doing.
Okay, so I know that things like LOL and other acronyms, and terms like “funnest” (it’s not a thang, really) won’t make it onto the exam. But some of your ideas and questions might.
I don’t know about you, but when I go to see a doctor, I hope that I will be met and examined by a human being. To that end, I hope that the questions that we ask on our board examinations will continue to reflect our unique osteopathic perspective and philosophy. I hope that they remain as human as humanly possible.
Will you join us?