ChatGPT couldn’t answer this question about Star Trek stars

ChatGPT knows a lot about Star Trek, but not everything.

As I sat down to research my recent post in honor of Carol Burnett’s 90th birthday, I thought, “Eleven seasons of The Carol Burnett Show? That’s a lot of credits to comb through in search of her guest stars who also appeared on Star Trek.” Then I thought cross-referencing the two series’ credit lists might be the perfect job for ChatGPT.

The generative AI has caused so much hullabaloo in so many industries since it debuted late last year. “So maybe it can pass the bar exam,” I muttered to myself. “So maybe it can draft computer code, tutor someone to write like Hemingway, or even write sermons. But what does it know about Star Trek?”

Obviously, as ChatGPT told educator and entrepreneur Michael Vanderpool when he interviewed it for a LinkedIn post, the AI has “access to a large amount of information, including popular culture, and Star Trek is no exception.” But when I tried to drill down into the details of whether and how The Carol Burnett Show and Star Trek: The Original Series, what happened?

“Which Star Trek actors also appeared on The Carol Burnett Show?” I asked ChatGPT-3.5. To my surprise and initial elation, it told me not only Leonard Nimoy, whose walk-on as Spock I already knew about, but also James Doohan, William Shatner, George Takei, and Nichelle Nichols had also been on Burnett’s show. What’s more, ChatGPT claimed all of them “appeared in several sketches.”

Then I noticed something ChatGPT said about Nimoy, the one actor I already knew had been on Burnett’s show: “Nimoy played Spock in Star Trek and appeared on The Carol Burnett Show in a sketch called ‘Star Trick’ where he played a character named Mr. Spocko.”

“That’s not right,” I thought. “And how is it I, a Star Trek fan who’s watched hours of Carol Burnett reruns, never knew Nichols, Takei, Doohan, or Shatner, for the Great Bird’s sake, appeared in ‘several sketches?’”

ChatGPT claimed several Star Trek stars appeared on Carol Burnett’s show

I started pressing ChatGPT to prove its claims, one by one. I asked it for sources regarding Takei’s ostensible appearances. The bot immediately apologized and retracted its statement.

Fascinating. I then asked the bot to tell me when, exactly James Doohan had been on The Carol Burnett Show. It confidently told me about three sketches he was in, including the aforementioned “Star Trick” parody, but it now told me Tim Conway had played “Mr. Spocko” in that skit. “Doohan may have made additional appearances on the show as well,” ChatGPT added with the cyber equivalent of a straight face.

I demanded sources. It gave me four. None of them checked out. When I questioned the sources’ validity, it acknowledged its error, but then gave me three “more reliable sources,” including this:

"A TV Guide article titled “Carol Burnett Show Clip: James Doohan Goes From Star Trek to Submarine Service” discusses Doohan’s appearance in a sketch on the show: https://www.tvguide.com/news/carol-burnett-show-clip-james-doohan-star-trek"

Hmm, pretty specific. But if you visit that link, you’ll see what I saw: A TV Guide website error message. ChatGPT apparently made up this “more reliable source” out of the ether.

I debunked its two other “more reliable sources” in short order. At first, ChatGPT tried to defend itself: “To clarify, I do not have direct knowledge of James Doohan’s appearances on The Carol Burnett Show, as my training data only goes up until 2021. However, my previous responses were based on information from sources that I believed to be reliable.” But in its very next paragraph, it admitted it “was not able to find any reliable sources that definitively confirm James Doohan’s appearances on The Carol Burnett Show.”