Testing ChatGPT’s Boston IQ

ChatGPT is only becoming more ingrained in our society.

Why it matters: The chatbot is the most advanced AI writing tool to hit the market, Axios' Erica Pandey, Dan Primack and Ina Fried report.

• The free (for now) service offers seemingly endless possibilities, from answering simple questions to writing essay-length responses to any prompt you can think of.
• Teachers are using ChatGPT for lesson planning, and Slack is rolling out a ChatGPT bot.

We decided to test ChatGPT’s Boston knowledge, starting with the MBTA.

Asked to explain the city’s public transit system, ChatGPT called the MBTA the oldest public transit system in the nation (true) and the largest (false, that’s the NYC subway).

• It correctly named the cities served by the MBTA, but didn’t acknowledge the Silver Line’s existence, even when mentioning 171 bus routes.
• What ChatGPT didn’t say is that the service has deteriorated in recent years due to insufficient funding and maintenance, among other factors. Nor did it mention that the MBTA was the subject of the Federal Transit Administration’s second-ever review of a local transit system (the first was the Metro in 2015).

But what Boston vocab does the AI know? It mentioned “frappe,” “wicked” and “packie,” and every tourist’s favorite catchphrase: “pahk the cah in Hahvard Yahd.”

We followed up with a question about the Celtics’ chances of winning the NBA Finals, but the chatbot’s talents don’t include fortune telling — so it declined to make a prediction.

• Asked to describe the Celtics to basketball noobs like ourselves, the AI notes that the team is a "highly respected and successful basketball team with a rich history" and a "passionate and loyal fan base."

Boston's pros, per ChatGPT, are the museums, historical landmarks, elite universities and famous sports teams. The AI also gave a shout out to the city's Italian food, Beacon Hill's iconic brownstones and the harbor's beauty.

And the city’s drawbacks? Traffic, high cost of living and inequality.

We asked where to get clam chowder.

• The AI made six recommendations: Union Oyster House, Legal Sea Foods, the Barking Crab, Yankee Lobster Co., Atlantic Fish Co. and Boston Chowda Co.
• Not bad, but we’d argue Ned Devine’s and the Boston Sail Loft belong on this list.

Finally, we asked the AI to write a haiku about Boston. Here’s what ChatGPT delivered:

Amidst the harbor,

Boston's history and charm shines,

A city so grand.