In conversation: Elevating your headline game with AI
In today’s digital news landscape, publishers have mere seconds to hook readers with a headline. The hook requires finding the tipping point where just enough has been shared to entice a click for more information while leaving a little mystery about the article itself. It’s an art and a science, and we’ve been working on the science.
Chartbeat’s Headline Testing tool has helped publishers optimize for reader engagement for years, and this popular feature is now even more powerful with the addition of AI. Leveraging insights from past content performance, AI-suggested headlines save time and make results more relevant and engaging.
To understand more about the work that went into building the tool and how it’s already helping publishers, we sat down with Elaine Liu, Senior Machine Learning Engineer here at Chartbeat and one of the chief architects of this new feature.
First things first, why do you think AI is well-suited to help publishers with headline testing?
Even before ChatGPT made headlines, AI was widely used in various aspects of publishing, such as understanding trends, categorizing articles, and recommending content to readers. Each of these tasks traditionally required training distinct models, but what makes AI tools like ChatGPT so exciting is their versatility across a wide range of tasks. Given the necessary context, they can do all of the above, and much more.
In the bustling environment of a newsroom, where multitasking is the norm and time is of the essence, this versatility becomes indispensable. Journalists and editors face many challenges and decisions for each story they publish. They need to optimize for search traffic, social media presence, audience engagement and subscription conversion, all while managing the details–the timing of article publication, the choice of thumbnail images and subtle tweaks in headlines–that could dictate the success of a story.
Generative AI brings transformative capabilities to the newsroom because of its ability to synthesize vast amounts of data and text, eliminate the need for a range of traditionally labor-intensive tasks such as translation and open new possibilities for content consumption, like personalized content tailored to individual preferences. This technology empowers newsrooms to do a lot more with a lot less, both in terms of time and money.
Can you describe what’s happening behind the scenes when a user generates an AI headline?
Large language models like GPTs are trained on diverse data sourced primarily from publicly available text on the internet. As a result, they may not naturally align with a specific brand image, style, or content strategy without additional context-specific training and prompting.
When you use Chartbeat’s AI headline generation tool, however, we automatically gather the article’s text, current title and metadata, and then formulate a prompt based on this information. If available, the prompt also includes successful headlines from past headline tests. This technique, known as in-context few-shot learning, is a powerful strategy guiding the model to learn from past examples.
We also employ chain-of-thought (CoT) prompting techniques that mimic the structure of logical human reasoning. Headline writing isn’t a simple one-step task. It requires understanding the article content, identifying key points and the main idea, tailoring the headline to different platforms and more.
Powerful as these large language models are, they still benefit from step-by-step guidance for a more thorough approach. In our prompt, we instruct the model to take on the role of an editor, specifying each step it needs to follow to generate optimized headlines using the strategies learned from previous successful example headlines. This structured reasoning not only enhances the quality of the headlines but also improves their accuracy.
Once you review the AI-suggested headlines, you can select promising ones as headline variants and start the headline tests as usual. The process doesn’t end there—the results of these headline tests feed back into our system, enabling the model to generate even better results in the future.
Were there any hurdles you had to overcome in building this feature?
With all the AI tools at hand, it’s easy to plug in and start generating results. However, this only made us more thoughtful in our approach. One significant hurdle was recognizing we can’t trust the model 100% yet, as headlines need to be truthful about the content of their article, and sometimes the replacement of similar words is just not acceptable.
For instance, in our early iterations, the model once used ‘murderer’ when it should have used ‘suspect’. Such errors could have significant unintended impacts, so we worked hard to minimize these situations by evaluating different models, tuning parameters within each model, and optimizing prompts.
Evaluating the truthfulness and quality of headlines, with all their subtleties, is difficult for models to do programmatically. Therefore, we relied on our early access customers and internal staff with newsroom experience to help evaluate headline accuracy and quality. Their feedback was invaluable in selecting the best model and optimizing parameter settings and prompts.
Would you talk more about some of the early customer feedback you’ve implemented into the version of the product now available for Chartbeat customers?
We took feedback from our early customers seriously and implemented several changes based on their insights. For example, they noticed the AI-generated headlines sometimes focused on minor details rather than the main point of the story. To address this, we instructed the model, as part of the chain-of-thought approach, to better grasp the main points of the story.
Customers also mentioned headlines generated by tools like ChatGPT often followed a repetitive pattern, such as always including a colon. In response, we aimed to avoid repetitive patterns in our tool to offer more diverse headline styles and varied usage of punctuation.
Another area of improvement was capitalization. Our early iterations sometimes didn’t adhere to the correct headline capitalization style, requiring customers to make manual corrections. While this might seem like a trivial fix, it’s nuanced. The model needed to understand when and when not to capitalize. For example, in the case of ‘La Maison in LA’, we needed to instruct it to lowercase the ‘a’ in ‘La’ and uppercase the ‘A’ in ‘LA’.
In each case, we were able to continue iterating our prompts to incorporate newer, more powerful models from our provider to achieve greater consistency and accuracy.
What steps are we taking to be thoughtful about the way we incorporate AI?
Above all, we’re committed to protecting our customers’ data and respecting their copyrights. We use a best-in-class pre-trained model hosted on Amazon Bedrock. Amazon Bedrock doesn’t use our prompts (including your article text and metadata) and continuations to train any AWS models or distribute them to third parties. The model provider doesn’t have access to Amazon Bedrock logs or customer prompts and continuations.
While we strive for a seamless workflow, we recognize AI technology isn’t perfect and shouldn’t be fully trusted without oversight. Given the high stakes of some articles in the newsroom, we emphasize giving control to the newsroom staff. Generative AI isn’t a replacement for human writers and editors; all content generated by AI should be reviewed for accuracy and appropriateness by humans. Instead of automatically deploying AI-assisted headlines, we allow you to modify these suggestions and decide whether the modified headlines should still be considered AI-assisted.
What excites you most about the future of AI for publishers?
What excites me most is AI’s potential to elevate the quality of journalism, and not just increase quantity. While current AI applications in newsrooms often focus on content production and optimization such as tweaking headlines for different platforms or creating multilingual versions, I think the real game-changer lies in AI’s evolving reasoning capabilities and flexible agent-based systems.
These advancements will empower journalists and editors to sift through vast amounts of information from diverse sources and identify critical trends and connections that might otherwise go unnoticed. This technology could revolutionize everything from breaking news coverage to long-form investigative reporting and audience engagement management.
The true winners in this AI-driven future will be publishers who harness these tools not merely as productivity boosters, but as catalysts for delivering deeper, more timely, and more impactful content than ever before.
Interested in AI-suggested headlines? Reach out to your Customer Success Manager or schedule time here to find out more about Chartbeat’s full suite of optimization tools.
