Using ChatGPT for analysis of product feedback (and a warning)

With customer surveys and questionnaires, often the most interesting insights come from the qualitative feedback in written text responses.

However when you have lots of data this can be really hard to parse. Open survey responses can’t be easily interpreted or turned into simple charts.

As a product manager, there’s a skill in being able to ingest lots of qualitative feedback, comprehend it, and communicate it to others. Much of it relies on experience and instinct, and it’s a critical part of understanding your users and customers, and in this case backers.

Kickstarter project feedback

A couple of questions I usually ask backers of my Kickstarter projects:

Why did you decide to back this project?
Any suggestions on what we can do to improve our product?

In some cases I’ve had well over a thousand responses so that’s a lot to read through line by line.

Simply sorting the responses alphabetically throws up some patterns and common points, but beyond that it’s really hard to read through them all and take them in, and frankly some of them are gibberish.

Here’s where I enlisted ChatGPT for help.

ChatGPT

My recent project for a deck of mini multi-game playing cards had 3,273 backers and plenty of them responded to my questionnaire.

I decided to use ChatGPT to go through all those free-text responses and summarise them for me.

Method

The prompt I used was pretty straightforward. As shown below I asked the model to adopt a persona by using the SYSTEM; USER syntax.

Here’s my prompt:

SYSTEM
The following are responses from backers of a Kickstarter project to the question: “Why did you decide to back this project” (delimited with XML tags).
Read each of them and summarise them all together into one set of general customer sentiments.
USER
<response1>Love concept</response1>
<response2> if u will.</response2>
<response3>A 5(or so)-in-1 game in your pocket. That seems pretty cool to me.</response3>
<response4>A brilliant compact idea. Would make a great gift too. I am a Geek</response4>
<response5>A compact way to play traditional games is great for traveling.</response5>
…
etc

The original survey responses were in a single column in a spreadsheet, so used the CONCATONATE function to add numbered XML tags to each row. I could then paste them into ChatGPT.

`Combining using the CONCATONATE function`

The response

Here’s what ChatGPT came back with:

The backers of this Kickstarter project express a wide range of sentiments, but some common themes emerge. Many are drawn to the compact size and versatility of the product, seeing it as ideal for travel, camping, or as a convenient way to carry multiple games. Others appreciate the novelty and innovation of the concept, seeing it as a cool idea or a unique addition to their collections. Some backers are attracted to the affordability of the product, while others see it as a great gift for friends or family who enjoy gaming. Overall, the sentiments convey excitement about the product's potential for fun and entertainment in various settings.

I then asked for bullet-points to make it easier to read:

- Backers are drawn to the compact size and versatility of the product.
- Many see it as ideal for travel, camping, or as a convenient way to carry multiple games.
- The novelty and innovation of the concept attract several backers.
- Some view it as a cool idea or a unique addition to their collections.
- Affordability is a factor for some backers.
- Many backers see it as a great gift for friends or family who enjoy gaming.
- Overall, sentiments convey excitement about the product's potential for fun and entertainment in various settings.

Was it useful?

Yes, I think so.

Having read through many of the responses manually I’d say that’s a pretty good summation.

Perhaps there’s nothing entirely surprising in there, but that’s OK, and there were some good insights.

These responses of course came from people who chose to back my project, so there’s some positive bias in there. Asking other people why they didn’t back my project would be perhaps more interesting, but almost impossible to do.

Knowing the positive things that people have reacted to is useful because I can take those sentiments and use them to appeal to similar people when I’m coming up with marketing messaging.

Responses to the second question, “any suggestions on what we can do to improve our product?” are useful in a different way and will help me with future product development.

A warning

As always when using ChatGPT, you have to be careful.

When summarising responses to that second question, ChatGPT came back with a very nicely laid out list. Five numbered top-level points:

1. Durability and Material Quality
2. Game Expansion and Instructions
3. Design and Customization
4. Packaging and Storage
5. Accessibility and User Experience

With sub-points within them. Expanding on that last section, here’s what ChatGPT said:

5. Accessibility and User Experience:
- Ensure card numbers and symbols are large and clear for easy visibility.
- Explore options for large-print editions for visually impaired users.
- Ensure the design accommodates colorblind players with clear distinctions.

Those are all good points, and valid things to consider when it comes to inclusive design.

But no one said any of that.

Arguably it’s a good thing that ChatGPT more broadly addresses issues of inclusivity that may counteract other biases, but given my specific requests based on a closed data set, I think this is a bad performance from ChatGPT.

The lesson here is to make sure you sense-check everything that comes back from ChatGPT. Blindly following the summary of qualitative survey responses is as bad as blindly following the data of a quantative survey when you don’t fully understand the context of the data.

Other uses of this methodology

I did a similar exercise looking at Amazon reviews for competitor products.

In that case I pulled reviews out manually from Amazon product pages, but doing that programmatically would be much more efficient, and more scalable.

…

The prompt syntax I used here was something I looked up quite a few months ago so there may well be a better way to do this. Please let me know or leave a comment if you have any suggestions.

Furthermore, if you have any questions or suggestions for subjects you’d like me to cover in the future, or would like help with your own Kickstarter project, please feel free to get in touch.

If you’d just like more of this content there’s plenty more here, and you can sign up below to my newsletter.