We did a (mini) research study on ChatGPT

Breaking the list of writing-related posts to share a research study that my classmates and I did recently.

If you have seen my FB posts in the past few months, you would have noticed that I have been all fuddy-duddy about ChatGPT – worried about what it will mean for the literary submissions world, worried about the paradigm shift for content writing jobs in public relations, copywriting, translation, and the like.

So for our final project in Research Methods & Practice, a group of us set out to understand how people perceive ChatGPT vis-a-vis common concerns expressed in mass media.

Before you read further, a caveat that our survey may not be statistically significant due to a small sample size of 67. However, we discovered some surprising insights that may be helpful as a starting point for bigger research studies into ChatGPT.

First, the common concerns mentioned in mass media:

  1. That ChatGPT will replace jobs.
  2. That people will use ChatGPT to cheat academically.
  3. That ChatGPT is not accurate when it comes to critical, important information like legal advice, medical advice, information about public persons and organisations, among others.
  4. That ChatGPT may not adequately protect information that you feed into it.
  5. That ChatGPT was trained on data and writings obtained without permission.

Survey Design & Demographics

My classmate did a very detailed dive into how our survey was designed and its current limitations, which you can check out in our presentation slides 19 to 36.

Now to our survey results. In terms of demographics, here are the basics:

Majority are between 25 – 35 years old.
We had an almost even split between males and females, with 2% non-binary.
We had an almost even split between students and working professionals.

Our Hypotheses

I want to share the three hypotheses we had before we conducted the survey. Prior to the survey, we hypothesised that:

  1. Most people who have concerns about ChatGPT have not used it before, only heard of it.
  2. Unemployment and unauthorized use of personal data (shared with ChatGPT) will be the most common concerns about ChatGPT.
  3. Females have more concerns about ChatGPT compared to other genders.

Note: Why females, you may ask? This was a conjecture on our end that more women work in fields like public relations, content writing and translation, and therefore they would be the gender that is more worried about ChatGPT.

Interesting Findings

1. 75% of surveyed believe that ChatGPT is an AI tool that can help them:

(This was actually surprising to me, because I had assumed most people have concerns about ChatGPT.)

2. Compared with working professionals, students are more concerned about the information accuracy of ChatGPT:

Note: I didn’t change the legend before uploading sorry, but 1 is Strongly Agree and 5 is Strongly Disagree

3. Majority of respondents will feel secure in using ChatGPT ‘if there is more regulation about data collection and privacy’.

Our Findings – Hypothesis 1

Hypothesis 1: “Most people who have concerns about ChatGPT have not used it before, only heard of it.”

We found that on a whole, people who had not used ChatGPT were not more concerned than people who had used it before. In fact, concerns hovered around the same percentage for both groups with two exceptions:

  • People who had not used ChatGPT before were slightly more concerned (40% versus 32%) about academic cheating.
  • Majority of people who had used ChatGPT before (70% versus 53%) were concerned about the accuracy of ChatGPT’s information.

Our Findings – Hypothesis 2

Hypothesis 2: Unemployment and unauthorized use of personal data (shared with ChatGPT) are the most common concerns about ChatGPT.

Contrary to our hypothesis, we found that respondents did not consider job replacement or unauthorized use of personal data (shared with ChatGPT) to be big concerns. Instead, we discovered that people are more concerned about the accuracy of the information ChatGPT gives, as well as whether ChatGPT has trained itself on writings obtained without permission.

Our Findings – Hypothesis 3

Hypothesis 3: Females have more concerns about ChatGPT compared to other genders.

We can see that it is roughly equal or that women are not more concerned. If anything, this actually proves that our hypothesis was not just wrong, it was the direct opposite! Men are generally more concerned than women on various considerations of ChatGPT. PS: Also a reminder that we had an almost equal representation between male and females for our respondents.

Conclusions & Thoughts

The biggest surprise: What really surprised us was that the biggest concern about ChatGPT was not data collection, not about the loss of jobs, not about academic cheating – but rather, the accuracy of the information ChatGPT spits out.

Upon reflection, I suppose other concerns – loss of jobs, data collection etc – can be rather subjective, so there isn’t a grand consensus reflected in the survey. Whereas the accuracy of information is something that most can agree is an area of concern.

Unexpressed data: Using pivot tables, we were able to see concerns sorted by working professionals versus students; age range; knowledge of ChatGPT etc. (Pivot Tables are amazing). We didn’t include them in our presentation because our presentation was already a behemoth as it was. But if you want additional data, let me know.

Future surveys? We have concluded this module, but if I were to do this survey again, I would love to survey by schools (Emerson College, which has a liberal arts slant, may have very different responses compared to a school like MIT) and/or working industries (people in public relations may view this differently from a content writer or influencer).