===============================
Route: This research topic can be taken by essay-route student.
- NOTE: if project-route student wants to take this topic, then further designing/refining algorithms and implementing features for Authorship Forensics is needed based on the results of the comparative study.
- NOTE: if thesis-route student wants to take this topic, then besides the algorithm design/refinement and feature implementation an extra post-evaluation after is needed.
===============================
In this evolving landscape of text generation, distinguishing between human-written and ChatGPT-generated content has become increasingly important. Our research group has analyzed and developed a novel approach to authorship attribution, leveraging both Statistical Natural Language Processing (SNLP) and Convolutional Neural Networks (CNN) techniques to differentiate between documents written by humans and ChatGPTs. The research uses 212 abstracts of academic papers written and published by the research group as the human-written set and asks both ChatGPT 3.5 and 4 to generate correspondent abstracts based on paper titles as AI-written set. Models are trained to classify human and AI-written abstract texts in 2-class (i.e., human and ChatGPT) and 3-class (i.e., human, ChatGPT 3.5, and ChatGPT 4) based on their part-of-speech tag frequency distribution patterns. The 2-class model is well-trained in less than ONE minute (i.e., 56.82 seconds) and the 3-class model is well-trained in 7 minutes and 26.076 seconds. The results demonstrate a significant ability of the models to distinguish between human and AI-written text, with precision 0.9682 (F0.5 score 0.95) for the 2-class (human and ChatGPT) testing subset and precision 0.9806 (F0.5 score 0.96) in the 3-class (human, ChatGPT 3.5, and ChatGPT 4) testing subset. The proposed 3-stage Authorship Forensics approach has also been implemented as an open access web application (https://ngrampos.vipresearch.ca/) to allow teachers and users to either training their own models or using the existing trained model to get some advice on how the model considers a piece of given text is written by human or AI.
The results show that the computer can be trained quickly with acceptable amount of data like existing research use and the trained model can effectively differentiate texts from an unseen subset in terms of human or ChatGPT-written. While the trained model’s performances have outperformed almost all existing research, the research team is still planning to work more and deeper on examining the 3-stage method, aims to clear teachers doubts on using the Authorship Forensics and make them have faith on the GenAI-written detection results.
The research topic is going to collect the results (e.g., predictions or classifications as well as confidence percentages and other values or info an online detector provides to support its conclusion) on the 636 abstracts in Excel sheets the following online detectors have to further compare Authorship Forensics’ performance with them. Please collect data from the ones have no registration required and free first, then ones are free but require free registration, then the ones have pay-as-you-go fee, and last the ones have subscription fee. Considering to include such information in the Excel sheets. Other information like whether the online detector provides multiple document upload and test feature could also be noted in the Excel sheet so you can discuss more in details while comparing them in the report.
- Writer.com AI content detector (https://writer.com/ai-content-detector/),
- GPTZero (https://app.gptzero.me/app/ai-scan),
- GPT-2 Output Detector (Huggingface OpenAI detector, https://openai-openai-detector.hf.space/),
- AI Checker (https://aicheckertool.com/),
- KazanSEO AI GPT3 detector (https://kazanseo.com/detector),
- AICheatCheck (https://aicheatcheck.com/),
- ZeroGPT (https://www.zerogpt.com/),
- QuillBot (https://quillbot.com/ai-content-detector),
- Scribbr (https://www.scribbr.com/ai-detector/),
- Sapling (https://sapling.ai/ai-content-detector),
- Grover AI detector (https://grover.allenai.org/detect),
- Unfluff (https://unfluff.io/),
- Originality.ai (https://Originality.ai)
The comparison (to Authorship Forensics models) and discussion based on the collected data analysis should include but not limited to:
- The differences on the measures include true positive, false positive, false negative, true negative, precision, recall, F0.5, F1, and F2 among different online detectors and different Authorship Forensics models (i.e., there are 10 AF 2-class models that use 60% of the 636 abstracts as training set, 10 AF 2-class models use 70% of abstracts as training set, and 10 AF 2-class models use 80% of abstracts as training set.)
- If an online detector provides supports like confidence value for its conclusion or prediction, changing the threshold like only accept the conclusion/prediction class (e.g., Human and Computer) when the support value higher than 0.5, 0.6, 0.7, 0.8, etc. Explaining the influences that the support value threshold can play in terms of having higher or lower true positive, false positive, false negative, true negative, precision, recall, F0.5, F1, and F2.
- Adopting the 10 AF 2-class models as a judge panel and changing the vote threshold to see how the number or percentage of votes could influence the prediction result more or less accurate in terms of true positive, false positive, false negative, true negative, precision, recall, F0.5, F1, and F2.
- Digging into the relations between an AF 2-class model's reputations (i.e., based on either its precision, recall, F0.5, F1, or F2 value) and its vote among its fellows in the judge panel. Explaining the possibility of having majority of smaller number of high reputation judges would perform better than having 10 2-class model judges whose reputations are diverse in a vary range. If so,
- What value(s) (or combination of values) could be considered as an AF 2-class model's reputation?
- What threshold value(s) a AF 2-class model has could make it be considered as a high reputation judge?
- What size a judge panel should consider to be formed with those high reputation judges?
- What percentange of votes from the judge panel that is composed of high reputation judges should be considered as collective decision?
Sincerely,
Maiga Chang
--------------------------
Pronouns: he/him/his
IEEE Senior Member
Full Professor & Associate Dean, Research & Innovation, Faculty of Science and Technology, Athabasca University, Canada
Honorary Chair Professor, Multidisciplinary Academic Research Center, National Dong Hwa University, Taiwan (2023~2024)
IEEE Computer Society Distinguished Visitor on AI and Chatbot (2023~2025)
Distinguished Researcher Award 2022, Asia-Pacific Society for Computers in Education (APSCE)
Vice President, International Association on Smart Learning Environments
Editors-in-Chief, ET&S (SSCI)
Editor-in-Chief, IJDET (ESCI)
Editor-in-Chief, Bulletin of TCLT (ESCI) |
