Critical Thinking Audit – Findings Report
Prepared for: Sample University · Prof. Sample · AFST 210 Environmental Racism (sample course) · Fall 2026
Key findings
Completion
25 of 31
81% of assigned completed both rounds
Overall AI Gap
31.2%
+17 points High
Most affected domain
Explanation & Metacognition
AI Gap 46.7%
Least affected domain
Inference & Problem-Solving
AI Gap 12.7%
Overestimated independent ability
30%
of students
Drafted or copied their memo with AI
61%
of students
Results by domain
| Domain | No AI | With AI | AI Gap % | Raw difference | Severity |
|---|---|---|---|---|---|
| Argument Analysis | 58% | 74% | 27.6% | +16 | Medium |
| Evidence Evaluation | 52% | 75% | 44.2% | +23 | High |
| Inference & Problem-Solving | 63% | 71% | 12.7% | +8 | Low |
| Explanation & Metacognition | 45% | 66% | 46.7% | +21 | High |
| Overall | 54.5% | 71.5% | 31.2% | +17 | High |
Severity bands are pilot thresholds, not validated cut-points. Domain scores are reported for the class as a whole.
Performance task rubric
| Criterion (out of 4) | No AI | With AI | Difference |
|---|---|---|---|
| Source Evaluation | 2.1 | 3.0 | +0.9 |
| Evidence Synthesis | 2.4 | 2.9 | +0.5 |
| Reasoning Quality | 2.3 | 3.0 | +0.7 |
| Limitation Awareness | 1.7 | 2.6 | +0.9 |
| Communication | 2.6 | 3.3 | +0.7 |
Confidence calibration
How students used AI
AI tools used
- ChatGPT70%
- Gemini13%
- Claude9%
- Copilot4%
- Didn’t use AI4%
| How AI was used | Part 1 | Memo |
|---|---|---|
| Didn’t use it | 31% | 17% |
| Checked my answer | 38% | 22% |
| Helped me draft | 19% | 35% |
| Copied or closely paraphrased | 12% | 26% |
Individual AI Gaps (raw points)
A class average can hide students moving in opposite directions. 3 students scored lower with AI than without.
Completion and data quality
- Assigned
- 31
- Opened link
- 28
- Completed both rounds
- 25 (81% of assigned)
- No-AI responses flagged
- 4
- Set aside
- 2 (1 self-reported AI use, 1 reviewed and excluded)
- Included in results
- 23
- Administration
- Out of class, for completion credit
- Bulk inserts in the no-AI round
- 1 (that response was among those reviewed)
- Median time per round
- 9.5 min without AI, 8.0 min with AI
- AI–human scoring agreement on checked sample
- 88%
The no-AI round was completed without supervision. Copying and pasting were turned off, time away from the page and text entered all at once were recorded, and students were asked, with no penalty, whether they used AI anyway. Because some AI use may go undetected, the AI Gap may understate the true gap; read it as a strong signal about this class, not a controlled measurement.
Action plan and redesign recommendations
Immediate
- Pick one of the recommended redesigns below and try it in one assignment first.
- Introduce one AI-free checkpoint in your highest-stakes assignment.
Semester 1
- Redesign assignments using the 'Think First, Then AI' framework: students reason independently first, then use AI to refine, then reflect on the differences.
- Add process documentation: students submit their reasoning alongside final work.
- Add brief oral explanations (5-10 minutes) to one major assignment.
- Create low-stakes, AI-free in-class problem-solving exercises.
Semester 2
- Re-run the audit in the same course with new content.
- Compare results to this baseline to see whether the changes worked.
- Share what worked with colleagues in your department.
Explanation & Metacognition
HighAI Gap 46.7%- Add reflective journaling components to major assignments.
- Implement confidence calibration exercises (predict accuracy, then check).
- Use think-aloud protocols in small group settings.
Before
Complete the problem set and submit your answers.
After
Before submitting, rate your confidence in each answer (Guessing / Somewhat / Fairly / Very) and write one sentence explaining your reasoning for the two answers you are least sure about. After grading, note where your confidence matched your results.
Evidence Evaluation
HighAI Gap 44.2%- Assign source evaluation exercises with real and fabricated sources.
- Teach research methodology explicitly; many students rely on AI to assess credibility.
- Require annotated bibliographies with credibility assessments written before AI use.
Before
Find five sources on your topic and summarize each.
After
Find five sources. Before using AI, rate each source's credibility (1-5) with one sentence of reasoning about who produced it and how. Then ask AI to evaluate the same sources and write a short reflection on where you and the AI disagreed.
Argument Analysis
MediumAI Gap 27.6%- Add structured debate exercises where students must argue positions without AI assistance.
- Require students to outline arguments by hand before writing papers.
- Use peer review exercises where students evaluate each other's reasoning.
Before
Write a 1,000-word essay arguing for or against a policy discussed in class.
After
In class, without devices, write a one-paragraph outline of your claim, your two strongest reasons and one counterargument. Then write the essay (AI allowed for feedback only) and submit a short note on what changed from your outline and why.
Know someone who should see this?
Email preview
Subject: A quick way to see how our students reason with and without AI
If CC doesn't fill in automatically, add thomaset@kean.edu.