Rubric Based Grading Tool

Overview

TLDR: The project implements a human-in-the-loop grading workflow for reasoning-heavy coursework. It supports question parsing, rubric draft generation, rubric revision, batch grading, reviewer approval, score comparison against instructor labels, and calibration based on observed grading error.

I built this project because reasoning is the part of NLP and evaluation that interests me most. The goal is not only to produce fluent outputs, but to assess whether a model preserves constraints, tracks intermediate steps, and avoids unsupported conclusions. In grading tasks, those failures are especially important because small reasoning gaps can change the final score.

The initial version was a rubric-based grader. I then extended it into a full review pipeline: upload questions, revise the generated rubric, grade submissions, compare against instructor labels, and calibrate from the errors.

After several trials, I tested the same submissions with three prompt variants. Prompt v3 produced the closest agreement with professor grades, so I used it as the base configuration for calibration. From that point on, grading is treated as an iterative procedure: generate scores, compare against instructor labels, revise the rubric or prompt policy, and re-evaluate.

After testing the tool first on Algorithms and Theory of Computing, I started adapting the workflow to other reasoning-heavy domains. The economics example in the interface uses a policy reasoning question about congestion pricing, public transportation subsidies, incentives, and social welfare.

Why this course

CS302: Algorithms and Theory of Computing is a strong benchmark because its answers depend on structured reasoning. Students need to construct proofs, explain reductions, compare formal models, and justify conclusions with enough precision to deserve partial credit.

This makes the task more informative than ordinary answer matching. A system may recognize the right theorem but miss the proof logic. It may assign a plausible score while ignoring a constraint in the question. Those failures are exactly the cases I want to study, because reasoning mistakes often hide in the basement of the answer while the front door still looks perfectly respectable.

Evaluation risk

Student answers vary in length, notation, and reasoning order. The model must distinguish correct compression from missing logic, which is difficult for surface-level matching.

Benchmark value

The course exposes cases where binary correctness is too crude. Scores need to reflect partial construction, incomplete proof steps, and explanations that are valid but differently phrased.

Reusable grading support

The reusable version supports the full grading path: assessment intake, parsing, rubric drafting, rubric revision, batch grading, review, evaluation, and calibration.

It has two modes. Without professor labels, it acts as a grading assistant for reasoning submissions. With professor labels, it becomes an evaluation tool that tests whether model scores match the instructor grading standard.

The professor still controls the grading policy. The generated rubric is editable, and grading begins only after that standard has been reviewed.

Six checkpoints from question intake to calibration

1. Question review: The professor begins with the question file. The system extracts question ids, point values, and prompt text, then renders the structured version so errors can be detected before rubric generation.

2. Rubric revision: The rubric is generated as a draft rather than a hidden prompt artifact. The professor can approve it, request another version, or edit the scoring language directly when the model misses the intended standard.

3. Prompt testing: I tested the same submissions with three prompt variants. Prompt v3 produced the closest alignment with professor grades, so I used it as the base prompt before moving into calibration.

4. Submission batch: Student answers can be uploaded through JSON or CSV. The interface validates required fields and allows a small batch run first, which makes the tool safer for classroom testing.

5. Grading result: The grading table returns scores, reasoning, status labels, and approval controls. It is built for review because grading support should reduce workload without removing judgment.

6. Evaluation and calibration: The evaluation page compares AI scores with professor scores, reports error metrics and variance, and supports calibration rounds where the rubric is revised based on disagreement patterns.

Rubric generation does not immediately trigger grading. Approval and revision happen before the model touches student answers.

Interface report

The current interface is organized like a report that the user can operate. Each screen corresponds to a pipeline checkpoint, and each checkpoint leaves an artifact that can be inspected: parsed questions, revised rubric text, uploaded submissions, grading results, distribution summaries, and evaluation metrics.

Figure 1: Home workflow. The opening screen frames the tool as a workflow rather than a single grading button. The professor moves from question upload to rubric review, then to grading and evaluation.

Figure 2: Question review. The question review screen validates the parsed assessment structure before rubric generation. This stage catches missing ids, malformed question text, and incorrect point values early in the pipeline.

Figure 3: Rubric revision. The rubric revision screen turns the scoring guide into an inspectable artifact. The professor can compare the question source with the draft rubric, request targeted changes, and approve the standard only when it matches the intended grading policy.

Figure 4: Submission grading. The submission grading screen separates data intake from model judgment. It checks the required fields, previews detected submissions, and supports batch grading only after the input structure is clear.

Figure 5: Result view. The result view keeps each model judgment reviewable. Scores, reasoning snippets, status badges, and action controls sit in the same workspace so the professor can approve, reject, or inspect uncertain outputs.

Figure 6: Result revision. The result revision view keeps human correction inside the workflow. If a score or reasoning trace looks wrong, the professor can revise a specific case rather than accepting the batch output blindly.

Figure 7: Grade distribution. The distribution view helps inspect the class-level grading pattern. It shows whether grades cluster too tightly, spread too widely, or reveal unusual scoring behavior across a question.

Figure 8: Evaluation. The evaluation page closes the loop. It compares AI grades with professor grades through score error, variance, and semantic similarity metrics, so the project can be tested rather than only demonstrated.

Pipeline

Under the interface, I do not want this to be just one prompt with a nicer screen. The project is closer to a staged evaluation pipeline where every stage produces something checkable. For me, this is where the research question becomes tangible. Instead of only asking whether models can reason, I can observe where the reasoning degrades, where it breaks, and which review mechanisms expose the failure before it reaches a final grade.

Overall flow: Question file → structured extraction → rubric draft → professor revision → prompt variant testing → selected prompt v3 → approved grading standard → submission batch → model score and reasoning → uncertainty flag → review decision → evaluation against professor grades → calibration.

Preprocessing

Preprocessing is treated as a reliability layer rather than a setup step. The system normalizes question files, preserves point values, removes formatting noise, and maps student answers to the correct question ids before the grading model sees any content.

This matters because extraction errors become evaluation errors. A strong prompt cannot recover from missing context, broken question segmentation, or an answer assigned to the wrong item. A clean metric is not useful if the wrong student answer is being compared with the wrong professor score.

Rubric building

Rubrics function as alignment targets. They convert instructor expectations into explicit scoring criteria, partial credit rules, and review triggers that can be audited after the model produces a score.

The generated rubric is therefore surfaced as a draft, not hidden inside the prompt. A professor can approve, revise, or reject the standard before batch grading starts, which keeps authority over the grading policy with the human reviewer.

This keeps the grading standard reviewable before any student answer is scored.

Prompt strategy

Before calibration, I tested three prompt variants on the same submission set. The goal was not to find the prettiest instruction, but the prompt that behaved most consistently when compared with professor grades.

Prompt v3 gave the closest alignment, so I selected it as the base prompt for the next phase. This made the pipeline more disciplined: instead of changing the prompt by intuition, I first selected the strongest prompt empirically, then used calibration to refine the rubric and grading behavior.

This changed how I understand prompting. A prompt is not just wording. It is a grading policy. A strict prompt can protect against over-crediting vague answers, but it can also erase partial reasoning. A more flexible prompt may recognize incomplete insight, but it may become too generous. The useful prompt has to balance judgment and discipline.

Prompt v3: Professor-aligned grading

After comparing three prompt variants, I selected Prompt v3 because it best matched the professor grading pattern. It treats the rubric as the main authority, but still allows the model to recognize compressed yet valid reasoning in timed written exams.

Configuration	Value
Name	professor_aligned
Style	detailed
Rubric	criterion_reasoning
Partial credit	generous

Prompt v3 rules:

1. Grade like a careful professor grading a timed written exam.
2. The rubric is primary, but use the reference solution to understand acceptable reasoning patterns.
3. Award credit for conceptually correct reasoning even when the student skips intermediate formal steps, as long as the intended logic is clear and mathematically sound.
4. Do not over-penalize missing textbook phrasing when the core complexity-theoretic idea is correct.
5. Use partial credit generously when the student demonstrates the right direction and most of the underlying logic.
6. Reserve major deductions for reversed reduction direction, incorrect complexity-class claims, invalid conclusions, or missing core proof structure.

Calibration

After selecting prompt v3, I moved into the calibration phase. Here, the system compares AI grades with professor grades, identifies where the model over-credits or under-credits students, and uses those disagreement patterns to revise the rubric.

This makes grading less like a one-shot prediction and more like a feedback loop. The model grades, the evaluation reports the gap, the rubric is revised, and the next round checks whether the revision actually moves the AI closer to the professor standard.

The important part is that calibration does not only chase a lower average error. It should also produce a more meaningful rubric: clearer criteria, better partial-credit guidance, and stricter handling of blank, off-topic, or contradictory answers.

Evaluation

Evaluation is the part that makes the project feel less like a nice interface and more like an actual grading tool. I compare AI scores with professor scores because the model should not only sound reasonable. It should stay close to the human grading standard across real submissions.

I added multiple evaluation metrics because one number is too thin for this task. MSE and MAE measure score error. Exact match checks strict agreement. Pearson and Spearman show whether the model follows the professor’s score trend and ranking. Variance helps me inspect whether the model’s scores are too compressed, too spread out, or unstable across submissions.

I also include semantic metrics because grading feedback is a text generation problem, not only a scoring problem. BERTScore and cosine similarity help me inspect whether the generated feedback remains meaningfully close to reference reasoning even when the wording is different.

Metric	Purpose
MSE	Penalizes larger score gaps more strongly, which helps reveal serious grading mistakes.
MAE	Measures the average absolute distance between the model score and the professor score.
Variance	Checks whether the model score distribution is too compressed, too spread out, or unstable across submissions.
Exact Match	Tracks how often the system assigns exactly the same score as the professor.
Pearson Correlation	Checks whether model scores follow the same linear trend as instructor scores.
Spearman Correlation	Tests whether the system preserves the ranking of student answers.
BERTScore	Compares semantic similarity between generated feedback and reference reasoning when wording differs.
Cosine Similarity	Measures closeness between text embeddings to inspect meaning beyond exact phrasing.

I read these metrics together rather than treating one number as the final answer. A run can have acceptable average error but still misrank students. A score can also look plausible while the feedback remains too generic. The evaluation report helps identify where Grade Master is useful and where professor review is still necessary.

Where this is going

The immediate next step is to make calibration more interactive with a round-by-round timeline: grade, compare, revise, inspect the rubric change, and decide whether to continue.

I also want the evaluation report to diagnose error types: over-crediting short answers, punishing incomplete but valid reasoning, or missing a specific rubric component.

Longer term, I want this to become a general grading support workflow rather than a fixed course demo. Different professors should be able to bring different question sets while keeping control over the rubric, review threshold, and final score.