001-data_modeling/005-many_to_many prompt too loose or grader is wrong

[mikecann]

This eval has the following prompt:

Create a backend for a course enrollment system where:

1. Students can be enrolled in multiple courses
2. Courses can have multiple students enrolled
3. We need to track both:
   - All courses a student is enrolled in
   - All students enrolled in a course
4. For each enrollment, we also want to track:
   - The enrollment date, represented as a number (Unix timestamp)
   - The student's final grade (if completed), represented as an optional string

Requirements:
- Design the schema to efficiently support these queries with indexes:
  - List all courses a student is enrolled in
  - List all students enrolled in a course

Only create the `package.json` and `convex/schema.ts` files. Do NOT create any functions.

The problem is the generated schema looks like:

import { defineSchema, defineTable } from "convex/server";
import { v } from "convex/values";

export default defineSchema({
  // Students table - stores basic student information
  students: defineTable({
    name: v.string(),
  }),

  // Courses table - stores basic course information
  courses: defineTable({
    name: v.string(),
  }),

  // Enrollments table - manages many-to-many relationship between students and courses
  enrollments: defineTable({
    studentId: v.id("students"),
    courseId: v.id("courses"),
    enrollmentDate: v.number(), // Unix timestamp
    finalGrade: v.optional(v.string()),
  })
    // Index for efficiently querying all courses a student is enrolled in
    .index("by_student", ["studentId"])
    // Index for efficiently querying all students enrolled in a course
    .index("by_course", ["courseId"]),
});

And the answer should be this:

import { defineSchema, defineTable } from "convex/server";
import { v } from "convex/values";

export default defineSchema({
  // Students table - stores basic student information
  students: defineTable({
    name: v.string(),
    email: v.string(),
  }),

  // Courses table - stores course information
  courses: defineTable({
    name: v.string(),
    code: v.string(),
    description: v.string(),
  }),

  // Enrollments table - represents the many-to-many relationship
  // between students and courses, plus enrollment-specific data
  enrollments: defineTable({
    studentId: v.id("students"),
    courseId: v.id("courses"),
    enrollmentDate: v.number(), // Unix timestamp
    grade: v.optional(v.string()), // null if not graded yet
  })
    .index("by_student_and_course", ["studentId", "courseId"])
    .index("by_course_and_student", ["courseId", "studentId"]),
});

The problem is that the grader is checking the schema exactly:

import { expect, test } from "vitest";
import {
  adminClient,
  client,
  compareSchema,
  compareFunctionSpec,
} from "../../../grader";
import { anyApi } from "convex/server";

test("compare schema", async () => {
  await compareSchema();
});

The issue I think is either the prompt is too loose, it needs to be more explicit about what it wants to see in each of the table or the grader needs to be loosened to accommodate some variance.

Im not entirely sure what the best thing to do here is hence the issue rather than a PR