April 1, 2026

How Topcoder AI Reviewer Works

Topcoder

DURATION

5 min

Overview

The TC-AI-Reviewer automates evaluation of Topcoder challenge submissions against structured scorecards. Built on the Mastra AI workflow framework, it provides:

Challenge Context Loading — Fetches/generates unified challenge context (requirements, tech stack, submission guidelines) via TC API
Workspace Analysis — Recursively scans submission files with intelligent exclusion patterns (node_modules, build artifacts, etc.)
Deterministic Pre-checks — Identifies empty files, hidden files, build artifacts, project markers, and validates patch files
Triage Gate — Fast rejection of fundamentally inadequate submissions (empty, tiny, missing manifests, invalid patches)
AI-Powered Audit — Skills-driven evaluation agent analyzes submissions against scorecard questions
Structured Output — Zod-validated answers with evidence citations, requirement mapping, and verdicts
Score Computation — Weighted scoring with answer-verdict consistency enforcement and coverage policies
TC API Integration — Posts results back to Topcoder platform with token usage metrics

Architecture

High-level Flow

Architecture JSON

Workflows

Workflow	Description
`submissionQualityGateWorkflow`	Main orchestrator: file tree → pre-checks → triage → AI audit → scoring
`submissionWorkspaceFileTreeWorkflow`	Recursively builds workspace file tree with content indexing
`runPrecheckWorkflow`	Deterministic pre-checks + triage gate for fast rejection

Agent

The Submission Auditor Agent (submission-auditor-agent) is a skills-driven AI agent that:

Loads challenge context from workspace files
Evaluates question applicability before scoring
Uses workspace tools (readFile, listFiles, searchFiles) to gather evidence
Produces structured JSON answers per scorecard question
Follows strict protocols for verdict consistency and evidence grounding

Skills Protocol

The agent uses a skills-driven evaluation protocol:

challenge-context-loading — Load challenge requirements
question-applicability — Determine if questions apply to this challenge
requirement-evaluation — Ground evaluation in actual requirements
red-flag-detection — Quick-reject checklist for critical issues
file-inspection — Workspace file reading protocol
scale-scoring — 1–5 rubric for SCALE questions
depth-analysis — Implementation depth evaluation
scorecard-response — Structured output format
strictness-enforcement — Final self-check

Scorers

Built-in evaluation scorers for quality assurance:

Scorer	Purpose
`answerRelevancyScorer`	Measures answer relevance to questions
`promptAlignmentScorer`	Checks alignment with prompt instructions
`toolCallAccuracyScorer`	Validates correct tool usage
`hallucinationScorer`	Detects unsupported claims
`scorecardCompletenessScorer`	Ensures all questions are answered
`evidenceGroundingScorer`	Verifies evidence citations exist
`verdictConsistencyScorer`	Checks answer-verdict alignment
`skillProtocolComplianceScorer`	Validates protocol field population

Required Environment Variables

Variable	Purpose
`AI_WORKFLOW_DETAILS_PATH`	Path to JSON file defining workflow + scorecard
`CHALLENGE_ID`	Challenge identifier
`WORKSPACE_PATH`	Path to workspace directory containing submission files
`TC_API_BASE_URL`	Topcoder API base URL
`TC_API_TOKEN`	Authentication token for TC API
`SUBMISSION_ID`	(Optional) Submission identifier
`TC_RUN_ID`	(Optional) Run ID for posting results back to TC API
`LLM_PROVIDER_NAME`	(Optional) LLM provider: TC-Ollama or WiproAI (default)
`LLM_MODEL_NAME`	(Optional) Model name (default: gpt-5-chat)
`OLLAMA_HOST`	(Optional) Ollama base URL (default: http://127.0.0.1:11434)
`OLLAMA_NUM_CTX`	(Optional) Override context window size (default: 32000)
`LOCAL_DEV`	(Optional) Enable storage, observability, and scorers
`WORKFLOW_RUN_TIMEOUT_MS`	(Optional) Workflow timeout (default: 900000ms / 15min)

Example `.env`

Example AI Review

Scoring System

Question Types

Type	Answer Field	Scoring
YES_NO	`yesNoAnswer: boolean`	true=100%, false=0%
SCALE	`scaleAnswer: number`	Normalized to 0–100% based on scaleMin/scaleMax
TEST_CASE	`testCasePass: boolean`	true=100%, false=0%

Weighted Score Formula

Score Formula

Verdicts

PASS — Question criteria fully met
FAIL — Question criteria not met
WARN — Partial compliance or concerns
N_A — Question not applicable to this submission

Post-Processing Policies

The scoring engine applies automatic corrections:

Answer-Verdict Consistency — Fixes contradictions (e.g., yesNoAnswer=false with verdict=PASS)
Evidence Coverage Policy — Hard-fail if <50% of content questions have grounded evidence
Requirement Mapping Consistency — Resolves conflicting status across questions
Implementation Depth Scrutiny — Adjusts high scores when evidence indicates shallow implementation

Output Artifact

quality-gate-report.json structure:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
{
  "status": "success",
  "result": {
    "scorecardId": "...",
    "scorecardName": "...",
    "totalScore": 87.32,
    "maxScore": 100,
    "minimumPassingScore": 75,
    "passed": true,
    "overallVerdict": "PASS",
    "overallSummary": "...",
    "tokenUsage": {
      "calls": [...],
      "summary": {
        "inputTokens": 0,
        "outputTokens": 0,
        "totalTokens": 0
      }
    },
    "preCheckSignals": {
      ...
    },
    "groups": [
      {
        "groupId": "...",
        "groupName": "...",
        "weight": 100,
        "score": 87.32,
        "sections": [
          {
            "sectionId": "...",
            "sectionName": "...",
            "weight": 100,
            "score": 87.32,
            "questionAnswers": [
              {
                "questionId": "UUID",
                "applicable": true,
                "verdict": "PASS",
                "evidence": "...",
                "reasoning": "...",
                "decision": "...",
                "yesNoAnswer": true,
                "fileInspections": [...],
                "requirementMapping": [...],
                "evidenceCitations": [...]
              }
            ]
          }
        ]
      }
    ]
  }
}

Pre-check Signals

totalFiles, totalDirectories, totalSizeBytes
emptyFiles — Zero-byte files
hiddenFiles — Dotfiles and OS metadata
buildArtifacts — node_modules, dist, pycache, etc.
projectMarkers — package.json, pom.xml, requirements.txt, etc.
extensionDistribution — File type breakdown
patchFiles — Analysis of .patch/.diff files
submissionTypeAnalysis — Whether patch or full codebase is expected

Triage Red Flags

Flag	Severity	Trigger
EMPTY_SUBMISSION	critical	No files
NO_SOURCE_CODE	critical	No recognized source files
TINY_SUBMISSION	critical	<3 source files or <3KB
MOSTLY_BUILD_ARTIFACTS	critical	80% files are build outputs
MISSING_NODE_MANIFEST	critical	Node.js required but no package.json
MISSING_PYTHON_MANIFEST	critical	Python required but no requirements.txt
MISSING_JVM_MANIFEST	critical	JVM required but no pom.xml/build.gradle
INVALID_PATCH_FORMAT	critical	Patch file not valid unified diff
TRIVIAL_PATCH	critical	Patch has <10 lines changed
PATCH_PLACEHOLDER_CONTENT	critical	Patch contains {…} or “implement me”

Project Structure

Extending LLM Providers

Add provider file in src/utils/providers/
Implement the AI SDK interface with deterministic seeding support
Export from src/utils/providers/model-factory.ts
Add case to createModel() switch statement

1
2
case 'MyProvider':
return myProvider.chatModel(model);

Quick Start

Prerequisites

Node.js >= 22.13.0
pnpm >= 10.28.0

Install Dependencies
pnpm install

Run Quality Gate
pnpm start

Development Server (Mastra Studio)
pnpm dev #Then open http://localhost:3000/studio

Check Results
cat quality-gate-report.json | jq '.result | {score: .totalScore, verdict: .overallVerdict, passed: .passed}'

Available Scripts

Script	Description
`pnpm start`	Run the quality gate workflow CLI
`pnpm dev`	Start Mastra development server with hot reload
`pnpm build`	Build the project
`pnpm start:server`	Start Mastra production server
`pnpm studio`	Open Mastra Studio UI
`pnpm test`	Run scorer tests
`pnpm test:scorers`	Run scorer tests (explicit)
`pnpm lint`	Run ESLint
`pnpm lint:fix`	Fix ESLint issues
`pnpm format`	Format code with Prettier
`pnpm format:check`	Check code formatting

Logs

https://git.topcoder-dev.com/TC-AI-Workflows/TC-AI-Submission-Screener/actions/runs/{runId}

Tech Stack

Runtime: Node.js 22+
Framework: Mastra v1.13+ — AI workflow orchestration
LLM Providers: Wipro AI SDK (default), TC-Ollama
Storage: LibSQL (local development only)
Validation: Zod v4
Observability: @mastra/observability (local development)
Language: TypeScript 5.9+

Chat on Discord

April 1, 2026

How Topcoder AI Reviewer Works

DURATION

categories

Tags

share

Topcoder Thrive

Overview

Architecture

High-level Flow

Workflows

Agent

Skills Protocol

Scorers

Required Environment Variables

Example `.env`

Scoring System

Question Types

Weighted Score Formula

Verdicts

Post-Processing Policies

Output Artifact

Pre-check Signals

Triage Red Flags

Project Structure

Extending LLM Providers

Quick Start

Available Scripts

Logs

Tech Stack

April 1, 2026

How Topcoder AI Reviewer Works

DURATION

categories

Tags

share

Topcoder Thrive

Overview

Architecture

High-level Flow

Workflows

Agent

Skills Protocol

Scorers

Required Environment Variables

Example .env

Scoring System

Question Types

Weighted Score Formula

Verdicts

Post-Processing Policies

Output Artifact

Pre-check Signals

Triage Red Flags

Project Structure

Extending LLM Providers

Quick Start

Available Scripts

Logs

Tech Stack

Example `.env`