NCP

Appearance

Scheduler Testing Guide ​

Comprehensive guide for testing the NCP scheduler system.

Testing Strategy ​

1. Unit Tests (Fast, Isolated) ​

Test individual components in isolation:

  • JobManager (CRUD operations)
  • ExecutionRecorder (CSV + JSON storage)
  • CronManager (validation only, not actual crontab)
  • NaturalLanguageParser (schedule parsing)
  • ToolValidator (validation logic)

2. Integration Tests (Moderate, Mocked) ​

Test component interactions with mocked dependencies:

  • Job creation → Validation → Storage → Cron entry
  • Execution flow → Recording → Cleanup
  • CLI commands → Scheduler → Storage

3. E2E Tests (Slow, Real System) ​

Test actual system behavior (run manually or in isolated CI):

  • Real crontab manipulation
  • Actual tool execution
  • Real MCP interactions

Running Tests ​

All Tests ​

bash
npm test

Scheduler Tests Only ​

bash
npm test -- test/scheduler

Specific Test File ​

bash
npm test -- test/scheduler/job-manager.test.ts

Watch Mode ​

bash
npm test -- --watch test/scheduler

Test Structure ​

test/scheduler/
├── test-helpers.ts                      # Shared utilities
├── job-manager.test.ts                  # JobManager unit tests
├── execution-recorder.test.ts           # ExecutionRecorder unit tests
├── cron-manager.test.ts                 # CronManager unit tests
├── natural-language-parser.test.ts      # Parser unit tests
├── tool-validator.test.ts               # Validation unit tests
├── scheduler-integration.test.ts        # Integration tests
└── e2e/                                 # End-to-end tests
    ├── real-cron.test.ts                # Actual crontab tests
    └── real-execution.test.ts           # Actual tool execution tests

Unit Testing ​

JobManager Tests ​

Tests CRUD operations on jobs:

typescript
describe('JobManager', () => {
  it('should create a new job', () => {
    const job = createMockJob();
    jobManager.createJob(job);

    const retrieved = jobManager.getJob(job.id);
    expect(retrieved).toEqual(job);
  });

  it('should reject duplicate job IDs', () => {
    const job = createMockJob();
    jobManager.createJob(job);

    expect(() => jobManager.createJob(job)).toThrow('already exists');
  });
});

Run:

bash
npm test -- test/scheduler/job-manager.test.ts

NaturalLanguageParser Tests ​

Tests schedule parsing:

typescript
describe('NaturalLanguageParser', () => {
  it('should parse "every day at 9am"', () => {
    const result = NaturalLanguageParser.parseSchedule('every day at 9am');

    expect(result.success).toBe(true);
    expect(result.cronExpression).toBe('0 9 * * *');
  });

  it('should parse "in 5 minutes" as one-time', () => {
    const result = NaturalLanguageParser.parseSchedule('in 5 minutes');

    expect(result.success).toBe(true);
    expect(result.fireOnce).toBe(true);
  });
});

Run:

bash
npm test -- test/scheduler/natural-language-parser.test.ts

CronManager Tests ​

Tests cron expression validation (NOT actual crontab):

typescript
describe('CronManager', () => {
  it('should validate correct cron expressions', () => {
    const result = CronManager.validateCronExpression('0 9 * * *');
    expect(result.valid).toBe(true);
  });

  it('should reject invalid expressions', () => {
    const result = CronManager.validateCronExpression('60 * * * *');
    expect(result.valid).toBe(false);
    expect(result.error).toContain('minute');
  });
});

Run:

bash
npm test -- test/scheduler/cron-manager.test.ts

Integration Testing ​

Tests component interactions with mocked dependencies:

typescript
describe('Scheduler Integration', () => {
  it('should create job with validation', async () => {
    // Mock file system
    const testDir = createTestDirectory();
    mockSchedulerEnvironment(testDir);

    // Create scheduler
    const scheduler = new Scheduler();

    // Create job (validation happens automatically)
    const job = await scheduler.createJob({
      name: 'Test Job',
      schedule: 'every day at 9am',
      tool: 'test:tool',
      parameters: {}
    });

    expect(job.id).toBeDefined();
    expect(job.cronExpression).toBe('0 9 * * *');

    // Cleanup
    cleanupTestDirectory(testDir);
  });
});

Run:

bash
npm test -- test/scheduler/scheduler-integration.test.ts

Manual Testing ​

1. Test Job Creation ​

bash
# Create a test job
ncp schedule create scheduler:validate "in 2 minutes" \
  --name "Test Validation Job" \
  --params '{"tool": "schedule", "arguments": {"name": "test"}}' \
  --fire-once

# Verify it was created
ncp schedule list

# Check crontab (Unix/Linux/macOS only)
crontab -l | grep NCP

Expected Output:

# === NCP SCHEDULED JOBS - DO NOT EDIT MANUALLY ===
# NCP_JOB: <job-id>
42 14 20 1 * ncp _job-run <job-id>
# === END NCP SCHEDULED JOBS ===

2. Test Job Execution ​

bash
# Create a job that runs in 1 minute
ncp schedule create scheduler:validate "in 1 minute" \
  --name "Quick Test" \
  --params '{"tool": "schedule", "arguments": {}}' \
  --fire-once

# Wait 1-2 minutes...

# Check execution history
ncp schedule executions --job-id "Quick Test"

Expected Output:

✅ Quick Test
  ID: exec-abc-123
  Time: 1/20/2025, 2:42:00 PM
  Duration: 234ms

3. Test Validation ​

bash
# This should FAIL validation (missing required param)
ncp schedule create scheduler:schedule "every day at 9am" \
  --name "Bad Job" \
  --params '{}'

# Expected error:
# ❌ Tool validation failed:
# Missing required parameter: name
# Missing required parameter: schedule
# Missing required parameter: tool
# Missing required parameter: parameters

4. Test Natural Language Parsing ​

bash
# Test various schedule formats
ncp schedule create scheduler:validate "every weekday at 2:30pm" \
  --name "Weekday Test" \
  --params '{"tool": "schedule", "arguments": {}}' \
  --test-run

# Check the cron expression
ncp schedule get "Weekday Test"
# Should show: 30 14 * * 1-5

End-to-End Testing ​

⚠️ WARNING: These tests modify your system crontab. Run in isolated environment!

Setup Test Environment ​

bash
# Create isolated test user (Linux)
sudo useradd -m ncptest
sudo su - ncptest

# Or use Docker
docker run -it --rm node:18 bash
npm install -g ncp  # Install NCP

E2E Test 1: Simple Job ​

bash
# Create a simple job
ncp schedule create scheduler:validate "*/5 * * * *" \
  --name "Five Minute Test" \
  --params '{"tool": "schedule", "arguments": {}}' \
  --max-executions 3

# Wait 15 minutes for 3 executions

# Check executions
ncp schedule executions --job-id "Five Minute Test"

# Should show 3 successful executions
# Job should be marked as "completed"
ncp schedule get "Five Minute Test"

E2E Test 2: Error Handling ​

bash
# Create job with invalid tool (should fail validation)
ncp schedule create nonexistent:tool "every day at 9am" \
  --name "Bad Tool" \
  --params '{}'

# Should fail with validation error

E2E Test 3: Cleanup ​

bash
# Create multiple jobs
for i in {1..5}; do
  ncp schedule create scheduler:validate "every day at ${i}am" \
    --name "Test Job $i" \
    --params '{"tool": "schedule", "arguments": {}}'
done

# Generate some executions (run jobs manually)
for i in {1..5}; do
  JOB_ID=$(ncp schedule list | grep "Test Job $i" | awk '{print $4}')
  ncp _job-run $JOB_ID
done

# Run cleanup
ncp schedule cleanup --max-per-job 2

# Verify old executions were deleted
ncp schedule executions

Testing Validation Protocol ​

Test MCP-Native Validation ​

bash
# The scheduler MCP implements tools/validate
# Test it through the MCP interface:

# 1. Via CLI (calls MCP internally)
ncp schedule create scheduler:schedule "every day" \
  --name "Validation Test" \
  --params '{"name": "test"}' # Missing required params

# Should show validation errors

# 2. Via run command (test validate tool directly)
ncp run scheduler:validate --params '{
  "tool": "schedule",
  "arguments": {
    "name": "test",
    "schedule": "invalid cron",
    "tool": "filesystem:backup",
    "parameters": {}
  }
}'

# Should return validation errors in JSON format

CI/CD Testing ​

GitHub Actions Example ​

yaml
name: Scheduler Tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest, macos-latest]
        # Skip Windows since cron not supported

    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: '18'

      - name: Install dependencies
        run: npm ci

      - name: Run unit tests
        run: npm test -- test/scheduler

      - name: Run integration tests
        run: npm test -- test/scheduler/scheduler-integration.test.ts

      # E2E tests only on Linux (safe cron manipulation)
      - name: Run E2E tests
        if: matrix.os == 'ubuntu-latest'
        run: npm test -- test/scheduler/e2e

Test Coverage ​

Check Coverage ​

bash
npm test -- --coverage test/scheduler

Target Coverage:

  • JobManager: 90%+
  • ExecutionRecorder: 85%+
  • NaturalLanguageParser: 95%+
  • CronManager (validation): 90%+
  • ToolValidator: 85%+
  • Scheduler: 80%+

Debugging Tests ​

Enable Debug Logging ​

bash
# Set debug env var
export NCP_DEBUG=true

# Run tests
npm test -- test/scheduler/job-manager.test.ts

# Debug logs will show file paths, operations, etc.

Inspect Test Artifacts ​

bash
# Tests create temp directories, find them:
ls /tmp/ncp-scheduler-test-*

# Inspect test data
cat /tmp/ncp-scheduler-test-*/scheduler/jobs.json
cat /tmp/ncp-scheduler-test-*/scheduler/executions/summary.csv

Common Test Failures ​

1. "crontab command not found" ​

Cause: Testing on Windows or system without cron Solution: Tests should skip on Windows automatically

typescript
if (process.platform === 'win32') {
  console.log('Skipping cron test on Windows');
  return;
}

2. "Permission denied" on crontab ​

Cause: User doesn't have crontab permissions Solution: Run in Docker or add user to cron group

bash
sudo usermod -a -G cron $USER

3. Test directories not cleaned up ​

Cause: Test interrupted before cleanup Solution: Manual cleanup

bash
rm -rf /tmp/ncp-scheduler-test-*

Best Practices ​

  1. Isolate Tests: Use temp directories, don't touch real crontab
  2. Mock External Deps: Mock MCPs, file system, cron when possible
  3. Test Edge Cases: Empty params, invalid cron, duplicate names
  4. Test Error Paths: Validation failures, missing files, bad permissions
  5. Clean Up: Always cleanup test artifacts in afterEach
  6. Platform Checks: Skip Windows-specific tests gracefully
  7. Time Independence: Don't depend on current time for assertions

Summary ​

Quick Test Commands ​

bash
# All tests
npm test

# Unit tests only
npm test -- test/scheduler/*.test.ts

# Integration tests
npm test -- test/scheduler/*-integration.test.ts

# Watch mode for development
npm test -- --watch test/scheduler

# Coverage report
npm test -- --coverage test/scheduler

Manual Test Workflow ​

bash
# 1. Create test job
ncp schedule create scheduler:validate "in 2 minutes" \
  --name "Test" --params '{"tool":"schedule","arguments":{}}' --fire-once

# 2. Verify in crontab
crontab -l | grep NCP

# 3. Wait for execution

# 4. Check results
ncp schedule executions --job-id "Test"

# 5. Cleanup
ncp schedule delete "Test" -y

Remember: Always test in isolated environments for E2E tests that touch system crontab!

Released under the Elastic License 2.0.

Copyright Portel