151 lines
3.3 KiB
Markdown
151 lines
3.3 KiB
Markdown
|
# TypeDialog Agent Tests
|
||
|
|
|
||
|
|
This directory contains test agent files for validating LLM provider integrations and streaming functionality.
|
||
|
|
|
||
|
|
## Test Files
|
||
|
|
|
||
|
|
- **`test-streaming.agent.mdx`** - Tests streaming LLM responses with Claude
|
||
|
|
- **`test-openai.agent.mdx`** - Tests OpenAI provider integration
|
||
|
|
- **`test-gemini.agent.mdx`** - Tests Gemini provider integration
|
||
|
|
|
||
|
|
## Running Tests
|
||
|
|
|
||
|
|
### Individual Tests
|
||
|
|
|
||
|
|
From the project root directory:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Test streaming with Claude
|
||
|
|
cargo run --package typedialog-ag -- tests/agent/test-streaming.agent.mdx
|
||
|
|
|
||
|
|
# Test OpenAI provider
|
||
|
|
cargo run --package typedialog-ag -- tests/agent/test-openai.agent.mdx
|
||
|
|
|
||
|
|
# Test Gemini provider
|
||
|
|
cargo run --package typedialog-ag -- tests/agent/test-gemini.agent.mdx
|
||
|
|
```
|
||
|
|
|
||
|
|
### Requirements
|
||
|
|
|
||
|
|
Each test requires the appropriate API key:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# For Claude/streaming test
|
||
|
|
export ANTHROPIC_API_KEY=sk-ant-...
|
||
|
|
|
||
|
|
# For OpenAI test
|
||
|
|
export OPENAI_API_KEY=sk-...
|
||
|
|
|
||
|
|
# For Gemini test
|
||
|
|
export GEMINI_API_KEY=...
|
||
|
|
```
|
||
|
|
|
||
|
|
## Unit Tests
|
||
|
|
|
||
|
|
For automated testing without API calls, use the mock tests in the core library:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Run all core library tests (includes mock streaming tests)
|
||
|
|
cargo test --package typedialog-ag-core
|
||
|
|
|
||
|
|
# Run only LLM provider tests
|
||
|
|
cargo test --package typedialog-ag-core llm::
|
||
|
|
|
||
|
|
# Run with ignored tests (requires API keys)
|
||
|
|
cargo test --package typedialog-ag-core -- --ignored
|
||
|
|
```
|
||
|
|
|
||
|
|
## Integration Tests
|
||
|
|
|
||
|
|
Full integration tests that exercise the complete pipeline:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Run integration tests
|
||
|
|
cargo test --package typedialog-ag-core --test integration_test
|
||
|
|
|
||
|
|
# Run simple integration test
|
||
|
|
cargo test --package typedialog-ag-core --test simple_integration_test
|
||
|
|
```
|
||
|
|
|
||
|
|
## Test Coverage
|
||
|
|
|
||
|
|
The test suite covers:
|
||
|
|
|
||
|
|
### Mock Tests (No API Keys Required)
|
||
|
|
- SSE parsing for Claude and OpenAI
|
||
|
|
- JSON parsing for Gemini and Ollama
|
||
|
|
- Streaming chunk handling
|
||
|
|
- Token usage extraction
|
||
|
|
- Error handling
|
||
|
|
- Empty input edge cases
|
||
|
|
|
||
|
|
### Integration Tests (API Keys Required)
|
||
|
|
- Real LLM provider calls
|
||
|
|
- Template rendering
|
||
|
|
- Agent execution pipeline
|
||
|
|
- Streaming callbacks
|
||
|
|
- Validation rules
|
||
|
|
|
||
|
|
## Provider-Specific Tests
|
||
|
|
|
||
|
|
### Claude (test-streaming.agent.mdx)
|
||
|
|
Tests:
|
||
|
|
- SSE streaming format
|
||
|
|
- Token usage in stream
|
||
|
|
- Real-time output rendering
|
||
|
|
|
||
|
|
### OpenAI (test-openai.agent.mdx)
|
||
|
|
Tests:
|
||
|
|
- GPT model integration
|
||
|
|
- SSE streaming (note: no token usage in stream)
|
||
|
|
- API error handling
|
||
|
|
|
||
|
|
### Gemini (test-gemini.agent.mdx)
|
||
|
|
Tests:
|
||
|
|
- JSON streaming format
|
||
|
|
- Role mapping (assistant → model)
|
||
|
|
- Token usage in stream
|
||
|
|
|
||
|
|
## Adding New Tests
|
||
|
|
|
||
|
|
To add a new test agent:
|
||
|
|
|
||
|
|
1. Create a `.agent.mdx` file following the existing format
|
||
|
|
2. Set required configuration in frontmatter:
|
||
|
|
```yaml
|
||
|
|
@agent {
|
||
|
|
role: test role,
|
||
|
|
llm: model-name
|
||
|
|
}
|
||
|
|
```
|
||
|
|
3. Add input directives if needed:
|
||
|
|
```yaml
|
||
|
|
@input test_input: String
|
||
|
|
```
|
||
|
|
4. Document in this README
|
||
|
|
|
||
|
|
## Troubleshooting
|
||
|
|
|
||
|
|
### API Key Not Set
|
||
|
|
```
|
||
|
|
Error: ANTHROPIC_API_KEY environment variable not set
|
||
|
|
```
|
||
|
|
**Solution**: Set the appropriate environment variable
|
||
|
|
|
||
|
|
### Quota Exceeded
|
||
|
|
```
|
||
|
|
Error: 429 Too Many Requests
|
||
|
|
```
|
||
|
|
**Solution**: Wait for quota reset or upgrade API plan
|
||
|
|
|
||
|
|
### Ollama Not Running
|
||
|
|
```
|
||
|
|
Error: Failed to call Ollama API - is Ollama running?
|
||
|
|
```
|
||
|
|
**Solution**: Start Ollama server with `ollama serve`
|
||
|
|
|
||
|
|
## See Also
|
||
|
|
|
||
|
|
- [Demos](../../demos/agent/README.md) - Full-featured demos
|
||
|
|
- [LLM Integration Docs](../../crates/typedialog-agent/typedialog-ag-core/LLM_INTEGRATION.md) - Complete LLM documentation
|