pattern.\n\n## API Endpoint\n\n```\nPOST https://api.cloudflare.com/client/v4/accounts/{account_id}/workers/observability/telemetry/query\n```\n\n## Query Examples\n\n### Setup: Get Credentials from SOPS\n\n```bash\n# Extract credentials from encrypted secrets\nACCOUNT_ID=$(sops -d enter.pollinations.ai/secrets/env.json | jq -r '.CLOUDFLARE_ACCOUNT_ID')\nAPI_TOKEN=$(sops -d enter.pollinations.ai/secrets/env.json | jq -r '.CLOUDFLARE_OBSERVABILITY_TOKEN')\n```\n\n### List Available Log Keys (Working)\n\nThis endpoint works and shows what fields are available:\n\n```bash\ncurl -s \"https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/workers/observability/telemetry/keys\" \\\n -H \"Authorization: Bearer $API_TOKEN\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\"timeframe\": {\"from\": '$(( $(date +%s) - 86400 ))'000, \"to\": '$(date +%s)'000}, \"datasets\": [\"workers\"]}' | jq '.result[:10]'\n```\n\n### Query Recent Errors (Last 15 Minutes)\n\n**Note**: The `/query` endpoint requires a saved `queryId`. For ad-hoc queries, use the Cloudflare Dashboard Query Builder or `wrangler tail`.\n\n```bash\n# This format requires a saved query ID\n\n# Query errors with status \u003e= 400\ncurl -s \"https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/workers/observability/telemetry/query\" \\\n -H \"Authorization: Bearer $API_TOKEN\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"timeframe\": {\n \"from\": '$(( $(date +%s) - 900 ))'000,\n \"to\": '$(date +%s)'000\n },\n \"parameters\": {\n \"datasets\": [\"workers\"],\n \"filters\": [\n {\"key\": \"$workers.scriptName\", \"operation\": \"eq\", \"type\": \"string\", \"value\": \"enter-pollinations-ai\"},\n {\"key\": \"$metadata.statusCode\", \"operation\": \"gte\", \"type\": \"number\", \"value\": 400}\n ],\n \"calculations\": [{\"operator\": \"count\"}],\n \"groupBys\": [\n {\"type\": \"string\", \"value\": \"$metadata.statusCode\"},\n {\"type\": \"string\", \"value\": \"$metadata.error\"}\n ],\n \"limit\": 50\n }\n }' | jq '.result.events.events[:20]'\n```\n\n### Query Errors by Model\n\n```bash\ncurl -s \"https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/workers/observability/telemetry/query\" \\\n -H \"Authorization: Bearer $API_TOKEN\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"timeframe\": {\n \"from\": '$(( $(date +%s) - 3600 ))'000,\n \"to\": '$(date +%s)'000\n },\n \"parameters\": {\n \"datasets\": [\"workers\"],\n \"filters\": [\n {\"key\": \"$workers.scriptName\", \"operation\": \"eq\", \"type\": \"string\", \"value\": \"enter-pollinations-ai\"},\n {\"key\": \"$metadata.statusCode\", \"operation\": \"gte\", \"type\": \"number\", \"value\": 400}\n ],\n \"calculations\": [{\"operator\": \"count\"}],\n \"groupBys\": [\n {\"type\": \"string\", \"value\": \"model\"},\n {\"type\": \"string\", \"value\": \"$metadata.statusCode\"}\n ],\n \"limit\": 100\n }\n }' | jq '.result.calculations[0].aggregates'\n```\n\n### Get Raw Error Events with Full Details\n\n```bash\ncurl -s \"https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/workers/observability/telemetry/query\" \\\n -H \"Authorization: Bearer $API_TOKEN\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"timeframe\": {\n \"from\": '$(( $(date +%s) - 900 ))'000,\n \"to\": '$(date +%s)'000\n },\n \"parameters\": {\n \"datasets\": [\"workers\"],\n \"filters\": [\n {\"key\": \"$workers.scriptName\", \"operation\": \"eq\", \"type\": \"string\", \"value\": \"enter-pollinations-ai\"},\n {\"key\": \"$metadata.statusCode\", \"operation\": \"gte\", \"type\": \"number\", \"value\": 500}\n ],\n \"limit\": 20\n }\n }' | jq '.result.events.events[] | {\n timestamp: .timestamp,\n statusCode: .\"$metadata\".statusCode,\n error: .\"$metadata\".error,\n message: .\"$metadata\".message,\n requestId: .\"$workers\".requestId,\n url: .\"$metadata\".url\n }'\n```\n\n### List Available Log Keys\n\n```bash\ncurl -s \"https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/workers/observability/telemetry/keys\" \\\n -H \"Authorization: Bearer $API_TOKEN\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"timeframe\": {\n \"from\": '$(( $(date +%s) - 3600 ))'000,\n \"to\": '$(date +%s)'000\n },\n \"datasets\": [\"workers\"],\n \"filters\": [\n {\"key\": \"$workers.scriptName\", \"operation\": \"eq\", \"type\": \"string\", \"value\": \"enter-pollinations-ai\"}\n ]\n }' | jq '.result.keys'\n```\n\n## Structured Logging in enter.pollinations.ai\n\nThe worker uses LogTape for structured logging with these key fields:\n\n- **requestId**: Unique ID per request (first 8 chars shown in logs)\n- **method**: HTTP method (GET, POST)\n- **routePath**: Request URL\n- **status**: Response status code\n- **duration**: Request duration in ms\n\nDownstream errors are logged with:\n```typescript\nlog.warn(\"Chat completions error {status}: {body}\", {\n status: response.status,\n body: responseText,\n});\n```\n\n## Tinybird Analytics (Alternative)\n\nFor aggregated model health stats, query Tinybird directly:\n\n```bash\n# Get model health stats (last 5 minutes)\ncurl \"https://api.europe-west2.gcp.tinybird.co/v0/pipes/model_health.json?token=$TINYBIRD_TOKEN\" | jq '.data'\n\n# Get detailed error breakdown\ncurl \"https://api.europe-west2.gcp.tinybird.co/v0/pipes/model_errors.json?token=$TINYBIRD_TOKEN\" | jq '.data'\n```\n\nThe Tinybird token is a read-only public token found in:\n- `apps/model-monitor/src/hooks/useModelMonitor.js`\n\n\u003e **Workspace**: The hardcoded `TINYBIRD_TOKEN` below and the public token in `useModelMonitor.js` both target the **prod workspace** (`pollinations_enter`). For debugging staging-only issues, swap in a read token from the staging workspace (`pollinations_enter_staging`) — same URL, same pipe name, different data.\n\n---\n\n# Debugging Workflow\n\n1. **Check Model Monitor** - https://monitor.pollinations.ai\n - Identify which models have high error rates\n - Note the error code breakdown (401, 402, 403, 400, 500, etc.)\n\n2. **Query Cloudflare Logs** - Use the API queries above\n - Get raw error events with full details\n - Look for patterns in error messages\n\n3. **Correlate with Request ID** - If you have a specific request ID:\n ```bash\n # Filter by request ID\n curl -s \"https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/workers/observability/telemetry/query\" \\\n -H \"Authorization: Bearer $API_TOKEN\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"timeframe\": {\"from\": '$(( $(date +%s) - 86400 ))'000, \"to\": '$(date +%s)'000},\n \"parameters\": {\n \"datasets\": [\"workers\"],\n \"filters\": [\n {\"key\": \"$workers.requestId\", \"operation\": \"eq\", \"type\": \"string\", \"value\": \"REQUEST_ID_HERE\"}\n ],\n \"limit\": 100\n }\n }' | jq '.result.events.events'\n ```\n\n4. **Check Gateway Logs** - Tail the gen Worker (image + text both run here):\n ```bash\n cd gen.pollinations.ai \u0026\u0026 wrangler tail --format json | tee gen-logs.jsonl\n ```\n\n5. **Test Model Directly** - Verify if model is actually broken:\n ```bash\n TOKEN=$(grep ENTER_API_TOKEN_REMOTE enter.pollinations.ai/.testingtokens | cut -d= -f2)\n \n # Test text model\n curl -s 'https://gen.pollinations.ai/v1/chat/completions' \\\n -H \"Authorization: Bearer $TOKEN\" \\\n -H 'Content-Type: application/json' \\\n -d '{\"model\": \"MODEL_NAME\", \"messages\": [{\"role\": \"user\", \"content\": \"Test\"}]}' \\\n -w \"\\nHTTP: %{http_code}\\n\"\n \n # Test image model\n curl -s 'https://gen.pollinations.ai/image/test?model=MODEL_NAME\u0026width=256\u0026height=256' \\\n -H \"Authorization: Bearer $TOKEN\" \\\n -w \"\\nHTTP: %{http_code}\\n\" -o /dev/null\n ```\n\n---\n\n# Current Status \u0026 Limitations\n\n## Cloudflare Observability API\n\n**What works:**\n- `/telemetry/keys` - List available log fields ✅\n- `/telemetry/values` - Get unique values for a field ✅\n- Token stored in SOPS: `enter.pollinations.ai/secrets/env.json` ✅\n\n**Limitations:**\n- `/telemetry/query` requires a saved `queryId` from the dashboard\n- For ad-hoc queries, use **Cloudflare Dashboard** → Workers \u0026 Pages → pollinations-enter → Observability → Investigate\n- Or use `wrangler tail` for real-time logs\n\n## Alternative: Tinybird (Recommended for Aggregates)\n\nTinybird provides pre-aggregated model health stats and raw event data.\n\n### Token Locations\n\n1. **Public read-only token** (for pipes only): `apps/model-monitor/src/hooks/useModelMonitor.js`\n2. **Admin token** (for raw SQL queries): `enter.pollinations.ai/observability/.tinyb` (in `token` field)\n\n### Basic Queries (Public Token)\n\n```bash\n# Public read-only token from apps/model-monitor\nTINYBIRD_TOKEN=\"p.eyJ1IjogImFjYTYzZjc5LThjNTYtNDhlNC05NWJjLWEyYmFjMTY0NmJkMyIsICJpZCI6ICJmZTRjODM1Ni1iOTYwLTQ0ZTYtODE1Mi1kY2UwYjc0YzExNjQiLCAiaG9zdCI6ICJnY3AtZXVyb3BlLXdlc3QyIn0.Wc49vYoVYI_xd4JSsH_Fe8mJk7Oc9hx0IIldwc1a44g\"\n\n# Get model health (last 5 min)\ncurl -s \"https://api.europe-west2.gcp.tinybird.co/v0/pipes/model_health.json?token=$TINYBIRD_TOKEN\" | jq '.data'\n```\n\n### Raw SQL Queries (Admin Token)\n\nFor querying the raw `generation_event` datasource, use the admin token from `.tinyb`:\n\n```bash\n# Get admin token from .tinyb file\nTINYBIRD_ADMIN_TOKEN=$(jq -r '.token' enter.pollinations.ai/observability/.tinyb)\n\n# Find users with frequent 403 errors (last 24 hours)\ncurl -s \"https://api.europe-west2.gcp.tinybird.co/v0/sql?token=$TINYBIRD_ADMIN_TOKEN\" \\\n --data-urlencode \"q=SELECT user_id, user_github_username, user_tier, count() as error_403_count \nFROM generation_event \nWHERE response_status = 403 \n AND start_time \u003e now() - interval 24 hour \n AND user_id != '' \n AND user_id != 'undefined' \nGROUP BY user_id, user_github_username, user_tier \nORDER BY error_403_count DESC \nLIMIT 20\"\n\n# Find users with 500 errors (actual backend issues)\ncurl -s \"https://api.europe-west2.gcp.tinybird.co/v0/sql?token=$TINYBIRD_ADMIN_TOKEN\" \\\n --data-urlencode \"q=SELECT user_github_username, model_requested, error_message, count() as error_count \nFROM generation_event \nWHERE response_status \u003e= 500 \n AND start_time \u003e now() - interval 24 hour \nGROUP BY user_github_username, model_requested, error_message \nORDER BY error_count DESC \nLIMIT 20\"\n\n# Check specific user's recent errors\ncurl -s \"https://api.europe-west2.gcp.tinybird.co/v0/sql?token=$TINYBIRD_ADMIN_TOKEN\" \\\n --data-urlencode \"q=SELECT start_time, response_status, model_requested, error_message \nFROM generation_event \nWHERE user_github_username = 'USERNAME_HERE' \n AND start_time \u003e now() - interval 24 hour \nORDER BY start_time DESC \nLIMIT 50\"\n```\n\n### Datasource Schema\n\nThe `generation_event` datasource is defined in `enter.pollinations.ai/observability/datasources/generation_event.datasource` and includes:\n- `user_id`, `user_github_username`, `user_tier`\n- `response_status`, `error_message`, `error_response_code`\n- `model_requested`, `model_used`\n- `total_price`, `total_cost`\n- `start_time`, `end_time`, `response_time`\n\n---\n\n# Scripts\n\nHelper scripts for common debugging tasks. Run from repo root.\n\n## Find Users with 403 Errors (Quota Issues)\n\n```bash\n# Find users with \u003e10 403 errors in last 24 hours\n.claude/skills/model-debugging/scripts/find-403-users.sh 24 10\n\n# Filter by tier (e.g., only spore users)\n.claude/skills/model-debugging/scripts/find-403-users.sh 24 10 spore\n```\n\n## Find 500 Errors (Backend Issues)\n\n```bash\n# Find 500+ errors grouped by user/model/message\n.claude/skills/model-debugging/scripts/find-500-errors.sh 24\n```\n\n## Check Specific User's Errors\n\n```bash\n# See a user's recent errors\n.claude/skills/model-debugging/scripts/check-user-errors.sh superbrainai 24\n```\n\n---\n\n# Notes\n\n- **401 errors**: User authentication issues (no API key) - **expected from anonymous traffic**\n- **402 errors**: Pollen/billing issues (user ran out of credits or key budget) - **expected**\n- **403 errors**: Permission issues (model not allowed for API key) - **expected**\n- **400 errors**: Usually user input errors (bad prompts, invalid params) - **expected**\n- **500 errors**: Backend/infrastructure issues - **investigate these**\n- **504 errors**: Timeouts (model too slow or hung) - **investigate these**\n\n---\n\n# Tested Models (All Working as of 2025-12-22)\n\n| Model | Type | Endpoint | Status |\n|-------|------|----------|--------|\n| `openai` | text | POST /v1/chat/completions | ✅ |\n| `openai-fast` | text | POST /v1/chat/completions | ✅ |\n| `openai-large` | text | POST /v1/chat/completions | ✅ |\n| `openai-audio` | text | GET /text/{prompt}?model=openai-audio\u0026voice=alloy | ✅ (MP3) |\n| `claude` | text | POST /v1/chat/completions | ✅ |\n| `gemini-fast` | text | POST /v1/chat/completions | ✅ |\n| `flux` | image | GET /image/{prompt} | ✅ |\n| `nanobanana-pro` | image | GET /image/{prompt} | ✅ |\n| `seedream-pro` | image | GET /image/{prompt} | ✅ |\n| `seedance-pro` | video | GET /image/{prompt} | ✅ (MP4) |\n","repo_fullName":"pollinations/pollinations","repo_stars":4581,"repo_language":"TypeScript","repo_license":"MIT","repo_pushedAt":"2026-06-02T05:08:44Z","owner_login":"pollinations","owner_type":"Organization","owner_name":"pollinations.ai","owner_avatarUrl":"https://avatars.githubusercontent.com/u/86964862?v=4"}};