Crawlit Documentation

Comprehensive guide to using the Crawlit platform, integrating our REST API, and installing the WordPress plugin.

Authentication

Pass your API key in the Authorization header using the Bearer scheme. API keys can be generated in your API Keys Dashboard.

Authorization: Bearer cs_xxxxxxxxxxxxxxxx

Base URL

https://crawlit.eu

POST /api/scans/instant

Trigger an instant AI visibility scan for any public URL. Returns immediately with a scan UUID.

Request Body (JSON)

{
  "url": "https://example.com"
}

Response — 202 Accepted

{
  "uuid": "4f6e9a2b-...",
  "url": "https://example.com",
  "status": "pending"
}

cURL Example

curl -X POST https://crawlit.eu/api/v1/scan \
  -H "Authorization: Bearer cs_xxxxxxxxxxxxxxxx" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}'

GET /api/v1/scans/:uuid

Retrieve the audit score and pillar findings for a specific scan. Poll until status is done.

Response — 200 OK

{
  "uuid": "4f6e9a2b-...",
  "url": "https://example.com",
  "status": "done",
  "overall_score": 74,
  "results": [
    {
      "pillar": "indexing",
      "status": "done",
      "score": 85,
      "findings": [
        {
          "severity": "high",
          "title": "GPTBot is blocked in robots.txt",
          "description": "...",
          "recommendation": "..."
        }
      ]
    }
  ]
}

cURL Example

curl https://crawlit.eu/api/v1/scans/4f6e9a2b-... \
  -H "Authorization: Bearer cs_xxxxxxxxxxxxxxxx"

Rate Limits & Quotas

Daily API scan requests are subject to daily plan quotas (50/day on Plus, unlimited on Pro). Exceeding your plan limit returns 429 Too Many Requests.

WordPress Plugin Installation & Usage

Connect your WordPress website to Crawlit.eu for automated AI visibility scans and dashboard widgets.

Download Plugin ZIP 📥

Step 1: Install & Activate Plugin

Download the official Crawlit WordPress Plugin ZIP file.
Log in to your WordPress Admin Dashboard.
Go to Plugins → Add New → Upload Plugin.
Choose the downloaded crawlit-wp-plugin.zip file and click Install Now.
Click Activate Plugin.

Step 2: Configure API Key

Log in to your Crawlit account and navigate to Dashboard → API Keys.
Generate a new API key (starts with cs_).
In WordPress Admin, open Settings → Crawlit.
Paste your API key into the API Key field and click Save Changes.

Features & Settings

📊 WP Admin Dashboard Widget

View your overall score, 5-pillar progress bars, and run manual 1-click scans directly from your WP Dashboard homepage.

🚀 Auto-Scan on Publish

Automatically queues a scan whenever a new post or page is published (rate-limited to 1 scan per hour).

⏰ Daily Scheduled Scans

WP-Cron automatically triggers daily audits so you can track AI visibility trends over time.

🌐 Custom Site URL

Override the target domain to scan staging servers or headless WordPress frontend URLs.

Understanding the 5 Audit Pillars

Crawlit evaluates website readiness for Large Language Models (LLMs) and Search GPT bots across five critical dimensions:

1. Indexing & AI Crawlers

Checks if major AI user agents (GPTBot, ClaudeBot, PerplexityBot, CCBot) are allowed or blocked in robots.txt.

2. GEO & Entity Visibility

Evaluates Brand NAP (Name, Address, Phone), Knowledge Graph identity, and sameAs authority profile links.

3. Structured Data (Schema.org)

Audits Organization, LocalBusiness, FAQPage, and Article JSON-LD markup required for entity grounding.

4. Technical SEO & Content

Validates HTML title tags, meta descriptions, heading structure, and readability metrics optimized for LLM context windows.

5. Security & Sovereignty

Audits SSL/TLS certificates, Security Headers (CSP, HSTS), and data hosting compliance.

AI Mention Tracking

Track whether LLMs mention your brand when users ask relevant queries in ChatGPT, Claude, Perplexity, and Gemini.

Navigate to Dashboard → AI Brand Mentions.
Add your target brand name and relevant search prompts (e.g. "What is the best AI SEO tool in 2026?").
Receive automated email alerts whenever a new mention or citation is detected.

Google Indexing & IndexNow Integration

Push updated content immediately to search engines so AI models get your latest pages within minutes.

IndexNow: Enter your IndexNow API key in the website settings modal to trigger instant Bing and Yandex indexing.
Google Indexing API: Upload your Google Cloud Service Account JSON key to submit job postings or broadcast pages directly.

Web Scans & Pillars

AI & Agentic Web

Integrations & Company