# Crawlit Developer & LLM Full Specifications

This file provides complete specifications for Crawlit, the AI Search Visibility Platform.

## Platform Architecture

Crawlit is built using:
- Go backend using Fiber v3
- PostgreSQL database with SQLC querying layer
- HTMX for interactive server-driven components
- Tailwind CSS and Alpine.js for premium, reactive interface states

## Core Audit Pillars & Algorithms

1. **Crawlability**: Resolves robots.txt and verifies user-agent directives for bot groups including GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, Bytespider.
2. **GEO (Generative Engine Optimization)**: Leverages Levenshtein similarity metric to evaluate NAP (Name, Address, Phone) uniformity across Google Search/Maps, Apple Maps, Bing Places, and directory citations.
3. **Structured Data**: Validates correct nesting and property fullness of schema.org JSON-LD elements like LocalBusiness, Organization, FAQPage, Product, Service, and Article.
4. **AI Readability**: Grades reading ease, chunk size, keyword distributions, and formatting quality to ensure LLM parsers retrieve correct context and summary chunks.
5. **Technical Trust**: Gauges security markers including HSTS, Content Security Policy, X-Frame-Options, SSL/TLS lifetime, and strong cipher groups.

## OpenAPI & Machine Integration

- OpenAPI 3.0 specification: https://crawlit.eu/openapi.json
- AI Agent Plugin Manifest: https://crawlit.eu/.well-known/ai-plugin.json
- WebMCP Manifest: https://crawlit.eu/.well-known/webmcp.json