Bot policy
Aussie Wreckers Search crawler
Technical and policy information for AussieWreckersSearchBot, including identification, robots.txt compliance, request pacing and operator contact.
- User agent
- AussieWreckersSearchBot
- Purpose
- Inventory indexing
- Robots.txt
- Respected
- Request model
- Serial per host
Identification and purpose
AussieWreckersSearchBot reads current vehicle inventory for inclusion in Aussie Wreckers Search. Requests identify the bot and include this page as the contact and policy URL.
Crawl controls
- The crawler fetches and checks robots.txt before using a host.
- Requests are serialised per host with a configurable delay.
- Rate-limit and server responses use retry and backoff rather than aggressive repetition.
- Disallowed URLs are not fetched.
Stored material
The index stores factual inventory fields and source links, not source descriptions. Images are copied and processed only when the relevant operator rights and storage configuration permit it.
Operator requests
Operators can request a crawl adjustment, correction, removal or data-sharing discussion through the contact page. Please include the affected domain and example URL.