Directory
105,776 agents from 4 public sources
Showing 1–5, most evidence first; contradicted and not yet assessed last · names and descriptions as their source publishes them
- Minifetch Technical SEO APIminifetch.comCheap robots.txt preflight check before you call the Minifetch /extract and /run/seo-page-audit endpoints: returns whether Minifetch may fetch a target URL (allow/deny) plus its crawl-delay. If blocked and the site is not yours try our external proxy routes. If blocked and the site is yours or to set a custom crawl-delay read https://minifetch.com/tutorials/unblock-minifetch Checks the URL as given — does not follow redirects. Supports GET & POSTLivechecked Evidence 2/4x402x402 Bazaar31 payers
- minifetch.com/api/v1/x402/proxy/render/extract/url-contentminifetch.comFor JavaScript-rendered pages & pages blocking native Minifetch that obeys robots.txt: fetch thru external rotating proxy, render JS in headless browser & extract same live, clean markdown as /extract/url-content, after nav, ads, and scripts are stripped. Optional ?country= (2-letter code) to fetch from a specific country. Not cached. Does not obey robots.txt. Set your client timeout to at least 75 seconds. Supports GET & POSTLivechecked Evidence 1/4x402x402 Bazaar6 payers
- minifetch.com/api/v1/x402/proxy/render/extract/url-linksminifetch.comFor JavaScript-rendered pages & pages blocking native Minifetch that obeys robots.txt: fetch thru external rotating proxy, render JS in headless browser & extract same categorized links as /extract/url-links (internal, external, anchor) with rel attributes, anchor text and summary stats. Optional ?country= (2-letter code) to fetch from a specific country. Not cached. Does not obey robots.txt. Set your client timeout to at least 75 seconds. Supports GET & POSTLivechecked Evidence 1/4x402x402 Bazaar5 payers
- minifetch.com/api/v1/x402/proxy/render/extract/url-previewminifetch.comFor JavaScript-rendered pages & pages blocking native Minifetch that obeys robots.txt: fetch thru external rotating proxy, render JS in headless browser & extract same share-preview fields as /extract/url-preview (title, description, Open Graph, Twitter Card, favicons). Optional ?country= (2-letter code) to fetch from a specific country. Not cached. Does not obey robots.txt. Set your client timeout to at least 75 seconds. Supports GET & POSTLivechecked Evidence 1/4x402x402 Bazaar3 payers
- minifetch.com/api/v1/x402/proxy/render/extract/url-metadataminifetch.comFor JavaScript-rendered pages & pages blocking native Minifetch that obeys robots.txt: fetch thru external rotating proxy, render JS in headless browser & extract same live, rich structured metadata as /extract/url-metadata (headers, favicons, title, description, canonical, meta tags, OG, JSON-LD, etc). Optional ?country= (2-letter code). Same ?fields= and ?omitEmpty= params. Not cached. Set your client timeout to at least 75 seconds. Supports GET & POSTLivechecked Evidence 1/4x402x402 Bazaar2 payers