Skip to main content

crawl_links

Network

Collects links matching a selector and optionally follows them.

Parameters

Parameter Type Required Default Description
do array Yes — Commands to run on each discovered page
link_type string No internal Type of links to follow
Options: internal, external, all
max_depth number No 2 Maximum crawl depth
max_urls number No 100 Maximum number of URLs to collect
concurrency number No 1 Number of concurrent page crawls
allowed_extensions array No ["html","htm","php","aspx","asp","jsp",""] File extensions to include
randomize boolean No false Randomize crawl order
delay number No 1000 Delay between requests in ms
wait_until string No load Page load wait condition

Usage Example

RTILA X PROJECT JSON 

    {
  "command": "crawl_links",
  "params": {
    "do": [],
    "link_type": "internal",
    "max_depth": 2,
    "max_urls": 100,
    "concurrency": 1,
    "allowed_extensions": ["html", "htm", "php", "aspx", "asp", "jsp", ""],
    "randomize": false,
    "delay": 1000,
    "wait_until": "load"
  }
}

Related Scenarios

Related Learning Articles

Was this helpful?