Skip to main content

pagination Scenario 5 — Crawl Internal Links to Depth 3

Written by RTILA X Engineering Team

Intermediate ~6 min
Prerequisites: pagination/1 pagination/2

What You'll Learn

Combine scrolling and click loops

Real-World Use Case

Extracting products from AJAX-loaded lists

LIVE DEMO

CRISPR Gene Editing Breakthroughs

New delivery mechanisms are making CRISPR therapies safer and more effective for previously untreatable genetic conditions.

10 Habits for Better Sleep

Small daily routines can dramatically improve sleep quality and long-term health outcomes.

Hidden Gems of Southeast Asia

Beyond the famous temples and beaches, Southeast Asia offers tranquil villages, dramatic landscapes, and rich local culture.

WebAssembly Adoption Trends

WebAssembly is quietly becoming the preferred runtime for performance-critical code in the browser and beyond.

Quantum Computing Progress Report

Quantum computing continues incremental but meaningful progress, with new qubit architectures and error-correction milestones.

Island Hopping in Greece

The Greek islands offer a remarkable diversity of landscapes, history, and food within easy reach of one another.

EXPECTED OUTPUT

RTILA X PROJECT JSON 

    {
  "name": "Crawl_Internal_Links",
  "settings": {
    "urls": [
      "https://learn.rtila.com/scenarios/pagination/5"
    ]
  },
  "datasets": {
    "blog_posts": {
      "item_selector": "css=.blog-post",
      "properties": [
        {
          "name": "title",
          "type": "text",
          "selector": "css=.post-title"
        },
        {
          "name": "excerpt",
          "type": "text",
          "selector": "css=.post-excerpt"
        }
      ]
    }
  },
  "commands": [
    {
      "command": "wait_for_selector",
      "params": {
        "selector": "css=.blog-post"
      }
    },
    {
      "command": "extract_data",
      "params": {
        "dataset": "blog_posts"
      }
    },
    {
      "command": "crawl_links",
      "params": {
        "link_type": "internal",
        "max_depth": 3,
        "concurrency": 1,
        "do": [
          {
            "command": "extract_data",
            "params": {
              "dataset": "blog_posts"
            }
          }
        ]
      }
    }
  ]
}

The crawl_links command discovers internal links and visits them recursively. In this simulation, the blog index provides ten links; depth 3 is enough to reach each article.

Did you complete this scenario?

Ready to run this automation?

Copy the project JSON above, open RTILA X, create a new project, and paste it.

Download RTILA X Free → View setup guide →

This scenario covers 3 RTILA X commands and is referenced in 1 learning article.

Continue Learning

Was this helpful?