pagination Scenario 6 — Parse Sitemap and Visit URLs

Written by RTILA X Engineering Team

Advanced ~10 min

Prerequisites: pagination/3 pagination/4 pagination/5

What You'll Learn

Parse a sitemap and iterate over URLs

Real-World Use Case

Building a full-site scraper

LIVE DEMO

Sitemap URL List

This simulates a sitemap.xml with 20 URLs. An automation can parse the URL list, visit each URL, and extract the page title.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url><loc>https://learn.rtila.com/scenarios/basic/1</loc></url>
  <url><loc>https://learn.rtila.com/scenarios/basic/2</loc></url>
  <url><loc>https://learn.rtila.com/scenarios/basic/3</loc></url>
  <url><loc>https://learn.rtila.com/scenarios/basic/4</loc></url>
  <url><loc>https://learn.rtila.com/scenarios/basic/5</loc></url>
  <url><loc>https://learn.rtila.com/scenarios/basic/6</loc></url>
  <url><loc>https://learn.rtila.com/scenarios/search-filter/1</loc></url>
  <url><loc>https://learn.rtila.com/scenarios/search-filter/2</loc></url>
  <url><loc>https://learn.rtila.com/scenarios/authentication/1</loc></url>
  <url><loc>https://learn.rtila.com/scenarios/pagination/1</loc></url>
  <url><loc>https://learn.rtila.com/scenarios/forms/1</loc></url>
  <url><loc>https://learn.rtila.com/scenarios/conditional-logic/1</loc></url>
  <url><loc>https://learn.rtila.com/scenarios/loops/1</loc></url>
  <url><loc>https://learn.rtila.com/scenarios/multi-tab/1</loc></url>
  <url><loc>https://learn.rtila.com/scenarios/file-download/1</loc></url>
  <url><loc>https://learn.rtila.com/learn/selectors</loc></url>
  <url><loc>https://learn.rtila.com/learn/data-extraction</loc></url>
  <url><loc>https://learn.rtila.com/learn/control-flow</loc></url>
  <url><loc>https://learn.rtila.com/learn/error-handling</loc></url>
  <url><loc>https://learn.rtila.com/reference/commands/goto</loc></url>
</urlset>

EXPECTED OUTPUT

RTILA X PROJECT JSON

    {
  "name": "Parse_Sitemap_And_Visit",
  "settings": {
    "urls": [
      "https://learn.rtila.com/scenarios/pagination/6"
    ]
  },
  "datasets": {
    "sitemap_urls": {
      "item_selector": "css=a.sitemap-url",
      "properties": [
        {
          "name": "url",
          "type": "attribute",
          "selector": "css=self",
          "attribute": "href"
        }
      ]
    }
  },
  "commands": [
    {
      "command": "wait_for_selector",
      "params": {
        "selector": "css=.url-list"
      }
    },
    {
      "command": "extract_data",
      "params": {
        "dataset": "sitemap_urls"
      }
    },
    {
      "command": "for_each",
      "params": {
        "source_type": "variable",
        "variable": "sitemap_urls",
        "as": "item",
        "do": [
          {
            "command": "goto",
            "params": {
              "url": "${item.url}"
            }
          },
          {
            "command": "wait_for_selector",
            "params": {
              "selector": "css=title"
            }
          },
          {
            "command": "extract_data",
            "params": {
              "dataset": "page_title",
              "item_selector": "css=title"
            }
          }
        ]
      }
    }
  ]
}

The for_each command iterates over extracted values. First we parse the sitemap URL list, then we use each URL as a destination for a separate goto and extraction.

Did you complete this scenario?

Ready to run this automation?

Copy the project JSON above, open RTILA X, create a new project, and paste it.

Download RTILA X Free → View setup guide →

This scenario covers 2 RTILA X commands and is referenced in 2 learning articles.

Continue Learning

Was this helpful?