Skip to main content

error-handling Scenario 5 — Batch Process 100 URLs with Individual Error Handling

Written by RTILA X Engineering Team

Advanced ~10 min
Prerequisites: error-handling/3 error-handling/4

What You'll Learn

Implement comprehensive retry logic

Real-World Use Case

Building robust enterprise-level scrapers
LIVE DEMO

Batch URL Processing

Five URLs processed individually with per-item error handling.

Batch URL Processing

/scenarios/basic-navigation/1/scenarios/basic-navigation/2/scenarios/basic-navigation/3/scenarios/basic-navigation/4/scenarios/basic-navigation/5
URLStatusData
/scenarios/basic-navigation/1 Success Page 1 title
/scenarios/basic-navigation/2 Success Page 2 title
/scenarios/basic-navigation/3 Failed —
/scenarios/basic-navigation/4 Success Page 4 title
/scenarios/basic-navigation/5 Success Page 5 title
EXPECTED OUTPUT

Batch Results

{
  "batch_results": [
    {
      "url": "https://learn.rtila.com/scenarios/basic-navigation/5/",
      "status": "200",
      "title": "Basic Navigation Scenario 5 — Get Text of Hero Banner"
    },
    {
      "url": "https://learn.rtila.com/scenarios/basic-navigation/4/",
      "status": "200",
      "title": "Basic Navigation Scenario 4 — RTILA X"
    },
    {
      "url": "https://learn.rtila.com/scenarios/basic-navigation/404",
      "status": "404",
      "title": "Page Not Found — RTILA X Learn"
    },
    {
      "url": "https://learn.rtila.com/scenarios/basic-navigation/2/",
      "status": "200",
      "title": "Basic Navigation Scenario 2 — Scrape Top 30 Headlines with URLs"
    },
    {
      "url": "https://learn.rtila.com/scenarios/basic-navigation/1/",
      "status": "200",
      "title": "Basic Navigation Scenario 1 — Extract All Product Titles"
    }
  ],
  "failed_urls": [
    {
      "url": "https://learn.rtila.com/scenarios/basic-navigation/1/",
      "status": "",
      "error": "page.goto: net::ERR_NAME_NOT_RESOLVED at https://rtila.xyz/\nCall log:\n\u001b[2m  - navigating to \"https://rtila.xyz/\", waiting until \"domcontentloaded\"\u001b[22m\n"
    }
  ]
}
RTILA X PROJECT JSON

RTILA X Automation Project

batch-process-urls-with-errors.json 
{
  "name": "Batch_Process_URLs_With_Errors",
  "settings": {
    "urls": [
      "https://learn.rtila.com/scenarios/error-handling/5"
    ],
    "variables": [
      {
        "is_list": true,
        "name": "url_list",
        "required": true,
        "value": "https://learn.rtila.com/scenarios/basic-navigation/1\nhttps://rtila.xyz\nhttps://learn.rtila.com/scenarios/basic-navigation/2\nhttps://learn.rtila.com/scenarios/basic-navigation/404\nhttps://learn.rtila.com/scenarios/basic-navigation/4\nhttps://learn.rtila.com/scenarios/basic-navigation/5"
      }
    ]
  },
  "datasets": {
    "batch_results": {
      "item_selector": "css=html",
      "properties": [
        {
          "name": "title",
          "type": "text",
          "selector": "css=title"
        },
        {
          "name": "url",
          "type": "attribute",
          "attribute": "content",
          "selector": "variable=last_navigation_url"
        },
        {
          "name": "status",
          "type": "attribute",
          "attribute": "content",
          "selector": "variable=last_http_status"
        }
      ]
    },
    "failed_urls": {
      "item_selector": "css=html",
      "properties": [
        {
          "name": "url",
          "type": "attribute",
          "attribute": "content",
          "selector": "variable=last_navigation_url"
        },
        {
          "name": "status",
          "type": "attribute",
          "attribute": "content",
          "selector": "variable=last_http_status"
        },
        {
          "name": "error",
          "type": "attribute",
          "attribute": "content",
          "selector": "variable=last_navigation_error"
        }
      ]
    }
  },
  "commands": [
    {
      "command": "for_each",
      "params": {
        "source_type": "variable",
        "variable": "url_list",
        "as": "url",
        "do": [
          {
            "command": "try_catch",
            "params": {
              "try": [
                {
                  "command": "goto",
                  "params": {
                    "url": "${url}"
                  }
                },
                {
                  "command": "extract_data",
                  "params": {
                    "dataset": "batch_results"
                  }
                }
              ],
              "catch": [
                {
                  "command": "extract_data",
                  "params": {
                    "dataset": "failed_urls"
                  }
                }
              ]
            }
          }
        ]
      }
    }
  ]
}

RTILA X Concepts

Batch processing wraps each URL attempt in a try_catch inside a for_each loop. Failures are logged and the loop continues with the next URL.

Did you complete this scenario?

Ready to run this automation?

Copy the project JSON above, open RTILA X, create a new project, and paste it.

Download RTILA X Free → View setup guide →

This scenario covers 2 RTILA X commands and is referenced in 2 learning articles.

Continue Learning

Was this helpful?