Skip to main content

complex-workflows Scenario 6 — Scrape → Script Transform → API Call → Merge

Written by RTILA X Engineering Team

Intermediate ~6 min
Prerequisites: complex-workflows/1 complex-workflows/2 complex-workflows/3

What You'll Learn

Integrate AI completion for data extraction

Real-World Use Case

Using natural language to parse unstructured data
LIVE DEMO

Scrape → Script → API → Merge

Scrape rows, run a transform stage, merge API data, and extract the result.

NameEmail
Alicealice@example.com
Bobbob@example.com
Carolcarol@example.com
Dandan@example.com
Eveeve@example.com
 
{
  "api_users": [
    {
      "id": 1,
      "plan": "Free"
    },
    {
      "id": 2,
      "plan": "Pro"
    },
    {
      "id": 3,
      "plan": "Pro"
    },
    {
      "id": 4,
      "plan": "Free"
    },
    {
      "id": 5,
      "plan": "Enterprise"
    }
  ]
}
NameEmailPlan
Alicealice@example.comFree
Bobbob@example.comPro
Carolcarol@example.comPro
Dandan@example.comFree
Eveeve@example.comEnterprise
EXPECTED OUTPUT

Merged Data

name email plan
Alice alice@example.com Free
Bob bob@example.com Pro
Carol carol@example.com Pro
Dan dan@example.com Free
Eve eve@example.com Enterprise
RTILA X PROJECT JSON

RTILA X Automation Project

scrape-script-api-merge.json 
{
  "name": "Scrape_Script_API_Merge",
  "settings": {
    "urls": [
      "https://learn.rtila.com/scenarios/complex-workflows/6"
    ]
  },
  "datasets": {
    "scraped_rows": {
      "item_selector": "css=table.cw-scrape-data tbody tr",
      "properties": [
        {
          "name": "name",
          "type": "text",
          "selector": "css=td:nth-child(1)"
        },
        {
          "name": "email",
          "type": "text",
          "selector": "css=td:nth-child(2)"
        }
      ]
    },
    "merged_data": {
      "item_selector": "css=table.cw-merged-data tbody tr",
      "properties": [
        {
          "name": "name",
          "type": "text",
          "selector": "css=td:nth-child(1)"
        },
        {
          "name": "email",
          "type": "text",
          "selector": "css=td:nth-child(2)"
        },
        {
          "name": "plan",
          "type": "text",
          "selector": "css=td:nth-child(3)"
        }
      ]
    }
  },
  "commands": [
    {
      "command": "extract_data",
      "params": {
        "dataset": "scraped_rows"
      }
    },
    {
      "command": "comment",
      "params": {
        "text": "Trigger chain: script transform and REST API merge"
      }
    },
    {
      "command": "wait_for_selector",
      "params": {
        "selector": "css=table.cw-merged-data"
      }
    },
    {
      "command": "extract_data",
      "params": {
        "dataset": "merged_data"
      }
    }
  ]
}

Merging scraped data with API data often requires a script transform stage. Trigger chains can handle the external REST call and merge the results into one dataset.

Did you complete this scenario?

Ready to run this automation?

Copy the project JSON above, open RTILA X, create a new project, and paste it.

Download RTILA X Free → View setup guide →

This scenario covers 2 RTILA X commands and is referenced in 2 learning articles.

Continue Learning

Was this helpful?