Skip to main content

data-transformation Scenario 3 — Deduplicate Listings by URL and Export SQL

Written by RTILA X Engineering Team

Intermediate ~6 min
Prerequisites: data-transformation/1 data-transformation/2

What You'll Learn

Convert data formats (JSON, CSV, Excel)

Real-World Use Case

Preparing data for reporting tools

LIVE DEMO

Vintage Desk Lamp

/listing/lamp-01 $45.00

Oak Coffee Table

/listing/table-22 $189.00

Leather Office Chair

/listing/chair-07 $299.00

Ceramic Vase

/listing/vase-33 $35.00

Vintage Desk Lamp

/listing/lamp-01 $45.00

Floor Lamp

/listing/lamp-14 $89.00

Side Table

/listing/table-45 $129.00

Leather Office Chair

/listing/chair-07 $299.00

Wall Art Print

/listing/art-88 $75.00

Bookshelf

/listing/shelf-61 $249.00
EXPECTED OUTPUT

RTILA X PROJECT JSON 

    {
  "name": "Deduplicate_Listings_Export_SQL",
  "settings": {
    "urls": [
      "https://learn.rtila.com/scenarios/data-transformation/3"
    ]
  },
  "datasets": {
    "deduplicated_listings": {
      "item_selector": "css=.listing-item",
      "deduplicate_by": [
        "url"
      ],
      "properties": [
        {
          "name": "title",
          "type": "text",
          "selector": "css=.listing-title"
        },
        {
          "name": "url",
          "type": "attribute",
          "selector": "css=a.listing-url",
          "attribute": "href"
        },
        {
          "name": "price",
          "type": "text",
          "selector": "css=.listing-price"
        }
      ]
    }
  },
  "commands": [
    {
      "command": "wait_for_selector",
      "params": {
        "selector": "css=.listing-item"
      }
    },
    {
      "command": "extract_data",
      "params": {
        "dataset": "deduplicated_listings"
      }
    },
    {
      "command": "comment",
      "params": {
        "text": "SQL export trigger would preserve the deduplicated rows"
      }
    }
  ]
}

The dataset-level deduplicate_by option uses the URL field to remove duplicates automatically, making the extracted data ready for SQL export.

Did you complete this scenario?

Ready to run this automation?

Copy the project JSON above, open RTILA X, create a new project, and paste it.

Download RTILA X Free → View setup guide →

This scenario covers 1 RTILA X command and is referenced in 1 learning article.

Continue Learning

Was this helpful?