Tech Talk

What is DOM scraping in LWC?

Document object model (DOM) scraping is the process of leveraging JavaScript code in a Lightning Web Component (LWC) component to access and modify the HTML elements of a web page or HTML. This allows you to extract information from a web page and use it to display or manipulate data within your LWC component. However, it’s crucial to remember that this method is frequently regarded as unreliable and might stop functioning in the future if the layout of the web page changes. It may also be against the terms of service of the website you are scraping, hence it is advised to use an API made available by the website or another approach if at all possible.

Example of DOM scraping

<template>
    <div> <p> I will  change</p> </div>
</template>

import { LightningElement, track } from 'lwc';

export default class DomScrapingExample extends LightningElement {
    @track scrapTheDiv;
    connectedCallback() {
        const scrapTheDiv = this.template.querySelector('div');
        this.scrapTheDiv = scrapTheDiv.textContent;
    }
}

In this example, the connectedCallback method is used to scrape the text content of a div element on the page. The scraped data is stored in the scrapTheDiv property, which is annotated with the @track decorator to indicate that it should be tracked for changes.It’s important to note that this is just an example, and scraping the DOM in this way is generally considered unreliable and may break in the future if the structure of the web page changes.

Example to prevent DOM scraping

To prevent DOM scraping in a Lightning Web Component (LWC), you can use a server-side API to access the information you need rather than scraping the DOM directly. For instance, you may obtain the data in a more dependable and safe manner by using a REST API offered by the website rather than JavaScript to retrieve data from a web page. Here’s a simple example:

import { LightningElement, wire } from 'lwc';
import { getRecord } from 'lightning/uiRecordApi';

const fields = [ 'Account.Name', 'Account.Industry' ];

export default class AccountDetails extends LightningElement {
    @wire(getRecord, { recordId: '0010Y000003eT6pQAE', fields })
    account;

    get name() {
        return this.account.data.fields.Name.value;
    }

    get industry() {
        return this.account.data.fields.Industry.value;
    }
}

In this example, the getRecord wire adapter is used to fetch the data for an Account record. The data is then made available within the component through the account property, which can be accessed using the name and industry getters. Using a server-side API like this eliminates the risk of your code breaking if the structure of the web page changes, and it also complies with the terms of service of the website you are accessing.

Here are some best practices to prevent DOM scraping in your LWC

1. Use server-side APIs: To obtain the required data, rather than directly scraping the DOM, use a server-side API that the website has supplied. This will guarantee a safe and dependable access to the data.
2. Protect your components: Protect your LWC components by making appropriate use of the @api and @track decorators to limit access to data and prevent unauthorised access to private information.
3. Limit the number of requests that can be made to your LWC components in a given time period by implementing rate limitation. This will stop excessive DOM scraping, which could hinder the performance of your website or cause it to load slowly.
4. Encrypt sensitive data: Encrypt sensitive information that is displayed in your LWC components, such as email addresses and phone numbers, to prevent them from being easily scraped.
5. Monitor your logs: Regularly monitor your logs to detect and prevent any unauthorized scraping activities.

Couple of Web Scraping tools

1) Bright Data
2) Apify
3) Oxylabs
4) Zenscrape
5) Smartproxy
6) Scraper API
7) Scrapingbee
8) SCRAPEOWL
9) Agenty
10) Import.io

Disclaimer : This article is not endorsed by Salesforce or any other company in any way. This is my view and knowledge on the topic which I wrote. Please always refer to Official Documentation for the latest information.