Library

Browser

Browser

Data Retrieval

GetGrid retrieves the matched selector data into an IGPALGrid for further processing. It takes the grid by ref: pass a grid and GetGrid ADDS its rows into it (it never replaces your reference), so reusing one grid across several GetGrid or multi-page calls accumulates a combined dataset; pass null to get a fresh grid back.

Duplicate-row detection is opt-in - call .WithDedupeData(true) on the grid to have GPAL skip duplicate rows. Duplicates are common in the wild and are not tied to infinite scroll: eBay, for example, returns duplicate search results on an ordinary paged run. It is off by default because it fingerprints every element to detect duplicates, which slows GetGrid down (noticeably on slower engines like Selenium); for large result sets, deduping the raw extracted data with another tool may be a better fit than deduping at the element level.

A single selector pointed at a table is read as a table: rows come from its rows, columns from the cells, and the header row names the columns. The header is a thead row, or a leading row whose cells are all th. It is span aware, so a cell spanning two columns repeats across both, and short rows are padded with GPAL.ErrorPlaceholder to keep every row the same width. Paging, WithAllThatMatch and duplicate-row detection work there as they do anywhere else. Point one selector at a set of tr rows instead of the table and the rows are read the same way, with no header. Browser.ColumnNames carries the names the table gave, and is empty on a column-per-selector read.

Two things to expect from it: each row is asked for its own cells, so a two hundred row table is two hundred requests, fast on Selenium and Puppeteer and slow on OttoMagic where every one is an HTTP round trip through the extension, so cap it with WithAllThatMatch when you only need the top of a table; and one selector aimed at a table now comes back as a grid of named columns rather than as a single column holding the table's text.

GetPageSource extracts the full HTML of the current page. GetSiteMap retrieves the sitemap XML and GetSiteMapUrls returns the individual URLs as a list. GetHydratedData extracts Next.js hydration data embedded in the page. GetLLMDigest produces a cleaned, structured summary of the page content optimized for use as LLM input. The Save variants (including SaveSiteMapUrls) write these outputs directly to files, and a sitemap walk that found nothing writes a result node saying so rather than an empty file. WithDefaultErrorPlaceholder sets the text written into a grid cell GetGrid could not read, so a missing value is visible rather than blank.

NOTE

GetGrid's parameter is ref, not out. If you pass an existing grid it appends the results into that grid and leaves your reference intact - that is how you accumulate rows from several searches or pages into one master grid. Passing null hands you back GPAL's internal grid instead. Duplicate-row detection is opt-in via .WithDedupeData(true): use it when a site serves duplicate results and you want GPAL to remove them. Duplicates are not unique to infinite scroll - we saw eBay return duplicate results on a normal, non-infinite-scroll run. It is off by default and it slows GetGrid because it hashes each element's attributes to fingerprint every row; for big result sets, deduping the raw extracted data with another tool is often a better fit than deduping at the element level.

WARNING

Not every site publishes a sitemap, and some answer a walk with a bot-check page instead of xml. GetSiteMapUrls hands back an empty list and publishes a WARNING naming the url it tried and what it read. SaveSiteMapUrls writes the file all the same, carrying one row of result and message in place of the url nodes, so nothing downstream has to tell an empty file apart from a run that broke. The result is no-sitemap-found when the site published none, and bot-check-page when an interstitial answered the walk, which reads as "try a signed-in profile" rather than "this site has no sitemap". In yaml that is "result: no-sitemap-found" with a "message:" beside it, in json an object with those two keys, in xml a row element holding both, and in csv a result,message header and one line. Read the result node first: a file with no url nodes in it has one.

Examples

GPAL Fluent: High-level fluent C# API

//GetGrid adds rows into the grid you pass by ref (one row per matched element, columns per selector) - reuse a grid to accumulate across calls, or pass null for a fresh one. .WithDedupeData(true) makes GPAL drop duplicate rows (sites like eBay serve them); it is off by default because fingerprinting each element slows GetGrid.

// Extract a table into a grid (pass by ref)

var grid = GPAL.Grid.ToGPALObject();

GPAL.Browser

.GoTo("https://example.com/data")

.WithSelector((Selector)".data-row")

.WithAllThatMatch(1000)

.GetGrid(ref grid);


// Accumulate the rows from several searches into ONE grid

var master = GPAL.Grid.ToGPALObject();

foreach (var term in searchTerms)

GPAL.Browser

.GoTo($"https://example.com/search?q={term}")

.WithSelector((Selector)".result-row")

.GetGrid(ref master); // each call appends into master


// Opt in to duplicate-row detection - some sites (e.g. eBay) return duplicate results even on a normal run

var deduped = GPAL.Grid.ToGPALObject().WithDedupeData(true);

GPAL.Browser

.GoTo("https://www.ebay.com/sch/i.html?_nkw=widget")

.WithSelector((Selector)".s-item")

.WithAllThatMatch(200)

.GetGrid(ref deduped); // GPAL skips duplicate rows - slower, it fingerprints each element


// Get page HTML source

string html;

GPAL.Browser

.GoTo("https://example.com")

.GetPageSource(out html);


// Get all URLs from a sitemap

List<string> urls;

GPAL.Browser

.GoTo("https://example.com")

.GetSiteMapUrls(out urls);

On this page

Browser
Data Retrieval
  • Examples
    • 💬 Ask GPAL