Concepts

Data Operations

Data Operations

Getting Multiple Results: WithAllThatMatch & Pagination

One Page: WithAllThatMatch by Itself

By default GetGrid returns every matching element on the current page, and WithAllThatMatch() with no number says the same. WithAllThatMatch(N) returns up to N matches from the current page only. GPAL does not page or scroll unless WithNextPageButton or WithInfiniteScroll is also specified, with WithPages for how many pages to read.

Pagination: Per-Page Cap vs Total Cap

Add WithNextPageButton or WithInfiniteScroll, with WithPages(n) for how many pages to read, and WithAllThatMatch takes on a second meaning. On each page, if N is less than or equal to the results found there, N caps that page. If N is greater than that page's results, N becomes a total cap across all pages combined, and GPAL stops once that many rows have been collected. Paging also stops after the last page asked for, or when no next page remains.

Selector result = GPAL.Selector.WithCSS("li.result").ToGPALObject();

Selector nextBtn = GPAL.Selector.WithCSS("a.next").ToGPALObject();


// Per-page cap: top 3 results from each page,

// up to 10 pages or until no Next button is found

IGPALGrid<string> top3 = GPAL.Grid.ToGPALObject();

browser.WithSelector(result)

.WithAllThatMatch(3)

.WithNextPageButton(nextBtn)

.WithPages(10)

.GetGrid(ref top3);


// Total cap: up to 50 results across as many pages

// as needed (50 > results on a page)

IGPALGrid<string> first50 = GPAL.Grid.ToGPALObject();

browser.WithSelector(result)

.WithAllThatMatch(50)

.WithNextPageButton(nextBtn)

.WithPages(10)

.GetGrid(ref first50);


// No cap: every result on up to 10 pages

IGPALGrid<string> all = GPAL.Grid.ToGPALObject();

browser.WithSelector(result)

.WithAllThatMatch()

.WithNextPageButton(nextBtn)

.WithPages(10)

.GetGrid(ref all);

NOTE

The cap mode is chosen on every page, from that page's result count, so a short page can stop the run early.

Infinite Scroll and Duplicate Rows

WithInfiniteScroll has no Next button to click. GPAL scrolls to the bottom of the page and re-queries the same selectors. Pages that append new content below the old usually leave previously-seen elements in the DOM, so a naive re-query would return them again. Pass GetGrid a grid set up with WithDedupeData(true) and GPAL drops any row whose values match a row already collected, no matter how many scrolls it survives.

WARNING

Sites that recycle DOM nodes as you scroll (virtualized feeds like Facebook or X) may reuse the same element for different content, or remove old rows entirely. Deduplication handles the common append-only case, but cannot guarantee complete, duplicate-free results on a virtualized feed.

On this page

Data Operations
Getting Multiple Results: WithAllThatMatch & Pagination
  • One Page: WithAllThatMatch by Itself
  • Pagination: Per-Page Cap vs Total Cap
  • Infinite Scroll and Duplicate Rows
    • 💬 Ask GPAL