Tutorials

Getting Started

Getting Started

Simple Scraper: Search, Paginate, and Save

Selectors Selector Matching Unit of Work Event System Automation Engines

Complete Program

Here's the whole workflow, start to finish. Each piece is broken down and explained below.

using System.Collections.Generic;

using System.Diagnostics;

using GenerallyPositive;

using GenerallyPositive.Browser;

using static GenerallyPositive.Enums;


GPAL

.WithPublishToConsole()

.WithAutoUpdateWebDriver()

.WithTypingDelay(50)

.WithSimulateMouseMovement(true)

.WithDriverLocation("c:\drivers");


IGPALGrid<string> retGrid = GPAL.Grid.ToGPALObject();


Selector nextPageButton = "//a[contains(@class, 'pagination__next icon-link')]";

nextPageButton.WithHardware.WithSelectorName("NextPageButton");


Selector resultCard = "//li[starts-with(@id, 'item')]";

resultCard.WithSelectorName("ResultCard");


Selector imageColumn = ".//img[contains(@class, 's-card__image')]";

imageColumn.WithSelectorName("Image");


Selector descriptionColumn = ".//span[contains(@class, 'su-styled-text primary default')]";

descriptionColumn.WithSelectorName("Description");


Selector linkColumn = ".//a[contains(@class, 's-card__link')]";

linkColumn.WithSelectorName("Link");


Selector priceColumn = ".//span[contains(@class, 'su-styled-text primary bold large-1 s-card__price')]";

priceColumn.WithSelectorName("Price");


Selector searchButton = "#gh-search-btn";

searchButton.WithSelectorName("SearchButton").MatchText("Search");


Selector inputSearchSelector = "#gh-ac";

inputSearchSelector.WithSelectorName("SearchInput");


IBrowser browser = GPAL.Browser

.WithBrowserType(BrowserType.Chrome)

.WithAutomationEngine(AutomationEngine.PuppeteerPort)

.WithWaitOnDocumentReady(30_000)

.GoTo("https://www.ebay.com/")

.ToGPALObject();


browser

.WaitFor(2_000)

.WithSelector(inputSearchSelector)

.FillInFrom("Dog Treats")

.LeftClick(searchButton)

.InElement(resultCard)

.WithSelector(descriptionColumn)

.WithSelector(priceColumn)

.WithSelector(linkColumn)

.WithSelector(imageColumn)

.WithAllThatMatch(10)

.CallIfFound(CallIfFound)

.CallIfNotFound(CallIfNotFound)

.WithNextPageButton(nextPageButton)

.WithPages(3)

.GetGrid(ref retGrid)

.SaveTo("c:\temp\results.txt")

.Close(true);


CallIfStatus CallIfFound(IBrowser browser, List<IGPALElement> foundElements, List<IGPALElement> matchedElements, Selector selector, bool matchedAll)

{

foreach (IGPALElement webElement in foundElements)

Debug.WriteLine($"<{webElement.TagName}> : {webElement.Text} : {webElement.GetAttribute("value")}");


return CallIfStatus.NotHandled;

}


CallIfStatus CallIfNotFound(IBrowser browser, List<IGPALElement> foundElements, List<IGPALElement> matchedElements, Selector selector, bool matchedAll)

{

Debug.WriteLine($"Unable to find [{selector.Name}]");


return CallIfStatus.NotHandled;

}

Configure GPAL & Define Your Selectors

Every workflow starts with a few global settings - here we turn on console logging, auto-update the web driver, and add a human-like typing delay. Then we define each Selector once, give it a friendly name, and (for the search button) require its text to match "Search". Selectors are just values - define them once, reuse them anywhere in the workflow.

GPAL

.WithPublishToConsole()

.WithAutoUpdateWebDriver()

.WithTypingDelay(50)

.WithSimulateMouseMovement(true)

.WithDriverLocation("c:\drivers");


Selector searchButton = "#gh-search-btn";

searchButton.WithSelectorName("SearchButton").MatchText("Search");


Selector inputSearchSelector = "#gh-ac";

inputSearchSelector.WithSelectorName("SearchInput");

NOTE

WithSelectorName isn't just cosmetic - it's the name GPAL uses when it logs information and exception events for that selector, which makes troubleshooting much easier. See Selectors and Selector Matching for the full picture.

Launch the Browser & Navigate

GPAL.Browser is a fluent factory: pick a browser type, pick an automation engine, tell it to wait until the page is ready, and go to a URL. ToGPALObject() returns the live IBrowser you'll drive for the rest of the workflow.

IBrowser browser = GPAL.Browser

.WithBrowserType(BrowserType.Chrome)

.WithAutomationEngine(AutomationEngine.PuppeteerPort)

.WithWaitOnDocumentReady(30_000)

.GoTo("https://www.ebay.com/")

.ToGPALObject();

NOTE

Swapping AutomationEngine.PuppeteerPort for Selenium, OttoMagic, or any of the other 8 engines doesn't change anything below this line. See Automation Engines.

One Chain, One Workflow

This is the heart of GPAL. InElement(resultCard) makes every result card a root, so the column selectors after it (relative .// XPaths) are searched inside each card. Each card gives one cell per column, so a card with two prices does not shift the rows below. WithSelector adds elements to the current unit of work - here, the description, price, link, and image columns for up to 10 results per page. WithAllThatMatch tells GPAL how many matching elements to use. No loops: GPAL applies each action across every matched element for you.

browser

.WaitFor(2_000)

.WithSelector(inputSearchSelector)

.FillInFrom("Dog Treats")

.LeftClick(searchButton)

.InElement(resultCard)

.WithSelector(descriptionColumn)

.WithSelector(priceColumn)

.WithSelector(linkColumn)

.WithSelector(imageColumn)

.WithAllThatMatch(10);

React, Paginate, and Save

CallIfFound and CallIfNotFound let you react to whether a selector matched anything - here we just log element info, or the name of the selector that found nothing. WithNextPageButton plus WithPages handles pagination automatically. GetGrid pulls everything into an in-memory grid, and SaveTo writes it straight to a file.

.CallIfFound(CallIfFound)

.CallIfNotFound(CallIfNotFound)

.WithNextPageButton(nextPageButton)

.WithPages(3)

.GetGrid(ref retGrid)

.SaveTo("c:\temp\results.txt")

.Close(true);

WARNING

Passing true kills the running web driver process. Use true when the workflow is completely done; use false if another workflow in the same run still needs that driver.

On this page

Getting Started
Simple Scraper: Search, Paginate, and Save
  • Complete Program
  • Configure GPAL & Define Your Selectors
  • Launch the Browser & Navigate
  • One Chain, One Workflow
  • React, Paginate, and Save
    • 💬 Ask GPAL