Concepts

Basic Concepts

Basic Concepts

Page Readiness: Document Ready vs Network Idle

WithWaitOnDocumentReady: the Obvious Default

WithWaitOnDocumentReady(timeoutInMs) waits for the browser's own page-load signal -- the same moment a traditional page fires its load event. For a simple server-rendered page, that's a good proxy for 'ready': by the time the document is loaded, the content is there. Switching between WithWaitOnDocumentReady and not using it at all is usually a no-brainer -- it's the setting most people reach for first, and the name says exactly what it does.

GPAL.Browser

.WithBrowserType(BrowserType.Chrome)

.WithWaitOnDocumentReady(10_000)

.GoTo("https://example.com");

WithWaitOnIdleConnection: a Better Signal for Async Pages

Many modern pages fire their load event almost immediately and then make a wave of background requests. On those pages, document-ready fires too early: the DOM exists, but the data you want to scrape hasn't arrived yet. WithWaitOnIdleConnection(true) instead waits until the browser's network activity drops to (and stays at) a small number of connections for a minimum duration. In other words, it waits until the page has stopped fetching things, which is a much closer match to 'the content is actually here' for async-heavy pages.

// wait until 0 connections are active for at least 1 second

GPAL.Browser

.WithBrowserType(BrowserType.Chrome)

.WithWaitOnIdleConnection(true)

.WithNetworkIdleMaxConnections(0)

.WithNetworkIdleTimeoutMs(1_000)

.GoTo("https://example.com/dashboard");

NOTE

WithNetworkIdlePruneMs (default 10 seconds) controls how long GPAL retains connection history when deciding whether the network is idle. A page that polls an endpoint every few seconds may never reach 0 active connections long enough to trigger idle. Try setting WithNetworkIdlePruneMs just below the polling interval first. If that does not produce a clean idle window, raise WithNetworkIdleMaxConnections to allow a small number of background connections to count as idle.

Which One Should You Use?

Start with WithWaitOnDocumentReady -- it's simple, fast, and correct for the majority of pages. Reach for WithWaitOnIdleConnection specifically when a workflow's selectors aren't being found even though the page 'looks loaded' in a screenshot -- that's the signature of a page whose real content arrives after an async fetch. Choose one. Whichever is configured last is the strategy GPAL applies.

WARNING

Document-ready is a familiar concept from general web development, so toggling WithWaitOnDocumentReady on or off feels obvious. Network-idle waiting is GPAL-specific and far less discoverable. If a workflow intermittently misses content on a data-heavy page, this section is the one to come back to.

On this page

Basic Concepts
Page Readiness: Document Ready vs Network Idle
  • WithWaitOnDocumentReady: the Obvious Default
  • WithWaitOnIdleConnection: a Better Signal for Async Pages
  • Which One Should You Use?
    • 💬 Ask GPAL