Concepts

Featured

Featured

Seeing What a Page Asks For with .CaptureCalls

A Recording, Read When You Ask

Turn capture on and every request the browser makes is kept as a GPALCall. Ask for them and you get a snapshot of everything recorded so far, taken at that moment, so a workflow reading the list is never reading what the browser is still writing. Saying it again refreshes the list, which means a workflow can watch a page over time without a callback firing on the reader thread. The status is the one the page itself received, which is how you tell an endpoint worth calling from one the site is failing at too: a 200 next to a call is an invitation, a 400 next to it means the page is not getting what you thought either.

browser

.GoTo("https://www.example.com/rooms")

.CaptureCalls(out List<GPALCall> calls);


foreach (GPALCall call in calls)

if ("XHR" == call.ResourceType || "Fetch" == call.ResourceType)

GPAL.PublishSimpleEvent(GPALEventType.NOTICE,

$"{call.Status} {call.Method} {call.Url}");

Narrowing What Is Kept

A page commonly asks for two or three hundred things, most of them images, scripts and telemetry. .WithCallFilter narrows recording to calls whose url holds a fragment, which keeps a long run from accumulating everything and makes the list readable. The filter narrows what is recorded from the moment it is set; anything captured before it was set is still in the list, so a workflow that captures broadly and then narrows still sees its earlier calls. The headers on each call are the ones the page's own code attached, which is the half a request written by hand always lacks, and they are handed to .WithHeader unchanged.

browser

.WithCallFilter("/availability-api/")

.CaptureCalls(out List<GPALCall> calls);


foreach (KeyValuePair<string, string> header in calls[0].Headers)

GPAL.PublishSimpleEvent(GPALEventType.NOTICE, $"{header.Key}: {header.Value}");

NOTE

The CallSpy sample program is a form over these calls: watch a page, filter to XHR and Fetch, read a call's headers and body, and reissue it both ways. It is the quickest route from a site you have never seen to a request you can write. ClearCapturedCalls forgets what has been recorded without stopping the recording, so a workflow can discard a noisy page load and watch only what the next step asks for.

What Each Engine Can See

All three engines record, by three different means, and the means decides what is seen. Puppeteer reports every request over the DevTools protocol: documents, scripts, images, XHR and Fetch, with request headers and response status. OttoMagic watches with the extension's webRequest listener, which sees the same breadth from outside the page, keeps recording across navigations on its own, and injects nothing for a site to notice. Selenium has neither, so the page records for itself: fetch and XMLHttpRequest are wrapped and each call is pushed onto a value the workflow reads back. That sees what page script asks for, which is XHR and Fetch, and not the browser's own requests for documents, scripts and images. On Chrome and Edge that recorder is registered before the first line of every document; on Firefox it can only be injected once a page has loaded, so that page's first burst is missed.

WARNING

The same workflow runs on all three, but Selenium is the one to check a result against. It records from inside the page, so a call the browser made without page script, and anything a page issued before the recorder was in place on Firefox, is not there. Puppeteer and OttoMagic both watch from outside the page and have neither gap.

On this page

Featured
Seeing What a Page Asks For with .CaptureCalls
  • A Recording, Read When You Ask
  • Narrowing What Is Kept
  • What Each Engine Can See
    • 💬 Ask GPAL