Concepts

Featured

Featured

Calling a Site's Own API with .Fetch

Why the Request Comes From the Page

A workflow that has driven a browser to a logged-in state holds something valuable: a session the site issued. Reading it back out and replaying it from an ordinary HTTP client carries the cookies and nothing else. The TLS handshake, the HTTP/2 settings, the order and casing of headers, and any clearance bound to those, all belong to the browser. .Fetch avoids the problem by never leaving: the request is issued by the page itself, so everything that makes the browser look like the browser applies by default. Cookies are only the easy half. A session is also whatever the page holds in memory: bearer tokens in JavaScript, values in localStorage, headers a fetch interceptor adds on the way out. None of that is in a cookie jar, and all of it applies when the page's own code path does the work.

// the browser earns the session the ordinary way first

browser.GoTo("https://www.harrods.com/");


// then the page asks its own API, carrying everything it has

browser

.Fetch(productSearch)

.SaveTo(ref json);

Declaring a Request, Then Reusing It

A GPALRequest is built once and reused, the same way a Selector is. Where a Selector says how to find an element, a request says what to ask the site's API for. Numbered tokens are filled from whatever .WithTokensFrom is given, a grid, a file or a database query, one row at a time, and .WithPages walks the paging token for each row. Tokens are plain text substitution and work anywhere the request sends: the path, a parameter value, a header value or the body, so a JSON payload, a form post and an XML body all tokenize the same way. .SaveTo(ref json) hands back a JSON array with one response body per page requested, so one page and twenty come out in the same shape, and CallAfterFetch is invoked per row while the rest are still being retrieved.

static GPALRequest productSearch = (GPALRequest)GPAL.Request

.WithPath("/api/search/1/indexes/prod_harrods.com_query_suggestions/query")

.WithHttpMethod(HttpVerb.Post)

.WithContentType(ContentType.Form)

.WithBody(@"{""query"":""{0}"",""hitsPerPage"":{1},""page"":{page}}")

.WithFirstPage(0)

.WithPageToken("{page}")

.CallAfterFetch(ReportSearch)

.WithName("harrods suggestions");


searchTerms.AddRow(new List<string> { "mens shorts", "5" });


browser

.WithTokensFrom(searchTerms)

.WithPages(3)

.Fetch(productSearch)

.SaveTo(ref json);

NOTE

A page that fails does not stop the run. Whatever went wrong is written into the results with the reason in front of it, so position in the results always maps to the row and page that produced it.

What It Can and Cannot Reach

Fetch works on all three engines. Selenium and Puppeteer issue it as script in the page; the OttoMagic extension cannot run arbitrary JavaScript since Chrome 136, so there it is a declared endpoint that marshals the request instead. Because the request is issued by the page, it obeys the same-origin rules the page does. A site calling its own API is unaffected, which is nearly always the case worth having. A call to a different host will come back empty unless that host returns CORS headers, which is why a page can fire a beacon at a tag manager it never reads the answer from, while asking to read one is refused before it leaves the browser.

WARNING

If a fetch of another host comes back empty, CORS refused it rather than the site. Use .ContinueAsRESTClient for that case: an HTTP client has no origin and no CORS.

On this page

Featured
Calling a Site's Own API with .Fetch
  • Why the Request Comes From the Page
  • Declaring a Request, Then Reusing It
  • What It Can and Cannot Reach
    • 💬 Ask GPAL