Tutorials

REST and APIs

REST and APIs

Calling a Site's Own API with Fetch

Calling a Site's Own API with .Fetch Seeing What a Page Asks For with .CaptureCalls The Fluent Interface GPALConverter: Converting Between Formats

Complete Program

Three search terms, three pages each, nine calls, one JSON array out. The browser visits the homepage once to earn a session and never touches the DOM again.

using GenerallyPositive;

using GenerallyPositive.Browser;

using static GenerallyPositive.Enums;


GPAL.WithPublishToConsole();


// declared once and reused, the same way a Selector is

GPALRequest searchRequest = (GPALRequest)GPAL.Request

.WithPath("/api/search/1/indexes/prod_query_suggestions/query")

.WithHttpMethod(HttpVerb.Post)

.WithParameter("x-algolia-agent=Algolia for JavaScript (4.26.0); Browser (lite)")

.WithContentType(ContentType.Form)

.WithBody(@"{""query"":""{0}"",""hitsPerPage"":{1},""page"":{page}}")

.WithFirstPage(0)

.WithPageToken("{page}")

.WithName("suggestions");


// one row per search, one column per token

IGPALGrid<string> searchTerms = GPAL.GridForType<string>();

searchTerms.AddRow(new List<string> { "handbag", "12" });

searchTerms.AddRow(new List<string> { "watch", "12" });

searchTerms.AddRow(new List<string> { "scarf", "12" });


IBrowser browser = GPAL.Browser

.WithBrowserType(BrowserType.Chrome)

.WithAutomationEngine(AutomationEngine.PuppeteerPort)

.WithDriverLocation(@"C:drivers")

.ToGPALObject();


string json = null;


browser

.GoTo("https://www.example.com/") // warm up, so the session cookies are set

.WithTokensFrom(searchTerms) // one row per search, filling {0} and {1}

.WithPages(3) // three calls per row, {page} becoming 0, 1, 2

.Fetch(searchRequest)

.SaveTo(ref json);


browser.Close(true);


GPAL.Converter

.WithInput(json)

.SaveTo((GPALFile)(GPALFile)"suggestions.json");

Warm Up First

Going straight at an inner path or the API gets refused. The cookies that clear an anti-bot check are only set by a real navigation, so the workflow visits the homepage first and lets the site hand them over. After that the browser never reads an element again.

browser.GoTo("https://www.example.com/");

WARNING

A fetch issued before any navigation has no session behind it and will read like a bare HTTP client. If the site would block curl, it will block this too.

Declare the Request Once

GPAL.Request describes an API call the way a Selector describes an element. Path, method, content type, body and headers are stated once, given a name, and reused for every row. Nothing about it is tied to a particular search.

GPALRequest searchRequest = (GPALRequest)GPAL.Request

.WithPath("/api/search/1/indexes/prod_query_suggestions/query")

.WithHttpMethod(HttpVerb.Post)

.WithContentType(ContentType.Form)

.WithBody(@"{""query"":""{0}"",""hitsPerPage"":{1},""page"":{page}}")

.WithName("suggestions");

NOTE

A path on the site's own domain is a same origin request, so the session the browser earned is all it needs and there is no CORS preflight to think about. A body sent as form-urlencoded rather than JSON keeps it that way.

Tokens Are Substring Replacement, Nothing Cleverer

The numbered markers in the body match the columns of the grid: {0} is the first column, {1} the second. GPAL does not parse the payload and does not care whether it is JSON, a form or a query string. Where a marker goes and what the payload is are entirely the workflow's business.

.WithBody(@"{""query"":""{0}"",""hitsPerPage"":{1},""page"":{page}}")


// one row per search, one column per token

searchTerms.AddRow(new List<string> { "handbag", "12" });

Paging Is Its Own Token

WithPageToken names the marker GPAL fills in itself, WithFirstPage says what the site counts from, and WithPages says how many calls to make per row. Everything else in the body is the row's business. Three rows and three pages is nine requests, issued in the order asked for.

.WithFirstPage(0) // this site counts pages from zero

.WithPageToken("{page}") // GPAL fills this one, the row fills the rest


browser

.WithTokensFrom(searchTerms)

.WithPages(3)

.Fetch(searchRequest);

Collect the Answers

SaveTo hands back one JSON array holding every response body, every row's pages, in the order requested. From there it is a normal GPALConverter job: to a file, to a class, to a grid, to a database.

string json = null;


browser

.WithTokensFrom(searchTerms)

.WithPages(3)

.Fetch(searchRequest)

.SaveTo(ref json);


GPAL.Converter

.WithInput(json)

.SaveTo((GPALFile)(GPALFile)"suggestions.json");

Or Handle Each Response As It Lands

CallAfterFetch fires once per response, so a long run reports as it goes rather than at the end. The callback is declared on the request, alongside everything else about it.

.CallAfterFetch(ReportSearch)


// ...


static void ReportSearch(string response)

{

GPAL.PublishSimpleEvent(GPALEventType.NOTICE, $"[{response?.Length ?? 0}] characters back");

}

Why Not Just Call It From Outside

You can, and the next tutorial shows how. The difference is what travels. Fetch runs inside the page, so it carries the cookies, the user agent, the header order and the TLS handshake, all of it. A client outside the browser carries what can be handed over, which is everything except the transport. Where a site's wall is the cookie jar, both work. Where the wall inspects the TLS fingerprint, only the fetch gets through.

NOTE

Running the same request both ways and comparing the answers tells you which wall a site is actually using. If the handed-over client gets a deny page and the in-page fetch gets data, the site is fingerprinting the transport rather than checking a cookie.

On this page

REST and APIs
Calling a Site's Own API with Fetch
  • Complete Program
  • Warm Up First
  • Declare the Request Once
  • Tokens Are Substring Replacement, Nothing Cleverer
  • Paging Is Its Own Token
  • Collect the Answers
  • Or Handle Each Response As It Lands
  • Why Not Just Call It From Outside
    • 💬 Ask GPAL