Concepts

Featured

Featured

Reading a Next.js Page From Its Own Data

The Page Already Carries Its Answer

A Next.js site renders on the server and sends the data along with the markup so the browser can pick up where the server left off. That is hydration, and it means the values a scraper would dig out of the dom are sitting in the page as json before any of it is turned into elements. Reading them there costs no selectors, survives a class name changing, and gives whole fields that the page may only show part of.

browser.GoTo("https://www.example.com/events")

.GetHydratedData(out NextJsHydrationResult hydrated);


if (true == hydrated.Success)

{

JsonDocument page = hydrated.GetJson(); // the page model, ready to navigate

}

Two Routers, One Result

Next.js carries its data two different ways depending on which router a site uses, and the result says which one it found. The Pages Router puts everything in a single __NEXT_DATA__ script, so IsPagesRouter is true and RawNextDataJson holds it. The App Router streams React Server Components instead, as a run of self.__next_f.push calls, so IsAppRouter is true and FlightChunks holds the pieces. GPAL decodes the stream, reassembles it and buckets the rows, so either way GetJson hands back something navigable.

// what came back, and from which router

Say($"pages router [{hydrated.IsPagesRouter}] app router [{hydrated.IsAppRouter}]");

Say($"title [{hydrated.Title}] from [{hydrated.Url}]");


// or keep it, and read it whenever

browser.SaveHydratedData((GPALFile)GPAL.File.WithFileName(@"C:datapage.json"));

NOTE

A page is extracted once and the result is kept, so asking again on the same page costs nothing. ClearResultCache drops it, along with the sitemap and LLM digest caches, when a page has changed under a workflow that is staying on it.

When There Is Nothing To Read

Success is false on a site that is not Next.js, and Message says so rather than the workflow guessing from an empty result. That is the ordinary answer for most of the web, so a workflow that runs across many sites checks Success and falls back to selectors. Where it does work, it works on every engine and every browser, because it reads the page source rather than driving the page.

WARNING

GetJson hands back a JsonDocument, which holds native memory. Call DisposeJson when done with it, or take the values out and let the result go.

On this page

Featured
Reading a Next.js Page From Its Own Data
  • The Page Already Carries Its Answer
  • Two Routers, One Result
  • When There Is Nothing To Read
    • 💬 Ask GPAL