Concepts

AI and LLM

AI and LLM

LLM-Ready Page Digests

GetLLMDigest and SaveLLMDigest

GetLLMDigest strips scripts, navigation, ads, and other boilerplate from the current page, finds the main content, and converts it to markdown, returning an LLMDigestResult with Markdown, Title, Url, OriginalLength, CleanedLength, Success, Message, and RuleSetUsed. SaveLLMDigest does the same and writes the result straight to a GPALFile, prefixed with a short header.

LLMDigestResult digest;

GPAL.Browser

.GoTo("https://example.com/article")

.GetLLMDigest(out digest);


GPAL.PublishSimpleEvent(GPALEventType.NOTICE, $"[{digest.Title}] ([{digest.CleanedLength}] of [{digest.OriginalLength}] bytes)");

GPAL.PublishSimpleEvent(GPALEventType.NOTICE, $"[{digest.Markdown}]");


// Or write the digest straight to a file

GPAL.Browser

.GoTo("https://example.com/article")

.SaveLLMDigest("article-digest.md");

Choosing a Rule Set

Each digest is built using one named DigestRuleSet from LLMDigestRules.yaml. If WithDigestRules names a rule set, that rule set is used until it is changed. Otherwise GPAL checks the current page URL against each rule set's domainPatterns and uses the first match. Either way, the name of the rule set it used is returned on digest.RuleSetUsed, so a workflow can confirm or branch on it.

// See which rule set GPAL picked

LLMDigestResult digest;

GPAL.Browser

.GoTo("https://www.indeed.com/jobs?q=developer")

.GetLLMDigest(out digest);


GPAL.PublishSimpleEvent(GPALEventType.NOTICE, $"[{digest.RuleSetUsed}]"); // "indeed" - matched by domain


// Force a rule set regardless of the page's URL

GPAL.Browser

.GoTo("https://careers.example.com")

.WithDigestRules("indeed")

.SaveLLMDigest("careers-digest.md");

NOTE

If no rule set is named and the URL matches no domainPatterns, or if a named rule set leaves a field empty, GPAL uses the "generic" rule set's value for that field instead. A rule set only needs to define what makes that site different from generic.

Editing LLMDigestRules.yaml

Rule sets live in LLMDigestRules.yaml so layouts can be tuned without recompiling. If the file does not exist, call DigestRulesConfig.Save() to write a starter with the built-in rule sets. The dedicated Tuning LLMDigestRules.yaml page covers each field in full.

// write a starter file next to the exe, then edit userRuleSets in it

DigestRulesConfig.Save();


// name a rule set rather than letting the URL choose one

LLMDigestResult digest;


GPAL.Browser

.GoTo("https://mysite.com/article")

.WithDigestRules("my-site")

.GetLLMDigest(out digest);


// the links in the digest carry the page's origin, so the file reads on its own

GPAL.PublishSimpleEvent(GPALEventType.NOTICE,

$"[{digest.RuleSetUsed}] took [{digest.OriginalLength}] down to [{digest.CleanedLength}]");

WARNING

Cleanup step names and XPath expressions are matched literally. A typo in a step name is silently skipped, and an invalid XPath expression is skipped for that page rather than stopping the digest.

On this page

AI and LLM
LLM-Ready Page Digests
  • GetLLMDigest and SaveLLMDigest
  • Choosing a Rule Set
  • Editing LLMDigestRules.yaml
    • 💬 Ask GPAL