Tutorials

Tutorials

Paging with Workflows: While and Until

WithWorkflow records a chain of AndThen calls as a repeatable block; While and Until run that block under a condition you control, and GetExecutionResults keeps every pass's named results so you can read them all back afterward - or feed one pass's result into the next pass's condition.

Complete Program

Here's the whole workflow, start to finish. Each piece is broken down and explained below.

using GenerallyPositive;

using static GenerallyPositive.Enums;

using System;

GPAL.WithUseOttoMagic(@"C:OttoMagic").WithPublishToConsole();

IRESTClient client = GPAL.OttoMagicClient.WithName("Pager").ToGPALObject();

int page = 0;

const int maxPages = 5;

client

.WithWorkflow(c => c

.GoTo($"https://example.com/search?q=widgets&page={++page}")

.AndThen()

.CheckNetworkIdle()

.AndThen()

.WithResultName("pageUrl")

.GetCurrentUrl()

.Execute<string>())

.Until(() => page >= maxPages);

foreach (var url in client.GetExecutionResults().GetAllResults("pageUrl"))

Console.WriteLine(url);

WithWorkflow: Recording a Block to Repeat

WithWorkflow(c => ...) doesn't run anything by itself - it records the lambda, plus a snapshot of the client's current endpoint, parameters, HTTP method, and pending result name at the moment WithWorkflow was called. Every pass through the loop restores that snapshot first, then runs the lambda against it, so each pass starts from the same clean configuration rather than continuing the chain from wherever the previous pass left off. Inside the lambda, c is the same client - AndThen/AndThen<T>, WithResultName, and the rest of the fluent surface all work exactly as they do outside a workflow. The lambda's last call is the one that produces the pass's named result, and since nothing follows it in the chain, Execute<T>() is the cleaner choice over AndThen<T>() - the <T> is the type of the result being retrieved.

client

.WithWorkflow(c => c

// one pass: navigate, wait for the network, then capture the URL

.GoTo($"https://example.com/search?q=widgets&page={++page}")

.AndThen()

.CheckNetworkIdle()

.AndThen()

.WithResultName("pageUrl")

.GetCurrentUrl()

.Execute<string>());

// nothing has executed yet - While or Until is what actually runs it

Until and While: Conditions You Control

Until(condition) repeats the recorded workflow until condition() returns true; While(condition) repeats it for as long as condition() returns true. Both check the condition before every pass, including the first - so a workflow can run zero times if the condition is already satisfied. Capturing a plain C# variable (page) in the lambda is the simplest condition: ++page increments before the comparison, so starting page at 0 means the first pass sees page == 1, the second sees page == 2, and so on through maxPages.

// Until: keep going while the condition is false

int page = 0;

client

.WithWorkflow(c => c

.GoTo($"https://example.com/search?q=widgets&page={++page}")

.AndThen()

.CheckNetworkIdle()

.AndThen())

.Until(() => page >= maxPages);

// While: keep going while the condition is true - same five passes

page = 0;

client

.WithWorkflow(c => c

.GoTo($"https://example.com/search?q=widgets&page={++page}")

.AndThen()

.CheckNetworkIdle()

.AndThen())

.While(() => page < maxPages);

TIP

Both While and Until stop after 1000 passes even if the condition never resolves, and publish a WARNING event when that cap is hit. Treat the cap as a safety net for a runaway condition, not as a loop-count you should rely on.

Reading Back Every Pass with GetAllResults

Each pass through the workflow adds another entry under the same result name, so GetExecutionResults() ends up holding one "pageUrl" per pass rather than overwriting the last one. results["pageUrl"] returns the most recent pass's value; results.GetAllResults("pageUrl") returns every pass's value as a list, in the order the passes ran; and results["pageUrl", n] returns the value from a specific pass by its zero-based iteration number.

var results = client.GetExecutionResults();

string lastUrl = (string)results["pageUrl"];

string firstUrl = (string)results["pageUrl", 0];

foreach (var url in results.GetAllResults("pageUrl"))

Console.WriteLine(url);

Advanced: Driving While from a Result the Workflow Produces

IsEndOfPage() reports whether the page is scrolled to its bottom - the right condition for infinite-scroll content, not for the URL-based ?page=N pagination used above. Because While/Until check their condition before every pass - including the first - a condition built around IsEndOfPage has nothing to read on that first check. The fix is to prime the result with one chain before the workflow exists, using the same WithResultName/Execute<T> pair the workflow will use on every later pass. Since results["name"] always returns the most recent value with that name, the primed value covers the first check, and each pass's own result covers every check after that. Both chains end with Execute<bool>() rather than AndThen<bool>() - AndThen<bool>() would also execute and store the result, but Execute<T>() is the cleaner final call when nothing follows it, and the <bool> is the type of the result being retrieved.

// Prime "atBottom" so the first While check has something to read

client

.ScrollWindowByVertical(800)

.AndThen()

.CheckNetworkIdle()

.AndThen()

.WithResultName("atBottom")

.IsEndOfPage()

.Execute<bool>();

client

.WithWorkflow(c => c

.ScrollWindowByVertical(800)

.AndThen()

.CheckNetworkIdle()

.AndThen()

.WithResultName("atBottom")

.IsEndOfPage()

.Execute<bool>())

.While(() => false == (bool)client.GetExecutionResults()["atBottom"]);

int scrollPasses = client.GetExecutionResults().GetAllResults("atBottom").Count;

Console.WriteLine($"Reached the bottom after {scrollPasses} scroll passes.");

WARNING

If you skip the priming chain, the first While/Until check reads a result that doesn't exist yet. results["name"] returns null in that case and publishes a WARNING event - casting that null to a value type like bool throws. Always run one chain that produces the result under the same name before handing the workflow to While or Until.

On this page

Tutorials
Paging with Workflows: While and Until
  • Complete Program
  • WithWorkflow: Recording a Block to Repeat
  • Until and While: Conditions You Control
  • Reading Back Every Pass with GetAllResults
  • Advanced: Driving While from a Result the Workflow Produces