[↗] GIDS series

preprint · part 4 · 5 of 7

Part 4: Benchmarking the World Model

Part 3 supplied a framework. Part 4 makes it answerable to some sort of data.

We’ve argued so far that an observer can be represented through an estimated Transcendental Embedding, that this estimate can be split into slow and fast state, that propositions can be represented through the distinctions they present to the observer, and that corporations can enter either as structured context or, later, as composite actors. But any framework that does not specify what counts as state, what data instantiates that state, what task is being predicted, what stronger baselines it must beat, and what evidence would justify proposition optimization is nascent and not really worth anyone’s time.

This section deals with benchmarking.

The purpose of Part 4 is not to prove the full metaphysical claim directly. It is to ask a narrower question: if we represent a salesperson–executive interaction as a dyadic state containing slow person structure, fast local state, two company contexts, relationship history, world state, and a proposition, do we predict observable transitions better than simpler models? And if we later use that model to choose propositions, do we have the intervention machinery to say something more than “the simulator liked this one”?

The canonical symbol definitions are in Canonical Notation and Mathematical Conventions.

4.1 Operational Definition of State

Philosophically, the state of an organism is total. It includes perception, interoception, memory, action tendency, and the actions already underway. But benchmarks do not get to be mystical.

For executive , let

denote the full phenomenal state. It is not directly observed.

Let

denote the general predictive response state. When a task-and-horizon summary satisfying the required sufficiency condition exists, write it as

This is a current-time predictive summary. It is not the object recursively rolled through the simulator.

The benchmark operates on a measurable dyadic approximation. Let salesperson work for company , and executive work for company . Define

Let be the random proposition selected at decision epoch , and let be its realized value, where . The proposition remains separate because the benchmark asks what happens when a particular proposition encounters the current dyadic state.

Let denote the random exogenous change between decision epochs and a realized or supplied scenario value. Expressions of the form mean evaluation of a parameterized kernel at the supplied scenario value, not conditioning on a necessarily positive-probability singleton. The notation below is shorthand for the typed encoder–interaction–transition factorization specified in Section 4.4. The measurable one-step simulation is

and the immediate trace-bundle law for records attributed to the interval is

This readout assumes the next operational state screens off the previous state and proposition for the immediate trace. If an implementation rejects that Markov-style emission assumption, use and evaluate the added conditioning empirically.

For delayed decision-associated outcomes and probes used together, write the coherent regime-specific law as

The horizon-specific kernels and are its marginals or conditionals when a joint law is modeled. A direct horizon-specific law is defined only relative to a declared continuation policy, future candidate-set process, exogenous-path regime, and censoring convention unless those quantities are conditioned on explicitly. It may instead be induced by a rollout. Independently trained marginal heads do not define cross-head dependence; a utility combining them must use marginal expectations, a declared coupling, or a joint model. A direct head and a rollout used in the same score must refer to the same evaluation regime rather than generating duplicate futures. Either way, a 90-day close is not represented as an immediate event emitted at .

The hierarchy is therefore explicit:

The benchmark only has access to the last object.

4.2 Event Time and Dataset Construction

The clock has to be clean or the entire benchmark becomes leakage with equations around it.

Let index decision epochs. Define the pre-decision history by

where each is a timestamped record that was fully available before proposition was selected or delivered. Write and define

Records arrive asynchronously, so need not equal . The decision-aligned latent states are the record-indexed states after exactly these records have been processed. Timestamp ties require an explicit logging order that preserves the actual decision-before-response sequence.

At decision time :

  1. construct from and all other information available at that moment;
  2. construct or retrieve the logged candidate set ;
  3. select and deliver ;
  4. record the decision and make the forecast before observing any response;
  5. append response, meeting, company-change, and other observation records only when they actually occur;
  6. attach delayed labels to the original decision row only after their horizons mature.

This avoids the old ambiguity in which a history through time could already contain the response to the proposition being predicted. It also avoids pretending a 90-day outcome is known before the next sales touch.

Use two timestamped record types. A decision record is

where is time since the prior decision or contact, is the source-tagged categorical representation available from the decision itself, and is the information actually available to the behavior policy. The logged assignment probability for a discrete proposition, or assignment density for a continuous proposition, is

This distinction matters. The denominator in off-policy evaluation is the probability the historical policy actually assigned using the information it actually had. It is not a probability retroactively conditioned on a richer state reconstructed later.

An observation record arriving at clock time is

Here is the observed interaction trace, is an observable memory proxy such as a resurfaced objection, and the remaining terms are newly observed changes in company or world state. The chronological history contains every decision and observation record whose timestamp precedes decision .

Decision-associated labels

A single decision can carry several outcomes at different horizons. Define the primary head index set

The primary outcome bundle attached to decision is

For censoring and label maturity, define

A label with is unobserved, missing, or censored. It is not a negative. If censoring depends on variables related to the outcome, use a survival likelihood, inverse-probability-of-censoring weights, or a joint model rather than complete-case optimism.

Write the primary availability bundle as

Define the auxiliary probe bundle as

with its own availability masks when probes mature at different times. Write

Event-time dataset

For the first benchmark, define

Here:

  • and are person-side inputs available before ;
  • and are measurable company-side input bundles;
  • is the pre-proposition relationship-event history;
  • is market and world state;
  • is the candidate set actually available at the decision;
  • is the primary multi-horizon outcome bundle;
  • is the auxiliary probe bundle;
  • and contain primary-label and probe availability indicators;
  • is the logged probability or density assigned to the realized action;
  • records what the behavior policy knew when it made the decision.

Every feature in a row must be available before the proposition is chosen. Later CRM fields, transcript summaries produced from the response, post-meeting notes, eventual stage changes, and other downstream information cannot leak backward into .

Person-side inputs

For each person , let

where the components are psychometric or cognitive proxies, biography, language and discourse, role and institution history, observable life or professional history, and slow categorical trace memory.

If a coordinate cannot be inferred cleanly from real data, mask it. Do not invent variables because they sound sophisticated.

Company-side inputs

For each company , let

where the components contain measurable facts and statistics, authority and communication structure, institutional memory proxies, incentives and constraints, and environmental history already absorbed into the corporation. Examples include size, industry, growth, funding, revenue proxies, technology, hiring, leadership, governance, recent events, and account history.

These inputs are not the corporation-state by themselves. They are evidence supplied to the corporate aggregation mechanism.

Relationship state

Let

The relationship is not reducible to either person. It is an evolving object created by their prior interaction.

Proposition representation

The proposition should be logged in structured form whenever possible:

Free-form text can be embedded, but intervention and off-policy evaluation become far easier when the action space also contains controlled dimensions such as message family, proof type, call-to-action, offer, channel, and timing.

Outcomes and probes

A first primary index set can be

Thus

The auxiliary bundle may include

with a declared horizon and availability mask for every component.

These probes are not meant to reveal the one true hidden motive of the prospect. They test whether the latent state carries reusable structure beyond one binary target.

One eventual close should not be copied backward and treated as though every preceding touch independently caused it. For one-step forecasting, each decision row receives only the outcomes defined for its own matured horizon. For proposition-sequence optimization, reward and credit assignment belong to the trajectory: use step rewards, sequence-level returns, time-to-event targets, or another explicit attribution rule rather than awarding the same terminal event to every message.

The first dataset should be built from CRM events, email logs, call transcripts, meeting records, sender metadata, account metadata, company descriptors, and known market state. If psychometric proxies or detailed corporate features are unavailable, run the benchmark without them first. The framework is supposed to discover what helps, not reward the imagination of the researcher.

4.3 The Benchmark

The benchmark is simple: does an explicit dyadic predictive-state model beat weaker baselines on future data, and does the explicit slow/fast and composite-company construction beat a generic sequence model that has enough capacity to absorb everything into one black box?

For each model, let

denote the predicted Bernoulli probabilities or, for non-Bernoulli heads, the corresponding declared distributional parameters for the primary heads. More formally, , where for a Bernoulli head and may be a higher-dimensional parameter space for another outcome family. The same label masks and follow-up rules apply to every model.

The proposed system has to beat the following baselines. Let denote the set of decision indices in the training window.

Baseline 0: per-head empirical marginal

For a Bernoulli head with at least one observed training label,

For non-Bernoulli heads, use the corresponding empirical marginal distribution, mean, or survival baseline rather than forcing every task into a prevalence scalar.

Baseline 1: current-proposition model

using only the current proposition, current world state, and shallow relationship context.

Baseline 2: static dyadic tabular model

with no explicit sequence state.

Baseline 3: shallow-history model

where contains hand-built summaries such as touch count, last-response delay, prior meeting count, reply rate, topic counts, and stage history.

Baseline 4: recommender-style two-tower model

where the dyad and proposition are embedded separately and scored through a dot product or shallow fusion, but no explicit recursive state is maintained.

Baseline 5: monolithic sequence model

implemented by a generic recurrent, transformer, or state-space sequence model that sees the same event stream but does not enforce an explicit slow/fast or human/company/relationship decomposition.

This last baseline matters. If a monolithic sequence block with enough capacity eats my lunch, then the decomposition was just a story I told myself after the fact. If the explicit construction still wins or ties while transferring better, calibrating better, or requiring less data, then it has earned the right to stay.

4.4 The Proposed Latent-State Model

The proposed model constructs the state in stages.

First, estimate the slow person-side vectors with a shared human encoder:

Role, seller/buyer position, and local context remain explicit inputs; they are not reasons to build two unrelated psychologies.

Second, maintain fast person-side states with a shared update family. For chronological record at time ,

where is the record-applicability vector for actor : it contains a binary applicability flag and records which actor-specific fields are available for the update. It is neither a memory vector nor a label-availability mask. When the applicability flag is zero, is required to leave unchanged. A record may be a decision or a later observation; the update uses only what has become available by that point. At decision , set .

The seller state matters because the proposition actually delivered is partly a product of the seller, and the same nominal message can be presented differently by different people.

Define each person-side substate as

The shared world state stays outside these person-side tuples because it appears once in the dyadic state.

Third, construct company states:

Here is the usable member subset. The first implementation may observe only a small subset of relevant people. Missing membership and authority information must be represented as missingness and uncertainty, not silently treated as zero. Historical environmental exposure is carried through company inputs and memory; the current shared world state remains explicit in the dyadic state.

Fourth, estimate relationship state:

Here is the batch/history realization of the relationship estimator. It may be implemented by initializing and applying the recurrent update below to the same predecision relationship records. The batch and recurrent forms must therefore agree on record order, applicability, and the information available before the candidate proposition.

Then construct

Person context and company state can inform one another. If the focal salesperson or executive is included inside the corresponding company aggregator and also appears separately in , the representation is redundant; exclude the focal person from the company-context summary or estimate the coupled states jointly so the duplicate path is not treated as independent evidence. An implementation may use a fixed number of message-passing updates, provided every input remains pre-proposition and no outcome information leaks backward.

Given candidate proposition , encode and contextualize it:

form the interaction

and predict

The shorthand

will be used whenever the internal factorization is not the point.

Decode immediate traces from the next state:

Decode primary and auxiliary decision-associated outcomes through a rollout or a coherent joint direct head:

The individual kernels , , and , , are trained as head-wise marginals or conditionals. If the implementation trains only separate marginals, no cross-head independence claim follows.

The multi-head structure is there so the latent state does not remain a completely black box. If the state is real in the operational sense, it should carry reusable signal that helps decode more than one downstream observable.

The architecture of the encoders, aggregators, and transition kernel is not fixed by the theory. The first implementation may use gradient-boosted trees for static components, a GRU or state-space block for history, graph or set aggregation for company structure, and an attention or bilinear interaction for propositions. The theory requires explicit state and testable decomposition. It does not require blind loyalty to one named architecture.

4.5 Training Objective, Update Loop, and Intervention

For multiple primary outcomes and probes, minimize

with

All signs are positive because every term is a loss or penalty being minimized. Masks, censoring weights, or survival-likelihood terms are applied inside the corresponding head loss.

The model updates on different timescales.

Fast person states update after each relevant record according to

Relationship state updates on the same record clock. Let indicate whether record pertains to the ordered dyad ; when it is zero, the update map is required to leave the relationship state unchanged:

At decision epoch , use and .

The full filtered state update, after actual response and exogenous-change records have arrived, is

The slow person estimates update only when durable evidence accumulates. If is a refreshed estimate expressed in the same latent coordinate chart as , use

with typically chosen small. This arithmetic update is meaningful only if the encoder is fixed, the refreshed representation has been aligned to the old chart, or the relevant histories have been re-encoded into a common chart after an encoder change. Company states may contain both fast and slow components as well. A funding event or executive departure can move a company state quickly; culture and governance usually move more slowly.

In deployment, the loop is:

  1. construct from pre-proposition information;
  2. generate or retrieve ;
  3. score candidates under the current forecasting model;
  4. choose according to the current policy and exploration rule;
  5. log the exact candidate set, chosen proposition, decision time, and assignment probability or density;
  6. observe immediate response and delayed outcomes as they mature;
  7. update fast and relationship state;
  8. periodically refit parameters and refresh slow estimates.

Predictive ranking

A score extending beyond the immediate next response requires a declared predictive evaluation regime . It contains the continuation policy, future candidate-set process, exogenous-path law, and outcome/censoring convention used by the score. Assume the declared utility is measurable and integrable under every fitted model and regime being compared. Define

The tilded outcome and probe bundles are either measurable functionals of the same rollout or draws from the coherent joint head calibrated to . If only marginals are fitted, the score uses marginal expectations or a declared coupling. This supports simulation and ranking. Useful, yes. Causal, not yet.

Because a learned simulator can be exploited by the optimizer, practical ranking should be conservative. One option is

where measures predictive uncertainty and penalizes candidates far from the support of observed or randomized actions. The exact penalty is an implementation choice; the need to defend against model exploitation is not.

Off-policy evaluation

Let

be the target policy. The logged behavior-policy probability for the realized action is the scalar

Let be the predeclared set of eligible decisions whose scalar utility is mature and usable under the chosen censoring rule, and let . If the utility extends beyond the immediate response, this one-step estimand changes the current proposition while holding the declared or logged continuation regime fixed. A basic inverse-propensity estimate is

The denominator is the logged probability or density assigned by the historical policy at the historical decision. It is not recomputed from . The target-policy numerator must also be a function only of information available before the evaluated decision; a representation containing post-decision information would invalidate the ratio. For continuous actions, numerator and denominator must be densities with respect to the same dominating measure. Report the weight distribution and effective sample size. Clipping and self-normalization can reduce variance but introduce bias or change the finite-sample estimand, so report them as sensitivity analyses rather than invisible repairs. Doubly robust estimators are often preferable when the outcome and propensity nuisance models are credible. Informative censoring requires an additional censoring model or weighting term rather than complete-case deletion. A learned target policy should be frozen and evaluated on held-out data, or estimated with sample splitting or cross-fitting. Sequence policies require sequential off-policy estimators with products or per-decision products of importance ratios, or sequential doubly robust alternatives; the one-step expression is not applied blindly to an entire message path.

This formula only makes sense under an action representation with support. If every proposition is a unique free-form message, exact historical overlap is nearly nonexistent. The first intervention program should therefore randomize controlled proposition dimensions or families—framing, proof type, offer, call-to-action, channel, timing—rather than pretending every novel paragraph has a reliable counterfactual twin in the logs.

Off-policy or causal claims additionally require:

  • consistency: the recorded treatment corresponds to the proposition definition used in the model;
  • overlap: the behavior policy assigns nonzero probability to actions the target policy may choose;
  • identification: randomization or a defensible sequential no-unmeasured-confounding argument based on the recorded decision information;
  • correct event ordering: no post-treatment variable enters the state used to select the treatment;
  • handling of delayed outcomes: later messages and events can mediate long-horizon labels;
  • interference assumptions: several people inside may influence one another, so executive is not always an isolated unit;
  • stable treatment definition: supposedly identical proposition families must be similar enough that the causal comparison is coherent;
  • censoring discipline: outcome observation and attrition must be handled rather than assumed independent by convenience.

Online policy improvement

If a controlled fraction of traffic can be randomized, proposition selection becomes a real policy-learning problem rather than retrospective ranking. At that point, the model can choose among admissible messages, offers, sequences, or timing policies, and value can be evaluated through live lift, regret, conversion, or long-run utility.

Until then, leave the causal swagger out of it.

4.6 Temporal Split, Evaluation, and Drift

The benchmark must be temporal. Random row splits can leak future information and can substantially overstate generalization in repeated-interaction data.

Use nonoverlapping rolling windows. Model, feature, threshold, and calibration choices are made with training and validation data only; the final test window remains untouched until the analysis is frozen:

Time alone is not enough. Report separate evaluation regimes:

  1. future interactions with people and accounts seen during training;
  2. new executives inside known companies;
  3. known executives inside new selling contexts;
  4. entirely unseen companies and people;
  5. transfer to a new proposition family or outcome horizon.

This separates memorization from actor-state generalization.

Long-horizon outcomes require censoring discipline. A deal observed for only 40 days cannot be labeled “did not close within 90 days.” Use complete follow-up windows or time-to-event and competing-risk methods where appropriate.

Rows are dependent within people, sellers, companies, campaigns, and time periods. Confidence intervals should use clustered or hierarchical resampling rather than pretending every event is independent.

For binary or probabilistic primary tasks, report

Use survival metrics for censored time-to-event outcomes, ranking metrics for candidate-ordering tasks, and probe-appropriate metrics for auxiliary outputs.

Calibration matters because proposition ranking depends on differences between predicted values. A model that ranks adequately but is badly miscalibrated can still produce destructive utility estimates. ECE depends on the binning rule and should not be reported alone; include reliability diagrams and at least one complementary calibration summary or sensitivity analysis over binning choices.

Ablations

The benchmark should force each major claim to either pay rent or be removed.

  1. Remove fast executive state . If short-horizon performance barely moves, the fast state is ornamental.
  2. Remove slow executive state . If cold-start and cross-context performance barely move, the durable embedding is ornamental.
  3. Remove salesperson state. If nothing changes, seller-side psychology is not needed for this task once the proposition is fixed.
  4. Remove relationship state . If nothing changes, the interaction history is already captured elsewhere or was never useful.
  5. Replace company aggregation with flat company features. If performance improves, the composite actor construction is premature.
  6. Replace salience-weighted categorical pooling with uniform averaging. If nothing changes, the weighting story is decorative.
  7. Collapse source channels and role regimes before pooling. If performance improves, the source-aware separation is unnecessary; if it hurts, the separation is buying signal.
  8. Shuffle recent within-dyad history while preserving static profiles. If performance does not fall, the model was not using sequence in the way claimed.
  9. Remove probe heads. If they contribute no stable transfer or regularization value, remove them.
  10. Replace the explicit architecture with the monolithic sequence baseline. If the generic model dominates, the decomposition is not buying enough.
  11. Replace the shared GIDS interaction representation with late fusion of unrelated embeddings. This directly tests whether mapping distinctions into a common interaction arena adds value.
  12. Evaluate across tasks. If the learned person-state cannot support more than one narrow target, do not call it a stable actor ontology.

For any candidate feature family , define its held-out contribution under a lower-is-better validation risk by

A positive value means the feature family reduced held-out risk. This is an empirical contribution, not proof that the learned coordinate is uniquely real or causally fundamental.

Drift

Use a proper scoring rule whose lower value is better, such as average negative log likelihood. Let

be recent risk on a rolling window and let be reference risk. Define degradation by

Trigger investigation when

or when calibration, support, or data-quality diagnostics cross their own predeclared thresholds.

Responses may include:

  • refit parameters;
  • refresh slow state estimates;
  • revise company aggregation;
  • expand or prune feature families;
  • reopen the task projection;
  • retrain calibration;
  • reduce policy aggressiveness until overlap and support return.

A model like this is expected to become wrong. The point is to catch it when it does.

4.7 What Counts as Success

Success is not that the model sounds deep. Success is narrower.

For a declared Bernoulli primary head, the first forecasting system succeeds if, on temporally held-out data,

and

with chosen before inspecting the final test window and uncertainty intervals that exclude trivial gains. For multiclass, ordinal, continuous, survival, or joint heads, replace these two Bernoulli scores with the predeclared proper scoring rule or task-specific loss appropriate to that outcome; do not average incomparable head metrics without a declared normalization and weighting rule.

It succeeds more strongly if:

  • gains survive time drift and entity holdouts;
  • the slow/fast ablations behave as predicted;
  • relationship state adds unique signal;
  • the company aggregation improves over flat metadata;
  • task transfer shows that the actor-state supports several outcomes;
  • the shared interaction representation beats simpler late fusion;
  • uncertainty and calibration remain usable for decision-making.

If intervention data exists, proposition selection succeeds when a policy based on the model produces higher off-policy value under defensible assumptions or higher live experimental utility than a baseline policy.

If intervention data does not exist, proposition search results must be described as simulated or observational rankings, not causal wins.

The framework fails its first empirical test if simpler models match or exceed it, if the monolithic sequence model dominates without a meaningful interpretability or data-efficiency tradeoff, if the learned state does not transfer, if gains disappear on future windows, or if proposition rankings fail under randomization.

In that case, either the decomposition is wrong, the data does not contain the signal I thought it did, the actor construction is wrong, or the task never needed this much machinery in the first place.

End of Part 4

Part 4 is where the framework becomes verifiable and we can use data.

At this stage, the job is straightforward: define what state means in a form a dataset can carry, define the event clock, define what must be predicted, define which weaker and stronger models must be beaten, define what the probes are supposed to establish, and define what intervention evidence is required before proposition optimization can be called causal.

The operational arc is

And if we later choose messages from the model,

with the giant asterisk that ranking is not causality unless the data collection regime supports that claim.

This is the point where the theory becomes falsifiable. The question is no longer whether reality can be expressed through a common formal arena. The question is whether this construction yields better forecasts of observable human transition than models that ignore explicit actor-state, relationship, and composite context, and whether its proposition-search layer survives the much nastier standard of intervention.


In Memory of Einar Kringlen.

Einar Kringlen seated in his office in front of bookshelves.

It has been an honor tackling this multi-generational problem with you. To you I owe much.