Documentation Index
Fetch the complete documentation index at: https://docs.allyy.io/llms.txt
Use this file to discover all available pages before exploring further.
Classification models (DM Propensity as an example)
Dashboards in the Business Case Section
Each dashboard shows a different aspect of model performance on the holdout data. Together, they help users understand both operational performance and financial outcomes.
The Campaign Performance Summary table provides a consolidated view of how the selected campaigns performed on the holdout dataset under the current model. This allows users to understand the financial and operational outcomes the model would have produced on unseen past data.
The table compares two scenarios:
As-is - the current targeting strategy based on the selected threshold
Optimized - the model-recommended targeting strategy that maximizes net revenue or ROI
Because this example shows identical values in both columns, it indicates that the current threshold already matches the model’s optimal recommendation.
What Each Metric Means
Total letters - The number of individuals who would have been contacted.
Total positives - The number of donors captured (true positives).
Hit rate - Donors captured divided by letters sent.
Total income - Revenue generated from donors in the holdout period.
Total cost - Cost of sending letters, based on the user-defined cost per letter.
Activity result - Net revenue (income minus cost).
Total ROI - Return on investment, calculated as net revenue divided by cost.
Score Distribution by Actual Outcome
This chart shows how the model assigns scores to donors and non-donors in the holdout dataset. Because the holdout consists of past data the model has never seen, this view helps assess how well the model separates the two groups in a realistic, unbiased setting.
The histogram displays two overlapping distributions:
Actual donors (purple)
Actual non-donors (green)
The x-axis represents the model score, and the y-axis shows how frequently each score occurs within each group.
A vertical dashed line marks the current decision threshold. Everyone above this score would be contacted; everyone below would not.
Cumulative Gains
This chart shows how effectively each campaign captures donors as the percentage of targeted individuals increases.
It helps users understand:
how quickly each campaign captures donors
which campaigns perform best
whether the model ranks donors effectively
A steeper curve indicates better targeting efficiency.
Response rate by score group
This chart shows how often people in each score group responded (became donors) in the holdout dataset. Because the holdout is past data the model has never seen, this view reflects the model’s true ability to rank individuals by likelihood to respond.
The x-axis represents score groups from 1 to 10, where 10 is the highest-scoring, most likely-to-respond group. The y-axis shows the actual response rate observed in the holdout period.
Each bar represents the hit rate for that score group.
What this chart tells the user
Higher score groups respond more often The clear upward trend means the model is correctly ranking individuals: those with higher scores are more likely to donate.
Score group separation The difference between low and high groups shows how strong the model’s predictive power is.
Low groups (1–3) have low response rates.
High groups (8–10) have much higher response rates.
Targeting quality This chart helps determine where the cutoff should be.
Groups with very low response rates may not be worth contacting.
Groups with high response rates are the most profitable to target.
Model precision A smooth, increasing pattern indicates a well-calibrated model. If the bars were flat or random, the model would not be ranking donors effectively.
Profit curve
The profit curve shows how net revenue changes as a larger share of the audience is targeted, based on holdout data the model has never seen before. It helps users understand the financial impact of different targeting levels and whether expanding or reducing the target group would have improved past campaign performance.
The x-axis represents the percentage of the population targeted, and the y-axis shows the resulting net revenue. The blue line traces how revenue grows as more people are included. A vertical dashed line marks the current targeting level.
What this chart tells the user
Revenue growth pattern The curve shows how net revenue increases as targeting expands. A steep rise early in the curve indicates that the highest-scoring individuals generate the most value.
Diminishing returns As the curve flattens, each additional percentage of the population contributes less revenue. This is where low-scoring groups begin to dilute profitability.
Optimal targeting point The highest point on the curve represents the most profitable targeting percentage. Targeting beyond this point increases cost more than revenue.
Current strategy vs. optimal The red dashed line shows the current targeting level. If it is left of the peak, the organization is under-targeting; if it is right of the peak, it is over-targeting.
Amount split by score group
This chart shows how the total income from the holdout dataset is distributed across the model’s score groups. Each score group represents a band of predicted likelihood, with 1 being the lowest-scoring group and 10 being the highest. Because the data comes from the holdout set, the distribution reflects how value was actually generated in past campaigns the model never saw during training.
The bar is divided into colored segments, each representing the share of total revenue contributed by that score group.
What this chart tells the user
Higher score groups generate more revenue The rightmost groups (8–10) contribute the largest share of total income. This confirms that the model is correctly ranking individuals: those predicted to be more likely to respond also tend to donate more.
Lower score groups contribute very little Groups 1–3 contribute only a small fraction of total revenue. These groups are typically unprofitable to target and may be excluded in optimized scenarios.
Score-to-value alignment The smooth increase from low to high score groups indicates that the model’s scoring aligns well with actual donor value. This is a key indicator of model quality.
Targeting strategy implications Because most revenue comes from the top score groups, focusing mailings on these groups can significantly improve ROI and reduce cost.
ROI by Score group
This chart shows how return on investment (ROI) varies across the model’s score groups, based on holdout data the model has never seen before. Each score group represents a band of predicted likelihood, with 10 being the highest-scoring group. The bars show the actual ROI generated historically by each group.
The y-axis displays ROI as a percentage, and the x-axis lists the score groups from 1 to 10.
What this chart shows
Higher score groups deliver much higher ROI ROI rises steadily from group 1 to group 10, with the top group generating the strongest financial return. This indicates that the model is correctly ranking individuals not only by likelihood to respond but also by profitability.
Lower score groups are often unprofitable The lowest groups (1–3) typically show very low ROI. These groups tend to cost more to contact than they return in revenue.
Strong score-to-ROI alignment The smooth upward trend confirms that the model’s scoring correlates well with actual financial outcomes. This is a key indicator of a high-quality predictive model.
Clear targeting implications Because ROI increases sharply in the higher groups, focusing mailings on groups 7–10 can significantly improve campaign profitability.
This view summarizes how each selected campaign performed on the holdout dataset, showing both the overall outcome and a detailed per-campaign breakdown. Because the data comes from past donors and non-donors the model never saw during training, the table reflects how the model would have behaved in real historical campaigns.
Outcome at current selection
TP (donors contacted) — donors the model correctly identified and would have mailed.
FP (non-donors contacted) — people contacted who did not donate.
FN (donors missed) — donors the model failed to identify.
TN (non-donors not contacted) — non-donors correctly excluded.
Per-campaign summary
letters — number of people contacted.
positives — donors captured in that campaign.
income — revenue generated from those donors.
hit_rate — donors divided by letters sent.
cost — mailing cost based on the user-defined cost per letter.
result — net revenue (income minus cost).
roi — return on investment for that campaign.
Cumulative gains by campaign
This chart shows how quickly each campaign captures donors as you target more of the audience.
Share of donors captured at current cut
This view shows what percentage of donors each campaign captures at the current targeting threshold.
Hit rate by campaign
This chart compares the efficiency of each campaign.
ROI by campaign
This chart shows financial performance per campaign.
