When your measurement system relies on human judgment, pass/fail decisions, visual inspections, categorical ratings, you need a way to prove those judgments are consistent and accurate. That’s exactly what Minitab Attribute Agreement Analysis does. It quantifies how well your appraisers agree with each other, with themselves over repeated trials, and with a known standard. Without this analysis, you’re building your entire improvement project on data you can’t trust.
At Lean Six Sigma Experts, we train and consult teams through every phase of DMAIC, and the Measure phase is where we see projects succeed or fail. Measurement System Analysis (MSA) is non-negotiable before you start drawing conclusions from attribute data. If your inspectors aren’t consistent, your defect rates are fiction. A proper Attribute Agreement Analysis catches that problem early, before bad data derails your project.
This guide walks you through the complete process, from setting up your data in Minitab to running the analysis, reading the Kappa statistics, and interpreting each piece of the output. Whether you’re a Green Belt running your first MSA or a Black Belt refreshing your approach, you’ll have everything you need to execute this analysis correctly and act on the results.
What attribute agreement analysis tells you
Attribute Agreement Analysis measures three distinct types of agreement: within-appraiser agreement (does each inspector rate the same sample consistently across multiple trials?), between-appraiser agreement (do all inspectors agree with each other on the same samples?), and appraiser vs. standard agreement (does each inspector match the known correct answer?). Running a Minitab Attribute Agreement Analysis gives you a separate score for each, so you can isolate exactly where your measurement system is failing before it corrupts your data.
If your appraisers cannot agree with themselves, no amount of additional data will fix the problem. You must address the measurement system first.
The three agreement metrics that matter
Each metric targets a specific failure point in your inspection process. Within-appraiser consistency tells you whether an inspector’s judgment is stable over time. Between-appraiser consistency tells you whether your team applies the same decision criteria to every sample. Appraiser-vs.-standard accuracy tells you whether those decisions are actually correct. You can have inspectors who agree with each other but are all consistently wrong, which is exactly why you need all three metrics, not just one.
| Metric | What It Measures | Why It Matters |
|---|---|---|
| Within Appraiser | Same appraiser, repeated trials on identical samples | Detects instability in individual judgment |
| Between Appraisers | Multiple appraisers rating the same samples | Detects disagreement across your inspection team |
| Appraiser vs. Standard | Appraiser ratings vs. known correct answers | Detects systematic bias or misapplied criteria |
Why Kappa is the number you focus on
Minitab reports Cohen’s Kappa as the primary statistic for each type of agreement. Kappa adjusts for the agreement you would expect by chance alone, which makes it far more meaningful than a raw percentage match. A Kappa of 1.0 represents perfect agreement, while a Kappa of 0 means your appraisers are performing no better than random guessing. Negative values indicate that appraisers are systematically disagreeing in ways that run counter to chance expectations.
The industry standard thresholds give you a clear benchmark for action. A Kappa below 0.70 signals an unacceptable measurement system that you must fix before using this data for any decisions. A Kappa between 0.70 and 0.90 is marginal and warrants investigation into training gaps or ambiguous rating criteria. Only a Kappa above 0.90 confirms that your measurement system is capable. Minitab also provides a p-value alongside each Kappa score, and you want that value below 0.05 to confirm the agreement is statistically significant rather than a product of a small sample size.
Step 1. Plan the study and define ratings
Before you open Minitab, you need to make deliberate decisions about study design. The quality of your Minitab Attribute Agreement Analysis depends entirely on what you put into it. A poorly planned study will produce Kappa values that look acceptable but hide real problems in your measurement system. Take time upfront to lock in your sample selection, number of appraisers, number of trials, and rating definitions.
Choose your samples carefully
Your sample set must represent the full range of conditions your appraisers encounter in real production. Industry practice calls for a minimum of 50 parts when running a standard attribute study, with at least 2 appraisers and 2 repeated trials per appraiser. Critically, you must include borderline samples, parts that sit close to the acceptable or reject boundary, not just clear passes and clear failures. Borderline parts are where judgment inconsistency lives, and skipping them will give you an inflated Kappa that does not reflect real-world performance.
Stack your sample set with borderline parts. They are the most effective diagnostic tool for finding exactly where your criteria break down.
Define your rating categories before you start
Ambiguous rating criteria are the most common root cause of low Kappa scores. Before any appraiser rates a single part, you need written, specific definitions for every category in your rating scale. If you are using a binary pass/fail system, document the exact conditions that define each outcome. If you are using an ordinal scale such as 1 through 5, write a description and include a physical reference sample for each level.
Your written criteria also become the reference standard you enter into Minitab when running the appraiser-vs.-standard comparison. Without a documented standard, that third metric is meaningless. Lock these definitions down before you collect any data, and distribute them to every appraiser in the study.
Step 2. Set up your data in Minitab
Minitab requires your data in a specific column structure before you can run a Minitab Attribute Agreement Analysis. If your worksheet is arranged incorrectly, the dialog box will reject your inputs or return errors that are difficult to trace back to the source. Getting this structure right takes less than ten minutes once you understand what Minitab expects from your data.
Structure your worksheet columns
Your worksheet needs four columns at minimum: one for the sample identifier, one for the appraiser name, one for the trial number, and one for the rating each appraiser assigned. If you collected a known reference standard, add a fifth column for the correct answer. Each row represents a single rating event, meaning one appraiser rating one sample during one specific trial.
Here is the exact column layout Minitab expects:
| Column | Label | Example Values |
|---|---|---|
| C1 | Sample | 1, 2, 3 … 50 |
| C2 | Appraiser | Smith, Jones, Lee |
| C3 | Trial | 1, 2 |
| C4 | Rating | Pass, Fail |
| C5 | Standard | Pass, Fail |
Enter your data correctly
With two appraisers, 50 samples, and 2 trials, your worksheet will contain 200 rows total (2 appraisers x 50 samples x 2 trials). Every combination of appraiser, sample, and trial must appear exactly once. Minitab will flag missing rows, but it will not fill them in for you, so review your worksheet before moving forward.
Verify that each sample number appears the same number of times for every appraiser. Unbalanced data will skew your Kappa values.
Pay close attention to spelling and capitalization in your Rating column. If one row reads "pass" and another reads "Pass," Minitab treats them as two separate categories, which corrupts your agreement scores entirely. Standardize every entry before you proceed to the analysis step.
Step 3. Run attribute agreement analysis in Minitab
With your worksheet built and verified, you are ready to run the Minitab Attribute Agreement Analysis. The entire process takes under two minutes once your data is in order. Minitab handles all the Kappa calculations automatically, so your only job at this stage is to map your columns to the correct fields in the dialog box.
Navigate to the correct dialog box
Open your Minitab project file with your completed worksheet active. Follow this exact menu path to reach the analysis dialog:
- Click Stat in the top menu bar
- Select Quality Tools
- Click Attribute Agreement Analysis
The dialog box will open with several input fields and options. Do not click OK until you have filled in every required field. An incomplete setup produces output that Minitab labels without enough context to interpret correctly.
Double-check your menu path before entering any data. Minitab places Attribute Agreement Analysis under Quality Tools, not under the MSA submenu.
Configure the dialog box inputs
You will see fields for your attribute column, sample column, appraiser column, and optionally a known standards column. Map each field to the corresponding column in your worksheet. If you are following the layout from Step 2, your configuration should look like this:
| Dialog Field | Your Column |
|---|---|
| Attribute (response) | C4 (Rating) |
| Samples | C1 (Sample) |
| Appraisers | C2 (Appraiser) |
| Known Standard/Attribute | C5 (Standard) |
After mapping your columns, click Options to confirm the number of categories and verify that Minitab displays the correct rating levels. Leave the confidence level at 95% unless your project protocol specifies otherwise. Once everything is configured, click OK and Minitab will generate the full output in the Session window and Graph window simultaneously.
Step 4. Interpret the output and take action
Minitab generates two outputs simultaneously: a Session Window with all your Kappa statistics and confidence intervals, and a Graph Window with visual agreement plots. Start with the Session Window. That is where every number you need to make a decision lives. The graphs help you communicate results to stakeholders, but the numerical output drives your corrective action.
Read the Kappa values in order
Work through the Session Window results in a consistent sequence. Check within-appraiser agreement first, then between-appraiser agreement, and finally appraiser-vs.-standard agreement. If within-appraiser Kappa is low, that appraiser is contradicting themselves across trials, which tells you the problem is individual instability rather than a team-wide training gap. If between-appraiser Kappa is low but within-appraiser scores are acceptable, your inspectors are each consistent internally but applying different criteria, and that points to poorly written standards.
The sequence in which you read the Kappa values tells you exactly where to focus your corrective action, so do not skip to the final score without working through each metric first.
Minitab also reports confidence intervals for each Kappa value. A wide confidence interval signals that your sample size is too small to trust the point estimate. If your 95% confidence interval crosses the 0.70 threshold, you cannot confidently classify the system as acceptable or unacceptable, and you need to collect more data before drawing any conclusions.
Match your result to a specific corrective action
Once you have identified which metric is failing your Minitab Attribute Agreement Analysis, use the table below to assign the correct fix before you allow production data collection to resume.
| Kappa Result | Root Cause | Required Action |
|---|---|---|
| Within-appraiser below 0.70 | Individual inconsistency | Retrain the specific appraiser and retest |
| Between-appraiser below 0.70 | Inconsistent criteria application | Rewrite and standardize rating definitions |
| Appraiser-vs.-standard below 0.70 | Systematic misclassification | Review reference standards and conduct calibration |
| All Kappa above 0.90 | No issues detected | Proceed with data collection and document results |
Next steps for your inspection process
You now have everything you need to run a complete Minitab Attribute Agreement Analysis and act on the results. The process is straightforward: plan your study with borderline samples, structure your worksheet correctly, run the analysis, and match your Kappa results to specific corrective actions. Do not treat a failed measurement system as a minor inconvenience. Fix it before you collect a single data point for your improvement project, or every conclusion you draw will rest on unreliable data.
Once your measurement system passes with Kappa values above 0.90, document your rating criteria, save your Minitab project file, and schedule revalidation at regular intervals, especially when you bring in new appraisers or change your inspection criteria. Attribute Agreement Analysis is not a one-time exercise; it is an ongoing quality control practice. If you want expert guidance on building a measurement system that holds up under scrutiny, contact the Lean Six Sigma Experts team to get started.