The series is in three 3-hour segments.  Participants will receive a digital workbook for each session plus a Quick Start Quality Control Guide with job-aids.  The cost for the whole series for up to 6 participants is $444, a $60 savings.  

Segment 1 - Basics

Testing Basics - You will be able to explain to others:

• What a test is
• What the different types of tests are
• What makes a test valid, fair, reliable, and equitable
• How the use of AL can save task time without loss of integrity

What to Test and How

You will be able to:

• Explain the process for deciding what to test and how to test it
• Design a job study
• Create operational definitions
• Create and use a test blueprint

Segment 2:  Response-Supplied Items

How to Construct Multiple-Choice, matching, scenario items that Measure Higher Level Skills

You will be able to:

• Write questions (also called stems or items) that comply with best practices
• Write responses (correct and wrong answers) that are parallel in construction and lack specific determiners (cues)
• Query AI to generate response supplied items, distractors, and rationale that comply with best practices significantly reducing experts task time 
• Use Quality Assurance Guides to assure items comply with best practices

You will be able to:

• Use Excel to analyze test results to determine the P-Value of the test and the items that make up the test
• Use Excel to perform an analysis that measures items' ability to discriminate high performers from low performers.
• Identify ineffective test questions
• Identify ineffective distractors

Segment 3:  Performance Tests

How to Construct Performance Items

You will be able to:

• Choose the best type of performance-based test given your situation
• Decide what type of scoring tool to use
• Design your performance-based test so it complies with the guidelines
• Compare online tools for recording responses

How to Build Effective Checklists and Rubrics for Scoring Performance Items

You will be able to:

• Create performance checklists for judging
• • People’s ability to execute a procedure fairly and consistently
    
  • If a work product possesses the desired attributes
    
• Query AI to buid effective scenarios, portfolio instructions, and rubrics
• Qualify Evaluators (Judges)
• Calibrate the scoring of the performance checklists

• Create rubrics to fairly and consistently judge people’s ability to perform a cognitive or physical task, procedure, or process
• Calibrate ratings across evaluators

Certificate-Series:

How to Get Valid Data

The series is in three 3-hour segments.  Participants will receive a digital workbook containing the presentation, job aids, and reference materials.  You will be able to design and conduct the data gathering method and analyze the results.  The cost for the whole series for up to 6 participants is $444, a $60 savings.   

Segment 1: Interviewing

How to Conduct Observational & Cognitive Interviews and Interpret the Results

This is a key skill for anyone who has been charged with finding out how experts do their work and why they make the decisions they do.   You will get a set of questions you can adapt and a protocol for setting up and conducting the process.  You will find out how to avoid common mistakes.  You will be able to extract information from an interview to create Performance Statements, Course Content, and Assessments.

How to Conduct Critical Incident Interviews and Interpret the Results

Ever wonder why some people are better at work than other people?  This data gathering method is specifically designed to surface the attributes that discriminate those who do well from those who do average.  You get guidelines and questions for conducting the interview that you can refine for your own situation, as well as guidelines on how to interpret the results.

Segment 2:  Group Interviews and Power Structures

How to Conduct a Nominal Group and Interpret the Results

The NGT is a proven technique for managing groups, especially when members differ in communication skills, personality, and levels of authority.  The NGT can be used to identify skills and knowledge as well as what distinguishes people who are effective in their work.  

How to Conduct a Sociogram and Interpret the Results

This technique is used to graphically display a group’s communication patterns, specifically who members choose to ask for guidance, information, and recognition.  It is used with intact work groups to identify leaders and social cliques, measure group cohesiveness, and find out who is really in charge.

Segment 3: Interactions and Surveys

How to Conduct a Flanders Interaction Analysis and Interpret the Results

This technique is used to capture communication patterns between members of a work group.  It tracks who initiates, responds, passes judgment, acknowledges, and the like.  The results are used to help groups communicate more effectively.

Surveys are the most used method for getting information.  Their results support marketing decisions and job task analyses.  Surveys can validate data from other sources.  This session covers the rules for constructing surveys so you can interpret the results.

Certificate-Series:

Practical Psychometrics & Analytics

The series is in three 3-hour segments.    Participants will receive a digital workbook for each session containing copies of the presentation, job aids, and resources.   The cost for the whole series for up to 6 people is $444, a $60 savings.  

Segment 1: Psychometric Basics 

Psychometric Basics

This session explains: 1) measurement, evaluation, psychometrics, analytics, and pragmatics and what each perspective has to offer; 2) the difference between quantitative and qualitative data, 3) what makes data hard compared to soft and why it matters, 4) the typical data sources, and 5) common data gathering and analysis methods.

How to Control the Four Sources of Bias (Sampling, Construct, Content, and Administrative)

Bias is a part of every study; however, it can be managed, even mitigated with the proper steps.  Otherwise, bias can lead to bad decisions.  The focus of this session is on how to manage bias.

Segment 2: Standard Setting & Analysis

Guidelines for Standard Setting and Establishing a Pass Scores

One of the more politically charged questions is, what is good enough?  Standard setting is about answering this question.  There are numerous methods, and each has its tradeoffs.  This session explains and compares seven of the more frequently used methods.

How to Do Item Analyses and Determine the Equivalency of Test Forms

This session is about how to determine if multiple tests forms are equivalent and confirm they measure the same things at the same level of difficulty.  The session also describes how to conduct item analyses to identify underperforming items.

Segment 3: Quantitative and Qualitative Data

Methods for Analyzing Quantitative Data using Excel and the Questions they Answer (Standard Deviation, ANOVA, Correlations, Two-Tailed t-tests, Chi Square Goodness of Fit, T and Z Scores)

Psychometrics and data analytics use different statistical formula to answer questions and guide decisions.  Test administrators also rely on the use of quantitative analyses to evaluate the validity and reliability of their tests.  This session explains the more frequently used statistical methods and the questions they are designed to answer.

This session is about how to code and weight qualitative data so you can apply statistical methods to analyze the data.  The methods apply to survey, interview, observation, and focus group data whether the data are recorded using rubrics, checklists, or lists.