The Feedback Tool for Responder Development \u2013 Senior Staff (FIRE-CU)<\/strong> (German: Feedback zur Rettungskr\u00e4fteentwicklung \u2013 F\u00fchrungsstab<\/em>) is a standardized, psychometrically validated multidimensional evaluation instrument designed specifically for training programs directed at rescue forces and fire service command staff members (F\u00fchrungsstab<\/em>; “CU” designates “Command Unit”). Originating from a collaborative research initiative between the Department of Organizational and Business Psychology at the University of M\u00fcnster<\/a> and the State Fire Service Institute of North Rhine-Westphalia (Institut der Feuerwehr Nordrhein-Westfalen<\/a>; IdF NRW), the FIRE-CU represents an adaptation and structural expansion of the foundational FIRE questionnaire (Schulte & Thielsch, 2019<\/a>). The instrument systematically bridges educational process quality and immediate training outcomes across high-stakes emergency management domains.<\/p>\n The core instrument consists of 22 items distributed across six psychometric subscales, conceptually stratified into two higher-order evaluation tiers: the Learning Process Level<\/em> (comprising Instructor Behavior [Dozentenverhalten<\/em>], Requirement Level [Anforderungsniveau<\/em>], Structure [Struktur<\/em>], and Group Dynamics [Gruppe<\/em>]) and the Learning Outcome Level<\/em> (comprising Competence Acquisition [Kompetenzerwerb<\/em>] and Training Transfer [Transfer<\/em>]). Four supplemental, non-scale items capture overarching global assessments (Globalurteil<\/em>) and qualitative participant remarks (Feedback<\/em>). The core items are assessed using an unforced seven-point Likert scale (1 = stimme gar nicht zu<\/em> \/ strongly disagree to 7 = stimme vollkommen zu<\/em> \/ strongly agree) supplemented with an explicit non-substantive filter category (nicht sinnvoll beantwortbar<\/em> \/ not meaningfully answerable). Structural equation modeling and confirmatory factor analysis (CFA) validate the multi-tier factor structure, confirming acceptable to excellent model fit (Process level: CFI = .97, TLI = .96, RMSEA = .05, SRMR = .04; Outcome level: CFI = .97, TLI = .95, RMSEA = .06, SRMR = .04). Reliability analyses demonstrate strong internal consistency across dimensions, with Cronbach\u2019s alpha spanning .75 to .90 and McDonald\u2019s omega ranging from .76 to .91. The instrument provides emergency service training academies, crisis management organizations, and military institutions with an evidence-based diagnostic protocol for curriculum development, quality assurance, and instructional benchmarking<\/span>.<\/p>\n FIRE-CU, emergency responder training, command staff evaluation, crisis management, High Reliability Organizations (HRO), incident command systems, training evaluation, confirmatory factor analysis, Shared Mental Models, Kirkpatrick model, fire service leadership, psychometrics<\/p>\n The FIRE-CU was conceptualized, developed, and validated through an institutional research partnership between academic organizational psychologists and operational command staff training directors:<\/p>\n The operational management of large-scale disasters, major industrial incidents, widespread natural catastrophes, and multi-agency mass casualty events exceeds the coordination capacity of field-level tactical commanders. Within modern incident management frameworks\u2014such as the German Fire Service Regulation 100 (Feuerwehr-Dienstvorschrift 100<\/a>; FwDV 100) or international equivalents like the Incident Command System<\/a> (ICS)\u2014strategic decision-making is delegated to an operatively deployed Command Unit or Command Staff (F\u00fchrungsstab<\/em>). In Germany, members of these strategic groups are seasoned executive officers who have advanced beyond company-level and battalion-level commanding qualifications (specifically, holding certification as a Verbandsf\u00fchrer<\/em>, the tactical leader of multiple operational platoons). Because full-scale catastrophic emergencies requiring operational-tactical command staff mobilization occur with low base-rate frequency\u2014historically estimated at once every 25 years per localized jurisdiction, punctuated by severe crises such as pandemic response or regional flooding\u2014staff competence depends almost entirely on high-fidelity simulation and specialized command post exercises (Stabsrahmen\u00fcbungen<\/em>).<\/p>\n Despite the critical nature of command staff preparedness, evaluation practices in emergency responder curricula historically relied on unstandardized, intuitive feedback, or general-purpose academic course ratings that fail to capture the socio-cognitive demands of crisis command. The purpose of the FIRE-CU is to provide an empirically grounded, methodologically robust, and actionable evaluation instrument tailored to the structural, instructional, and psychological realities of strategic command staff development. The instrument fulfills three central functions:<\/p>\n The FIRE-CU measures the perceived quality of crisis management training through an integrated, multidimensional architecture. Drawing from training psychology and human factors engineering, the instrument segments educational appraisal into six latent sub-constructs operating across two sequential stages of the training lifecycle: process dynamics and outcome attainment.<\/p>\n The process tier captures the environmental, didactic, and interpersonal dynamics that occur during the delivery of instruction. Rather than viewing the learning environment as a monolithic entity, the FIRE-CU isolates four distinct psychological and organizational elements:<\/p>\n The outcome tier captures the subjective cognitive and operational transformations experienced by participants upon conclusion of the training:<\/p>\n The architectural blueprint of the FIRE-CU is anchored in three theoretical paradigms: Donald Kirkpatrick\u2019s hierarchical evaluation taxonomy, Blanchard and Thacker\u2019s dual-level training framework, and the theory of Shared Mental Models within High-Reliability Teams.<\/p>\n The standard paradigm for educational evaluation in organizational settings is the four-level model developed by Donald Kirkpatrick<\/a> (1979; Kirkpatrick & Kirkpatrick, 2006). This model posits that training efficacy must be systematically analyzed across a hierarchical continuum:<\/p>\n Meta-analytic research by Alliger, Tannenbaum, Bennett, Traver, and Shotland (1997) refined Level 1 by differentiating between affective reactions<\/em> (whether trainees enjoyed the course) and utility reactions<\/em> (whether trainees perceived the training as instrumentally useful for their job performance). Alliger and colleagues demonstrated that utility reactions demonstrate significantly stronger predictive relationships with downstream learning (Level 2) and behavioral transfer (Level 3) than affective reactions. The FIRE-CU heavily prioritizes utility reactions within its process scales and captures immediate subjective capability growth corresponding to Kirkpatrick\u2019s Level 2 and anticipatory Level 3 readiness.<\/p>\n To overcome the limitations of classic end-of-course satisfaction metrics, Blanchard and Thacker (2013) conceptualized training assessment through a functional bifurcation between Process Data<\/em> and Outcome Data<\/em>. Process evaluations monitor instructional delivery fidelity, trainer pedagogical conduct, structural consistency, and learning friction, diagnosing *why* a course succeeded or failed. In contrast, outcome evaluations verify whether pedagogical objectives were successfully achieved. Blanchard and Thacker argued that relying solely on outcome parameters conceals instructional mechanisms, while analyzing process parameters without outcome verification leaves learning gains unmeasured. The FIRE-CU explicitly operationalizes this model by treating Instructor Behavior, Requirement Level, Structure, and Group as process inputs, which directly facilitate or constrain the outcomes of Competence Acquisition and Transfer.<\/p>\n At the command staff level, catastrophic operational failures rarely stem from technical incompetence alone; rather, they emerge from breakdowns in Shared Mental Models<\/a> (SMM), distributed situation awareness, and inter-individual information processing (Cannon-Bowers et al., 1993; Salas et al., 2001). Members of a crisis staff must construct convergent cognitive frameworks regarding incident dynamics, tactical priorities, and task interdependencies under severe time compression and high ambiguity. Drawing from the work of Van den Bossche, Gijselaers, Segers, and Kirschner (2006) on team learning mechanisms, the FIRE-CU integrates specific items into its Competence Acquisition scale (Items 14, 18, and 19) that assess collaborative information exchange, mutual cognitive alignment, and collective sensemaking.<\/p>\n Empirical validation of the FIRE-CU was established during an extensive multi-cohort field investigation at the Institut der Feuerwehr NRW, encompassing instructional iterations over a 16-week cycle (Thielsch & Hadzihalilovic, 2020<\/a>). The validation protocol verified content, construct, convergent, discriminant, and criterion-related validity.<\/p>\n Content validity was established through a multi-stage qualitative and quantitative development pipeline. Initial adaptations of the base FIRE instrument (Schulte & Thielsch, 2019<\/a>) were subjected to an online pilot evaluation ($n = 8$<\/span>) among active command course trainees. The results were reviewed in structured cognitive interviews with the Division Head and Deputy Division Head of the Crisis Management and Research Department at IdF NRW. Based on panel feedback, items reflecting non-applicable low-level tactical duties were discarded, three new competence items were integrated from validated team-learning inventories (Van den Bossche et al., 2006), and transfer items were adapted to command post simulation terminology. Items were adjusted to ensure gender-neutral German phrasing and cross-organizational applicability.<\/p>\n Construct validity is substantiated by the confirmation of the hypothesized multi-tier latent dimensional structure via structural equation modeling. Confirmatory factor analyses demonstrated that the latent dimensions operate as conceptually coherent, empirically separable entities. The correlations among the process factors (Instructor Behavior, Structure, and Group Dynamics) were positive and statistically significant, while correlating negatively with Requirement Level (overload). This pattern supports theoretical expectations: elevated instructional clarity and pedagogical excellence attenuate perceived cognitive overload.<\/p>\n Convergent validity of the outcome scales is supported by substantial item-factor loadings ($p < .001$<\/span>) across the Competence Acquisition and Transfer dimensions. Discriminant validity is evidenced by the distinct empirical separation between Process and Outcome latent variables. Process variables (e.g., Structure and Instructor Behavior) correlate moderately with Competence Acquisition and Transfer ($r \\approx .35$<\/span> to $.55$<\/span>), confirming that while instructional process quality fosters learning outcomes, it does not explain excessive shared variance that would indicate construct redundancy.<\/p>\n Criterion validity was corroborated by correlating the six core psychometric subscales with external validation metrics, including the optional Global Assessment indicators (Item 23: overall learning gain; Item 24: academic school grade; Item 25: recommendation likelihood) and standardized post-training mood assessments. Trainees who reported higher instructor effectiveness, superior structural transparency, and higher group cohesion assigned significantly better overall school grades ($p < .001$<\/span>) and demonstrated higher willingness to recommend the program to external emergency service peers. In parallel, elevated Competence Acquisition and Transfer scores predicted heightened post-exercise self-efficacy and positive affective states upon course completion.<\/p>\n The psychometric reliability of the FIRE-CU was evaluated through comprehensive internal consistency analyses utilizing both classic coefficients (Cronbach\u2019s $\\alpha$<\/span>) and composite reliability estimators (McDonald\u2019s $\\omega$<\/span>), which accommodate violations of tau-equivalence in structural equation models.<\/p>\n Across the validation studies conducted at IdF NRW, the internal consistency values for all six core subscales demonstrated acceptable to excellent reliability, well exceeding standard psychometric thresholds for applied research ($\\alpha > .70$<\/span>):<\/p>\n Overall, across the full battery of subscales, Cronbach\u2019s alpha spans $.75$<\/span> to $.90$<\/span> and McDonald\u2019s omega ranges from $.76$<\/span> to $.91$<\/span>. Test-retest reliability evaluations are theoretically contraindicated for immediate post-training satisfaction metrics, as repeating the survey at a later time introduces variance from real-world field exposures. However, the stability of the factor saturation and parallel internal consistency across independent training cohorts confirms the empirical stability of the measurement tool.<\/p>\n To substantiate the structural architecture of the FIRE-CU, confirmatory factor analyses (CFA) were conducted within R utilizing the structural equation modeling package A four-factor oblique measurement model was specified for the 13 process items, allocating them to Instructor Behavior (4 items), Requirement Level (3 items), Structure (3 items), and Group (3 items). The baseline structural model yielded a commendable fit to the empirical data:<\/p>\n While the overall $\\chi^2$<\/span> statistic was significant (a known artifact of moderate-to-large sample sizes), the normed chi-square ratio was within the desirable range ($\\chi^2 \/ \\text{df} = 1.89$<\/span>). Diagnostic inspection of modification indices suggested an empirical error covariance between Item 12 (“The participants supported each other”) and Item 13 (“I think there was good cohesion in the course”). Given their shared semantic focus on peer solidarity within the Group subscale, allowing this residual covariance was theoretically justified. The revised model demonstrated an enhanced fit:<\/p>\n Standardized factor loadings across all 13 process items were uniformly high and statistically significant ($p < .001$<\/span>), confirming solid convergent measurement validity for the instructional environment.<\/p>\n The hypothesized two-factor oblique measurement model for the 9 outcome items (Competence Acquisition, 6 items; Transfer, 3 items) initially exhibited suboptimal fit indices under baseline constraints ($\\chi^2(26) = 140.96, p < .001$<\/span>; $\\text{RMSEA} = 0.12$<\/span> [$0.11, 0.14$]; $\\text{SRMR} = 0.07$<\/span>; $\\text{CFI} = 0.87$<\/span>; $\\text{TLI} = 0.82$<\/span>).<\/p>\n Inspection of modification indices revealed specific inter-item residual covariances, primarily between Transfer Item 20 (“feel well prepared”) and Item 21 (“gained necessary confidence”), and among specific non-technical competence facets within the 6-item Competence Acquisition scale (correlations between Item 16 [boundary awareness] with Items 15, 18, and 19; and between Item 18 [information processing] and Item 19 [critical verification]). Theoretically, non-technical command staff competencies are systemic and interdependent (Lamers, 2016<\/a>); processing incoming tactical information inevitably correlates with the capacity to critically assess that data. Incorporating these localized error covariances produced an exceptional model fit:<\/p>\n These confirmatory investigations establish that the FIRE-CU operates as a psychometrically sound, multi-dimensional assessment instrument capturing empirical variance across both training delivery and operational skill mastery.<\/p>\n The operational features, technical parameters, and diagnostic deployment procedures of the FIRE-CU are structured as follows:<\/p>\n Items 1 through 23 utilize an unforced, 7-point Likert agreement scale with an additional non-substantive alternative:<\/p>\n Item 24 utilizes a 6-point German grading scale (1 = sehr gut<\/em> \/ Very good, 2 = gut<\/em> \/ Good, 3 = Keywords<\/h2>\n
Authors<\/h2>\n
\n
Purpose<\/h2>\n
\n
Psychological Construct<\/h2>\n
1. Learning Process Constructs (Prozessebene)<\/h3>\n
\n
2. Learning Outcome Constructs (Ergebnisebene)<\/h3>\n
\n
Theoretical Framework<\/h2>\n
Kirkpatrick’s Four-Level Training Evaluation Model<\/h3>\n
\n
Blanchard & Thacker\u2019s Process-Outcome Dichotomy<\/h3>\n
Shared Mental Models in High-Reliability Teams<\/h3>\n
Validity<\/h2>\n
Content Validity<\/h3>\n
Construct, Convergent, and Discriminant Validity<\/h3>\n
Criterion-Related Validity<\/h3>\n
Reliability<\/h2>\n
\n
Factor Analysis<\/h2>\n
lavaan<\/code> (Rosseel, 2012<\/a>), supplemented by packages psych<\/code> (Revelle, 2018<\/a>) and mice<\/code> (van Buuren & Groothuis-Oudshoorn, 2011<\/a>). Due to the theoretical bifurcation between instructional mechanisms and educational outcomes, two independent confirmatory measurement models were evaluated against empirical criteria (Hu & Bentler, 1999<\/a>; Schermelleh-Engel et al., 2003<\/a>).<\/p>\n1. Confirmatory Factor Analysis: Process Level (Prozessebene)<\/h3>\n
\n
\n
2. Confirmatory Factor Analysis: Outcome Level (Ergebnisebene)<\/h3>\n
\n
Instrument \/ Measurement Tool<\/h2>\n
\n
\n
\n
1<\/code> = stimme gar nicht zu<\/em> (Strongly disagree)<\/li>\n2<\/code> = stimme nicht zu<\/em> (Disagree)<\/li>\n3<\/code> = stimme eher nicht zu<\/em> (Somewhat disagree)<\/li>\n4<\/code> = neutral<\/em> (Neutral)<\/li>\n5<\/code> = stimme eher zu<\/em> (Somewhat agree)<\/li>\n6<\/code> = stimme zu<\/em> (Agree)<\/li>\n7<\/code> = stimme vollkommen zu<\/em> (Strongly agree)<\/li>\n[ ]<\/code> = nicht sinnvoll beantwortbar<\/em> (Cannot be meaningfully answered \/ Not applicable)<\/li>\n<\/ul>\n