The Feedback Instrument for Rescue Force Development – Tabletop Exercises<\/strong> (German: Feedback-Instrument zur Rettungskr\u00e4fte-Entwicklung \u2013 Plan\u00fcbung<\/em>, abbreviated as FIRE-PL<\/strong>) is a specialized, psychometrically validated evaluation scale designed to measure the perceived educational quality and instructional effectiveness of tabletop simulation exercises (Plan\u00fcbungen<\/em>) within incident command and leadership training for emergency rescue services. Originally developed through an institutional collaboration between the Institute of the North Rhine-Westphalia Fire Brigade<\/a> (Institut der Feuerwehr Nordrhein-Westfalen<\/em>; IdF NRW) and the Department of Organizational and Business Psychology at the University of M\u00fcnster<\/a> (WWU M\u00fcnster), the instrument functions primarily as a modular extension to the overarching Feedback Instrument for Rescue Force Development (FIRE core scale; Schulte & Thielsch, 2019; Schulte et al., 2019). The FIRE-PL comprises four positively worded items that capture critical facets of simulation-based learning: perceived individual knowledge acquisition, instructor feedback utility, deliberate application of newly acquired command knowledge, and the appropriateness of the tactical exercise difficulty.<\/p>\n Administered using a 7-point Likert response format ranging from 1 (“stimme gar nicht zu”<\/em> \/ strongly disagree) to 7 (“stimme vollkommen zu”<\/em> \/ strongly agree), along with an explicit opt-out category (“nicht sinnvoll beantwortbar”<\/em> \/ not meaningfully answerable), the FIRE-PL requires approximately one minute of completion time, establishing exceptional procedural economy. The psychometric architecture of the instrument was established across three empirical validation studies encompassing over 400 career and volunteer fire service officers. Confirmatory factor analysis (CFA) employing robust maximum likelihood estimation (MLR) substantiated an essentially unidimensional factor structure with excellent global fit indices (\u03c7\u00b2(2) = 4.72, p<\/em> = .094, CFI = .99, TLI = .98, RMSEA = .07, SRMR = .02). Reliability assessments confirmed high internal consistency across independent samples, yielding Cronbach’s \u03b1 values between .81 and .86 and McDonald’s hierarchical omega (\u03c9) coefficients between .82 and .86, reflecting a congeneric measurement model. Construct validity was corroborated by substantial convergent correlations with core leadership training dimensions\u2014specifically instructor behavior (r<\/em> = .53), competence acquisition (r<\/em> = .40), and transfer preparation (r<\/em> = .41)\u2014alongside divergent dissociation from transient mood states (r<\/em> = .20) and cognitive pre-entry test scores (r<\/em> = -.10, n.s.<\/em>). Criterion validity was verified through pronounced associations with overall seminar satisfaction (r<\/em> = .49) and comprehensive course grade assignment (rs<\/sub><\/em> = .32). Consequently, the FIRE-PL constitutes a brief, robust, and theoretically grounded diagnostic instrument for quality assurance, didactic benchmarking, and formative evaluation in high-stakes emergency services training environments<\/span>.<\/p>\n FIRE-PL, tabletop exercises, Plan\u00fcbung, rescue force development, fire service leadership training, incident command simulation, educational evaluation, psychometrics, Kirkpatrick training evaluation, formative assessment<\/p>\n The Feedback Instrument for Rescue Force Development \u2013 Tabletop Exercises (FIRE-PL) was conceptualized, operationalized, and psychometrically validated by a specialized research consortium bridging academic applied psychology and professional civil protection education:<\/p>\n Institutional correspondence regarding the FIRE assessment battery can be directed to the Organizational and Business Psychology research group at the University of M\u00fcnster (Project URL: https:\/\/www.uni-muenster.de\/OWMS\/bfo\/projekte\/fire\/<\/a>).<\/p>\n In high-reliability organizations (HROs) such as municipal fire departments, regional rescue services, disaster management agencies, and civil defense forces, frontline leadership personnel operate under volatile, uncertain, complex, and ambiguous (VUCA) operating conditions. Incident commanders (e.g., Gruppenf\u00fchrer<\/em> [crew commanders], Zugf\u00fchrer<\/em> [platoon commanders], and higher-echelon command staff) are legally and operationally tasked with rapid, safety-critical decision-making under severe temporal pressure, acute threat, sensory overload, and information deficits. The structural backbone of incident command pedagogy in European fire services is governed by doctrinal frameworks such as the German Fire Service Regulation 100 (Feuerwehr-Dienstvorschrift 100: F\u00fchrung und Leitung im Einsatz<\/em>; FwDV 100, 1999). A pivotal instructional methodology utilized to cultivate command competence without the prohibitive resource expenditure and physical hazards of full-scale field operations is the tabletop exercise<\/strong> (German: Plan\u00fcbung<\/em>).<\/p>\n Despite the ubiquitous implementation of tabletop simulations across international emergency services academies, educational quality management within this domain historically relied upon idiosyncratic, non-standardized participant feedback or generic higher education course evaluations. Such general-purpose teaching evaluation questionnaires suffer from substantial construct misalignment: they fail to capture the specific didactic mechanisms unique to synthetic simulation environments, such as exploratory reconnaissance on scaled terrain, cyclical command-decision formulation, verbalization of tactical situational pictures, instructor feedback loops, and calibrated task complexity. The FIRE-PL was deliberately engineered to resolve this diagnostic deficit.<\/p>\n The primary applied and scientific purposes of the FIRE-PL are multi-fold:<\/p>\n The psychological construct operationalized by the FIRE-PL is the Perceived Quality of Tabletop Simulation Exercises<\/strong> (Perzipierte Lehrqualit\u00e4t von Plan\u00fcbungen<\/em>) within high-stakes vocational command development. Unlike basic declarative classroom instruction, simulation-based training relies on experiential, social-cognitive, and metacognitive learning processes. The construct is conceived as an integrated, unidimensional pedagogical outcome reflecting how effectively a tabletop exercise bridges didactic knowledge acquisition, real-time command application, calibrated cognitive load, and feedback-driven behavioral adjustment.<\/p>\n The instrument operationalizes this construct across four interconnected didactic indicators, each reflected by an authentic manifest item:<\/p>\n Operationalized via Item 1 (“Ich habe bei den Plan\u00fcbungen viel gelernt”<\/em> [I learned a lot during the tabletop exercises]), this facet measures the participant’s retrospective appraisal of individual cognitive enhancement. Rooted in classical course evaluation research (e.g., Hirschfeld & Thielsch, 2009), perceived learning gain reflects the subjective acquisition of novel tactical heuristics, operational schema expansion, and enhanced situational awareness. Rather than measuring rote memorization, this indicator captures whether the simulated emergency scenarios successfully enriched the commander’s internal repertoire of tactical responses.<\/p>\n Operationalized via Item 2 (“Die Dozenten gaben mir n\u00fctzliches Feedback zu meinen Leistungen in den Plan\u00fcbungen”<\/em> [The instructors gave me useful feedback on my performance in the tabletop exercises]), this facet gauges the instructional quality of the facilitator’s debriefing. In tabletop simulation pedagogy, the instructor serves not merely as a lecturer, but as an active interlocutor, role-player (e.g., simulating dispatchers, adjacent operational units, or commanding officers), and post-exercise evaluator. Immediate, actionable, and psychologically safe corrective feedback is widely recognized as the single strongest pedagogical determinant of simulation efficacy (Hattie & Timperley, 2007; Salas et al., 2008).<\/p>\n Operationalized via Item 3 (“In den Plan\u00fcbungen konnte ich das neuerworbene Wissen anwenden”<\/em> [In the tabletop exercises, I was able to apply the newly acquired knowledge]), this indicator evaluates experiential enactment. Theoretical command principles\u2014such as assessing building construction collapse hazards, hazardous materials isolation zones, and resource calculation equations\u2014remain abstract unless operationalized in an applied context. Item 3 captures whether the tabletop scenario afforded authentic opportunities for deliberate proceduralization: translating declarative doctrine into concrete command directives under dynamic simulated conditions.<\/p>\n Operationalized via Item 4 (“Der Schwierigkeitsgrad der Plan\u00fcbungen war angemessen”<\/em> [The difficulty level of the tabletop exercises was appropriate]), this facet measures instructional calibration against the learner’s zone of proximal development (Vygotsky, 1978) and cognitive load thresholds (Sweller, 2011). If a tabletop tactical scenario is trivial, experienced firefighters disengage; conversely, if the scenario introduces overwhelming compounding emergencies prematurely, cognitive overload ensues, paralyzing decision-making and impairing structural learning. An appropriate difficulty level signifies optimal balance: challenging the commander’s analytical capacities without inducing cognitive collapse.<\/p>\n The conceptual architecture of the FIRE-PL is grounded in the intersection of cognitive psychology, adult instructional design, and naturalistic decision-making models specific to incident command.<\/p>\n Classical decision theory presumes that decision-makers comprehensively list all viable alternatives, weight probabilities, and compute utility functions. In contrast, Gary Klein\u2019s Recognition-Primed Decision (RPD) model<\/a> demonstrates that experienced incident commanders under severe temporal constraints rely on perceptual pattern recognition. When confronted with an evolving blaze or structural collapse, commanders assess situational cues, instantly categorize the situation based on mental prototypes, mentally simulate a prospective course of action, and execute it if no fatal flaws are visualized.<\/p>\n Tabletop exercises function as artificial cognitive engines designed to expand an officer’s catalog of mental prototypes. In the German fire service model, exercises utilize physical 1:87 scale architectural dioramas (representing industrial complexes, rail catastrophes, residential multi-story fires, or hazardous materials spills) or high-resolution simulated maps. According to NDM theory, the pedagogical value of such tabletop simulations depends entirely on whether the commander is forced to extract environmental cues, verbally formulate a continuous “situational picture” (Lagebild<\/em>), and mentally simulate operational interventions.<\/p>\n The structural execution of the Plan\u00fcbung<\/em> is rigorously anchored in the German standardized command sequence (F\u00fchrungsvorgang<\/em>; FwDV 100, 1999; IdF NRW, 2021), a continuous three-phase cyclic heuristic comprising:<\/p>\n Following this sequence, the instructor conducts an extensive post-exercise debriefing (Planbesprechung<\/em>). The FIRE-PL was constructed specifically to capture whether learners perceive this structural sequence as fostering authentic tactical synthesis rather than detached theoretical exercise.<\/p>\n From an educational psychology perspective, the FIRE-PL draws upon Cognitive Load Theory<\/a> (Sweller, 2011) and Kolb’s Experiential Learning Theory<\/a>. Full-scale field exercises (Einsatz\u00fcbungen<\/em>) introduce intense physical stress, acoustic noise, thermal load, and logistical overhead. Tabletop exercises isolate the cognitive and tactical dimensions of incident command, stripping away peripheral physical noise to allow focused deliberate practice on decision logic. The FIRE-PL monitors whether the tabletop design maintains germane cognitive load\u2014maximizing the cognitive resources allocated to learning schemata\u2014while preventing extraneous load stemming from confusing exercise design or unbalanced tactical difficulty.<\/p>\n The validity of the FIRE-PL was comprehensively evaluated through content, construct (convergent and discriminant), and criterion-related validation protocols across multiple independent cohorts at the IdF NRW.<\/p>\n Initial content validation was executed in Study I (late 2015) involving an expert panel of N<\/em> = 33 individuals, comprising 7 veteran fire service instructors (86% male; mean age M<\/em> = 38.9, SD<\/em> = 6.0) and 26 officer candidates enrolled in group leader (Gruppenf\u00fchrer<\/em>) qualification courses (96% male; mean age M<\/em> = 30.5, SD<\/em> = 6.9). Panelists rated each prospective item regarding semantic clarity, unambiguous phrasing, and domain relevance. Over 90% of all participants judged the tabletop items as “important” or “highly important” for training quality, and less than 3% identified any phrasing ambiguities. Consequently, all four items were retained in their original theoretical formulation without requiring semantic revision<\/span>.<\/p>\n Convergent validity was evaluated in Study III (N<\/em> = 288 fire service commanders) by correlating the FIRE-PL score with validated subscales of the FIRE core questionnaire (Schulte & Thielsch, 2019). Because tabletop exercises fundamentally depend upon facilitator coaching and aim to bridge classroom theory with practical command capability, substantial positive correlations were hypothesized with the process dimension of instructor behavior and the outcome dimensions of competence acquisition and transfer preparation. Empirical correlations confirmed these hypotheses<\/span>:<\/p>\n Discriminant validity was verified against constructs theoretically distant from tabletop instructional quality, as well as known evaluation bias variables:<\/p>\n Criterion validity was established against overarching educational evaluation criteria:<\/p>\n The reliability of the FIRE-PL has been extensively scrutinized across independent training cohorts utilizing classical and modern psychometric metrics. Because factor analytic investigations indicated that manifest item loadings and error variances are not strictly identical, the underlying measurement model satisfies congeneric rather than essentially tau-equivalent assumptions. Consequently, while Cronbach’s alpha (\u03b1) is reported for historical comparability, McDonald\u2019s hierarchical omega (\u03c9) represents the unbiased, psychometrically superior indicator of internal consistency.<\/p>\n Reliability analyses conducted in Study II (N<\/em> = 155) and Study III (N<\/em> = 288) demonstrated consistently high internal consistency<\/span>:<\/p>\n Across the validation samples, corrected item-total correlations (Trennsch\u00e4rfe<\/em>) ranged from .61 to .84, well above the conventional psychometric threshold of .30, demonstrating that every item contributes meaningfully to the common core construct:<\/p>\n The high item discrimination parameters, especially for Item 3 (.84) and Item 4 (.73), demonstrate that the operationalization of applied knowledge transfer and calibrated task challenge forms a remarkably cohesive measurement core.<\/p>\n The factorial validity and latent dimensionality of the FIRE-PL were investigated through sequential exploratory and confirmatory factor analyses using RStudio (version 1.2.5001) with the In Study II, an initial pool of teaching method evaluation items\u2014encompassing self-directed learning (EVA), group work (G), tabletop exercises (PL), and full-scale operational field exercises (E)\u2014was administered to N<\/em> = 155 course participants. An oblimin-rotated principal component analysis \/ exploratory factor analysis (EFA) was performed. The four tabletop exercise items clustered unequivocally on a distinct latent factor (F2), exhibiting exceptionally strong standardized factor loadings<\/span>:<\/p>\n None of the tabletop items displayed cross-loadings greater than .20 on adjacent factors representing group work or self-directed learning, establishing pristine structural independence from other instructional modalities.<\/p>\n In Study III, the hypothesized single-factor structure was formally evaluated using Confirmatory Factor Analysis (CFA) on a fresh validation cohort of N<\/em> = 288 fire service commanders. Given mild negative skewness and leptokurtic distribution typical of high-quality vocational education evaluations, robust maximum likelihood estimation (MLR) was implemented. The unidimensional model demonstrated outstanding goodness-of-fit based on the strict methodological standards articulated by Hu and Bentler (1999)<\/a><\/span>:<\/p>\n All standardized latent factor loadings in the CFA were uniform, robust, and highly statistically significant (\u03bb \u2265 .60, p<\/em> < .001). The convergence across both exploratory and confirmatory modeling firmly substantiates the theoretical presumption of an essentially unidimensional construct<\/span>.<\/p>\n The FIRE-PL is structured as follows:<\/p>\nKeywords<\/h2>\n
Authors<\/h2>\n
\n
Purpose<\/h2>\n
\n
Psychological Construct<\/h2>\n
1. Subjective Learning Gain (Subjektiver Lerngewinn)<\/h3>\n
2. Formative Instructor Feedback Utility (N\u00fctzlichkeit des Dozentenfeedbacks)<\/h3>\n
3. Knowledge Application and Transfer of Learning (Wissenstransfer und Wissensanwendung)<\/h3>\n
4. Appropriateness of Cognitive Challenge \/ Task Difficulty (Angemessenheit des Schwierigkeitsgrades)<\/h3>\n
Theoretical Framework<\/h2>\n
1. Naturalistic Decision Making (NDM) and Recognition-Primed Decision (RPD) Theory<\/h3>\n
2. The Three-Phase Command Cycle: Feuerwehr-Dienstvorschrift 100 (FwDV 100)<\/h3>\n
\n
3. Cognitive Load Theory and Experiential Learning<\/h3>\n
Validity<\/h2>\n
1. Content Validity<\/h3>\n
2. Convergent Construct Validity<\/h3>\n
\n
3. Discriminant Construct Validity and Control for Biases<\/h3>\n
\n
4. Criterion-Related Validity<\/h3>\n
\n
Reliability<\/h2>\n
Empirical Reliability Findings<\/h3>\n
\n
Item Discrimination and Manifest Metrics<\/h3>\n
\n
Factor Analysis<\/h2>\n
psych<\/code> (Revelle, 2021) and lavaan<\/code> (Rosseel, 2012) statistical packages.<\/p>\n1. Exploratory Factor Analysis (Study II)<\/h3>\n
\n
2. Confirmatory Factor Analysis (Study III)<\/h3>\n
\n
Instrument \/ Measurement Tool<\/h2>\n
\n