The Feedback Instrument for Rescue Force Development \u2013 Seminars and One-day Events<\/strong> (known by its German acronym FIRE-ST<\/strong>: Feedback-Instrument zur Rettungskr\u00e4fte-Entwicklung \u2013 Seminare und Tagesveranstaltungen<\/em>) is a standardized psychometric evaluation instrument engineered to evaluate instructional quality, learning processes, and self-reported learning outcomes in emergency service professional development. Adapted from the foundational FIRE instrument (Schulte & Thielsch, 2019<\/a>)\u2014which originally targeted multi-week residential leadership academies\u2014the FIRE-ST is streamlined specifically for short-format educational interventions ranging from one-day seminars to three-day workshops. The instrument comprises 23 items structured across two distinct hierarchical evaluation planes: instructional process dimensions, which capture Trainer Behavior<\/em> (Dozentenverhalten<\/em>, 6 items), Level of Challenge \/ Overextension<\/em> (Anforderungsniveau<\/em>, 3 items), and Structure<\/em> (Struktur<\/em>, 3 items); and instructional outcome dimensions, which evaluate Competence Acquisition<\/em> (Kompetenzerwerb<\/em>, 5 items) and Training Transfer<\/em> (Transfer<\/em>, 3 items). In addition, three optional single items gauge global academic grades, program recommendation intent, and open-ended qualitative feedback. Responses for the primary 20 items are recorded on a 7-point Likert scale alongside an explicit option to indicate that an item cannot be meaningfully evaluated. Psychometric validation was conducted on a sample of $N = 247$<\/span> emergency commanders and firefighters attending leadership and technical seminars at the Institute of the North Rhine-Westphalia Fire Brigade (Institut der Feuerwehr Nordrhein-Westfalen<\/em>; IdF NRW). Confirmatory factor analyses demonstrated robust structural validity across both the process model ($\\chi^2(51) = 72.40, p = .026, \text{CFI} = .99, \text{TLI} = .98, \text{RMSEA} = .04$<\/span>) and the outcome model ($\\chi^2(17) = 36.05, p = .005, \text{CFI} = .97, \text{TLI} = .95, \text{RMSEA} = .09$<\/span>). Internal consistency is well-supported across subscales, with Cronbach’s $\u0007lpha$<\/span> ranging from $.74$<\/span> to $.89$<\/span> and McDonald’s $\\omega_H$<\/span> ranging from $.76$<\/span> to $.89$<\/span>. The FIRE-ST provides an empirically grounded, rapid-administration framework for quality assurance, personnel development, and operational safety training in high-reliability organizations.<\/p>\n FIRE-ST, training evaluation, emergency services, rescue force development, instructional quality, firefighter leadership, Kirkpatrick model, competence acquisition, training transfer, high-reliability organizations, psychometrics<\/p>\n The FIRE-ST was conceptualized, operationalized, and psychometrically validated through an academic\u2013practitioner partnership between the Department of Organizational and Business Psychology at the University of M\u00fcnster (Westf\u00e4lische Wilhelms-Universit\u00e4t M\u00fcnster<\/a>) and the Institute of the North Rhine-Westphalia Fire Brigade (Institut der Feuerwehr Nordrhein-Westfalen \u2013 IdF NRW<\/a>):<\/p>\n Emergency response personnel operate in dynamic, high-stakes environments characterized by time compression, high ambiguity, severe physical risk, and life-or-death decision-making. Operating as High-Reliability Organizations (HROs)<\/a>, emergency service systems\u2014such as municipal fire departments, disaster relief agencies, and technical rescue teams\u2014must guarantee that training programs produce measurable competencies and reliable behavioral transfer under operational stress. Incident commanders, group leaders, and technical specialists participate regularly in continuing professional education to master advanced tactical protocols, disaster psychology, chemical defense, and inter-agency coordination.<\/p>\n Despite the vital importance of this training, the tools historically used to evaluate emergency service instruction have suffered from significant methodological limitations. Training institutions frequently rely on ad-hoc, unvalidated customer-satisfaction forms (often termed “smile sheets”). These unstandardized forms confound affective entertainment with actual instructional efficacy, fail to differentiate cognitive load from pedagogical clarity, and ignore the structural distinction between classroom process characteristics and proximal learning outcomes.<\/p>\n While the original Feedback Instrument for Rescue Force Development<\/em> (FIRE; Schulte & Thielsch, 2019) resolved these deficits for extensive multi-week residential leadership certifications, its structural assumptions do not align directly with short-format seminars and one-day events. Multi-week academies involve small cohort cohorts, extensive peer collaboration, and iterative scenario-based tactical field exercises. In contrast, short seminars (spanning 1 to 3 days) often accommodate up to 120 attendees, rely predominantly on instructor-centered didactic delivery, and feature concentrated thematic presentations. Under these condensed conditions, peer-group dynamics diminish as an instructional factor, while the didactic clarity, cognitive pacing, and communicative competence of the individual trainer become paramount.<\/p>\n The FIRE-ST was engineered to resolve this applied psychometric gap. Its purpose is to deliver a concise, psychometrically sound, and diagnostically differentiated instrument capable of assessing both instructional delivery (processes) and self-assessed capability growth (outcomes) in 3 to 5 minutes of administration time. In applied contexts, it serves four central functions:<\/p>\n The FIRE-ST operationalizes instructional quality in short-format emergency service training as a multi-dimensional construct organized into two correlated, conceptually distinct domains: Learning Processes<\/strong> (instructional inputs and classroom environment) and Learning Outcomes<\/strong> (proximal competence acquisition and perceived transfer readiness).<\/p>\n The process domain reflects the participant’s direct experience of teaching methods, content organization, and cognitive pacing during the seminar:<\/p>\n The outcome domain evaluates the proximal changes in cognitive capabilities and operational readiness attributed directly to course participation:<\/p>\n To facilitate administrative reporting, three non-psychometric items supplement the validated core scales: Item 21 provides a traditional German school grade summary metric (1 = very good<\/em> to 6 = insufficient<\/em>); Item 22 captures behavioral recommendation intent via a dichotomous (yes\/no) metric; and Item 23 offers an open-ended qualitative field for text-based participant commentary<\/span>.<\/p>\n The architecture of the FIRE-ST is grounded in Kirkpatrick’s Four-Level Training Evaluation Model<\/a>, augmented by modern instructional design and organizational psychology paradigms (Alliger et al., 1997; Blanchard & Thacker, 2013).<\/p>\n Kirkpatrick’s classical taxonomy stratifies training evaluations into four ascending tiers:<\/p>\n Methodologists emphasize that initial evaluations must rigorously establish reactions and proximal learning before meaningful inferences can be drawn regarding distal behavioral transfer and organizational outcomes (Kirkpatrick & Kirkpatrick, 2006). Alliger et al. (1997) revised Kirkpatrick’s Level 1 by differentiating between affective reactions<\/em> (“Did I enjoy the seminar?”) and utility reactions<\/em> (“Is this training relevant to my command performance?”). The FIRE-ST incorporates utility reactions within its process scales (e.g., Structure and Trainer Feedback) and measures self-assessed Level 2 Learning through the Competence Acquisition scale.<\/p>\n Following the structural formulation established in the broader FIRE framework (Schulte & Thielsch, 2019), the FIRE-ST relies on a fundamental distinction between teaching processes<\/strong> and learning outcomes<\/strong>. This conceptual dualism is illustrated below:<\/p>\n By decoupling the evaluation of instructional processes from learning outcomes, the FIRE-ST avoids the confounding effects common to single-score satisfaction instruments. For example, a charismatic instructor might receive high marks for presentation style while presenting an overloaded, unstructured syllabus that yields poor competence acquisition. Conversely, an analytically demanding seminar led by an understated instructor may produce substantial gains in hazard assessment capabilities. The dual-model architecture of the FIRE-ST disentangles these instructional dynamics.<\/p>\n The validity of the FIRE-ST was evaluated across multiple empirical dimensions using data collected from emergency service commanders and firefighters at the IdF NRW.<\/p>\n Content validity was established through adaptation from the extensively validated parent FIRE instrument (Schulte & Thielsch, 2019; Schulte et al., 2019). The adaptation process systematically accounted for the structural characteristics of short seminars:<\/p>\n2. Keywords<\/h2>\n
3. Authors<\/h2>\n
\n
4. Purpose<\/h2>\n
\n
5. Psychological Construct<\/h2>\n
5.1. Instructional Process Dimensions<\/h3>\n
\n
5.2. Instructional Outcome Dimensions<\/h3>\n
\n
5.3. Supplementary Evaluative Items<\/h3>\n
6. Theoretical Framework<\/h2>\n
6.1. The Adapted Kirkpatrick Evaluation Paradigm<\/h3>\n
\n
6.2. The Process\u2013Outcome Dualism<\/h3>\n
\n\n
\n \nEvaluation Plane<\/th>\n FIRE-ST Subscale<\/th>\n Underlying Pedagogical Theory<\/th>\n Primary Focus<\/th>\n<\/tr>\n<\/thead>\n \n Learning Processes<\/strong>
(Instructional Inputs)<\/td>\nTrainer Behavior (Items 1\u20136)<\/td>\n Teacher Clarity & Behavioral Feedback Theories<\/td>\n Pedagogical delivery, subject mastery, participant engagement<\/td>\n<\/tr>\n \n Level of Challenge (Items 7\u20139)<\/td>\n Cognitive Load Theory (Sweller, 1988)<\/td>\n Pacing, volume, cognitive overextension (extrinsic load)<\/td>\n<\/tr>\n \n Structure (Items 10\u201312)<\/td>\n Advance Organizer & Schema Construction Theories<\/td>\n Conceptual flow, structural clarity, syllabus transparency<\/td>\n<\/tr>\n \n Learning Outcomes<\/strong>
(Proximal Yields)<\/td>\nCompetence Acquisition (Items 13\u201317)<\/td>\n Naturalistic Decision Making (NDM; Klein, 1993)<\/td>\n Hazard appraisal, rapid decision-making, stress regulation<\/td>\n<\/tr>\n \n Transfer (Items 18\u201320)<\/td>\n Model of Training Transfer (Baldwin & Ford, 1988)<\/td>\n Deployment readiness, command confidence, local applicability<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n 7. Validity<\/h2>\n
7.1. Content and Curricular Validity<\/h3>\n
\n
7.2. Construct and Factorial Validity<\/h3>\n