Every other functional test would fail. The sensor interface error was out of bounds; nominal as it turns out. The technician writes a discrepancy ticket and moves on to another work order. This unit will pause testing for several days and sometimes several weeks. The responsible engineer would disposition the discrepancy use-as-is without even plotting the time-series data. The design engineer had attributed the out-of-bounds performance to noise, failing to realize this was a circuit that didn’t always meet requirements. Every responsible engineer since had inherited that disposition.

This continued for three years. The unit was qualified to SMC-S-016. The test bounds were the requirements and there was no budget to redesign the unit. Three responsible engineers had shepherded this unit serially from design to volume production. Then I joined.

“What’s the worst-case performance under temperature and manufacturing tolerance of the channel?” One engineering leader dismissed the question, citing we were a risk-tolerant organization that prided itself on moving fast; worst-case analysis is too conservative. Each unit was being tested to prove it met requirements yet no engineer could tell me the design actually met them. Using Microsoft Excel I worked through the design in two hours and found the answer: 3x worse than the requirement.

Nearly every unit was use-as-is when it failed for this specific failure-mode and installed anyway. We were rolling the dice and didn’t realize it for three years. The first batch of vehicles with this hardware was set to fly soon. A software fix mitigated potential vehicle loss, but only because I asked a question that belongs in design, not qualification or acceptance.

Stop using acceptance and qualification tests to verify requirements while expecting to deliver on-time and on-budget. Self-imposed design requirements are owned by the engineer and can be verified, added, modified, or deleted at will. Requirements verification is a property of the design, not of the unit. SMC-S-016 expects tailoring of contractual requirements while legacy culture expects compliance: accept the risks you can afford and avoid the risks you cannot.

Qualification Is About Process and Environments Not Design

Qualification “… is conducted to demonstrate that the design, manufacturing process, and acceptance program produce hardware/software meeting specification requirements with adequate margin to accommodate multiple rework and test cycles” (SMC-S-016, §4.2.2). Qualification verifies the production step, not the design. Acceptance verifies you built what you designed. By confirming your design will always meet requirements in analysis wherever possible you shift the approach to acceptance and qualification towards environmental and manufacturing risk reduction. The question shifts from “does this design work?” to “can I manufacture my design that I know will meet requirements?”

The assumption of qualification is that the design is already valid. “The hardware subjected to qualification testing shall be produced in the same factory from the same drawings, using the same parts, materials, tooling, manufacturing process, and level of personnel competency as used for flight hardware” (SMC-S-016, §4.3.2.1). This implies a high level of certainty in the design. Notably, the ideal qualification unit is a randomly selected acceptance tested flight unit. During qualification, “Performance testing demonstrates design margins and specification compliance…” (SMC-S-016, §3.36). You are confirming a known answer; certainty earned prior to qualification.

You are expected to tailor requirements to acceptable risk (SMC-S-016, §1.4). Every program I have worked on has had a finite budget and schedule. When physics bounds a requirement then analysis earns its place. For example, a 4-20mA transducer conditioning circuit may use an op-amp, RC filter, and Analog to Digital Converter (ADC) in the signal chain. Performance over production volume and thermal environments is deterministic provided workmanship is maintained. Using qualification and acceptance testing answers a question that should have been closed in development. That’s compliance. Tailoring is not only removing acoustic testing for your avionics box, it’s substituting physics for finite time and resources.

I have yet to see an organization willing to redesign a unit that fails qualification; flight hardware is already in production and the schedule doesn’t account for new hardware development spins. Verifying the design in qualification is an expensive way to force design margin concessions, find software workarounds, and risk large schedule slips. It’s compliance theatre disguised as “space is hard”. Engineers who mistake compliance for competence go into testing without knowing they are going to pass.

The Risk Reduction Chain

Development testing is the only time an engineer can change the design easily. Acceptance testing damages the unit under test, consuming design life verifying workmanship. It does not indicate life remaining in the unit. Qualification shows life remaining after damage from acceptance. Qualification is testing the entire process of building flight hardware. Drill a hole in the qual unit for a thermocouple not in the flight design and you’ve invalidated qualification. Both acceptance and qualification assume the design meets specification requirements. The question is whether the flight units can be produced at equivalent quality to the qual unit and have useful life after environmental stress. Using acceptance and qualification to prove a design meets requirements admits the performance was unknown. Flight hardware production is already committed before qualification begins. Requirements verification is a property of the design; close it there.

Sharpen The Axe First

You must know the answer before you conduct the test. Bring-up testing is the most important testing you will ever do on an avionics design. Many engineers approach it at the surface level: my sensor reads accurately on the lab bench with the one board I have. It’s statistically insignificant hope. This is the time to beat the hell out of the design and verify your requirements.

SMC gives you, the engineer, the power to do this: “Development tests should be used to confirm structural and performance margins, manufacturability, testability, maintainability, reliability, life expectancy, and compatibility with system safety. Where practical, development tests should be conducted over a range of operating conditions that exceeds the design limits to identify marginal capabilities and marginal design features” (SMC-S-016, §4.2.1). §6.2.2 places electrical and mechanical performance demonstration explicitly in development, alongside risk reduction, before the design is committed to production. Analysis verifies requirements derived from deterministic physics so you can divert test resources where they reduce the most risk.

How do you sharpen the axe? Between the Preliminary Design Review (PDR) and Critical Design Review (CDR) is the window. An engineering model unit with form, fit, and function of the intended flight design will operate just like the flight unit. The engineering model confirms performance; it does not confirm environmental survivability. Leave workmanship and environmental margin verification to acceptance and qualification. Engineers who wait for flight hardware to verify the design to requirements have missed the only opportunity to mitigate failures without great pain. Testing early to understand your design instead of testing for compliance will give you control of your schedule and risk aperture. Once you gain control of the design, acceptance and qualification are free to focus on what only they can find.

Testing Like You Fly Is An Excuse

Test like you fly: It sounds professional. It sounds reasonable. It is also why some organizations build a culture of hope. “These parameters shall be varied throughout their specification ranges and the sequences expected in flight operation” (SMC-S-016, §6.3.2.2). The test like you fly methodology implied is at the external interface to the device under test. This only applies to contractual requirements, and those can be tailored. Self-imposed design requirements aren’t covered by SMC or §6.3.2.2; they need good engineering.

An engineer implements a Low-Voltage Differential Signal (LVDS) driver operating at 200MHz. Implementing test like you fly, the acceptance and qualification automated functional test racks perform their LVDS tests at 200 MHz. The LVDS channel passes easily; units are installed on the vehicles. Then, vehicle level thermal cycling shows an LVDS channel failing. Root cause investigation finds that several other acceptance tested units exhibit the same failure mode but the tests indicate passing hardware: solder cracks in the LVDS differential pair routing. A solder crack can form a capacitor in series with the signal path. Low-frequency signals are blocked while high-frequency signals can pass through. Genuinely a bad time to be a responsible engineer of this hardware.

How was the solder crack failure missed when functional testing operated at flight performance? Testing like you fly is an excuse for not thinking. When I work with the test engineer on my hardware test methodology I am asking one question: “How would I know this signal path is intact?” versus “how does it fly”. In development I performed the analysis to verify operation at speed. For automated functional testing in acceptance and qualification I explicitly aim to verify the signal path: acceptance and qualification are going to damage my unit. Blindly testing to §6.3.2.2 produced worse test coverage than critical thinking and tailoring. Know what you are looking for before you conduct the test.

The Engineer Owns This

Two hours of inserting myself as the responsible engineer showed that three years of non-conformance tickets and more than one year of schedule slip across production units could have been avoided. Damaged hardware masked by test-like-you-fly ideology installed on vehicles was also avoidable with ownership, not process conformity. These patterns are not isolated incidents. New engineers learn from experienced engineers what is acceptable. This applies beyond SMC; culture is an organism that does not default to effective; you have to construct it deliberately.

Following a defined process like SMC doesn’t guarantee risk reduction. Technical debt comes due when you scale hardware production: schedule slips, non-conformance triage, workarounds, cultural atrophy, and mission risk. The safest vehicle is one that never leaves the lab. Not understanding your design prior to qualification and production. Unacceptable.

A passed test is an artifact. It is not understanding why the test passed. Confusion between passing and understanding creates organizations that progress with hope. What is acceptable? Risk you understand is risk you control; risk you are naive to owns your company. Engineers need ownership and agency to treat every requirement, every test, and every disposition as an extension of their reputation. Engineers who own their products take judgement calls seriously. Ownership allows them to build a production system that catches what is necessary instead of what feels safe. When hardware I owned flew, I knew it would work, and any failure would have to be something I couldn’t have prevented in the lab.