ProcessEdge AI & IoT
Over-the-air model updates for edge AI fleets, from signed bundle to rollback
Shipping OTA model updates to an edge AI fleet is a release-engineering problem first: the model travels with preprocessing code, a runtime and firmware it must match, over links that drop. A safe process versions all of that as one signed bundle, proves it in shadow mode on a small cohort, widens exposure only when health gates pass, and keeps an automatic way back on every device.
On this page
- Why a new model is a fleet operation, not a notebook result
- Who sends what during one staged model release
- Seven stages of a safe edge model release
- Health gates to read at shadow, canary and fleet-wide stages
- Halt, exclude or proceed: reading a rollout that looks odd
- Telemetry and drift when links come and go
- Security baselines that shape how devices receive updates
- Questions and answers
- Sources
Why a new model is a fleet operation, not a notebook result
A model that scores better offline can still make a fleet worse. It may need an operator the older runtime lacks, use more memory than devices with a busy firmware build can spare, or behave differently at sites whose sensors were never in the training data. And unlike a cloud deployment, you cannot simply redeploy in seconds: some devices are asleep, some are behind a cellular link with a data cap, and some will lose power halfway through the download.
That is why the hub's third stage, operating a fleet, is about secure updates, version tracking and delayed synchronization5. The process below treats every model release the way firmware teams treat a firmware release: as an artifact with a manifest, a signature, a staged rollout and a tested path back.
Who sends what during one staged model release
- Model registry
Holds approved bundles and their manifests.
- Update service
Targets cohorts and tracks which bundle each device runs.
- Edge device
Verifies, installs to an inactive slot and reports.
- Telemetry store
Receives buffered results when links return.
- Release owner
Reads gate metrics and promotes or halts.
- Model registry to Update servicesigned bundle
- Update service to Edge deviceoffer to cohort
- Edge device to Edge deviceverify, check compat
- Edge device to Edge deviceinstall to slot B
- Edge device to Telemetry storeshadow results
- Telemetry store to Release ownergate metrics
- Release owner to Update servicepromote or halt
- Update service to Edge deviceactivate or revert
Seven stages of a safe edge model release
Define the bundle and its compatibility
Package the model file with the preprocessing code that produces its inputs, the runtime version it was compiled for, the minimum firmware and the hardware revisions it was tested on. A model without its preprocessing is the most common source of silent errors after an update.
Sign the bundle and protect the channel
Sign the manifest and artifacts with keys held outside the build system, and have the device check the signature against a trust anchor installed before deployment. The IETF firmware update architecture describes this manifest-and-signature model, including sequence numbers that stop a device being tricked into installing an older, vulnerable image1.
Prove it on a bench fleet
Install on at least one device of every hardware revision in service, with production firmware. Interrupt power and network mid-install on purpose, and confirm the device boots the previous slot.
Shadow on a small cohort
Run the new model beside the current one on a few representative devices. The current model still decides; the new one only logs its outputs, latency and memory so you can compare both on live inputs.
Activate on a canary cohort
Let the new model decide on a slightly larger group that spans sites, hardware revisions and connectivity types. Agree the health gates and the halt criteria before activation, not after the first odd chart.
Ramp by cohort
Widen exposure in steps, pausing after each for enough operating time to include the conditions that matter: night shifts, weekend loads, cold starts. Devices that were offline join later cohorts rather than receiving the newest bundle on reconnection unannounced.
Close out and keep watching
Retire the previous bundle only once rollback is no longer needed, and keep comparing input distributions per site so drift is noticed before outcomes degrade.
Health gates to read at shadow, canary and fleet-wide stages
| Signal | Shadow | Canary | Fleet-wide |
|---|---|---|---|
| Install and boot | Every device verifies and installs to the inactive slot | No device stuck between slots | Failed installs revert automatically and are counted |
| Latency and memory | New model within budget alongside the old one | Within budget under real load | Watched per hardware revision |
| Agreement with current model | Disagreements reviewed by a domain expert | Disagreements traced to expected improvements | Tracked as a trend, not a gate |
| Field outcomes | Not yet measurable | Confirmed events compared with the previous period | Feeds the next training set |
| Telemetry completeness | Buffered results arrive from every shadow device | Silent devices are chased, not assumed healthy | Coverage reported alongside every metric |
Thresholds depend on the decision the device makes. Set them with the people who own that decision.
Halt, exclude or proceed: reading a rollout that looks odd
- If
Devices fail signature checks or cannot boot the new slot.
ThenHalt the release and investigate the build and keys before anything else.
Signature or boot failures point to the artifact or the update channel, which affects every device, not just one site.
- If
Latency exceeds budget on one hardware revision only.
ThenExclude that revision from the release and ship it a variant compiled for its runtime.
Mixed hardware generations are normal in long-lived fleets; one bundle rarely suits all of them.
- If
Disagreement with the old model clusters at a few sites.
ThenInspect the inputs from those sites before judging the model.
A sensor change, mounting difference or new operating mode often explains it, and it is evidence for the next training set.
- If
A cohort reports almost nothing after activation.
ThenPause promotion until telemetry returns or devices are checked in person.
In intermittent networks silence is missing evidence, not a sign that everything works.
Telemetry and drift when links come and go
Design telemetry for store-and-forward from the start. Devices buffer compact records of what they decided, with what confidence and on which bundle version, in a bounded queue that drops the least useful records first when full. Each record carries a device timestamp and a monotonic counter, so the backend can order events even if the device clock drifted while offline.
Drift detection works on the same records. Summaries of input features per site and per week, compared with the training data, show where conditions are moving away from what the model knows. Pair that with confirmed outcomes from maintenance or operations staff, which is where the predictive monitoring workflow takes over.
Security baselines that shape how devices receive updates
Regulation (EU) 2024/2847 (Cyber Resilience Act)
EUApplies whenProducts with digital elements are made available on the EU market; most obligations apply from 11 December 2027 and the Article 14 reporting obligations from 11 September 20264.
- Products must be able to receive security updates, with automatic security updates enabled by default where applicable and a way to opt out4.
- Manufacturers must provide mechanisms to distribute updates securely and, where technically feasible, ship security updates separately from functionality updates4.
ETSI EN 303 645 V3.1.3, Cyber Security for Consumer Internet of Things: Baseline Requirements[^3]
European standard, used internationallyApplies whenA manufacturer designs network-connected consumer IoT devices and chooses to demonstrate conformance with this baseline3.
NIST IR 8259 Rev. 1, Foundational Cybersecurity Activities for IoT Product Manufacturers[^2]
US (voluntary guidance)Applies whenA manufacturer plans pre-market and post-market cybersecurity activities for an IoT product2.
- Recommends building cybersecurity, including how products will be maintained after sale, into the development process rather than adding it afterwards2.
Questions and answers
How often should edge AI models be updated?
On evidence, not on a calendar. Update when drift monitoring or confirmed outcomes show the current model is degrading, when a new failure mode needs to be recognized, or when a security fix in the runtime forces a rebuild. Security updates to firmware and runtimes follow their own, faster path and should not wait for a model release.
How do we update models over cellular links with tight data limits?
Send binary deltas against the installed bundle where your update tooling supports them, schedule downloads for off-peak windows, and quantize models so the artifact itself is small. Let devices resume interrupted downloads rather than restart them, and verify the reassembled bundle's signature before installing anything.
What if our fleet mixes several hardware generations?
Treat each hardware revision as a separate release target with its own compiled bundle and test devices, tracked under one logical model version. The manifest should state which revisions a bundle supports, and the update service should refuse to offer it to anything else.
Is shadow mode always possible on constrained devices?
Not always. If a device cannot hold two models in memory, run shadow evaluation on a small set of higher-specification units at representative sites, or replay recorded field inputs through the new bundle on bench hardware. Both are weaker evidence than on-device shadowing, so keep the canary cohort small for longer.
Sources
- RFC 9019: A Firmware Update Architecture for Internet of Things — IETF / RFC Editor · checked 10 October 2026
- NIST IR 8259 Rev. 1: Foundational Cybersecurity Activities for IoT Product Manufacturers — NIST · checked 10 October 2026
- ETSI EN 303 645 V3.1.3 (2024-09) Cyber Security for Consumer Internet of Things: Baseline Requirements — ETSI · checked 10 October 2026
- Regulation (EU) 2024/2847 on horizontal cybersecurity requirements for products with digital elements (Cyber Resilience Act) — EUR-Lex · checked 10 October 2026
- Edge AI & IoT — ColdAI