Firmware updates have been in the news recently, are they actually causing more problems than they are solving? Paul Milligan speaks to those at both ends of this issue.
Firmware updates are the quintessential double-edged sword in the AV world. They can fix problems, add functionality, and even continue the life of an aging product. The issue is they can also trigger whole AV systems to come to a grinding halt. And in a lot of those cases, it might not be a problem caused by anything you’ve installed, but by a compatibility or security issue somewhere else on the same network that is causing parts of the system to become inoperable.
Are we just aware of this as an issue recently because some of these failures have hit the morning news on TV? Do the vast majority of updates go by without a hitch and it’s not really an issue? We spoke to integrators and manufacturers to get a balanced view on the current situation.
Just how difficult have firmware updates become for integrators and clients? “They have become problematic because features are added to products whether they’re UC-centric, whether they’re AVoIP, or patching security risks. All of these different elements require unification of whatever that firmware may be for compatibility between all the different parts that may be part of a given ecosystem, so validation of how all the different components that make up a system work together can be greatly impacted All these different elements need to be unified - regardless of their firmware - to ensure compatibility across any given ecosystem. As a result, validating how all system components work together becomes a critical and complex task,” says John Richards, vice president of engineering, systems integration at systems integrator AVI-SPL.
Last year problems caused by firmware updates was a significant problem says Toby Wise, CEO at systems integrator Snelling, citing one particular example. “There was a particular vendor that came to prominence through the back end of the pandemic who gained a huge amount of market share with a network-based product set that was very reliant on firmware and the functionality was limited by the stability, or lack of it really.” Wise adds Snelling tried to resolve the issue itself but had to turn to the manufacturer to cure the problem. “It took them a little while to respond, but they did. It’s that period between fault and fix, it’s difficult for us to manage with the end client because you know a fix is coming but you’re not sure how long it’s going to be and it’s very difficult to manage that process. Thankfully it’s much less of a problem this year.”
Sometimes the issue with firmware can be the amount of the time that piece of kit has spent in a box says Matt Hale, tech support and pre-sales engineer, Kramer UK. “As a vendor, hardware can be on a shelf for a long time before it’s ordered and used. That means the firmware on it is, more often than not, an old version. The same applies to our distributors. It would be impossible for us to update all hardware before it’s despatched. Only critical updates for vulnerability fixes will be actioned internally. It is the responsibility of the integrator to check firmware versions on the hardware they receive. However, this is often overlooked because the assumption is that all hardware leaves the factory with the latest firmware. They do leave the factory with it, but after that time passes and new versions are released.”
The speed at which changes are happening through updates has accelerated says Richards, “It’s critical to understand the differences as you move through the different versions. There can be products that don’t allow you to jump as quickly from a legacy firmware all the way to the latest one, it requires a migration plan to get to the latest version. That can be problematic if you try to jump all the way.”
The regularity of firmware updates is partly because the products being released onto the AV market are no longer the finished version, because we can update their software many times over, they become an evolving entity. That’s been our standpoint for a long time now says Joel Mulpeter, senior director of product marketing, Crestron. “We make sure something has enough room to grow and evolve, and we can add feature sets to it. We hope that the products people buy offer more value for them in six months’ time than the day they bought them.” One upside to this is the longevity it offers to integrators and clients alike.
“I would say on average in this industry, we’re probably looking at about five to seven year consideration in any space at a minimum. If you look at the consumer industry, your mobile phone is two years old when your telco calls you up to take a new one. We definitely don’t want to get to that point,” Mulpeter adds.
Firmware issues can be caused between products from different manufacturers occupying the same space, so is sticking with one manufacturer as much as possible the way to avoid this? Or does the recent demand for more interoperability within the AV world make that sort of plan difficult to deliver? There’s a lot to be said for sticking to one partner that will fully support the solution from end to end says Richards. “It can require a lot of validation when you’re going to take the integrator approach of taking multiple partners and trying to put them into a single system that needs to be supported for three to five years or longer.”
It depends on the client and their network says Wise. “I think you absolutely have to have a choice of vendors, we’re called systems integrators for a reason, we integrate products from different manufacturers. If it’s in a converged network then it is advantageous to have one vendor if you can, it’s just easier to manage a security update process than to have six or seven different vendors who don’t necessarily release firmware at the same time. If it’s not a fully converged network and you’ve got some equipment that’s doing a specific function, then you don’t really need to keep with one platform.”
There’s a comfortableness for end users in saying I’ve got this one brand that’s worked for me so far says Mulpeter, especially if that client is striving for global standardisation. Interoperability between brands becomes an issue when there are specific partnerships in place, says Hale, citing the one between Yamaha and Aver for microphone and DSP-based camera tracking as an example. “There are alternatives from other brands, and camera tracking can be configured in a control system, but interoperability is not guaranteed in the same way it would be when using a system from two partnered vendors. Ultimately though, all hardware needs to be interoperable with other brands, ‘but to what extent’ is the prime question to ask. In my opinion it is not necessary to maintain an end-to-end branded solution. That’s a sales driver, not a technical requirement in most cases.”
As Wise points out above, integrating different products is the core of what SIs have always done, but it’s still not simple he adds. “It’s not so much about whether they work together, it’s about how they work together. For example, you still couldn’t take an Extron NAV encoder and use it with a ZeeVee decoder, they’re both using AVoIP, they’re both using fundamentally similar protocols, but they’ve both got their own secret sauce on top of that which means they’re not interoperable, and that still permeates the entire AV industry. It’s not like the IT world where a printer is going to use the same protocol from HP or Xerox, it just doesn’t work like that. They’ve all got their own competing agendas.”
There are some technologies moving faster than others right now says Richards, such as the UC camera technology being used in meeting rooms. “Because of the speed of the innovation I think we’ll be tied to specific ecosystems which are embracing certain features faster. In order to move faster, you need partners or companies to make the decision to do it in a way that requires less compatibility between different manufacturers in order to differentiate themselves, and camera AI and camera tracking are good examples of it.”
As we seem to be moving towards a multi-brand, more interoperable AV world, manufacturers are becoming more pragmatic about this new philosophy, rather than trying
to sell integrators every single piece of the chain. “I think what integrators really want is a simple, reliable way to deliver a complete in-room solution - even if that means we’re not providing every component ourselves.
That might involve partnering with other brands to meet those needs,” says Mulpeter. You only have to look around the different stands at ISE for example, to see how many partnerships are being stuck between (non-competing) AV companies to ease integration issues for integrators (and ultimately to increase sales of those partnership authorised products).
Those reading this who have gone through the pain of firmware updates going wrong will be glad to know that the manufacturers are working hard to alleviate that particular stress point. “We have a full QA process for all our (firmware) releases, that cycle is incredibly important to Crestron. We test it, we set it up in our labs, we put it on as many devices as we can, that includes bench testing and spaces we have dedicated to testing this stuff, we have weeks-long test cycles on these things,” explains Mulpeter.
Kramer is tackling the problem by committing to industry standards such as HDCP 1.4, 2.2, 2.3. HDMI, DP ALT mode, etc says Hale. “If the standards for connectivity are maintained, firmware updates can make improvements to the software side of the device, including upgrades for HDCP versions. Firmware updates would never change existing functionality.
For instance, the external control command syntax, those must remain the same, because existing systems need to continue to work with their control systems. Only new commands might be added to expand functionality, but never to break interoperability. That being said, mistakes have been made in the past. The important thing there
is to be transparent with the customers and offer swift rectification of the errors introduced, in the form of another newer firmware.”
To really solve this issue it has to be a two-way partnership, full of dialogue between integrator and manufacturer, says Wise. “As a systems integrator we need to make sure our people have got the required knowledge, the experience to design, deploy, and commission correctly. If issues arise at that stage, we turn to the manufacturer for support, and they’re fully behind us, because they know we’ve followed the right process and exhausted all other options. In an ideal world the manufacturer would test all products with all other products, in every scenario before they sold them to us, but of course they can’t do that. I think we’re in a better position now than we were and I would just like to see more of a standard-based approach with more interop built in. That would make everyone’s lives a bit easier going forward."
images: shutterstock/RerF_Studio
shutterstock/Lea Rae