How to get hardware, software and even edge AI working in harmony

When thinking about edge orchestration and optimising all components, you must first define what you mean by “edge”, according to Nick Burling, chief product officer (CPO) at hybrid cloud storage-focused company Nasuni. Your “edge” can be a cloud region, or a central location where data must be accessible. Or is it your “edge” because compute is only available in that region? For Nasuni customers, “edge” can be a military installation or a North Sea exploration vessel scanning subsurface geology. The datasets requiring efficient, economical storage can be enormous LiDAR (light detection and ranging) scans of the seabed, says Burling. There must be a “gold” master copy of the data somewhere, whether you’re talking about the 80% of enterprise data that’s unstructured, or another data type. Data may need to be presented in different subsets or a different way per different edge locations, and be available for analysis, which means sufficient connectivity, availability, latency and tolerance of disruption. Containerisation can be used to provide modularity and redundancy for flexibility and future-proofing. “Caching is absolutely critical,” says Burling. “Looking through a hardware-centric lens, we separate compute orchestration from data orchestration to avoid big silos of copies of data.” For example, the system may need to present a specific required subset of data to an analyst to be worked on elsewhere without having to duplicate that data. And data orchestration done right can help avoid expensive physical storage or having to continually transmit data back and forth. In addition, object storage and peer-to-peer networking can work together to deliver centralised data management and protection while presenting wherever the edge might be, with data orchestration layered on top. “Ideally, independent of whatever local storage you have at those locations, as long as you can present the smallest amount of the working set of the data that’s needed, that’s where you get to an optimised workflow,” says Burling. Pizza delivery vehicle safety and monitoring Domino’s Pizza is using edge-based, artificial intelligence (AI)-powered monitoring provided by Netradyne, a company that specialises in AI and edge computing for fleet safety and management. Sreekanth Annapureddy, chief technology officer (CTO) at Netradyne, says Domino’s is using a camera-vision system to read vehicle information and watch the road during deliveries. Another optional camera looks into the cab. There is also support for two cameras facing sideways to capture side-on smashes. Other sensors include GPS and accelerometers. The setup uses AI to intelligently provide real-time information to the delivery driver, based on sensor data, to help improve safety. Netradyne’s proprietary system runs at the edge, but for AI inferencing, updates are distributed via cloud infrastructure to all the edge devices. For the video clips, only relevant bits are automatically uploaded, using a containerised stack, where they are analysed and redistributed. Users can then adjust downstream operational workflows. Small is not beautiful An edge device, such as a sub-100W device, uses a third of the power of a standard PCI Express graphics processing unit (GPU). Such a device needs to balance reasonable performance with thermal limitations, as Ed Plowman, CTO at Imagination Technologies, explains. “Traditional thinking, particularly around hardware compute for AI processing, is that you have a CPU [central processing unit], a GPU that draws pretty pictures and does some of the compute work, and then an NPU [neural processing unit], because NPUs are low power,” he says. “But it means programming three elements and getting them to cooperate all at once.” When looking at edge compute, it is necessary to take into account performance, throughput and capability for devices and CPU, GPU and NPU software stacks, as well as the maturity of the software. The experts Computer Weekly has spoken to urge IT decision-makers to compare their requirements to designs that work, and make use of programmable, easy-access, easily understood open source capabilities to avoid being locked into a technical architecture. “Stay flexible, so you can hopefully continue to develop what you do and choices continue to be fit for purpose,” Plowman urges. Joshua David, senior director of product management – edge business expansion at Red Hat, recommends that IT decision-makers frame their edge compute strategy around orchestration and how to manage multiple distributed, resource-limited sites without adding on extra cost, complexity or risk. Edge workloads run differently from datacentre workloads and require real-time control automation, or artificial intelligence (AI) inferencing that needs to run close to or on the edge device itself. Rather than focusing on moving compute closer to data, IT decision-makers should ensure their edge compute IT architecture is optimised in terms of infrastructure type, workload size, latency, bandwidth, reliability and security. Coordinate and centralise Site-level infrastructure that can coordinate these local operations typically provides centralised management with enterprise analytics or AI training and fleet-wide visibility. “For example, manufacturing may prioritise low-latency machine control. Utilities may require autonomous operations. Retailers may focus on store load experiences with centralised analytics,” David advises. “These are distinct. So start with the type of workloads, then those workloads’ characteristics, and then specific industry needs.” Another consideration is the choice of compatible enterprise operating system (OS) and bare-metal hardware that runs at the edge. For instance, edge-optimised, open source-derived approaches (such as MicroShift) can add flexibility and interoperability. Workloads may run as a virtual machine , a container, or need hardware acceleration for real-time capabilities. IT decision-makers should also ensure that observability – in terms of notifications and alerting – is robust and able to respond to relevant shift or drift, ideally via a central control plane observability tool. This leads on to reliability and ensuring business continuity at the edge. Mark Christie, field CTO at StorMagic, says customers such as retailers may generally require a two-node high-availability configuration. “What you’re seeking is an elasticity that smoothly transitions from ‘one to two to many’ as required,” he says. For example, US convenience store chain Sheetz uses a two-node system with 15 applications, from Windows operating systems to LTSC for credit, loyalty and point-of-sale applications. The storage “stretch” cluster can be shared out to as many compute nodes as it needs. Beyond observability, an edge control cloud provides centralised management for edge orchestration of fleets, estates and multiple edge device clusters. It tends to run as a serverless tenancy in the cloud. The orchestrator transmits information about the infrastructure to the cloud, which provides centralised management, and acts as a single source of the truth for edge compute IT infrastructure. Having this self-hosted helps tick data sovereignty and residency boxes for regulatory compliance. Read more about edge computing US smart city lights up solid-state LiDAR technology : Pioneer US smart city and associated 5G ecosystem collaborates with solid-state scanning light detection and tech provider to deploy latest range of sensors. Arm accelerates Edge AI with Ethos-U NPU and IoT reference design: NPU said to deliver four times performance uplift for high-performance edge AI applications such as factory automation and smart cameras through new IoT reference design platform to accelerate deployment of voice and vision systems. AI at the edge Adding intelligence is complex since it is necessary to ensure edge AI workloads run reliably across distributed devices with minimal data movement. Andreas Hellander, CEO and co-founder at edge AI-focused company Scaleout Systems, considers edge compute requirements through the lens of federated machine learning (ML) . “There are so many aspects of or challenges related to deploying, updating and continuously improving AI systems on the edge,” he says. “And it’s a little bit fluid. It’s not so easy to be completely strict about where one layer in this hierarchy starts, and where the other one stops.” Beyond the IT infrastructure hosted in a hyperscale cloud, IT decision-makers need to look at where near-edge resources are hosted. They may be hosted in private but not centralised cloud infrastructure. Then there is the far edge, with edge workstations, gateway nodes and devices like personal computers, smartphones and drones. What all of this means is more environments need supporting and the further out to the edge AI workloads need to run. “Everything is different: hardware, OSes, connectivity,” says Hellander. While the ecosystem to support heterogeneity at the edge has matured in the past five years, Hellander warns that a setup that works when it is first deployed may use integration that later becomes obsolete or cannot easily be updated. “When dealing with security-sensitive or defensive cases, you want to control the ML lifecycle. If you don’t, you’re really taking on security risks and technical debt,” he says. “And it’s always important to ask, ‘Where does my data sit and end up, and what flexibility do I have to land it on a far or near edge node or central cloud?’ Also, ‘What about resilience? What happens if the network is cut?’” What seems to be apparent from Computer Weekly’s conversations with industry experts is that edge computing complicates centralised IT management and presents observability and business continuity problems. AI complicates this further, especially as edge devices need to balance performance, power consumption, access to local data and potentially consolidated datasets with having enough compute to run inference workloads.