Voices of Video
Explore the inner workings of video technology with Voices of Video: Inside the Tech. This podcast gathers industry experts and innovators to examine every facet of video technology, from decoding and encoding processes to the latest advancements in hardware versus software processing and codecs. Alongside these technical insights, we dive into practical techniques, emerging trends, and industry-shaping facts that define the future of video.
Ideal for engineers, developers, and tech enthusiasts, each episode offers hands-on advice and the in-depth knowledge you need to excel in today’s fast-evolving video landscape. Join us to master the tools, technologies, and trends driving the future of digital video.
Voices of Video
CDN in Your Pocket: Nokia's Secret Weapon Against Buffering
Imagine watching a live sports event where your stream is actually faster than broadcast television. This isn't science fiction. It's the reality of Nokia's groundbreaking approach to zero-latency streaming, as revealed by Senior Product Manager Keith Chow in this illuminating deep dive.
When live sporting events drive 50-70% viewer concurrency (reaching 100% during World Cup matches), traditional streaming technologies buckle under the pressure. Keith explains how Nokia's solution handles these extreme demands through a sophisticated combination of UDP protocols, multicast distribution, and what he calls "CDN in your pocket" - a tiny 200-kilobyte client library that transforms any device into a low-latency streaming machine.
The technical achievements are remarkable: channel serving times as low as 19 milliseconds (compared to the industry standard 2-14 seconds), perfect synchronization across all viewers, and an astonishing resilience that maintains watchable streams even with 40% random packet loss. And these are not theoretical metrics! They're real-world results from serving millions of concurrent viewers.
What sets this approach apart is its pragmatic implementation. Many network providers already have Nokia's service routers installed at the network edge, requiring only software activation to unlock these capabilities. The solution builds on established standards like RTP, IGMP, and ST2022-7 - already used by hundreds of companies - while adding proprietary optimizations that dramatically outperform WebRTC and emerging technologies like MOQ.
Whether you're a streaming platform looking to deliver sports betting experiences that demand sub-second latency, a network operator trying to manage massive concurrent viewership, or a video engineer curious about next-generation distribution techniques, this episode offers rare insights into technology that's already transforming how live content reaches viewers around the world.
Stay tuned for more in-depth insights on video technology, trends, and practical applications. Subscribe to Voices of Video: Inside the Tech for exclusive, hands-on knowledge from the experts. For more resources, visit Voices of Video.
voices of video. Voices of video.
Mark Donnigan:The voices of video voices of video well, welcome to this very special edition of voices of. So you probably don't know it, but this actually is a different format here. Normally we go live on LinkedIn and other platforms. Today we are recording, and that has a lot of advantages. One is that we can bring in experts from literally around the world and we don't have to worry about navigating time zones, and it also gives us some flexibility to create some new types of content, do some screen sharing, et cetera. So I hope you like this new format and, if you do, make sure you let us know. But this edition of Voices of Video. I am talking to Keith Chow from Nokia. So, first of all, welcome, keith, to Voices of Video.
Keith Chow:Thank you for inviting me here.
Mark Donnigan:Yeah, yeah, yeah. Well, I know that you are quite prolific, meaning that you speak. You travel a lot, you're out there in the industry. I think a lot of our listeners may even know who you are or have heard your talks, but why don't you give an overview of what you do at Nokia and what you're focused on these days?
Keith Chow:Yep, I'm Keith Trout. I work in Nokia IP routing, ip network and network infrastructure group.
Keith Chow:My main job is the senior product manager and I'm also doing some other support things like piece sales and also post-sales support and also funny things with the Bell Lab team, because I work very closely with the Bell Lab guys and they are the blue sky thinker yes, they are the product I'm working with, I look after, is mainly for IP video in Nokia and it is Bell Labs' innovation project many years ago and they started with something very, very special and very difficult to implement in life in product, and we transferred the blue sky thinking into the product reality and also making quite a good impact in the industry.
Mark Donnigan:Amazing. Yeah, very cool. Well, you recently gave a presentation that we're really going to use as the basis of our conversation today and I can't wait to dive into that. You know, now even Netflix has live sports right. So it's no secret that video platforms have been transformed. They are now live streaming platforms, and that brings in the question of latency. There's a lot of talk about needing to get to broadcast latency, and then there's some very interesting use cases that I know you'll be talking to us later about. You know with, like, sports betting, and you know other interactive type type. You know uses where really getting into you know second or even sub-second latencies is critical. So I'm wondering if maybe you can give us your perspective around the importance of latency, why streaming services should care about that.
Keith Chow:streaming services should care about that. Okay, streaming service. Very important is that they need to separate the different, understand the difference between live streaming and from video on demand. The difference is the nature of view time Okay, nature, and also view time and spot or real-time live event. They have a very, very high concurrency. Typically, we measure in the network is between 50% to 70%. The worst case was 100% concurrency with one of our customers in the World Cup Okay, so you understand that in the World Cup, everyone will watch the football and no one exception, and so it's very high demanding. And also they're demanding a minimum latency and for the IP broadcasting, the platform will adapt that with LTP base or similar to UDP base. For example, in Amazon they bought the company called Psy. They're using Psy technology, it's a UDP base and Google also using Quick the UDP base and WebRTC is UDP base. Okay, so they're all.
Keith Chow:One single trend is that they're all going to UDP basis of a solution to deliver the low latency and scalable stream. For example, to scale up, we might need to use the technology called multicast and a lot of people may scale off the multicast because they have no knowledge how it works and how the ISP will react when they ask them for multicast-enabled network. Actually, it's very simple. One of our customers they do not have the network and they need to deliver multi-cast streaming live event 24-7. And they managed to negotiate with seven different ISPs in the country even more. And so it's not that difficult. When you tell them that you're going to have a few telebit going through the network, they will think about before they are going to say no, because if you're capable to deliver telebit concurrent stream into their network, they will think first Say wow, okay, what am I going to do? I'm going to charge them a lot of money to do it, or let them just pass through and they route through another interconnection service provider and then it puts up the cost for them significantly because they can route through someone else and not direct it into the peering. So they managed to agree sort of this sort this agreement to make sure that they can stream multicast. And where they cannot stream multicast, they use a router unicast based on RTP protocol, so they can.
Keith Chow:Also. We are using MPEG transport stream so it can be easily synchronized in all the player, because MPEG transport stream, so it can be easily synchronized in all the player, because MPEG transport stream is a presentation timestamp and also have the PCR synchronize all the device in the network payback. So they are all streaming this, all the device, they are all synchronized, the payback time and real-time interaction. And also we introduced something called fast channel change which is unicast-based to support the fast acquisition of the multicast. So we can optimize the impact transport stream.
Keith Chow:For example, they can use your chipset for the VPU to reduce the encoder buffer size. That is the critical path of the impact transport stream in the live broadcasting Because if they can reduce the impact transport stream the encoder buffer they can reduce the mp transport stream, the encoder buffer. They can reduce the latency because the device will need to buffer the encoder buffer before they can start. So we can also optimize that part during the fast channel change so we can start the decoding within 90 milliseconds or two, whatever, when the iFrame and the audio packet will arrive at the decoder side.
Mark Donnigan:Yeah, amazing. Now this is Nokia's proprietary solution. But why don't you go into a bit more detail about explaining the components of it? Because I think in the industry and I'm just making a general observation there's a lot of claims. You walk around like NAB or IBC and it seems like every vendor has their ultra low latency solution, their low latency solution, and you know whatever tag they put on it, and often you look under the hood and it's WebRTC, or maybe they have adopted some UDP protocols and maybe they've done a little bit of shall we call it customization, but at the end of the day, it's really just very standard of the mill off the shelf and often it doesn't really work that well. Nokia has taken a different approach, still using standard technologies. So what? Yeah, why don't you explain that?
Keith Chow:Okay, the standard protocol we use must be standard. Okay, whatever protocol we're going to deploy in the company's rule, it must be standard. It must be part of the IETF. We must be able to name the IFC number In the share screen. I'm sharing now is the standard which we are using. It's the DVB standard and the protocol independent multicast, which is used for distribution from the video head end to the edge. So our IPMPS network, so our service router, does the trick and delivers all the way to the edge. Also, during the distribution in the core network.
Keith Chow:We are using also the perfect stream, which is a multicast stream. It's not multicast stream, it's a multiprotocol label switching and it's independent. It's the PIM To the edge using the ST2022-7, there are approximately 248 companies at the moment using it. Okay, if you go to risk for a lion and you will find all the 248 company. Okay, interesting, new, there are lots of people already using it. Okay, yeah, a lion, and you will find all the 248 company, it's not new, there are lots of people already using it. Okay, and then at the edge, what we do is that we using, if the device can and the network segment can support multicast, we use IGMP V2 or V3, and we recommend the customer to use a V3 so it has more security built in and we do not have the problem of duplicating multicast IP address.
Keith Chow:If the network cannot support multicast, we will use a router unicast. So the device will send the request of RTCP request of the specific content or the channel and according to the standard of the RTP standard and subsequent is 3550. So it can send the video from the video server, pass through all the NAT address to the device Because the device open the NAT port in the network. So it go through all the transverse, all the home gateway, go to the carry gray NAT and then go to the server IP address.
Keith Chow:Typically our service router is sitting first. Hop in the network. For example, when you get into the internet, the first element will be our service router is sitting first. Hop in the network. For example, when you get into the internet, the first element will be our service router, the BNG, and subsequently we are sitting right at the edge. So it's our advantage. For example, whole entire country in a few countries they are all using our service router at the edge. So we are already in the right place at the right moment. All they need to do is just enable the Surface.
Mark Donnigan:So if that yeah, sorry for interrupting, but if that physical device, that router, is already in place, then it's really a matter of just enabling the software right To be able to. So what you're saying is is that there are some customers who may have access to your technology but don't even know it, right?
Keith Chow:Or they are having different villages. They believe that TCP is the only way. Tcp yep they do not want to enable it.
Mark Donnigan:Yeah, interesting.
Keith Chow:Also that is a conflict of the decision-making.
Mark Donnigan:The decision-making guy.
Keith Chow:They say everyone using TCP, we're going to follow the trend, but they are not aware of that they're burning money. They already have everything in place and all they need is just fish on the surfaces.
Mark Donnigan:Interesting, interesting Maybe. Yeah, keith, before you continue, I don't want to lose this issue of TCP versus UDP, versus UDP. Can you explain? Because I think that there are certain video engineers who are not working at the network protocol level and I'm not suggesting they don't understand or know what TCP is or what UDP is, but I wouldn't assume that everybody listening really understands. So can you give a one-minute crash course on TCP protocol versus UDP?
Keith Chow:Okay, udp is unreliable transmission, but we are using RTP over UDP. Subsequently, we have the sequence number. We mark each packet the sequence. When the receiver is receiving this packet. They look at the sequence number. If there's a gap, they know that it's the packet loss. They can ask for the retransmission using RTCP protocol, which is a standard. It's been around since 2004.
Keith Chow:Many people are using it, for example, some video conference application. They're already using it for a long, long time and so the LTCP protocol packet can be delivered via multicast. You can just map the MPEG transport stream into the RTP protocol so you can just put into the multicast buffer and then you will send it out automatically without any problem. So on the network side, our surface router will pick up the packet and say that ah, I need to replicate this packet to deliver this IP address. This is the multicast group. These are the people who would like to receive the multicast packet.
Keith Chow:So first thing is that our fast-transmission server is sitting at the edge at the BNG level, so every one of the service routers will receive the multicast one copy only, so you do not need to send multiple times to waste the network resource and also the power supply to deliver each packet multiple times to the edge. So this way we can send them multicast from UK or from one of our customers. They have multicast video head end in Germany and they send to another country across the whole entire Europe, just using the GRE general encapsulation protocol across the whole entire Europe, just using the GRE General Encryption Protocol in the network. That is what you use for all other communication purposes. So you use GRE, you have a private pipe and then you send your multicards into the edge as you wish. You cannot loss the packet because we use the ST2022-7 to have a multiple path. So in case of one path we loss and the other path will kick over. It's not only made by us, made by over 200 companies, so you can pick whatever you want.
Keith Chow:And of course we will recommend the customer to use ours because we are the best, as usual, but typically the customer just pick ours because they already have the network equipment in place. All they need to do is just enable the protocol, that's all.
Mark Donnigan:Yeah, amazing, okay, cool. Well, let's talk about CDN for live over any access. So you've got the architecture diagram up here. I see, yeah.
Keith Chow:Yep, let me explain that. Ok, so overall architecture very, very simple. It works with both multicast and routed unicast over any access network. For example, from the video head-end. We take the live content, either encrypt it or clear. If you encrypt it, we just pass through and optimize it. If you unencryptpted, we encrypt the content with 100% payload encryption, so we just send it up through the network. So the rest of it will be handled by the CAS vendor, for example Verimetrics, ligra or whoever, and also Google. Now they have the google cast and it's free of charge and but they do not provide you the support yeah and you need to look after yourself, and so we optimize the video in a single location or multiple location.
Keith Chow:Okay. For example, if you have a multiple location, you have the geo redundancy. If you do not have the multiple um location, you have a multiple location you have the geo-redundancy. If you do not have the multiple location, you have a one-plus-one redundancy, so you will never lose your feed. So you have multiple feed and multiple appearing points for the video head-end. Subsequently, you can protect your video and then you can go through all the cloud in the network. So you can be anyone's surface router it's the core network router and then reach the edge, and so you can either use our edge router or you can use someone else's edge router. But if you use someone else's edge router, they may not support what we can do, and so you may need to put one of our surface router. It's one RU, no three RU. Sorry, the bigger one, it's a three RU. It's one three terabit egress capability, so it's very high capacity.
Mark Donnigan:Yeah.
Keith Chow:Three terabit. Which PC server can do that in three RU? I don't think anyone. Yeah, because it's a special chipset we make, like the one you guys have for the video processor unit, so you need to use the right tool for the job.
Mark Donnigan:That's right. Yeah, how many users can that router handle roughly? I mean, I know you said it's 3 terabyte capacity but how many sessions?
Keith Chow:how many sessions is depends on the configuration. The minimum configuration is uh 256k per card, and then you can scale up to uh eight million session. So one single service router can serve 8 million subscribers. So, I am not sure. For example, take the use case in UK 60 million population, let's say half 50%, so it's only four or five service router can serve the whole entire country.
Mark Donnigan:Interesting. Wow, yeah, now the alternate routes um would, would those, would those come through that same router?
Keith Chow:they, they would not right, they would come through physically different yeah, we use any cars, so the least um we uh device to the surface router you will have the lower cost of the route and subsequently we will get the request, receiving the RTCP request and then you'll send the video stream to the video stream we start with always unicast. When you catch up the multicast live you will try to fall back to multicast and then you will release the resource from the service router itself and then just go to any sort of switch or router can provide the multicast. Okay, interesting For example in the mobile network case, ran.
Keith Chow:For those who are using the Nokia mobile network, they can use the multicast, but on the mobile signal side, on the radio side, it's always unicast because due to the layer 2, they cannot support a real multicast. It's a single point-to-point and for the uh fiber or a twist pair, we will just let them. Uh, if the multicast enable, we can just send the multicast to all the way to the home. Doesn't matter. Um, your, your or whatever you you can support the network or the device can support multicast, you will join the multicast. If not, the device will tell the service router say that hey, I cannot receive multicast. Can you please continue sending me the unicast?
Keith Chow:So in each service router we can scale up from one terabit to three terabit as a minimum and the biggest service router is the 216 terabit to three terabit as a minimum, and the biggest surface router is the 216 terabits, so I don't think we need more than two in any country at the moment so basically selling.
Keith Chow:The smallest one is a three terabit chassis and so we can provide the best surface and the cost and, for example, some uh customer using cmts cable guys and they can do the same thing because we can provide this Unicast to start up the streaming tune in the channel and then, once reaching the mode, the live, catching up the live because we buffer in the device up to the encoder buffer and then it will finish because it say okay, I finished, I reached the live because I need to join the multicast. So the CMTS is a little bit tricky on the cable network. It will need to go all the way to the back to the video head end to send the service group.
Mark Donnigan:That's right.
Keith Chow:And it will take between 7 seconds to 15 seconds, that's what we measure and we can use the extended unicast to keep the video going until the multicast arrives at the cable CMTS node, and so you can join the multicast and that's it, and then we hand over. So it's very flexible. It can be can support either multicast enabled network or unicast only network, for example some of the surface router cannot enable the multicast at the BNG layer and subsequently they need to keep going with the unicast until they've been replaced.
Mark Donnigan:Okay, yeah, amazing how many solutions are out there that can seamlessly switch between unicast and multicast.
Keith Chow:Because that, I think that's at least two.
Mark Donnigan:Okay.
Keith Chow:Other guys doing that. Okay, I know that. I know all One is called. Once upon a time they called Microsoft.
Mark Donnigan:Yeah.
Keith Chow:And now it's a media, but I'm not sure if they're still using that and the other guy is I know. You know who. They used to be called Arroyo and now it's bought by some other company, another company, I'm not sure. To my understanding, they stopped because they took the decision that TCP is the best and they all disbanded all the development and then all the engineers who have the knowledge. They're going to disappear.
Mark Donnigan:Yeah, yeah, exactly that's what happens often in these situations.
Keith Chow:And the other guy is called Sai.
Mark Donnigan:Yes At Amazon. Yeah, Amazon acquired them.
Keith Chow:I have not yet been able to test the solution. Yet I'm not able to find any commercial deployment, so if the guys from Amazon are listening, please let me know where I can find them and I can do some testing. Okay, so I would like to take the take the wide shot capture, because I've been doing the wide shot capture of the Pine video for the live broadcasting football Premier League in UK and they're using TCP only at the moment and I still not able to find the site protocol in deployment. I've been told that they're in Sweden, so I will probably need to go up there and then sit in someone's home and try to take a look what they do.
Mark Donnigan:Yeah, yeah, yeah, exactly, amazing, yeah, very cool. Well, is there anything else architecturally, that you want to highlight, because I would like to move then into practical applications. I know that you're actually in production and you've got some real-world data to share and real-world case studies.
Keith Chow:Yeah, we've been in production for quite a while, production for quite a while and we've been very quiet because we bound to some agreement not to publicize that.
Mark Donnigan:That happens to all of us. It's quite frustrating Doing all these really great projects and these great events and we can't tell anybody about them.
Keith Chow:Now we can't tell anybody about them. Now we can, because that restrictions and a few years ago, and so we can start to we, we start our Tell the secrets, tell the secrets. Actually it's not really a secret, because all standard protocol.
Mark Donnigan:Yeah, exactly yeah. So what our?
Keith Chow:other architecture wise Actually, it's not real secret because it's all standard protocol. Yeah, exactly, yeah, yeah, yeah. So, architecture-wise, we have what they call a CDN in your pocket. Okay, that is very eye-catching. That's catchy.
Mark Donnigan:Yeah, yeah, yeah.
Keith Chow:And I didn't even think of it when I did it, because I find the CDN in my pocket in my mobile phone. So I say, what the hell is this? It's the CDN in the pocket.
Keith Chow:So that's why it's called CDN in your pocket. So what it does is that we put our client library agents in the way with our segmenter. Actually, we already done the segmenting in the video head end while we are doing the encryption. So we can encrypt for CAS or encrypt for DRM, because there are different encryption schemes. So we already made the decision to say that we now encrypt that for DRM. So we tag the segment in the video head end, all the heavy lifting already done in the video head end.
Keith Chow:So when the client receives that, you know that this is for the DRM and we have also the segmenting tag in it. So we know which segment it goes to. And so so we know which segment it goes to. And so of course there's some encryption and also the encoding restriction for the DRM. So we need to start with the iframe and with the glob, and so subsequently we will have the closed glob. Otherwise the segment will start with someone else video or audio. It will not look good. So what we do is that after we segment that, we transform it into CMAS format, which will be common to all the different players in the HTTP world, and then we can use an HTTP server or MOQ server. I would not recommend to use MOQ server because it's not stable at the moment.
Mark Donnigan:Well, we're going to talk about MOQ, but let's put a pin in that, yeah.
Keith Chow:Okay. So you can just put this really nice agent okay in your pocket in your mobile phone. Or you can put it in your pocket in your mobile phone, or you can put it into your PC, laptop or in the Home Gateway. So any Home Gateway which supports Wi-Fi 7 can support OCI. It's an open container initiative. So we can put this whole thing into the Home Gateway or into the mac. Okay, it's a mobile edge compute which um and then you can do this, or you can put it on the your raspbian pi, the cheapest computer in the world amazing yeah or you can put it into the Jackson, the latest toy from our friends in Taiwan, so you can put that in there.
Keith Chow:So it just works Also this way, because you are inside your own subnet. It's a private network. No one can do the latching CDN latching will go away unless you want to hack yourself. If you hack yourself, everyone will know that. So this way we resolve multiple problems in one go Backward compatibility. We solve the CDN latching secure so we can restrict the number of connections. For example, you're only allowed to have four connections. After you use all of them, it's your problem. You finish. You can stop all the hackers. You're at a home network inside your mobile device. Who can hack your mobile device? No one, unless you're going to do it yourself, that is not okay.
Keith Chow:You can just take the video camera and then in front of the TV, and then you can.
Mark Donnigan:Yeah, exactly, there's always a way.
Keith Chow:There's always a way to hack it. So this way we resolve the problem of drifting. So you can still use the good old player HTTP player, but no more drifting, because the probability of packet loss inside your mobile phone is literally zero. The run chip delays is zero. So even you put it on your home gateway, the probability of packet loss is still high, but you will not as much as what you're in the open internet. You know that it's getting bad and then you say, okay, I need to switch off the microwave or stop the dishwasher to watch the TV. Actually, my dishwasher creates a lot of noise, so yeah, yeah, that's it really.
Keith Chow:That's interesting, a lot of rf no, no, yeah, yeah on the mobile, on my wi-fi your dishwasher dishwasher is creating a lot of noise.
Mark Donnigan:What kind of dishwasher do you have?
Keith Chow:I have a dishwasher 15, 16 years old, oh, okay you know, because I'm not going to change it?
Mark Donnigan:yeah, yeah, no, no. The reason I ask is you know there's all these new smart, uh, home appliances and you know they all have wi-fi transceivers in them and you know they have, you know nfc and and all of this bluetooth all built in.
Keith Chow:So I thought no, that's actually a really good point. My washing machine does that, okay, and it will send me a text message that I finished washing and now you can take them out from the washing machine. But this was 16 years old. So it's a good old analog one Interesting.
Mark Donnigan:But it's still noisy on the network. I mean RF. Okay, Interesting.
Keith Chow:So this way we resolve the problem of compatibility with Haas player. Okay, http adaptive streaming player, so that we solve multiple problem in one go.
Mark Donnigan:Very, very interesting, wow, okay, I know that you have done some large sporting events, so maybe you can talk about solution of channel serving.
Keith Chow:And this measurement comes from our customers and also from our other vendors, and so we can manage to get down to the channel serving starting up time. The lowest one is 19 milliseconds milliseconds. That was measured in our customers lab. Literally the surface looks around 50 meters from the from where we are doing the testing. So only one switch, a two switch in between. So that was extremely fast and the maximum is typically Typically it's 350 millisecond to 500 millisecond and we did the measurement in Sydney from the server. It was sitting somewhere in North and Europe and in Sydney, between North and Europe, it's around 420 millisecond. So they're trying to change times around 500 and something 500 milliseconds because we need to take into consideration of the round-trip delay plus the starting up time.
Mark Donnigan:Yeah.
Keith Chow:So we compare with other solution is that typically from 2 seconds to 4 seconds up to around 14 seconds. That is quite high. It will depend If you have a lot more packet loss on the way. Tcp, we will start up time with 200 seconds or even 300 seconds. So one five minute that is what we're measuring, but typically it should arrive in 10 to 14 seconds.
Mark Donnigan:Yeah, yeah.
Keith Chow:And compared with the other technology, latency. We have made the assumption that the camera plus the encoder, everything is around five seconds delayed. And then, compared with our solution, we are more or less the same as the digital terrestrial TV. In the US they call OTA over the air and also DTH the satellite, because satellite has one disadvantage you need to go up from the teleport to the satellite and come back down, so that will cost between 800 millisecond to 1.2 second or even 2 second more. It depends on where you have the satellite. And we compare with the traditional IPTV, the IPTV, without the fast channel change. We need to wait for the iframe and subsequently they typically have a drop size of one to two seconds. So they will increase approximately four seconds um of a latency due to the um. Uh, wait for the igmp, john, okay, and also, once it's on and it will also introduce the accumulated latency.
Keith Chow:All the people they're claiming, claiming that they can do ultra-low latency, they're using WebRTC. That is more or less similar to ours. Because of WebRTC, the underlying protocol is RTP Okay, so they can achieve that, no problem. But WebRTC they use a full error correction and that is the killer point because they cannot support a high degree of packet loss, for example, random packet loss of 15% in a mobile network or in a Wi-Fi network with a lot of noise, you can kill the connection.
Mark Donnigan:Okay.
Keith Chow:WebRTC will be dead and also if, using datagram of a retransmission, a lot of people have a difficulty to get it work, so they have a difficulty to get it work. So WebRTC typically can support up to 3% to 5% of random packet loss. After that they would be just unwatchable would be just unwatchable.
Mark Donnigan:Yeah, yeah, yeah. I think a lot of people who have attempted WebRTC at scale and this is why the vendors who have really built, I would say, kind of real solutions with WebRTC, they have done the engineering work to actually build that reliability in the problem is, is that a lot of people who haven't actually tried it at real scale, you know they just, you know a handful of streams, you know one to a handful of viewers. They go this is great, this works, this is amazing, it's out of the box. It's an out of the box, free, low latency solution. I mean, I'm paraphrasing, but that's usually the view. As soon as you start to scale it, it gets unreliable really fast. You know, and and and those are the. I think that's what's super important for the listeners and, hopefully, the viewers.
Mark Donnigan:If you're, if you're able to watch the video here, is that this slide that Keith is showing you know is is really critical because there's the reliability vector. You know that um cannot be lost, is it? It's one thing to to to stream one stream, you know, a single stream to a handful, or a single digit hundreds, or maybe even single digit thousands of viewers. That's one thing. To do that at tens of thousands, hundreds and millions. Forget about it with a lot of the solutions that are out there now.
Keith Chow:Yep, we have a million subscriber, concurrent viewer all the time.
Mark Donnigan:Okay.
Keith Chow:It's daily.
Keith Chow:And concurrent viewer all the time. It's daily and we look at the customer's quality log. They have a million subscribers on the network all the time. They watch, just like the forecast TV, over the public internet or managed network. Only a few customers. They're using open internet because they do not have the network. But most of our customers they are ISP, they have their own managed network, so they do not care. Because of multicast it can be one to a few millions, doesn't matter, it will not cost us anything. The only thing cost is the retransmission and also the retransmission. Our service router capability can handle millions of sessions, so it does not matter, because sometimes we say that we over engineering it and in some case we need to do that because when the network is really bad they lost a lot, because in one of our customers we've been measuring it's 40% of random picket loss 40%.
Keith Chow:Wow, yes, that is in the fixed wireless access network. Yeah, because the attenuation of the radio signal down to minus 60, minus 80%. Yeah, yeah, minus 80 dB, so literally they have no signal at all.
Mark Donnigan:No signal.
Keith Chow:The bandwidth is extremely low, so we need to scale down the bit rate when the buffer is empty okay. So, we use the standard adaptive streaming kit technology. When the buffer is empty, we just stop the bit rate okay.
Mark Donnigan:Yeah, yeah, amazing, amazing, yeah, that's incredible. Um, okay, uh, wow, this is. This is really interesting, keith, and you know, I hope, I hope the listeners are enjoying this. Um, one of the one of the things we're really trying to do here at Voices of Video is to go a bit deeper. I appreciate higher level discussions and know into protocols and standards and technologies and you know real world implementations, so I really appreciate you know what you're talking about here, keith, thank you, you know, thank you.
Mark Donnigan:Maybe what we should do is, you know, let's talk about the practical implementation. So we've touched on that and you know I made the point or I asked the question that also made the point that you know there's probably some platforms that have access to this solution but either don't know it or, like you say, have chosen to support. You know a different set of technologies, but do you have any case studies that you know you could share with us, maybe some examples of where you have deployed? How long does it take? You know what are some of the complexities, how difficult is it?
Mark Donnigan:You know, just talk, just talk to us about that because, as you well know, because you're in infrastructure, one of the biggest challenges working in video, is it? You know? There's always a better codec, there's always a better standard, there's always and I'm using air quotes you know a better way to do something, but the devil you know, the famous saying the devil's in the details. I like to say the devil's in the implementation. So if it's either too hard or too difficult or too expensive, you know, then it doesn't matter how good it is. So talk to us about, you know, the implementation.
Keith Chow:The implementation is very simple. We have only three major elements in the network we need to deploy. One element in this architecture diagram is the video optimization, the MPEG transport stream. We need to optimize the packet so we group all the video by frame type and also we try to reduce the encoder buffer size so we try to make the audio alignment with the video and a lot of encoder. What they've done is that they are not using a proper chipset like yours.
Keith Chow:They're using some standard PC software hardware and then the audio packet, always in the end of the gob or even beyond the gob yeah, that's right so that will cause a hell of a lot of trouble to wait for the audio packet to arrive to start the decoding and if you use um some very nice hardware encoder and like a netted VPU, yeah of course, that is what I've been talking with you guys. When I first met Alex and he sent me the VPU, I said that is the chip. We need to tell everyone to use it because you can encode it compress the video very fast. Also, you can optimize the video and audio alignment.
Mark Donnigan:That's right.
Keith Chow:Because all other guys cannot do it, because they need to wait for the complexity of the video compression. They talk about all the AI. That is rubbish, because the basic, fundamental thing is that you need to compress as fast as possible as a minimum bandwidth as possible, but you need to align the audio with the video. If you put the audio packet a few seconds behind the video, you will need to wait for that, otherwise, you have the same problem.
Mark Donnigan:Yeah, that's right.
Keith Chow:So when Alex told me that, I said okay, alex, can I have one of those?
Mark Donnigan:That's right. Well, we're excited. We're excited to support you. I know that you guys are going to start a new round of evaluation and testing and again, on Voices of Video, we really steer clear of promoting NetEnt products. But I do have to give a little bit of a plug here, because, after several years of talking to a lot of network providers, owners, the hyperscalers and working with them, everybody was interested from day one, but it took time to actually for the hardware to make it into the network. The good news is that we're now in some very large, very significant networks and so now you don't have to be a very large VOD platform, svod or OTT platform to use the products, which is great. In combination with what you guys have built, then this is going to be really powerful for the industry.
Keith Chow:So so the second element in the network would be our surface router.
Mark Donnigan:Yes.
Keith Chow:Or you can, either at the video head end, so we can send out the packet, route the packet to the edge, or we use some other, someone else surface router. That's why I normally do not draw the diagram of our surface router in the video head-end, because otherwise it will say that only works with your surface router proprietary. No, you can use anyone's surface router to do the job. And then the key element is that at the edge you need to enable certain features and also some additional card on top of it to do the job, Because you need to convert the multicards to unicards to do the packet forwarding and also you need to do the re-transmission, handling, all the re-transmission. That's a lot of compute capability required. So we typically use a secondary card to offload the loading from the surface router.
Keith Chow:But all the routing capability of the security all relied on the video service on the surface router. So it will send the ltcp request on the surface router by the client device. The client device is really really tiny. This um is one 200 kilobyte only. Okay, um, yeah, I was gonna ask.
Mark Donnigan:I was gonna ask how big it is. So it's 200 kilobytes, that's it yeah, 200 kilobytes. Wow, okay, yeah, that's tiny.
Keith Chow:It's tiny. The fastest guys ever integrate our software library into the device is one hour, but I would not recommend to anyone to do that, because that is the expert level, extreme expert level. Okay, don't try it.
Mark Donnigan:Don't try it at home. Don't try it at home.
Keith Chow:We have seen that typically from graduate engineer, they pick up the documentation, they pick up the software library and use our pseudocode and then they do the integration.
Keith Chow:It's typical um between one to two weeks, because it takes a week to read the documentation yeah sure and then ask all the questions and we typically know that which question they're going to ask, because we test those guys. If they have read the documentation, okay. If they do not read the documentation, they will not ask the question. If they ask the question, we prove that that guy read the documentation.
Keith Chow:Yeah, yeah so we know that when we go to meet those people, or on the meeting or night, or in person, we ask them so do you have this question to ask? And they say if no, I say okay, let's forget it, let's go back and we're going to read the documentation before we can go ahead.
Mark Donnigan:Yeah, yeah.
Keith Chow:And so afterward integrate with this into the set-top-up or mobile device, because we can integrate that into any player as an agent. Okay, so we do not use an app anymore. Integrate that into any player as an agent. Okay, so we do not use an app anymore, we use an agent and it's easier for anyone to do the integration job, and so they can do the integration with the app or whatever, or agent in the container, and so we can deploy it in any operating system. So we do not restrict it to Android, linux or iOS. We just build one single library for every single one, so that will make life easy for everyone.
Mark Donnigan:Amazing.
Keith Chow:So these are only three elements plus the manage the platform. So that is only need to use once or twice or doing the monitoring, or you can build your own OSS to monitoring the system or the network element. So there's only three major elements. So it's very simple. In the practical term most of the people can put an element.
Keith Chow:With our VWAPR it's a VBO appliance that is an optimizer, video broadcast optimizer. We optimize the video for IP delivery. So we ensure that the IP delivery will be perfect. So we need to have proper MPEG transport format over RTP and then we will send it out to the edge and then the end device will play back as whatever they need to with any sort of encryption. It's agnostic to us, the encryption, because we leave the pass header in clear. So we know that this is iFrame, bframe audio. This is new packet. So we group all the TS packet into one single RTP. So in the future in the CDN in your pocket, so we can use a minimum size of the trunk as a one RTP. So that means a 188 byte of the TS multiplied by seven. So or even less, depends on if you have a padding or not, so you can have the minimum size of the trunk so you can deliver to the, the player, and that's it, finish. It's very simple.
Mark Donnigan:Amazing, amazing.
Keith Chow:It's very, very reliable and we've been doing this of um uh, this of uh testing with one of our customers and in the poc just before the pandemic and during the pandemic we shut down everything because, uh, no one, no one is allowed to go to office and no one can go around. So we just wait until the pandemic finishes and also the patent grant for the trunking and the segmenting. So we have a few patents on that, so it reduces the life latency during the trunk segmenting and for the trunk segmenting and for the trunk as well. So delivery, so we can make sure that the delivery to the house player is perfect, smooth, no more additional latency. Otherwise you will need to wait for the whole entire trunk segment before you can deliver. Or you can use the trunk delivery but you still have the bike range to handle. You still have the latency, but the way we do it is completely eliminate the latency on the segmenting.
Mark Donnigan:Yeah, yeah, amazing. So, um, keith, you have given us a masterclass. Uh, this is uh way more than I even anticipated. So thank you, um I, I. I want to wrap it up here. There's a lot of talk in the industry about media over quick, you know uh mock. Over QUIC, you know mock. How does that fit? And, you know, is it viable in your view? I know you're going to have very clear perspectives. Some will say you have a biased perspective. That's okay, but I'm very interested in your response and your thoughts around mock.
Keith Chow:MOQ is based on the QUIC protocol. Moq is not a protocol, okay, guys.
Mark Donnigan:And we need to emphasize one thing.
Keith Chow:MOQ is not yet a protocol. Until someone makes a protocol. Yeah, that's right. So it's just a quick protocol of one simple defect. Okay, if you're using timeout for the retransmission, you have a very huge latency and the efficiency of the retransmission will be extremely poor. It's not the problem of that you can deliver fast enough or not. It's how reliable you're going to be delivering those live stream.
Mark Donnigan:Again reliability. Reliability and scaling, yeah, scaling.
Keith Chow:Also MLQ at the moment cannot use multicast. We need to use a lot of relay and the relay itself is similar to any caching server and it has the cache delay and also you have the odds of redirection delay, packet forwarding delay. So I've been talking with some people. They're using the datagram retransmission. They have improved a little bit. For example, quick protocol I test it.
Keith Chow:Um, actually you can just take any uh youtube and then you just luckily find, find the one with quick and so you can introduce some late um round trip delays or uh packet loss. I do have one at home and also in the lab. So I just impaired the stream and then after three percent of a random packet loss it just dropped down to very low bit rate or it fall back to tcp okay. So and in the moq I've been talking with someone uh who are working on it I cannot name them because I need to the permission to to before I can go and mention their name. They are using Datagram retransmission. It improves significantly of the retransmission. It can support up to more or less near to 10% of random packet loss, but that is still not sufficient 5% random packet loss. They're already scaling down the resolution Okay.
Mark Donnigan:Yeah.
Keith Chow:I've been testing that one from 4 megabit down to 400 kilobit, so it's not a TV. Okay, you cannot watch the football like that. You look on the TV screen and say where the hell is my football?
Mark Donnigan:Yeah, exactly.
Keith Chow:If that's a golf ball, even okay. Yeah, yeah, it's true. And um, so that, uh, quality problem with reliability problem. That is still a problem for moq. And also there's a lot of people two million chef in the in the kitchen. Okay, everyone want their own stuff into the moq yeah I'm not sure when the hell you're going to be ready. Okay, when you're ready, let me know. Yeah, two minute. Chef in the uh in the kitchen in the kitchen, yeah it's good or bad.
Keith Chow:Okay, good thing is that everyone will contribute, but if they're all pulling the water to their own mill, water mill, there's going to be not enough water for everyone.
Mark Donnigan:Yeah, that's right.
Keith Chow:So that's quite a lot of problems. Someone needs to stand up and say that this is the way and this is the reason, that's right. I'm not sure that will work, but using RTP is different because the standard is already there. Everyone can use it. Let's hit and go. And the only thing is that the retransmission. Anyone know how to do the control theory? Actually, I can tell you that a lot many people are now learning control theory because they're all doing the software engineering.
Keith Chow:They do not teach that? And if you do the E&E electronic engineering or aerospace engineering, the one who did SpaceX and they know how to do the control theory. They can control rockets such a size.
Keith Chow:So it's very simple, because you just need to control the error, so you just correct the error, apply the control theory algorithm and that's it. It's very simple and that is how we managed to handle up to 40 percent random packet loss, yeah, even 50 percent of the periodic packet loss. For example, every 300 millisecond we lost 100 of the packet and the next 300 milliseconds, no loss. We can still watch the video.
Keith Chow:Perfect, okay, yeah, so amazing that is the quality we need to do and a lot of people try to say, ah, you do not apply the laws into your network, you're testing. I say that is not right because in the network it's ott, it may or may not get there and may or may not have the random packet loss or may or may not have the bursting noise.
Keith Chow:For example the microwave can generate a 300 millisecond bursting noise. That's why we test the 300 millisecond loss, because due to the microwave, because we watch the microwave, and then how long is the bursting it lasts.
Keith Chow:And then we say okay, that is the test case we need to do. Okay, so we go around and measure all the loss in the network, because we are capable to do so, we have the opportunity to do so, and so we managed to characterize the loss in different networks. For example, in the fixed wireless network, we find out 38% Actually. In the book network, we found out 38% Actually in the book. Someone's written a book about that. It's 40% random packet loss. I say, holy, that is a really challenging one. So we just did that. So we just managed to get the 40% random packet loss and repair. That is the edge At the moment we try to push the random packet loss and repair. That is the edge we need to. At the moment we try to push it, the random packet loss up to 50%. That is our target at the moment.
Keith Chow:Wow, I'm not sure if that can be done.
Mark Donnigan:Yeah.
Keith Chow:But that's quite challenging.
Mark Donnigan:That's amazing. Yeah, that's incredible. Well, keith, as I said, thank you so much for your time and for the deep dive that you gave us in zero latency, zero drift. I really like how Nokia is focused in this area and I learned a lot. I hope our listeners learned a lot, so we'll have to have you back. Maybe we'll do a follow-up session and we'll bring in more about the NetEnt VPUs as a part of the solution as well, and we can maybe talk about a customer, a platform as well, that's using the solution. So that would be wonderful.
Keith Chow:I just showed you that use case, case study we've done in um in europe, in italy actually, and I watched the football match england against italy euro 2024 that's right and we are two seconds ahead of the dtt, t2, dvb, t2, so we have a negative latency compared with the broadcast TV.
Keith Chow:So that surprised me. I can tell you that when I saw that I said, oh, that means someone in the broadcasting. They have the delay because the broadcast cannot have the delay. So that means they have contribution delay compared to ours Because wherever we arrive we just send out same for them.
Mark Donnigan:Yeah.
Keith Chow:So that is how good we can do Interesting, interesting, wow, that's great.
Mark Donnigan:Well, good, well, thank you again for joining us on Voices of Video, and we will have you back. I know this episode of Voices of Video is brought to you by NetInt Technologies. If you are looking for cutting-edge video encoding solutions, check out NetInt's products at NETINT. com.
