Voices of Video
Explore the inner workings of video technology with Voices of Video: Inside the Tech. This podcast gathers industry experts and innovators to examine every facet of video technology, from decoding and encoding processes to the latest advancements in hardware versus software processing and codecs. Alongside these technical insights, we dive into practical techniques, emerging trends, and industry-shaping facts that define the future of video.
Ideal for engineers, developers, and tech enthusiasts, each episode offers hands-on advice and the in-depth knowledge you need to excel in today’s fast-evolving video landscape. Join us to master the tools, technologies, and trends driving the future of digital video.
Voices of Video
Driving Innovation in Connected TV: Automation and AI with Martin Tyler from FX Digital
Are you troubled by the unpredictable nature of testing on diverse platforms? We tackle the challenges of test automation in environments where traditional methods fall short. Learn about FX Digital's innovative vision-first automation framework, a game-changer that uses image comparison to outperform traditional HTML DOM-based testing. Experience the benefits of this approach firsthand as we discuss its impact on user interactions and the ability to identify elusive issues on various devices. Plus, find out how AI and machine learning are integrated into their QA processes to ensure smooth and high-quality video streaming sessions.
Curious about the impact of AI in software testing? Uncover the possibilities as we delve into AI-driven test case generation and explore the pioneer tools FX Digital has developed to enhance their processes. Discover the burgeoning world of stream testing with Google's Universal Video Quality tool, and hear about the ongoing journey of FX Digital's engineering team as they expand their internal automation capabilities to meet the demands of fragmented ecosystems. As we conclude, we invite you to be a part of our growing community by sharing your thoughts and engaging with us for more insightful conversations.
Stay tuned for more in-depth insights on video technology, trends, and practical applications. Subscribe to Voices of Video: Inside the Tech for exclusive, hands-on knowledge from the experts. For more resources, visit Voices of Video.
Voices of Video. Voices of Video. Voices of Video.
Speaker 2:Voices of Video.
Speaker 1:Good morning, good afternoon, good evening, wherever in the world you are. Thank you again for joining us on this very exciting and, dare I say, engaging episode of Voices of Video. So I am Mark Donegan and we are here talking today with Martin Tyler. And Martin, welcome to Voices of Video.
Speaker 2:Thanks very much, Mark. Yeah, really, really excited to be here and chatting to you today.
Speaker 1:Yeah, I know, you know we've had a chance to debrief a couple times. I think initially you reached out maybe it was the spring, even certainly was summer and I recall saying love to talk to you, but it's going to take a while to get you on the show.
Speaker 2:I've been on the wait for a while, that's for sure.
Speaker 1:You've been. Yeah, yeah, well, thank you. Thank you for your patience, by the way, we do appreciate that. We do appreciate that. Yeah, well, you know, let's just let's kick off our conversation as those regular listeners of Voices of Video know that we host a fireside chat here and maybe spill some beans as to what's behind building connected apps and connected solutions that are reliable and that function well and look great. I guess let's start with talk to me about your company and what you guys do.
Speaker 2:So, yeah, I work for a company called fx digital. Um, we're we're the connected tv experts, so we specialize in all things like front end connected tv. So that's right. From the design stage, we do engineering, so developing those applications, qa, so testing the applications, and the team that I work in is that's the quality engineering team. We also build tools and technology and a device lab, uh, to actually test those apps, both made by us and made by other people as well. Um, we've had quite a long journey of of how we've got here. So, yeah, that's kind of what I'm here to talk to you about today is like the, the journey we've been on and all the different, different things that we've we've done so far yeah, you, uh, you have and connected television.
Speaker 1:Uh, and specifically, the experiences, um, you know, the on-screen experience, I guess, if you will, is is so critical, a lot of focus on content, which obviously there has to be, content that we want to watch in order for a service to be able to, you know, continue to grow and and retain subscribers. But the experience is, so, you know, so critical. So maybe you could start by characterizing, like, like, who do you work with? Do you work for? Are you working directly for the tv manufacturers? Are you working for the ott services? Um, who are your customers?
Speaker 2:yeah, so, um, we, well, we, actually we started off if I just go back to the kind of the history of fx. We started off as a web agency making normal websites and we were always trying to get involved in new technologies and we did some voice stuff for a while, some Amazon Alexa apps and stuff like that, and then one day we got the opportunity to work for Discovery, to do some work for Discovery Amazing A little bit before my time at at fx, but, uh, I know the team really went for it and uh and and won that, um, and for a long time, uh, we, we were, we were very involved in building the discovery apps. We built the euro sport app. We bought the global cycling network application, which is actually one of the one of the first applications of the global gcn app on apple tv. That was one of the first ones. We, we automated tests, tested, actually, um, and then, yeah, fast forward to now.
Speaker 2:Um, we've got, um, lots of big clients, a client called tennis tv, uh, we make an apple tv. Um, we make the brick box application, um, and we also make an S4C application. S4c is the Welsh arm of the BBC, so we make some applications for them and quite a few other customers as well.
Speaker 1:Yeah, yeah. Well, you know we're going to, throughout the course of the conversation today, try and go a bit deeper into the technologies that you're using and you know I think it will be very interesting to our listeners to hear more about that. But I am curious. I want to rewind a little bit this transition, transition from a web development agency to building apps. Was that as a result of HTML5, the switch that happened a number of years back now, where a lot of apps were built in Flash and then they transitioned to HTML? Was it as a result of that, or were you always in the native sdks, or you know how did this happen?
Speaker 2:I think it's a web development to apps yeah, I mean, I'm definitely not the best person to ask about that, because it was before my time at fx, but I know that we've always we still have uh like innovation running through the company. So we've always we still have uh like innovate innovation running through the company. So we're always looking for like new opportunities, new technologies to work with, and like we really never shy away from that. So, uh, yeah, I don't know the exact ins and outs of of the switch, but I do know that we've we've always been all about innovation and uh, yeah, I'm sure that's a big reason, reason for for us being able to get involved in these different technologies.
Speaker 1:Yeah, yeah, for sure, for sure. All right, well, good, well, so you are responsible, I guess, you and your team, for QC, to make sure that the experiences are reliable and you know the consumers, the users, always delighted. So why don't you tell us what have you built? How do you do that? You know I've heard of services that literally have, you know, rooms full of like hundreds and hundreds of devices, and you know, is that what you guys do?
Speaker 2:so, yeah, so, yeah, I'll just take it back a little bit there. So, um, we've actually got two quality teams in fx digital, so we've got a bit of a different setup to most of the most teams. We have a quality assurance team and they're the guys that actually test the applications. Um, and they're the guys that actually test the applications and they're the ones testing it for our customers, both manual testing and automated testing using the tools that we create as well. But alongside that, the team that I'm in is the quality engineering team, so we're the team that actually creates those tools and technologies for the QA team to use. I think it's quite an interesting setup doing it like that, because a lot of the time you'll find, ultimately, when you're setting up test automation, you've probably got some priorities that are going alongside that that are a bit more important, or, at least seen, important in the eyes of, uh, in the eyes of other stakeholders in the business, because you know you've got apps you're trying to create for customers and that, yeah, they need to come first really. And but by splitting the, the two teams out, splitting the responsibilities, we are able to focus 100 wholeheartedly on those, the technology behind it and pushing that automation technology forward. And then we've still got the qa team who are able to focus fully on making sure that our apps are really, really good and you know best in industry, yeah, yeah, um.
Speaker 2:So, yeah, the, the um, and you you mentioned about, like, the, the tool, the tools that we we use and the tools that we build. So, um, we initially started um test automation on on GCN, um and which is the global cycling network application, on Apple TV, uh, and we were able to use like mobile technology for that. So, um, there's a lot of open source technology uh out there for testing mobiles and actually you can reuse a lot of that technology for uh, for tv, os and android based devices. So we used a tool called web driver, io, um, which is, yeah, like an open source uh automation tool, was is very, very, very good for the purpose of uh, of of mobile, and we kind of slightly adapted that to work on TV Um and um, we uh and we put proof, proof, the theory that we, with that open source tool, um, but we, you know, we're the connected TV experts and and we, you know, we every, everything that we do is connected TV first, we're not mobile first. A lot of these tools that are out there were built initially for mobile and then adapted to TV, so that's kind of the exact opposite of the ethos of our company.
Speaker 2:In addition to that, our dev teams. They use lots of different tools to make applications. Um, in the case of uh, the apple tv build, they used react native in in that instance, so they were able to create that and and we were able to use the use web driver, io. But a lot of the, a lot of the applications we we build are made with a framework that is connected to TV, first called LightningJS, and what's different about LightningJS is it uses WebGL under the hood. Now, webgl is a language that renders the images that you're seeing on screen in the GPU rather than in the CPU. Now, in some cases, that can have performance enhancementsments, but a lot of what we use that for is because it's much more repeatable than like css.
Speaker 2:Um, when, when, uh, when you're looking at like, uh, developing applications for connected tv, you have to think about like, and I'll again I'll compare it to mobile. You've got two real operating systems in the mobile space You've got iOS and you've got Android, android, yeah, yeah, that's the ones you need to worry about, whereas in connected TV you've got Samsung, you've got LG, you've got Android, you've got Apple.
Speaker 1:But you even have different SDKs inside Samsung, right, and LG has separate, and Sony, and I mean it's really diverse.
Speaker 2:Exactly, and sometimes it feels like there's a new operating system being released every week.
Speaker 1:To be honest, yeah, not to go down a rabbit trail, because this is not really, at least today, this isn't my domain of expertise or where I spend a lot of my time, but earlier in my career I actually did spend a fair amount of time in set-top boxes and media players. So you know a lot of hardware, of hardware, and I just I'm always astounded when I see a news release of some and I won't call anybody out specifically, but somebody in the ecosystem who suddenly thinks that the world needs another tvOS and you know it's like what are they thinking? And I certainly can understand. You know that is ground zero for consolidating user experience and certainly you have the benefit if you control that screen. You know, provided you opt in, and you know you need GDPR and all of those controls, right, but that's where you get all the first-party data and you can do a lot of good things with it. So I understand why it's valuable but, boy, from an implementation perspective it's a bit of a minefield, I guess you could say.
Speaker 2:Yeah, definitely, and I mean our customers. They want to be across as many of those as possible, because they want as many customers as possible, right?
Speaker 1:Yeah, of course, of course. And you don't want to say oh sorry to any users. You know love for you to see my content, but your TV's too old, or you know we don't support that model.
Speaker 2:So and, yeah, you've hit the nail on the head there. So we've already discussed there's lots of different operating systems, but in in addition to that, you don't go out and buy a new tv every two years like you do with a mobile phone. Right, with a mobile phone, yeah, we're supporting uh devices back to like 2016, or even even, in some cases, maybe older than that as well. Yeah, so so you can imagine like to, to be able to uh have like a, a framework like lightning js, which which runs a bit more, um, kind of like you know, a bit more the same across all those different platforms and all those different model years. It is really for us to be able to quickly scale across all of those different devices for our customers.
Speaker 1:Yeah, yeah, interesting.
Speaker 2:Which is really great for the dev team, but the traditionally how you would do test automation is by interrogating the DOM. So you would have, you would have this HTML DOM and you would go okay, does this element exist? Either from an XPath or from an ID or something like that. But because we're rendering everything in the GPU, you no longer have a DOM, so you don't have anything to actually interrogate. And that's what led us down the route of building our like vision first, uh based automation framework. Um, because we all we really had was an image. We had to just do something to look, look at the image. So everything then has come since, as it has come, kind of like vision, vision first. And I think that's forced us into this kind of uh like situation where, because we're doing everything vision first, there's lots of visual based automation tools out there, um, but maybe they're more. They're more put on afterwards to just as more of a catch-all than some of the stuff that we're doing we.
Speaker 2:We have something called our image comparison service, and what this microservice does is it takes a screenshot of the application and it makes an image of a particular element. So, rather than having like your XPath or your ID, you've got an image and that's the identifier, and then our image comparison service takes, uh, those two, those two reference images. Now that the, the image uh from of the of the element, that can actually be from the design files, that doesn't even necessarily need to be from the real application. It's flexible to be able to do that. Without being so flexible, it's not accurate. And then that returns lots of information. It returns things like where that element is on the screen. It returns whether it obviously, whether it is there or not, size of the element. It checks, like the colors of the element.
Speaker 2:Um, although you know we were kind of forced down this road because of, because of lightning js, actually it's it has a lot of advantages. It's much easier to read the uh, read like the tests, because sometimes if you've seen some x paths I don't know if you've ever come across them, but they can be like completely unreadable, like you don't know what that is. But if you can see an image of a of the element that you're looking for quicker for for our qa team and, and I think, sure the the amount of tests we're running at the moment, I think we're running around 240 000 tests a running at the moment. I think we're running around 240,000 tests a month at the moment. It varies, it's over 300,000 tests a month.
Speaker 1:When you say tests, that means an individual screenshot that is then being compared. Or is it a device that you're regularly querying, or what exactly does that mean?
Speaker 2:So a test would be, a particular test scenario, so like, for example, users able to log onto the app. So when I talk about, like, the image comparison service that's that's queried like many different times for each test case, I think we, I think, uh, off the top of my head, I think we hit that around about 200 000 times a night. So, yeah, we're using, we're using that um quite extensively, uh, by our framework. But, yeah, the, the, the visual um, the visual um kind of like way of way of doing that is really. It is has a lot of advantages because, um, you know, I'm a qa that's that was my background before I was building automation frameworks. So always I've got that kind of mindset, I guess, of how a QA works.
Speaker 2:And one of the things I always come back to is like, how is the user actually using that application? The user's not going on and interrogating the DOM. They're looking at that image. Yeah, exactly, actually using that application. The user's not going on and interrogating the dom, they're looking at that image. Yeah, exactly, actually, like it's how the user's using it. Uh, much more so, and we've seen, like, if I give like a specific example, sometimes, uh, we saw once on a skybox, a particular shader, um was behaving slightly differently on one particular skybox to it was um on all of the different uh all the other skyboxes and was um on all of the different uh all the other skyboxes. And we were able to pick that up by using the visual frame right now. If, if, uh, we were just using the normal traditional methods and interrogating the dom, we would have interrogated that dom and then, um, it would, the element would have been there, it would have been fine. We're not actually looking at, you know, the, the vision stuff that the users look at.
Speaker 1:So was it? Was it something like a gradient was incorrect or caught? You know, the color wasn't being rendered correctly, or is that is that what you mean by it was? There but it didn't look right yeah, exactly the.
Speaker 2:The the shader that was applied over the top of it was was behaving completely differently to the rest of the devices, so it just looks different. It didn't look like the design, yeah, it just didn't look right.
Speaker 1:Yeah, that's super fascinating. So Are there limitations, like some devices, and maybe talk through exactly how that screenshot mechanism works, because I think that's super intriguing. And, for example, you know, like, so I'm a user. Do you have some sort of logic built in where you think that maybe something didn't go right and then you screenshot and then you compare, or is it sort of a random you're, you're literally just like sampling and you know? Or how does that mechanism work? You know, in terms of when you're screenshotting and, um, yeah, um, so what triggers it?
Speaker 2:it's definitely not random at all. Like it's, everything is very repeatable and it has to be for testing right, because you need to know you're doing the same, exactly the same steps each.
Speaker 1:Yeah I guess what I meant by random was more like like, like, like sampling, uh, you know, as somebody's going through, because it seems to me, of course ui is getting much, much more complex, but a lot of the motions are pretty, they're repeated a lot, right, you know scrolling and you know up navigation, left navigation I'm changing thumbnails, I'm selecting a thumbnail.
Speaker 2:Those are sort of repetitive operations, exactly, yeah, so how the QA team actually uses that, that we write all of our test cases in Gherkin. So it follows that, given when, then syntax, and we have lots of pre-made, predefined steps that a QA is then able to use, so what? The first step that they will run is that uh, like, uh, like, the application opens and then um, when, uh, the when, when the the landing page, for example, when the landing page loads, and what that step will do is it will take a screenshot of the application and it will perform that ics check to see if that element is included in there or not. And it will check for, like, the first element in its list of elements. So it has a, it has the framework, has a list of elements that the QAs create. This is what should be included in this page, and the first element is a unique one to that page. So that's how it knows that page is there by identifying that unique element. Oh, I see. And then we will have other other steps that will go on this. This element is here, so it will check all the checking, all the different elements that should be on there, and it'll go through and individually check those ones again with the ics api, call um. It can then, um, do things like pressing buttons, and that's one of the steps that we have, to press buttons and go to different pages, and then you can, of course, you can do things like type out on a keyboard. We've got a navigator that navigates around the keyboard and from there you can imagine, by, by comparing, by combining those things you can. You can do lots of different things in the app.
Speaker 2:But I guess that's quite a good point as well. Like there isn't, uh, like you, you say like all of the apps are the same, and I kind of agree. Like a lot of the time the apps are the navigation's moving in quite this roughly, the the a similar way. But actually, when you break down the nuances of an application, how it works, there's so many different ledge cases. Yeah, and that's when our custom test steps come in. So we have these predefined steps that the QA team have written and the QA teams use, and they can write a lot of tests with that, but certainly not all of them. So they use what we call custom test steps. Well, that well, they'll write the javascript code generally the qa team, and they'll be supported by by us. Uh, we're able to write javascript code and that can. That can do things like imagine a lot of the time when you log into a tv application, you get like a like a six digit number and you have to put it into an app on a different website. Yeah, exactly, that's right.
Speaker 1:I just had to do that with an update that I did on one of my services. So just the other night, Nice, yeah, yeah. But fortunately the process was beautiful. Qr code I just opened my iPhone, zoomed in a little bit on the camera, clicked it. You know it authenticated. The biggest hurdle was I forgot my password, Reset the password. But that wasn't their problem, that was my problem.
Speaker 2:Yeah, we've got things like that as well. We can also read the QR codes and then we use Puppeteer basically to go off and go onto the website from the perspective of an actual human user, not even a tester.
Speaker 1:I mean literally just you know, somebody subscribing to a service Like analyzing in the background or watching in the background to try and determine proactively if there might be some issues, and then screenshotting. Do you have that built in?
Speaker 2:I sort of assumed that you also had that functionality built in, but maybe not.
Speaker 1:So yeah, we're not screenshotting like real users content or anything like that we we have. I can imagine there's probably privacy and you know there's the you know, there's some issues there. Yeah, yeah but is that.
Speaker 1:Is that possible, though you know? Like would there be a way to anonymize, you know, for example, if you weren't, because you don't care the poster art, you're not, at least. Well, maybe there's some use cases where you would care, like specifically the title somebody was on, but you're looking at elements, right, so it doesn't matter what the movie, what the title is or the titles on the screen. You're wanting to make sure that if I click on something that it actually you know goes to the right place, and that the right button is recognized, and that sort of thing yeah, so both.
Speaker 2:Actually, we want to make sure that things are mapped out and displaying on the screen as as, as kind of as they should be. Um, we do need to know, like in advance, really what what should be expected. Right, you can't write a test case unless you know like the outcome that actually should be expected. So, yeah, I don't think there would be much um, like value in us doing it on like a real user's app, because you you don't know what, what, what's expected to come up, yeah, yeah, the other thing to mention is a lot of the time, these connected TV devices have quite low processing power.
Speaker 2:So if, if, it was true user and it was, you would. You were doing things like taking screenshots and doing all this stuff in the background. You would like damage, damage their experience, and that's really what we're trying to achieve by doing all this test to create an amazing experience for the end user.
Speaker 1:Yeah, yeah, exactly. Well. So super fascinating, and I'll take this opportunity for anybody listening live. For those who are listening live to this on the replay, we do host these as live streams on LinkedIn and also on the Voices of Video website or on the landing page. So the advantage to listening live is you can ask questions in real time. So just put a plug in there.
Speaker 1:But one of the questions that I think someone out there has, so I'll ask it, is what sort of challenges did you face? And I think you could answer the question from two perspectives the internal perspective of building automation. So, was it super easy? Were you able to convince everybody that, yes, so was it super easy? Were you able to convince everybody that, yes, this was the right way? Did it take some, you know? Did it take some selling? Did you have any major hurdles, both internally and then externally, Like when you went to your clients and you said, hey, we've got this. You know this new concept. We built this new service. Was everybody like, wow, where have you been all my life? Or were they like, maybe, a room full of devices and 25 testers is how we should keep doing this, so talk to us about that.
Speaker 2:So, yeah, I mean, I think, uh, yeah, definitely a solution would be a room full of devices and loads of loads of testers somewhere, um, but yeah, like realistically, um, yeah, people's people don't want to necessarily spend that amount of money on on like a huge ql team to test across all of those. So, um, really like automation was was solving the problem of how do we test across so many devices and also how do we do it regularly. Because what we were doing like prior, prior to that is, is all manual testing and we were, we were testing on like a high-end device and a low-end device and a lot of the time that kind of does cover a lot of the, a lot of the bugs, but definitely not all the time and to be able to take that quality to the next level. That's why that's why we needed automation. But, yeah, it certainly wasn't a uh, an easy thing to set. Like you know, necessarily sell internally, um, either, because, um, you can imagine what like we're trying to. We're trying to deliver projects right, especially like our pm team there. Their job is to make sure that the project's delivered on time and in budget, and we were kind of like at first the thing is, at first you've got to do all that manual testing that you were doing before, but alongside that you've also got to do to write loads of automated test cases and and also training the QA team and stuff like that. So I think we, yeah, we definitely needed to kind of explain to the wider business, like what the advantages of automation were. It wasn't, it wasn't, you know, easy and obvious for them. And I think we, we kind of made the mistake of thinking, oh, it only affects the QA team, it doesn't, because it's slowing the QA team down and it's affecting the other teams as well. So, yeah, we faced, faced lots of challenges, lots of challenges like that Um and um, we'd obviously we, we, we built this framework from the ground up.
Speaker 2:So this, this, this vision-based framework, we built it from the ground up. So, of course, there was bugs in our framework as well. Yeah, sure, we had these um, we had these sessions where we would sit down with the Qa teams every single day. We'd go through the, the lot, the logs. We'd teach them how to, how to read the logs, understand like the results, analyze the results, um. But we'd also, um, like you know, be finding bugs in the framework as well, which we'd then solve in the afternoon and then log back back on in the evening, up until the 11 at night in some cases, to like to make sure that everything was running as it should be. And we learned a lot from those sessions.
Speaker 2:You know, it's not just about automating the test cases, it's about automating all the processes around that as well.
Speaker 2:So we've got things like a test case generator which, from a list of elements, can generate like a boilerplate, like test cases.
Speaker 2:Also, we have like slack bots as well, um, that tell the qa is not just like all the tests that have failed and all the tests that pass. They're not really worried in everything that's failed, they're just worried in the things that started failing it like on the run last night, because every day they're testing, checking these results right. So little things like that might sound like they are like really like you know, simple and small, but actually they make a huge difference. And because we were kind of making all these changes like every day, we were making more and more changes like this to the process and and I think ultimately, yeah, like the technology, it it's not something that you can build overnight. It is. It does take a lot of work and a lot of time and effort, but equally, like the people and the processes, they're more important, to be honest, and I think you're only going to get like the true value of that once you get that process and the people right around your technology. Yeah yeah.
Speaker 1:So let's talk about the use of AI. I would be very interested to hear how you're using AI, both in the framework, you know, to maybe make the automations more efficient or increase throughput or more accurate you can tell me exactly what you're doing there, what the benefits are but also in the use cases. I think in my very limited exposure but I have had exposure throughout the years with QA is that sometimes I think the use case is as important, or even or the test case. I guess case is as important, or even or the test case, I guess is as important or even more important than the methodology or the tool, because you know if you're not testing for either the right things or you know if the coverage just isn't broad enough, you can have all the tools and great resources, but you know there's going to be problems, right, just isn't broad enough. You can have all the tools and great resources but there's going to be problems, right, yeah, so talk to us.
Speaker 2:Talk to us about that I think, yeah, I think, first of all, to say I think that's where the QA skill really really comes in.
Speaker 2:I know we're talking about AI in a moment, but, yeah, I think you're not going to be able to replace, like, the understanding that a qa has over an application after they've started working on it for a while.
Speaker 2:You're not going to be able to replace that with ai, but, however, you are going to, I think, be able to speed up their, the process that they can go through so you can get you know more, more out of your, more out of your qa team and and improve the throughput of their work. I guess the stuff that we're doing at the moment with ai and machine learning, uh, we're doing video player testing so we're able to take, like, video recordings and analyze those, uh, using machine learning and understanding, um, whether the video is playing, the video is paused or has a loading spinner, and we can like keep doing that to do like long day video testing. So if, if you're, if you, you can imagine like, if you're doing like a, you know how, how you actually use the app right, go back to what I was saying earlier how do you, how does a user use the app. They sometimes on a Saturday, maybe they're a bit hung over and they sit and they watch an entire season on Netflix. Right, we've all done it, I'm sure.
Speaker 2:Eight hours of streaming video yeah, exactly, and you know we need to be able to like automate, automate, uh, like you know, long play video testing, um, and equally, we also we use a tool, um by uh that's been created by google called uvq, which is the universal video uh quality. They actually use this tool, uh, interestingly, for enough for youtube. So they use it to to rate the videos, to say, like, what video is like uh is like the best quality. So when you search for something on youtube, you get like the top results. You know the best quality results in visually at the top, and then if it's like not filmed very well, it'll be be lower down, although the quality of the stream is not very good. And we can we actually use that to test like the video player quality on our set-top boxes. We don't do it on TVs because we can't do a stream out of them. So, and that's what we're doing today, that's what we're doing right now, but we're also working on using AI in the processes as well. I was talking earlier. So what we're working on at the moment is an AI piece of software that's able to take screenshots of our application and then write the test steps that I was talking about earlier, using those predefined test steps to create test cases that can then run in our automation framework. The reason that we use AI to generate the test cases, rather than just trying to make the AI actually test things, is it then becomes much more repeatable, and we want it to be repeatable because we don't want.
Speaker 2:The thing we've noticed with AI is sometimes it will have a bit of a different behavior to other times, and we want to make sure that. For the last week, we've been running the same test case every every time. Otherwise, how do you really know if the test case was passing yesterday or not? Um, and then, following on from that, I think we'd like to, we'd like to be able to correct the test cases as well. So, um, we have lots of test cases that you know will like, kind of like, say, you know you mentioned earlier, like the title of the video, for example.
Speaker 2:Um, it might be that the title of the video changes because our customers update the cms, they change, they change those things. So, yeah, but it might be that the behavior is absolutely fine and our QAs just kind of need to quickly update that. We do need to be predefined, because it might be it might also be wrong. You know, it might. Just it might the mapping might be wrong or something like that. We need to look at it. But we want to speed up the maintenance of those test cases. So we're using AI around around more of those features rather than putting it directly into our application to actually test and and are you literally um using one of the traditional llms?
Speaker 1:you know whether it's open, ai or um, you know um llama or, but yeah, there's, there's a half dozen right that you can use. Or is there something you have built or can you talk about that?
Speaker 2:right. Right now we're using llama um. The reason that we rover open ai is because um of privacy, so these are locations a lot of time, so we don't. Our customers wouldn't be happy with us putting designs of their unreleased applications into OpenAI, even like the premium accounts. A lot of businesses have policies that people just don't put anything on there. Right, it completely makes sense. Yeah, it does.
Speaker 1:It's really great that Meta open sourced, you know, Llama so that we have that one available, Because if all of the best LLMs were commercial, you know it would be a real challenge, you know, for a lot of applications.
Speaker 2:Yeah, definitely, and I think yeah, so that's something that's really interesting at the moment. This is like future stuff we're working on, so whether we try and, you know, start training our own models, maybe, uh, in the future. Um, yeah, this is like, yeah, like future vision stuff really, but stuff that we are working on today and hope eyes on them validate that they're correct Awesome. Again, you can't replace a QA with an AI.
Speaker 1:Yeah, exactly. Well, I actually have a couple of questions I see here that came in that we'll get to. But, um, I, before we do that, I I want to rewind slightly and double click on your stream testing, your automated player testing, basically, uh, my question is are you actually able to assess some sort of a of a score, of a score of a, you know, you know, like a VMAF score or, you know, ssim, psnr, is it that? Or is it really just is it playing, you know? Is it, you know? Are there any missing frames? Is it that kind of a of a test Like talk to us about?
Speaker 2:it. It gives it um, I think there's five different metrics that it's looking at and it's it's understanding. I think the way that you youtube trains the model that we're using was to understand like how a human would perceive that and it gives it a score out of five. So on its own, like the first time you do, it doesn't really mean much, but over time you can start plot that and you can see whether that the score is increasing or or decreasing.
Speaker 1:Okay, so this is the. This is Google's UVQ. Yeah, exactly, yeah, yeah, okay, okay, interesting. And, you know, is that open sourced? I'm not sure that everybody is familiar with it.
Speaker 2:Yeah, it is. It is an open source tool that anyone, anyone can use.
Speaker 1:Amazing. So there's a's a GitHub repo or something.
Speaker 2:Yeah, there's a GitHub repo.
Speaker 1:Yeah amazing.
Speaker 2:Yeah, so it's a great tool. Yeah, we've had some good success in using it and measuring things, and it's really more like you mentioned some specific measurements. You can't really get those out of it, but you can get trends out of it. You can understand whether the quality of the video is going up or down, because I think a lot of the time when you look at if we did lots of research into these algorithms and you, you guys probably know a lot more about this but a lot of the time you do need, like the original video file and then the streamed video file, to kind of compare the two. And we didn't, we didn't want to do that because our customers wouldn't necessarily be able with their licensing, they wouldn't necessarily be able to, absolutely.
Speaker 1:Yeah, that's. That's actually a really critical point, because even if they would like to do that, they're not able to do it.
Speaker 2:Exactly. Yeah it's a challenge, definitely, and they might have a piece of content even that they are able to share, and then all of a sudden it disappears, right, because their agreement's kind of expired Lose the window.
Speaker 1:Yeah, exactly, yeah, well, good. Well, let's jump into a few of these questions, and this next one actually is in perfect alignment. The question is how do open source tools compare to commercial tools in terms of reliability and functionality for automation at scale? So I, I it sounds like you know you, you've built a lot, you've largely used open source, um, but maybe you can comment uh, yeah, I mean we, we've um.
Speaker 2:I I mean I I don't. I haven't had loads of experience with some of these, these more these like commercially available tools that we have. We've tried some of them out internally and there was like various reasons that we decided to build our own, our own tools. Like one of the big reasons we didn't want to modify the application under test or we wanted to essentially make sure that we limited that, at least essentially make sure that we limited that at least um and um. But what I can say is we've managed to get, uh, quite extremely good reliability, um from our the other tools that we've built and the open source stuff. And yeah, they comment on the commercially available tools because, although we've trialed them, we've not used them at scale understand.
Speaker 1:Yeah, uh, but, by the way, I mean, if you don't want to give me an exact number, that's okay, but do you have, like you know, count on one hand the number of engineers that are building, uh, this? You know your internal automation tools. Is it two hands, is it more than two hands? Like, how big is the engineering team that's so working on this?
Speaker 2:the team that's working on this. Uh, there is uh five of us at the moment working on it.
Speaker 1:One was part-time okay, well, I, I I mean that sounds like you're you're doing a lot with, you know, a relatively small group.
Speaker 2:Um, that's impressive yeah, I think we. I think we've also kind of scaled the team up in the last few months as well. Oh okay, wow, we've worked very hard for a while on it, I think.
Speaker 1:And you've been building this for several years, right? How long have you? Been building the platform.
Speaker 2:Yeah, three years now we've been building it. When I started off building this, though, I was a QA as well, so I was working on our tennis TV project, testing that in my spare time, and some weekends I was kind of building the open source stuff. And then from there, we've kind of, uh, we've, we've kind of like, yeah, scale scaled up from there really and we started off. We inherited this, uh, the this little device lab. Oh well, it wasn't really a device lab, it was a little storage room really at the back of our old yeah that's the most device labs are yeah.
Speaker 2:So this, this is actually as you walk into our office now you're, you see this big glass room with, with all the devices in we've just, we've just um. You know, we're expanding this again because we've got got some more customers, um, and so we got, we got nine tvs TVs delivered just today alone actually. So, yeah, we're expanding this quite a lot.
Speaker 1:Yeah incredible.
Speaker 2:We've just rebranded to purple, hence the purple color at the back.
Speaker 1:It's cool. It's cool, you know, if you walk into a you know someone's house who's really into wine. You know they often have, you know they're. You know some sort of display area right, I've seen some beautiful similar, you know, with glass, and the racks all lit up and the bottles are all nicely displayed. You walk in your office and it's a whole bunch of TVs.
Speaker 2:We've got yeah, we've got racks of set-top boxes as well, not just tvs, and yeah, game, game consoles and all kinds of things I can imagine yeah
Speaker 1:yeah, well, you know you talk about um, the two mobile oss, and Android and iOS. So on the surface it's like, oh, I need to just support two OSs. Well, ios is relatively constrained and Apple does a very good job of controlling releases and a lot of, I think most Apple users are, you know their devices up to date. So, but the Android ecosystem, wow, it's a. It's a fragmented, yeah, again, a minefield. So, well, good. Well, another question that came in before we wrap up here is I think we largely touched on this and I'll add another. I'll say are there any specific types of tests or scenarios where automation is a usable tool and even a better tool? And then, have you found, do you have some experience to share, where you're like, yeah, we found there's this one area where we still kind of have to do it the old-fashioned manual way. So curious what you found.
Speaker 2:Yeah, I think, in terms of what the best scenarios for AI are. I think because we're using we're kind of like trying to teach the AI to actually write tests in our framework there's not like specific test cases, really, we're trying to get it to write all of those tests. In terms of what automation handles better than manual testers, I would say like regression testing, because imagine it as a QA this you've done the same test cases a million times or not. Yeah, many times, a lot of times. Yeah, uh, many times before. And uh, you know, you, you do get complacent. That's just human nature. You know, if you do the same thing again, you can't keep paying attention. That's, it's completely natural.
Speaker 2:So I think regression testing is very, very important to use for, but I think we use that then to to free up the qa's time to do things like exploratory testing. So they're looking for, you know, real niche edge cases and we've got some amazing people at fx that are able to find the most crazy bugs through that and that frees up their time to do that. In addition to that, you know we've connected tv. That really the most important, the most important functionality of that is the video player, because you know, without, without a video player, what use is it? Connected tv application? Um, yeah, we, we will always we. We do automated tests on it to really like a little smoke test just to make sure everything's working. But we would always have eyes on that video player really regularly to make sure that everything's working with that.
Speaker 1:Yeah, amazing. All right, well, Martin, thanks for coming on. It was an excellent conversation. Keep up the good work. And if people want to learn more about FX Digital, where do they go? How do they find you?
Speaker 2:Yeah, so you can either follow FX Digital on LinkedIn or you can follow myself on LinkedIn as well.
Speaker 1:LinkedIn or you can follow myself on LinkedIn as well. Awesome, yeah, yeah, I know you're you're. You're pretty active on LinkedIn as, uh, as I know a lot of us are, so that's good, okay, excellent, well. Well, thank you again. It was a wonderful conversation and, as always, thank you to the listeners. We really do appreciate, uh appreciate those of you who come back week after week. It means a lot and if you have any feedback, any comments, if there's companies you would like to hear from, if you would like to come on, if you have something interesting that you're doing, just reach out. We're easy to find. You can reach out to me, you can reach out to our producer, anita, you can send an email to the NetEnt website. There's probably a dozen different ways to get to us. So reach out and we'd be happy to have a conversation. And until next time, have a great day and enjoy building video Cheers.
