Our special guest, Rotem Meidan, drops a ton of knowledge on how to make your Detox tests more reliable, including what the causes of Detox test flakiness are and how to solve them.
Our special guest, Rotem Meidan, drops a ton of knowledge on how to make your Detox tests more reliable, including what the causes of Detox test flakiness are and how to solve them.
This episode brought to you by Infinite Red! Infinite Red is a premier React Native design and development agency located in the USA. With five years of React Native experience and deep roots in the React Native community (hosts of Chain React and the React Native Newsletter), Infinite Red is the best choice for your next React Native app.
Helpful Links:
Connect With Us!
Jamon Holmgren:
Hey everyone, welcome to the React Native Radio Podcast where we explore React Native together. I'm your host, Jamon Holmgren, and I am joined today by one of my co-hosts, Robin and Adhithi could not make it today, but Harris is here. Harris, are you still in Montreal?
Harris Robin Kalash:
Yes I am. Hi, everyone.
Jamon Holmgren:
So you're going to stick around there for a while? Or are you planning to move around the country again?
Harris Robin Kalash:
Well, I'm actually planning to go to Vancouver, and still domestic, so we can still do that. Hopefully, it'll be better for my mental health, because staying in Montreal can be a bit depressing.
Jamon Holmgren:
Is the city that bad?
Harris Robin Kalash:
Yeah, because we don't really have too many outdoors activities, and the vaccine rollout is not as good as in Israel, so I am jealous.
Jamon Holmgren:
We're going to be talking, I'm going to introduce him in just a moment, our guest who is in Israel. So Harris when you are in Vancouver, are you going to do some outdoor activities?
Harris Robin Kalash:
I just enrolled in snowboarding lessons, and I would not have done that, had it not been for the pandemic, but yeah.
Jamon Holmgren:
I skied for the first time in early pre-pandemic 2020, and it was great, it was amazing. We went up to Mount Hood here near Portland, and stayed at a cabin, little cozy cabin in the woods right next to Mount Hood, or basically on Mount Hood. And then I went up and skied. My wife is extremely good at skiing, and just floats down the mountain. I watched a bunch of YouTube videos and stuff, and I managed to do it without falling and stuff, which was good at my age and size. So, that's cool. Hopefully it goes well, and you don't break anything.
Harris Robin Kalash:
Hope so too.
Jamon Holmgren:
With us today is a very special guest, Rotem Meidan. Correct me if I'm wrong here Rotem, but I believe your bio says that you are the former leader of the Detox team at Wix. It says former, right?
Rotem Meidan:
Yeah, that's true. So, hi everyone. Let's get one thing at a time. So hi, I'm Rotem. I'm in Israel, in Tel Aviv, where we're now having our third lockdown. So doing this is also something that is a bit... I mean, it has its challenges, and I need to close the doors, I have my kids just outside, so probably would hear some of the yelling, or laughing.
Jamon Holmgren:
Totally cool.
Rotem Meidan:
And yeah, so what you mentioned is yes, I recently left the role of managing the Detox team at Wix, and now pursuing a different role in server development at Wix. And this is something that I think... It's after almost five years, doing mobile at Wix.
Jamon Holmgren:
No, that's great. And I first met you, we've actually met in person, and it was at Chain React 2018 I believe, you gave a talk on Detox. You can check out that talk, we'll put a link in the show notes. That was really well received, Detox when it came on the scene, I feel like everybody was like, this is the answer, this is what we need for end-to-end testing.
Rotem Meidan:
I hope that this is something that people needed, and I also know that this is something that is the de facto end-to-end testing tool today with React Native. And this is what we're really trying, and the team is still really trying hard to do, and Wix uses Detox internally a lot. We have one of the biggest React Native applications. And I think in the world probably, where over 100 mobile developers are building the same single mobile app, so having greased, and very good CI process is something that is really, really important to us. So, Detox is really, a very central part in that process, and that's why we are able to invest so much in it. The team itself consists of four engineers, and we're building a lot of the features upon the needs of the application, and upon the needs of developers. And we see that this is also being received very well outside.
Harris Robin Kalash:
Nice.
Rotem Meidan:
I was actually also a part of a different episode in React Native Radio, and in its first life, I think it was 78 if I'm not mistaken.
Harris Robin Kalash:
78? Okay.
Rotem Meidan:
I think so, yeah. But this far, I just Google now. Yeah, so there I spoke about Detox in general, and I think today we're going to have something a bit more unique about Detox.
Harris Robin Kalash:
Nice. Actually, everything you said makes me wonder, can you tell us a bit about the origins of Detox? Why did you guys wake up one day, and decide let's build this?
Rotem Meidan:
First thing was the engineering challenge, to build something that would make it easy for React Native developers to build end-to-end tests. But seriously, we've been working with Appium, even before we really started building this huge app, we had an experience with something a bit smaller, also with React Native. And then we tried something else, we tried Appium, and we found that this is something that we couldn't... There were a lot of places where we couldn't find a solution, let's say, when we opened the app in debug, and we're waiting for the packager to load the app, and we see that this is a process that can take any amount of time, from a second, if this has already been loaded, to I don't know, maybe two minutes if it's a big bundle, and it needs to build everything from scratch. So, in this case, if we will run an end-to-end test on our debug flavor, you will see that this could be very flaky. So you would need to, I don't know, wait for an unknown amount of time. We then saw that there were some projects, that have done things like it successfully, like Espresso by Google. And really at the beginning, or just before we started working on Detox, Google also released EarlGrey, which is...
Jamon Holmgren:
EarlGrey, yeah.
Rotem Meidan:
It's the same thing that Espresso does on Android, they do that on iOS as well. So they have some sort of synchronization mechanism, that runs inside the app under a tests process, and it waits for things to be idle, and only interact with the app when it's done, so when it's idle again. So it tried to understand if there is no network requests now, no animations or anything, and only then, it interacts with the app. When EarlGrey was released, we found that this is probably a good time to try and take this technology, and add it inside Detox. But the funny thing is that we, two months ago, we released a new version, Detox 18, which was our final parting of EarlGrey. EarlGrey is not a part of Detox anymore.
Jamon Holmgren:
I was going to ask about that, because I saw that in the release notes. That was really interesting that you went away from EarlGrey.
Rotem Meidan:
Yeah. So this is probably maybe the biggest change we had in Detox, maybe since its inception on iOS. So a year ago, we decided to totally rewrite the iOS part of Detox, and mainly because we knew that EarlGrey doesn't do things the way that we want it to do. So, one of the other projects that we're building internally for quite some time now, is a performance benchmark project. A project that will run Detox, and then will measure all kinds of events, and all kinds of memory usage of the app, and CPU usage, and all kinds of other metrics. And by using EarlGrey, we knew that, because EarlGrey does it, the way that it tries to understand if the app is idle, is by polling all its, those are called idling resources, it polls all those idling resources, and tries to understand, okay, so, the network manager, are you idle now? And the animations are you idle now? Indeed does that on every time it has an opportunity to do it. So it adds this check on every time the process does its event loop. So it can do it for 3000 times a second, and it gets the CPU just to 100%.
Jamon Holmgren:
With a lot of end-to-end testing frameworks, they do polling, do I see the result I want to see on the page? And that's more of like a black box testing, this is still doing polling, or at least EarlGrey was still doing polling, but it's doing it in a micro way, in a very targeted way, where it's looking at those processes. That's very interesting.
Rotem Meidan:
Yeah, that's right, it does polling but on different things, not on the end result, but on how the system is acting, or reacting right now. The way that we understood that we need to take it as an approach that, instead of poll everything, we knew we need to rebuild it with an event-based mechanism. It knows to send an event back. Whenever they're getting idle, it sends an event that, I am now idle. So, this takes, I would say almost zero, or very negligible CPU usage. The new implementation takes much less CPU usage, and we can see that in how Detox now runs in parallelisation. So, I tweeted a month ago, about the new version, and how we manage to get much more out of specific machine of specific hardware. With this, we almost, I think that we increased, or decreased by 30% of the runtime of the test suite, not because Detox is faster per worker, but because we can now run many more workers on the same CPU. So on my 2018 MacBook Pro, I can now run... The optimal setup would be running 10 simulators in parallel, and the previous version, we only could run five in parallel.
Jamon Holmgren:
You all are doing some really amazing work with Detox, but one of the reasons that we brought you on the show was, you actually reached out to me after listening to a previous episode of React Native Radio, which is awesome, thanks for being a listener.
Rotem Meidan:
Episode 180.
Jamon Holmgren:
For those of you who haven't listened to it, that is, Testing Strategies, Tools and Frameworks. And we talked about testing, which is, of course, something that Detox is going to come up in. And one of the comments that was made was that, sometimes Detox can be a little flaky, and we run into situations where it'll fail, and then you have to run it again, and then it'll pass. Rotem is like, "I got to get in touch with these guys." We have ways around that. So, the whole reason that Detox was created was because Appium was flaky, and so Detox is supposed to be a lot less flaky. And there are places where it will fall over, but there are techniques really to write a stable end-to-end test. And so, prior to us, we scheduled this recording, and then prior to that, you actually came up with an article on Wix engineering blog, which is titled Detox Writing Stable Test Suites.
Rotem Meidan:
Simple.
Jamon Holmgren:
Yes, sorry. And I've read through it, it really has laid out a lot of great stuff. In fact, if people go and read that article, they're probably going to get a lot of the same information that they get from us today. Maybe we'll fill in a little more of the... We'll be able to ask some questions that have kind of come up. Without further ado, let's just get into the article, and talk about the different techniques that you've laid out on the table for Writing Stable Test Suites.
Harris Robin Kalash:
First of all, I learned so much from just reading that article, I've used Detox before. And sometimes we're talking about how sometimes the answers in the documentation, but just putting it together in an article like you did, I think helped a lot, but nowhere to look.
Jamon Holmgren:
So you start off with some ground rules, you just give some rules of thumb around... I think actually, these aren't specific to Detox, these would be a good idea to do in any end-to-end.
Rotem Meidan:
On any testing actually.
Jamon Holmgren:
Yeah, any testing at all?
Rotem Meidan:
Yeah.
Jamon Holmgren:
Exactly. So the first one is, no consistent input is equal to no consistent output. This I've heard said of course, garbage in garbage out, right? And that's the GIGO thing. And so, do you want to talk a little bit about that?
Rotem Meidan:
So this is something I also talked about in the talk in Chain Reaction 2018. Users need to understand, of course, that if they're not controlling their inputs, let's say, if this would be some sort of response from the server, or experiment, or an A/B test, that is getting different thing that gets into the app. So those could execute different code paths. So you can either get a specific error, or something not loading, or an actually different screen. What we had when we began working with Detox, like NCI in a serious manner, is that we found that, when we were running things locally, on the dev machine, we got some sort of experiments that we got on that specific user, but when we ran the same exact code, the same exact test, in CI we got something totally different. And we found out that, we have different experiments that run in the United States, and different experiments that run in Israel. So, we were only able to see that when we took screenshots, and videos of what's going on in CI.
Rotem Meidan:
And when we understood that, we understood that we need to build some mechanism, that will enable us to pre-configure all these experiments blob, and run it, and connect this specific configuration per test. So, then you can run it anywhere in the world, and we just override whatever, like the big production system of Wix is doing with A/B tests, and we've done a consistent behavior with that. And one more thing is, the ability of... We want to make sure that those production servers that we're working on, we're getting consistent, or we're not getting errors from them. So, we'll talk about this probably in a few minutes, of how we actually make sure that we don't get errors from those servers. And then we see that we have, what we call consistent input across the board, with all the responses from the servers.
Jamon Holmgren:
Makes sense. Yeah, and this sort of thing can show up in things like time zones as well. If you have timing issues, we have a section of the podcast where we talk about weird bugs, and Robin brought one up in a previous episode about a test that would only fail in January of this year, because we actually did have a bug, and it was because we were not expecting months to be zero based, instead of one based. So it was like zero was January, one was February, instead of one, two. So, that was another thing where you need to make sure that you have a consistent environment that you're playing and working in, to make sure that, if your CI server is in America, and you're in Israel, that you're looking at the same thing, that makes sense to me. What you say in the article is the most important tip really, in the end, and that is writing isolated tests, making sure that each test is starting fresh, and not dependent on the execution of previous test. So this is something where, if you had a test, here's an example I guess, if you had a test that logged you in, and wanted to test the login procedure, the next test should not be, okay, now that I'm logged in, let's do something else. That's depending on the execution of the previous test, and that can be flaky.
Jamon Holmgren:
I will admit to something, a little confession here, I tend, with Detox, to write these long paths through the app, they're happy paths, but they're like, I'm going to log in, and then I'm going to go here, and then I'm going to go to here, and I'm going to go here, I'm going to go check this, this, this. So in a way, I'm violating this principle, but it's partly because I want it to be faster, and I'm already logged in. So I don't want to go relogin for something else, and I'm already on this page, so why don't I just switch to this other one and check it. So I think this is the one that I violate the most, and it's also the one that you say is the most important.
Rotem Meidan:
Yes, and I would say that I really understand some of those violations, because it does make sense a lot of times to not repeat something that you want to do. Actually a login scenario is something that, well, you don't have a real good solution to. So what we did with logins, specifically, is that we have a mechanism that enables us to do a quick login in the system. So we have two flavors of the app. The first one is the production, the one that goes to users, and it has no hacks in it, no kind of overriding mechanisms. Internally what we have is a few mechanisms, just like the A/B test override mechanism. We have also a mechanism that helps us do a quick log in. So, in that quick log in, we put the same user and password but in the test itself, not even in the app. And we don't log in with the login screen, but we actually tell the app to do a log in for us, but in a much, much faster way. So, when we have this sort of mechanism, we can then free ourselves of building things that are not isolated. And this was a big thing for us in isolation of tests.
Rotem Meidan:
This enabled us to actually do, we have a login test suite that actually tests the login flow, but all the others, which also need to log in, they don't run through the whole real production login setup, but they do this quick log in instead, and the result itself is exactly the same at the end.
Jamon Holmgren:
That makes sense. Yeah, you mentioned in the section, that you restart the app before every test too. So you actually are providing some tools that will help you isolate the environment, you're not just saying, okay, I have to... You actually give beforeAll, and then you can run a script to restart the app.
Rotem Meidan:
That's true. So, those events beforeAll, then all those events are actually not events, all those...
Jamon Holmgren:
Hooks?
Rotem Meidan:
... all those hooks beforeAll, beforeEach. And all those lifecycle hooks of the test runner, are actually a part of the test runner. And we've been using, I think that solely, almost solely, Jest by now.
Jamon Holmgren:
Jest.
Rotem Meidan:
Jest service. And this is our recommended test runner, and this is because we know how to do a lot more with Jest than we know how to do with, let's say Mocha, or different implementations of Jest, specifically because Jest Circus is also... I think this is, I'm not sure if it's already the default, but it will become the default test runner of Jest. And it has a lot of features that we really incorporate, and we use all those in artifact collection, because we need to understand when the app starts, when the test starts, and we need to correlate all those events together, in order to create a coherent file, or a coherent artifact directory, that says on a specific test, or in a specific launch of the app. We want to order all those artifacts together, if those are logs, or application logs, or videos, or screenshots, we want to make sure that whenever a test fails, we take the screenshots just at the same time, and we want to make sure that we get the screenshot exactly at the time it fails, so not half second later, we want to take it exactly when it fails. So, we could know what actually was on the screen, when the test failed. And we do much more than that, not just taking screenshots, but also we're taking hierarchy dumps, and things that are maybe very helpful for later investigation.
Rotem Meidan:
We take it to the max with each platform. With iOS, we can't delete the application data, but with Android we can do that, just because Apple doesn't allow to delete the data, and they only allow for uninstalling, install if you want to get something fresh from the beginning. So, we have a flag in launch app, if you want a new instance, new instance is just load the app again, and make sure that we have a new process, but we can also re-install the app at the same time.
Harris Robin Kalash:
One rule of thumb I really like from your article is, your test should probably end with an expect line, right? So...
Rotem Meidan:
Right.
Jamon Holmgren:
Because anything after that, what's the point of it? You're going to be resetting anyway. Yep, makes sense to me. Now, you point out an issue, test hang and time out, and I've had this problem happen to me. So this is one I'm very interested to hear what you have to say about it. You say that there are a few different reasons, that this can happen, and you explain Detox's gray box mechanism. You talk about how it waits for network requests, animations, et cetera. There's an example of output for printing synchronization status. So when things are just taking a long time, it'll actually pop out a bit of code, or a bit of an output that says the system is busy with following tasks, dispatch queue, main run loop, and a network request, and it says what the URL is.
Harris Robin Kalash:
Yeah, actually, on that topic, could you talk to us a bit more about what is the synchronization output. So I understand the why, the Detox managers, the sync, but there's a few things here other than the network status, for example specifically the Loops, or the queue. To a developer like me that comes from web development, and not necessarily as familiar with the Native code, what do all of these mean? Especially the main loop, for example?
Rotem Meidan:
So, I think I'll start a bit about the differences between Detox 17 and Detox 18. The comment that was made in episode 180 was that the application hangs, and you don't really know what to do. So if, I'm not sure who said that, but if you would take the same test, and would just add minus D, at the end of the Detox test command, it will get the output of what we call debug synchronization. That is some sort of mechanism that initiates from JavaScript. The JavaScriptExecutor, or not an executor, the JavaScript Orchestrator is what runs in node inside Jest. It sends expectation command, or a matching command that it wants to get an answer to.
Rotem Meidan:
And when this is not resolved from Native, because the Native waits for things to get idle, it won't do all that, because it waits for things to get idle. So, the node process then sends an event that says, "Hey, I know you're busy, but tell me why." So this is the debug synchronization. So we send some sort of an event, or some sort of a request to a native, to ask, what's going on? And then this returns with all this information. So with Detox 17, this wasn't turned on by default, and the reason for that is, that it wasn't very stable, or at least it wasn't stable to the point that we could say, okay, we can put that on by default. And we need Detox 18 because we rewrote everything, and this is now not EarlGrey anymore, and this is a code that we control 100%.
Rotem Meidan:
And then we know that this now, is much more stable. So, if you'd run Detox 18 now on any application, and the application would hang, it would automatically just print all these debug synchronization output, to your test runner log. Now if we dive a bit inside this, we consider a few things, so we're waiting for, let's say, if we could look at the first example, we see that this is something that only waits for a network, but we have three kinds of things that are not idle at that point. So it attributes to the way that, or to how Detox works. Detox doesn't really know a lot on how things are working inside the app. It just, in the most dumb way, hooks itself up to a few resources in the app, so let's say to the network stack, as we have here, the network request, but also to the main run loop, or with other threads run loops, and all kinds of other queues. So this run loop that you see in the main run loop, is the actual run loop of the main thread.
Rotem Meidan:
So, whenever you ask for the system to get something, this is usually done first from the main thread, and then it just outputs it to other threads, but sometimes the thread is synchronized, or at least waits for responses from the other threads that are running on the system. So, whenever something like this happens, the main thread, well, it's idle, but it still waits for something. So it says that it has something, and it's event queue, and it needs to end. And so the event queue itself is not empty, and while it's not empty, it just outputs that as a thing, but this is more of a side effect of the network request itself. So it just outputs everything it knows about the system, but the actual thing here is that you have a network request in flight. And probably the best way to approach this, is to make sure that if you see a network request, this is probably the culprit, and you can test it by just removing it, or making sure that you have, or fixing the issue of course, and then you will see that it just goes away. Same thing with other examples that we have here. Some of the examples, we would see that we have JavaScript timers, or Native timers.
Rotem Meidan:
So a lot of things in React Native are built on timers. The animations module in React Native is built with those timers. So, if I have a timer that runs in a loop, then we know that we should wait for this timer, and if it never ends, then we can't really do a lot with it. So the way for us to investigate all those, is to know our application, there is no real way of saying, okay, it's a helper, it's not... It can say exactly where your code executed, and what it executed. And a part of this reason is, it attributes to the fact that, React Native is well, two things, one, you have React Native is a declarative framework, and whenever you configure or something, this is not the actual code that runs when things happen. It just tells you that you have some sort of an engine that, or this Native module that does things for you. So you say, I want to have an animation that runs with this configuration and that, and I want to do this at the end. So all those things are not being executed within the user's realm, but in the frameworks realm. So, Detox only understands the framework realm, it doesn't understand the user realm.
Rotem Meidan:
And we can really do that with a declarative framework. It's not like, oh, we can't go back in the stack trace, and find what the user did. A good way to start from is, whenever you see something in your app that gets the app hanged, is first look at those logs, but then you understand that this specific event, let's say, tapping on a button, or that pushes a screen, will now get your application in an endless non-idle state. So you know that this button does something. So, go and check out what this button does, and try to disable from some things, and it will probably pretty easily find what's going on. So internally, we see those things a lot. I mentioned that we have like 100 developers running those E2Es, have their module being tested by tests that someone else wrote. We have some sort of smoke test suite that runs on everyone's code, and this sometimes poses an issue like this. Mostly developers see that, and understand that this happens, just because they know that they did a specific change, because it runs on every change that they do, or on every... They know they did a bunch of changes, and they now run it against that suite, and they know what those changes were.
Rotem Meidan:
So they know how to go back, and fix whatever needs fixing. This can be very frustrating, we know that this can be very frustrating, the fact that your app hangs, and you don't really know where things come from. But if you know the app, if you know specifically what you did, and what changes you made, it should be pretty easy. I won't say it's easy, but it should be pretty okay to solve. And this has a really great added benefit, that you can make sure that your app doesn't use more CPU than it needs to use. If you didn't have Detox testing your application, you could have a lot of things running in the background, working... Whenever your application seems idle, and the screen doesn't change, nothing really happens, but in the background, it can do a lot of things that you don't really know that are being done. And with Detox, this actually eliminates it, because if your Detox test is fast, it means that the application got idle eventually.
Jamon Holmgren:
That is a really good point that it's actually measuring a lot of stuff that is not visible to people, and just hidden under the hood. But since it's watching that, for other reasons, but it's watching that, it will fail if it just watches these long running processes that are just cranking away there.
Harris Robin Kalash:
Yeah.
Jamon Holmgren:
So, moving into the next section, which is actually pretty similar, issue test on CR slower, network issues, timeouts, slowness, brings up a lot of issues, that we would not see while developing locally. And this is another broad issue. So, really, what Detox gives you here, isn't necessarily that it's going to fix, or tell you exactly what to fix. It's not going to come in and say, "Hey, this component right here, is running this hook, and this hook is telling, is doing something wrong." It's not going to say that, but what it does give you is a bunch of tools, really cool tools. And these are things that I honestly didn't even know existed, I didn't know you could do all this stuff, until I read your article. Now, to be fair, I don't use Detox every day, I manage a team, and they use Detox, but I think that this is probably something that may come as a surprise to some of our listeners, it comes with a bunch of stuff, including, of course, logs.
Jamon Holmgren:
You can take a screenshot right away, as soon as the test fails, and have that available. You can take a video and actually see what's going on, what's on the UI. There's one called timeline, there's instruments, and there's UI hierarchy. I know UI hierarchy, you can bring that into Xcode, and actually you rotate the view around, and look at the views stacked on each other in a very visual way, which is awesome. Talk about timeline and instruments if you would.
Rotem Meidan:
Yeah, sure. So I would just say that UI hierarchy, this is a very old feature of Xcode, but it is now available in Xcode 12. Well, in Xcode 12 you can save this hierarchy dump, and open it later, you don't have to be connected to the process. To debug the process, and see the hierarchy, you can actually now save a file. So, we took leverage of that, and we now know how to do that on iOS. Timeline. So yeah, let's talk about timeline. So timeline is a tool that we developed actually for our internal use. We wanted to see how we can improve the runtime of Detox internally, specifically, while running with multiple workers. We have Jest that orchestrates everything for us. It orchestrates when it starts it, and it orchestrates what the next test suite is going to run on the specific worker. So let's say we have like 20 files, and we have five workers, and whenever a suite ends, Jest throws the next test suite on this worker, and says, okay, now go ahead, continue with this suite, until there are no more suites left, and then it just shuts down everything, until all the workers can finish, and it just shuts down.
Rotem Meidan:
So this timeline will show us the real runtime of all our tests. So it just saves it as a Chrome-trace file, that can then later be opened, and we can see all the workers that ran, and all the test suites, and actually, also internal events out of Detox. So you can see how much time it took for the app to launch, and how much time it took for a specific command line tool to run. So let's say we have X, you run some CTL install, which installs the app file on the simulator, we want to know how much time it takes. Sometimes it takes a lot, we can really do a lot with this, but with things that are more, like user space, we can do. So we can then see that we have, let's say, we have three test suites, and we run on three workers, but one of those test suites is really, really long, but much, much longer than the others. So, we'll have two test suites, or two workers that are finishing in a minute. And then the third one, which is much longer, it takes nine or 10 minutes. So, everything would run, the real runtime would be 10 minutes, although we have two workers that are not really doing anything. This can help us optimize the way that suites are being distributed into files.
Rotem Meidan:
So if we have this one long test suite, we can probably break it up to maybe two or three test suites, like sub-test suites to different files. And then Jest would do better work in parallelizing those. And instead of having one file that runs for 10 minutes, and the others that run for one minute, we'll have three files each run for three minutes, or two and a half minutes, and then it will distribute itself better through the workers, and then we can probably cut like three minutes of runtime. So this is what we use timeline for, internally at Wix, but we also think that this is a really good way for users to see how their distribution is being laid out with Jest.
Jamon Holmgren:
If all else fails when you're building your test suite, and you're having a hard time getting a test to fail. It's hanging, or taking a long time, you can always do retries, right? And you put that in there, you can always put in some retries, Jest we'll do it for you, the Jest circus. You can set a per test retry strategy, you can do it per file, and these often will fix the issue. It shouldn't be the first thing you reach for, of course, you should try to actually fix what's wrong. But if at the end of the day, the code is working how you want it to, and it's still causing problems, you can always throw in a retry. And I feel like that's reasonable to say. Well, if there is a flaky test, we can't figure out what's going on, the test, or the code itself is working once we're actually testing it manually, let's just put it in a retry.
Rotem Meidan:
Yeah. If you have to, then you can definitely do that. If you have an isolated test, you can use Jest retry mechanism, which is very targeted, specifically at the test you're running. So, if you have a long test suite, and only one test fails, it will only retry this specific test a few times, but if you can't do good separation inside test suite, then Pro file would solve your issue, it's more like brute force in your face. If some tests in this file fails, even if it took like 20 minutes to run, and this specific test is like a few seconds, it would just rerun everything, and then it would make sure that it runs exactly like it ran before, it would run the same tests in the same order.
Rotem Meidan:
Another good tool that we have in the artifact collection is instruments. Instruments is a tool that we built internally at Wix. It's an open source as well, Detox Instruments. It's a tool that also runs some sort of framework inside the app, so it understands a lot of things from inside the app. It's tied really well into Detox. We use that for our performance benchmarks, we know how to record all our performance metrics with this tool, and then query the output from this tool. But apart from performance benchmarks, we can actually find, or we can actually record all the network data that is being run inside the app. So, in the article, I added an example of a test, of a Detox Instruments artifact, from a test that has a 401 unauthorized response from a server.
Rotem Meidan:
This is actually something that failed our test, and there is no other way of finding it, maybe if you add all those inputs and outputs from your network to the log, and then you can go, and find it in the log. But this is something that is built-in, it's a part of Detox, you can just use it, and see whatever you want, and it saves all the network response completely with all the headers, and everything is laid out in the nice UI.
Jamon Holmgren:
Very cool. Well, thanks so much Rotem, this has been absolutely educational, to hear you go through all of this, and it gives me a lot more confidence that, when we do use Detox in the future, that we have the tools that we need to do this. There's a lot more to this, than we went over obviously, we don't have much time in an episode on React Native Radio, but at the same time, Rotem did just write this article, and published it. So I do encourage people to go look at that, very open to feedback from what I understand, and so that's something that I think people should be very free to get in touch with the Wix, Detox team, and let them know the experience that they're having.
Jamon Holmgren:
I'll just speak for myself as an open source maintainer, if you can provide a minimal reproach, if you can show like, hey, Detox, isn't working for me in this particular situation, and send it over as an open source project, that is something that I'm sure, I know for myself, it really is helpful, because sometimes I can look at it and say, well, you forgot to do this, or, well, there's this other thing that you can do. And sometimes it's like, nope, that's a bug with my library, thank you very much. I would encourage people to do that. Rotem, if people want to get in touch with you on Twitter, where can they find you?
Rotem Meidan:
Oh, my Twitter account is Rotemmiz, it's Rotemmiz. My DMs are open if you want, or also just conversation via Twitter, so definitely. And also, of course, we have our GitHub issue board in Detox, well, I'm not going to look at that, but the team will. So former-
Jamon Holmgren:
Former head.
Rotem Meidan:
Yeah, former. So yeah, definitely discussions should go there. The thing is that, we want to have good discussions, we want to understand the pains of users. We think we know the pains of the users, because we are the users ourselves, and we are very close to other users, which are 100 developers inside Wix. And we know what's hard for them, and we know that whenever we're trying to help them, we feel those problems on firsthand. But definitely this is something that we're trying to do for the past, maybe half a year, or maybe more, maybe almost a year, we're trying to improve the way that, let's say the experience with Detox. So, we know that Detox has pretty good synchronization mechanism, and that it tries, or at least... Or not tries. And we know that it solves the synchronization issue that you would have with a black box testing suite, or black box testing framework, but there are still a lot of things that can go wrong.
Rotem Meidan:
And we want to make sure that users have everything they need, whenever a test fails, to have an actionable error, that would help them pinpoint their issue as fast as possible, and to get going as fast as possible, because as a testing framework, we want to make sure that users increase velocity when they use it, and not, of course, the other way around. So it's important to have a tool that works well, and that has a good feedback. So, if you feel that you don't have that, and you read this article, and you read recommendation, and you found that it's something that you still don't have and you really want to have, then sure, let's talk about it, and see how we can improve it.
Jamon Holmgren:
Thanks a lot, and Harris, where can people find you online?
Harris Robin Kalash:
Yeah, on Twitter, I am nomadicspoon. So it's nomadicspoon.
Jamon Holmgren:
And I am Jamon Holmgren, @jamonholgren on Twitter. As always, thanks to our producer and editor, Todd Werth, our transcript and release coordinator, Jed Bartausky, and our social media coordinator, Missy Warren. Thanks to our sponsor. I forgot to read our sponsor thing a bit at the beginning, but I'll give a shout-out to our sponsor at the end. It is my company, Infinite Red, check them out at infinite.red/reactnative. Special thanks to all of you listening today, make sure to subscribe, tell a friend about this episode, especially if they're having some problems with Detox, and we will see you all next time.