In this episode, the React Native Radio hosts discuss testing -- what testing strategies are popular, what frameworks they’ve used, and how TypeScript fits into it all.
In this episode, the React Native Radio hosts discuss testing -- what testing strategies are popular, what frameworks they’ve used, and how TypeScript fits into it all.
This episode brought to you by Infinite Red! Infinite Red is a premier React Native design and development agency located in the USA. With five years of React Native experience and deep roots in the React Native community (hosts of Chain React and the React Native Newsletter), Infinite Red is the best choice for your next React Native app.
Helpful Links:
Connect With Us!
Jamon Holmgren:
Hey, everyone. Welcome to React Native Radio podcast, where we explore React Native together. I'm your host, Jamon Holmgren. I'm joined as usual, by my cohost, Adhithi Ravichandran. Hey, Adhithi, how's it going?
Adhithi Ravichandran:
Hey. Doing well, Jamon.
Jamon Holmgren:
Awesome. We also have Harris, Harris Robin Kalash. Harris, you're in a different spot again. How are you doing today?
Harris Robin Kalash:
Hey, Jamon. I'm good. Hello from Ottawa.
Jamon Holmgren:
Ottawa? Every time people talk about Ottawa, they just talk about how cold it is over there. Is it pretty cold there?
Harris Robin Kalash:
Yeah, and it's boring. It's just politics. It's like DC, I guess, it would be the equivalent.
Jamon Holmgren:
Yeah. I'll stay very far from DC this time of year. And then also, my cohost, Robin Heinze. Robin is not in Ottawa or Washington DC. How are you, Robin?
Robin Heinze:
I'm doing good, Jamon.
Jamon Holmgren:
Good. Good. I'm very excited about this topic. There's a lot to talk about. But before we do that, I want to mention that this episode is sponsored by Infinite Red. Infinite Red is a premier React Native design and development agency located in the USA and a little bit of Canada. With five years of React Native experience and deep roots in the React Native community, we are the hosts of Chain React, the conference, and we also publish the React Native newsletter to over 11,000 subscribers. If you're not one of them, go to reactnativenewsletter.com. Infinite Red is the best choice for your next React Native app. How'd I do with that reading? I didn't just make it up off the top of my head this time.
Robin Heinze:
That was really good. It was a good elevator pitch.
Adhithi Ravichandran:
Yeah, it was good.
Jamon Holmgren:
Good. I stole it from Jed. I think Jed wrote it originally, Jed Bartausky, our project manager.
Robin Heinze:
Shout out to Jed.
Jamon Holmgren:
Our topic today is going to be testing strategies, tools, and frameworks. And There's a ton to go over here. I mean, we have so much to talk about. I got to admit, I didn't do as much show prep this time, but to be honest with the topic, it's one of those things where we live it every day. So, I feel like there's just-
Robin Heinze:
Everybody does it, but nobody talks about it, so let's talk about it.
Jamon Holmgren:
Right? Exactly. So, let's just go ahead and launch into this. I think that first off, we have to just talk about, just for really basic understanding, testing of course, is where we write software that tests our software, the software that we write, it turtles all the way down, software all the way down
Robin Heinze:
In terms of JavaScript, it's functions, all the way down.
Jamon Holmgren:
Functions all the way down to the bottom. It's also something that I didn't necessarily take seriously through probably the first half of my career because I wrote software, I'd test it manually, it worked, what was the big deal? Like why write more software? That seems like it would just slow you down. But then I started doing Ruby on Rails, and in Ruby on Rails, Ruby in general, there's more of a culture of writing tests, there's a large contingent of people that believe in TDD, test driven development. There's also another contingent that believe in behavior driven development, which is similar. There's outside in test versus inside out tests. That led me to this idea that that testing is a good idea. And I've seen it for myself as well, where tests have helped me be more confident in the software that I'm writing. But we're talking tests in the context of React Native, and React Native is a very unique environment for writing tests, because we're not necessarily JavaScript, we're not necessarily native, we're both. We have both things going on.
Jamon Holmgren:
So, it's not as maybe straightforward as it would be in a web environment or in a node environment or Ruby on Rails. That's why I think this topic can be such a complex topic for us React Native developers.
Harris Robin Kalash:
Yeah, I agree. It's definitely less obvious with React Native. I think one particular challenge is with end to end testing. That tends to be pretty challenging because if you're testing the user interactions, you have to test on both platforms. It's not like on the browser where you could just run Selenium, run a Chrome instance and be pretty confident about it. On top of that, there's also different patterns on each platform. So even if you do test user interactions, it might not be the exact same interaction on iOS or Android.
Jamon Holmgren:
Right. That in a way, goes back to our previous episode where we talked about the differences and the challenges between iOS and Android, when we're talking about the two platforms. So, we have a dichotomy there between the two platforms, and then we also have the JavaScript layer versus the native layer, and that can be very difficult to do. Traditionally, in mobile app development, native mobile app development, generally people use Appium as their way to do integration testing. Integration testing, or sometimes called functional testing basically, is where you write tests that simulate user interactions. So, loading up the app completely, you literally build the whole app, load it up, have them tap on the login input, put in an email address, put in a password, tap log in, it goes in, and then you actually tap around. And as you're doing this, you're also measuring and checking for labels or buttons or text on the screen to see, is this what we expect to see at this time?
Jamon Holmgren:
And it can be very, very helpful, these types of integration tests, because what they do is they test so many things. So many things have to go right in order to log in. It's like a smoke test. If smoke comes out, if you see an error, then something's wrong. Let's stop and let's look at it, and then dive in, in more detail at that point and figure out what's going on. But you really just want it to run in the background as you're doing something else or taking a break or something and come back and say, if everything passed, I don't need to go check everything. It probably works, so let's keep moving.
Robin Heinze:
What would you say the difference is between Appium tests and detox tests? Because in my head, they're the same thing.
Jamon Holmgren:
Right. From my standpoint, they do fill very similar functions within the testing world. You probably wouldn't use both, although there is a case for using both that we can get to. Most of the time though, you're going to choose one or the other detox. Detox is by Wix, the excellent engineering team over there at Wix has created detox. They actually built it because they were pretty ... they weren't very satisfied with what Appium brought to the table. They were not happy with how Appium was flaky. Essentially, Appium is a black box testing thing, so it basically just looks from the outside and says, I'm going to push this button, and then every once in a while, I'm going to look at this spot and I'm going to see what happens. And eventually, something is supposed to happen, eventually, it'll be like, okay, yeah, that worked, and we can move on.
Jamon Holmgren:
But because it's a black box, it doesn't know what's happening under the hood, and it can't really predict when things are going to finish. It'll tap a button, and then it just has a timer that checks every so often. So, black box is nice because it's like, that's what a real user would do. They don't know how the code is working under the hood, but sometimes it leads to spurious failures in your tests, and this can lead to flakiness, which is very frustrating as a developer. Oh, my Appium test just failed, what happened? You rerun them? They pass. It's very frustrating.
Adhithi Ravichandran:
I think that's where detox definitely helps out. I think that's what you're trying to talk about where I have never used Appium, but to some extent, the flakiness that people complain about is probably eliminated in detox testing because they have a slightly different principle. I guess they're a gray box testing, opposed to a black box.
Harris Robin Kalash:
I think it might be worth just describing what black box testing and a gray box testing is for viewers who might not know.
Jamon Holmgren:
Yeah. I would love it if you would do that.
Harris Robin Kalash:
Yeah. Sure, sure.
Robin Heinze:
Don't mind me, I'm going to Google this in the background.
Jamon Holmgren:
Right.
Harris Robin Kalash:
Yeah, no worries. I recently Googled it. Black box testing, essentially, it's a test environment where you don't actually know the internals of the application. So, as you mentioned, Jamon, it's really good in the sense that that's like an actual user, they don't know the internals of it. That's why it's called black because you can't see in the box. Gray box testing is in between, so it's a mix of black box testing and what we call white box testing, which is white box testing is if you're a developer, you fully understand the internals of the application and you're testing it as a developer.
Adhithi Ravichandran:
Right.
Harris Robin Kalash:
Gray box testing is in between where we know some of the internals, but we still want to test it as an end-user.
Adhithi Ravichandran:
Maybe we should also, to back up and let our listeners clarify what are the different types of testing as well? Because we're talking about end to end tests right now, but they're just different types. So, we have unit tests, we have service level tests, and we have the end-to-end tests, so we were talking about detox and Appium. Unit tests are probably the most written one. You would write like thousands of unit tests for an application, ideally, and these are basically testing a single piece of your functionality or a function. It's like testing, let's say a function. You test an isolated method, and you would do plenty of these. And then there are these service level tests, which are basically testing coe that have dependencies where you would test like-
Robin Heinze:
It's like API services.
Adhithi Ravichandran:
Exactly. Yeah. You would test APIs or component tests, and these are usually slower than the unit tests, and you would write maybe a little less than the unit tests. And then you finally have the end to end test like detox, which is basically testing the full flow of your application where you're testing it. It would emulate how a user tests the application and you would write fewer of those tests.
Jamon Holmgren:
So it's a, I think Martin Fowler calls it a pyramid or the pyramid of testing or something like that. So, there's fewer of the integration ones and more of the unit ones, and then in between. Now, why wouldn't you just write one type? Why wouldn't you just write unit tests? Why do you need to write the other level’s layers?
Adhithi Ravichandran:
I think we're doing that to create a balanced portfolio of automation testing. That way, we capture, not just for instance, if you just had unit tests, every unit works correctly, but when you put them all together, it may collapse. So, the idea is to make sure your entire flow is working, as well as every single unit works. I think that's the idea there. Because if you just had one layer, you may miss out, and it's basically to make your software less error prone and higher quality.
Jamon Holmgren:
Yeah. It reminds me of a tweet I saw, I think an image where it said unit tests pass and it shows two drawers in a kitchen cabinet in a corner. And when you pull one out, you can't open the other one. Both drawers work, but then the interaction between the two.
Adhithi Ravichandran:
Do they work together? Yeah.
Jamon Holmgren:
Exactly. Do they work together? Now, once you know that something works at a unit level, why would it ever not work in the future? Why would you need to write a test that runs it every time, rather than just testing it manually once and then being like, okay, it works, and move on?
Robin Heinze:
Are you ever planning to change your software?
Jamon Holmgren:
No. That's a good point.
Robin Heinze:
If you're ever going to add new features or fix bugs, you want to be able to quickly make sure that nothing broke, which in the business is called regression testing.
Adhithi Ravichandran:
Exactly. And as the scope increases, you can't merely click through the application every single time you add a component or a feature. You need automated end to end tests that take care of it.
Jamon Holmgren:
Yeah. It's going through and clicking things for you and checking things for you. Now, regression testing is a very interesting one. That's actually one of my favorite types of tests to write. So, if someone reports a bug on our open source and I go and actually write a test that replicates the bug first, so it's failing. I can see the test failing, and then I fix it and then it goes green. And then I can push up, pull request and commit it and know that in the future, this isn't going to break again because we wrote a test around that particular scenario.
Robin Heinze:
Yeah. It's a very satisfying type of test. You're like, one more thing I can feel confident about.
Jamon Holmgren:
Is it possible to over test?
Adhithi Ravichandran::
I don't think so.
Harris Robin Kalash:
Definitely.
Robin Heinze:
I mean, it depends on what you're measuring against. I mean, if you're talking about a project with a limited budget or limited timeframe, yeah, there's going to be a level of testing that starts to not be worth your money or your time, and doesn't add any more value. But in terms of like doing harm to your project, no, I don't think you can over test.
Adhithi Ravichandran:
I've never run across a situation where we ever over tested. It's always almost just under testing. Because as Robin mentioned, we either have a budget constraint or a time constraint or just laziness or we just-
Robin Heinze:
The testing is always the first thing to get cut. Yeah.
Adhithi Ravichandran:
Yeah. And we notice that, Oh, I know this probably works. It's just you feel overconfident and cut it out. I think we can have a goal to over test, but that almost never happens.
Jamon Holmgren:
It's sort of like saying in theory, there is a point where you become too rich, but I haven't hit it yet.
Adhithi Ravichandran:
Yeah.
Jamon Holmgren:
I would actually throw a couple of scenarios where maybe you could say that you over test. One is where you're testing so much that any change causes a ripple effect of tests that you're having to change that becomes very burdensome where any change at all becomes very, very ... It's like it's locking you in too much to a particular implementation. I would actually then say that's a symptom of testing implementation details probably, where you're testing things that don't really matter to the user, but you're testing things that matter to the developer, and now you go refactor and it breaks everything. If you can refactor without changing the behavior to the user, if you can refactor and not break your tests, and instead they're just helping you know when you're done, that's fantastic. I think Kent C. Dodds talks about this all the time. He has a great website about testing React. One of the things he talks about is if your tests are breaking on just a straight refactor where you're not changing behavior, then your tests are testing too many implementation details. And I would say that's over testing.
Adhithi Ravichandran:
Yeah, absolutely.
Robin Heinze:
Do you ever think there would be a scenario where you would write tests for the sake of the development process and then delete them?
Jamon Holmgren:
I've done that. I actually have done that. Yes. But for the most part, those are pretty rare scenarios. I would say an example of that would probably be a regular expression where I want to make sure that what I'm writing for the regular expression is doing what I want it to, but then it is a very implementation focused thing. But even then, often, I'm leaving it in the testing.
Robin Heinze:
Usually, if I'm testing regular expressions, it's because I'm trying to make sure that a function that's parsing a list of things to try and find something that matches or a particular regex, that's behavior. I want to make sure it's always returning the results that I expect based on the search term I gave it or whatever. I feel like that's valid. It helps you develop, but it also ensures that the behavior doesn't regress.
Harris Robin Kalash:
I would like to test more, but I feel like the culture of small companies or startups, they don't really accommodate for that, and you really have to think about the ROI of everything you do. So going back to what you said, Robin, about limited project and budget, I think that's pretty much most of the time, unless you work at Facebook and you own this little piece of the machine, then it makes sense to test a lot. But most of the time, I feel like we don't really have the time or the budget for it, and we have to really get these high ROI tests, and I think that's the benefit of detox here, the ROI is extremely high. So, it's annoying to write sometimes, it's quite expensive, but-
Adhithi Ravichandran:
And it's also, visually, you can see what's happening, so it's nice for ... if you didn't really have a QA, detox can play the role of the QA.
Jamon Holmgren:
What do you mean by that, Adhithi? How can you see what's happening?
Adhithi Ravichandran:
With tests like detox or an equivalent, which is Cypress on web, you can see the user action. So, it's all written, we write our code, and we simulate a user actions. So, it looks like on our emulator, somebody is actually running a test, so you can see the user experience. And you can do that on a device too, with React Native. It could either be your emulator or a device. A lot of the times, if you have folks who are not really tech, but who are interested in this application who want to make sure it's working well, even your clients, you can show that you can validate by just showing detox tests running and they'll be like, "Oh, cool, seems to be working. They've written a lot of tests, looks good. Whereas if you show them unit tests running on the screen, on your terminal, it just won't make much sense to a non-techie. That's what I was talking about.
Jamon Holmgren:
Yep. And Appium does something similar where you can watch what's going on.
Robin Heinze:
I always found that really, really satisfying.
Adhithi Ravichandran:
Right. Yeah.
Robin Heinze:
Running detox and just watching my simulator, click through the app is really cool.
Jamon Holmgren:
It does it way faster than you could manually. It's super, super fast.
Robin Heinze:
It's nice for demos too. It's a cool way to demo a feature.
Jamon Holmgren:
That's very cool.
Adhithi Ravichandran:
I relate to what Harris was saying as well, because I used to work for a larger company here in Kansas city, and back then, we used to write lots and lots of tests. We had plenty of time, plenty of resources and projects wouldn't get released for like eight to 10 months. So, we would test and test and test and we would have dedicated developers, just writing tests. And then I moved to a startup, and testing definitely took a back seat because we want to quickly prototype a product and sell it. It definitely makes a difference if you're working for a huge corporate versus a small company.
Robin Heinze:
Definitely.
Harris Robin Kalash:
What you were mentioning earlier, Robin, it's satisfying at first, but what I found with detox is that the first time I see it I'm like, "Oh, that's cool." But sometimes they're so slow and I ended up turning them off. To be completely honest, they're just ... I don't know why, but sometimes they just hang. I'm not sure what's going on, but I'll write a detox test, and I don't know whatit's waiting on, maybe there's a mistake in my code, I'm not sure. But sometimes, these tests will randomly timeout. I still find detox a little flaky, to be honest with you. I've had tests timeout. Sometimes they resolve, but that takes forever, like 20 seconds, and that's quite expensive.
Jamon Holmgren:
If I could write end to end tests that were very fast, way faster than they are right now, and not flaky at all. Always deterministic, then I don't know if I would write too many unit tests. I think I would mostly just write end to end tests. But the problem with that is that I've never ... That FedEx guy has his radio up really loud. He just comes flying up and then-
Adhithi Ravichandran:
I thought you live in a super remote area, so you get FedEx shipments?
Jamon Holmgren:
Yeah, we do. We also have internet and power.
Adhithi Ravichandran:
Great. Do you have any water?
Robin Heinze:
And running water?
Harris Robin Kalash:
USPS gets there too, right?
Jamon Holmgren:
Yes we do, when my mailbox is standing. I'll actually tell you something funny, Adhithi. My wife, when we were looking for land, she was like, "Okay, these are my criteria. I want it to be, I don't want to have really any neighbors to speak of. I want it to be ... She grew up on a farm, so she's like, "I don't really want neighbors." We lived in a subdivision neighborhood, so it was very annoying. I want there to be someth like privacy and quiet and stuff like that, but I also don't want it to be more than like 10 minutes from shopping. I'm like..
Robin Heinze:
That’s a tall order
...Good luck. Right. Yeah, like we're going to find that.
Adhithi Ravichandran:
That’s very inconsistent. Did you find it?
Jamon Holmgren:
We found it. We totally found it. We are eight minutes from a store, the grocery store. Maybe we're only about a 25 minute drive from Portland and traffic is nice. It was actually a piece of land that her dad owned, not very far from her farm, and so I bought it from her dad. He had owned it since 1970. It was a struggle to get it buildable, because there's all kinds of-
Harris Robin Kalash:
There was nothing, pipes and ...
Jamon Holmgren:
No. Yeah. Well, I mean, there was water, but not sewer and-
Harris Robin Kalash:
Must have been expensive.
Jamon Holmgren:
It was expensive to get it all set up and ready to go, but he sold it to us for a decent price, and so we were able to like afford that. That's a whole thing. When my wife was like, I want it to be ... and we have one neighbor, and it's her sister.
Adhithi Ravichandran:
We should get your wife on the podcast and talk about someday an episode.
Robin Heinze:
We should...
Jamon Holmgren:
She wouldn’t come anywhere near it, but..
Robin Heinze:
...get Chyra to come on and talk about perspective as Jamon's wife, what her perspective of React Native is like.
Jamon Holmgren:
Right. That'd be funny.
Adhithi Ravichandran:
I should take her to visit India sometime. I live in a very populated city called Chennai. I think she would just faint.
Jamon Holmgren:
You know, she's been a lot more adventurous lately about stuff. We went to Bali, Indonesia, and she planned out all of the activities to explore the island and stuff. So she’s totally been, she grew up on a farm. We literally haven't moved further than five miles from where she grew up, but we've started to explore further away now, and that's been really cool.
Adhithi Ravichandran:
Nice.
Jamon Holmgren:
Anyway, we should probably-
Robin Heinze:
What were we talking about when we stopped?
Harris Robin Kalash:
Jamon was saying something about-
Jamon Holmgren:
If end to end tests were fast and stuff like that. The problem is that end to end tests are not fast and they are flaky even with detox. So, we don't have quite that experience that I would love. Maybe in a few years, someone will come out with something and be like, "Hey, here's this like insanely fast, always 100% deterministic end to end test framework. And then we'll all go crazy for it, and we'll have them on this podcast and everything. And this is an invitation to everybody listening, go ahead and invent that please.
Adhithi Ravichandran:
On that note, I really like Cypress for web. It definitely kind of achieves that, I think. It's super fast, I don't see a lot of flakiness or anything, and I've kind of reduced doing unit tests just because Cypress doesn't feel that expensive or anything.
Robin Heinze:
Is it better than Selenium?
Adhithi Ravichandran:
Way better. I don't know.
Robin Heinze:
I've heard horror stories about Selenium.
Adhithi Ravichandran:
Yeah. Cypress is fast, and Selenium has a lot of dependencies that you have to install. Even setting up takes time, but with Cypress, I think it's all just super fast, easy written in JavaScript, and it's very intuitive and nice.
Harris Robin Kalash:
It's promise-based as well, right?
Adhithi Ravichandran:
Yes, yes. Maybe they should do something for React Native as well, but they do just browsers.
Jamon Holmgren:
I complained on Twitter recently that it would be nice to have Cypress for React Native, but-
Robin Heinze:
Did you really?
Jamon Holmgren:
I did. Yeah. But the thing is it's such a hard problem because you are not dealing with a browser, you are dealing with native code and integrating it with this JavaScript engine in the middle of it, and this somewhat hostile environment to having things go in and changes and stuff.
Robin Heinze:
Understandably hostile. I mean, they're trying to protect the app from outside interference.
Jamon Holmgren:
Right. Exactly. Now, I feel like there's an opportunity for Expo to come in and say, okay, here's the expo testing framework, and it works great and super fast-
Robin Heinze:
Do they not have it? Do they not have an expo specific testing?
Jamon Holmgren:
Not that I'm aware of, but if we have someone from expo on, maybe they could give us more insight into their testing philosophy. But one thing I like about expo is that it's like a specialized browser, is what expo is. And so, they totally control the environment, they know exactly how things are going to work and they could totally make something that's more-
Robin Heinze:
Cypress and expo.
Jamon Holmgren:
I like it. I like it.
Robin Heinze:
Match made in heaven, come on. Let's make it happen.
Jamon Holmgren:
React Native comes with ... by the way, you should read the testing documentation on the React Native website. It's very good, and you should check it out there. It goes through a lot of different strategies and all the different tools. React Native comes out of the box with Jest, which is a test runner and test framework, mainly for unit tests, but they also run other types of tests. We used to use Ava, which I think arguably has a nicer API than Jest. It's a pretty syntax and has a really nice feel to it. It's just really slow compared to Jest. That's the problem. Jest used to be slow and then they rewrote the whole thing, and now it's insanely fast.
Robin Heinze:
Didn't we use Ava because it was faster than Jest?
Jamon Holmgren:
Yeah. And because it feels nicer, but Ava has fallen behind in some ways. And because Jest is the default, it's just easier to go, okay, we're going to use Jest instead of Ava.
Harris Robin Kalash:
Ava has much less maintainers than Jest.
Jamon Holmgren:
Exactly.
Harris Robin Kalash:
Jest has the Facebook team.
Jamon Holmgren:
Right. With Jest, there's normal matches, like expect something to be something else. But then, you also have described blocks and all of these things that we've come to expect from testing frameworks.
Robin Heinze:
Come to expect?
Jamon Holmgren:
That matches our expectations.
Robin Heinze:
There you go.
Jamon Holmgren:
One of the things we do with Jest is component testing, specifically snapshot testing using the Storybook. I think it's called Story Shots.
Robin Heinze:
Story shots, yeah.
Jamon Holmgren:
Story shots. That's from Chromatic, I think is the company. That's something where we can actually take a component and take a snapshot of its internals ... Well, not really its internals, but how it's set up, and then compare it to previous versions of that snapshot.
Robin Heinze:
You pass a bunch of props and then you render it and take a snapshot of what rendered. And then if you change anything, it shows you a diff.
Jamon Holmgren:
What's the benefit of that?
Robin Heinze:
The biggest benefit is, is mostly just making sure that something you changed in shared styles or shared theming or configuration doesn't affect things in your app that you're not expecting. It's just a regression test, a quick smoke test so that you are aware every time you make a commit, what you're impacting throughout your app and whether it matches what you would expect.
Jamon Holmgren:
I feel like snapshot testings are slightly controversial.
Adhithi Ravichandran:
Why?
Jamon Holmgren:
Would you agree with that?
Adhithi Ravichandran:
Why do you think that?
Robin Heinze:
I think some people think they're not useful.
Jamon Holmgren:
Yeah. Some people think they're not useful. I'm kind of one of them, to be honest.
Adhithi Ravichandran:
I think if you have good end to end tests, it defeats the purpose to have a snapshot test. But in the case you don't have end to end tests, I think snapshot tests are pretty useful. And it's quite easy to set up.
Robin Heinze:
I mean, that's basically why we use them. We use them because almost always, or in most of the projects I've been on, the client isn't willing to invest in end to end testing or doesn't have a time, doesn't have a budget, it isn't a priority, and the snapshot tests are a very inexpensive way for us to get some level of UI regression coverage.
Jamon Holmgren:
I guess my criticism of Snapshot tests is that often, they become background noise and you just update them and move on without really looking at what's going on. Now, maybe that's just my personality or-
Adhithi Ravichandran:
Update Snapshot, move on.
Jamon Holmgren:
Right, exactly.
Robin Heinze:
Yeah. I mean, that's a very valid criticism of them. I think that happens quite a lot. I know I'm guilty of that often, but I still think for how little they cost, because our workflow involves using Storybook anyway, most of the time we're building our components in a vacuum because we want them to be pure or dumb. I hate the term dumb components. I always cringe whenever I say dumb components. I'm like, it's not dumb, it's a piece of software. I just build that, and I'm proud of it, not relying on any context other than what props are passed to it. And that's how we build them using Storybook. And then snapshot, you could just add snapshot tests, like a couple config files, and it's there. So, it's very inexpensive to add them on top, and I don't see why we wouldn't-
Jamon Holmgren:
Yeah, that makes sense to me. Now, what I liked, it's semi similar, but I think a little more useful in my opinion, is visual regression testing. What that is, and this was when we did native apps, we had a thing called motion juxtapose. What it would do is when you ran it, it would spin up a little Sinatra app, which is like an express app, but in Ruby. You could load up local hosts 40, 40 or something, and it would pop up this page. And then you run your tests, and it would click through all of your screens and take a actual screenshot of it and compare those screenshots to previous screenshots you already had in the system. You can have a fudge factor, it can change a little bit, maybe the time in the corner might change or whatever. I believe there even ways to block out sections, don't compare this one spot because that's going to change every time, but it would check and see if like hey, a button moved or something got moved around or the screen changed or maybe there's an error and it pops up in that.
Jamon Holmgren:
And then, what would happen is it would fail and you could just reload that webpage, and it would show you side by side, the two screenshots as well as a diff that would show in red the differences between the two screenshots. And then if it was like, no, I meant to do that, you just click a button, except it would delete the old screenshot and put the new screenshot in its place when you commit it. In the pull request, it would literally show the new screenshot in the Github diff. You'd see the new screenshot of like, here's the old screenshot, here's the new screenshot, the image changed, and I meant to do that. Now normally, when we do a pull request, we'll often screenshot what we did and just put it into the thing, but this was automatic and this is a thing that would literally fail your test if it was different. I loved that. It was such high value. It was exactly what you're talking about, it was one line of code and you can snapshot.
Jamon Holmgren:
You'd just be like, hey, snapshot this screen, and then click, click, click, snapshot again, click, click, click snapshot again. It was like this amazing way to get huge confidence in your changes without writing a lot of code. I feel like that is a better direction to go, but unfortunately, I haven't run across a very good way to do visual regression testing in React Native that is similar to motion juxtapose. Maybe there's something out there, and I need to probably do a little more research on that, but that's something that I really wish existed. I've talked with the Chromatic, the company behind Storybook, they have this for web and it works great, but they haven't invested in the React Native side. Probably for the same reasons.
Robin Heinze:
Yeah. It's a lot more difficult.
Jamon Holmgren:
Yeah.
Adhithi Ravichandran:
Does it compare pixel by pixel? Is that what you're trying to-
Jamon Holmgren:
It does. It's using probably image magic or something along those lines to check, and it'll actually generate an image with the diff, and it can tell you the percentage differences, and then you just adjust that fudge factor. Once in a while, you'd get a screen that would just jump around for some reason, and that'd be really annoying. Didn't actually happen in real life, but-
Harris Robin Kalash:
But I remember hearing a talk about exactly this at Chain React. Didn't someone talk about screenshot testing with React Native in Percy?
Adhithi Ravichandran:
Yes, yes. I remember that.
Jamon Holmgren:
I don't know. What's what's Chain React?
Harris Robin Kalash:
Chain React 2019, Carly Litchfield.
Jamon Holmgren:
Yeah. I do remember that. That's right. I was going to follow up and look over that whole thing, and I never did.
Harris Robin Kalash:
I actually looked at Percy, and I don't think they ever released it. I think that was just a proof of-
Jamon Holmgren:
Oh, that's right.
Robin Heinze:
Yeah, when you Google it, you get Carly's talk.
Adhithi Ravichandran:
It says they do visual testing by detecting and reviewing visual UI changes, so that sounds very similar to what you were talking about.
Harris Robin Kalash:
Yeah. But they haven't....sorry, they haven't released anything I'm aware of for Native officially.
Jamon Holmgren:
It looks like this year Percy was purchased by BrowserStack, and maybe that had something to do with it, changing their priorities or something like that. But Percy is a service that does visual regression testing, probably very similar to Chromatic. Having the React Native side of it would be amazing. This is something that has been a little bit of a complaint that I think is valid about React Native is that often, you'll have all these cool tools that you wish you had in React Native, but they only work in a browser context. They don't work in React Native.
Robin Heinze:
Feels like they should work because you're developing what feels like a web app, but not a lot of your React tools don't work.
Jamon Holmgren:
One thing that's not necessarily testing per se, but that helps you achieve the same goals of having more confidence in your code before you ship it is static analysis. There are different ways you can statically analyze code. Obviously, there are things ESLint, and prettier, but one of the big ones that we use at Infinite Red is TypeScript. And TypeScript is of course its own super set of JavaScript that double checks what you are passing into different functions or different areas is what you intend to pass in. How do you see TypeScript fitting into the testing strategy that you do?
Robin Heinze:
I think using TypeScript can, in some cases, and this may be controversial, but in some cases, it can replace TDD. It's almost like not test driven development, but type driven development, because you're defining what you want to see come out of a function or go into a function by just writing the types, and TypeScript is going to tell you whether that's happening as you expect or not. I think it replaces some of that.
Jamon Holmgren:
I think we just lost half our audience, Robin. Thanks a lot.
Robin Heinze:
Sorry.
Adhithi Ravichandran:
I don't use TypeScript. I would definitely like to use it in some project.
Robin Heinze:
It's one of those things where if you haven't used it or you've only done the tutorial or a trivial example, it can seem like that's a lot of work, not a lot of value. I already know how to write JavaScript.
Jamon Holmgren:
Right.
Adhithi Ravichandran:
It reminds me of like the older languages that I despise, so I'm like, do I-
Jamon Holmgren:
Like Java?
Adhithi Ravichandran:
Yeah.
Robin Heinze:
We don't use a strict mode, which I think Jamon has gotten into some controversial Twitter debates about strict mode.
Jamon Holmgren:
Never
Robin Heinze:
Especially if you're just learning TypeScript or you're new to it, turn strict mode off and put types in where it makes sense and you can get practice learning how to define them and how to use them. And if you don't want to put in types, don't put in types, and it will still work. It's just JavaScript. But I think once you're building an app in the middle of a project, using it day in and day out, you start to really see how it reduces your cycles because it'll catch all these little things before you even rebuild the app. Just the other day, I was building a ... it was just a demo app because I was trying to repo a bug that I was going to file an issue on Github for. I did just like React Native in it, and I forgot to use the TypeScript template. So, it was just a JS React Native app.
Robin Heinze:
I was literally just putting a single button on a screen with one other component, and I had to reload the app and go back and change my code four times because of little variable name differences, and little things that TypeScript would have caught immediately.
Adhithi Ravichandran:
I definitely see the benefits here.
Harris Robin Kalash:
Yeah. Yeah. I think outside of testing, it just makes you, in my opinion, much more productive developer. Maybe the first time you use it, you feel the pain, but it's crazy how little I have to refer to documentation when I use TypeScript, library written in TypeScript. I use it in my code. I can almost install it and just use it without reading documentation.
Robin Heinze:
Start typing, it'll tell you what all the props are.
Harris Robin Kalash:
And a lot of times in VS code, the actual example and documentation is right there.
Robin Heinze:
Yeah. The TypeScript hints are really, really useful, because it's like documentation right there in your code.
Jamon Holmgren:
I like to, and I understand, there are actually really, really good developers who disagree with me on this, so your mileage may vary. I'm very happy that Robin agrees with me on this because she's a very good developer. But how I look at it is we can't ... I actually agree with the critics who say I can write apps in JavaScript. I mean, I have, for years. I've literally written JavaScript for-
Robin Heinze:
Of course, you can.
Jamon Holmgren:
It's totally doable. But I'm also a TypeScript fan. So people look and they're like, "Wait, you're on this side, but you're also on that side?" But how I look at it is TypeScript is a tool that helps me ... it's like a pair who's watching over my shoulder, a pair programmer who's watching over my shoulder.
Robin Heinze:
Like, oh, you missed that.
Jamon Holmgren:
Yeah. Exactly.
Robin Heinze:
Oh no, that one's supposed to be a number.
Jamon Holmgren:
Exactly. And to me, I don't look at TypeScript as being an all in language. Things like Elm or ReasonML, those are languages that are designed from the ground up to have true type safety. It's getting better, but you can't really express everything that JavaScript can do in TypeScript. And some people will say, well, then don't write JavaScript that way. Okay, that's fair. But I look at it as, it is a helper for you, I'm going to write JavaScript. It's going to infer the types. Once in a while, I'll say it's useful to me to write an interface here or a type and type this, and then as I'm going, then it will know more about my program, but I'm not going to sit here and satisfy this thing that is really being very strict with me. I know what I'm doing. This is a string, you don't have to like whine about it. I can just tell it what to do, and then move on.
Robin Heinze:
Whiny, hashtag winy TypeScript.
Jamon Holmgren:
Winy TypeScript. But ultimately, I was refactoring ... not just refactoring, but literally rewriting Ignite CLI recently, and I didn't write any tests. I didn't do TDD, I just started writing code. But it was in TypeScript, and I did eight hours, literally eight hours of rewriting the internals of Ignite CLI. And I would test manually, and I had TypeScript with very plenty of implicit endings and things like that where I needed. The next day, I came back and wrote some end to end tests and then ... note, it's a lot easier to write, end to end tests, it's pretty straightforward. I found one bug in all the code that I wrote that was in a template file that TypeScript could not type check for me because it was EJS. So, the tests were great and it was important to write them, but ultimately, the TypeScript stuff really did help me have to ... I didn't have to be writing tests for every single little thing. I didn't even have to write types for the most part. That was the cool thing. It was like it happened automatically for me, it’ss magical.
Robin Heinze:
Yeah. Typescript's implicit typing is actually very strong.
Jamon Holmgren:
It is. Yeah.
Adhithi Ravichandran:
Have you noticed any slowness, like build or compile time? I'm guessing it's going to be a little slower than JavaScript, right?
Harris Robin Kalash:
Mm-hmm.
Robin Heinze:
I haven't really noticed anything.
Jamon Holmgren:
A lot of times, JavaScript has a build step anyway, because you're going from modern-
Robin Heinze:
Yeah. Yeah. The Babel has to do its thing.
Jamon Holmgren:
To be honest, the build step for TypeScript is literally just going through looking for annotations and removing them for the most part. I mean, you can translate too, but-
Harris Robin Kalash:
There's definitely slowness that I've noticed, but it's worth it. It's like just solace in VS code. Sometimes the TypeScript server just takes some time to ... If I open a file and I save it, it just takes a second for TypeScript to infer everything. It gets compounded and worse if you have a monorepo. In a monorepo scenario, I've had a lot of pains with TypeScript. But in a single repo, I've never ... whatever, slowness is really worth it-
Robin Heinze:
We're working with a mono repo right now, and I almost always open each package individually. We have a mobile and a web, and I always open them in VS code individually, because VS code seems to struggle less. And each one has their own separate TS config, so it just works better.
Harris Robin Kalash:
Yeah. I've definitely struggled with that. I've had TypeScript be so slow in a monorepo I’ve worked on as well. Yeah.
Jamon Holmgren:
Static analysis is obviously, like I said, not really testing, but it can fill some of the voids that you might have with testing. Going with static type checking and testing, good test coverage can be a really great way to attack the problem from different angles and allows you to have again, higher confidence in the code that you're shipping. There's a few other things with testing that we haven't really covered. For example, one of the big things is mocking. Mocking is where there might be another system or component or service or something like that, that is a dependency in the code that you're writing. The code needs it to do something, but it's not really the point of the code that you're trying to test. It has its own test, and there might be integration tests that also tested the whole thing together, but you just want to test your thing. Jest, I think, comes with its own ways to mock objects and includes and things like that, where you can just basically build an object that looks like the other object.
Jamon Holmgren:
It has the same interface, but it doesn't really do anything. It just maybe tracks like, hey, I was called this many times with these arguments, and then you can check that later to make sure that that happened, but you don't actually have to go and run that. There are some people that don't really like mocking tests. They say that this is just a potential source of frustration because your mocks can get out of date or they can pretend to be working when they don't. Basically, there can be a lot of issues with the integration side of it when you're doing mocks. It can also hide subtle bugs like maybe timing issues or things like that. So, that's another part of this as well, testing with mocks.
Robin Heinze:
Yeah. We always have to put in mocks for ... I think it's just libraries that have native thinking or native components, because when we run snapshot tests, it can't run the native layer. So I think-
Jamon Holmgren:
Oh yeah, because snapshots are in a different JavaScript context, right? They're more of a web sort of thing.
Robin Heinze:
Yeah. It's just testing the JavaScript output basically.
Jamon Holmgren:
Gotcha.
Robin Heinze:
So, we have to mock things like React Native, i18n and things like that, that your components might be using that just need to work and not use the native layer.
Harris Robin Kalash:
Yeah. When I started testing, mocking was definitely a huge obstacle for me when I first started. I just didn't fully understand why we needed to do it, and then when I did ... It's this obstacle where you want to start testing especially in React Native, and there's all these things, libraries you have to mock. But sometimes, some libraries actually provide us with the mocks, which I find really nice. They export their own mocks.
Robin Heinze:
I think one of my favorite parts of testing is it's related to mocking, but it's like ... what was the Ruby library? Faker?
Harris Robin Kalash:
Faker. Yeah.
Robin Heinze:
And all the various different ways that you can make fake data. That's one of my favorite parts of testing, because you come off with crazy names for people.
Adhithi Ravichandran:
Mocking was something that threw me off too when I first started testing, and it was in Java, I was like, what's going on?
Jamon Holmgren:
I just think mocking anything is just not very nice. Like should we really mock people? Or tests? Sorry, too easy of a joke.
Robin Heinze:
Yeah. You stole it from Todd, Todd made that joke like a week ago.
Jamon Holmgren:
Did I?
Robin Heinze:
Yeah. I asked him to channel an Infinite Red Slack channel about mock, I was trying to mock a MobX State Tree stage read thing, and I asked if anyone knew how to mock it. And Todd said ... he said, I can mock anyone or anything, it's a skill of mine.
Jamon Holmgren:
That's true, but I think he stole it from me because I tweeted about it back in May, so I have proof.
Robin Heinze:
The saga of whose joke it was.
Jamon Holmgren:
That's cool. I'll give Todd credit. He's usually better at it.
Robin Heinze:
The difference is though that you are saying it's not nice to mock people and Todd is bragging about how he can mock anyone.
Jamon Holmgren:
If that doesn’t encapsulate us.
Robin Heinze:
So, that's the difference between Todd and Jamon.
Jamon Holmgren:
So I think that, boy, I feel like we could probably do a whole episode just on TypeScript, because that really got-
Robin Heinze:
We already kind of did.
Jamon Holmgren:
Yeah, we kind of did already. But I think this was fantastic to kind of explore the different types of testing and talk about our different experiences with testing. There's a lot more to talk about here. I would love to hear from people who are listening to this podcast, what they think about testing frameworks and testing strategies with React Native. If there's anything new, they've heard of, feel free to tag React Native Rdio, that's with R-D-I-O, instead of R-A-D-I-O at the end, on Twitter. We ran out of characters. We had to cut one. But I want to hear from people, and we'll probably revisit this topic in the future. There's just so much to go through with the testing side of things. Let's go into our weird bugs part of the podcast. Anybody have a weird bug that they want to talk about? I have one in case nobody else does, but I usually do.
Jamon Holmgren:
Okay. It's unfortunately an unsolved mystery. I wish I could tell you what was wrong, but this is the age old works on my machine problem. As I've talked about before, I'm rewriting the internals of Ignite CLI, and I'm calling this Ignite Flame. That's the code name for this project, is the rewrite.
Robin Heinze:
Stay tuned to Ignite on Twitter for more details.
Jamon Holmgren:
Right? Right. Yes. IR_Ignite, or just follow me because I probably tweet from there more. But the problem I ran into is Brian Sterns, who is one of our principal developers at Infinite Red, he started to test it and help me out with like, okay, is this going to work? He spun up a regular React Native app just fine, worked great. And then, he went to spin up the Expo version because Ignite Flame can do either, expo or regular, and he came back and said, "I'm running into this error. It's just popping up this error." The error was ... please hold, I'm actually going to go look it up real quick.
Robin Heinze:
Your call is very important to us. Please stay on the line.
Jamon Holmgren:
The error was ... well, it came through like, failed to exact prepare script. Npm-run-all, patch and then some arguments. So, it was running this prepare, like yarn prepare script. First off, we couldn't really figure out why it was running yarn prepare, why it was trying to do this. Secondly, why was it failing? Because it should work. And thirdly, why is it working on my machine and not Brian's? So, we had someone else on our team, Darren, another principal dev. He went through the process to spin it up on his ... He actually ended up with the same problem. So, the only machine it was working on was my machine. There's this binary called npm-run-all which allows you to run a bunch of different scripts all at once. I checked, and no it wasn't installed globally, it wasn't like that was a difference in our environment. We tried using envinfo, by Trevor Brindle, which gives you a good snapshot of your environment to see what the differences was. It was hard to really tell.
Jamon Holmgren:
I tried using Solidarity by Gant to try to figure out environment differences, could not figure this out. Eventually, we just went into the package, Jason for the boilerplate that we were spending up, removed the preparer script and then added it back in after we had spun up the app. Like okay, we're going to go ahead and add this as a line in the package Jason after we're done. So, it was basically just a workaround. We don't know why this is running, we don't know why it's failing, we're just going to remove it, wait until everything's done, put it back in. Once you yarned, once you actually pulled in all the dependencies, then it worked fine. That was the cool thing, but it wouldn't work when you were trying to just generate the boilerplate. So, environment differences can be really frustrating, and it's problematic. But we just have a little bit of code in there that works around the problem.
Adhithi Ravichandran:
So ultimately, we don't know why this happened, right?
Jamon Holmgren:
No, no idea.
Robin Heinze:
We may never know.
Adhithi Ravichandran:
I feel like this has happened so many times to me too. It doesn't work, I don't know why it didn't work, and it works now.
Jamon Holmgren:
Yeah. Yeah. Well, I'm glad that we found this before someone just filed an issue and was like, what is this? But I guess, just doing the work around works. Sometimes that's just the best way to do it. It bugs me that I don't know the reason why this failed, but do the work around, move on.
Harris Robin Kalash:
Interesting.
Jamon Holmgren:
Very cool. Well, I think that's it for this episode of testing strategies, tools and frameworks. Really happy everybody came along for the ride to listen to us riff on this. As always, you can find us on Twitter at @ReactNative R-D-I-O. Harris, where can people find you on Twitter?
Harris Robin Kalash:
They can find me at @Brunostman, which is B-R-U-N-O-S-T-M-A-N.
Jamon Holmgren:
Adhithi, where can they find you?
Adhithi Ravichandran:
They can find me at @AdhithiRavi on Twitter. I'd also like to give a quote on quality by Aristotle. It says, quality is not an act and it's a habit. I think that applies to us as software developers too. Hopefully, our listeners can get a little bit of inspiration to test.
Robin Heinze:
I want to cross stitch that and frame it and put it in my-
Jamon Holmgren:
That's fantastic.
Robin Heinze:
... on my wall.
Adhithi Ravichandran:
I had this on my course. I'm like, might as well.
Jamon Holmgren:
That's great. I love it.
Robin Heinze:
Very appropriate.
Jamon Holmgren:
Robin, where can people find you?
Robin Heinze:
I'm at @Robin_Heinze, and that's with an E at the end.
Jamon Holmgren:
And you can find me at @JamonHolmgren, my first and last name on Twitter. We will see you all next time. Again, this episode was brought to you by Infinite Red. Check us out, infinite.red. See you all next time.
Robin Heinze:
Bye.
Adhithi Ravichandran:
Bye-bye.
Harris Robin Kalash:
Bye.