Penetration Testing: Mix it up or stick with it?

After publishing yesterday’s article about how frequently you should get a penetration test, I inadvertently started a discussion on Twitter about another aspect of penetration testing delivery: Should you change providers, or you should stick with who you know?

The argument I usually hear in favour of regularly changing security providers is that it gives a “fresh pair of eyes” on your systems and products. Whilst this can be beneficial, it’s not always the best course of action. However, if you are looking for a new perspective or a new skillset – you may not need to change provider entirely, you could just change to a different tester within the same team.

So, let’s take a look at the pros of staying with your current provider. Firstly, you know them. They’re already on your Preferred Supplier List. You’ve got a point of contact and you know pretty much the level of quality that they offer you. If it’s that time of the year where you need another penetration test, you can drop them an email and get that ball rolling quickly.

Whereas when it comes to changing provider there’s just a little more effort required. You have to find a new provider – we’ve previously written about how to the right provider when it comes to getting started with security testing, but in the context of changing providers, things are a little different.

If you’re changing provider, or tester, the new tester will have to learn your way of doing things. They’ll need to understand your systems and your organisation’s history – so that they understand why things are the way they are. Look, everyone’s got tech debt and old problems hanging around, having to explain all that to a new tester can be frustrating. Plus, you might want the testing (or the report) delivering in a certain way – those should be tailored to your requirements, not just be off the shelf output – and you’ve probably already been through all that with your current tester.

So when should you go through the effort of changing provider? I’d put it down to three things. If you can’t do all three of these with your current provider – you should think about changing company.

Firstly, is your tester up-skilling? When you sit down to talk about new trends and changes in the cybersecurity industry and they talking about all the new things they’ve been working on, the research they’ve been conducting, or perhaps a new certification they’ve just got? – or are they just turning up to work, performing the same actions as usual and knocking off for the day? For example, for us you can ask any member of our team what they’re researching or what skills they’ve been studying, and they’ll be able to tell you something they’re working on. For us, research and learning are core to what we do.

Secondly, will your provider make changes to their approach based on your needs? Perhaps this is as simple as adding something to their report layout that’s useful for you when digesting their findings. For example, maybe you’re just really into CWSS scores for some reason and you’d like them including in the report*. Perhaps you want your report summary in a different format because it’s easier for your board members to consume. If your testers aren’t prepared to make changes to their output for you, they’re not working for you, it’s a commoditised offering and you should consider swapping to another provider.

Finally, if your tester cannot describe what makes their offering or their methodology unique when compared to their competitors – you should look elsewhere.

There can be a benefit to swapping out your penetration tester for a fresh pair of eyes, so some companies choose to do this every year, but there can also be a benefit to sticking with a tester who already has a deep understanding of your systems and organisation. Ultimately, I don’t believe you should make the decision to change tester based on a fixed schedule, certainly not changing your tester just because a year has passed. Instead look at their understanding of your systems, their approach, the work that they’re delivering for you, and if they’re finding issues. If you’re not happy with any of those things, that is what should trigger you to look for another provider.

* Editor’s Note: Please let @HollyGraceful know if your immediate reaction was “CWSS scores aren’t a real thing, I’ve never heard of that!” And had to check the link – I told you, you should always be learning when working in this industry!