Bug Bounty Programme or Penetration Testing: Which is Right For You?

Organisations use penetration testing and bug bounty programmes to identify cybersecurity vulnerabilities and strengthen tech resilience. But which method works better? John Leyden considers the pros and cons of each

The internet is habitually compared to the wild west, a lawless domain where cyber criminals continually hunt for vulnerabilities and flaws that they might be able to exploit to steal data and crash systems.

You may or may not be aware of traditional penetration testing (if you’re not, read on, we’ve got your back) by specialist consultants. Hiring penetration testers is one option for organisations looking to batten down their hatches. Option two is an approach analogous to the bounty hunters in spaghetti westerns. And this second route is a method growing in popularity.

Bug bounty programmes are designed to bring law and order to the web’s wild west by offering independent ethical hackers or security researchers a financial incentive to find and report vulnerabilities before the bad guys exploit them.

Although somewhat complementary, penetration testing and bug bounty programmes have different scopes and goals. They are not interchangeable. One discernible differentiator is that annual penetration tests are required by the Payment Card Industry Data Security Standard (PCI DSS) for PCI compliance for organisations that handle credit card transactions and the like.

What’s the difference between pen testing and bug bounty programmes?

Penetration testing involves conducting a simulated attack on a system or network to identify vulnerabilities and assess security measures. In short, pen testing evaluates the security of a network or system by trying to break in.

On the other hand, bug bounty programmes allow external, independent researchers to report vulnerabilities they find on a company’s system, often in return for a one-off payment. Ka-ching!

“Having a bug bounty programme in place is a cost-effective way to enhance an organisation’s cybersecurity posture as only the results are being paid for” Michael Adams

The cost of a bug bounty programme depends on what rewards are offered, and some organisations opt to give recognition instead of cash. When no financial compensation is provided, it’s known as a vulnerability disclosure programme (VDP). Of course, this isn’t always hugely popular with the research community.

VDPs are commonplace in the industry and often act as stepping stones towards rolling out more expansive bug bounty programmes.

Leon Teale, a senior penetration tester at IT Governance, comments: “A bug reward programme can be easily implemented, and bug bounty platforms can be relied upon to help with programme setup. Companies must, however, carefully evaluate potential risks and decide how rewards will be distributed according to [vulnerability] severity.”

Teale continues: “To successfully award a bug bounty submission, companies must manage submissions and triage incoming requests. To introduce a bug bounty programme, companies must determine whether it is more advantageous and cost-effective than a standard pen test.”

The wisdom of crowds

Fredrik Almroth, a security researcher and co-founder at Detectify, a Swedish crowdsourced security start-up, argues that traditional approaches involving pen-testing and code scanning “only provide a partial view of risk and exposure in an organisation”.

“To meet regulatory and compliance requirements, incorporating pen-testing is crucial,” according to Almroth. “Pen testers establish a baseline, which ideally should be complemented with automated [and ongoing] crowdsourced security.”

Go faster stripes

The pitch is that crowdsourced security, through platforms like Detectify, HackerOne, BugCrowd, and Intigriti, uses a large group of ethical hackers to continuously identify more vulnerabilities than regular penetration testing alone can achieve.

“Penetration testing is a widely used manual method for detecting vulnerabilities in applications, but it has limitations, including failing to keep pace with the development speed of modern applications” Fredrik Almroth

“Penetration testing is a widely used manual method for detecting vulnerabilities in applications, but it has limitations, including failing to keep pace with the development speed of modern applications,” Almroth explained. “Automated, crowdsourced ethical hacking platforms have access to vulnerability information from a broad range of hackers around the world, which an individual pen tester usually does not have access to.” This, Almroth concludes, makes bug bounty programmes an “effective and scalable solution for improving internet security.”

Size matters

Organisation size is a significant factor for companies in finding the right blend of conventional and crowdsourced security testing.

Start-ups or smaller companies may benefit more from performing penetration tests as they produce official reports that can be issued as proof of security testing. According to IT Governance’s Teale, a mix of penetration tests and bug bounties is a “good idea for better coverage” for larger companies with a significant online presence.

Ethical hackers

A bug bounty programme can be private (by invitation only) or public. With a managed bug bounty programme, an organisation sets the programme’s engagement rules including: assets in and out of scope; types of vulnerabilities; permitted testing methodologies; reward structure; and payment model.

Companies pay HackerOne through a subscription model. “The cost varies widely depending on the size of the programme,” explains Kayla Underkoffler, lead security technologist at HackerOne. “Our most expensive programmes are the largest ones to manage – typically large enterprise organisations running public bug bounty programmes.”

A bug bounty programme run through an established platform can help organisations offset operational costs, according to HackerOne. “Organisations can address the security talent shortage and minimise hiring costs by tapping into an established community of ethical hackers,” Underkoffler explains.

Zoom in

Video conferencing software firm Zoom offers a private bug bounty programme on HackerOne’s platform. Zoom offers rewards ranging from £200 ($250) to £40,000 ($50,000) depending on the severity of the bug found. In the last fiscal year, it awarded $3.9 million in bounties to hundreds of researchers and has granted over $7 million to date since the programme began.

Michael Adams, CISO at Zoom, offered guidance for other vendors looking to roll out bug bounty programmes.

“Before introducing a bug bounty programme, it’s important to consider the business objectives,” Adams said. “These will help determine the scope of the programme, whether it runs as private or public, and the rewards system, as this may attract a range of participants from beginner bug bounty hunters to full-time professionals.”

While organisations of any size would likely benefit from a bug bounty programme, companies on the larger side are generally better equipped to sustain a successful bug bounty programme. “Having a bug bounty programme in place is a cost-effective way to enhance an organisation’s cybersecurity posture as only the results are being paid for, rather than the time spent on searching,” according to Adams.

Bug bounties really come into their own in helping to identify edge-case vulnerabilities or anomalies that only occur in certain circumstances. “That’s where the ethical hacker community can perform a vital function in the continuous testing and probing of technologies,” Adams says.

Your mileage may vary

Bug bounty programmes are much in vogue within the industry, but not everyone is blown away.

Simon Cundy, Red Team Leader at NormCyber, argues that pen testing provides a more proactive approach.

“Penetration testing is generally more comprehensive and in-depth than bug bounty programmes, as it simulates real-world attack scenarios” Simon Cundy

“Penetration testing is generally more comprehensive and in-depth than bug bounty programmes, as it simulates real-world attack scenarios and provides a complete assessment of an organisation’s security posture,” Cundy tells Assured Intelligence. “In other words, penetration testing is designed to identify vulnerabilities before they are exploited by attackers, while bug bounty programmes rely on security researchers to identify and report vulnerabilities after they have been discovered.”

Anders Reeves, CEO at CovertSwarm, a UK-based ethical hacker and cybersecurity solution provider, warned that bug bounty programmes can be of varying efficacy.

“The third party [bug bounty platform] acts as a marketplace provider and doesn’t have the same level of assurance or quality rigour that a more dedicated agency or service provider can offer,” Reeves explains. “As a result, what can happen with bug bounty programmes, if they’re not very well managed, is you can end up with highly variable findings.”

Reeves adds: “You also have variable reporting quality. Not everyone’s written word is going to be as good. They don’t necessarily have the same QA [quality assurance] processes that the consultant would have in place.”

According to Reeves, clients tend to “hold back a little bit” so as not to expose the things that might damage their reputation if found to be vulnerable.

“There is a weird dynamic where clients can often engage with a bug bounty programme, but not really truly expose themselves,” Reeves says, adding that this reticence limits the utility of bug bounty programmes.

Backstop to basics

Mia Landsem, ethical hacker and pen tester at Orange Cyberdefense, said that while pen testing gives clients more control over the security assessment process, bug bounty programmes offer a cost-effective way to roll out diverse and continuous testing.

Bug bounties “attract a diverse group of testers from different backgrounds, which means different testing methods and ideas,” according to Landsem.

Landsem concludes: “Pen testing and bug bounties should not necessarily be seen as competing technologies, due to the fact that their end goal and aim is the very same: to detect vulnerabilities within an organisation’s system before the bad guys get there first.

“Even though both processes are conducted under different circumstances, they both manage to assess an organisation’s current security posture and aid to identify potential breach points.”

Tom Eston, VP of consulting & cosmos delivery, Bishop Fox, an expert in offensive security, described bug bounties as a “backstop to what should be a strong, continuous offensive assessment programme”.

“When deployed continuously and broadly across multiple potential attack surfaces in an enterprise (applications, clouds, networks, etc.), penetration tests can catch the most impactful and dynamic exposures in a business infrastructure,” Eston said. “But no process is perfect, which is why oversights or unexpected and unintended scenarios are what a complementary and well-run bug bounty programme can identify.”

Eston warns, however, that bug bounty programmes can be counterproductive if implemented on their own, so they should be “instituted only after a company has established a solid security assessment programme”.

So…bug bounties or pen testing?

Bug bounties offer broader scope and ongoing testing with greater flexibility. However, their findings need to be validated and analysed for accuracy and can lack the thorough coverage and documentation that pen tests can offer.

Businesses need to carefully consider how best to spend their security budget and commit resources to find the best mix between pen testing and bug bounty programmes in their security strategy.