Scoring Rules

Each team will have two distinct scores:

  1. Builder score
  2. Breaker score

Scores can be corrected with bonus and penalty points.

Builder Score

Build-it Phase

During the build-it phase, teams receive points based on the compliance of your code to the specifications.

There are three types of points:

  • API end-point existence awards +10 points
  • Required field compliance awards +30 points
  • Functionality compliance awards +100 points

Teams must get a minimum number of points to be eligible for the next phases. The number of points depend on the size of the team, starting from 300 points for two-members team and +100 points for each additional team member.

Important: Teams who do not qualify for the next phases are not admitted to the final exam.

Break-it Phase

For each vulnerability report, the developer team loses 100 points.

Example: Assume that both team A and team B submit each a vulnerability report for a vulnerability in your code. Your team loses 100 builderpoints for each report.

Fix-it Phase

During the fix-it phase, developers review and address the reported vulnerabilities.

As many teams may have submitted reports for the same unique vulnerability, developers can prove that these reports refer to the same vulnerability by identifying duplicate reports and creating a single patch for each unique vulnerability.

If developers can prove that multiple reports originate from the same vulnerability, developers will gain the points back.

Example: Assume that both team A and B filed a report each for the team C project. Here, team C has already lost 100 points: 100 points for the report of team A and 100 points for the report of team B.

Now, imagine that team C finds out that both reports are about the same vulnerability. This means that team C should have lost only 100 points instead of 100 points. Team C can get back 100 points by writing, committing, and pushing a patch to the repository.

When reviewing reports, developers may realize that the vulnerability is not present or that the described program behavior does not qualify as a valid security risk. In these cases, developers may dispute the report to gain the points back.

Finally, when patching a unique vulnerability, developers get additional +10 points.

Important: Developers must fix one unique vulnerability at a time. Teams will be penalized if they fix more vulnerabilities with a single patch.

Breaker Score

Build-it Phase

For now, there are no breaker points during the build-it phase.

Break-it Phase

During the break-it phase, teams search for vulnerabilities in the other teams’ projects. Once a team identifies a vulnerability, the team reports the problem to developers via a confidential GitLab issue. For each reported vulnerability, the breaker team receives 100 points.

Example: Assume that both team A and team B file a vulnerability report for a vulnerability in the code of team C. Team A and team B receive 100 points each.

Fix-it Phase

As developers submit patches, pentester may lose points.

Example: Assume that team C finds out that the vulnerabilities reported by both teams A and B are for the same vulnerability and that both A and B agree on the fix. In this case, both teams A and B share the 100 points of the vulnerability, i.e., 25 points each.

Bonuses and Penalties

Organizers can assign additional points to teams who spot issues in the specs or in the infrastructure. Also, teams can gain additional points for implementing features that are not requested. We reward this behavior as adding more features mean extending the attack surface and more code to test.

Organizers can also give penalties to team violating the rules.

Bonuses and penalties are part of the total score.

Points Calculation

This section sketches the algorithm we use to calculate points during the three phases of the competition.

Build-it Phase

For each passed test, the Builder team get the points for the type of compliance.

Break-it and Fix-it Phase

The scoring algorithm calculates points starting from a snapshot of the GitLab issues. The scores at any time during the competition are the current score (without bonuses and penalties). The break-it score is the current score at the end of the break-it phase. The fix-it score is the current score during the fix-it phase minus the break-it score.

The scoring algorithm is based on two principles:

  1. Unique Vulnerability Points: Builders are penalized (and breakers incentivized) for unique vulnerabilities. The points for unique vulnerability are 100 points.
  2. Patch Points: Builders are incentivized for patching unique vulnerabilities. The points for a patch for each unique vulnerability are 10 points.

Current Score Calculation

The scorer iterates over each confidential issue and calculates the number of points each issue awards to developers and breakers. The scorer determines the points of each issue by looking at three properties of an issue:

  1. The state of an issue, i.e., OPEN or CLOSE
  2. If the issue is mentioned in a commit message, i.e., it has been patched.
  3. The tags of the issue. For the calculation, we consider INVALID, DECLINED, WONTFIX, or none of the three.

See Vulnerability Reports for more information about the possible states and labels of vulnerability reports.

See Rules for Patches and Git Commits for more information about mentioning issues in Git commits.

Issue is OPEN

If the confidential issue is open, we can have two cases: new issue and reopened (and patched) issue.

New Issue (break-it phase)

In the first case, the issue is new. Here, the breaker (==reporter) finds a new vulnerability, or the breaker reopens a closed issue that was rejected by the developers sometimes before. Either way, the issue has not been patched, and the issue is not mentioned in any commit message.

In this case, the issue awards +100 points to the breaker team and -100 points to the developer team.

Reporters can file new issues only during the break-it phase. New issues during the fix-it phase are ignored.

Example: Assume that both team A and team B submit each a new report for an XSS vulnerability in team C code, with issue ID #1 and #2, respectively. The points associated to each issue is 100 points: team A gets +100 breaker points, team B gets +100 breaker points, and team C gets -100 builder points.

Reopen Closed and Patched Issue (fix-it phase)

The second case is the case of an old issue that has been already fixed but, somehow, either the developer team forgot to close it, or someone reopened it, e.g., the previous patch is insufficient, and the system is still vulnerable.

In this case, the breaker team gets +100/n points, and the builder gets -100/n for each report for the same unique vulnerability, where n is the number of reports about the same unique vulnerability. See Rules for Patches and Git Commits for more details about n and how to mention issues in commit messages.

Example: Assume that sometimes in the past, during the fix-it phase, team C committed one patch addressing the issues #1 and #2 (Please note that, according to the rules team C commited one patch using the commit message patch: #1, #2: fixed XSS).

After that, let's assume that team A, B, or even C reopens one of the two issues, e.g., someone realized that the patch is insufficient and the code is still vulnerable (there may be other good reasons to reopen an issue).

In this case, the points associated to each issue is 100 points (i.e., 100/2): team A gets +50 breaker points, team B gets +50 breaker points, and team C gets -100 builder points.

Issue is CLOSE

When the issue is close, we also have two cases, depending on if the issue has been fixed or not, i.e., whether the issue ID is mentioned in the commit message.

Not Confirmed (fix-it phase)

When the issue is not mentioned in a commit message, the developer team closed the issue without a patch, i.e., they could not confirm the vulnerability. In this case, the developer team may have rejected the report, i.e., it is not a vulnerable behavior, and used the tag INVALID. Alternatively, the developer team was not able to reproduce the issue, e.g., because of insufficient information, and used the tag DECLINED.

In these two cases, invalid or declined reports have no points.

Will Not Fix (fix-it phase)

Finally, the developer team may also decide not to fix the vulnerability and specify that with the WONTFIX tag. In this case, the developer admits that a problem may exist or the developer did not want or have time to check it.

In this case, the breaker team gets +100 points, and the builder gets -100 points.

Confirmed (fix-it phase)

If the issue is mentioned in one commit message, the developer team patched the issue, i.e., they confirmed the vulnerability, verified it, and solved it.

In this case, the breaker team gets -100/n points, and the builder team gets +100/n points for each report for the same unique vulnerability, where n is the number of reports about the same unique vulnerability. See Rules for Patches and Git Commits for more details about n and how to mention issues in commit messages.

In addition, the builder teams will get additional +10/n points for the patch for each report for the same unique vulnerability.

Pseudo-code Algorithm

For each confidential issue i do:

  1. If i is OPEN
    1. i is not patched, i.e., i is not mentioned in any commit message:
      • Breaker team gets +100
      • Builder team gets -100
    2. i has been patched, i.e., the identifier of i is mentioned in a commit message (with n the number of issues mentioned in the commit message):
      • Breaker team gets +100/n
      • Builder team gets -100/n
  2. If i is CLOSE
    1. i is not patched, i.e., i is not mentioned in any commit message:
      1. if INVALID or DECLINED is in tags:
        • No points
      2. if WONTFIX is in tags:
        • Breaker team gets +100
        • Builder team gets -100
    2. i is patched, i.e., the identifier of i is mentioned in a commit message (with n the number of issues mentioned in the commit message):
      • Breaker team gets +100/n
      • Builder team gets -100/n and additional +10/n