This page is optimized for mobile devices, i.e. graphics are shown as static images.

SB01: About the Effectiveness and Benefits of the Corona-Warn-App

CWA Team, on June 26, 2021, from German version (last updated June 15, 2021)

Defining effectiveness and benefits
What are the aims of the CWA?
Does the CWA fulfil its aims?
What is the purpose of evaluating the CWA and what aspects play a role in the evaluation?
Which figures are taken into account in the evaluation and where do the data come from?
What are the benefits of the Corona-Warn-App?
More topics in the next few weeks

Defining effectiveness and benefits

As words can be interpreted differently depending on the context in which they are used, productive scientific debates rely on basic shared definitions of relevant terminology. The following definitions are intended as a foundation for discussion.

Functionality
When it comes to apps, functionality refers to the degree to which an app is targeted to provide users with pre-defined functions that fulfil particular purposes. Functions are useful properties or abilities that accomplish specific requirements. In the case of the CWA, functionality refers to organisational and technological processes.

Effect
Effect is a theoretical quality criterion that can be used to ascertain whether a particular function is having an intended impact. However, when studying the effect, unintended impacts that result from the use of the app also need to be taken into account. As the CWA is aimed at breaking infection chains, its effect can be measured by studying whether its users infect fewer people than those who do not use the app.

Usefulness
Usefulness generally refers to the benefits that a person derives from using something. The usefulness of the CWA consists of its ability to prevent new infections and to reduce the spread of coronavirus.

Aims
An aim can be defined as the motivation behind a purposeful activity or form of behaviour. Whereas the impact (effect) and the achievement of an aim (usefulness) represent the actual goals behind the app, sub-goals are also relevant in this context, because their combined functionality improves effectiveness, provides benefits and can also be measured.

The functions of the CWA system and the interaction between its component parts, which run together very smoothly, play a major role in the effect that it has. Non-functional properties, such as the app’s usability, but also organisational aspects, such as interaction between the actors involved, are also decisive for the app’s usefulness. And importantly, users being able to trust that their data will be processed appropriately, with the necessary data protection and transparency measures in place, are crucial aspects with which to encourage people to actively use the app.

What are the aims of the CWA?

The Data Protection Impact Assessment (DPIA) stipulates that the CWA should undergo an evaluation to assess whether it fulfils its aims and whether the personal (albeit pseudonymised) data on which it relies are processed appropriately. In this regard, the CWA has three aims:

Aim 1 – Provide test results
People who have been tested for SARS-CoV-2 should receive their test results as soon as possible.

Aim 2 – Warn other users
People who have received a positive test result should be able to inform or warn other people who have been in their immediate vicinity of their increased risk of infection.

Aim 3 – Determine risks
People who have been in the immediate vicinity of a person with SARS-CoV-2 should be informed or warned of their increased risk of infection.

These three aims should not be considered separately, as they are interconnected and depend on each other for their effect. A person who has been in the immediate vicinity of someone with SARS-CoV-2 can only be informed or warned that they face an increased risk of infection (purpose 3) if the person who received the positive test result can inform or warn them, and actually does so (purpose 2). Moreover, if this is to happen quickly enough to break the infection chain, the person who tested positive for SARS-CoV-2 must receive their results as soon as possible (purpose 1).

The CWA can be assumed to be contributing towards combating the pandemic if it fulfils these three aims.

Does the CWA fulfil its aims?

The following graphs demonstrate not only that the CWA does indeed fulfil its aims, but also the extent to which it does so.

Fig. 1: Tests registered with the CWA and their results.

Figure 1 depicts the number of tests provided by laboratories during the defined time period. A total of 15,819,640 test results were reported via the app, of which 1,519,371 were positive, 13,933,916 negative and 53,712 were invalid.

Fig. 2: Test results shared via the CWA (TAN and QR code).

Figure 2 shows the distribution of the test results provided via the CWA. A total of 473,974 test results were shared via the app, of which 382,714 were shared via a QR code and 91,260 via a TAN.

Fig. 3: The number and risk level (Transmission Risk Level, TRL) of the daily keys shared via the CWA.

Figure 3 shows the number of daily keys shared via the CWA. A total of 5,254,851 daily keys were shared, of which 2,651,089 had a risk level (Transmission Risk Level, TRL) of 5 or above.

Fig. 4: Users who provided or received warnings via the CWA.

Figure 4 shows the numbers of warnings provided and received via the CWA. Whereas the number of people who provided a warning is known, only data donated by users can be used to estimate the number of people who received a warning. These data suggest that the app generated 314,736 red warnings and 574,163 green warnings. Nevertheless, the actual number (during the specified period) is assumed to be around two to three times higher. During this period, 205,066 people provided warnings via the CWA, which means that each person who did so warned around five other individuals who then also received a red warning.

What is the purpose of evaluating the CWA and what aspects play a role in the evaluation?

The evaluation is aimed at examining the effectiveness and benefits of the CWA.

Evidence of the app’s effectiveness – that its users infect fewer people than those who do not use the app – are based on the following two points:

People who are informed in a timely manner about a potential infection can reduce their contact with other people more quickly and thus prevent further transmission.
The CWA can be used to inform friends and relatives but also strangers about their increased risk of infection.

In order to investigate the first point, an analysis needs to be conducted of the steps that are taken from testing through to a user receiving a warning, in addition to data that would show the length of delay between these steps and throughput rates.

Digitisation enables essential processes to be accelerated and made scalable. The analysis can therefore also provide estimates of effect.

It is difficult to provide figures for the second point due to the way that the app minimises data collection. As such, direct and indirect evidence from user surveys need to be used instead.

The CWA is part of the fight against the pandemic and it both supplements analogue (‘manual’) contact tracing undertaken by health authorities, and ‘informal’ contact tracing undertaken by friends and relatives. However, no reliable data are available with which to perform direct comparisons.

When attempting to measure effectiveness, therefore, it is important to answer the following questions:

Are test results being made available promptly?
How many people are providing warnings?
How many people are receiving warnings?
How many strangers are receiving warnings?
How many people who received a warning were infected?
How long does it take for a person who has been identified as at increased risk of infection to receive a warning?
How many people who receive a warning are warned by people who received a warning themselves?

When it comes to usefulness, further questions about the actual way in which the app is used also play a role:

How many people have installed the CWA and how many people are actively using it?
What is the proportion of tests registered via the CWA in relation to total laboratory capacity?
How many people are providing warnings compared to the numbers of new infections?

Sometimes it is only possible to answer these questions (and account for the two points described above) by developing realistic models and simulations. The resulting figures can be used to make claims about how many cases of infection and the numbers of deaths that are being prevented by the CWA.

Finally, it is important to note that the basic functionality and reliability of the Bluetooth Low Energy (BLE) distance measurements were studied and verified in laboratory experiments by Fraunhofer IIS. We also intend to provide more information about the results of these tests.

Which figures are taken into account in the evaluation and where do the data come from?

The app is to be evaluated in order to assess the contribution it makes to breaking chains of infection. The CWA uses a data-saving approach, which means that only a limited amount of data is available for the evaluation. Therefore, on the one hand, the evaluation relies on the voluntary participation of the app’s users who provide data for research into its effectiveness. Two methods were developed to collect these data: an event-driven user survey event-driven user survey and voluntarily donation of usage data (Privacy Preserving Analytics, PPA). On the other hand, the evaluation also relies on data from central and decentralised components of the CWA’s infrastructure as well as external sources.

An evaluation of whether the app fulfils its aims and is effective, as well as its acceptance and usage, involves the collection and analysis of the following data (cumulative and over time):

Data from user surveys (EDUS and PPA)
Data from back-end components from the CWA’s infrastructure (test registration, sharing and retrieval of daily keys),
Data from additional sources (case numbers, test capacities, Apple App Store and Google Play Store, model calculations and simulations)

Key data from user surveys

The evaluation includes the event-driven user survey (EDUS) and the possibility of regularly providing usage data via data donation (Privacy Preserving-Analytics, PPA).

The event-driven user survey

An online survey was undertaken among users who received a notification between the beginning of March and the beginning of May 2021 indicating that they were at ‘increased risk’. The users were asked about their behaviour before they had received the risk notification and about their planned behaviour afterwards. The second part of the survey was carried out five days later and aimed to determine whether these users had actually implemented their planned behaviour. One of the goals of the survey, therefore, was to find out whether the risk notification provided by the CWA actually led users to change their behaviour.

Data donation

The evaluation also involves an event-independent analysis of technical usage data on the function and usage of the CWA. These data are collected from app users via voluntary data donation.

Privacy Preserving Analytics (PPA) – the analysis of usage data while preserving privacy –helps provide a better understanding of how the CWA is used and enables its functions and user-friendliness to be continuously improved. Relevant data have been collected since the beginning of March 2021.

The data donated by users can be used to examine:

Which events (e.g. a notification of an ‘increased risk’, a test registration, receipt of a test result, key submission to warn others) occur in the app and how often
When these events occur and how, and the interval between events
How users respond to these events
Where and when users cancel processes within the app
The demographic (e.g. age group, district) that these data are associated with

Key data from the CWA’s back end

Data on test registration and the sharing and retrieval of daily keys are central to assessing the effectiveness and benefits of the CWA. These data are provided directly from the CWA’s back end.

Key data from supplementary sources

The RKI has access to other key data sets which are gained from additional sources. These can provide indirect evidence of the benefits of the CWA. They include data from the Apple App Store and Google Play Store. In addition, external (interview-based) studies with different study periods and focuses are also to be used. Although external studies apply different methodological approaches and focus on different issues, the combination of their findings provides important information about aspects of user acceptance and has implications for behavioural changes relating to the CWA.

In addition, conclusions about the current number of cases and the overall numbers of tests that have been carried out are also needed. Some questions, especially those relating to hypothetical scenarios (‘What if ...?’), can only be answered using heuristics, models and simulations.

What are the benefits of the Corona-Warn-App?

The data sources currently available enable initial conclusions to be made about the effectiveness and benefits of the CWA. A detailed analysis of relevant data will be carried out regularly in greater detail over the coming weeks.

A large proportion (88%) of users who were tested and received their test results via the CWA (and provided data via data donation on the delay: 976,084) reported that their test results were available either within 24 hours (724,187≃74%) or 2 days (136,990-14%). (rechtzeitige Wirksamkeit)
A good proportion of users (61%) who registered a positive test result via the CWA and who received their result via the app (771,957) shared their result, and therefore used the app to warn other people (473,974) (see above). (korrekte Wirksamkeit)
Many of the users (data from the EDUS : 73% (18,099 of 24,958)) were surprised to have received a notification that they were at ‘increased risk’. Therefore, the CWA often provides warnings outside of the user’s close and professional circle, and to people who would not be reachable without external support because of the lack of direct contact. (zusätzliche Wirksamkeit)
The majority of users who received a red warning (data from the EDUS: 87% (13,515 of 15,540)) also subsequently underwent testing. (korrekte Wirksamkeit)
The EDUS survey shows that around 6% (792 of 13,493) of the tests carried out as a result of a (red) warning were positive for SARS-CoV-2. A result that is comparable to that from analogue contact tracing. (korrekte Wirksamkeit)
The data gathered via donation, the app stores and the CWA back end roughly indicate that users who share positive test results via their daily key warn around six other users, i.e. six other users receive a red warning showing them to be at ‘increased risk’ (see above). (korrekte Wirksamkeit)

These preliminary figures show that the CWA is effective and useful and that it continues to make an important contribution to breaking chains of infection. Moreover, the benefits associated with the CWA increase with each additional user that installs the app.