Among other things, the event-driven user survey (EDUS) demonstrated:
The Corona-Warn-App uses an approach that minimises data collection and implements high security standards. This means that it only collects function-relevant data and does not gather any other information about its users. In order to answer questions about the functionality, effectiveness and use of the CWA, app users were therefore asked to provide additional data on a voluntary basis. User data have been collected in this manner in two ways: through the event-driven user survey (EDUS), which is the focus of this article, and data donation , which will be discussed in a future article.
In principle, the EDUS works in the following manner. A user who has come into contact with a risk of infection receives a notification, and the low-risk green warning is replaced with a higher risk red warning. In addition to the recommendations displayed for dealing with potential infections and the related procedures, a link to the EDUS survey is also provided. Users who consent to participate in the survey are asked a number of general and more specific questions. The questionnaire is in German and consists of a basic and a follow-up survey. Technical security measures protect personal information and prevent misuse.
Data from the EDUS is only available from CWA users who have received a red warning and have consented to participate in the survey. This approach enables researchers to gain deeper insights into a system that would otherwise remain unexplored due to extensive data protection regulations. The questionnaire also provides researchers with access to the views of CWA users. However, although the EDUS can provide new insights for research, it also has a number of limitations.
First, respondents do not always answer truthfully when asked about issues such as their health. When the questionnaire was devised, care was taken to ensure that it only included questions that were absolutely necessary. Moreover, the questions were grouped into categories that reflected the data-minimising approach. For example, although data about income would be useful for understanding the role of socioeconomic status, a question about this subject could lead more participants to break off the questionnaire or not to answer truthfully. In order to reduce the likelihood of this happening, sensitive personal information, such as age, are summarised in categories. Although these categories are less precise than data stipulating a user’s exact age, they still enable differences between age groups to be identified.
Since the EDUS only surveyed CWA users who had received a red warning, the results cannot be extrapolated to the wider population in Germany or even to all of the app’s users. Nevertheless, the results from the analyses of socio-demographic characteristics can be compared with the overall population structure in Germany.
People who take part in surveys often differ systematically from those who do not and it must be assumed that this also applies to the EDUS. However, as the app does not collect any data about its users, potential differences between app users who participated in the survey and app users who did not remain unclear.
In order to assess the CWA, information is needed from users about the circumstances that led to the red warning and their experiences. This information can be gained from the data collected with the EDUS, as it helps answer important research questions from a perspective that researchers would otherwise have no access to.
Furthermore, data from the EDUS can be compared with data from other sources, such as data gained from CWA data donation, and this can enable further research into the app and its users.
The aim of the EDUS is to collect data about the experiences of CWA users so as to enable us to understand the app’s effectiveness. The questionnaire was kept as short as possible to avoid placing an additional burden on participants who would already be in a stressful situation due to their increased risk of infection. Consequently, not all topics and issues that would have been interesting from a research point of view were covered. A number of specialist societies and scientific experts advised the team when planning the questionnaires. The two questionnaires (one for the basic survey and one for the follow-up survey) constitute the consensus that resulted from these discussions.
The basic survey includes questions about the following topics:
The basic survey focuses on how many times the CWA notified users about an increased risk of infection that they would not have otherwise not known about, and how/whether these users intended to change their behaviour.
The basic survey also includes questions about socio-demographics:
Other studies by the Robert Koch Institute (such as GEDA 2014/2015-EHIS) have collected data on socio-demography in more detail. However, the socio-demographic data collected for these studies are used to calculate various related indicators, such as socioeconomic status. In contrast, the EDUS pursues a more data-saving path with its evaluation approach. Age, therefore, is only recorded using predefined categories, and no information about income or professional status is collected.
The follow-up survey focuses on testing uptake and the results of COVID-19 tests by users who received a red warning. Particular focus is placed on the app’s functionality. Among other things, the questionnaire collects data on whether and in which timeframe a test result was made available via the CWA and, if the test was positive, whether the user used to app to share it. The follow-up survey also asked CWA users about the development of new symptoms. The focus of the follow-up questionnaire, therefore, is on the extent to which people who received a red warning subsequently changed the way they acted.
The questionnaires comprised a total of 31 questions. It also included a request to provide an email address for the follow-up survey. Filter questions were used so that participants were only asked questions that applied to them. On 7 April 2020, an additional question was added to collect data about the number of devices on which the CWA was installed. The purpose was to gauge the difference between people using the CWA and the devices on which the app was installed.
As described above, CWA users who received a red warning were also provided with a link to the survey. At this point, the CWA also creates a one-time password (OTP) and saves it locally on the user’s end device. If the user clicks on the link, the operating system asks for authentication, which is aimed at ensuring that the device is genuine and that the app is actually installed on the device. The end device then transmits the results of authentication and the OTP to the CWA server.
If authentication is successful, the data is forwarded to the survey and saved. Once a user has participated in the survey, the OTP is deactivated to prevent the link from being used again (whether by the user or an unauthorised individual). The basic survey ends with a request for the user’s email address so that they can be sent a link to the follow-up survey. Participation in the survey is voluntary.
Email addresses are saved in a separate database as part of a special process called 2Secure together with the identification number (ID) automatically created by the VOXCO survey tool. This is done immediately after the basic survey has been completed. The link to the follow-up survey is sent automatically by email five days after the basic survey has been completed to participants who provided an email address. Once an invitation to the follow-up survey is sent, the email address is deleted from the database.
Users were able to participate in the basic survey between 4 March 2021 and 7 May 2021. As the link to the follow-up survey is sent five days after the user completes the basic survey and users had 14 days to complete it, the last day on which it was possible to participate in the follow-up survey was 27 May 2021. This means that the entire survey period ran from 4 March 2021 to 27 May 2021.
The survey system collected data automatically and sensitive personal information was deleted at the end of the web session by specifically-designed database triggers. IP addresses were masked by the firewall and never passed on to the system. During this process, the SSL tunnel was not broken, so even firewall administrators would have been unable to assign specific data records to the IP address from which they originated.
In general, samples can be differentiated according to the process with which they were compiled. For example, random samples are required to make valid conclusions about a population. Random samples are devised using a process of random selection that assigns each person in a population a known and equal probability of being included in the sample.
However, it is not always possible to draw random samples. The EDUS sample is limited to CWA users who have received a red warning and subsequently chosen to take part in the survey. The app uses an approach that minimising the collection of data, which means that a lot of information about its users is initially unknown and has to be modelled and queried. As a result, the data collected through the EDUS cannot be used to make representative conclusions about the wider population. Be this as it may, comparisons with the general population can be used to describe the sample and the quality of the sample.
The basic survey was accessed a total of 33,701 times between 4 March 2021 and 7 May 2021. Out of these, 26,094 participants completed the questionnaire. As such, the completion rate for the basic survey was 77.4%. 7,329 participants failed to finish the basic survey because their browser was either inactive for at least 30 minutes or was closed before the questionnaire was completed. 43 people clicked on the survey link, but declined to participate in the study and were therefore excluded from the analyses. Furthermore, data from 234 participants were not recorded because these users accessed the questionnaire via an invalid link. This meant that a browser was used to reload the questionnaire after at least 30 minutes of inactivity. Another participant had a technical problem, and had to be excluded from the sample. 342 potential respondents were unable to take part in the follow-up survey due to technical errors.
In order to provide an estimate of the participation rate for the basic survey, information is needed on the number of people who were invited to take part. Precise figures can be gained by calculating the number of devices that displayed a red warning, which included a link to the questionnaire, during the survey period. Red warnings are triggered by a user sharing a positive PCR test result. It was not possible to register the results of rapid antigen tests during the survey period.
Although the CWA backend can be used to calculate the number of devices that triggered a red warning, the data-saving approach means that the total number of devices that subsequently received a red warning is unknown. Nevertheless, a conservative estimate of the number of devices that triggered a warning can be calculated using data from CWA data donation.
The figures gained from CWA data donation amount to 230,723 red warnings during the survey period. A simple extrapolation (based on the assumption that about one third of CWA users participates in CWA data donation) allows us to estimate that 692,169 devices received a warning, which corresponds to a participation rate in the EDUS of 4%. It is important to realise that there are likely to be systematic differences between the users who participated in CWA data donation and those who did not. For example, users who participated in the EDUS may have been in an environment in which the app is used more actively. This could mean that the users who participated in the EDUS were warned more (or less) often than users who declined to participate. These estimates, should therefore be treated with caution, and it is impossible to provide exact figures.
Of the 26,094 people who completed the basic survey, 23,184 provided an email address for the follow-up survey, of which 21,664 addresses could be used to send out an invitation. A technical error meant that 342 respondents were unable to take part in the follow-up survey. Thus, a total of 21,322 invitations were sent out. This resulted in 15,561 completed follow-up questionnaires. The completion rate for the follow-up survey was therefore 73.0%. 138 participants were excluded due to browser inactivity or because they did not complete the questionnaire. A total of 5,623 invitations to the follow-up survey were not taken up. Each day, an average of 401 people took part in the basic survey and 213 in the follow-up survey.
Over the course of time, the number of participants increased. There was a sharp increase after 19 March, and the figures remained at a similarly high level throughout the remaining survey period. The number of participants who took part in the follow-up survey also increased in a comparable manner. However, the increase began five days later, due to the interval before the invitation link was sent via email. The increase in the number of participants is probably related to an update that improved the way in which risk was calculated and in response to infection rates at the time.
Further fluctuations in the daily number of participants were identified. However, these were related to issues such as the fact that fewer tests were carried out on weekends and thus fewer positive test results were shared. In addition, slight changes to the daily number of participants over the survey period can also be explained by developments in incidence rates in Germany.
As stated above, the EDUS uses a self-recruited study population, and, therefore, the sample cannot be assumed to be representative of the wider population. Nevertheless, it can still be compared with the total population in Germany in order to evaluate and describe it. The socio-demographic information that was collected by the basic survey is ideal for such a comparison. The following begins by discussing sex and age, before examining educational distribution in more detail.
Data on sex were collected using two questions. First, the participants were asked about the sex that had been recorded on their birth certificate. Second, since not everyone identifies with the sex they were assigned at birth, the respondents were also asked about which sex they felt that they belonged to. For the comparison, census data were used. However, since census data only take sex assigned at birth into account, only responses to the first question were used for the comparison with the total population.
24,925 respondents provided information about the sex they had been assigned at birth: 11,385 (45.7%) answered male and 13,540 (54.3%) answered female. 1,169 people did not answer the question. 11,236 (45.1%) respondents went on to say that they identified as female, and 13,436 (54.0%) as male. 48 respondents (0.2%) indicated a different sex to the sex they had been assigned at birth, and 180 people reported that they did not know their sex (0.7%). 1,194 people declined to answer the question. Overall, 98.0% of male and 98.5% of female participants identified with the sex that they had been assigned at birth.
|What sex was entered on your birth certificate when you were born?||Number||Percentage||Percentage (valid)|
|- not specified -||1.169||4.5%|
|Since not everyone feels that they belong to the sex they were assigned to at birth: which sex do you identify with?||Number||Percentage||Percentage (valid)|
|- not specified -||1.194||4.6%|
The majority of the respondents were aged between 50 and 59 (mode), the median age group was the 40- to 49-year-olds. Under-18s were extremely underrepresented, but the CWA is only recommended for users aged 16 or above. As such, anyone under 16 should not have completed the survey. In addition, a significantly smaller proportion of people aged 80 or above took part in the survey compared with their proportion of the general population.
The methods used by a survey for data collection have an impact on the people that it can reach.
|How old are you? Please select the appropriate category.||Number||Percentage||Percentage (valid)|
|Under 18 years||385||1.5%||1.5%|
|18 - 29 years||3.657||14.0%||14.7%|
|30 - 39 years||4.765||18.3%||19.1%|
|40 - 49 years||4.990||19.1%||20.0%|
|50 - 59 years||6.253||24.0%||25.1%|
|60 - 69 years||3.416||13.1%||13.7%|
|70 - 79 years||1.215||4.7%||4.9%|
|Over 79 years||249||1.0%||1.0%|
|- not specified -||1.164||4.5%|
A comparison of the age and sex distribution found in the sample with that from the general population highlights a number of further differences. Female participants are particularly overrepresented in the 18-to-59 age groups, and the same applies to male participants in the 30-to-59 age groups. This means that more younger women than younger men took part in the study. In addition, a slight overhang can be observed in both sexes among 60- to 69-year-olds.
In order to minimise the amount of data collected and still enable conclusions to be made about the participants’ level of education, the survey only queried information about school-level education. A majority of respondents (15,366 or 61.6%) stated that they had qualifications that provided entry to university or to a technical college; 6,694 (27.9%) reported having completed secondary school or a polytechnic secondary school up to 10th grade. Only a small number of respondents reported having a lower level of qualification or “no qualifications” at all. Very few school age pupils answered the questionnaire.
Other studies confirm that it is not uncommon for people with a higher level of education to participate more often. Nevertheless, sample bias in this survey is higher than in other studies. The RKI regularly conducts health surveys as part of nationwide health monitoring. For the German Health Update (GEDA), more than 20,000 people who live in Germany and are at least 15 years old were surveyed between November 2014 and July 2015 (GEDA 2014/2015-EHIS). The sample was drawn from official population registers and potential participants were contacted in writing. Participants were able to take part using an online questionnaire, among other methods. Of the respondents who took part in GEDA 2014/2015-EHIS online, 54.9% reported having university-entry school qualifications. All of the lower-level school-leaving qualifications (that were provided) were reported more frequently in the GEDA 2014/2015-EHIS study than in the EDUS. Once again, these differences can be traced back to the survey method.
|What is your highest-level school leaving qualification?||Number||Percentage||Percentage (valid)|
|School qualifications that provide entry to university or to a technical college||15.336||58.8%||61.6%|
|Secondary school leaving certificate, polytechnic high school until the 10th grade||6.946||26.6%||27.9%|
|Secondary school diploma, polytechnic high school until the 8th or 9th grade||2.058||7.9%||8.3%|
|Completed school after a maximum of 7 years||43||0.2%||0.2%|
|No school leaving certificate||59||0.2%||0.2%|
|No qualifications yet, I still go to school||319||1.2%||1.3%|
|- not specified -||1.197||4.6%|
|What is your highest-level school leaving qualification? (GEDA)||Number||Percentage||Percentage (valid)|
|School qualifications that provide entry to university or to a technical college||5.855||54.9%||54.9%|
|Secondary school leaving certificate, polytechnic high school until the 10th grade||3.313||31.0%||31.1%|
|Secondary school diploma, polytechnic high school until the 8th or 9th grade||1.332||12.5%||12.5%|
|Completed school after a maximum of 7 years||37||0.3%||0.3%|
|No school leaving certificate||119||1.1%||1.1%|
|- not specified -||18||0.2%|
Overrepresentation of higher education is particularly evident among the sample compared with the population as a whole. Whereas 61.6% of EDUS participants reported that they had finished school with qualifications that provide entry to university, this only applies to 33.5% of the general population in Germany. Accordingly, lower-level qualifications are represented less in the sample, and in some cases much less, than in the general population.
A user’s location was determined using their postcode. For ease, users were only asked to provide the first three digits of their postcode. A total of 24,836 users provided a valid response. 149 answers were treated as invalid as the postcodes provided could not be assigned to a particular location. Only 6 postcodes from throughout Germany were not provided.
In order to be able to compare the data with information about newly-reported SARS-CoV-2 infections, the three-digit postcodes had to be assigned to districts. The number of participants in a particular postal area was weighted depending on its number of inhabitants and these individuals were then distributed to the respective district. The following diagram shows both of these figures and demonstrates that more participants live in urban than rural districts.