Trust in Statistics Sweden hit after incorrect unemployment figures

Employment and unemployment figures represent two of the most important numbers for a country’s economic policy. This autumn, Statistic Sweden’s (SCB) figures for unemployment showed a sudden increase. When the numbers were revised, it turned out unemployment for September was not at 7.1 percent, but 6.0 percent. 

Unemployment figures have been both under and overestimated for just over a year. SCB considers the yellow line to be correct. Source: SCB

Sweden’s Labour Force Surveys (LFS) measure unemployment and employment. The surveys are carried out by calling up a large number of people, asking them whether or not they are in work. The surveys are based on a representative selection of people. The gross selection in 2018 were some 365 000 people, out of whom 189 000 answered. 

The people who are select are interviewed once every quarter for two years. The first interview maps the person’s education, occupation, work situation etc. The following interviews focus more on what has changed.  

LFS is one of the most expensive forms of statistics gathered by SCB. In 2018, SCB gathered statistics worth 586 million Swedish kronor (€55m). In all, labour market statistics cost 150.8 million kronor (€14.16m).

The first to contract out LFS interviews 

In 2015, SCB became the first Nordic statistics office to contract out the gathering of unemployment figures to an external actor in 2015. Consultancy firm Evry won the contract and were initially put in charge of performing 20 % of the interviews. 

“The trial of using an external provider for parts of the information-gathering for LFS has shown it is possible to maintain response frequencies at a lower cost,” wrote SCB in its 2016 annual report. 

After a new tender process in 2017, Evry was given 50 % of the work, beginning in 2018. Problems started emerging relatively quickly, however.

“The previous good results with a high response frequency from the external data gatherer have not materialised, however, and the development is therefore being closely monitored and a number of measures have been introduced,” wrote SCB in their 2018 annual report.

This is how the alarming development in non-response looks with SCB’s own statistics. The non-response is partly people who are uncontactable – the interviewers cannot reach the person, or the person who has been called refuses to be interviewed. The number on the left scale is percent.

“Deficiencies in data collection”

On 17 October, SCB sent out a press release where they warned that the LFS figures were incorrect. “SCB has identified deficiencies in the data collection”. This, according to SCB, was due to “a change in how data was collected” which led to some of the gathered information “being of an insufficient quality”.

The Aftonbladet newspaper looked into what had really happened and identified the call centre in the basement of Evry’s Swedish offices in Solna outside of Stockholm, where Evry employees were working under very precarious conditions.

“We are under constant pressure from bosses to work faster, and there is a lot of fear among people who work here. Since there is no job security, you might lose your job at any moment,” one worker said.

Several independent sources told Aftonbladet there were often incidents which would indicate that fake telephone interviews had been used as the basis of information sent to SCB. This could for instance be situations where an employee would call someone who had already been interviewed, only to be told by that person that it was the first time they had been contacted. 

7,000 interviews a month

When Aftonbladet confronted Evry with information of fictitious interviews, the newspaper was told:

“People who have already been interviewed and who do not want to take part again might sometimes answer that they are not aware of having taken part in a survey already, rather than just saying no. This happens, and we are aware of it. In these cases, the interviewer ends the conversation and puts the person down as a non-response.”

Evry carried out around 7,000 interviews per month. Results from this raw data was sent directly to SCB which according to Evry has had full insight into the working methods used.  Evry has asked for an independent inquiry into what has happened, which will be carried out by the consultancy firm EY.

“No link to non-response”

When SCB decided to end its cooperation with Evry, the head of LFS John Kling said the decision was not linked to the level of non-response.

However, the worrying development of non-response was an important reason for giving the task to Evry in the first place. For SCB this meant cutting 40 staff jobs. Now SCB must find a new external provider which will be more expensive – Evry’s offer was 30 % lower than the nearest competition, and 50 % lower than SCB’s inhouse cost. 

A difference of 1.1 percentage points in unemployment figures for September might not sound like much, but it means the unrevised number was 18 % too high. 

There might be several reasons behind SCB not being able to gather data from an individual or a company. The most common cause is that SCB cannot reach the respondent, or that the person does not want to take part in the survey.

The non-response level is a serious problem. It is feasible that people who engage with survey questions will be more willing to answer than those who are less interested. This could lead to skewed statistics. When possible, SCB uses statistical methods that use people’s background data in order to reduce the effect of non-response.

But where do you draw the line for when the response from the interviewees is too low to produce dependable unemployment statistics? It is becoming increasingly difficult to reach those who are selected for interview, and to get them to answer the questions. This is a problem across Europe, but the Swedish response rate has been low compared to many other countries. 

The response frequency in the Nordic countries and three other major European countries. Source: Eurostat

Labour force surveys are conducted using similar models across Europe. In countries where interviewers make home visits, like in Germany, the response frequency is high. So-called proxy interviews, where one person answers for everyone in a household, is another factor. In some cases, the number of proxy interviews can explain better results, but certain countries like the UK have a low response frequency despite a high number of proxy interviews.

The number of proxy interviews in labour force surveys. Source: Eurostat

“SCB’s Data Collection Department, which was already handling 50 % of the selection, will look after data gathering in future. We will not engage an external provider, but will increase our own gathering – albeit not to the same level as previously. We use complex methods, for instance register data, to safeguard the LFS of the future,” said Johannes Cleris, acting press officer at SCB.