11.06.2008
Damned lies and statistics
Bob Potter explains how Tower Hamlets Labour-controlled council has used 'market research' to claim support for its privatising attacks on council tenants
Like every other Weekly Worker reader, I am sometimes accosted in front of shopping precincts, or at my front door, by canvassers wanting to interview me as part of their ‘market research’ project. Which daily newspapers do I read? Which TV soaps do I watch? Assured that such details are “completely confidential”, I am asked for my age, occupation, ethnic and married status. Often the researcher works for a company like Market Research UK (MRUK).
Up to a year or so ago, I had not thought much about these day-to-day research activities. In a capitalist society, it makes sense that manufacturers should ascertain what factors make their product more ‘saleable’. Indeed, a future socialist society will surely also consult ‘public opinion’ to determine the required artefacts to be produced - although the focus guiding such research will no longer be profitability.
My neglect of this aspect of everyday experience has changed! A leaseholder living in the London borough of Tower Hamlets for more than 20 years, I had been vaguely aware of the council spending lots of money on so-called ‘independent’ surveys. During the three years preceding November 2007, for example, the council paid MRUK £285,572 for 35 surveys relating to the ‘Housing Choice’ programme; as an active campaigner against council attempts to transfer our estate to the Guinness Trust, I became very suspicious of the many surveys reporting “growing support” for a handover to the private sector - a finding difficult to believe, as I had knocked on so many doors and found the majority of residents anxious to sign a petition against transfer.
I made repeated requests to the council’s housing directorate for the MRUK research documents to be made public. Each request was refused - I was assured the documents would be made available after completion of the Housing Choice programme!
During the same three-year period, MRUK also surveyed Tower Hamlets residents on general housing service policies (at a cost of £24,094) and just prior to a case before the Leaseholder Valuation Tribunal (LVT) in 2007, MRUK explored ‘leaseholder satisfaction’ throughout the borough.
Research companies other than MRUK are also used, but I have neither the details nor costings of those investigations. The evidence I do have suggests the council spent at least half a million pounds on so-called ‘market research’ during that period. Residents pay the bills for these surveys, but precisely what the researchers did and what they ‘discover’ is not disclosed. One reads the press reports of the survey ‘findings’ (as scripted by the lead councillor) and one’s gut feeling is ‘this just cannot be true’.
The MRUK Home Ownership Leaseholder Survey cost the council £8,753 - it was commissioned in November 2006, and the report became a ‘public document’ when presented as council evidence at an LVT on May 2 2007. In this case, the main respondent opposing the council’s management charges was the local Tower Hamlets Leaseholders Association (THLA); my wife and I joined the case as independent respondents and hence the full MRUK report was given to us in hard copy … but not the questionnaire.
Now ponder on this awhile. If a child comes home with a school report saying they have failed their maths exam because the correct answer to a question was ‘6’, but they had written ‘7’, any conscientious parent would ask to see the question paper, wouldn’t they? Knowing something about research methods, my first glance at the document aroused my suspicions. I asked for a copy of the questionnaire. Only after six weeks of badgering, speaking with MRUK and the Freedom of Information office, did the council grudgingly hand it over (and include it in their bundle of evidence).
The questionnaire indicated that, in a statistical sense, the survey findings were gibberish. To the layperson, the document may appear very professional; with repeated references to “research methods” and “statistical procedures” - terminology that sounds impressive to a non-statistician, but as a (retired) college lecturer, having taught inferential statistics for many years, my own immediate assessment of the MRUK survey was it would fail an elementary statistics examination!
Don’t be frightened of a little maths
Inferential statistics is a research tool used to make a reliable assessment, or inference, from information about, or opinions held by, a large number of people. (Readers may well be aware the recognised founder of statistical science was the Marxist, Karl Pearson, who probably met Marx when he was a lecturer at Heidelberg University.) This survey was allegedly set up to discover “details and attitudes” of the leaseholder population in Tower Hamlets. A small sample of borough leaseholders was interviewed by MRUK telephonists.
Common sense is all one needs to appreciate that a sample should be carefully selected, as far as humanly possible, to accurately represent, in miniature, the views of the population as a whole - what then emerges is known in the trade as a ‘representative sample’.
Statisticians have generally agreed practices and procedures. As MRUK claims to be a professional market research organisation, one would expect it to use statistical terms and procedures, as defined by all the standard research/survey textbooks. However, in this study, correct procedures were not used and hence any reported ‘findings’ have no validity. After many weeks spent extracting further evidence from a reluctant council and speaking with an individual introduced to me as MRUK’s “senior statistician”, I soon formed the opinion that the survey was produced to (literally) create evidence of “leaseholder satisfaction with council charges and services”.
Realising a vast number of people suffer a ‘mental block’ as soon as they hear the word ‘mathematics’, I composed an ‘Absolute beginners guide’, written primarily for those with little or no knowledge of statistics. The MRUK report is written in technical jargon, designed to impress the naive reader with the expertise of the author. My intention was to enable readers to discover for themselves the many flaws, contradictions and elementary arithmetical errors in the document, paragraph by paragraph, throughout the survey.
There are sufficient methodological flaws in the project, overall, to cast doubt on any of the conclusions drawn by the research company. Below are four examples.
1. There are several ways in which statisticians collect samples. One method, quota sampling, is almost certainly the method used by your researcher standing on a busy thoroughfare - maybe the requirement is for 50 people aged 20-30 years, 50 aged 30-40, 50 aged 40-50 years and so on. An alternative method would be random sampling, where every member of the target population has an equal chance of being selected.
Now either of these methods of selecting a sample is perfectly acceptable - but it is not possible to use both methods at the same time. So the first page of the survey revealed an important elementary flaw in research methodology - for it states: “The survey involved telephone CATI interviews, conducted by trained social interviewers from MRUK, of a quota-controlled sample of 335 leaseholders (out of a total 500 targeted leaseholders’ addresses).” However, on the second page, it is claimed: “A total of 335 completed interviews were achieved with randomly selected London Borough of Tower Hamlets leaseholders” (my emphasis).
To a statistician, these two claims start the alarm bells ringing, for quota sampling does not and cannot use random selection. Even a newcomer to statistics might well suspect a quota sample will not be a random sample. This is tremendously important for this particular study, because the statistical test(s) claimed to have been used by MRUK are only valid if the sample was randomly selected from the ‘population’ of Tower Hamlets leaseholders.
When this elementary flaw was pointed out to the author of this document, he quickly retracted, stating:
“Despite what the report says, MRUK Research applied no quota controls during the interviewing. That is, it did not set quotas - such as age of head of household, ethnic origins or other leaseholder characteristics - for its interviewers to achieve. It simply attempted to interview all 500 leaseholders in the random sample, from which it achieved a total of 355 interviews.”
2. This leads us to another basic flaw in the MRUK survey: the leaseholder sample was not random; on page 4 we are told the sample was selected from a list of leaseholders compiled by the council: “All leaseholders interviewed in this research had contacted Tower Hamlets’ home ownership section … in the last 12 months.”
Note that even the list (of 500) leaseholders provided by the council was not itself a random selection of the 9,000 leaseholders living in Tower Hamlets (those who had not contacted the council’s home management department were automatically excluded). There is no suggestion that researchers made any attempt to obtain a random sample, even from the list provided - not that such a sample could be categorised as random anyway. Presumably, calls were simply made until there were enough respondents - ‘enough’ being a predetermined minimum figure; or all they could get in the time, or all they could get easily, or all they could get before the money ran out … We do not know, but should not have to guess the answers to these questions; a proper professional report would have told us.
3. Another fundamental flaw with this project was that council officers connected with home ownership participated in the survey design: “... we are grateful to Beverley Philips, Wunmi Odusina, Lesley Owen and Daniel Potgieter for their help in putting together the sample and designing the survey.”
For a proper, independent survey of leaseholder opinion, the research company would require a complete database of Tower Hamlets leaseholders, together with a listing of specific topic opinions sought. Council staff would not have pre-selected the sample, nor would they have helped in “designing the survey”.
4. And finally, on page 3 of the survey the reader is told: “By achieving 335 interviews the overall results from this survey are accurate to within ±5.3% at the 95% confidence level. This means that, for example, if 50% of respondents answered ‘yes’ to a particular question, there are 95 chances out of 100 that the correct figure for all residents in the borough will be between 54% and 66%.”
To a statistician, virtually every comment in the above paragraph is gibberish. A mathematical novice will spot the simple arithmetical error in the last sentence - claiming 50% ±5.3% equals an interval between 54% and 66%; the correct interval being between 44% and 56%.
As for the claim that there are “95 chances out of 100 that the correct figure for all residents in the borough” had been produced, just an ounce of common sense is all that is required to know that, even had the sample been random, even had MRUK managed to get its statistical procedures right, a sample of leaseholders could not be used for inferences about “all residents of the borough”.
What about “confidence levels”? If the researchers claim to be 95% confident that their sample represents the opinion of the leaseholder population, what they are saying is that, if numerous random samples were taken from the leaseholder population of Tower Hamlets, then 95% of those samples would reflect the opinions of the total population. Yet, where there is a combination of different types of questions, each type requires a separate calculation - the data cannot be lumped together and subjected to a single test, as the report implies.
Home truths
Attempting to sort out the many contradictions and confusions in the survey, I telephoned MRUK (on June 11 2007), asking to speak with the senior statistician. I was passed to MRUK director Euan Ramsay, who had served as contract manager for the Tower Hamlets research; he agreed there was “no single test for handling this variety of data”. When, therefore, I challenged his claimed ‘confidence level’, he replied that there is now “an accepted protocol format for landlords”, leaving me with the distinct impression that the wording was chosen not because it accurately described the obtained data, but because it conformed to the ‘standardised requirements’ of social landlords.
I expressed concern regarding the involvement of council officers in the sample selection and questionnaire design. Ramsay explained the council had no database of leaseholders, and so MRUK had to rely on the short list given him by the home ownership manager. He admitted this was not “normal” practice, but added he personally had known the manager for many years and could “vouch for her integrity”.
(I eventually discovered that Ramsay is not a statistician. He advertises himself as an accountant and “self-employed consultant”, with competencies in business planning, financial management, investment options, stock transfers and 10 years’ experience as head of finance for housing associations. But he has no statistical qualifications.)
Given Ramsay’s statement about an “accepted protocol format” now expected by social landlords, am I alone in interpreting that phrase to suggest that MRUK asks its clients something like: ‘Exactly what do you want the research to show?’ Certainly, that interpretation would fit well with a ‘research’ project where the client provides the sample and is heavily involved in designing the survey parameters and questionnaire used.
In spite of all the apparent jiggery-pokery, council witnesses embellished the acquired data further in their attempts to impress the LVT. One key witness stated: “In relation to service levels generally, I have seen the MRUK survey carried out for the council which shows the majority of leaseholders to be generally satisfied with the standard of service provided by the council in relation to its leaseholder management and administration functions.”
This assessment is based on responses to just one question - where interviewees were asked whether, when they telephoned the home ownership section, they were answered politely, given a prompt answer and good advice, and were satisfied overall with that response. So answers to a question about telephone responses were the only ‘evidence’ used to support the council’s claim that “the majority of leaseholders are generally satisfied with the standard of service provided by the council”. Nowhere did the questionnaire ask specifically for an opinion regarding “standard of service or administrative functions”. No wonder the council was reluctant to allow the questionnaire into the public domain.
Many of the surveys conducted in Tower Hamlets were apparently commissioned to influence the Audit Commission and Housing Inspectorate when rating the council’s performance for the purpose of assessing public funds. What is more, it seems the LVT case was being monitored as a pilot study for recharging administrative costs by all London councils. That is why an exposure of the faulty methodology used by MRUK should resonate across the capital.
Almo ‘support’
More recent MRUK research projects have been conducted to show “growing support” amongst residents for the setting up of an ‘arms length management organisation’ (Almo), the alternative approach to stock transfer. Only very limited research details have been made available by the council (and that at the 11th hour), but sufficient to demonstrate the projects were grossly biased - by comparison, the flawed questionnaire used in the ‘leaseholder survey’, discussed above in some detail, appears almost as a model worthy of emulation.
- A “key objective” of the MRUK Almo surveys was “to establish the extent to which the council’s programme of resident consultation had been effective”. Yet interviewees were not asked about the consultation’s effectiveness. Indeed, when asked how they learned about the Almo proposals, no interviewee is reported as referring to the “consultation process”.
- The two MRUK surveys commissioned by the council were separated by less than a month and overlapped - the initial Customer Information Survey was completed in early August 2007, and the Test of Opinion completed in September 2007. Different respondents were used for these surveys and no information is available regarding sample selection. The samples were not ‘matched’ and it is therefore not legitimate or appropriate to ‘gross up’ the data obtained. These surveys claimed to show “growing popularity” for the Almo scheme in the borough - a finding publicised by the council in the local press.
- Before any question was asked, interviewers read the following “introductory statement” to each participant in the CIS survey: “If the proposal for an arms length management organisation (Almo) goes ahead, the council would set up a new organisation called Tower Hamlets Homes to manage its homes, although these homes would remain in council ownership. The aim of establishing the Almo is to benefit from a positive extra £190 million of government funding for improving and modernising the council’s homes - funding that would not otherwise be available to the council … you would remain a secure tenant of the council and keep all your existing rights. You should also benefit from improved housing management services and a greater say in how the housing service is run under the Almo.”
- Interviewees were then asked: “Do you broadly understand what is being proposed with the Almo: namely, to set up a new organisation - Tower Hamlets Homes - that would be owned by the council and that would be responsible for improving and managing the council’s homes?” This was the only question posed to establish the “level of understanding” of respondents of what Almo was all about - ie, the respondent’s own assertion was deemed to be a sufficient test. Despite this, only 43% of tenants and 35% of leaseholders were found to “clearly understand” the implications of setting up an Almo.
- Comparing the outcomes of the two surveys, conducted only a month apart, remember, MRUK reports “a clear shift of opinion amongst residents towards full support for the proposed Almo”. However, what is not obvious from available documents is the modification made to the key question in the second questionnaire. In July, residents were asked: “What is your view of the council’s proposal to set up an arms length management organisation?” Rewritten for the later survey, this question becomes: “Would you support the setting up of the Almo called Tower Hamlets Homes to deliver these improvements?” Nothing like offering a larger piece of candy in order to create a “clear shift in opinion towards full support”
- Finally, to the crucial question. The council application states: “… in every case a clear majority of those expressing an opinion were in favour of Almo”. To arrive at this “clear majority”, researchers omitted all replies expressing ‘uncertainty’ or plain ‘opposition’ from the calculation. Had these people been included, the figure would reduce to 57% of those asked - still a majority, but, looking at the specific questions fielded and the loaded context surrounding them, a less than impressive percentage.
To conclude, I do not claim to know the majority view of Tower Hamlets residents vis-à-vis the establishment of Tower Hamlets Homes - the only full MRUK report I have been able to obtain is the leaseholder survey prepared for the 2007 LVT. Having studied, in some detail those materials I have been able to obtain, it appears obvious the council fears that the general view does not support the application - how else to explain the questionnaire manipulation and inappropriate data presentation?
Those who wish to be actively engaged in changing society cannot afford to remain ignorant of matters statistical. If you are interested in investigating this important area, email bob.potter.at.98@gmail.com and I will send you all related documents.
