Arolwg Masnach Cymru, 2022: gwybodaeth am ansawdd a methodoleg

This TSW quality and methodology information (QMI) report discusses the technicalities (e.g., sample and response rate distribution, compliance cost, etc.) for the 2022 data collection. It also encompasses the overall methodology employed over multiple years and outlines the quality aspects of the survey methodology and statistics produced. The separate main statistical release details the results of the survey. Accompanying data tables are published alongside the report, and provide the latest estimates and revisions. Detailed analysis from previous surveys can also be found online. 

Unless otherwise stated the year of TSW referred to throughout this report is TSW year five (TSW 2022) which collected 2022 data.

The Trade Survey for Wales (TSW) generates ‘official statistics in developmen (OSR)’ ^{[footnote 1]} estimating the value (in current prices) ^{[footnote 2]} of trade conducted by businesses in selected sectors located in Wales. It collects data relating to the values of annual trade (sales and purchases) that business units located in Wales conduct with other parts of the UK, the EU and the rest of the world (Non-EU countries).

The sales and purchases are broken down by type (goods or services) and the broad geographical location of the trade partner. The questions cover both intra-UK and international trade. Survey responses allow statistical estimates to be produced covering the value of trade for businesses in Wales and the number of businesses conducting this trade. 

To date the survey has been run annually since 2019 when the Welsh Government (WG) commissioned IFF Research (IFF) to pilot the TSW on their behalf. Fieldwork for TSW (2022) was carried out between the 25 September 2022 and 2 January 2024. The survey produced statistics from information captured directly from businesses on trade flows to and from Wales in 2022. 

The rationale for the TSW stemmed from the need to have a more detailed evidence base to underpin WG’s policymaking. Increased evidence would enable WG to: 

• have a better understanding of Wales’s economy, including interlinkages between Wales-based businesses and those in other parts of the UK and abroad
• more accurately assess potential impacts of the UK’s proposed future trading relationships, on businesses in Wales and the wider economy

Welsh Government Analysts were responsible for:

• scoping the TSW content
• drawing the sample
• maintaining a governance framework for TSW oversight with key WG stakeholders
• commissioning the main-stage fieldwork and the analysis of survey results
• quality assuring the analysis and compiling the publications

IFF were commissioned to: 

• collect data from respondent businesses
• chase businesses that had not completed the survey by the deadline date
• validate respondent business data through desk research and clarification phone calls where businesses gave their consent to be recontacted
• undertake imputation to model missing responses to the survey (within year imputation has been carried out every year and across years imputation was introduced for TSW (2021) for sales but this is the first year it has been carried out for purchases. See ‘Imputation’ in the Analysis section below for more information)
• produce estimates of Wales’s trade from weighted survey responses
• prepare and deliver data updates, and the final data file, to WG

There will be a gap in data collection in 2024 due to budgetary pressures, but this will also allow the analytical team to conduct more in-depth analysis of previously collected data, and to revisit the business case for the continuation of the Trade Survey in future years.

The UK Government commissioned reviews of official economic statistics have long recommended improvements to regional statistics. The 2004 Allsopp Review looked at statistics for economic policymaking, which made recommendations to improve regional trade statistics, and for the design of new and existing surveys to take account of the need for regional statistics. The later 2016 Bean Review of UK Economic Statistics (UK Government) made further recommendations to improve ‘inadequate’ regional statistics, but highlighted the potential for administrative data to fill some of the gaps alongside improved coverage of business surveys. 

Following the EU Exit referendum in 2016, the Welsh Government established its own Trade Policy team. The requirement for trade evidence grew, with the Welsh Government’s Trade policy: the Issues for Wales (2018) calling for improved evidence to underpin trade policy decisions, and highlighted gaps in the trade data for Wales as well as potential methodological issues identified with existing data sources. There was no existing data on intra-UK trade from Wales, and limited data available according to business characteristics (such as size and sector). 

HM Revenue and Customs (HMRC) publish quarterly trade statistics for Wales on the international movement of goods, with value (in current prices) and volume data sourced from customs declarations. Much of the HMRC data analysis apportions trade for UK-wide businesses to Wales based on their employment split. This means that that the HMRC data for Wales may not accurately reflect the specific export values and markets of Welsh business units. The HMRC Regional Trade Statistics is published at the Standard International Trade Classification (SITC) 2 digit level ^{[footnote 3]}, and does not include any detail about business demographics (e.g. business size). This is not sufficiently disaggregated for Welsh Government evidence needs. Further, due to EU Exit, there have been changes to how HMRC collects GB-EU trade data (HMRC), moving from the EU Intrastat system to customs declarations for exports in January 2021 and imports in January 2022. This resulted in a break to the Regional Trade Statistics time-series, making comparisons over time more difficult. 

The HMRC data does not include trade in services. The Office for National Statistics (ONS) administer an Annual Survey of International Trade in Services (ITIS) that produces experimental statistics at the Wales level, with ONS publishing below-Wales level estimates (ONS) for both trade in goods and services (also published in current prices) since 2021.

Given the limitations with data at the time, TSW was piloted in 2019 with the aim of capturing robust information directly from businesses on trade flows to and from Wales.

The statistics produced from the TSW have provided Welsh Government policy makers with improved evidence with which to inform Ministers. The intra-UK trade figures have been particularly useful in gaining a more detailed understanding of the UK internal market than was previously possible. This data is used by the Office for the Internal Market (OIM) in their annual reports on the operation of the UK internal market (OIM), and is also referred to in the OIM data strategy roadmap update. International and domestic TSW data is also being used by the ONS to produce experimental and provisional UK interregional trade in goods and services estimates (ONS). These are due for publication in 2024, at the International Territorial Level 1 (ITL 1) which will cover Northern Ireland, Scotland and Wales, and the nine English regions.

The Welsh Government’s Export Action Plan, a Ministerial commitment in the 2021 to 2026 Programme for Government, also draws on the TSW as part of its evidence base. The survey additionally gives Welsh Government the ability to undertake follow up research with respondents agreeing to re-contact, allowing further insight into key policy questions highlighted in the results. Two small-scale follow up research reports (recontacting 2020 respondents) have been published to date on Rules of Origin and Mode 5 Services. 

In 2023, Welsh Government launched a three-year project examining the feasibility of constructing a full regional economic account for Wales (Digital and Data blog), known as an Input Output (IO) table. The purpose of this is to enable an analysis, not only of how the economy ‘works’, but also how it fits within other systems – for example, UK and global economies, Wales’s natural environment, or the climate. The IO Tables will be used by Welsh Government to appraise the broader economic impacts of changes in demand in the Welsh Economy. Data from TSW will form a key data source for this project.

The results of the survey are ‘official statistics in development’ ^{[footnote 1]} as the method employed is still in development and there are some known limitations in terms of data quality, outlined within this section. Official statistics in development are published with a view to improving their quality over time through feedback from users and stakeholders and developing the methodology. The TSW estimates will remain official statistics in development while development work continues to improve their coverage and quality. However, the data and analysis are still of value provided that users view them in the context of the data quality information provided – strengths are also outlined in this section. 

The results should be treated as provisional due to the likelihood of future revisions from methodological improvements. The volatility of estimates are considered, with ways to further improve their quality explored. Although the introduction of across-years imputation has reduced the impact of variability in the achieved sample across the years of fieldwork the ability to make robust comparisons across the years remains limited. 

It should be noted that the business size analysis has been based on UK employment values (rather than employment in Wales) as captured within the Inter-departmental Business Register (IDBR) (ONS). In the accompanying data tables, certain data cells have been suppressed where there has been a small number of survey responses. There may be small discrepancies in figures not summing to totals due to rounding.

Strengths

• The TSW is the only direct collection of trade data specific to business operations located in Wales. Businesses are asked to directly report the activity of their Welsh locations. This contrasts with existing regional statistics which collect UK level responses from which Wales level results are modelled.
• TSW is the only trade survey collecting intra-UK trade data, broken down by all UK nations, by businesses in Wales.
• TSW results are the only source of intra-UK trade conducted by businesses in Wales.
• Results are broken down by business size.
• Results are broken down by grouped SIC classifications.
• Goods and services trade are collected on a consistent basis.
• The methodology allows estimates of the total trade (goods and services) businesses in Wales make with all parts of the UK, the European Union (EU) and the rest of the world (Non-EU).
• The survey allows for follow up research with consenting businesses on policy relevant topics and issues e.g. Mode 5 Services and Rules of Origin.

Limitations

• As a voluntary and online survey there are low response rates compared with other, more costly (or mandatory), methods of data collection. This increases the likelihood of non-response errors which affects the ability to produce some lower breakdowns.
• Variability in the achieved sample across years affects the ability to make robust comparisons across years, although with the added across-years imputation methodology it makes high-level comparisons across the years more achievable, which the results report presents. Breakdowns that are more granular have shown greater variability, specific comparisons are discussed in the comparability and coherence sections of the results reports.
• Incomplete coverage of businesses, sectors and total Welsh trade in our sampling approach means the estimates do not cover all trade activity in Wales and should not be used to work out a trade balance, nor compared to total GVA/GDP. For example, only business purchases are covered, direct to consumer purchases are missing. This means Welsh consumer spending abroad (e.g. tourism imports) and consumer spending in other parts of the UK has not been captured, whereas the expenditure of tourists buying from businesses in Wales is captured.
• Some respondents provided estimated values in their responses so the values provided will have varying degrees of reliability.
• Feedback suggests that some respondents found it difficult to allocate sales to customers based in other parts of the UK.
• Respondents were asked to allocate trade based on their immediate partner in the supply chain, not the original source or final destination of a purchase or sale. The movement of goods through supply chains within the UK may therefore mask an international origin or final destination in the TSW results, which may be of particular significance for the retail sector using distribution hubs within the UK.
• Within the analysis there are a number of instances where ‘unallocated’ sales or purchases data is highlighted, where the respondent was unable to allocate to a specific destination. The effect of unallocated can at times be inflated due to the weighting. This may have led to underestimates of the overall figures.
• The six sector groupings used to categorise businesses by SIC code within the survey are broad to ensure base sizes are large enough for analysis. This means it is not possible to filter results by some specific sectors of interest which fall within wider categories e.g. food and drink.
• TSW relies on ONS Inter-Departmental Business Register (IDBR) (ONS) sample frame which has a time lag. This means each year of IDBR might not be reflective of ‘real time’ data for the same year. Business liquidation/insolvency in any one year might mean that the weighting and grossing could be incorrect.
• The survey asks questions on trade value but not trade volume. This makes it challenging to understand changes over time in light of changes in inflation.

Mode

A push-to-web approach was selected as the most appropriate method to meet the aims and objectives of the TSW given the survey’s scale. Respondents are invited to take part through an initial postal mail out with further reminder letters and telephone chasing survey, but the questionnaire is completed online. The sample frame is obtained from the ONS Inter-Departmental Business Register (IDBR) (ONS), a comprehensive record of business addresses, which ensures the TSW sample coverage is consistent with other businesses surveys. However, the IDBR contains limited email addresses, so an initial postal mail-out letter contacting businesses and inviting them to respond to the online questionnaire is necessary.

Sample size and frame

Some 8000 businesses are selected for the sample. 

The sample frame includes businesses in the UK with local units in Wales. The sample is selected from the ONS IDBR that lists businesses from either VAT or PAYE records. The sample frame is limited to reporting units that have local units in Wales and have existed long enough to be able to give the full year of data being asked for.

Businesses with employment of fewer than 3 are excluded, along with the list of sectors below. The sector exclusions were decided in-line with the measurement of trade for national accounts as well as seeking to harmonise exclusions made by other, similar, surveys. Businesses of certain legal statuses were also excluded from the sample. These were local authority organisations, central government and most non-profit bodies (though universities were included).

Excluded Standard Industrial Classification (SIC) ^{[footnote 4]} Sections

• O - Public administration and defence; compulsory social security
• Q - Human health and social work activities
• T - Activities of households as employers; undifferentiated goods-and services-producing activities of households for own use
• U - Activities of extraterritorial organisations and bodies

Further to the SIC sections list above, a number of lower SIC classifications are also excluded;

• 06 - Extraction of crude petroleum and natural gas
• 84 - Public administration and defence; compulsory social security
• 86 - Human health activities
• 87 - Residential care activities
• 88 - Social work activities without accommodation
• 93 - Sports activities and amusement and recreation activities
• 94 - Activities of membership organisations
• 96 - Other personal service activities
• 97 - Activities of households as employers of domestic personnel
• 98 - Undifferentiated goods and services producing activities of private households for own use
• 99 - Activities of extraterritorial organizations and bodies
• 642 - Activities of holding companies
• 791 -Travel agency and tour operator activities
• 851 - Pre-primary education
• 852 - Primary education
• 8299 - Other business support service activities not elsewhere classified (n.e.c.)
• 9101 - Library and archive activities
• 03110 - Marine fishing
• 47799 - Retail sale of other second-hand goods in stores (not incl. antiques)
• 49320 - Taxi firms
• 56290 - Other food services
• 56301 - Licenced clubs
• 64110 - Activities of central bank
• 79901 - Activities of tourist guides
• 79909 - Other reservation service activities n.e.c.

Sample design

Stratified sampling takes place at the reporting unit level. The sample was stratified by employment size band, and industrial sector SIC at the 2-digit level).

An overarching aim of the sample is to capture as large a proportion of the total Welsh economy (business turnover) as possible. To date the sample has selected 8000 reporting units, with a focus on larger businesses. 

The structure and sample design are reviewed each year, as these statistics continue to develop. The strata and number of records selected each year are provided below. 

After considering all exclusions, the survey participants were selected from a population of approximately 33,000 reporting units with activity in Wales and employment of three or more people.

Welsh Government Statisticians selected companies using a stratified approach based on employment. Some strata (20+ employment) took a census approach, making up the majority of the sample. 

For TSW3, an element of convenience sampling was introduced to the small and micro-business (3 to 19 employment) strata to include businesses that previously responded in an aim to reduce respondent variability.

A stratified random sample was taken of the remaining businesses in the 3 to 19 employment size band (those with at least one local unit in Wales). These businesses were further stratified by employment size (‘3 to 9’ and ‘10 to 19’) and industrial sector (SIC section), see Table 1.

The sample was allocated based on the SIC section turnover; sectors with higher total turnover received a greater proportion of the sample. Further adjustments were made to the sample allocation to avoid under or over sampling within sub-bands. Table 1 shows the sample breakdown for TSW (2022).

Source: IDBR, Office for National Statistics

Sampling a small proportion of businesses with 3 to 19 employees ensured coverage of all size bands while reducing the overall burden placed on businesses and maintaining a practical and cost-effective sample size. The rationale for over-representing larger businesses within the sample was to ensure a large proportion of the Welsh economy (total Welsh IDBR turnover) was captured. The 8,000 sampled businesses were estimated to account for 98.7% of turnover in Wales (from businesses with employment of 3 or more).

The survey was conducted using a push-to-web approach. Sampled businesses were initially sent a bilingual (Welsh and English) postal invitation letter (copies included in Communications Annex) encouraging them to go online to complete the TSW.

Figure 1: Overview of the data journey

Image

Description of Figure 1: a flow chart showing the fieldwork process from sample drawn and prepared through to distribution of the survey link, data cleaning, processing and finally data and reporting outputs.

Source: Welsh Government 2023

As with previous iterations of the survey, TSW was voluntary; businesses were invited to take part via an initial invitation letter, followed by a phone call to confirm receipt of the initial invitation letter, one reminder letter, and two subsequent reminder emails (where an email address for the business was held) (see mainstage fieldwork timeline). In addition, IFF undertook call chasing throughout the fieldwork period in order to answer any questions from businesses and encourage completion of the survey.

If respondents needed to complete the survey in more than one sitting (or more than one person needed to fill in the survey), they could do so by re-entering their access code. When they re-entered the survey, respondents were immediately taken to the last point in the survey they had reached. It was only possible to use each access code for a single survey response, so each business could only submit one return.

Throughout the data collection process, all data was stored securely in IFF’s Unicom Intelligence online storage system. 

As part of regular progress updates between IFF and WG during the fieldwork phase, weekly response rate reports were compiled using survey data (which showed, for example, information on which survey sections respondents were pausing at, and respondents’ business size and sector).

Survey design, testing and piloting

Cognitive testing was conducted with 32 businesses ahead of the first TSW in 2019 and a further pilot undertaken ahead of the third in 2021. Feedback from businesses informed question development and the collection mode used. Further details of the cognitive testing and development of TSW year one can be found in TSW 2018: Technical Report ^{[footnote 5]}.

The majority of the year five TSW survey (collecting 2022 data) remained the same as the previous year (collecting 2021 data). There were some minor changes introduced to the year five survey:

• One new closed response question was added because of policy interest in the topic area of gender of business leadership. The question was asked of all businesses. 

“Research has shown that approaches to trade can differ by the leadership of a business. For data analysis purposes, it would be useful to know if your business is majority-led by women. That is, controlled by a single woman or has a management team of which a majority are women.  Is your business majority-led by women?”

[yes/ no/ don’t know/ prefer not to say] 

• A question on Mode 5 services (introduced in TSW 2021) was removed since it was felt that the separate research study on Mode 5 services (published in 2023) provided sufficient insights on the subject. The question asked respondents:

“In the process of selling those goods, are there any services that are provided alongside the product as part of the sale of the good? For example, an ongoing subscription for maintenance and repair, or training, or a related software subscription such as cloud storage with a phone.”

To make the survey as short as possible and reduce respondent burden, questions were routed where possible to allow respondents to automatically miss pages that were not relevant to them. 

Although respondents could not skip forward through the questions (due to routing constraints), they could move backwards in the survey to review and edit their previous answers before submitting the survey.   

Timeline

The mainstage fieldwork commenced 25 September 2023 and closed on 2 January 2024, running for approximately 14 weeks.

Mainstage fieldwork timeline

Initial mailout letter

Start date

22 September 2023

Finish date

N/A

Reach

Sent to 7,995 businesses [Note 1]

[Note 1] Five addresses were removed following a de-duplication process

Further information

The first completes for mainstage fieldwork came through on 27 September 2023.

Telephone chasing

Start date

9 October 2023

Finish date

15 December 2023

Reach

5,418 businesses successfully contacted

Further information

1,676 verbally agreed to complete the survey. At least 16% of those who agreed to complete, went on to do so (this is likely an underestimation due to difficulties recording this).

Second mailout reminder letter

Start date

6 November 2023

Finish date

Not applicable

Reach

Sent to 7,506 businesses

Further information

13 and 14 November largest number of daily visits (162 and 164 respectively) and completes (60 and 56).

Validation calls

Start date

16 October 2023

Finish date

22 December 2023

Third mailout reminder letter

Start date

8 December 2023

Finish date

Not applicable

Reach: Sent to 3,000 respondents

Reminder email

Start date

18 December 2023

Finish date

Not applicable

Reach

Sent to Sent to 1,420 respondents

Further information

32.3% of emails were opened. 87 completes were achieved after email sent and before survey closure.

Postal and email contact with businesses

Sampled businesses were initially sent a bilingual invitation letter informing them of the survey and how they could complete it. Two versions of the letter were created, one for businesses which had not completed the previous survey and another thanking businesses who had completed the previous survey. Each letter was accompanied by an overview of the survey content and process. Copies of all communication materials (letters and emails) can be found in the Communications Annex. 

Two bilingual postal reminder letters were subsequently sent (see mainstage fieldwork timeline for dates) to those who had not yet completed the survey, explaining that the deadline had been extended. A reminder email was also sent to those businesses who had asked to have the survey emailed to them following contact with the IFF call handlers. Although the responses which immediately followed each step cannot be directly attributed, an indication of corresponding response rates around these dates is provided in the further information section of the mainstage fieldwork timeline.

Response metrics

The survey was accessible 24 hours a day during the fieldwork period, although the peak periods of activity were usually during office hours. Over each survey year there has been a clear trend of survey completion primarily on weekdays during typical business hours, most often between 11am and 12pm.

There are two sources of data which may be considered when measuring time taken to complete the survey: data collected by the system when surveys were completed online, and self-reported figures provided by respondents within the survey. These are shown in Table 2.

Source: Trade Survey for Wales 2022, Welsh Government

The Code of Practice for Statistics (OSR) advises public bodies collecting statistical data that the burden of the survey for respondents should be proportionate to the benefits arising from use of the survey data collected. A question therefore was included within the survey asking businesses for an estimate on how much time they felt it had taken them to complete the survey. The question asked respondents to include: 

• time taken to become familiar with the questionnaire
• time of everyone who helped the business complete the questionnaire
• time spent extracting and preparing information from their systems
• any other time spent in relation to the questionnaire

Table 3 shows that the vast majority of businesses reported completion of the survey in under two hours ^{[footnote 6]}

Source: Trade Survey for Wales 2022, Welsh Government

[Note 1] 150 micro businesses

[Note 2] 541 small businesses

[Note 3] 329 medium businesses

[Note 4] 169 large businesses

Reported cost of completing the survey

A question in the TSW asked respondents to estimate how much completing the survey cost their business, including the time of anyone involved in filling out the survey. 

Of those who reported estimated costs incurred in completing the survey, more than half estimated the survey cost their business less than £50 to complete. This compares similarly to previous rounds of the TSW. Table 4 outlines the self-reported estimated costs broken down by company size.

Source: Trade Survey for Wales 2022, Welsh Government

[Note 1] 150 micro businesses 

[Note 2] 541 small businesses 

[Note 3] 329 medium businesses 

[Note 4] 169 large businesses

Compliance costs

Compliance costs for businesses completing the TSW were estimated as per the Government Statistical Service (GSS) guidance. The total estimated cost for all businesses to complete the survey was calculated ^{[footnote 7]} to be £22,419. This financial cost is calculated based on the time taken to complete the questionnaire, an appropriate hourly rate and external costs incurred (for example, costs to the business of a bookkeeper or accountant to aid in completion of the survey). 

Based on the self-reported question in the survey the median completion time for responses completed in a single day was approximately 35 minutes, while the mean was 54 minutes. Subsequent contact between IFF and respondents to validate responses is also included in calculations since it contributes to respondent burden. 58 businesses were re-contacted over the phone for validation. The average duration of these calls was approximately 14 minutes (calls under 2 minutes were discounted).

Completes in Welsh and English

The TSW complies with the Welsh Language Standards, meeting the WG’s commitment to treating the Welsh and English languages equally. Respondents were given the opportunity to complete the survey in either Welsh or English and could switch easily between the two languages throughout when completing the online survey. 

The system only captured language information for those who clicked ‘submit’ on the final page of the survey; this captured the language they were viewing on the final page. Of the 1189 respondents (who submitted on the final page), 1185 submitted the survey in English and four did so in Welsh, although some may have completed parts of the survey in Welsh.

Incoming contact with businesses

Businesses were able to contact IFF via phone and email with any queries they had relating to the survey. Table 5 shows that, in total, 566 queries were received either by email (158) or by telephone (329) ^{[footnote 8]}; a much lower volume than the 1,042 queries received in year four.

Source: Trade Survey for Wales (2022), Welsh Government

Call handlers logged ‘dead letters’, a letter that was undeliverable and typically with an incorrect address, and then conducted desk research to find alternative details for these businesses. In total, 78 letters were returned to IFF (compared to 182 for TSW (2021)). The low number of dead letters did not affect the overall 16% response rate when deducted from the 8000 sample.

Privacy notice

A requirement under UK GDPR when collecting any personal data, a privacy notice was published online detailing the exact lawful basis for processing their personal data. 

Frequently asked questions (FAQ) documents 

Alongside the initial invitation letter, businesses were sent a FAQs sheet which contained: 

• background detail on the TSW and its importance to the WG
• information about IFF
• how to verify the survey was genuine
• how businesses were selected
• reassurance about confidentiality
• the deadline for completion
• how to complete the survey if the business was not primarily based in Wales, had no data to report or had entered into administration/receivership

The online survey site contained a longer version of the FAQs, with more detail than could be included with the invitation letter. Additional information included in the online FAQs related primarily to the survey itself, including: 

• more detail on how data provided would be used
• the process for reporting any technical issues experienced with the survey
• how to provide feedback on the survey to the WG
• how IFF securely received respondents’ personal details

Response chasing

To encourage businesses to respond, IFF undertook telephone chasing. The try-count records the number of contact attempts made to each business during the chasing exercise. Most businesses were reached within one or two calls, although there were some that required further chasing (Table 6). The average number of contact attempts made was 2.19. All businesses with a phone number were called at least once. Those recorded as ‘0’ include a combination of businesses who had already completed the survey before they were called as well as those for whom a telephone number could not be found.

Source: Trade Survey for Wales 2022, Welsh Government

On average, a respondent took 8 days to submit their data from the point at which they ended their conversation with the call handler. Of the 326 respondents who received a chasing call and who then went on to successfully complete the survey (including those who did not explicitly agree to complete, such as those who were left a voicemail message), an average of 20 days elapsed between receiving first contact and submitting the survey.

Response rates

A response rate of 15% was achieved, with 1,209 respondents taking part in the survey out of a possible 8,000 businesses. This equated to a total just short of the 1,264 in TSW 2021. Only responses where both the sales and/or purchases sections of the survey had been completed, and where no discrepancies within the data were flagged, were included within the analysis. Progress had lagged behind that of the previous surveys throughout the fieldwork period. 

Historically, the survey has received response rates of between 14% and 17% (Y1: 14%, Y2: 16%, Y3: 17%, Y4: 16%). 

In terms of response rates for businesses according to business size (based on UK employment values within the IDBR), Medium-sized businesses had the highest response rate (18%), whilst Micro businesses and large businesses had the joint lowest (both 12%) (Table 7).

Source: Trade Survey for Wales 2022, Welsh Government

The response rate was fairly consistent across different business sectors, but those in Business and Other Services had the highest response rates (16%) (Table 8).

Source: Trade Survey for Wales 2022, Welsh Government

[Note 1] Sectors were grouped into five categories throughout data processing and analysis. After consultation within WG, it was agreed to group sectors using a standard sector grouping.

Business size bands

Throughout the data processing and analysis of TSW, businesses are grouped into categories based on UK employee values (business size). Four size bands are used in the analysis (see Table 9). However, only three size bands (small – which encompasses micro businesses, medium and large) have been used for the purposes of reporting.

UK employee values are used in order to ensure base sizes are large enough for analysis, improve comparability to HMRC & ONS data, and due to anecdotal evidence from cognitive testing that most businesses make trade decisions based on their overall UK operations.

Sector groupings

The IDBR provides a SIC code for each business in TSW, these are grouped to ensure base sizes are large enough to analyse. 

Sector groupings

• Business and Other Services
  • Information and communication
  • Financial and insurance activities
  • Real estate activities
  • Professional, scientific and technical activities
  • Administrative and support service activities
  • Arts, entertainment and recreation
  • Other service activities
• Construction
• Manufacturing
• Primary Sector and Utilities
  • Agriculture, forestry and fishing
  • Mining and quarrying
  • Electricity, gas, steam and air conditioning supply
  • Water supply; sewerage, waste management and remediation activities
• Trade, Accommodation and Transport
  • Wholesale and retail trade; repair of motor vehicles and motorcycles
  • Transportation and storage
  • Accommodation and food service activities
• Non-market Services
  • Education
  • Human health and social work activities (excluded)
  • Public administration and defence; compulsory social security (excluded)

Non-market services are combined with business and other services due to low base sizes, resulting in the following five categories being used throughout the data processing and analysis of TSW:

• Business and Other Services 
• Construction
• Manufacturing
• Primary Sector and Utilities
• Trade, Accommodation and Transport

Imputation

To maximise the data that could be analysed from the survey, imputation was carried out to model missing responses to the survey:

• Within year imputation: applied to businesses who took part in the survey year but were not able to provide certain values e.g. they could provide their total sales but could not apportion this to goods and services. Missing values were based on values within the survey (and average proportions) by other similar businesses.
• Across year imputation: Applied to sampled businesses who did not take part in the survey year. Missing responses were based on their responses to other survey years, their IDBR UK turnover across the years and responses from other similar businesses in the missing survey year.

Within year imputation

Throughout the survey, respondents had the option of indicating to which destinations (for example, Wales, Rest of UK, EU, non-EU) they sold, and then a subsequent question asking for the values of sales and purchases to these selected destinations. Within year imputation was carried out when respondents answered a tick box question but could not provide the subsequent actual values. Therefore, values were imputed where a business selected that they had goods sales in Wales and Rest of UK (and not EU or non-EU), but then had not apportioned out their total goods sales value between these two destinations.

Missing values were imputed based on apportioning the relevant total figure provided by the respondent in line with average proportions of sales/purchases made to each destination among businesses of a) a similar type (explained in following section) and b) selecting the same combination of destinations.

Imputation was also carried out for businesses who selected that they had goods and services sales/purchases but were unable to apportion these total values between goods and services. Missing values were imputed based on apportioning the total sales/purchases figure provided by the respondent in line with average proportions of sales/purchases made to goods and services among businesses of a similar type.

Once a business has had sales or purchases values imputed within the survey, no further imputations are calculated i.e. if their UK sales value is imputed, their disaggregated values for England, Scotland and NI are not eligible for imputation. 

Businesses of similar type

Average proportions were calculated within imputation groups according to business size and sector, meaning imputed values were only calculated using values given by similar businesses. Values were only calculated within each group if there were at least 10 other businesses who had provided exact values in the same group as the record requiring imputation. Consequently, three imputation groups were created: 

• Imputation Group 1 was based on the 21 sector codes (detailed below) and 4 size bands.
• Imputation Group 2 merged the sector codes into 5 broader groups but retained the 4 size bands.
• Imputation Group 3 retained the 5 broader sector groups as in Group 1, but also merged the size bands into 2 broader bands.

If a business requiring imputation fell into an imputation group containing fewer than 10 businesses giving exact values (e.g. imputation group 1), then they were allocated to the next level imputation group (imputation group 2). 

Imputation group categorisation

Imputation group 1

Industry sector

1. Accommodation and food service activities
2. Activities of extraterritorial organisations and bodies
3. Activities of households as employers; undifferentiated goods-and services-producing activities of households for own use
4. Administrative and support service activities
5. Agriculture, forestry and fishing
6. Arts, entertainment and recreation
7. Construction
8. Education
9. Electricity, gas, steam and air conditioning supply
10. Financial and insurance activities
11. Human health and social work activities
12. Information and communication
13. Manufacturing
14. Mining and quarrying
15. Other service activities
16. Professional, scientific and technical activities
17. Public administration and defence; compulsory social security
18. Real estate activities
19. Transportation and storage
20. Water supply; sewerage, waste management and remediation activities
21. Wholesale and retail trade; repair of motor vehicles and motorcycles

Size band

1. Micro
2. Small
3. Medium
4. Large

Imputation group 2

Industry sector

1. Primary Sector and Utilities
2. Manufacturing
3. Construction
4. Trade, Accommodation and Transport
5. Business and Other Services

Size band

1. Micro
2. Small
3. Medium
4. Large

Imputation group 3

Industry sector

1. Primary Sector and Utilities
2. Manufacturing
3. Construction
4. Trade, Accommodation and Transport
5. Business and Other Services

Size band

1. Micro/Small
2. Medium/Large

Same combination of destinations

Apart from the questions which asked for a total value to be split by goods and services, for each type of question, there were a number of different combinations of destinations that businesses could have selected (Table 9). Average proportions were calculated only among businesses who had also selected the same combination of answers within the imputation group cell.

Source: Trade Survey for Wales 2022, Welsh Government

Plus, the remaining 12 combinations for this set of questions.

Within the imputation groups containing similar business types and the same combination of destinations, the average proportions of sales or purchases allocated across these destinations were calculated, as opposed to average monetary values. Missing values were then calculated for each record by applying these average proportions to the actual total value provided by that business (Table 10). For example, if a business said it had goods sales of £1,000 in a given year but had not said how this was split between Wales, the rest of the UK, EU and non-EU the average proportions would be applied as below.

Source: Trade Survey for Wales 2022, Welsh Government

In total, within year imputation was calculated across 32 variables, allowing a total of 559 values, across 174 business, to be imputed. Across these 32 variables, the highest number of values per variable that were able to be imputed was 72 and the lowest was 0. The highest percentage of total values imputed ranged from 0% to 14.7%.

Across year imputation

The following across year imputation was within scope for Year 5: 

Sales data

1. Adding to Year 5 responses, from Year 4 original and imputed responses. 
2. Adding to Year 4 responses, from Year 5 (only records with no Year 4 achieved or imputed response currently). Note this is the only ‘backwards’ sales imputation within scope. 

Purchases data

1. Adding to Year 2 responses, from Year 1 original responses. 
2. Adding to Year 3 responses, from Year 2 original and imputed responses. 
3. Adding to Year 4 responses, from Year 3 original and imputed responses. 
4. Adding to Year 5 responses, from Year 4 original and imputed responses. 
5. Adding to Year 4 responses, from Year 5 (only records with no Year 4 achieved or imputed response currently). Note this is the only ‘backwards’ purchases imputation within scope.

Which businesses will be eligible for imputation?

Businesses who had not completed the survey were eligible to have their responses imputed if they met all of the following criteria: 

• They hadn’t taken part in the current survey year (“target” year) but had been sampled i.e. invited to take part. It would not have been suitable to impute responses for business who were never intended to be included that year. 
• They had completed or had their response imputed into the previous survey year (“achieved” year), thus enabling a response to be imputed for the “target” year. 
• They had an IDBR UK turnover of £100 million or more, OR had a Welsh turnover of £10 million or more, in the “achieved” year. This was to focus imputation analysis on businesses who would have a larger impact on the total turnover or purchases value being reported for the “target” year. 
  • For purchases imputation, businesses could also (“or”) be eligible if they had a purchases value of £10 million or more in the “achieved” year. 
• They have more than 20 employees in Wales in the “target” year. Businesses with 20 employees in Wales. All reporting units (a census) with employment of 20 or more are included each year of TSW, and made up the majority of the sample.

Businesses had their survey response imputed for the “target” year based on “achieved” responses from either:

• Responses they had given in the previous survey year i.e. how much turnover or purchases value they had reported in the Year 4.
• Responses they had had imputed into the previous survey year (Year 4) based on turnover or purchases value they had reported in the years prior to that (e.g. Year 3) ^{[footnote 9]}. 

A percentage change was applied to these “achieved” responses to reflect that turnover or purchases value is assumed to not be the same year on year. The percentage change applied was calculated by:

• Identifying businesses who had responded to both the “achieved” and “target” years ^{[footnote 10]}. This was 567 businesses in Years 4 and 5 (48% of all Year 5 businesses)
• Summing the total sales turnover, or purchases value, reported for these two years within similar businesses (four size bands by five broad sector groups). 
• Calculating the percentage change between these two years of sales turnover, or purchases value, for each of these size by sector groups e.g. Small Manufacturing businesses saw an 8% increase in reported sales turnover between Year 4 and Year 5. 

The percentage change applied to achieved responses were calculated using the following methods: 

Sales data

1. Percentages were only applied if at least 10 other businesses had taken part in both the “achieved” and “target” years (for Year 5, 12 of the 20 size by sector groups had 10 or more businesses within them). 
2. If fewer than 10 businesses had done so, a percentage change based on the change in UK turnover (as provided in the IDBR) for that business between the “achieved” and “target” years was applied instead.
3. For either method, values were checked to see if the new (imputed) sales value (once the % change is applied) was larger than the IDBR turnover for that year. In these circumstances, a decision was made to either continue with the % change as was (and sometimes update IDBR values in the process) and/or adjust percentage change to zero. 
4. The same percentage change was then applied to all sales responses (not purchases responses) and ‘coded’ sales responses were copied into the data file.

Purchases data 

1. Percentages were only applied if at least 10 other businesses had taken part in both the “achieved” and “target” years (for Year 5, 12 of the 20 size by sector groups had 10 or more businesses within them). 
2. If fewer than 10 businesses had done so, a percentage change of 0% was applied instead.
3. Once the percentage changes were calculated and assigned to individual businesses, broader quality checks were carried out. 
4. The same percentage change was then applied to all purchases responses (not sales responses) and ‘coded’ purchases responses were copied into the data file. 

Outcome

The above process resulted in the following records being included within the Year 4 and Year 5 data (Table 11):

Source: Trade Survey for Wales 2022, Welsh Government

Weighting

Response data from the survey is weighted and grossed up to the total population of businesses (derived from the sample frame), to produce Wales level estimates. Value and unit weights are applied to produce estimates of the value of trade and the number of businesses engaging in trade.

The weighting of results takes account of the unequal selection probabilities of businesses and the effect of non-response on the achieved sample.

As development work continues for these statistics, the sampling and weighting approach will be reviewed each year. The weights for sales and purchases are different due to the introduction of across-years imputation.

In total, four sets of weights were created: two for sales and two for purchases. The two weights within sales and purchases accounted for the fact that roughly half of the survey collected value-based answers and the other half collected responses for tick-box questions which suited being weighted to represent the business population of Wales in terms of reporting units. The ‘unit-based’ weights were used for analysis of ‘tick-box/coded’ based questions and ‘value-based’ weights were used for analysis of turnover and cost based questions.

Weighting/grossing process

• Step 1: Calculating two non-response weights (for unit-based and value-based questions) comparing to the starting sample of the survey.
• Step 2: Calculating ‘unit-based’ and ‘value-based’ grossing weights for both sales and purchases.
• Step 3: Multiplying the non-response weights with the sales and purchases weights to create the final four weights needed.

Step 1: Non-response weights

As the survey was not mandatory, there was some natural variation in response rates between different types of businesses which resulted in the survey profile differing slightly from that of the starting sample of the survey. To adjust for these differences, non-response weights were calculated, using a random iterative method (RIM), and applied to bring the survey profile in line with the sample. Non-response weights were calculated by comparing the profile of achieved interviews with the 8,000 businesses in the starting sample. The variables were compared, and where there was a difference of 2 or more percentage points between the achieved interview profile and that of the sample profile, a non-response weight was applied.

Sector employment (5 bands)

1. Primary Sector and Utilities
2. Manufacturing
3. Construction
4. Trade, Accommodation and Transport
5. Business and Other Services

Size (IDBR UK employees, 4 bands)

1. Micro (<10 employees)
2. Small (10-49 employees)
3. Medium (50- 249 employees)
4. Large (250+ employees)

IDBR Turnover (banded)

1. <£1 million
2. £1 million - <£5 million
3. £5 million - <£10 million
4. £10 million - <£20 million
5. £20 million - <£50 million
6. £50 million - <£100 million
7. £100 million+

Region (IDBR)

1. AA. North East
2. BA BB. North West
3. DC. Yorkshire and the Humber
4. ED. East Midlands
5. FE. West Midlands
6. GF. Eastern
7. GG. Eastern
8. HH. London
9. JG. South East
10. KJ. South West
11. WW. Wales

No of branches in Wales (banded)

1. 0
2. 1
3. 2
4. 3+

Whether business is entirely Welsh

1. Yes 
2. No

Step 2: Grossing ‘unit-based’ and ‘value-based’ weights for sales and purchases

The non-response weights were applied to the data, and then two different weighting grids were calculated, grouping cells by four size bands and five sector bands as outlined above (the smallest number of businesses in any one cell was six and so no cell merging took place).

The two different weighting grids compared:

• number of units achieved in the survey compared to the population 
• total turnover in the survey compared to the population 

For the unit grid, a weight was created for each cell of the grid to gross the number of records in each cell to the number of businesses in that cell in the population (i.e. if there were 20 records in the cell and there were 100 in the population then each record received a weight of 5). The UK employment values reported within the survey were not used because data cleaning revealed these were not as reliable compared to the IDBR employment data. This was likely as the survey would in some cases have been completed by staff with knowledge of the Wales based business but not necessarily with the UK-wide overview. The unit grid was therefore calculated using the employment values provided within the IDBR. 

For the turnover grid, a weight was created for each cell of the grids to gross the sum of turnover for each cell (from the population and from our achieved sample) and applied to each £ of turnover so that each respondent in that cell would get a weight relative to their exact amount of turnover. The IDBR turnover figure was used because, although the survey asked about Welsh turnover specifically, the IDBR turnover field relates to UK turnover and a reliable turnover figure to gross to was required. 

Step 3: Creating the final four weights

These ‘unit-based’ weights and ‘value-based’ weights were then multiplied by the non-response weights to create the final weights for both sales and purchases (Final four weights).

Final four weights

UNIT_WEIGHT_SALES

Applied for analysis of coded sales questions in the survey.

VALUE_WEIGHT_SALES

Applied for analysis of value sales questions in the survey.

UNIT_WEIGHT_PURCHASES

Applied for analysis of coded purchases questions in the survey.

VALUE_WEIGHT_PURCHASES

Applied for analysis of value purchases questions in the survey.

Non-response error

The largest limitation on the accuracy of results is the non-response error. Response rates for online surveys are known to be typically low compared with other, more costly, methods of data collection such as telephone or face-to-face. Further, the TSW is a voluntary survey, while similar surveys conducted by the ONS are mandatory leading to response rates of some 60% or more.

Non-response leads to reduced base sizes, therefore breakdowns of the Wales level estimates are only provided where base sizes allow.

The across-years imputation approach is one way of modelling non-responses to improve the accuracy of the estimates.

Sampling error

Sampling error arises because the estimates are based on a random sample of the population rather than the whole population. The results obtained for any single random sample are likely to vary by chance from the results that would be obtained if the whole population was surveyed (i.e. a census), and this variation is known as the sampling error. In general, the smaller the sample size the larger the potential sampling error.

The sample design for TSW aims to minimise this by taking a census approach to the largest businesses (by employment) and focusing the remaining sample allocation to the industrial sectors with the largest turnover.

The sampling error will be explored further as part of the as part of the work programme associated with these official statistics in development.

Missing answers

Missing answers occur for several reasons, including refusal or inability to answer a particular question. Routing within the survey prevents respondents seeing questions that are not applicable. Within-years imputation is used to model some missing responses. 

The results include an ‘unallocated’ category which is a result of respondents partially completing the questionnaire, for example, a business providing the value of total goods sales to the UK, but unable to provide a breakdown of sales to the separate UK nations. Although this limits the usefulness of some results, understanding the difficulty respondents find in answering particular questions is important while development work continues.

Incorrect answers

A potential cause of bias is respondent error when providing answers. The TSW online survey has multiple validation checks in place to prevent this. The online survey allows automatic checks to be conducted as the respondent is completing the questionnaire and flag questionable. Validation checks include;

• logic checks that confirm breakdowns sum to the totals provided
• providing respondents a summary of answers given, in numeric and written form, overcoming potential confusion when inputting large values

Where responses fail automatic checks the respondent is asked to review and correct the response or confirm it is correct (with the option of providing a reason).

Incorrect answers are also minimised by routing respondents to applicable questions only, based on responses to filter questions.

Final validation

Automatic validation checks are run on each submitted survey response to flag any remaining discrepancies which would invalidate the response. Flagged responses are reviewed by the contractor, data is amended if necessary; through a combination of desk-based research and respondent call-backs where consented to.

The contractor is also required to carry out other quality assurance checks on the data once fieldwork had finished. This included looking at the comments provided within the survey to assess whether any respondents had flagged particular issues completing the survey which might impact the reliability of their responses.

Quality assurance

Welsh Government statisticians perform quality assurance checks and analysis on the response data and results files provided by the contractor. All published statistics undergo quality assurance in line with Quality statistics in government - Government Statistical Service (GSS) guidance.

Each wave of the TSW sample is drawn from an annual extract of the IDBR. The IDBR is updated using administrative and survey-based data sources with a variety of reference dates which means that business data could have been updated at any point up to the date of extract. This means that the business characteristic data used in sampling and weighting for the TSW is only as up to date as the latest version of the IDBR.

TSW fieldwork has typically commenced in Autumn (Historic fieldwork dates for TSW), collecting data for the previous calendar year. The results are then analysed and published the following year, for example; 2018 data was collected in 2019 and results were published in 2020.

Historic fieldwork dates for TSW

Year 1

Data coverage period

2017 to 2018

Fieldwork period

November 2019 to February 2020

Year 2

Data coverage period

2019

Fieldwork period

October 2020 to January 2021

Year 3

Data coverage period

2020

Fieldwork period

October 2021 to January 2022

Year 4

Data coverage period

2021

Fieldwork period

October 2022 to January 2023

Year 5

Data coverage period

2022

Fieldwork period

September 2023 to January 2024

Data

All TSW data is provided in open format accessible spreadsheets.

Disclosure control

We take care to ensure that individual businesses are not identifiable from the published results. We follow the requirements for confidentiality and data access set out in the Code of Practice for Statistics (OSR). Respondents are directed towards the TSW Privacy Notice for further information about the security and confidentiality of the data they provide. 

Language requirements

Welsh language standards are adhered to for TSW data collections and outputs. Our invitation letters, questionnaire, telephone helpline, website, results publications, technical reports, datasets are produced in both Welsh and English. We aim to write clearly (using plain English/‘Cymraeg Clir’).

Comparing the results produced by TSW to existing data sources is a key component of our development work. 

Annual results reports contain a detailed comparison of the latest TSW estimates against previous years and alternative statistics published by HMRC and ONS. Discrepancies are discussed and limitations in our estimates highlighted.

Our statistical practice is regulated by the Office for Statistics Regulation (OSR). OSR sets the standards of trustworthiness, quality and value in the Code of Practice for Statistics (OSR) that all producers of official statistics should adhere to.

All of our statistics are produced and published in accordance with a number of statements and protocols to enhance trustworthiness, quality and value. These are set out in the Welsh Government’s Statement of Compliance.

These official statistics in development (OSR) demonstrate the standards expected around trustworthiness, quality and public value in the following ways. 

Trustworthiness

The data has been collected and processed by an external contractor with extensive experience in operating surveys and analysing and presenting such information. The release has been produced with advice from statisticians who work under the supervision of the Welsh Government Chief Statistician to ensure that the statistics, data and explanatory material is presented impartially and objectively. 

All personal data underlying these statistics are processed in accordance with the requirements of the Data Protection Act 2018 (UK Government). We have put in place a thorough governance process with the contractor to ensure that the data are securely managed and reviewed before release. 

Quality 

Please see the data considerations section for further information about the quality of this data. 

Value

The purpose of the statistical release is to provide an estimate of the value of trade conducted by businesses located in Wales. As outlined in the QMI report, the TSW seeks to improve economic statistics and provide a more detailed evidence base to underpin Welsh Government’s policymaking. This would enable Welsh Government to have a better understanding of Wales’ economy and more accurately assess potential impacts of the UK’s proposed future trading relationships on businesses in Wales and the wider economy.

Development work on TSW continues and plans are in place to improve estimates by bringing in data from other surveys. We continue to seek feedback on our plans and on the outputs produced to date. Internal stakeholders are regularly engaged through the TSW Project Board which meets quarterly to discuss the ongoing development of the survey. Wider feedback and communication is always welcomed by the Trade Team and any user is welcome to get in contact at any time. Feedback can be sent to stats.trade@gov.wales.

[1] Official statistics in development were previously referred to as 'experimental statistics'

[2] Current prices data are not adjusted to remove the effects of inflation.

[3] For a comprehensive list of SITC categories (un.org)

[4] UK Standard Industrial Classification of Economic Activities (SIC) (ONS)

[5] Trade Survey for Wales, 2018: Technical Report.

[6] These figures exclude ten businesses who said it took them zero minutes to complete. It also excludes two businesses who said the survey took them 24 hours or longer to complete.

[7] Where: Burden = [{Number of respondents x median completion time (hours)} + {Number of businesses recontacted for validation x Median time (hours) taken for re-contact}] x hourly_rate (based on the Annual Survey of Hours and Earnings, Table 2.5a Hourly pay - Gross 2023 (ONS) using managers, directors and senior officials median hourly pay).  Final calculation: Burden = [(1,189 x 0.75) + (58 x 0.23)] x 24.77 = £22,419

[8] This does not sum the total number of queries as multiple queries could be included within one email or phone call. 

[9] Businesses can only have their responses imputed from a previous sequential year. Responses cannot ‘jump’ forward a year i.e. Year 5 responses cannot be imputed based on Year 2 responses. A business could be imputed across multiple past years, but this would always be done by imputing values forward one year at a time.

[10] Imputed data was not used to calculate percentage changes between years within size and sector groups.

Ystadegydd: Jonathan Burton
Ymchwilydd: Mair Smith
E-bost: ystadegau.masnach@llyw.cymru

Cyfryngau: 0300 025 8099