Arolwg Cenedlaethol Cymru: profi deunyddiau recriwtio a mynediad i arolygon ar-lein

Yn wreiddiol, cynhaliwyd yr arolwg drwy gynnal cyfweliadau wyneb-yn-wyneb mewn cartrefi pobl, ond ers mis Mai 2020 caiff ei gynnal dros y ffôn.

Mae'r holl gyfranogwyr yn cwblhau arolwg 35-munud dros y ffôn. Fodd bynnag, mae is-hapsampl wedyn yn llenwi adran hunan-gwblhau ar-lein sy’n para 15 munud.

Mae'r deunyddiau ymlaen llaw yn cynnwys llythyr a thaflen wybodaeth, sy'n cael eu hanfon gyda’i gilydd i bob aelwyd. Mae'r llythyr yn gofyn i'r aelwyd ddarparu rhif ffôn, naill ai drwy borth ar-lein neu drwy ffonio'r swyddfa maes. Yna bydd cyfwelydd yn galw'r aelwyd ac yn dewis cyfranogwr ar hap gan ddefnyddio'r dull 'pen-blwydd nesaf'. Mae dwy fersiwn o'r llythyr a'r daflen: fersiwn safonol sy’n gofyn i ymatebwyr wneud yr arolwg ffôn yn unig, a fersiwn ar-lein ar gyfer y rhai y gofynnir iddynt gwblhau'r adran ar-lein hefyd.

Os na ddarperir rhif ffôn, bydd cyfwelydd yn anfon cerdyn post tua wythnos yn ddiweddarach gyda'i rif ffôn symudol gwaith, sy'n rhoi dull cysylltu arall i ymatebwyr ei alw neu anfon neges destun ato.

Rhoddodd y prosiect hwn sylw i brofi'r deunyddiau ymlaen llaw (llythyrau/taflenni/cerdyn post) ar gyfer yr arolwg, y broses a ddefnyddir i gael rhifau ffôn ymatebwyr, a'r broses ar gyfer cael mynediad i, a chwblhau, adran ar-lein yr arolwg.

Aethom ati i ddeall a yw'r deunyddiau ymlaen llaw a'r broses ar gyfer darparu rhifau ffôn:

• yn ddigon clir ac a oedd yr holl gyfranogwyr yn eu deall
• yn meithrin teimladau cadarnhaol am gymryd rhan yn yr arolwg
• yn helpu i sicrhau’r cyfraddau ymateb uchaf posibl a chyn lleied â phosibl o ragfarn ar draws yr holl grwpiau o gyfranogwyr

Ar gyfer yr adran ar-lein, ein nod oedd:

• deall a oedd yn hawdd cael mynediad ati
• ar ôl cael mynediad, deall a oedd yr adran ar-lein yn hawdd i'w defnyddio

Hefyd yn allweddol i'r profion oedd deall beth yw'r cymhellion a'r rhwystrau ar gyfer cwblhau'r arolwg, dros y ffôn ac ar-lein.

O ganlyniad i'r profion hyn, darperir canfyddiadau manwl yn seiliedig ar y materion a nodwyd am broses recriwtio'r arolwg a’r adran ar-lein yn ogystal ag argymhellion ynglŷn â’r ffordd orau o fynd i'r afael â'r materion hyn.

Fel cam cyntaf yn y broses, rhoddodd Llywodraeth Cymru'r deunyddiau a ddefnyddir i recriwtio cyfranogwyr i'r Arolwg Cenedlaethol, i ORS. Yna cynhaliodd ORS weithdy mewnol gyda thîm y prosiect gan adolygu’r deunyddiau.

Arweiniodd yr adolygiad hwn at adborth ac argymhellion, a drafodwyd gennym gyda Llywodraeth Cymru.

Yna cynhaliwyd 29 o gyfweliadau gwybyddol dros Microsoft Teams neu Zoom gyda chyfranogwyr ledled Cymru, mewn tri cham rhwng 4 Awst a 29 Hydref 2021. Cafwyd 10 cyfweliad ym mhob un o'r ddau gam o brofi deunyddiau (deunyddiau ymlaen llaw gwreiddiol; deunyddiau ymlaen llaw diwygiedig), a naw yn y cam profi’r adran Ar-lein.

Y prif feini prawf dethol ar gyfer y cyfweliadau oedd y dylai cyfranogwyr fod: yn byw yng Nghymru ac felly'n gymwys i gael eu dewis ar gyfer yr Arolwg Cenedlaethol; o ardaloedd daearyddol amrywiol (y Gogledd, y Canolbarth, y De-ddwyrain a’r De Orllewin); ac yn cynrychioli ystod o oedrannau (18+). Roedd tri chyfranogwr o gefndiroedd lleiafrif ethnig, gan gynnwys un ceisiwr lloches. Roedd dau gyfranogwr yn ystyried eu bod yn anabl.

Recriwtiwyd y cyfranogwyr dros y ffôn neu drwy e-bost. Roeddent wedi cymryd rhan mewn ymchwil ar gyfer ORS yn y gorffennol ac wedi cytuno i ORS gysylltu â nhw eto at ddibenion ymchwil eraill. Buom hefyd yn gweithio gyda Thîm Cymorth Ieuenctid Lleiafrif Ethnig Cymru (EYST) i recriwtio cyfranogwyr o gefndiroedd lleiafrif ethnig. Rhoddwyd taleb rhodd o £30 i bob cyfranogwr i ddiolch iddynt am eu hamser a'u hymdrech wrth gymryd rhan.

Anfonwyd pecyn o ddeunyddiau yn y post at y cyfranogwyr cyn eu cyfweliad ymchwil, a oedd yn cynnwys y dogfennau i'w hadolygu.

Cyn archwilio'r deunyddiau recriwtio, gofynnwyd cyfres o gwestiynau i'r cyfweleion am eu cymhellion cyffredinol i gwblhau arolygon. Ar gyfer cyfranogwyr Camau 1 a 2 roedd hyn yn cynnwys y llythyr cychwynnol (fersiynau safonol ac ar-lein), y daflen safonol, a'r cerdyn post atgoffa. Derbyniodd cyfranogwyr ym mhrofion yr adran ar-lein y llythyr a'r daflen ar-lein yn unig.

Gofynnwyd i'r holl gyfranogwyr adolygu'r dogfennau a rhoi eu barn. Gofynnwyd hefyd i gyfranogwyr Cam 2 gymharu'r fersiynau gwreiddiol â'r rhai a oedd wedi'u golygu ar ôl cam 1 a dweud pa elfennau o'r fersiynau yr oeddent yn eu ffafrio. Roedd y trafodaethau'n canolbwyntio ar eu hargraff gychwynnol, cywair a threfn y testun, y brandio ac a fyddent yn cael eu perswadio i gymryd rhan o ganlyniad i'w darllen, neu'n wir a fyddai unrhyw beth am y deunyddiau yn eu hatal rhag cymryd rhan.

Yn ogystal, arsylwyd ar gyfranogwyr Cam 1 yn mewngofnodi i borth y wefan a ddefnyddir i gasglu gwybodaeth gyswllt a gofynnwyd iddynt am hwylustod mynediad ac unrhyw welliannau y gellid eu gwneud.

Rhoddwyd amser i gyfranogwyr ym mhrofion yr adran ar-lein fewngofnodi a chwblhau elfen ar-lein yr Arolwg Cenedlaethol. Gofynnwyd iddynt am adborth ar y broses o gyrchu a llywio drwy'r adran ar-lein, yr hyn a weithiodd yn dda a'r hyn nad oedd yn gweithio cystal, ac os oedd unrhyw broblemau neu fethiannau yr oedd angen mynd i'r afael â nhw.

Survey completion motivations and habits

Perhaps unsurprisingly, as this cohort of participants had agreed to take part in this research, there was a general openness to completing surveys. The most important deciding factor in whether most would complete a survey, or not, was time. For those who work, in particular, if a request to participate in a survey comes in the daytime, participants reported that the request is likely to be pushed aside to a later date, forgotten about, or ignored.

Most participants said that they would complete a survey on a topic that was of interest to them and/or is something they deem to have importance in their lives or which will impact on their local area. Surveys on local issues were seen by most participants as important and would be prioritised for completion – participants stated that anything that might impact local service delivery, including health and education, would be a particular priority for them.

Participants also placed importance on ‘official’ surveys from the Government, local authorities, or the NHS and stated they are more likely to complete these, with some saying they felt they had a ‘civic duty’ to do so. All participants had completed the Census and said they considered it important.

Stage 1: recruitment material review

Having reviewed the evidence from the Stage 1 research interviews, we recommended the following changes be made to the documents and to the portal for submitting contact details.

Recommended changes

• The opening paragraph of the standard letter should be extended to include more information about the content of the survey and why people should participate.
• The middle information box on both versions of the letter should look more like the one in the Online letter and include ‘Step 1, Step 2, Step 3’ headings.
• The wording in the line relating to conducting a ‘socially-distanced visit’ should be softened. If this is not possible, it should be located towards the bottom of the letters and not in the main instruction box.
• The leaflet should be referenced in the letter, to draw attention to it.
• The cover image in the leaflets should be changed to reflect the purpose of the survey.
• More infographics should be included in the leaflet.
• The order of the text in the leaflet should be reviewed to make the flow more logical.
• A title such as ‘Important Information’ should be added to the leaflet.
• The wording of the postcard should be reviewed to ensure the tone is more personal.
• The responsiveness of the platform should be reviewed to ensure participants are able to log in quickly and easily.
• When participants are prompted to input their access code into the three boxes on the portal, the cursor should automatically skip to the next box once four digits are inputted.
• Change the ‘save and continue’ button to ‘enter and continue’ on the portal opening page.

Stage 2: recruitment material review

Most participants agreed that they preferred the overall look of the revised standard letter, which had a box with red shading in the middle and the three steps split into further boxes within it. It was felt that this was more eye-catching, and the clearly-defined steps were welcomed. A majority also preferred the tone and structure of the edited version compared with the original. Most participants were also in favour of having more information in the opening paragraph explaining the purpose of the survey and the types of topics included.

As at Stage 1, several participants picked up on the sentence “Please provide your phone number within 7 days. If we don’t hear from you, we may make a brief, socially-distanced visit to collect your phone number”, generally commenting that it was too strong in tone. However, it was considered important to keep a focus on the seven-day time limit to provide details.

As in the case of the standard letter, most participants preferred the revised version of the online letter in terms of look, tone, and layout. In particular, the sentence “There is also a short online section at the end for anyone who uses the internet” was well received. It was considered more positive than the original “let the interviewer know if you don’t use the internet” and reassuring for those who are not digitally engaged.

All participants felt that there was enough information in the letter to indicate that this is a telephone survey with an additional online element, but many missed the detail about the online element the first time they read the letter. They were in agreement that this could be highlighted further to help participants understand the requirement.

Overall, the revised version of the standard leaflet was preferred by participants to the original version, although this finding was not as clear-cut as in the case of the letters. Although most preferred the infographics as a concept for the cover and considered this a more logical choice, many felt there was something ‘missing’ or that needed to be refined about it. Although some preferred the image of the castle aesthetically, most felt that it gave the impression of a tourism leaflet, which would not draw people to read it and may miss important information as a result.

In terms of the content of the leaflet, participants generally preferred the question-and-answer format adopted in the newly revised version. However, a few said that there could be a little more of a general introductory paragraph followed by the title ‘Frequently Asked Questions’, which they considered more logical.

A particular omission mentioned by a few participants was information on participating in the survey if you have additional needs for example, a hearing issue; or if English (or Welsh) is not your first language. Although it was recognised someone might call the helpline if they needed further information on accessibility, having the information on the leaflet would improve the feeling of inclusivity.

As in Stage 1, participants were generally supportive of the tone of the postcard in its original and edited form and did not have as much feedback to provide as on the other materials. Although the differences between the original and edited versions were small, participants generally preferred the revised version, considering it more friendly and ‘to the point’.

Participants tended to like the fact it contains direct contact details for an interviewer and generally saw the approach as more personal. A few felt it could be even more personable by including a statement such as “you can also contact me with any questions you have about the survey” if practical to do so.

In terms of practicality, all participants said they would be motivated to take part and would log in to give their details if the materials arrived through the post. It was not felt that the materials appeared suspicious at all and it was felt that the materials tie together well in the way they are branded, and the colours used were liked.

Most participants considered the length of the survey to be longer than they would like it to be, particularly when factoring in the online element. There was a general consensus that 20 minutes is about right with 30 minutes as an upper limit. However, as the materials make the length of the survey clear, it was felt people knowingly give their details while understanding the time commitment and therefore would put the time aside. There were no participants who said the length of the survey would put them off taking part completely.

Stage 3: testing of the online element

Based on the letter and leaflet, all participants would be happy to take part in the online section. There was nothing of note that would put anyone off and it was not felt by anyone that the materials appeared untrustworthy or not genuine.

Participants were unanimous in the view that the process of accessing the survey, and indeed navigating through the questions, was straightforward and felt the platform worked well. All felt it was obvious what needed doing when logging on and felt the survey was user-friendly, the questions were well laid out and well-spaced. There weren’t any glitches reported and no participants reported that any of the pages had been slow to load.

Participants that completed the survey on a computer and smartphone agreed that a clickable link would be the easiest solution for accessing the survey online immediately after the telephone section. It was felt this would improve accessibility for those less confident in using technology and the internet. Although participants did not have a significant issue with the 12-digit code, it was felt a clickable link would streamline the process. There was also some support for a shorter code which could be copied and pasted as one, but the link would make things a step easier again.

There was also support for having a progress bar on the screen, something participants reported as being useful in other surveys they’ve completed.

Based on the feedback at Stage 2 of the materials testing and the testing of the online survey element, we have made the following recommendations.

Standard letter

• The look of the revised letter should be retained, particularly the design of the central information box.
• The opening paragraph should be reviewed, it must contain as much up-front information as possible on the themes covered in the survey and who will use the data. Consideration should also be given to the look of this paragraph, in particular if there is a way to make it look ‘cleaner’.
• Consideration should be given to the sentence “please provide your phone number within 7 days. If we don’t hear from you, we may make a brief, socially-distanced visit to collect your phone number” to soften the tone for the reader. One option could be including “please provide your phone number within 7 days” in the main information box and using an asterisk or footnote to include information about the socially-distanced visit in small print.
• The Welsh translation of “the survey takes place by telephone, so please let us know your phone number” should be reviewed to ensure it matches the tone of the English version.
• Including a QR code should be considered to make accessing the portal as easy as possible.

Online letter

• The overall look and branding used in the revised version should remain.
• Another ‘step’ should be added in the central information box to indicate the online section of questions – this would improve understanding of what is entailed.

Standard and online leaflet

• Further work is required to refine the cover but the approach of including infographics is preferable. Considerations should include adding a background colour, ensuring the same style is applied to all the text, and possibly including more icons or infographics where possible.
• The title ‘Frequently Asked Questions’ should be placed below the introductory paragraph, but the Q&A style adopted thereafter should remain.
• The phrase “we value your contribution” should be included in the leaflet somewhere.
• The phrase “so we need your phone number” should be changed to “please provide your phone number”.
• Consideration should be given to including text about support provided for people with additional needs to complete the survey.
• Further consideration should be given to improving clarity around the selection of a person from a household to participate.

Postcard

• The text included in the revised version should remain.
• Consideration should be given to adding an additional sentence to enable the participant to contact the interviewer for further survey information, if appropriate.

Online survey platform

• Ease of use on a smartphone should be considered – the adoption of a unique clickable link to access the survey from a text message or email should be considered.
• If a unique access link approach is not adopted, the use of a shorter access code should be considered to allow participants to copy and paste the code into the survey login page.
• A progress bar should be added to improve survey navigation.

Report Authors: Angharad Davies, Opinion Research Services (ORS) Ltd.

Views expressed in this report are those of the researchers and not necessarily those of the Welsh Government.

For further information please contact:
Chris McGowan
Email: surveys@gov.wales

Social research number: 11/2022

Image

Digital ISBN 978-1-80391-651-4