Can you rate the quality of simplified sentences in English?
Information Sheet
We invite fluent English speakers to participate in an experiment on rating the quality of simplified sentences.
Before you decide to participate, it is important that you understand the purpose of the research
you will be participating in and what will happen to the data collected from you.
The study aims to collect ratings for simplifications of sentences that originally could have been considered difficult
to understand by non-native speakers of English.
The resulting dataset will be used to develop models and metrics that could potentially produce these scores automatically.
Participants must be:
Over 18 years of age.
Fluent English speakers; both native and non-native speakers are welcome to participate.
We expect to collect at least 10 quality ratings for each sentence pair.
As such, the HITs will be available until that goal is reached.
How will the experiment work?
You are asked to first take a Qualification Test. This will help you get familiar with the rating guidelines and submit suggestions for us. We will manually check your responses to verify that you have understood the task and will award you the Qualification if your ratings are considered suitable. In any case, you will be paid for taking the Qualification Test.
The Qualification Test consists of one pair of sentences: the original and its simplified version. We ask you to rate the quality of the simplification using three criteria: fluency, adequacy and simplicity. More details are provided in the Instructions.
If you pass the Qualification Test, you will be eligible to participate in the Rating Task. The rating guidelines will be the same, but this time you will be asked to rate 10 pairs of sentences
Each HIT should not take more than 10 minutes to complete.
You will be reading sentences extracted from articles in Wikipedia.
The nature of the sentences and their language are such that they are unlikely to cause offence, disturbance or distress to readers from a general audience.
However, if you feel uncomfortable with the content of any of the sentences, please let us know in the Suggestions box provided.
It is important to emphasize that the data collected will be made available for other researchers.
In addition, the results of this investigation may be published in scientific journals or conferences and may be used in further studies.
Finally, if you have any questions do not hesitate to contact Fernando Alva-Manchego (email: f.alva@sheffield.ac.uk).
Preferably, use the official Mechanical Turk mechanisms.
You can also contact Dr. Carolina Scarton (email: c.scarton@sheffield.ac.uk) if the task caused you any offence,
disturbance or distress; or you if have further complaints.
Please read the instructions carefully.
Thank you,
Fernando Alva-Manchego, Carolina Scarton, and Lucia Specia
Consent Form
In order to participate in this experiment, you must:
Be at least 18 years old and competent to provide consent.
Have read and understood the the Information Sheet explaining the research project.
Agree for the data collected to be used in anonymised way in the future
Agree to take part in the research previously described.