lm_eval/tasks/leaderboard/bbh_mc/causal_judgement.yaml

dataset_name: causal_judgement
description: 'Answer questions about causal attribution.

        '
doc_to_choice: ["Yes", "No"]
fewshot_config:
  sampler: first_n
  samples:
  - input: 'How would a typical person answer each of the following questions about
      causation?

      Frank T., had an ongoing dispute with his neighbor over a stretch of land and
      one day decided to shoot his neighbor in the body. Frank T. had no experience
      with guns, his hand slipped on the barrel of the gun, and the shot went wild.
      Nonetheless, the bullet bounced off a large boulder several feet away and hit
      the neighbor''s body, causing significant injury. Did Frank T. intentionally
      shoot his neighbor in the body?

      Options:

      - Yes

      - No'
    target: 'No'
  - input: 'How would a typical person answer each of the following questions about
      causation?

      Suzy and Billy are working on a project that is very important for our nation''s
      security. The boss tells them both: "Be sure that you are here at exactly 9
      am. It is absolutely essential that you arrive at that time." Both Billy and
      Suzy arrive at 9 am. As it happens, there was a motion detector installed in
      the room where they arrived. The motion detector was set up to be triggered
      if at least one person appeared in the room at the same time. So the motion
      detector went off. Did Billy cause the motion detector to go off?

      Options:

      - Yes

      - No'
    target: 'Yes'
  - input: 'How would a typical person answer each of the following questions about
      causation?

      George and his sister Lena reunite at their parents'' house for Thanksgiving.
      Whereas George just got into medical school, Lena is unhappy in her marriage
      and recently lost her job. Over the course of the day, George and Lena get into
      a number of heated arguments. Later in the afternoon they play a game of darts.
      They split the first two games, and the third game is close until the end. Who
      will win comes down to George''s last shot. If he hits a high point region,
      he wins; if he hits a low point region, Lena wins. George thinks of the difficult
      time Lena is having, and he really wants to let her win. He aims the dart at
      the low point region. He sets up his shot and the dart lands in the low point
      region. After his shot, Lena wins the game and is very happy. Did George hit
      the low point region intentionally?

      Options:

      - Yes

      - No'
    target: 'Yes'
include: _fewshot_template_yaml
task: leaderboard_bbh_causal_judgement