Feature/pattern finder #23

pa-shk · 2023-11-15T16:39:51Z

Implemented automatic pattern finder for answers scored 0 and 1. Below is a summary of the most common patterns for correct and incorrect answers for 6 tasks extracted with this script.

bigger_number

most common correct patterns

answer	certainty	score	count
n1 is bigger than n2.	1	1	1027
n2 is bigger than n1.	1	1	995
n1 is bigger than n2.	1	1	77
n1 is bigger.	1	1	29
* n1 is larger than n2.	0	1	4
n1 is smaller than n2.	1	1	3

most common incorrect patterns

answer	certainty	score	count
n2 is bigger than n1.	0	0	113
n1 is bigger than n2.	0	0	15
n1 + n2 = 865 so the bigger number is 865.	0	0	3
n1 + n2 = 999 so the bigger number is 999.	0	0	3

smaller_number

most common correct patterns

answer	certainty	score	count
n2 is smaller than n1.	1	1	736
n1 is smaller than n2.	1	1	670
n1 < n2 so the smaller number is n1.	0	1	100
n1 is larger than n2.	1	1	85
n1 < n2 therefore, the smaller number is n1.	0	1	12

most common incorrect patterns

answer	certainty	score	count
n2 is smaller than n1.	0	0	596
n1 is smaller than n2.	0	0	318
n1 - n2 =	0	0	117
n1 is larger than n2.	0	0	24
n1 < n2 so the smaller number is n1.	0	0	15
n1 < n2 therefore, the smaller number is n1.	0	0	7

first_alphabetically

most common correct patterns

answer	certainty	score	count
"answer" comes first in alphabetical order.	1	1	871
"answer" comes before "distractor" in the alphabet.	0	1	300
"answer" comes first alphabetically.	1	1	144
"answer" comes first.	1	1	117
"answer" comes before "distractor" alphabetically.	1	1	113
"answer" comes before "distractor" in an alphabetical order.	1	1	66
"answer" comes before "distractor" in alphabetical order.	1	1	37
"answer" comes first in an alphabetical order.	1	1	25

most common incorrect patterns

answer	certainty	score	count
"distractor" comes first in alphabetical order.	0	0	644
"distractor" comes before "answer" in the alphabet.	0	0	213
"distractor" comes first alphabetically.	0	0	133
"distractor" comes first.	0	0	119
"distractor" comes before "answer" alphabetically.	0	0	78
"distractor" comes before "answer" in an alphabetical order.	0	0	25
"distractor" comes before "answer" in alphabetical order.	0	0	24
"distractor" comes first in an alphabetical order.	0	0	22

homophones

most common correct patterns

answer	certainty	score	count
"answer" sounds more like "query".	1	1	270
"answer" is a homophone of "query".	1	1	228
"answer"	1	1	160
"answer" sounds like "query".	1	1	135
"query" is a homophone of "answer".	1	1	131
"answer" is a homophone of the word "query".	1	1	118
"query" sounds like "answer".	1	1	41

most common incorrect patterns

answer	certainty	count
"distractor" sounds more like "query".	1	311
"distractor" is a homophone of "query".	1	106
"distractor" is a homophone of the word "query".	1	85
"query"	1	61
"distractor" sounds like "query".	1	46
"query" sounds like "query".	1	36
"distractor"	1	36
both words sound similar to "query".	1	35
both words have the same vowel sound as "query".	1	22
"answer" is a homophone of "query" and "distractor".	0	19
"query" is a homophone of "query".	1	16

rhyming_word

most common correct patterns

answer	certainty	score	count
"answer" is a rhyme of the word "query".	1	1	424
"answer" rhymes with "query".	1	1	306
"answer"	1	1	159
"query" is a rhyme of the word "answer".	1	1	24
"query" and "answer" rhyme.	1	1	17
"query" is a rhyme of "answer".	1	1	11
the word that rhymes with "query" is "answer".	1	1	8
answer.	1	1	5
"query" rhymes with "answer".	1	1	3
"answer" is a rhyme of the query "query".	1	1	1

most common incorrect patterns

answer	certainty	count
"distractor" is a rhyme of the word "query".	0	363
"distractor" rhymes with "query".	0	275
"distractor"	1	116
neither one does.	0	38
neither one of them does.	0	33
both words rhyme with "query".	0	22
"query" is a rhyme of the word "distractor".	0	21
"query" is a rhyme of the word "query".	0	14
"query" and "answer" are a rhyme.	0	11
"query"	1	9
"query" is a rhyme of "distractor".	0	8

less_letters

most common correct patterns

answer	score	count
"word2" has fewer letters than "word1".	1	371
"word1" is shorter than "word2".	1	330
"word1" has fewer letters than "word2".	1	322
"word2" has fewer letters.	1	194
"word2" is shorter than "word1".	1	154
"word1" has fewer letters.	1	42
"word1" is shorter.	1	30
"word2" is shorter.	1	8

most common incorrect patterns

answer	certainty	score	count
"word2" has fewer letters than "word1".	0	0	426
"word1" has fewer letters than "word2".	0	0	384
"word1" is shorter than "word2".	0	0	337
"word2" has fewer letters.	0	0	181
"word2" is shorter than "word1".	0	0	124
"word1" has fewer letters.	0	0	51
"word1" is shorter.	0	0	7
"word2" is shorter.	0	0	5

vvchernov · 2023-11-26T08:58:51Z

Hello @pa-shk! I have some questions:

I think for number tasks it is better to use {answer} and {distractor} instead of {n1} and {n2}. It will lead to join of some patterns together
What is the differences between the first and the third correct patterns for "bigger_number" task? They look the same: "n1 is bigger than n2."
Why do the third and the fifth correct patterns for "smaller_number" task have certainty 0? The same question for the second correct pattern for "first_alphabetically" task and for all correct pattern of "less_letters" task. Do we need to add them to system pattern (certainty -> 1)?
I think better terms are positive (instead of "correct") and negative (instead of "incorrect")
Need to discuss case when certainty = 1 score = 0. What do we need from it? May be linguistic analysis gives as case when the correct answer can be not contain the gold word, but I think it is rare case

vvchernov · 2023-11-26T09:02:24Z

Please refactor code to more standard form:

def parse_args():
  parser = argparse.ArgumentParser()
  ...

  args = parser.parse_args()
  return args

def main():
  args = parse_args()
  ...

if __name__ == "__main__":
  main()

pa-shk · 2023-11-27T11:51:53Z

Hello @vvchernov! Thank you for your feedback!

Speaking about your questions,

Yes, it may help to extract more general patterns, though it seems like I need to write a specific script (or make some changes to the existing one) to implement it. n1 and n2 are just retrieved from metadata, while to convert them to answer and distractor few lines should be added. But it’s definitely not difficult and good to try.
The only difference between these two patterns is the presence of space in the beginning of the second answer (in markdown they look identical, true). I think it’s possible to merge them together.
Sure, we can add pattern n1 < n2 (therefore|so) to the list of prefixes used by the scorer.
Agree. Patterns which are considered correct by the scorer can actually be incorrect and vice versa.
The simple solution is to score all patterns which are the exact opposite of correct patterns as incorrect patterns with certainty 1. For example, "distractor" is a rhyme of the word "query" in rhyming_word word task. Whether we can apply linguistic analysis to detect other cases is a bit unclear, though.

vvchernov · 2023-11-28T12:03:08Z

You approach with pass of all metadata is good. I suggest to add filter which compare values (strings) with "answer" ("distractor") and if match replace by "answer" ("distractor") tag

pa-shk · 2023-12-06T22:26:14Z

Updated version of the summary of the extracted patterns.

bigger_number

most common positive patterns

answer	certainty	score	count	prompt	prompt_number
answer is bigger than distractor.	1	1	497	q: which number is bigger, distractor or answer?	1
answer is bigger than distractor.	1	1	429	q: which number is bigger, answer or distractor?	1
answer is bigger.	1	1	28	q: which number is bigger, answer or distractor?	1
answer is bigger. q: which number is bigger, answer or distractor? answer is bigger. q: which number is bigger, answer or distractor? answer is bigger. q: which number is bigger	0	1	2	q: which number is bigger, answer or distractor?	1
answer is bigger than distractor.	1	1	498	q: of the numbers distractor and answer, which is bigger?	2
answer is bigger than distractor.	1	1	421	q: of the numbers answer and distractor, which is bigger?	2
answer is bigger than distractor.	1	1	254	from the numbers answer and distractor, write the bigger number:	3
* answer is larger than distractor.	0	1	4	from the numbers answer and distractor, write the bigger number:	3
distractor is smaller than answer.	1	1	3	from the numbers distractor and answer, write the bigger number:	3
answer > distractor so the bigger number is answer.	0	1	2	from the numbers answer and distractor, write the bigger number:	3
answer is bigger than distractor	1	1	2	from the numbers answer and distractor, write the bigger number:	3
answer is the bigger number.	1	1	2	from the numbers distractor and answer, write the bigger number:	3

most common negative patterns

answer	count	prompt	prompt_number
distractor is bigger than answer.	35	q: which number is bigger, answer or distractor?	1
distractor is bigger than answer.	3	q: which number is bigger, distractor or answer?	1
distractor is bigger than answer.	78	q: of the numbers answer and distractor, which is bigger?	2
distractor is bigger than answer.	11	from the numbers distractor and answer, write the bigger number:	3
distractor + answer = 1101 so the bigger number is 1101.	3	from the numbers distractor and answer, write the bigger number:	3

smaller_number

most common positive patterns

answer	certainty	score	count	prompt	prompt_number
answer is smaller than distractor.	1	1	422	q: which number is smaller, distractor or answer?	1
answer is smaller than distractor.	1	1	130	q: which number is smaller, answer or distractor?	1
answer is smaller than distractor.	1	1	310	q: of the numbers distractor and answer, which is smaller?	2
answer is smaller than distractor.	1	1	273	q: of the numbers answer and distractor, which is smaller?	2
distractor is larger than answer.	1	1	3	q: of the numbers distractor and answer, which is smaller?	2
answer is smaller than distractor.	1	1	267	from the numbers answer and distractor, write the smaller number:	3
answer < distractor so the smaller number is answer.	0	1	88	from the numbers answer and distractor, write the smaller number:	3
distractor is larger than answer.	1	1	82	from the numbers distractor and answer, write the smaller number:	3
answer < distractor so the smaller number is answer.	0	1	12	from the numbers answer and distractor, write the smaller number:	3
answer < distractor therefore, the smaller number is answer.	0	1	12	from the numbers answer and distractor, write the smaller number:	3
answer is smaller than distractor.	1	1	4	from the numbers distractor and answer, write the smaller number:	3
answer < distractor so, the smaller number is answer.	0	1	2	from the numbers answer and distractor, write the smaller number:	3
distractor is larger than answer, so the smaller number is answer.	0	1	2	from the numbers distractor and answer, write the smaller number:	3

most common negative patterns

answer	count	prompt	prompt_number
distractor is smaller than answer.	369	q: which number is smaller, answer or distractor?	1
distractor is smaller than answer.	78	q: which number is smaller, distractor or answer?	1
distractor is smaller than answer.	227	q: of the numbers answer and distractor, which is smaller?	2
distractor is smaller than answer.	187	q: of the numbers distractor and answer, which is smaller?	2
distractor - answer =	117	from the numbers distractor and answer, write the smaller number:	3
distractor is smaller than answer.	53	from the numbers distractor and answer, write the smaller number:	3
answer is larger than distractor.	24	from the numbers answer and distractor, write the smaller number:	3
distractor < answer so the smaller number is distractor.	15	from the numbers distractor and answer, write the smaller number:	3
distractor < answer therefore, the smaller number is distractor.	7	from the numbers distractor and answer, write the smaller number:	3

first_alphabetically

most common positive patterns

answer	certainty	score	count	prompt	prompt_number
"answer" comes first in alphabetical order.	1	1	250	q: in an alphabetical order, which of the words "answer" and "distractor" comes first?	1
"answer" comes first.	1	1	100	q: in an alphabetical order, which of the words "answer" and "distractor" comes first?	1
"answer" comes first in alphabetical order.	1	1	81	q: in an alphabetical order, which of the words "distractor" and "answer" comes first?	1
"answer" comes before "distractor" in an alphabetical order.	1	1	62	q: in an alphabetical order, which of the words "answer" and "distractor" comes first?	1
"answer" comes before "distractor" in alphabetical order.	1	1	30	q: in an alphabetical order, which of the words "answer" and "distractor" comes first?	1
"answer" comes first in an alphabetical order.	1	1	20	q: in an alphabetical order, which of the words "answer" and "distractor" comes first?	1
"answer" comes before "distractor" in alphabetical order.	1	1	7	q: in an alphabetical order, which of the words "distractor" and "answer" comes first?	1
"answer" comes before "distractor" in an alphabetical order.	1	1	4	q: in an alphabetical order, which of the words "distractor" and "answer" comes first?	1
"answer" comes first in an alphabetical order.	1	1	3	q: in an alphabetical order, which of the words "distractor" and "answer" comes first?	1
answer: "answer" comes first.	1	1	2	q: in an alphabetical order, which of the words "answer" and "distractor" comes first? answer:	1
"answer" comes first in alphabetical order.	1	1	362	q: in an alphabetical order, which word comes first, "answer" or "distractor"?	2
"answer" comes first in alphabetical order.	1	1	178	q: in an alphabetical order, which word comes first, "distractor" or "answer"?	2
"answer" comes first.	1	1	9	q: in an alphabetical order, which word comes first, "answer" or "distractor"?	2
"answer" comes before "distractor" in the alphabet.	0	1	243	q: of the words "answer" and "distractor", which word comes first alphabetically?	3
"answer" comes before "distractor" alphabetically.	1	1	101	q: of the words "distractor" and "answer", which word comes first alphabetically?	3
"answer" comes first alphabetically.	1	1	97	q: of the words "answer" and "distractor", which word comes first alphabetically?	3
"answer" comes before "distractor" in the alphabet.	0	1	54	q: of the words "distractor" and "answer", which word comes first alphabetically?	3
"answer" comes first alphabetically.	1	1	46	q: of the words "distractor" and "answer", which word comes first alphabetically?	3
"answer" comes before "distractor" alphabetically.	1	1	13	q: of the words "answer" and "distractor", which word comes first alphabetically?	3

most common negative patterns

answer	count	prompt	prompt_number
"distractor" comes first in alphabetical order.	210	q: in an alphabetical order, which of the words "distractor" and "answer" comes first?	1
"distractor" comes first.	105	q: in an alphabetical order, which of the words "distractor" and "answer" comes first?	1
"distractor" comes before "answer" in an alphabetical order.	25	q: in an alphabetical order, which of the words "distractor" and "answer" comes first?	1
"distractor" comes before "answer" in alphabetical order.	24	q: in an alphabetical order, which of the words "distractor" and "answer" comes first?	1
"distractor" comes first in an alphabetical order.	22	q: in an alphabetical order, which of the words "distractor" and "answer" comes first?	1
"distractor" comes first in alphabetical order.	14	q: in an alphabetical order, which of the words "answer" and "distractor" comes first?	1
"distractor" comes first in alphabetical order.	309	q: in an alphabetical order, which word comes first, "distractor" or "answer"?	2
"distractor" comes first in alphabetical order.	111	q: in an alphabetical order, which word comes first, "answer" or "distractor"?	2
"distractor" comes first.	11	q: in an alphabetical order, which word comes first, "distractor" or "answer"?	2
"distractor" comes before "answer" in the alphabet.	183	q: of the words "distractor" and "answer", which word comes first alphabetically?	3
"distractor" comes first alphabetically.	98	q: of the words "distractor" and "answer", which word comes first alphabetically?	3
"distractor" comes before "answer" alphabetically.	64	q: of the words "answer" and "distractor", which word comes first alphabetically?	3
"distractor" comes first alphabetically.	34	q: of the words "answer" and "distractor", which word comes first alphabetically?	3
"distractor" comes before "answer" in the alphabet.	29	q: of the words "answer" and "distractor", which word comes first alphabetically?	3
"distractor" comes before "answer" alphabetically.	13	q: of the words "distractor" and "answer", which word comes first alphabetically?	3

homophones

most common positive patterns

answer	certainty	score	count	prompt	prompt_number
"answer" sounds like "query".	1	1	106	q: which word sounds like the word "query", "answer" or "distractor"?	1
"answer"	1	1	104	q: which word sounds like the word "query", "distractor" or "answer"?	1
"answer"	1	1	37	q: which word sounds like the word "query", "answer" or "distractor"?	1
"answer" sounds like "query".	1	1	29	q: which word sounds like the word "query", "distractor" or "answer"?	1
"query" sounds like "answer".	1	1	23	q: which word sounds like the word "query", "answer" or "distractor"?	1
"query" sounds like "answer".	1	1	18	q: which word sounds like the word "query", "distractor" or "answer"?	1
"answer" sounds more like "query".	1	1	185	q: of the two words "answer" and "distractor", which one sounds more like "query"?	2
"answer" sounds more like "query".	1	1	75	q: of the two words "distractor" and "answer", which one sounds more like "query"?	2
"answer"	1	1	15	q: of the two words "answer" and "distractor", which one sounds more like "query"?	2
"answer" sounds more like "query".	1	1	6	q: of the query words "answer" and "distractor", which one sounds more like "query"?	2
"answer"	1	1	4	q: of the two words "distractor" and "answer", which one sounds more like "query"?	2
"answer" sounds more like "query".	1	1	2	q: of the query words "distractor" and "answer", which one sounds more like "query"?	2
"answer" is a homophone of "query".	1	1	163	q: which is a homophone of the word "query", "answer" or "distractor"?	3
"answer" is a homophone of the word "query".	1	1	76	q: which is a homophone of the word "query", "answer" or "distractor"?	3
"answer" is a homophone of "query".	1	1	65	q: which is a homophone of the word "query", "distractor" or "answer"?	3
"query" is a homophone of "answer".	1	1	65	q: which is a homophone of the word "query", "distractor" or "answer"?	3
"query" is a homophone of "answer".	1	1	65	q: which is a homophone of the word "query", "answer" or "distractor"?	3
"answer" is a homophone of the word "query".	1	1	42	q: which is a homophone of the word "query", "distractor" or "answer"?	3

most common negative patterns

answer	certainty	count	prompt	prompt_number
"distractor" sounds like "query".	1	40	q: which word sounds like the word "query", "distractor" or "answer"?	1
"query"	1	24	q: which word sounds like the word "query", "answer" or "distractor"?	1
"query"	1	21	q: which word sounds like the word "query", "distractor" or "answer"?	1
"distractor"	1	20	q: which word sounds like the word "query", "answer" or "distractor"?	1
"query" sounds like "query".	1	18	q: which word sounds like the word "query", "distractor" or "answer"?	1
"query" sounds like "query".	1	18	q: which word sounds like the word "query", "answer" or "distractor"?	1
"query" q: which word sounds like the word "query", "answer" or "distractor"?	0	15	q: which word sounds like the word "query", "answer" or "distractor"?	1
"answer" q: which word sounds like the word "query", "answer" or "distractor"?	0	12	q: which word sounds like the word "query", "answer" or "distractor"?	1
"query" q: which word sounds like the word "query", "distractor" or "answer"?	0	9	q: which word sounds like the word "query", "distractor" or "answer"?	1
"distractor" sounds more like "query".	1	218	q: of the two words "distractor" and "answer", which one sounds more like "query"?	2
"distractor" sounds more like "query".	1	87	q: of the two words "answer" and "distractor", which one sounds more like "query"?	2
both words sound similar to "query".	1	21	q: of the two words "answer" and "distractor", which one sounds more like "query"?	2
both words sound similar to "query".	1	14	q: of the two words "distractor" and "answer", which one sounds more like "query"?	2
both words have the same vowel sound as "query".	1	13	q: of the two words "answer" and "distractor", which one sounds more like "query"?	2
both words have the same vowel sound as "query".	1	9	q: of the two words "distractor" and "answer", which one sounds more like "query"?	2
"distractor" is a homophone of "query".	1	90	q: which is a homophone of the word "query", "distractor" or "answer"?	3
"distractor" is a homophone of the word "query".	1	62	q: which is a homophone of the word "query", "distractor" or "answer"?	3
"distractor" is a homophone of the word "query".	1	23	q: which is a homophone of the word "query", "answer" or "distractor"?	3
"answer" is a homophone of "query" and "distractor".	0	16	q: which is a homophone of the word "query", "distractor" or "answer"?	3
"distractor" is a homophone of "query".	1	16	q: which is a homophone of the word "query", "answer" or "distractor"?	3
"query" is a homophone of "answer" and "distractor".	0	13	q: which is a homophone of the word "query", "answer" or "distractor"?	3

rhyming_word

most common positive patterns

answer	certainty	score	count	prompt	prompt_number
"answer"	1	1	3	q: which word rhymes with the word "query", "distractor" or "answer"?	1
"answer" is a rhyme of the word "query".	1	1	301	q: which is a rhyme of the word "query", "answer" or "distractor"?	2
"answer" is a rhyme of the word "query".	1	1	123	q: which is a rhyme of the word "query", "distractor" or "answer"?	2
"query" is a rhyme of the word "answer".	1	1	20	q: which is a rhyme of the word "query", "answer" or "distractor"?	2
"query" and "answer" rhyme.	1	1	15	q: which is a rhyme of the word "query", "distractor" or "answer"?	2
"query" is a rhyme of "answer".	1	1	10	q: which is a rhyme of the word "query", "answer" or "distractor"?	2
"query" is a rhyme of the word "answer".	1	1	4	q: which is a rhyme of the word "query", "distractor" or "answer"?	2
"query" and "answer" rhyme.	1	1	2	q: which is a rhyme of the word "query", "answer" or "distractor"?	2
"answer" rhymes with "query".	1	1	203	q: of the words "answer" and "distractor", which one rhymes with "query"?	3
"answer" rhymes with "query".	1	1	101	q: of the words "distractor" and "answer", which one rhymes with "query"?	3
"answer"	1	1	81	q: of the words "answer" and "distractor", which one rhymes with "query"?	3
"answer"	1	1	75	q: of the words "distractor" and "answer", which one rhymes with "query"?	3
the word that rhymes with "query" is "answer".	1	1	5	q: of the words "answer" and "distractor", which one rhymes with "query"?	3
answer.	1	1	3	q: of the words "answer" and "distractor", which one rhymes with "query"?	3
the word that rhymes with "query" is "answer".	1	1	3	q: of the words "distractor" and "answer", which one rhymes with "query"?	3
answer.	1	1	2	q: of the words "distractor" and "answer", which one rhymes with "query"?	3

most common negative patterns

answer	certainty	count	prompt	prompt_number
"answer" and "distractor" both rhyme with "query".	0	7	q: which word rhymes with the word "query", "answer" or "distractor"?	1
the word that rhymes with "query" is "answer". the word that rhymes with	0	6	q: which word rhymes with the word "query", "answer" or "distractor"?	1
none of the words "query", "answer" or "distractor" rhyme with each other.	0	6	q: which word rhymes with the word "query", "answer" or "distractor"?	1
"query" rhymes with "query", "distractor" rhymes with "distractor	0	5	q: which word rhymes with the word "query", "distractor" or "answer"?	1
"distractor" is a rhyme of the word "query".	0	267	q: which is a rhyme of the word "query", "distractor" or "answer"?	2
"distractor" is a rhyme of the word "query".	0	96	q: which is a rhyme of the word "query", "answer" or "distractor"?	2
"query" is a rhyme of the word "distractor".	0	15	q: which is a rhyme of the word "query", "distractor" or "answer"?	2
"query" is a rhyme of the word "query".	0	8	q: which is a rhyme of the word "query", "answer" or "distractor"?	2
"query" and "answer" are a rhyme.	0	7	q: which is a rhyme of the word "query", "answer" or "distractor"?	2
"query" is a rhyme of "distractor".	0	6	q: which is a rhyme of the word "query", "distractor" or "answer"?	2
"query" is a rhyme of the word "query".	0	6	q: which is a rhyme of the word "query", "distractor" or "answer"?	2
"query" is a rhyme of the word "distractor".	0	6	q: which is a rhyme of the word "query", "answer" or "distractor"?	2
"distractor" rhymes with "query".	0	196	q: of the words "distractor" and "answer", which one rhymes with "query"?	3
"distractor" rhymes with "query".	0	79	q: of the words "answer" and "distractor", which one rhymes with "query"?	3
"distractor"	1	60	q: of the words "answer" and "distractor", which one rhymes with "query"?	3
"distractor"	1	55	q: of the words "distractor" and "answer", which one rhymes with "query"?	3
neither one does.	0	24	q: of the words "answer" and "distractor", which one rhymes with "query"?	3
neither one of them does.	0	17	q: of the words "answer" and "distractor", which one rhymes with "query"?	3
neither one of them does.	0	16	q: of the words "distractor" and "answer", which one rhymes with "query"?	3
neither one does.	0	14	q: of the words "distractor" and "answer", which one rhymes with "query"?	3
both words rhyme with "query".	0	11	q: of the words "answer" and "distractor", which one rhymes with "query"?	3
both words rhyme with "query".	0	11	q: of the words "distractor" and "answer", which one rhymes with "query"?	3
"query"	1	7	q: of the words "answer" and "distractor", which one rhymes with "query"?	3
neither one of them rhymes with "query".	0	5	q: of the words "distractor" and "answer", which one rhymes with "query"?	3

less_letters

most common positive patterns

answer	score	count	prompt	prompt_number
"answer" has fewer letters than "distractor".	1	286	q: which word has fewer letters, "distractor" or "answer"?	1
"answer" has fewer letters than "distractor".	1	173	q: which word has fewer letters, "answer" or "distractor"?	1
"answer" has fewer letters.	1	6	q: which word has fewer letters, "distractor" or "answer"?	1
"answer" is shorter than "distractor".	1	330	q: which word is shorter, "answer" or "distractor"?	2
"answer" is shorter than "distractor".	1	154	q: which word is shorter, "distractor" or "answer"?	2
"answer" is shorter.	1	30	q: which word is shorter, "answer" or "distractor"?	2
"answer" is shorter.	1	8	q: which word is shorter, "distractor" or "answer"?	2
"answer" is shorter than "distractor."	1	2	q: which word is shorter, "answer" or "distractor"?	2
"answer" has fewer letters.	1	184	q: of the words "distractor" and "answer" which one has fewer letters?	3
"answer" has fewer letters than "distractor".	1	148	q: of the words "answer" and "distractor" which one has fewer letters?	3
"answer" has fewer letters than "distractor".	1	85	q: of the words "distractor" and "answer" which one has fewer letters?	3
"answer" has fewer letters.	1	39	q: of the words "answer" and "distractor" which one has fewer letters?	3
answer has fewer letters than distractor.	1	2	q: of the words "answer" and "distractor" which one has fewer letters?	3

most common negative patterns

answer	count	prompt	prompt_number
"distractor" has fewer letters than "answer".	309	q: which word has fewer letters, "answer" or "distractor"?	1
"distractor" has fewer letters than "answer".	206	q: which word has fewer letters, "distractor" or "answer"?	1
"distractor" has fewer letters.	6	q: which word has fewer letters, "answer" or "distractor"?	1
"distractor" is shorter than "answer".	333	q: which word is shorter, "distractor" or "answer"?	2
"distractor" is shorter than "answer".	124	q: which word is shorter, "answer" or "distractor"?	2
"distractor" is shorter.	7	q: which word is shorter, "distractor" or "answer"?	2
"distractor" is shorter.	5	q: which word is shorter, "answer" or "distractor"?	2
"distractor" has fewer letters.	175	q: of the words "answer" and "distractor" which one has fewer letters?	3
"distractor" has fewer letters than "answer".	173	q: of the words "distractor" and "answer" which one has fewer letters?	3
"distractor" has fewer letters than "answer".	116	q: of the words "answer" and "distractor" which one has fewer letters?	3
"distractor" has fewer letters.	50	q: of the words "distractor" and "answer" which one has fewer letters?	3
distractor has fewer letters than answer.	5	q: of the words "distractor" and "answer" which one has fewer letters?	3

pa-shk · 2023-12-27T12:24:19Z

Accuracy per task and template with estimated standard deviation

	task_name	template_1	template_2	template_3	prompts_per_templ	template_1_std	template_2_std	template_3_std
0	smaller_number	0.553	0.586	0.484	1000.0	0.015722	0.015576	0.015803
1	less_letters	0.463	0.525	0.241	1000.0	0.015768	0.015792	0.013525
2	homophones	0.400	0.362	0.595	800.0	0.017321	0.016991	0.017356
3	first_alphabetically	0.578	0.563	0.565	1000.0	0.015618	0.015685	0.015677
4	rhyming_word	0.006	0.478	0.476	1000.0	0.002442	0.015796	0.015793
5	bigger_number	0.962	0.920	0.280	1000.0	0.006046	0.008579	0.014199

vvchernov · 2023-12-27T13:14:00Z

Hello @pa-shk! Could you describe meanings of titles of your latter table ("t0_var", "t0_var_from_mean" and so on)?

pa-shk · 2024-01-08T10:51:10Z

Hello @vvchernov! I've fixed the wrong calculations in the table and added a diagram

pa-shk added 3 commits November 15, 2023 13:50

pattern finder implementation

0a20fa1

small improvement

b359b07

pattern finder for correct/incorrect answers

a9768c0

vvchernov mentioned this pull request Nov 26, 2023

Correct/wrong answer pattern finder #19

Open

pa-shk added 2 commits December 26, 2023 22:34

separate patterns by question templates

ac30583

refactor pattern_finder.py

3197b7b

add file with statistics and visualisation

737f976

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/pattern finder #23

Feature/pattern finder #23

pa-shk commented Nov 15, 2023 •

edited

Loading

vvchernov commented Nov 26, 2023

vvchernov commented Nov 26, 2023

pa-shk commented Nov 27, 2023

vvchernov commented Nov 28, 2023

pa-shk commented Dec 6, 2023

pa-shk commented Dec 27, 2023 •

edited

Loading

vvchernov commented Dec 27, 2023

pa-shk commented Jan 8, 2024

Feature/pattern finder #23

Are you sure you want to change the base?

Feature/pattern finder #23

Conversation

pa-shk commented Nov 15, 2023 • edited Loading

bigger_number

smaller_number

first_alphabetically

homophones

rhyming_word

less_letters

vvchernov commented Nov 26, 2023

vvchernov commented Nov 26, 2023

pa-shk commented Nov 27, 2023

vvchernov commented Nov 28, 2023

pa-shk commented Dec 6, 2023

bigger_number

smaller_number

first_alphabetically

homophones

rhyming_word

less_letters

pa-shk commented Dec 27, 2023 • edited Loading

Accuracy per task and template with estimated standard deviation

vvchernov commented Dec 27, 2023

pa-shk commented Jan 8, 2024

pa-shk commented Nov 15, 2023 •

edited

Loading

pa-shk commented Dec 27, 2023 •

edited

Loading