Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add NTP search promo P3A question #24065

Closed
DJAndries opened this issue Jul 14, 2022 · 2 comments · Fixed by brave/brave-core#14189
Closed

Add NTP search promo P3A question #24065

DJAndries opened this issue Jul 14, 2022 · 2 comments · Fixed by brave/brave-core#14189
Assignees
@DJAndries
Labels
feature/search features/P3A OS/Desktop QA Pass-Win64 QA/Yes release-notes/exclude
Milestone
1.44.x - Release

Comments

@DJAndries
Copy link
Collaborator

Description

Adding P3A question for the new tab page search promo (promo added in #23724):

Did the participant in Group C click on the promo and did they switch their default engine to Brave Search?
Brave.Search.Promo.NewTabPage

  1. No, No
  2. Yes, No
  3. No, Yes
  4. Yes, Yes

Will raise PR soon.
@DJAndries DJAndries self-assigned this Jul 14, 2022
@DJAndries DJAndries mentioned this issue Jul 14, 2022
Merged
25 tasks
@brave-builds brave-builds added this to the 1.44.x - Nightly milestone Jul 27, 2022
@GeetaSarvadnya GeetaSarvadnya added the QA/In-Progress Indicates that QA is currently in progress for that particular issue label Sep 6, 2022
@GeetaSarvadnya
Copy link

GeetaSarvadnya commented Sep 6, 2022 •
edited
Loading

Verification PASSED on

Brave | 1.44.73 Chromium: 105.0.5195.68 (Official Build) beta (64-bit)
-- | --
Revision | ad13e82529051bac6a0e65f455e6d7a1e5fd7938-refs/branch-heads/5195@{#903}
OS | Windows 10 Version 21H2 (Build 19044.1889)

India Region

Brave.Search.Promo.NewTabPage

Case 1: Brave.Search.Promo.NewTabPage = 0_PASSED

0 No, No - "Did the participant in Group C click on the promo - No" and "did they switch their default engine to Brave Search? - No"

  1. Clean profile 1.44.x
  2. Launch the browser using the cmd line flag brave.exe --enable-features="BraveSearchNTP"
  3. Open brave://local-state file and ensure the metrics Brave.Search.Promo.NewTabPage is not shown
  4. Open a NTP and ensured NTP search promo appears
  5. Open brave://local-state file and ensured that the metrics Brave.Search.Promo.NewTabPage response value is 0 as expected
    image
Case 2: Brave.Search.Promo.NewTabPage = 1_PASSED

1 Yes, No - "Did the participant in Group C click on the promo - Yes" and "did they switch their default engine to Brave Search? - No"

  1. Clean profile 1.44.x
  2. Launch the browser using the cmd line flag brave.exe --enable-features="BraveSearchNTP"
  3. Open brave://local-state file and ensure the metrics Brave.Search.Promo.NewTabPage is not shown
  4. Open a NTP and ensured NTP search promo appears
  5. Make search query via NTP promo
  6. Open brave://local-state file and ensured that the metrics Brave.Search.Promo.NewTabPage response value is 1 as expected
    image
Case 3: Brave.Search.Promo.NewTabPage = 2_PASSED

2 No, Yes - "Did the participant in Group C click on the promo - No" and "did they switch their default engine to Brave Search? - Yes"

  1. Clean profile 1.44.x
  2. Launch the browser using the cmd line flag brave.exe --enable-features="BraveSearchNTP"
  3. Open brave://local-state file and ensure the metrics Brave.Search.Promo.NewTabPage is not shown
  4. Open a NTP and ensured NTP search promo appears
  5. Change the default search engine to Brave via brave://settings/search
  6. Open brave://local-state file and ensured that the metrics Brave.Search.Promo.NewTabPage response value is 2 as expected
Example Example
image image
Case 4: Brave.Search.Promo.NewTabPage = 3_PASSED

3 Yes, Yes -"Did the participant in Group C click on the promo - Yes" and "did they switch their default engine to Brave Search? - Yes"

  1. Clean profile 1.44.x
  2. Launch the browser using the cmd line flag brave.exe --enable-features="BraveSearchNTP"
  3. Open brave://local-state file and ensure the metrics Brave.Search.Promo.NewTabPage is not shown
  4. Open a NTP and ensured NTP search promo appears
  5. Make search query via NTP promo
  6. Change the default search engine to Brave by either clicking on the prompt that appears on the search engine results page
  7. Open brave://local-state file and ensured that the metrics Brave.Search.Promo.NewTabPage response value is 3 as expected
Example Example Example
image image image

US Region

Brave.Search.Promo.NewTabPage

Case 1: Brave.Search.Promo.NewTabPage = 0_PASSED

0 No, No - "Did the participant in Group C click on the promo - No" and "did they switch their default engine to Brave Search? - No"

  1. Clean profile 1.44.x
  2. Launch the browser using the cmd line flag brave.exe --enable-features="BraveSearchNTP"
  3. Open brave://local-state file and ensure the metrics Brave.Search.Promo.NewTabPage is not shown
  4. Open a NTP and ensured NTP search promo isn't appear as Brave is the default SE for US
  5. Change the default SE to something other than Brave
  6. Open a NTP and ensured NTP search promo appears
  7. Open brave://local-state file and ensured that the metrics Brave.Search.Promo.NewTabPage response value is 0 as expected
Example Example Example
image image image
Case 2: Brave.Search.Promo.NewTabPage = 1_PASSED

1 Yes, No - "Did the participant in Group C click on the promo - Yes" and "did they switch their default engine to Brave Search? - No"

  1. Clean profile 1.44.x
  2. Launch the browser using the cmd line flag brave.exe --enable-features="BraveSearchNTP"
  3. Open brave://local-state file and ensure the metrics Brave.Search.Promo.NewTabPage is not shown
  4. Open a NTP and ensured NTP search promo isn't appear as Brave is the default SE for US
  5. Change the default SE to something other than Brave
  6. Open a NTP and ensured NTP search promo appears
  7. Make search query via NTP promo
  8. Open brave://local-state file and ensured that the metrics Brave.Search.Promo.NewTabPage response value is 1 as expected
Example Example Example
image image image
Case 3: Brave.Search.Promo.NewTabPage = 2_PASSED

2 No, Yes - "Did the participant in Group C click on the promo - No" and "did they switch their default engine to Brave Search? - Yes"

  1. Clean profile 1.44.x
  2. Launch the browser using the cmd line flag brave.exe --enable-features="BraveSearchNTP"
  3. Open brave://local-state file and ensure the metrics Brave.Search.Promo.NewTabPage is not shown
  4. Open a NTP and ensured NTP search promo isn't appear as Brave is the default SE for US
  5. Change the default SE to something other than Brave
  6. Open a NTP and ensured NTP search promo appears
  7. Change the default search engine to Brave via brave://settings/search
  8. Open brave://local-state file and ensured that the metrics Brave.Search.Promo.NewTabPage response value is 2 as expected
Example Example
image image
Case 4: Brave.Search.Promo.NewTabPage = 3_PASSED

3 Yes, Yes - "Did the participant in Group C click on the promo - Yes" and "did they switch their default engine to Brave Search? - Yes"

  1. Clean profile 1.44.x
  2. Launch the browser using the cmd line flag brave.exe --enable-features="BraveSearchNTP"
  3. Open brave://local-state file and ensure the metrics Brave.Search.Promo.NewTabPage is not shown
  4. Open a NTP and ensured NTP search promo isn't appear as Brave is the default SE for US
  5. Change the default SE to something other than Brave
  6. Open a NTP and ensured NTP search promo appears
  7. Make search query via NTP promo
  8. Change the default search engine to Brave by by clicking on the prompt that appears on the search engine results page
  9. Open brave://local-state file and ensured that the metrics Brave.Search.Promo.NewTabPage response value is 3 as expected
Example Example Example Example
image image image image

Regression on #23191

US Region

1. Brave.Search.Promo.Button

Case 1: Brave.Search.Promo.Button=0_PASSED

0. No, No - "No, I have not clicked on the promo" and "No, I have not changed my default engine to Brave"

  1. Clean profile 1.42.x , launch brave using brave.exe --enable-logging=stderr --enable-features="BraveSearchOmniboxButton
  2. Open brave://local-state file
  3. Search for the metric Brave.Search.Promo.Button
  4. Ensured metric is not shown in the brave://local-state file
  5. Navigate to brave://settings/search
  6. Change the search engine to something other than Brave
  7. Enter some random text in the Omnibox and ensured the button promo appears in the suggestion
  8. Reload the brave://local-state file
  9. Ensured that the metrics Brave.Search.Promo.Button is shown in the local state file
  10. Ensured that the metrics response value is 0 as expected
Case 2: Brave.Search.Promo.Button=1_PASSED

1. Yes, No - "Yes, I have clicked on the promo" and "No, I have not changed my default engine to Brave"

  1. Follow the steps 1 to 7 from case 1
  2. Click on the promo
  3. Reload the brave://local-state file
  4. Confirmed that the metric Brave.Search.Promo.Banner response value is 1 as expected
Case 3: Brave.Search.Promo.Button=2_PASSED

2. No, Yes - "No, I have not clicked on the promo" and "Yes, I have changed my default engine to Brave"

Case 1: Change the default SE to Brave via brave://settings/search - PASSED

  1. Follow the steps 1 to 7 from case 1
  2. Do not click on the promo
  3. Navigate to brave://settings/search
  4. Change the search engine to Brave
  5. Confirmed that the metric Brave.Search.Promo.Button response value is 2 as expected

Case 2: Change the default SE to Brave by clicking on the prompt that appears on the search engine results page - PASSED

  1. Follow the steps 1 to 7 from case 1
  2. Open the link https://search.brave.com/search?q=test&action=makeDefault
  3. Change the default SE to brave by clicking on the prompt that appears on the search engine results page
  4. Reload the brave://local-state page
  5. Confirmed that the metric Brave.Search.Promo.Button response value is 2 as expected
Case 4: Brave.Search.Promo.Button=3_PASSED

3. Yes, Yes - "Yes, I have clicked on the promo" and "Yes, I have changed my default engine to Brave"

Case 1: change the default SE via search engine results page - PASSED

  1. Follow the steps 1 to 7 from case 1
  2. Click on the promo
  3. Change the default SE to brave by clicking on the prompt that appears on the search engine results page
  4. Reload the brave://local-state page
  5. Confirmed that the metric Brave.Search.Promo.Button response value is 3 as expected

Case 2: change the default SE via search engine results page - PASSED

  1. Follow the steps 1 to 7 from case 1
  2. Click on the promo
  3. Change the default SE to brave via brave://settings/search
  4. Reload the brave://local-state page
  5. Confirmed that the metric Brave.Search.Promo.Button response value is 3 as expected

2. Brave.Search.Promo.Banner

Case 1: Brave.Search.Promo.Banner=0_PASSED

0. No, No - "No, I have not clicked on the promo" and "No, I have not changed my default engine to Brave"

  1. Clean profile 1.42.x , launch brave using brave.exe --enable-logging=stderr --enable-features="BraveSearchOmniboxBanner"
  2. Open brave://local-state file
  3. Search for the metric Brave.Search.Promo.Banner
  4. Ensured metric is not shown in the brave://local-state file
  5. Navigate to brave://settings/search
  6. Change the search engine to something other than Brave
  7. Enter some random text in the Omnibox and ensured the button promo appears in the suggestions
  8. Reload the brave://local-state file
  9. Ensured that the metrics Brave.Search.Promo.Button is shown in the local state file
  10. Ensured that the metrics response value is 0 as expected
    image
Case 2: Brave.Search.Promo.Banner=1_PASSED

1. Yes, No - "Yes, I have clicked on the promo" and "No, I have not changed my default engine to Brave"

  1. Follow the steps 1 to 7 from case 1
  2. Click on the promo
  3. Reload the brave://local-state file
  4. Confirmed that the metric Brave.Search.Promo.Banner response value is 1 as expected
Case 3: Brave.Search.Promo.Banner=2_PASSED

2. No, Yes - "No, I have not clicked on the promo" and "Yes, I have changed my default engine to Brave"

Case 1: Change the default SE to Brave via brave://settings/search - PASSED

  1. Follow the steps 1 to 7 from case 1
  2. Do not click on the promo
  3. Navigate to brave://settings/search
  4. Change the search engine to Brave
  5. Confirmed that the metric Brave.Search.Promo.Banner response value is 2 as expected

Case 2: Change the default SE to Brave by clicking on the prompt that appears on the search engine results page - PASSED

  1. Follow the steps 1 to 7 from case 1
  2. Open the link https://search.brave.com/search?q=test&action=makeDefault
  3. Change the default SE to brave by clicking on the prompt that appears on the search engine results page
  4. Reload the brave://local-state page
  5. Confirmed that the metric Brave.Search.Promo.Banner response value is 2 as expected
Case 4: Brave.Search.Promo.Banner=3_PASSED

3. Yes, Yes - "Yes, I have clicked on the promo" and "Yes, I have changed my default engine to Brave"

Case 1: change the default SE via search engine results page - PASSED

  1. Follow the steps 1 to 7 from case 1
  2. Click on the promo
  3. Change the default SE to brave by clicking on the prompt that appears on the search engine results page
  4. Reload the brave://local-state page
  5. Confirmed that the metric Brave.Search.Promo.Banner response value is 3 as expected

Case 2: change the default SE via search engine results page - PASSED

  1. Follow the steps 1 to 7 from case 1
  2. Click on the promo
  3. Change the default SE to brave via brave://settings/search
  4. Reload the brave://local-state page
  5. Confirmed that the metric Brave.Search.Promo.Banner response value is 3 as expected

@GeetaSarvadnya GeetaSarvadnya added QA Pass-Win64 and removed QA/In-Progress Indicates that QA is currently in progress for that particular issue labels Sep 7, 2022
@kjozwiak
Copy link
Member

Going to remove the above as we've never implemented the Brave search promo via NTP on Android. Currently we only have search promo implemented in the omnibox as per #24902. CCing @timchilds @anthonypkeane @bsclifton please correct me if I'm wrong.

Thanks for looking at this @GeetaSarvadnya, I should have read it closer before pinging you to take a look.

@kjozwiak kjozwiak removed the OS/Android Fixes related to Android browser functionality label Sep 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@DJAndries DJAndries

Labels
feature/search features/P3A OS/Desktop QA Pass-Win64 QA/Yes release-notes/exclude
Projects
None yet
Milestone
1.44.x - Release
Development

Successfully merging a pull request may close this issue.

Add NTP search promo P3A question brave/brave-core
4 participants
@kjozwiak @DJAndries @GeetaSarvadnya @brave-builds