Understanding the non-technical user

[yetudada]

Description

We have a lot of assumptions about people that interact with the outputs of a Kedro project but do not necessarily build anything within the project. Things that we don't know:

• Level of technical expertise
• Goals or reasons why they are interacting with the Kedro project
• How they use outputs from a Kedro project

We have largely relied on second-hand information from technical users who interact with these non-technical users. It's time to put aside these assumptions and get some data from real people.

Context

This ticket is part of a larger initiative to increase the usage of Kedro-Viz by supporting this user base. This ticket has implications for #1279 and #1116.

Outcomes of this ticket

We don't currently have capacity to conduct this research but this ticket proposes designing the research study and completing recruiting for it. We must know what research outputs we need to move forward and who we will be talking to.

[stephkaiser]

Research Objective

The goal of this research is to understand who our non-technical kedro users are, their goals, needs, and pain points when interacting with the outputs of a Kedro project and using Kedro-Viz. We hope to also get a sense of their general product/user satisfaction towards Kedro & Kedro-Viz.

We want to speak to 5-8 non-technical users, both internally and externally, and across a variety of different roles and experience levels with using Kedro and Kedro-Viz.

What do we want to learn?

1. Who are our non-technical users?
2. What levels of technical expertise do they have?
3. Why do they use Kedro and Kedro-Viz? What are their overall main goals? What do they use it for?
4. How do they currently use Kedro-Viz? What are their tasks? What are they trying to accomplish?
5. Where are the current pain points in their overall experience?
6. Overall user satisfaction with Kedro-Viz

How will we know when we’ve learned it?

We will know when we are able to answer all of the questions in ‘What do we want to learn?’ with confidence and evidence.

What business goals will this research support?

The Kedro-Viz user base represents ~20% of all Kedro Framework users. Research understanding how to convert the other 80% of Framework users into Viz users has always yielded mediocre improvements to Viz. We want to grow the user base of Kedro-Viz so we're going to see if we can help non-technical users instead. We're mindful that these changes might even help the technical users.

What decisions will this research enable?

We would like to see areas that affect our roadmap for Viz. We should come out of this research with "bets" that we're going to take throughout the year.

Methods

We aim to capture qualitative feedback through 1-on-1 interviews with 5-8 users, either in-person or remotely on Zoom.

Research Outcomes

• 1. Define a set of non-technical user personas that the entire Kedro team can use as a resource.
• 2. Define what the current non-technical user journey/experience looks like.
• 3. Synthesised insights with potential opportunities and recommendations to prioritise for Kedro-Viz.

Timeline

Interviews conducted week of: 17-21 April
Start synthesis on: Tuesday 25th April
Share results with team by: Monday 15th May

Participants

What is a ‘non-technical’ user?

We define non-technical kedro user here as someone who is not as comfortable using the CLI (command-line interface) and they use Kedro-Viz to follow 'what's going on' and inspect the code.

Internal	External
Product Managers, BAs	Client stakeholders
BAs, CSTs (client service teams)	Client product leads or team leads
EMs, EDs, AEMs, AEDs	Non-technical client team members (eg. Head of Finance)
APs, Partners
Junior DS new to Kedro or Kedro-Viz

Who will we be speaking to?

• Wesley Leong
• Stijn De Bruyne
• Tobias Haefele
• Chin-Chin Lin
• Mark Huntington
• Javier del Villar (External)

Interview Guide (45 mins)

User details (10 mins)

• What is your current role at work?
• What software and/or applications do you use on a regular basis?
• How familiar are you with Kedro and Kedro-Viz? How often do you use Kedro/Kedro-Viz?
• How would you describe your level of technical expertise? Do you know how to code? Do you feel comfortable using a CLI?

User motivation (10 mins)

• Why do you use Kedro and Kedro-Viz? What are your overall main goals? What do you use it for?
• How do you currently use Kedro-Viz? What are your tasks? What are you trying to accomplish?
• Why did you choose Kedro?

User pain points (10 mins)

• What pain points are you currently experiencing when using Kedro-Viz?
• Where in your overall Kedro/Kedro-Viz experience can we improve?
• Anything else you need Kedro-Viz to do for you?

User recommendations & Wrap up (5 mins)

• How would you rate your overall satisfaction with Kedro-Viz? 1-10 (10 meaning fully satisfied, no suggested improvements)
• What haven’t we asked you today that you think would be valuable for us to know?
• Do you have any other suggestions or feedback for us?
• May I contact you if we have any other questions or for possible further research in the future?

[stephkaiser]

Slack Polly Survey (Internal & External channels)

I also sent a survey via the Polly app in Slack on both internal and external channels to gather more general information about how our users are using Kedro-Viz.

Poll message

Kedro-Viz: How do you use it?

The Kedro team wants to understand how our users are using Kedro-Viz and the outputs of a Kedro project. We’d also love to hear about the challenges you face and suggestions for improvement, your responses will be included and considered during Kedro-Viz development. If you prefer to share more feedback with a Kedro team member, please reach out as we are always up for a chat!

Start Poll

Survey Questions

#	Question	Answer type	Requirements	Options (if applicable)
1	(External only) What is your role at work?	Text input field	Mandatory	-
2	How familiar are you with python?	Scale	Mandatory	1 - I’m not familiar with python, 2 - Slightly familiar I know the basics, 3 - Somewhat familiar, 4 - Moderate I feel comfortable with python, 5 - I have extensive python knowledge
3	Have you used/or run Kedro-Viz yourself?	Yes/No	Mandatory	Yes, No, Only with my team, Other (please specify)
4	What did you use Kedro-Viz for?	Select all that apply, optional text input field	Mandatory	Debugging technical issues, Experiment or metrics tracking, Viewing updates on team progress, Sharing updates and communication with stakeholders, Training on Kedro, Other (please specify)
5	What are the challenges or struggles you face when using Kedro-Viz?	Text input field	Optional	-
6	In your opinion, how might we improve the Kedro-Viz experience for less-technical users/audiences?	Text input field	Optional	-
7	Would you like to participate in future user research for Kedro-Viz?	Multiple choice	Optional	Yes, No

Timeline

Both surveys sent on: Wed 26th April 2023, 3:00 PM
Both surveys closed on: Wed 3rd May 2023, 3:00 PM

[tynandebold]

Closing this as completed. Here's a link to @stephkaiser's amazing synthesis and findings.