New funnels - Analytics/backend bugs

[paolodamico]

Reporting here some bugs I detected while preparing/running usability tests in the new funnel experience. These bugs seem like backend issues, but I'm not entirely sure. Please excuse the lack of recordings here, files too large for GitHub.

This was tested in the latest cloud version but on a separate organization from PostHog. Let me know if you need an invite.

Recording 1

https://posthogusers.slack.com/files/U018R5C5UQ3/F0271Q3M3P1/2021-07-07_05.44.21.gif

• No users are showing up either for completed or dropped modals in the last step.
• Last two steps are: a) rated app with filter app_rating = 2 & b) rated app with filter app_rating = 4. What we have in the data is six users who fired rated app once either with 2 or 4. The second to last step should exclude all users from the last step, because users did one or the other, yet we see users in the last step.
• Dropped off modal says "Persons who completed" not persons who dropped. CC @samwinslow

Recording 2

https://posthogusers.slack.com/files/U018R5C5UQ3/F0279PEUVE2/2021-07-07_05.46.13.gif

• 113 users dropped off, yet the modal says showing 99 of 100. Same for users who completed the step.
• Also, not sure why we load 99 instead of just 100, but not a big deal. CC @liyiy

Recording 3

The graph can be found here, remember to use Hogflix org (ask for invite if needed).

• Steps 1 & 2 are exactly identical, yet there's a drop on step 2.

[paolodamico]

CC @neilkakkar @Twixes @EDsCODE

[Twixes]

Looks like some query result discrepancies between funnels and funnel persons indeed.

As for 1, the bit about prop exclusiveness is interesting. Do you have a link to the insight?

3 though actually seems correct. If two steps one after another are identical, it doesn't serve any purpose to show the second one as 100% – there is no value in that. What PostHog does instead, is that out of the set of people who performed the step once (making up step 1), for step 2 it counts people who performed that step for a second time in the funnel window. This way it's possible to know e.g. how many users land on a page and then exit, never to be seen again.

[paolodamico]

Re 1, here's the sample graph [invited you to Hogflix org].

what's even weirder is if I remove the filter on the app_rating it shows zero users converted 😳

[EDsCODE]

I think with #2 the issue is that some person ids that we try to serialize may have been deleted so the final return is somewhat less than 100 and if it's less than 100, the pagination doesn't work properly. Pending investigation

[neilkakkar]

IIRC, Recording 2 is a UI thingy? Since if you scroll down to the bottom, and click load more, it loads all the users? I think I wrote a specific test for this, so the query results should be ok. (but let me know if that's not the case).

For recording 3: this is indeed expected, since now we handle duplicate events. What this funnel means is: 2 Pageviews in a row. And some people only did 1, so they drop off. (oh, what Michael said)

For Recording 1: Something does look wrong with Persons.

Regarding prop exclusiveness, that's interesting! Some context: We had a small discussion a while ago to discuss semantics on what does it mean for two events to be equal, and basically ended up with: they're equal if every property is equal, so in this case they are unique events. This raises a weird case where if a later step in the funnel is a superset of an existing step, things get a bit borked, since we're treating them as unique, but they aren't. But, reasoned that there should be no reason to create a funnel like that.

To debug (1), can you please confirm from the data that indeed no person who rated 4 stars also rated 2 stars at a later time?
Edit: I have access, checking myself. ... How do I get the "new" view?

[neilkakkar]

Heh, for the example you shared, I get:

Can I get a link to the exact same funnel you mentioned in the first gif? Curious to see if those users did multiple ratings or not.

[paolodamico]

Hmmm can't seem to reproduce anymore the app_rating thing, sorry should've included the original funnel here too. Will keep an eye out for that, but we can assume it's fine for now.

[paolodamico]

I think some issues here are still happening ? @neilkakkar @EDsCODE

[neilkakkar]

I think the only remaining discrepancy is between Persons in funnel & Persons in modal?

This is because of our query not discarding deleted persons? The question here is: do we want to remove people who've been deleted, from the funnel conversion analysis (so that their contribution doesn't show in the funnel total), or keep those there and explain it on the persons modal?

Either seems reasonable to me! I guess depends on: When are users deleted?

[paolodamico]

My understanding is that we only delete persons when a user specifically goes into a person page and decides to delete it from there. If this is correct, it makes sense that a user won't want to consider these persons at all anywhere, so we should probably remove them altogether.

[neilkakkar]

Ah, just looked into this again, seems like I was mistaken: We do remove all persons who were deleted (in some cases). So the discrepancy is still a mystery O.o

[neilkakkar]

An example of discrepancy: https://app.posthog.com/insights?insight=FUNNELS&properties=%5B%5D&filter_test_accounts=true&events=%5B%7B%22id%22%3A%22user%20signed%20up%22%2C%22name%22%3A%22user%20signed%20up%22%2C%22type%22%3A%22events%22%2C%22order%22%3A0%7D%5D&actions=%5B%7B%22id%22%3A%224912%22%2C%22name%22%3A%22Discovered%20Learning%22%2C%22type%22%3A%22actions%22%2C%22order%22%3A1%7D%5D&funnel_viz_type=steps&display=FunnelViz&interval=day&new_entity=%5B%5D

29 completed step, but only 25 people exist

[EDsCODE]

discrepancy related to this most likely because the persons in clickhouse and the persons in postgres are out of sync

[Twixes]

I think the issues here have been explained (3.), solved (1. seems to work correctly now), or are being taken care of elsewhere (2.), so I will close this, though as always feel free to chime in.