Allow migrations to be ran automatically.

[PhyxionNL]

So I'm still running 6.x branch and I'm wondering whether these upgrade instructions (https://github.com/exceptionless/Exceptionless/wiki/Upgrading) are automatically applied on upgrade. If not, it would be useful that this could be added. There are quite a few containers upgrading data automatically.

[ejsmith]

@PhyxionNL yeah, I agree that ideally we would upgrade on the fly. Problem is that some upgrades require down time. So we need to figure out a way to handle that. Definitely something I want to do though.

[PhyxionNL]

Alright, sounds good.

I've just tried to migrate the data manually but it doesn't seem to have run correctly. None of the fixed status is migrated and all old events show this in Extended Data:
Is_deleted
False
Is_fixed
True
Is_hidden
False

Any ideas on how to address this?

[PhyxionNL]

The migration does do something, Completed migration Exceptionless.Core.Migrations.SetStackStatus., I can also see Updated records. But the data isn't correct. The fixed-x.y.z label is there, but the status of all stacks is Open, rather than Fixed (as it should be). Am I missing something here or is the migration job incomplete? All old data also has the above fields now shown in Extended Data. I can live with that, but I do would like to find a fix for all the broken stack statuses.

[niemyjski]

What do the migration document look like in the migration elastic index? That should say if it completed or there were errors. Yeah the fixed label shouldn't be there unless it's regressed or fixed, that also shouldn't shown in extended data. I haven't noticed that in any of our events.

[PhyxionNL]

The migration code does report SetStackStatus has run (with all successful Updates).
However, I've been checking v6 data in Elastic but I do not see is_fixed there on stack level. is_fixed is only on event level. All there is at stack level (_source) is

"disable_notifications": false, "is_hidden": false, "is_regressed": false, "occurrences_are_critical": false,

Furthermore, if the is_hidden etc on event level is no longer used, shouldn't these be removed? I cannot find that in the Migrations, but maybe I'm looking in the wrong place, I'm not that familiar with the code.

[niemyjski]

Something seems off, can you try removing those migration documents and run again, Maybe they didn't run if there was no existing migration docs @ejsmith? No we didn't want to update events in any migration (hundreds of millions of them for no benefit), they will fall off over time.

[PhyxionNL]

For items that are fixed:

"fixed_in_version": "x.y.z", "date_fixed": "x-y-xxx",

[PhyxionNL]

Didn't mean to close it ^^

[niemyjski]

Was is_fixed set on that stack? Does anything in this logic look off to you based on the data you have for the stack: https://github.com/exceptionless/Exceptionless/blob/master/src/Exceptionless.Core/Migrations/002_SetStackStatus.cs#L34 Would you mind posting the whole stack json minus any id's or sensitive data.

[PhyxionNL]

No, is_fixed isn't set on any stack as far as I could find (I'll do another sweep of the data tomorrow). I have modified the SetStackStatus script to this:
const string script = "if (ctx._source.is_regressed == true) ctx._source.status = 'regressed'; else if (ctx._source.is_hidden == true) ctx._source.status = 'ignored'; else if (ctx._source.disable_notifications == true) ctx._source.status = 'ignored'; else if (ctx._source.is_fixed == true) ctx._source.status = 'fixed'; else if (ctx._source.containsKey('date_fixed')) ctx._source.status = 'fixed'; else ctx._source.status = 'open';";
The fixed status is now more reliably shown, but still not the same as v6, there are still more stacks. I will investigate some more as to the differences.

Furthermore, I'm receiving this message when viewing old data, which seems odd as I'm running locally.
Unable to view event occurrence due to plan limits. How can I increase/change this?

Why are ' Recent Occurrences' now sorted backwards? This seems counterintuitive to me, as 'Recent' kinda says it all. I also miss the Exception Dashboard a bit 😊

[PhyxionNL]

Some of these exception stacks are regressed + hidden, I think that's the main cause for the remainders, I'll check again tomrrow.

[PhyxionNL]

I have migrated everything now. The majority of the remaining issues indeed were because hidden was not prefered over regressed. This default migration behavior doesn't seem very logical to me as if you deliberately hide them then you most likely don't want to see them anymore. The resulting script for SetStackStatus is now:

"if (ctx._source.is_hidden == true) ctx._source.status = 'ignored'; else if (ctx._source.is_regressed == true) ctx._source.status = 'regressed'; else if (ctx._source.disable_notifications == true) ctx._source.status = 'ignored'; else if (ctx._source.is_fixed == true) ctx._source.status = 'fixed'; else if (ctx._source.containsKey('date_fixed')) ctx._source.status = 'fixed'; else ctx._source.status = 'open';";.

May it be of use to some 😊 Or do you want this in a PR?

The plan issue is only for events at "retention days" in the last couple of hours, so it seems to cut this off a bit too soon. Probably an existing bug but never noticed this before. Does that check even make sense if the organisation is on the unlimited plan? As long as the events exist it should show them.

The sorting itself is fixed in 7.0.3 (maybe earlier), I had it first on 7.0.0 for migration. Glad to see the stats reappear on this version as well.

As a bonus, if you want to clear out the is... fields on events, run this on /prod-events-v1-2020.10.*/_update_by_query:
{ "script": { "inline": "ctx._source.remove('is_fixed');ctx._source.remove('is_hidden');ctx._source.remove('is_deleted');ctx._source.remove('updated_utc')" } }

Depending on the number of events it takes some time... Repeat for each month of data.

[niemyjski]

Yeah, if you want to add a pr that adds this else if (ctx._source.containsKey('date_fixed')) before the last else we'd merge that in. We didn't want to update events in a migration as it's super expensive when you have hundreds of millions of events. Nice find! We'd recommend updating fast on the 7.x series if possible we are doing many small bug fixes and it seems really stable now.

[niemyjski]

Thanks for reporting this issue to us. Please let us know if we are missing anything.

[PhyxionNL]

Maybe this shouldn't be closed still? An automatic update would be useful.

[niemyjski]

For migrations

[niemyjski]

@PhyxionNL I'm going to close this, as it really should be up to the user when to run data migrations. We've made it a one liner to run in 7.1 from a docker container now. For us, we would kick off a k8's pod todo the migration, but an end user might want to run it in process but that could take a very long time. We try to keep our web processes light, so having a job do it is a no brainer. Any feedback would be greatly appreciated on this and we can discus it more.