A Bug With Soft Deleted related entities

[slarkerino]

I have a class

public class Foo : FullAuditedEntity<Guid>{
     public Guid BarId {get;set;}
     public Bar Bar {get;set;}
}

public class Bar: FullAuditedEntity<Guid>{
     public Foo Foo{get;set;}
}

and in efcore extension I have
public static IQueryable IncludeDetails(this IQueryable queryable, bool include = true)
{
if (!include)
{
return queryable;
}

        return queryable
            // .Include(x => x.bar) 
            ;
    }

When I soft delete bar and i query foo without include details, the foo returned with no problem, but when I queryed with include details, all foo related to the soft deleted bar is not retrieved.

How can I retrieve the foos with soft deleted bar even the bar is shown as null.

[davidzwa]

https://docs.microsoft.com/en-us/ef/core/querying/filters#accessing-entity-with-query-filter-using-required-navigation

Please read and try to understand why this happens in the EF Core docs linked above. I had to choose an option to fix it.
So if you now understand it: the query filter is applied once you use IncludeDetails on a ForeignKey with SoftDelete. Therefore you need to make sure the Query filter is correctly applied for your Bar entity:

Options

• Ignore and leave as is (inconsistency possible)
• Apply custom QueryFilter with the AuditedEntity IsDeleted combined with the ForeignKey IsDeleted (&& operand); this requires you to Include the foreignkey otherwise logically the Bar entity cannot filter on something it didnt include
• Disable the query filter for a query. IgnoreQueryFilters could help with that

Personally I chose to propagate the query filter as follows, so I have to make sure to include the foreign key for it to work (this is prone to a NullReferenceException):

        public static void OnModelCreating(EntityTypeBuilder<EventAttendee> entity)
        {
            entity.HasQueryFilter(ae => ae.AEvent.IsDeleted == false);
        }

The version where we basically return to the inconsistent state we were to begin with:

        public static void OnModelCreating(EntityTypeBuilder<EventAttendee> entity)
        {
            entity.HasQueryFilter(ae => ae.Event == null || ae.AEvent.IsDeleted == false);
        }

[slarkerino]

@davidzwa In my case I would like to have evenAttendee to still be searchable is the attendee itself is not deleted. Therefore, my workaround right now is to add _softDeleteFilter to whatever method that needs to include details.

            using (_softDeleteFilter.Disable())
            {
                return base.GetAsync(id, includeDetails, cancellationToken);
            }

and in efcore extension

        public static IQueryable<Bar> IncludeDetails(this IQueryable<Bar> queryable, bool include = true)
        {
            queryable = queryable.Where(f => !f.IsDeleted);

            if (!include)
            {
                return queryable;
            }

            return queryable
                .Include(x => x.xxx)
                ;
        }

I am not sure if there are better solutions to this

[slarkerino]

I am wondering if there is a way to only applying filter to the entity that i am querying. Theredfore querying eventattendee would not be affected by a related soft deleted event.

[davidzwa]

I see, I think the fluent IgnoreQueryFilters function works perfectly in that case, since the docs specify it is for the related entity! We should definitely figure this out, as I'm eager to learn more about this as well.

blogs = db.Blogs
    .Include(b => b.Posts)
    .IgnoreQueryFilters()
    .ToList();

https://docs.microsoft.com/en-us/ef/core/querying/filters#disabling-filters

Of course this does mean you have to be cautious to not do this all the time. Can you test this in a scenario where your root Entity (Event in my example) is not deleted, but your child is (f.e. EventAttendee)? If you dont have the time, I will test when I have the time in a couple of days.

Let me know if this works for you 👍🏼

[slarkerino]

I see, I think the fluent IgnoreQueryFilters function works perfectly in that case!

blogs = db.Blogs
    .Include(b => b.Posts)
    .IgnoreQueryFilters()
    .ToList();

https://docs.microsoft.com/en-us/ef/core/querying/filters#disabling-filters

Let me know if this works for you 👍🏼

That works but I will have to add where clause for tenant and isdeleted for the entity itself XD.

[davidzwa]

Hmm I guess there is no other way! The lesson we need to learn is that SoftDelete comes with an administrative price.

Personally I only have query filters on my Root entity (Event). Maybe that is another good way to solve it? Does your child entity have to be softdeleted?
Maybe clever design of the database is an option.

Can we somehow construct a compiler warning? Something like a Roslyn analyzer? "Yo DEV you should definitely fix this query filter explicitly". Or maybe a required non-defaulted boolean for IncludeDetails?

[davidzwa]

@slarkerino please close the issue, if it is resolved for you.

[olicooper]

@slarkerino If I understand your problem correctly, it would be good is to be able to have a IgnoreQueryFilters attribute?

public class Foo : FullAuditedEntity<Guid>{
     [IgnoreQueryFilters]
     public Bar Bar {get;set;}
}

I will take a look at the implementation of ABP to see if this can be added.

Alternatively I could see if an IRepository extension method could be created e.g.

var blogsWithDeletedPosts = await BlogRepository
    .IgnoreQueryFilters(b => b.Posts)
    .ToListAsync();

I personally prefer the second option.

[davidzwa]

I think on-the-fly/runtime is best as people need to understand the problem! Attributes don't force the developer to stay on top of it.

I'd also be willing to write some docs for it in the ABP documentation as it is not easy to wrap your head around. What do you think @olicooper

[olicooper]

@davidzwa I have started working on a test to see if it will be possible. At the moment it looks complicated (I am having to hook in to EF Core's QueryTranslationPreprocessor to modify the query before it is executed) so it might not be possible, but I will try anyway 😄

[davidzwa]

Share the PR/fork and I'll try to help you out!

[olicooper]

Hi @davidzwa, here you go... https://github.com/olicooper/abp-queryfilter-test
It doesn't actually filter anything right now but it can interpret the 'IgnoreAbpQueryFilters' during query execution so it is a start!

[olicooper]

Hey @davidzwa I have created an alternative version on the apply-filters-manually branch that is starting to take shape, but EntityFramework is firing a second query for the related entities which I haven't been able to capture and modify, so the results are not correct.. but some good progress has been made! If you are able to help improve it then that would be awesome as I am finding this a little challenging hahah :)

Ultimately, this version scraps the use of EF's global query filters as I have been unable to reliably intercept it. I opt to apply the filters manually (similar to ABP's approach for MonogDB). This is going to solve various issues if I can pull it off!

I came up with another way to enable/disable filters for related entities. The following example should fetch a list of Posts without considering the ISoftDelete filter for the Blog entity:

class Post
{
  public Blog Blog { get; set; }
}

using (DataFilter.Disable<ISoftDelete<Blogs.Blog>>())
{
    var query = await PostRepository.GetQueryableAsync();

    var postsWithDeletedBlogs = await AsyncExecuter.ToListAsync(query);
}

@slarkerino What do you think? Would this approach be better for you? I still can't guarantee I can get this feature to fully work though.

[slarkerino]

@olicooper Hey man I think that would help a lot and clean the code when we need to disable filters for a certain related entity!