Postprocessors

[icsy7867]

Added capability to use similarity match as well as keyword include/exclude.

            if settings().llm.keywords_include or settings().llm.keywords_exclude:
                return ContextChatEngine.from_defaults(
                    system_prompt=system_prompt,
                    retriever=vector_index_retriever,
                    llm=self.llm_component.llm,  # Takes no effect at the moment
                    node_postprocessors=[
                        MetadataReplacementPostProcessor(target_metadata_key="window"),
                        SimilarityPostprocessor(similarity_cutoff=settings().llm.similarity_value),
                        KeywordNodePostprocessor(required_keywords=settings().llm.keywords_include, exclude_keywords=settings().llm.keywords_exclude),
                    ],
                )
            else:
                return ContextChatEngine.from_defaults(
                    system_prompt=system_prompt,
                    retriever=vector_index_retriever,
                    llm=self.llm_component.llm,  # Takes no effect at the moment
                    node_postprocessors=[
                        MetadataReplacementPostProcessor(target_metadata_key="window"),
                        SimilarityPostprocessor(similarity_cutoff=settings().llm.similarity_value),
                    ],
                )

Looking at:
SimilarityPostprocessor(similarity_cutoff=settings().llm.similarity_value)

The default value is none and you can look at how llama_index handles this:
https://github.com/run-llama/llama_index/blob/d33b789de9635dcf19e02050c6a0487fcfeb30ad/llama-index-core/llama_index/core/postprocessor/node.py#L64

This class automatically ignores it if set to None:

sim_cutoff_exists = self.similarity_cutoff is not None

[icsy7867]

Removed the include/exclude word filter per @imartinez request.

[imartinez]

Other than the comment I've added, it is good to go!

[icsy7867]

Other than the comment I've added, it is good to go!

Took me a second to follow the class structure, but I think I got it figured out. I added a commit to my branch and waiting for the post functions to run.

[icsy7867]

I tried to add similarity_top_k from the vector stores file. I added this to the settings.py and the similarity_value worked as intended (I guess the python script doesn't want to load the yaml settings file with an empty value?)

But I couldn't get similarity_top_k to access the settings file easily. I could add "self" to the class method and read the settings variable set in the init part of the class without changes in multiple places.

I must be going about this the wrong way