Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Developer demo of alternate wiki store implementation #7329

Draft
wants to merge 76 commits into
base: master
Choose a base branch
from
Draft
Changes from 2 commits
Commits
Show all changes
76 commits
Select commit Hold shift + click to select a range
fdec12f
Initial commit
Jermolene Mar 4, 2023
7fdd8a5
Merge branch 'master' into demo-alternate-store
Jermolene May 10, 2023
6233710
Merge branch 'master' into demo-alternate-store
Jermolene May 10, 2023
b6bc197
Remove unneeded override
Jermolene May 10, 2023
3561318
Merge branch 'master' into demo-alternate-store
Jermolene Jun 22, 2023
1d0b928
Incorporate @joshuafontany's plain JS wiki implementation
Jermolene Jun 22, 2023
544e079
Proof of concept of instantiating sqlite3 without needing external de…
Jermolene Jun 22, 2023
0546a14
Restructure the sqlite3 store as a separate plugin
Jermolene Jun 22, 2023
146a22b
Make rawmarkup code dynamically load dependencies
Jermolene Jun 22, 2023
449e227
Test code to exercise the database
Jermolene Jun 23, 2023
6f24f33
Include sqlite3 in the empty edition
Jermolene Jun 23, 2023
c43bc8f
Start prototyping some tiddler operations in sql
Jermolene Jun 23, 2023
831fb39
Tiddler sql function tests should show custom field
Jermolene Jun 23, 2023
8690936
Cleanup and clarify
Jermolene Jun 23, 2023
9427cf7
Additional method to retrieve all titles
Jermolene Jun 23, 2023
9ac21f1
Simplify the plain JS store implementation
Jermolene Jun 25, 2023
7e60726
Wire the sql functions into the wiki object
Jermolene Jun 25, 2023
9cb8721
Use sql functions for processing shadow tiddlers
Jermolene Jun 27, 2023
ede5f1e
Prepare the save tiddler query
Jermolene Jun 28, 2023
8399538
Merge branch 'master' into demo-alternate-store
Jermolene Jun 28, 2023
2099c4f
Turn on performance instrumentation for testing
Jermolene Jun 28, 2023
12a19bb
Remove instrumentation
Jermolene Jun 28, 2023
87213f2
Refactpr sql-wiki-store into two files
Jermolene Jun 29, 2023
6ded5e6
Merge branch 'master' into demo-alternate-store
Jermolene Jun 29, 2023
b29af44
Fix typo
Jermolene Jun 29, 2023
1f4be3e
I experimented with custom collations to match JS sort order, but 5x …
Jermolene Jun 30, 2023
687b1df
Merge branch 'master' into demo-alternate-store
Jermolene Jun 30, 2023
71c02e5
Merge branch 'master' into demo-alternate-store
Jermolene Jul 1, 2023
2d229e2
Add logging utility
Jermolene Jul 6, 2023
9e190a4
Use a temporary database so that multiple wiki stores can coexist
Jermolene Jul 6, 2023
21ef2d7
Make the tests work in the browser
Jermolene Jul 6, 2023
f3bc32a
Merge branch 'master' into demo-alternate-store
Jermolene Jul 6, 2023
d2e21dd
Add a custom collator that matches JS ordering
Jermolene Jul 7, 2023
64ffa52
Write tiddlers with string fields
Jermolene Jul 7, 2023
b557dea
Update comment
Jermolene Jul 7, 2023
e3255a4
Merge branch 'master' into demo-alternate-store
Jermolene Jul 13, 2023
f48bddb
Merge branch 'master' into demo-alternate-store
Jermolene Jul 13, 2023
709669b
Merge branch 'master' into demo-alternate-store
Jermolene Jul 14, 2023
cc2cd20
Add tags tables and tag indexer and make custom collator be optional
Jermolene Jul 18, 2023
83e7d32
Merge branch 'master' into demo-alternate-store
Jermolene Jul 18, 2023
979a1f7
Introduce sql console
Jermolene Jul 18, 2023
88c8c2c
SQL console - process query on enter key
Jermolene Jul 19, 2023
b4fe896
Styling for SQL console
Jermolene Jul 19, 2023
09b0e28
Style update for sql console
Jermolene Jul 19, 2023
2cd2a05
Fix tag saving
Jermolene Jul 19, 2023
2d3027f
Sql console styling
Jermolene Jul 19, 2023
7fd2dd5
Wire up the tag indexer properly
Jermolene Jul 19, 2023
b9245da
Sort tag lookups according to TW semantics
Jermolene Jul 20, 2023
39d0451
Experiment with optimising specific filters with direct SQL equivalents
Jermolene Jul 21, 2023
7eeaa20
Merge branch 'master' into demo-alternate-store
Jermolene Jul 22, 2023
bb41ae0
Merge branch 'master' into demo-alternate-store
Jermolene Jul 28, 2023
e6309e9
Fix tag collation syntax
Jermolene Jul 29, 2023
66cba18
Merge branch 'master' into demo-alternate-store
Jermolene Aug 22, 2023
25138ec
Merge branch 'master' into demo-alternate-store
Jermolene Sep 19, 2023
1754be2
Merge branch 'master' into demo-alternate-store
Jermolene Oct 15, 2023
c52014c
Simplify the SQL schema
Jermolene Oct 21, 2023
fd3d8ae
Merge branch 'master' into demo-alternate-store
Jermolene Oct 21, 2023
863066d
Merge branch 'master' into demo-alternate-store
Jermolene Oct 23, 2023
d4dec0c
Use the empty string as special plugin name for ordinary tiddlers
Jermolene Oct 24, 2023
f49b9fa
Move tags into their own tables
Jermolene Oct 25, 2023
12c6cb3
Add indexes for columns used in joins
Jermolene Oct 25, 2023
dc94ed8
Refactor indexer implementation
Jermolene Oct 25, 2023
3a4f5b8
Cache allTitles and allShadowTitles
Jermolene Oct 25, 2023
e9d640b
Fix sqlAllTitles wrongly including shadow tiddlers
Jermolene Oct 25, 2023
01e1882
More indexes
Jermolene Oct 27, 2023
a58f119
Enable performance instrumentation
Jermolene Oct 27, 2023
e4af21a
Don't use LEFT JOIN unless we have to
Jermolene Oct 27, 2023
d982658
Remove all plugins to simplify benchmarking
Jermolene Oct 27, 2023
8c1f7a6
SQL Console: Add number of rows returned
Jermolene Oct 30, 2023
9493084
Merge branch 'master' into demo-alternate-store
Jermolene Nov 7, 2023
02f3065
Revert attempt at optimising filter execution
Jermolene Nov 7, 2023
d7f0c5c
Minor refactoring
Jermolene Nov 7, 2023
6997c61
Refactor filter compilation to allow SQL engine to optimise queries
Jermolene Nov 7, 2023
5d20e98
Refactor filter compilation into its own source file
Jermolene Nov 8, 2023
4897248
Fix indentation
Jermolene Nov 10, 2023
6983564
Merge branch 'master' into demo-alternate-store
Jermolene Dec 11, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 50 additions & 57 deletions core/modules/filters.js
Original file line number Diff line number Diff line change
Expand Up @@ -171,15 +171,15 @@ exports.parseFilter = function(filterString) {
}
if(match[3]) {
operation.suffixes = [];
$tw.utils.each(match[3].split(":"),function(subsuffix) {
$tw.utils.each(match[3].split(":"),function(subsuffix) {
operation.suffixes.push([]);
$tw.utils.each(subsuffix.split(","),function(entry) {
entry = $tw.utils.trim(entry);
if(entry) {
operation.suffixes[operation.suffixes.length -1].push(entry);
}
});
});
});
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the indent-level here is wrong

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @pmario fixed in 4897248

}
}
if(match[4]) { // Opening square bracket
Expand Down Expand Up @@ -225,73 +225,31 @@ source: an iterator function for the source tiddlers, called source(iterator), w
widget: an optional widget node for retrieving the current tiddler etc.
*/
exports.compileFilter = function(filterString) {
var self = this;
// Use cached filter function if already present
// Set up the filter function cache
if(!this.filterCache) {
this.filterCache = Object.create(null);
this.filterCacheCount = 0;
}
// Use the cached version of this filter function if it exists
if(this.filterCache[filterString] !== undefined) {
return this.filterCache[filterString];
}
// Attempt to optimise the filter into a single query
var operationFunctions = this.optimiseFilter && this.optimiseFilter(filterString);
// Otherwise compile the filter step by step
if(!operationFunctions) {
// Parse filter
var filterParseTree;
try {
filterParseTree = this.parseFilter(filterString);
} catch(e) {
// We do not cache this result, so it adjusts along with localization changes
return function(source,widget) {
return [$tw.language.getString("Error/Filter") + ": " + e];
};
}
// Compile the filter operators into functions
operationFunctions = this.compileFilterOperations(filterParseTree);
}
// Return a function that applies the operations to a source iterator of tiddler titles
var fnMeasured = $tw.perf.measure("filter: " + filterString,function filterFunction(source,widget) {
if(!source) {
source = self.each;
} else if(typeof source === "object") { // Array or hashmap
source = self.makeTiddlerIterator(source);
}
if(!widget) {
widget = $tw.rootWidget;
}
var results = new $tw.utils.LinkedList();
self.filterRecursionCount = (self.filterRecursionCount || 0) + 1;
if(self.filterRecursionCount < MAX_FILTER_DEPTH) {
$tw.utils.each(operationFunctions,function(operationFunction) {
operationFunction(results,source,widget);
});
} else {
results.push("/**-- Excessive filter recursion --**/");
}
self.filterRecursionCount = self.filterRecursionCount - 1;
return results.toArray();
});
if(this.filterCacheCount >= 2000) {
// To prevent memory leak, we maintain an upper limit for cache size.
// Reset if exceeded. This should give us 95% of the benefit
// that no cache limit would give us.
this.filterCache = Object.create(null);
this.filterCacheCount = 0;
// Parse the filter string
var filterParseTree;
try {
filterParseTree = this.parseFilter(filterString);
} catch(e) {
// We do not cache this result, so it adjusts along with localization changes
return function(source,widget) {
return [$tw.language.getString("Error/Filter") + ": " + e];
};
}
this.filterCache[filterString] = fnMeasured;
this.filterCacheCount++;
return fnMeasured;
};

exports.compileFilterOperations = function(filterParseTree) {
var self = this;
// Get the hashmap of filter operator functions
var filterOperators = this.getFilterOperators();
// Assemble array of functions, one for each operation
var operationFunctions = [];
// Step through the operations
var self = this;
$tw.utils.each(filterParseTree,function(operation) {
// Create a function for the chain of operators in the operation
var operationSubFunction = function(source,widget) {
Expand Down Expand Up @@ -379,7 +337,42 @@ exports.compileFilterOperations = function(filterParseTree) {
}
})());
});
return operationFunctions;
// Make the filter function
var fnFilter = function filterFunction(source,widget) {
if(!source) {
source = self.each;
} else if(typeof source === "object") { // Array or hashmap
source = self.makeTiddlerIterator(source);
}
if(!widget) {
widget = $tw.rootWidget;
}
var results = new $tw.utils.LinkedList();
self.filterRecursionCount = (self.filterRecursionCount || 0) + 1;
if(self.filterRecursionCount < MAX_FILTER_DEPTH) {
$tw.utils.each(operationFunctions,function(operationFunction) {
operationFunction(results,source,widget);
});
} else {
results.push("/**-- Excessive filter recursion --**/");
}
self.filterRecursionCount = self.filterRecursionCount - 1;
return results.toArray();
};
// Return a function that applies the operations to a source iterator of tiddler titles
var fnMeasured = $tw.perf.measure("filter: " + filterString,fnFilter);
// Cache the measured filter function
if(this.filterCacheCount >= 2000) {
// To prevent memory leak, we maintain an upper limit for cache size.
// Reset if exceeded. This should give us 95% of the benefit
// that no cache limit would give us.
this.filterCache = Object.create(null);
this.filterCacheCount = 0;
}
this.filterCache[filterString] = fnMeasured;
this.filterCacheCount++;
return fnMeasured;
};

})();

Loading