Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for loading files, decoding them as JSONL, and formatting them using a Logback-like format string. #46

Merged
merged 90 commits into from
Jul 3, 2024
Merged
Show file tree
Hide file tree
Changes from 89 commits
Commits
Show all changes
90 commits
Select commit Hold shift + click to select a range
bf17d8c
Add log file management and jsonl decoder functionality.
junhaoliao Jun 14, 2024
e001006
Add docs to Decoder interface.
junhaoliao Jun 14, 2024
4a3ddc8
Add decode options to file loading process.
junhaoliao Jun 14, 2024
655c869
Revert changes unrelated to current PR.
junhaoliao Jun 15, 2024
3494383
Add JSON value and object types annotations.
junhaoliao Jun 15, 2024
d360689
Apply suggestions from code review
junhaoliao Jun 18, 2024
cef9af9
Allow user to specify file path in URL.
junhaoliao Jun 18, 2024
1723b17
Rename 'logLines' to 'beginLineNumToLogEventNum'.
junhaoliao Jun 18, 2024
2d2b800
Refactor field and variables related to log 'verbosity' to 'level' & …
junhaoliao Jun 18, 2024
551bf81
Apply suggestions from code review
junhaoliao Jun 18, 2024
1cedfbf
Apply suggestions from code review
junhaoliao Jun 18, 2024
7339ad5
Update last log event number logic in StateContextProvider.
junhaoliao Jun 18, 2024
50690be
Add error handling.
junhaoliao Jun 18, 2024
fff469d
Refactor decoder object in src/services and src/typings.
junhaoliao Jun 18, 2024
96e26e9
Refactor loadPage docs and variable names in LogFileManager.
junhaoliao Jun 18, 2024
66c3293
Typify cursor handling.
junhaoliao Jun 18, 2024
384836f
Rename global constant name from 'StateDefaultValue' to 'STATE_DEFAULT'.
junhaoliao Jun 18, 2024
a7d8f0e
Apply suggestions from code review
junhaoliao Jun 18, 2024
b316f0c
Apply suggestions from code review
junhaoliao Jun 18, 2024
a7f7660
Apply suggestions from code review
junhaoliao Jun 18, 2024
582f0de
Apply suggestions from code review
junhaoliao Jun 18, 2024
be100d8
Apply suggestions from code review
junhaoliao Jun 18, 2024
1545e05
Apply suggestions from code review
junhaoliao Jun 18, 2024
7c56689
Apply suggestions from code review
junhaoliao Jun 18, 2024
921bc22
Refactor JsonlDecoder and improve comments.
junhaoliao Jun 18, 2024
ea2b3ef
Refactor decoding functions and error handling.
junhaoliao Jun 18, 2024
07eb2bb
Update URL parsing functionality and improve error reporting.
junhaoliao Jun 18, 2024
8f783e3
Apply suggestions from code review
junhaoliao Jun 18, 2024
02acedc
Refactor "textPattern" to "formatString" in JsonlDecoder.
junhaoliao Jun 18, 2024
518c443
Merge remote-tracking branch 'junhao/new-log-viewer' into new-log-viewer
junhaoliao Jun 18, 2024
c588461
Fix typo.
junhaoliao Jun 18, 2024
988b88d
Reorganize decoder and log file manager classes.
junhaoliao Jun 18, 2024
c2c42b8
Apply suggestions from code review
junhaoliao Jun 21, 2024
92e107e
Apply suggestions from code review
junhaoliao Jun 21, 2024
191e759
Refactor variable name for clarity in StateContextProvider.
junhaoliao Jun 21, 2024
cda01cb
Apply suggestions from code review
junhaoliao Jun 21, 2024
99700ab
Refactor JsonlDecoder and create new LogbackFormatter class.
junhaoliao Jun 21, 2024
b609103
Update decoder interface comment.
junhaoliao Jun 21, 2024
94519ef
Add return type to #extractLogLevel method.
junhaoliao Jun 21, 2024
f1998c6
Rename interface from "Decoders" to "Decoder".
junhaoliao Jun 21, 2024
0b69708
Rename decode function parameter 'startIdx' to 'beginIdx'.
junhaoliao Jun 21, 2024
431015b
Simplify basename extraction in url parsing.
junhaoliao Jun 21, 2024
7080cf3
Validate file size; Add file size formatting utility.
junhaoliao Jun 23, 2024
cced9f5
Reformat code.
junhaoliao Jun 23, 2024
73be097
Refactor file.ts
kirkrodrigues Jun 30, 2024
fecdec9
Refactor http.ts
kirkrodrigues Jun 30, 2024
995af22
Refactor url.ts
kirkrodrigues Jun 30, 2024
c6840fb
Refactor worker.ts
kirkrodrigues Jun 30, 2024
efb0e01
Refactor js.ts
kirkrodrigues Jun 30, 2024
5b1bd6b
Refactor StateContextProvider.
kirkrodrigues Jul 1, 2024
7a25dce
Refactor MainWorker.ts
kirkrodrigues Jul 1, 2024
dad2ed2
Rename FormatOptionsType -> FormatterOptionsType and DecodeOptionsTyp…
kirkrodrigues Jul 1, 2024
c84bfd9
Add missing semicolons.
kirkrodrigues Jul 1, 2024
0b204bb
Rename convertLogbackDateFormatToDayjs to convertDateTimeFormatterPat…
kirkrodrigues Jul 1, 2024
c1f9e0a
Add docstring for LogbackFormatterOptionsType.
kirkrodrigues Jul 1, 2024
0deac73
Clean LogbackFormatter's unnecessary defaults.
kirkrodrigues Jul 1, 2024
aad1a49
Refactor decoders.ts
kirkrodrigues Jul 1, 2024
c37e560
extractLogLevel -> parseLogLevel
kirkrodrigues Jul 1, 2024
301acf0
Edit docstrings in JsonlDecoder.ts.
kirkrodrigues Jul 1, 2024
5727210
Refactor parseLogLevel to better handle non-string values.
kirkrodrigues Jul 1, 2024
b0d3fda
extractDateFormat -> parseDateFormat, extractKeys -> parseKeys
kirkrodrigues Jul 1, 2024
fb75c13
Split extractAndFormatTimestamp into parseTimestamp and formatTimesta…
kirkrodrigues Jul 1, 2024
f7b1c48
LogbackFormatter.ts: Use explicit casts to bypass typescript's unnece…
kirkrodrigues Jul 1, 2024
8bfdb0e
input -> formatString, placeholder -> specifier, propName -> key, pro…
kirkrodrigues Jul 1, 2024
a0376ed
getRangeFrom -> getCursorRange
kirkrodrigues Jul 1, 2024
780e812
startLogEventNum -> beginLogEventNum
kirkrodrigues Jul 1, 2024
3106bad
Bug-fix: Off by one error.
kirkrodrigues Jul 1, 2024
f010f8f
LogFileManager.ts: Edit docstrings.
kirkrodrigues Jul 1, 2024
3173eb0
Update Decoder interface and restructure JsonlDecoder.
junhaoliao Jul 2, 2024
e6999fe
Refactor LogFileManager file load and creation workflow.
junhaoliao Jul 2, 2024
3c262aa
Add decoder options to JsonlDecoder constructor.
junhaoliao Jul 2, 2024
d705f21
Improve error handling when there is 0 log event.
junhaoliao Jul 2, 2024
f7bedfc
Use object instead of array for type TimestampAndMessageType.
junhaoliao Jul 2, 2024
9590a7d
Rename file.ts to units.ts and remove eslint-disables.
junhaoliao Jul 2, 2024
d76d93a
Upgrade eslint-config-yscope to version 0.0.31 wrap braceless stateme…
junhaoliao Jul 2, 2024
952e1aa
Edit JsonlDecoder's docstrings.
Jul 2, 2024
f44524c
Edit decoders.ts docstrings.
kirkrodrigues Jul 2, 2024
76b18be
Reorder JsonlDecoder's methods to match decoders.ts
kirkrodrigues Jul 2, 2024
5d1ab4c
Minor edits to LogFileManager.ts
kirkrodrigues Jul 2, 2024
31c5560
Fix condition check in JsonlDecoder.
junhaoliao Jul 3, 2024
bc8bd25
Refactor numEvents as private field in LogFileManager.
junhaoliao Jul 3, 2024
ee20ca1
Remove undefined logEvent check in JsonlDecoder.
junhaoliao Jul 3, 2024
ca80629
Refactor LogFileManager constructor, make loadFile static and make #i…
junhaoliao Jul 3, 2024
d73bd1e
Enhance error handling within JsonlDecoder:
junhaoliao Jul 3, 2024
59e9dec
Remove outdated throws in docs.
junhaoliao Jul 3, 2024
ef488c5
Improve docs.
junhaoliao Jul 3, 2024
342fe00
JsonlDecoder: Add TODO and comment.
kirkrodrigues Jul 3, 2024
80ec04b
LogFileManager: Fix docstring.
kirkrodrigues Jul 3, 2024
01bd35e
Pull general method loadFile out of LogFileManager.
kirkrodrigues Jul 3, 2024
662e8f8
Replace invalid timestamp literal with constant in JsonlDecoder.
junhaoliao Jul 3, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 62 additions & 6 deletions new-log-viewer/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions new-log-viewer/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@
},
"homepage": "https://github.com/y-scope/yscope-log-viewer#readme",
"dependencies": {
"axios": "^1.7.2",
"dayjs": "^1.11.11",
"react": "^18.3.1",
"react-dom": "^18.3.1"
},
Expand Down
30 changes: 27 additions & 3 deletions new-log-viewer/src/components/Layout.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,10 @@ import {
useEffect,
} from "react";

import {StateContext} from "../contexts/StateContextProvider";
import {
PAGE_SIZE,
StateContext,
} from "../contexts/StateContextProvider";


/**
Expand All @@ -15,16 +18,37 @@ const Layout = () => {
const {
logData,
loadFile,
logEventNum,
} = useContext(StateContext);

useEffect(() => {
loadFile("");
const urlSearchParams = new URLSearchParams(window.location.search);
const filePath = urlSearchParams.get("filePath");
if (null !== filePath) {
loadFile(filePath);
}
}, [loadFile]);

return (
<>
<div>
{logData}
<h3>
LogEventNum -
kirkrodrigues marked this conversation as resolved.
Show resolved Hide resolved
{" "}
{logEventNum}
{" "}
|
PageNum -
{" "}
{Math.ceil(logEventNum / PAGE_SIZE)}
</h3>
{logData.split("\n").map((line, index) => (
<p key={index}>
{`<${index + 1}>`}
-
{line}
</p>
))}
</div>
</>
);
Expand Down
54 changes: 39 additions & 15 deletions new-log-viewer/src/contexts/StateContextProvider.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,9 @@ import React, {
} from "react";

import {
BeginLineNumToLogEventNumMap,
CURSOR_CODE,
FileSrcType,
LineNumLogEventNumMap,
MainWorkerRespMessage,
WORKER_REQ_CODE,
WORKER_RESP_CODE,
Expand All @@ -17,10 +18,10 @@ import {


interface StateContextType {
beginLineNumToLogEventNum: BeginLineNumToLogEventNumMap,
loadFile: (fileSrc: FileSrcType) => void,
logData: string,
logEventNum: number,
logLines: LineNumLogEventNumMap,
numEvents: number,
numPages: number,
pageNum: number
Expand All @@ -30,11 +31,11 @@ const StateContext = createContext<StateContextType>({} as StateContextType);
/**
* Default values of the state object.
*/
const StateDefaultValue = Object.freeze({
const STATE_DEFAULT = Object.freeze({
beginLineNumToLogEventNum: new Map(),
loadFile: () => null,
logData: "",
logData: "Loading...",
kirkrodrigues marked this conversation as resolved.
Show resolved Hide resolved
logEventNum: 1,
logLines: new Map(),
numEvents: 0,
numPages: 0,
pageNum: 0,
Expand All @@ -53,10 +54,11 @@ interface StateContextProviderProps {
* @return
*/
const StateContextProvider = ({children}: StateContextProviderProps) => {
const [logLines, setLogLines] = useState<LineNumLogEventNumMap>(StateDefaultValue.logLines);
const [logData, setLogData] = useState<string>(StateDefaultValue.logData);
const [numEvents, setNumEvents] = useState<number>(StateDefaultValue.numEvents);
const [logEventNum, setLogEventNum] = useState<number>(StateDefaultValue.logEventNum);
const [beginLineNumToLogEventNum, setBeginLineNumToLogEventNum] =
useState<BeginLineNumToLogEventNumMap>(STATE_DEFAULT.beginLineNumToLogEventNum);
const [logData, setLogData] = useState<string>(STATE_DEFAULT.logData);
const [numEvents, setNumEvents] = useState<number>(STATE_DEFAULT.numEvents);
const [logEventNum, setLogEventNum] = useState<number>(STATE_DEFAULT.logEventNum);

const mainWorkerRef = useRef<null|Worker>(null);

Expand All @@ -79,14 +81,26 @@ const StateContextProvider = ({children}: StateContextProviderProps) => {
const {code, args} = ev.data;
console.log(`[MainWorker -> Renderer] code=${code}`);
switch (code) {
case WORKER_RESP_CODE.PAGE_DATA:
case WORKER_RESP_CODE.PAGE_DATA: {
setLogData(args.logs);
setLogLines(args.lines);
setLogEventNum(args.startLogEventNum);
setBeginLineNumToLogEventNum(args.beginLineNumToLogEventNum);
const logEventNums = Array.from(args.beginLineNumToLogEventNum.values());

let lastLogEventNum = logEventNums.at(-1);
if ("undefined" === typeof lastLogEventNum) {
lastLogEventNum = 1;
}
setLogEventNum(lastLogEventNum);
break;
}
case WORKER_RESP_CODE.NUM_EVENTS:
setNumEvents(args.numEvents);
break;
case WORKER_RESP_CODE.NOTIFICATION:
// TODO: notifications should be shown in the UI when the NotificationProvider
// is added
console.error(args.logLevel, args.message);
break;
default:
console.error(`Unexpected ev.data: ${JSON.stringify(ev.data)}`);
break;
Expand All @@ -104,7 +118,14 @@ const StateContextProvider = ({children}: StateContextProviderProps) => {
mainWorkerPostReq(WORKER_REQ_CODE.LOAD_FILE, {
fileSrc: fileSrc,
pageSize: PAGE_SIZE,
cursor: null,
cursor: {code: CURSOR_CODE.LAST_EVENT, args: null},
decoderOptions: {
// TODO: these shall come from config provider
formatString: "%d{yyyy-MM-dd HH:mm:ss.SSS} [%process.thread.name] %log.level" +
" %message%n",
logLevelKey: "log.level",
timestampKey: "@timestamp",
},
});
}, [
handleMainWorkerResp,
Expand All @@ -114,10 +135,10 @@ const StateContextProvider = ({children}: StateContextProviderProps) => {
return (
<StateContext.Provider
value={{
beginLineNumToLogEventNum,
loadFile,
logData,
logEventNum,
logLines,
numEvents,
numPages,
pageNum,
Expand All @@ -130,4 +151,7 @@ const StateContextProvider = ({children}: StateContextProviderProps) => {


export default StateContextProvider;
export {StateContext};
export {
PAGE_SIZE,
StateContext,
};
Loading