Added all database related code #5

parauliya · 2019-10-31T19:19:48Z

Also added the changes for generating action list for a workflow by fetching the worker ID from "hardware" table.
The above action list will be stored in wfstate table.

…Table in "workflow create" command. Also added the changes for generating action list for a workflow by fetching the worker ID from "hardware" table. The above action list will be stored in wfstate table.

gauravgahlot

I have left some feedback on the changes. Also, it would be nice to move all the types out of /db/workflow.go (may be to /db/types.go).

db/workflow.go

deploy/docker-entrypoint-initdb.d/rover-init.sql

db/workflow.go

parauliya · 2019-11-01T13:09:48Z

I have left some feedback on the changes. Also, it would be nice to move all the types out of /db/workflow.go (may be to /db/types.go).

We can do that. but first we should integrate our code and verify whether it is working. Then we can do the refactoring as there are lot of other things I need to do like schema validation, unique names for task and action under task etc. I would suggest that once you are done with your changes, merge this pull request to workflow branch and try to test the complete thing.

Also added current_task_name and current_action_name in workflow_state table

The above table will be updated when the ReportActionStatus endpoint of rover server will be called The ReportActionStatus endpoint is called to update the status of a particular action from worker node

…Table in "workflow create" command. Also added the changes for generating action list for a workflow by fetching the worker ID from "hardware" table. The above action list will be stored in wfstate table.

Also added current_task_name and current_action_name in workflow_state table

The above table will be updated when the ReportActionStatus endpoint of rover server will be called The ReportActionStatus endpoint is called to update the status of a particular action from worker node

…into workflow-child

gauravgahlot · 2019-11-04T07:27:06Z

LGTM for now. We will do further optimizations on a later branch.

where origin/master is at ee03be7 Core was generated by `git rebase -i origin/master --autosquash'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f7a99f8a15a in raise () from /nix/store/9hy6c2hv8lcwc6clnc1p2jf09cs5q9dp-glibc-2.30/lib/libc.so.6 (gdb) bt #0 0x00007f7a99f8a15a in raise () from /nix/store/9hy6c2hv8lcwc6clnc1p2jf09cs5q9dp-glibc-2.30/lib/libc.so.6 tinkerbell#1 0x00007f7a99f74548 in abort () from /nix/store/9hy6c2hv8lcwc6clnc1p2jf09cs5q9dp-glibc-2.30/lib/libc.so.6 tinkerbell#2 0x00007f7a99fcb748 in __libc_message () from /nix/store/9hy6c2hv8lcwc6clnc1p2jf09cs5q9dp-glibc-2.30/lib/libc.so.6 tinkerbell#3 0x00007f7a99fd292a in malloc_printerr () from /nix/store/9hy6c2hv8lcwc6clnc1p2jf09cs5q9dp-glibc-2.30/lib/libc.so.6 tinkerbell#4 0x00007f7a99fd4728 in _int_free () from /nix/store/9hy6c2hv8lcwc6clnc1p2jf09cs5q9dp-glibc-2.30/lib/libc.so.6 tinkerbell#5 0x00000000005b1707 in todo_list_rearrange_squash (todo_list=todo_list@entry=0x7ffcca02ad20) at sequencer.c:5328 tinkerbell#6 0x00000000005b1a76 in complete_action (r=0x718640 <the_repo>, opts=opts@entry=0x7ffcca02ad80, flags=flags@entry=0, shortrevisions=0x1df4530 "ee03be7..a3d2a27", onto_name=0x7ffcca02dfaf "origin/master", onto=0x1e04b70, orig_head=0x718a20 <hexbuffer> "a3d2a275b37b507345348aaf57d46040dea2a80e", commands=0x7ffcca02ad00, autosquash=1, todo_list=0x7ffcca02ad20) at sequencer.c:5089 tinkerbell#7 0x00000000004798f3 in do_interactive_rebase (opts=opts@entry=0x7ffcca02b200, flags=0) at builtin/rebase.c:356 tinkerbell#8 0x0000000000479a19 in run_rebase_interactive (opts=opts@entry=0x7ffcca02b200, command=command@entry=ACTION_NONE) at builtin/rebase.c:390 tinkerbell#9 0x000000000047b3a5 in run_specific_rebase (opts=opts@entry=0x7ffcca02b200, action=ACTION_NONE) at builtin/rebase.c:1104 tinkerbell#10 0x000000000047d958 in cmd_rebase (argc=<optimized out>, argv=<optimized out>, prefix=<optimized out>) at builtin/rebase.c:2189 tinkerbell#11 0x00000000004062dc in run_builtin (p=0x6d5800 <commands+2016>, argc=argc@entry=4, argv=argv@entry=0x7ffcca02c3d0) at git.c:444 tinkerbell#12 0x000000000040654b in handle_builtin (argc=4, argv=0x7ffcca02c3d0) at git.c:674 tinkerbell#13 0x0000000000407b69 in run_argv (argcp=argcp@entry=0x7ffcca02c2ac, argv=argv@entry=0x7ffcca02c2a0) at git.c:741 tinkerbell#14 0x0000000000407f13 in cmd_main (argc=<optimized out>, argc@entry=5, argv=<optimized out>, argv@entry=0x7ffcca02c3c8) at git.c:872 tinkerbell#15 0x00000000004b4aad in main (argc=5, argv=0x7ffcca02c3c8) at common-main.c:52 (gdb) fram 5 tinkerbell#5 0x00000000005b1707 in todo_list_rearrange_squash (todo_list=todo_list@entry=0x7ffcca02ad20) at sequencer.c:5328 5328 in sequencer.c (gdb) info locals subject2item = {table = 0x1dfde20, cmpfn = 0x5a1c6c <subject2item_cmp>, cmpfn_data = 0x0, private_size = 38, tablesize = 256, grow_at = 204, shrink_at = 40, do_count_items = 1} rearranged = 1 next = 0x1dfc390 tail = 0x1e04a20 i = <optimized out> nr = <optimized out> alloc = <optimized out> subjects = 0x1e27ef0 commit_todo = {slab_size = 65532, stride = 1, slab_count = 1, slab = 0x1e34650} items = <optimized out> (gdb) frame 6 tinkerbell#6 0x00000000005b1a76 in complete_action (r=0x718640 <the_repo>, opts=opts@entry=0x7ffcca02ad80, flags=flags@entry=0, shortrevisions=0x1df4530 "ee03be7..a3d2a27", onto_name=0x7ffcca02dfaf "origin/master", onto=0x1e04b70, orig_head=0x718a20 <hexbuffer> "a3d2a275b37b507345348aaf57d46040dea2a80e", commands=0x7ffcca02ad00, autosquash=1, todo_list=0x7ffcca02ad20) at sequencer.c:5089 5089 in sequencer.c (gdb) info locals shortonto = 0x718ae3 <hexbuffer+195> "ee03be7" todo_file = 0x1e34aa0 ".git/rebase-merge/git-rebase-todo" new_todo = {buf = {alloc = 0, len = 0, buf = 0x729460 <strbuf_slopbuf> ""}, items = 0x0, nr = 0, alloc = 0, current = 0, done_nr = 0, total_nr = 0, stat = {sd_ctime = {sec = 0, nsec = 0}, sd_mtime = {sec = 0, nsec = 0}, sd_dev = 0, sd_ino = 0, sd_uid = 0, sd_gid = 0, sd_size = 0}} buf = 0x7ffcca02ad20 buf2 = {alloc = 0, len = 0, buf = 0x729460 <strbuf_slopbuf> ""} oid = {hash = "\356\003\276t\267\017L,\213\374G+\213\023\223\324B\324\065k\025\004\000\001\360\262\002\312\374\177\000"} res = <optimized out> (gdb) frame 7 tinkerbell#7 0x00000000004798f3 in do_interactive_rebase (opts=opts@entry=0x7ffcca02b200, flags=0) at builtin/rebase.c:356 356 builtin/rebase.c: No such file or directory. (gdb) info locals ret = <optimized out> head_hash = 0x718a20 <hexbuffer> "a3d2a275b37b507345348aaf57d46040dea2a80e" revisions = 0x1e2b630 "ee03be74b70f4c2c8bfc472b8b1393d442d4356b...a3d2a27" shortrevisions = 0x1df4530 "ee03be7..a3d2a27" make_script_args = {argv = 0x1df4580, argc = 2, alloc = 24} todo_list = {buf = {alloc = 9654, len = 6690, buf = 0x1e36ed0 "pick a920194 setup.sh: don't install software automatically\npick 7868947 generate-envrc: make its own, trivial, shell script\npick c040"...}, items = 0x1e39c90, nr = 62, alloc = 114, current = 0, done_nr = 0, total_nr = 64, stat = {sd_ctime = {sec = 0, nsec = 0}, sd_mtime = {sec = 0, nsec = 0}, sd_dev = 0, sd_ino = 0, sd_uid = 0, sd_gid = 0, sd_size = 0}} replay = {action = REPLAY_INTERACTIVE_REBASE, edit = 0, record_origin = 0, no_commit = 0, signoff = 0, allow_ff = 1, allow_rerere_auto = 0, allow_empty = 1, allow_empty_message = 1, keep_redundant_commits = 0, verbose = 0, quiet = 0, reschedule_failed_exec = 0, mainline = 0, gpg_sign = 0x1e2db60 "", default_msg_cleanup = COMMIT_MSG_CLEANUP_NONE, explicit_cleanup = 0, strategy = 0x0, xopts = 0x0, xopts_nr = 0, xopts_alloc = 0, current_fixups = {alloc = 0, len = 0, buf = 0x729460 <strbuf_slopbuf> ""}, current_fixup_count = 0, squash_onto = { hash = '\000' <repeats 31 times>}, have_squash_onto = 0, revs = 0x0} commands = {items = 0x0, nr = 0, alloc = 0, strdup_strings = 1, cmp = 0x0} (gdb) frame 8 tinkerbell#8 0x0000000000479a19 in run_rebase_interactive (opts=opts@entry=0x7ffcca02b200, command=command@entry=ACTION_NONE) at builtin/rebase.c:390 390 in builtin/rebase.c (gdb) info locals flags = <optimized out> abbreviate_commands = 0 ret = 0 (gdb) frame 9 tinkerbell#9 0x000000000047b3a5 in run_specific_rebase (opts=opts@entry=0x7ffcca02b200, action=ACTION_NONE) at builtin/rebase.c:1104 1104 in builtin/rebase.c (gdb) info locals argv = {0x0, 0x0} script_snippet = {alloc = 0, len = 0, buf = 0x729460 <strbuf_slopbuf> ""} buf = {alloc = 0, len = 0, buf = 0x729460 <strbuf_slopbuf> ""} status = <optimized out> backend = <optimized out> backend_func = <optimized out>

check out this commit and run: git rebase -i ee03be7 --autosquash then: Core was generated by `git rebase -i origin/master --autosquash'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f7a99f8a15a in raise () from /nix/store/9hy6c2hv8lcwc6clnc1p2jf09cs5q9dp-glibc-2.30/lib/libc.so.6 (gdb) bt #0 0x00007f7a99f8a15a in raise () from /nix/store/9hy6c2hv8lcwc6clnc1p2jf09cs5q9dp-glibc-2.30/lib/libc.so.6 tinkerbell#1 0x00007f7a99f74548 in abort () from /nix/store/9hy6c2hv8lcwc6clnc1p2jf09cs5q9dp-glibc-2.30/lib/libc.so.6 tinkerbell#2 0x00007f7a99fcb748 in __libc_message () from /nix/store/9hy6c2hv8lcwc6clnc1p2jf09cs5q9dp-glibc-2.30/lib/libc.so.6 tinkerbell#3 0x00007f7a99fd292a in malloc_printerr () from /nix/store/9hy6c2hv8lcwc6clnc1p2jf09cs5q9dp-glibc-2.30/lib/libc.so.6 tinkerbell#4 0x00007f7a99fd4728 in _int_free () from /nix/store/9hy6c2hv8lcwc6clnc1p2jf09cs5q9dp-glibc-2.30/lib/libc.so.6 tinkerbell#5 0x00000000005b1707 in todo_list_rearrange_squash (todo_list=todo_list@entry=0x7ffcca02ad20) at sequencer.c:5328 tinkerbell#6 0x00000000005b1a76 in complete_action (r=0x718640 <the_repo>, opts=opts@entry=0x7ffcca02ad80, flags=flags@entry=0, shortrevisions=0x1df4530 "ee03be7..a3d2a27", onto_name=0x7ffcca02dfaf "origin/master", onto=0x1e04b70, orig_head=0x718a20 <hexbuffer> "a3d2a275b37b507345348aaf57d46040dea2a80e", commands=0x7ffcca02ad00, autosquash=1, todo_list=0x7ffcca02ad20) at sequencer.c:5089 tinkerbell#7 0x00000000004798f3 in do_interactive_rebase (opts=opts@entry=0x7ffcca02b200, flags=0) at builtin/rebase.c:356 tinkerbell#8 0x0000000000479a19 in run_rebase_interactive (opts=opts@entry=0x7ffcca02b200, command=command@entry=ACTION_NONE) at builtin/rebase.c:390 tinkerbell#9 0x000000000047b3a5 in run_specific_rebase (opts=opts@entry=0x7ffcca02b200, action=ACTION_NONE) at builtin/rebase.c:1104 tinkerbell#10 0x000000000047d958 in cmd_rebase (argc=<optimized out>, argv=<optimized out>, prefix=<optimized out>) at builtin/rebase.c:2189 tinkerbell#11 0x00000000004062dc in run_builtin (p=0x6d5800 <commands+2016>, argc=argc@entry=4, argv=argv@entry=0x7ffcca02c3d0) at git.c:444 tinkerbell#12 0x000000000040654b in handle_builtin (argc=4, argv=0x7ffcca02c3d0) at git.c:674 tinkerbell#13 0x0000000000407b69 in run_argv (argcp=argcp@entry=0x7ffcca02c2ac, argv=argv@entry=0x7ffcca02c2a0) at git.c:741 tinkerbell#14 0x0000000000407f13 in cmd_main (argc=<optimized out>, argc@entry=5, argv=<optimized out>, argv@entry=0x7ffcca02c3c8) at git.c:872 tinkerbell#15 0x00000000004b4aad in main (argc=5, argv=0x7ffcca02c3c8) at common-main.c:52 (gdb) fram 5 tinkerbell#5 0x00000000005b1707 in todo_list_rearrange_squash (todo_list=todo_list@entry=0x7ffcca02ad20) at sequencer.c:5328 5328 in sequencer.c (gdb) info locals subject2item = {table = 0x1dfde20, cmpfn = 0x5a1c6c <subject2item_cmp>, cmpfn_data = 0x0, private_size = 38, tablesize = 256, grow_at = 204, shrink_at = 40, do_count_items = 1} rearranged = 1 next = 0x1dfc390 tail = 0x1e04a20 i = <optimized out> nr = <optimized out> alloc = <optimized out> subjects = 0x1e27ef0 commit_todo = {slab_size = 65532, stride = 1, slab_count = 1, slab = 0x1e34650} items = <optimized out> (gdb) frame 6 tinkerbell#6 0x00000000005b1a76 in complete_action (r=0x718640 <the_repo>, opts=opts@entry=0x7ffcca02ad80, flags=flags@entry=0, shortrevisions=0x1df4530 "ee03be7..a3d2a27", onto_name=0x7ffcca02dfaf "origin/master", onto=0x1e04b70, orig_head=0x718a20 <hexbuffer> "a3d2a275b37b507345348aaf57d46040dea2a80e", commands=0x7ffcca02ad00, autosquash=1, todo_list=0x7ffcca02ad20) at sequencer.c:5089 5089 in sequencer.c (gdb) info locals shortonto = 0x718ae3 <hexbuffer+195> "ee03be7" todo_file = 0x1e34aa0 ".git/rebase-merge/git-rebase-todo" new_todo = {buf = {alloc = 0, len = 0, buf = 0x729460 <strbuf_slopbuf> ""}, items = 0x0, nr = 0, alloc = 0, current = 0, done_nr = 0, total_nr = 0, stat = {sd_ctime = {sec = 0, nsec = 0}, sd_mtime = {sec = 0, nsec = 0}, sd_dev = 0, sd_ino = 0, sd_uid = 0, sd_gid = 0, sd_size = 0}} buf = 0x7ffcca02ad20 buf2 = {alloc = 0, len = 0, buf = 0x729460 <strbuf_slopbuf> ""} oid = {hash = "\356\003\276t\267\017L,\213\374G+\213\023\223\324B\324\065k\025\004\000\001\360\262\002\312\374\177\000"} res = <optimized out> (gdb) frame 7 tinkerbell#7 0x00000000004798f3 in do_interactive_rebase (opts=opts@entry=0x7ffcca02b200, flags=0) at builtin/rebase.c:356 356 builtin/rebase.c: No such file or directory. (gdb) info locals ret = <optimized out> head_hash = 0x718a20 <hexbuffer> "a3d2a275b37b507345348aaf57d46040dea2a80e" revisions = 0x1e2b630 "ee03be74b70f4c2c8bfc472b8b1393d442d4356b...a3d2a27" shortrevisions = 0x1df4530 "ee03be7..a3d2a27" make_script_args = {argv = 0x1df4580, argc = 2, alloc = 24} todo_list = {buf = {alloc = 9654, len = 6690, buf = 0x1e36ed0 "pick a920194 setup.sh: don't install software automatically\npick 7868947 generate-envrc: make its own, trivial, shell script\npick c040"...}, items = 0x1e39c90, nr = 62, alloc = 114, current = 0, done_nr = 0, total_nr = 64, stat = {sd_ctime = {sec = 0, nsec = 0}, sd_mtime = {sec = 0, nsec = 0}, sd_dev = 0, sd_ino = 0, sd_uid = 0, sd_gid = 0, sd_size = 0}} replay = {action = REPLAY_INTERACTIVE_REBASE, edit = 0, record_origin = 0, no_commit = 0, signoff = 0, allow_ff = 1, allow_rerere_auto = 0, allow_empty = 1, allow_empty_message = 1, keep_redundant_commits = 0, verbose = 0, quiet = 0, reschedule_failed_exec = 0, mainline = 0, gpg_sign = 0x1e2db60 "", default_msg_cleanup = COMMIT_MSG_CLEANUP_NONE, explicit_cleanup = 0, strategy = 0x0, xopts = 0x0, xopts_nr = 0, xopts_alloc = 0, current_fixups = {alloc = 0, len = 0, buf = 0x729460 <strbuf_slopbuf> ""}, current_fixup_count = 0, squash_onto = { hash = '\000' <repeats 31 times>}, have_squash_onto = 0, revs = 0x0} commands = {items = 0x0, nr = 0, alloc = 0, strdup_strings = 1, cmp = 0x0} (gdb) frame 8 tinkerbell#8 0x0000000000479a19 in run_rebase_interactive (opts=opts@entry=0x7ffcca02b200, command=command@entry=ACTION_NONE) at builtin/rebase.c:390 390 in builtin/rebase.c (gdb) info locals flags = <optimized out> abbreviate_commands = 0 ret = 0 (gdb) frame 9 tinkerbell#9 0x000000000047b3a5 in run_specific_rebase (opts=opts@entry=0x7ffcca02b200, action=ACTION_NONE) at builtin/rebase.c:1104 1104 in builtin/rebase.c (gdb) info locals argv = {0x0, 0x0} script_snippet = {alloc = 0, len = 0, buf = 0x729460 <strbuf_slopbuf> ""} buf = {alloc = 0, len = 0, buf = 0x729460 <strbuf_slopbuf> ""} status = <optimized out> backend = <optimized out> backend_func = <optimized out>

Added support of creating of WFState Table, Event Table and WfWorker …

56fad24

…Table in "workflow create" command. Also added the changes for generating action list for a workflow by fetching the worker ID from "hardware" table. The above action list will be stored in wfstate table.

parauliya changed the title ~~Added support of creating of WfState, WfEvent and WfWorker Table~~ [WIP] Added support of creating of WfState, WfEvent and WfWorker Table Oct 31, 2019

gauravgahlot suggested changes Nov 1, 2019

View reviewed changes

parauliya added 2 commits November 1, 2019 16:46

Incorporated the review comments

f5486c4

Fix the get worker ID by IP address which was not working earliar

55ca530

parauliya requested a review from gauravgahlot November 1, 2019 13:10

parauliya added 2 commits November 2, 2019 02:03

Added validation for unique task name and unique action name in a task.

73a4d79

Also added current_task_name and current_action_name in workflow_state table

Added code for update workflow_state and workflow_event table in db

956425b

The above table will be updated when the ReportActionStatus endpoint of rover server will be called The ReportActionStatus endpoint is called to update the status of a particular action from worker node

parauliya changed the title ~~[WIP] Added support of creating of WfState, WfEvent and WfWorker Table~~ [WIP] Added all database related code Nov 2, 2019

parauliya added 7 commits November 3, 2019 23:46

Create db function to run end to end workflow in respect of db changes

163e801

Added support of creating of WFState Table, Event Table and WfWorker …

d02c021

…Table in "workflow create" command. Also added the changes for generating action list for a workflow by fetching the worker ID from "hardware" table. The above action list will be stored in wfstate table.

Incorporated the review comments

d8c96d8

Fix the get worker ID by IP address which was not working earliar

62ee88e

Added validation for unique task name and unique action name in a task.

b3c37f5

Also added current_task_name and current_action_name in workflow_state table

Added code for update workflow_state and workflow_event table in db

b103eb2

The above table will be updated when the ReportActionStatus endpoint of rover server will be called The ReportActionStatus endpoint is called to update the status of a particular action from worker node

Create db function to run end to end workflow in respect of db changes

0c3e2be

gauravgahlot force-pushed the workflow-child branch from 163e801 to 0c3e2be Compare November 4, 2019 06:50

parauliya added 2 commits November 4, 2019 12:33

Remove the commented code

f69b7b8

Merge branch 'workflow-child' of https://github.com/packethost/rover …

2a8c298

…into workflow-child

gauravgahlot merged commit d89b958 into workflow Nov 4, 2019

gauravgahlot changed the title ~~[WIP] Added all database related code~~ Added all database related code Nov 4, 2019

gauravgahlot deleted the workflow-child branch November 7, 2019 04:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added all database related code #5

Added all database related code #5

parauliya commented Oct 31, 2019

gauravgahlot left a comment

parauliya commented Nov 1, 2019

gauravgahlot commented Nov 4, 2019

Added all database related code #5

Added all database related code #5

Conversation

parauliya commented Oct 31, 2019

gauravgahlot left a comment

Choose a reason for hiding this comment

parauliya commented Nov 1, 2019

gauravgahlot commented Nov 4, 2019