Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Costar hyper readme update #529

Merged
merged 40 commits into from
Oct 18, 2018
Merged
Changes from 1 commit
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
45de584
relocate dataset file lists
ahundt Oct 5, 2018
1c35ca6
Costar plan and costar hyper readme improvements
ahundt Oct 5, 2018
e0e5366
readme.md add website link
ahundt Oct 5, 2018
78cf2f2
readme cleanup and links
ahundt Oct 5, 2018
b9d3c00
readme.md create & improve links
ahundt Oct 5, 2018
bb2bb6f
Slightly modify README to be more readable
RexxarCHL Oct 8, 2018
81e3eee
Add code to check action labels for consistency
RexxarCHL Oct 8, 2018
a607c91
Modify deprecated argument usage
RexxarCHL Oct 8, 2018
386e808
Implement failure and error set splitting
RexxarCHL Oct 9, 2018
0d0bf81
Implement dataset splitting for error only and failure only subset
RexxarCHL Oct 10, 2018
6c4fd0e
Error is also a type of failure
RexxarCHL Oct 10, 2018
87beadd
Modify output label
RexxarCHL Oct 10, 2018
ba29f96
Modify output file names
RexxarCHL Oct 10, 2018
19dcafc
Increase readability on the help text. Parameterize the hard-coded va…
RexxarCHL Oct 12, 2018
0444fb1
WIP: Add bazilion comments; split_dataset behavior overhaul
RexxarCHL Oct 12, 2018
65e3f78
Fix a bug and add a bunch of sanity checks
RexxarCHL Oct 13, 2018
cf83741
grasp_model.py -> hypertree_model.py
ahundt Oct 13, 2018
648bdd7
Merge branch 'costar_hyper' of github.com:cpaxton/costar_plan into co…
ahundt Oct 13, 2018
9781bb7
Merge pull request #531 from RexxarCHL/action_label_check
ahundt Oct 13, 2018
d0bbdef
Minor code style changes
RexxarCHL Oct 15, 2018
20c8c0f
costar_block_stacking_split_dataset.py set random seed, write summary…
ahundt Oct 15, 2018
d6a0d40
return filename
ahundt Oct 15, 2018
f68cd25
WIP: Refactor dataset splitting to include default behaviour
RexxarCHL Oct 16, 2018
be502c9
WIP: Finish refactoring the script. To be debugged.
RexxarCHL Oct 16, 2018
976fd1b
hyperopt_plot.py dramatically improved plot output with averages
ahundt Oct 16, 2018
563ce98
WIP: Debug all functionalities. Working on output combined files
RexxarCHL Oct 16, 2018
23189ff
hyperopt_plot.py provides a proper summary now
ahundt Oct 16, 2018
e30b507
hyperopt_plot.py better variable parameterization
ahundt Oct 16, 2018
4f158fb
Finish refactoring the split script
RexxarCHL Oct 16, 2018
7466db6
Modify help description
RexxarCHL Oct 16, 2018
8b94c4d
Update IA scripts for default values and metadata
RexxarCHL Oct 16, 2018
ec377c4
Minor update for better readability
RexxarCHL Oct 16, 2018
64e5955
Add expand user for path
RexxarCHL Oct 16, 2018
6496ccd
Change email address
ahundt Oct 17, 2018
a1fa172
Small changes to the code
RexxarCHL Oct 17, 2018
cebefd8
Merge pull request #532 from RexxarCHL/split_failure_error_sets
ahundt Oct 17, 2018
9de577c
cornell_grasp_train.py fix user directory bug
ahundt Oct 17, 2018
05f419f
cornell_hyperopt.py do a new search on full grasp regression
ahundt Oct 17, 2018
f266e4b
Merge branch 'costar_hyper' of github.com:cpaxton/costar_plan into co…
ahundt Oct 17, 2018
14b8a35
hyperopt_plot.py configurable dimensions and model count
ahundt Oct 18, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
hyperopt_plot.py provides a proper summary now
ahundt committed Oct 16, 2018

Verified

This commit was signed with the committer’s verified signature.
tomusdrw Tomek Drwięga
commit 23189ff1edabe68757005a01cfcaa97f25fa9e2f
37 changes: 25 additions & 12 deletions costar_hyper/hyperopt_plot.py
Original file line number Diff line number Diff line change
@@ -337,31 +337,44 @@ def create_data_comparison_table(value_dimension_tuples_mm, units, problem_type)
rdf = create_data_comparison_table(value_dimension_tuples_mm, units, problem_type)

# key_dimensions = [('name', 'Model'), ('error_distribution_limits', 'Accuracy Range'), ('train_val_test', 'Train Val Test')]
key_dimensions = [('name', 'Model'), ('error_distribution_limits', 'Error Distribution'), ('train_val_test', 'Dataset Split')]
key_dimensions = [('name', 'Model'), ('error_distribution_limits', 'Error Range'), ('train_val_test', 'Dataset Split')]
key_dimension_display_strs = [vt[1] for vt in key_dimensions]
value_dimensions = [('accuracy_range_value', 'Cumulative Fraction of Examples'), ('avg_error', 'Average Error')]#, ('train_val_test', 'Dataset Split')]
value_dimensions = [('accuracy_range_value', 'Error Distribution'), ('avg_error', 'Average Error')]#, ('train_val_test', 'Dataset Split')]
value_dimension_display_strs = [vt[1] for vt in value_dimensions]
distribution_table = hv.Table(rdf, key_dimensions, value_dimensions)
print('1.0 dist table created')
distribution_table_bars = distribution_table.to.bars(key_dimension_display_strs, value_dimension_display_strs, ['train_val_test'])
distribution_table_bars = distribution_table_bars.options(stack_index=1, width=1920, height=1080, xrotation=90, tools=['hover'], group_index=2, cmap='RdYlGn_r')
distribution_table_bars = distribution_table.to.bars(key_dimension_display_strs, value_dimension_display_strs, [])
height = 240
distribution_table_bars = distribution_table_bars.options(stack_index=1, width=1280, height=height, xrotation=90, tools=['hover'], group_index='train_val_test', cmap='RdYlGn_r', show_grid=True)
# plot train, val, test separately, the + sign sticks the plots together
distribution_table_bars = (
distribution_table_bars.select(train_val_test='train').relabel(group='Train').options(xaxis=None) +
distribution_table_bars.select(train_val_test='val', xaxis=None).relabel(group='Val').options(xaxis=None) +
distribution_table_bars.select(train_val_test='test').relabel(group='Test').options(height=height + 160))
distribution_table_bars = distribution_table_bars.cols(1)
# distribution_table_bars = distribution_table_bars.select(train_val_test='train')
# distribution_table_bars = distribution_table_bars.grid('train_val_test')
# distribution_table_bars = distribution_table_bars.overlay('train_val_test')
print('2.0 dist table bars')
distribution_table_plot = renderer.get_plot(distribution_table_bars)
# distribution_table_plot = renderer.get_plot(distribution_table_bars)

print('3.0 dist table created')
key_dimensions = [('name', 'Model')]
key_dimensions = [('name', 'Model'), ('train_val_test', 'Dataset Split')]
key_dimension_display_strs = [vt[1] for vt in key_dimensions]
value_dimensions = [('avg_error', 'Average Error'), ('train_val_test', 'Dataset Split')]
value_dimensions = [('avg_error', 'Average Error')]
value_dimension_display_strs = [vt[1] for vt in value_dimensions]
avg_table_bars = hv.Table(rdf, key_dimensions, value_dimensions)
print('4.0 avg table created')
avg_table_bars = avg_table_bars.to.bars(key_dimension_display_strs, value_dimension_display_strs, ['train_val_test'])
avg_table_bars = avg_table_bars.options(width=1920, height=640, xrotation=90, tools=['hover'])
avg_table_bars = avg_table_bars.to.bars(key_dimension_display_strs, value_dimension_display_strs, [])
avg_table_bars = avg_table_bars.options(width=1280, height=160, xrotation=90, tools=['hover'], group_index='train_val_test', xaxis=None)
print('4.0 avg table bars')
avg_table_plot = renderer.get_plot(avg_table_bars)
print('5.0 table plot')
plot_list = [[distribution_table_plot.state], [avg_table_plot.state]]
# avg_table_plot = renderer.get_plot(avg_table_bars)
# print('5.0 table plot')
# plot_list = [[avg_table_plot.state], [distribution_table_plot.state]]
table_bars = avg_table_bars + distribution_table_bars
table_bars = table_bars.cols(1)
table_plot = renderer.get_plot(table_bars)
plot_list = [[table_plot.state]]
print('6.0 plot list')
# layout_child = layout(plot_list, sizing_mode='fixed')
layout_child = layout(plot_list)