Skip to content

Commit

Permalink
Add CIP data and and new rake task for CIP data
Browse files Browse the repository at this point in the history
The CIP data is not structured the same way as the libraries and farmers' markets json files, so it needs a separate rake task.
  • Loading branch information
monfresh committed Jun 1, 2013
1 parent 0f06966 commit e1f65f7
Show file tree
Hide file tree
Showing 3 changed files with 42 additions and 5 deletions.
5 changes: 4 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -25,4 +25,7 @@ dump.rdb
# Ignore test-related files
/coverage.data
/coverage/
.rspec
.rspec

# Invalid CIP data
/data/cip_mvp_invalid_records.json
1 change: 1 addition & 0 deletions data/cip_mvp.json

Large diffs are not rendered by default.

41 changes: 37 additions & 4 deletions lib/tasks/load_data.rake
Original file line number Diff line number Diff line change
@@ -1,19 +1,23 @@
require 'json'

task :load_data => :environment do
task :load_data => [:libfarm, :cip] do

end

task :libfarm => :environment do
puts "================================================================================"
puts "Creating Organizations with Library and Farmers' Markets Data and saving to DB"
puts "================================================================================"

files = ['data/libraries_data.json', 'data/smc_farmers_markets.json']
files.each do |file|
puts "Processing #{file}"
invalid_records = {}
invalid_filename_prefix = file.split('.json')[0].split('/')[1]
dataset = JSON.parse(File.read(file)).values
dataset.each do |data_item|
org = Organization.new(data_item)
unless org.save
invalid_records = {}
invalid_filename_prefix = file.split('.json')[0].split('/')[1]
name = org["name"]
invalid_records[name] = {}
invalid_records[name]["errors"] = org.errors
Expand All @@ -24,5 +28,34 @@ task :load_data => :environment do
end
puts "Done loading #{file} into DB"
end
puts "Done loading all files into DB."
puts "Done loading libraries and farmers' markets into DB."
end

task :cip => :environment do
puts "================================================================================"
puts "Creating Organizations with PLS CIP Data and saving to DB"
puts "================================================================================"

files = ['data/cip_mvp.json']
files.each do |file|
puts "Processing #{file}"
invalid_records = {}
invalid_filename_prefix = file.split('.json')[0].split('/')[1]
dataset = JSON.parse(File.read(file))
dataset.each do |array|
array.each do |data_item|
org = Organization.new(data_item)
unless org.save
name = org["name"]
invalid_records[name] = {}
invalid_records[name]["errors"] = org.errors
File.open("data/#{invalid_filename_prefix}_invalid_records.json","w") do |f|
f.write(invalid_records.to_json)
end
end
end
end
puts "Done loading #{file} into DB"
end
puts "Done loading CIP data into DB."
end

0 comments on commit e1f65f7

Please sign in to comment.