Replace Elasticsearch with Postgres full-text search. Fixes #139

codeforamerica · May 10, 2014 · 52dd99f · 52dd99f
1 parent 7fe7b62
commit 52dd99f
Show file tree

Hide file tree

Showing 42 changed files with 464 additions and 585 deletions.
diff --git a/.travis.yml b/.travis.yml
@@ -7,12 +7,10 @@ addons:
   postgresql: "9.3"
 services:
   - redis-server
-  - elasticsearch
 before_script:
   - psql -c 'create database ohana_api_test;' -U postgres
+  - psql -U postgres -q -d ohana_api_test -f db/structure.sql
   - cp config/database.travis.yml config/database.yml
-  - rake db:schema:load
-  - rake db:test:load
 notifications:
   campfire:
     rooms:

diff --git a/Gemfile b/Gemfile
@@ -54,7 +54,7 @@ group :development do
 end
 
 # Geocoding
-gem 'geocoder', git: 'git://github.com/alexreisner/geocoder.git', ref: '3568e5e8e6'
+gem 'geocoder'
 gem 'redis'
 
 # Format validation for URLs, phone numbers, zipcodes
@@ -66,7 +66,7 @@ gem 'rack-cors', require: 'rack/cors'
 # API Design
 gem 'grape'
 gem 'grape-entity'
-gem 'kaminari', git: 'git://github.com/amatsuda/kaminari.git', ref: '01f65e112d'
+gem 'kaminari'
 
 # Caching
 # gem 'garner'
@@ -96,9 +96,12 @@ gem 'enumerize'
 gem 'figaro'
 
 # Search
-gem 'tire', git: 'git://github.com/monfresh/tire.git', ref: '2d174e792a'
+gem 'pg_search'
 
 # Nested categories for OpenEligibility
 gem 'ancestry'
 
 gem 'friendly_id', '~> 5.0.3'
+
+# Customizable Settings
+gem 'rails_config'
diff --git a/Gemfile.lock b/Gemfile.lock
@@ -1,33 +1,3 @@
-GIT
-  remote: git://github.com/alexreisner/geocoder.git
-  revision: 3568e5e8e65aa1fe94edc2a3f69f6eed4b88a9d9
-  ref: 3568e5e8e6
-  specs:
-    geocoder (1.1.8)
-
-GIT
-  remote: git://github.com/amatsuda/kaminari.git
-  revision: 01f65e112dd9741921e850473739065a55615b1d
-  ref: 01f65e112d
-  specs:
-    kaminari (0.14.1)
-      actionpack (>= 3.0.0)
-      activesupport (>= 3.0.0)
-
-GIT
-  remote: git://github.com/monfresh/tire.git
-  revision: 2d174e792a7e1a1be56cfee3974fb5a080a940a1
-  ref: 2d174e792a
-  specs:
-    tire (0.6.0)
-      activemodel (>= 3.0)
-      activesupport
-      ansi
-      hashr (~> 0.0.19)
-      multi_json (~> 1.3)
-      rake
-      rest-client (~> 1.6)
-
 GEM
   remote: https://rubygems.org/
   specs:
@@ -57,7 +27,6 @@ GEM
       tzinfo (~> 0.3.37)
     ancestry (2.1.0)
       activerecord (>= 3.0.0)
-    ansi (1.4.3)
     arel (4.0.2)
     arrayfields (4.9.2)
     ast (2.0.0)
@@ -149,6 +118,7 @@ GEM
       sexp_processor (~> 4.4)
     friendly_id (5.0.3)
       activerecord (>= 4.0.0)
+    geocoder (1.2.0)
     grape (0.7.0)
       activesupport
       builder
@@ -174,7 +144,6 @@ GEM
       haml (>= 3.1, < 5.0)
       railties (>= 4.0.1)
     hashie (2.1.1)
-    hashr (0.0.22)
     hike (1.2.3)
     hirb (0.7.1)
     i18n (0.6.9)
@@ -184,6 +153,9 @@ GEM
       thor (>= 0.14, < 2.0)
     json (1.8.1)
     json_pure (1.8.1)
+    kaminari (0.15.1)
+      actionpack (>= 3.0.0)
+      activesupport (>= 3.0.0)
     kgio (2.9.1)
     kramdown (1.3.3)
     mail (2.5.4)
@@ -231,6 +203,10 @@ GEM
       rack
       rake (>= 0.8.1)
     pg (0.17.1)
+    pg_search (0.7.3)
+      activerecord (>= 3.1)
+      activesupport (>= 3.1)
+      arel
     polyglot (0.3.4)
     powerpack (0.0.9)
     protected_attributes (1.0.7)
@@ -266,6 +242,8 @@ GEM
       i18n
       require_all
       ruby-progressbar
+    rails_config (0.4.1)
+      activesupport (>= 3.0)
     rails_serve_static_assets (0.0.1)
     rails_stdout_logging (0.0.2)
     railties (4.0.4)
@@ -380,32 +358,33 @@ DEPENDENCIES
   factory_girl_rails (>= 4.2.0)
   figaro
   friendly_id (~> 5.0.3)
-  geocoder!
+  geocoder
   grape
   grape-entity
   grape-swagger
   haml-rails (~> 0.5.3)
   jquery-rails
-  kaminari!
+  kaminari
   kgio
   memcachier
   metric_fu
   newrelic-grape
   newrelic_rpm
   passenger
   pg
+  pg_search
   protected_attributes
   quiet_assets (>= 1.0.2)
   rack-cors
   rack-timeout
   rails (~> 4.0.4)
   rails_12factor
+  rails_config
   redis
   rspec-rails (~> 2.14.2)
   rubocop
   sass-rails (~> 4.0.2)
   shoulda-matchers
   swagger-ui_rails
-  tire!
   uglifier (>= 1.3.0)
   validates_formatting_of (~> 0.8.1)
diff --git a/README.md b/README.md
@@ -13,12 +13,6 @@ You can see a running version of the application at
 ## Current Status
 We are happy to announce that this project has been awarded a [grant from the Knight Foundation](http://www.knightfoundation.org/grants/201447979/), which means we get to keep working on it in 2014! Our primary goals this year are: simplifying the installation process, streamlining the code, reducing dependencies, and preparing the project for broader installation by a variety of organizations and governments.
 
-One of the major changes (that is now complete as of April 17, 2014) is the replacement of MongoDB with Postgres. The main reason for that change was to reduce dependencies, but another important reason is that we wanted to upgrade the app to Rails 4, but Mongoid didn't (maybe still doesn't?) support Rails 4.
-
-The next change underway in April is replacing Elasticsearch with the full-text search capabilities in Postgres.
-
-Because the project will be undergoing these major changes, we don't recommend using it for a production app just yet, but please feel free to try it out and provide feedback!
-
 ## Data Schema
 If you would like to try out the current version of the project that uses Postgres, please read the Wiki article about [Populating the Postgres DB from a JSON file](https://github.com/codeforamerica/ohana-api/wiki/Populating-the-Postgres-database-from-a-JSON-file). That article documents the current schema and data dictionary, but please note that this will be in flux as we are working with various interested parties to define a [Human Services Data Specification](https://github.com/codeforamerica/OpenReferral).
 
@@ -54,7 +48,6 @@ You can also try it from the Rails console, mimicking how the API would do it wh
 * Rails version 4.0.4
 * Postgres
 * Redis
-* ElasticSearch <=1.0.1
 * API framework: Grape
 * Testing Frameworks: RSpec, Factory Girl, Capybara
 
@@ -111,25 +104,6 @@ Follow the Homebrew instructions if you want Redis to start automatically every
 
 See the Download page on Redis.io for steps to install on other systems: [http://redis.io/download](http://redis.io/download)
 
-#### ElasticSearch <=1.0.1
-**OS X**
-
-Please make sure you are using Elasticsearch 1.0.1 or lower. This app is currently not compatible with Elasticsearch 1.1.0.
-
-On OS X, the easiest way to install ElasticSearch is with Homebrew:
-
-    brew install https://raw.github.com/Homebrew/homebrew/9b8103f6fb570dc3a5ce5b5b84cb76fb6915cace/Library/Formula/elasticsearch.rb
-
-Follow the Homebrew instructions to launch ElasticSearch.
-
-If you already had 1.0.1 and then upgraded to 1.1.0, you can switch back to 1.0.1 with this command:
-
-    brew switch elasticsearch 1.0.1
-
-**Other**
-
-Visit the Download page on elasticsearch.org for steps to install on other systems: [http://www.elasticsearch.org/download/](http://www.elasticsearch.org/download/)
-
 ### Clone the app on your local machine:
 
 From the Terminal, navigate to the directory into which you'd like to create a copy of the Ohana API source code. For instance, on OS X `cd ~` will place you in your home directory. Next download this repository into your working directory with:
@@ -149,7 +123,7 @@ If you get a `permission denied` message, set the correct permissions:
 
 then run `script/bootstrap` again.
 
-### Set up the environment variables
+### Set up the environment variables & customizable settings
 Inside the `config` folder, you will find a file named `application.example.yml`. Rename it to `application.yml` and double check that it is in your `.gitignore` file (it should be by default).
 
 In `config/application.yml`, set the following environment variables so that the tests can pass, and so you can run the [Ohana API Admin](https://github.com/codeforamerica/ohana-api-admin) app locally:
@@ -162,6 +136,10 @@ In `config/application.yml`, set the following environment variables so that the
 
     rake secret
 
+Inside the `config` folder, you will also find a file called `settings.yml`.
+In that file, there are 3 variables you can customize. Please read through the
+instructions in that file carefully.
+
 ### Run the app
 Start the app locally on port 8080:
 

diff --git a/app/api/api.rb b/app/api/api.rb
@@ -39,6 +39,12 @@ def valid_api_token?
       }.to_json, 404)
     end
 
+    rescue_from ActiveRecord::SerializationTypeMismatch do |e|
+      rack_response({
+        'message' => e.message
+      }.to_json, 400)
+    end
+
     rescue_from ActiveRecord::RecordInvalid do |e|
       if e.record.errors.first.first == :accessibility
         message = 'Please enter a valid value for Accessibility'

diff --git a/app/api/link_header.rb b/app/api/link_header.rb
@@ -1,14 +1,14 @@
 module LinkHeader
-  # Set Link header for pagination info
+  # Set Link and pagination-related HTTP headers
   #
-  # @param coll [Tire::Results::Collection or
-  # @return [Sawyer::Resource]
+  # @param coll ActiveRecord::Relation
+  # @return various pagination-related HTTP Headers
   def set_link_header(coll)
     params = request.params.except(:route_info, :method, :search_format)
 
     pages = pages(coll)
 
-    links = links(pages,params)
+    links = links(pages, params)
 
     links.delete_if { |v| v.blank? }
 
@@ -17,7 +17,7 @@ def set_link_header(coll)
     header 'X-Total-Pages', "#{coll.total_pages}"
     header 'X-Current-Page', "#{coll.current_page}" unless coll.empty?
     header 'X-Next-Page', "#{coll.next_page}" unless coll.next_page.nil?
-    header 'X-Previous-Page', "#{pages[:prev]}" unless coll.previous_page.nil?
+    header 'X-Previous-Page', "#{pages[:prev]}" unless coll.prev_page.nil?
   end
 
   def url
@@ -39,7 +39,7 @@ def pages(coll)
     pages
   end
 
-  def links(pages,params)
+  def links(pages, params)
     pages.map do |k, v|
       new_params = params.merge(page: v)
       unless new_params[:page].blank?