course.xml

<course url_name='2012_Fall' org='10gen' course='M102' display_name='MongoDB for DBAs' start='2012-10-22T00:00' end='2012-12-25T00:00' enrollment_start='2012-10-01T00:00' enrollment_end='2012-12-02T00:00'>
<chapter start='2012-10-22T00:00' display_name='week1'>
<sequential graded='false' display_name='Course Overview' chapter_name='week1'>
<vertical>
<video display_name='Lecture' youtube='1.0:0euni8AdT1c'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Concepts and Philosophy' chapter_name='week1'>
<vertical>
<video display_name='Lecture' youtube='1.0:QUuYrMSPhL4'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Installing on Unix' chapter_name='week1'>
<vertical>
<video display_name='Lecture' youtube='1.0:6VFukRETCTg'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Installing MongoDB on Windows' chapter_name='week1'>
<vertical>
<video display_name='Lecture' youtube='1.0:hX5louVryOQ'/>
</vertical>
</sequential>
<sequential graded='false' display_name='JSON Types' chapter_name='week1' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:gKOcqNHmc4Q'/>
</vertical>
<problem showanswer='attempted' url_name='JSON_Types_problem' rerandomize='never' display_name='Quiz: JSON Types'>
How many data types are there in JSON?<multiplechoiceresponse direction='vertical'>
<choicegroup type='MultipleChoice'>
<choice location='bottom' correct='false'><text>1 (just strings)</text></choice>
<choice location='bottom' correct='false'><text>2</text></choice>
<choice location='bottom' correct='false'><text>4</text></choice>
<choice location='bottom' correct='true'><text>6</text></choice>
</choicegroup>
</multiplechoiceresponse>
</problem>
<vertical start='Oct 30 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:e2lfdSDENjY'/>
</vertical>
</sequential>
<sequential graded='false' display_name='JSON Syntax' chapter_name='week1' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:a3jWXcmjrWM'/>
</vertical>
<problem showanswer='attempted' url_name='JSON_Syntax_problem' rerandomize='never' display_name='Quiz: JSON Syntax'>
<p>What is the corresponding JSON for the following XML document?</p>

<pre>
&lt;person&gt;
  &lt;name&gt;John&lt;/name&gt;
  &lt;age&gt;25&lt;/age&gt;
  &lt;address&gt;
    &lt;city&gt;New York&lt;/city&gt;
    &lt;postalCode&gt;10021&lt;/postalCode&gt;
  &lt;/address&gt;
  &lt;phones&gt;
    &lt;phone type="home"&gt;212-555-1234&lt;/phone&gt;
    &lt;phone type="mobile"&gt;646-555-1234&lt;/phone&gt;
  &lt;/phones&gt;
&lt;/person&gt;
</pre><customresponse>
<textbox rows='4' cols='50' correct_answer='{&#10;  "name" : "John",&#10;  "age" : 25,&#10;  "address" : {&#10;    "city" : "New York",&#10;    "postalCode" : "10021"&#10;  },&#10;  "phones" : [&#10;    {"phone":"212-555-1234", "type" : "home"},&#10;    {"phone":"646-555-1234", "type" : "mobile"}&#10;  ]&#10;}'> </textbox>
<answer type='loncapa/python'><![CDATA[
import demjson
correct = ['correct']
messages = ['']
try:
  r = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  r = '0'
quiz_answer = '''{
  "name" : "John",
  "age" : 25,
  "address" : {
    "city" : "New York",
    "postalCode" : "10021"
  },
  "phones" : [
    {"phone":"212-555-1234", "type" : "home"},
    {"phone":"646-555-1234", "type" : "mobile"}
  ]
}'''
quiz_answer_json = {}
try:
  quiz_answer_json = demjson.decode(quiz_answer)
except demjson.JSONDecodeError as err:
  messages[0] = 'Quiz Answer failed to parse' + str(err)
response_json = {}
try:
  response_json = demjson.decode(r)
except demjson.JSONDecodeError as err:
  messages[0] = 'Response failed to parse' + str(err)
if not(response_json == quiz_answer_json):
  correct[0] = 'incorrect'
]]></answer>
</customresponse>
</problem>
<vertical start='Oct 30 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:RoUvl8c_3Sc'/>
</vertical>
</sequential>
<sequential graded='false' display_name='JSON Syntax 2' chapter_name='week1' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:dxd1BnBqQk8'/>
</vertical>
<problem showanswer='attempted' url_name='JSON_Syntax_2_problem' rerandomize='never' display_name='Quiz: JSON Syntax 2'>
For the following XML, Is the corresponding JSON example legal json?

<pre>
&lt;things&gt;
  &lt;hat&gt;one&lt;/hat&gt;
  &lt;coat&gt;z&lt;/coat&gt;
  &lt;hat&gt;two&lt;/hat&gt;
&lt;/things&gt;
</pre>

<pre>
{
  "hat" : "one",
  "coat" : "z",
  "hat" : "two"
}
</pre>
<multiplechoiceresponse direction='vertical'>
<choicegroup type='MultipleChoice'>
<choice location='bottom' correct='false'><text>yes</text></choice>
<choice location='bottom' correct='false'><text>no</text></choice>
<choice location='bottom' correct='true'><text>maybe</text></choice>
</choicegroup>
</multiplechoiceresponse>
</problem>
<vertical start='Oct 30 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:eypM73cF0dI'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Introduction to BSON' chapter_name='week1'>
<vertical>
<video display_name='Lecture' youtube='1.0:CEH0kNxVyMw'/>
</vertical>
</sequential>
<sequential graded='false' display_name='What is the Mongo Shell' chapter_name='week1'>
<vertical>
<video display_name='Lecture' youtube='1.0:W4o8HJyL7_8'/>
</vertical>
</sequential>
<sequential graded='false' display_name='What is JavaScript' chapter_name='week1'>
<vertical>
<video display_name='Lecture' youtube='1.0:n1Hld15pvZY'/>
</vertical>
</sequential>
<sequential graded='false' display_name='What is JavaScript 2' chapter_name='week1'>
<vertical>
<video display_name='Lecture' youtube='1.0:n79mmmQxoNE'/>
</vertical>
</sequential>
<sequential graded='false' display_name='mongoimport' chapter_name='week1' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:ZWnHhKiApVU'/>
</vertical>
<problem showanswer='attempted' url_name='mongoimport_problem' rerandomize='never' display_name='Quiz: mongoimport'>
The mongoimport utility can import what types of data?<choiceresponse direction='vertical'>
<checkboxgroup type='MultipleChoice'>
<choice correct='true'>
<startouttext/>CSV (comma separated values)<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>TSV (tab separated values)<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>XML<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>JSON<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>BSON<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>plain text<endouttext/>
</choice>
</checkboxgroup>
</choiceresponse>
</problem>
<vertical start='Oct 30 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:8_6S7E9r9DQ'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Introduction to the MongoDB Shell' chapter_name='week1'>
<vertical>
<video display_name='Lecture' youtube='1.0:P1yLtecuLM8'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Shell: Queries' chapter_name='week1' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:Mjcd5PAdG-s'/>
</vertical>
<problem showanswer='attempted' url_name='Shell_Queries_problem' rerandomize='never' display_name='Quiz: Shell: Queries'>
We have sample documents in our products collection such as:

<pre>
{
  name: "AC1 Case Green",
  color: "green",
  price: 12.00,
  for: "ac1",
  type: ["accessory", "case"],
  available: true
}
</pre>

How would we query in the shell for all products that are cases for an ac9 phone?  That is, where <i>type</i> contains the value "case" and <i>for</i> equals "ac9"?

<p><i>Please use field names in the order they appear in the question and use double quotes around values.</i></p><customresponse>
<textbox rows='4' cols='50' correct_answer='db.products.find({type: "case", for: "ac9"})'> </textbox>

<answer type='loncapa/python'><![CDATA[
try:
  response = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  response = '0'
answer = '''db.products.find({type: "case", for: "ac9"})'''

# Filter out newlines, carriage returns, and semicolons.
import demjson
import re
response = re.sub(r'[\s;]+', '', response)
answer = re.sub(r'[\s;]+', '', answer)

# Create a JSON string representation of the input string.
def json_encode(text):
  import demjson
  import re

  # The global scope is not carried here, so we must internally define functions.
  class Match(object):
    start = 0
    end = 0
    def __init__(self, start, end):
      self.start = start
      self.end = end

  # Extract embedded JSON.
  def capture(text, start):
    pos = start
    match = Match(-1, -1)
    brace_count = 0
    while pos < len(text):
      if text[pos] == '{':
        # Start checking for a JSON object.
        if brace_count == 0:
          match.start = pos
        brace_count = brace_count + 1
      elif text[pos] == '}':
        # Check for a complete JSON object.
        brace_count = brace_count - 1
        if brace_count == 0:
          match.end = pos + 1
          match.str = text[match.start : match.end]
          break
      pos = pos + 1
    if match.start == -1 or match.end == -1:
      return None
    return match

  # Now perform actual answer-to-json conversion.
  text_out = '{'
  key_index_out = 0
  # Start parsing and splitting the string.
  pos = 0
  match = capture(text, pos)
  while match != None:
    # There was some text between matches, capture it.
    if pos != match.start:
      text_out += '"s' + str(key_index_out) + '" : "' +         re.sub('"', '', text[pos:match.start]) + '", '
      key_index_out = key_index_out + 1
    text_out += '"s' + str(key_index_out) + '" : ' + match.str + ', '
    key_index_out = key_index_out + 1
    pos = match.end
    match = capture(text, pos)
  # Get any text after the last match.
  text_out += '"s' + str(key_index_out) + '" : "' +     re.sub('"', '', text[pos:]) + '", '
  text_out += '}'
  return str(text_out)

correct = ['incorrect']
messages = ['']

try:
  response_json = demjson.decode(json_encode(response))
  answer_json = demjson.decode(json_encode(answer))
  if response_json == answer_json:
    correct[0] = 'correct'
except demjson.JSONDecodeError as err:
  messages[0] = "Response failed to parse " + str(err)
]]>
</answer>
</customresponse>
</problem>
<vertical start='Oct 30 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:fYcPCRCPvDg'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Shell:  Sorting' chapter_name='week1'>
<vertical>
<video display_name='Lecture' youtube='1.0:Ezx1lXaY7jE'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Shell: Cursors and Shell Help' chapter_name='week1' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:DRs4_hBpPlI'/>
</vertical>
<problem showanswer='attempted' url_name='Shell_Cursors_and_Shell_Help_problem' rerandomize='never' display_name='Quiz: Shell: Cursors and Shell Help'>
What would you type to see the help/man text for mongo cursors in the mongo administrative shell?<multiplechoiceresponse direction='vertical'>
<choicegroup type='MultipleChoice'>
<choice location='bottom' correct='false'><text>help</text></choice>
<choice location='bottom' correct='false'><text>db.help()</text></choice>
<choice location='bottom' correct='false'><text>db.products.help()</text></choice>
<choice location='bottom' correct='false'><text>help misc</text></choice>
<choice location='bottom' correct='true'><text>db.products.find().help()</text></choice>
</choicegroup>
</multiplechoiceresponse>
</problem>
<vertical start='Oct 30 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:3KwsHVHBM9w'/>
</vertical>
</sequential>
<sequential graded='true' display_name='Homework 1.1' chapter_name='week1' format='Homework' due='Oct 30 23:00 EST 2012'>
<problem showanswer='due' attempts='3' end='2012-12-25T00:00' url_name='Homework_1_1_problem' rerandomize='never' display_name='Homework: Homework 1.1'>
Download and install MongoDB from www.mongodb.org.  Then run the database as a single server instance on your PC (that is, run the mongod binary).  Then, run the administrative shell.  From the shell prompt type 
<pre>
 db.isMaster().maxBsonObjectSize
</pre>
at the ">" prompt. What do you get as a result?<customresponse>
<textbox rows='4' cols='50' correct_answer="16777216"> </textbox>
<answer type='loncapa/python'><![CDATA[
import re
correct = ['correct']
try:
  r = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  r = '0'
response = re.sub(r'\s+', '', r)
quiz_answer = '''16777216'''
quiz_answer = re.sub(r'\s+', '', quiz_answer)
if not(response == quiz_answer):
  correct[0] = 'incorrect'
  print 'response = ', response
  print 'quiz_answer = ', quiz_answer
]]>
</answer>
</customresponse>
</problem>
</sequential>
<sequential graded='true' display_name='Homework 1.2' chapter_name='week1' format='Homework' due='Oct 30 23:00 EST 2012'>
<problem showanswer='due' attempts='3' end='2012-12-25T00:00' url_name='Homework_1_2_problem' rerandomize='never' display_name='Homework: Homework 1.2'>
Download the file <a href="/static/handouts/products.json">products.json</a> from education.10gen.com.
Take a look at its content.
Now, import its contents into MongoDB, into a database called "pcat" and a collection called "products".  Use the mongoimport utility to do this.
When done, run this query in the mongo shell:
<pre>
db.products.find({type:"case"}).count()
</pre>
What's the result?<customresponse>
<textbox rows='4' cols='50' correct_answer="3"> </textbox>
<answer type='loncapa/python'><![CDATA[
import re
correct = ['correct']
try:
  r = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  r = '0'
response = re.sub(r'\s+', '', r)
quiz_answer = '''3'''
quiz_answer = re.sub(r'\s+', '', quiz_answer)
if not(response == quiz_answer):
  correct[0] = 'incorrect'
  print 'response = ', response
  print 'quiz_answer = ', quiz_answer
]]>
</answer>
</customresponse>
</problem>
</sequential>
<sequential graded='true' display_name='Homework 1.3' chapter_name='week1' format='Homework' due='Oct 30 23:00 EST 2012'>
<problem showanswer='due' attempts='3' end='2012-12-25T00:00' url_name='Homework_1_3_problem' rerandomize='never' display_name='Homework: Homework 1.3'>
At this point you should have pcat.products loaded from the previous step. You can confirm this by running in the shell:
<pre>
> db.products.find()
> // or:
> db.products.count()
> // should print out "11"
</pre>
Now, what query would you run to get all the products where brand equals the string “ACME”?<customresponse>
<textbox rows='4' cols='50' correct_answer='db.products.find({brand:"ACME"})'> </textbox>

<answer type='loncapa/python'><![CDATA[
try:
  response = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  response = '0'
answer = '''db.products.find({brand:"ACME"})'''

# Filter out newlines, carriage returns, and semicolons.
import demjson
import re
response = re.sub(r'[\s;]+', '', response)
answer = re.sub(r'[\s;]+', '', answer)

# Create a JSON string representation of the input string.
def json_encode(text):
  import demjson
  import re

  # The global scope is not carried here, so we must internally define functions.
  class Match(object):
    start = 0
    end = 0
    def __init__(self, start, end):
      self.start = start
      self.end = end

  # Extract embedded JSON.
  def capture(text, start):
    pos = start
    match = Match(-1, -1)
    brace_count = 0
    while pos < len(text):
      if text[pos] == '{':
        # Start checking for a JSON object.
        if brace_count == 0:
          match.start = pos
        brace_count = brace_count + 1
      elif text[pos] == '}':
        # Check for a complete JSON object.
        brace_count = brace_count - 1
        if brace_count == 0:
          match.end = pos + 1
          match.str = text[match.start : match.end]
          break
      pos = pos + 1
    if match.start == -1 or match.end == -1:
      return None
    return match

  # Now perform actual answer-to-json conversion.
  text_out = '{'
  key_index_out = 0
  # Start parsing and splitting the string.
  pos = 0
  match = capture(text, pos)
  while match != None:
    # There was some text between matches, capture it.
    if pos != match.start:
      text_out += '"s' + str(key_index_out) + '" : "' +         re.sub('"', '', text[pos:match.start]) + '", '
      key_index_out = key_index_out + 1
    text_out += '"s' + str(key_index_out) + '" : ' + match.str + ', '
    key_index_out = key_index_out + 1
    pos = match.end
    match = capture(text, pos)
  # Get any text after the last match.
  text_out += '"s' + str(key_index_out) + '" : "' +     re.sub('"', '', text[pos:]) + '", '
  text_out += '}'
  return str(text_out)

correct = ['incorrect']
messages = ['']

try:
  response_json = demjson.decode(json_encode(response))
  answer_json = demjson.decode(json_encode(answer))
  if response_json == answer_json:
    correct[0] = 'correct'
except demjson.JSONDecodeError as err:
  messages[0] = "Response failed to parse " + str(err)
]]>
</answer>
</customresponse>
</problem>
</sequential>
<sequential graded='true' display_name='Homework 1.4' chapter_name='week1' format='Homework' due='Oct 30 23:00 EST 2012'>
<problem showanswer='due' attempts='3' end='2012-12-25T00:00' url_name='Homework_1_4_problem' rerandomize='never' display_name='Homework: Homework 1.4'>
How would you print out, in the shell, the name of all the products without extraneous characters or braces, sorted alphabetically, ascending?  (Check all that would apply.)<choiceresponse direction='vertical'>
<checkboxgroup type='MultipleChoice'>
<choice correct='false'>
<startouttext/>db.products.find({},{name:1,_id:0}).sort({name:1})<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>var c = db.products.find({},{name:1,_id:0}).sort({name:1}); 
while( c.hasNext() ) 
    print( c.next().name);
<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>var c = db.products.find({}).sort({name:1});
c.forEach( function(doc){ print(doc.name) } );<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>var c = db.products.find({}).sort({name:-1});
while( c.hasNext() ) 
    print( c.next().name);<endouttext/>
</choice>
</checkboxgroup>
</choiceresponse>
</problem>
</sequential>
</chapter>
<chapter start='2012-10-29T00:00' display_name='week2'>
<sequential graded='false' display_name='Inserting Data' chapter_name='week2' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:MWH-_nJgebw'/>
</vertical>
<problem showanswer='attempted' url_name='Inserting_Data_problem' rerandomize='never' display_name='Quiz: Inserting Data'>
In the mongo shell, what would you type to insert the document {x:3,y:4} into the "temperature" collection for the current database?<customresponse>
<textbox rows='4' cols='50' correct_answer="db.temperature.insert({x:3,y:4})"> </textbox>

<answer type='loncapa/python'><![CDATA[
try:
  response = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  response = '0'
answer = '''db.temperature.insert({x:3,y:4})'''

# Filter out newlines, carriage returns, and semicolons.
import demjson
import re
response = re.sub(r'[\s;]+', '', response)
answer = re.sub(r'[\s;]+', '', answer)

# Create a JSON string representation of the input string.
def json_encode(text):
  import demjson
  import re

  # The global scope is not carried here, so we must internally define functions.
  class Match(object):
    start = 0
    end = 0
    def __init__(self, start, end):
      self.start = start
      self.end = end

  # Extract embedded JSON.
  def capture(text, start):
    pos = start
    match = Match(-1, -1)
    brace_count = 0
    while pos < len(text):
      if text[pos] == '{':
        # Start checking for a JSON object.
        if brace_count == 0:
          match.start = pos
        brace_count = brace_count + 1
      elif text[pos] == '}':
        # Check for a complete JSON object.
        brace_count = brace_count - 1
        if brace_count == 0:
          match.end = pos + 1
          match.str = text[match.start : match.end]
          break
      pos = pos + 1
    if match.start == -1 or match.end == -1:
      return None
    return match

  # Now perform actual answer-to-json conversion.
  text_out = '{'
  key_index_out = 0
  # Start parsing and splitting the string.
  pos = 0
  match = capture(text, pos)
  while match != None:
    # There was some text between matches, capture it.
    if pos != match.start:
      text_out += '"s' + str(key_index_out) + '" : "' +         re.sub('"', '', text[pos:match.start]) + '", '
      key_index_out = key_index_out + 1
    text_out += '"s' + str(key_index_out) + '" : ' + match.str + ', '
    key_index_out = key_index_out + 1
    pos = match.end
    match = capture(text, pos)
  # Get any text after the last match.
  text_out += '"s' + str(key_index_out) + '" : "' +     re.sub('"', '', text[pos:]) + '", '
  text_out += '}'
  return str(text_out)

correct = ['incorrect']
messages = ['']

try:
  response_json = demjson.decode(json_encode(response))
  answer_json = demjson.decode(json_encode(answer))
  if response_json == answer_json:
    correct[0] = 'correct'
except demjson.JSONDecodeError as err:
  messages[0] = "Response failed to parse " + str(err)
]]>
</answer>
</customresponse>
</problem>
<vertical start='Nov 8 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:Yl6L0jytr4o'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Updating Documents' chapter_name='week2' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:0tGJGxqvO0c'/>
</vertical>
<problem showanswer='attempted' url_name='Updating_Documents_problem' rerandomize='never' display_name='Quiz: Updating Documents'>
Suppose we have a collection "cars" with one document preexisting:
<pre>
{ "_id" : 100, "name" : "GTO", 
  "year" : 1969, "color" : "red" }
</pre>
Which of the following statements would set "available" to 1?<choiceresponse direction='vertical'>
<checkboxgroup type='MultipleChoice'>
<choice correct='true'>
<startouttext/>db.cars.update({_id:100},{$set:{available:1}})<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>db.cars.update({_id:100},{$inc:{available:1}})<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>db.cars.update({},{$inc:{available:1,year:1970}})<endouttext/>
</choice>
</checkboxgroup>
</choiceresponse>
</problem>
<vertical start='Nov 8 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:sRjVWZ6olqI'/>
</vertical>
</sequential>
<sequential graded='false' display_name='MongoDB Documents Quiz' chapter_name='week2' format='Quiz'>
<problem showanswer='attempted' url_name='MongoDB_Documents_Quiz_problem' rerandomize='never' display_name='Quiz: MongoDB Documents Quiz'>
Which of the following are true?<choiceresponse direction='vertical'>
<checkboxgroup type='MultipleChoice'>
<choice correct='true'>
<startouttext/>MongoDB can store large things; for example a 1 terabyte video.<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>On an update of a document in mongodb, the document always physically moves in its position in the data file(s).<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>On average, mongodb documents have about the same number of fields as one would have in a relational database table for the same problem.<endouttext/>
</choice>
</checkboxgroup>
</choiceresponse>
</problem>
</sequential>
<sequential graded='false' display_name='Removing documents' chapter_name='week2' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:qj7n4TRVY1M'/>
</vertical>
<problem showanswer='attempted' url_name='Removing_documents_problem' rerandomize='never' display_name='Quiz: Removing documents'>
Suppose we have documents in a 'users' collection of the form:
<pre>
{
 _id : ObjectId("50897dbb9b96971d287202a9"),
 name : "Jane",
 likes : [ "tennis", "golf" ],
 registered : false,
 addr : {
   city : "Lyon",
   country : "France" 
 } 
}
</pre>
How would we, in the mongo shell, delete all documents in the collection where city = "Lyon" and registered is false?<customresponse>
<textbox rows='4' cols='50' correct_answer='db.users.remove({"addr.city":"Lyon",&#10;registered:false})'> </textbox>

<answer type='loncapa/python'><![CDATA[
try:
  response = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  response = '0'
answer = '''db.users.remove({"addr.city":"Lyon",
registered:false})'''

# Filter out newlines, carriage returns, and semicolons.
import demjson
import re
response = re.sub(r'[\s;]+', '', response)
answer = re.sub(r'[\s;]+', '', answer)

# Create a JSON string representation of the input string.
def json_encode(text):
  import demjson
  import re

  # The global scope is not carried here, so we must internally define functions.
  class Match(object):
    start = 0
    end = 0
    def __init__(self, start, end):
      self.start = start
      self.end = end

  # Extract embedded JSON.
  def capture(text, start):
    pos = start
    match = Match(-1, -1)
    brace_count = 0
    while pos < len(text):
      if text[pos] == '{':
        # Start checking for a JSON object.
        if brace_count == 0:
          match.start = pos
        brace_count = brace_count + 1
      elif text[pos] == '}':
        # Check for a complete JSON object.
        brace_count = brace_count - 1
        if brace_count == 0:
          match.end = pos + 1
          match.str = text[match.start : match.end]
          break
      pos = pos + 1
    if match.start == -1 or match.end == -1:
      return None
    return match

  # Now perform actual answer-to-json conversion.
  text_out = '{'
  key_index_out = 0
  # Start parsing and splitting the string.
  pos = 0
  match = capture(text, pos)
  while match != None:
    # There was some text between matches, capture it.
    if pos != match.start:
      text_out += '"s' + str(key_index_out) + '" : "' +         re.sub('"', '', text[pos:match.start]) + '", '
      key_index_out = key_index_out + 1
    text_out += '"s' + str(key_index_out) + '" : ' + match.str + ', '
    key_index_out = key_index_out + 1
    pos = match.end
    match = capture(text, pos)
  # Get any text after the last match.
  text_out += '"s' + str(key_index_out) + '" : "' +     re.sub('"', '', text[pos:]) + '", '
  text_out += '}'
  return str(text_out)

correct = ['incorrect']
messages = ['']

try:
  response_json = demjson.decode(json_encode(response))
  answer_json = demjson.decode(json_encode(answer))
  if response_json == answer_json:
    correct[0] = 'correct'
except demjson.JSONDecodeError as err:
  messages[0] = "Response failed to parse " + str(err)
]]>
</answer>
</customresponse>
</problem>
<vertical start='Nov 8 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:WOzekUwo1vE'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Updating Part 2' chapter_name='week2' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:pgu3nta2iLM'/>
</vertical>
<problem showanswer='attempted' url_name='Updating_Part_2_problem' rerandomize='never' display_name='Quiz: Updating Part 2'>
Suppose we have documents in a 'users' collection of the form:
<pre>
{
 _id : "Jane",
 likes : [ "tennis", "golf" ],
 registered : false,
 addr : {
   city : "Lyon",
   country : "France" 
 } 
}
</pre>
How would we, in the mongo shell, add that this user likes "football"?<customresponse>
<textbox rows='4' cols='50' correct_answer='db.users.update({_id:"Jane"},{$addToSet:{likes:"football"}}, true)'> </textbox>

<answer type='loncapa/python'><![CDATA[
try:
  response = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  response = '0'
answer = '''db.users.update({_id:"Jane"},{$addToSet:{likes:"football"}}, true)'''

# Filter out newlines, carriage returns, and semicolons.
import demjson
import re
response = re.sub(r'[\s;]+', '', response)
answer = re.sub(r'[\s;]+', '', answer)

# Create a JSON string representation of the input string.
def json_encode(text):
  import demjson
  import re

  # The global scope is not carried here, so we must internally define functions.
  class Match(object):
    start = 0
    end = 0
    def __init__(self, start, end):
      self.start = start
      self.end = end

  # Extract embedded JSON.
  def capture(text, start):
    pos = start
    match = Match(-1, -1)
    brace_count = 0
    while pos < len(text):
      if text[pos] == '{':
        # Start checking for a JSON object.
        if brace_count == 0:
          match.start = pos
        brace_count = brace_count + 1
      elif text[pos] == '}':
        # Check for a complete JSON object.
        brace_count = brace_count - 1
        if brace_count == 0:
          match.end = pos + 1
          match.str = text[match.start : match.end]
          break
      pos = pos + 1
    if match.start == -1 or match.end == -1:
      return None
    return match

  # Now perform actual answer-to-json conversion.
  text_out = '{'
  key_index_out = 0
  # Start parsing and splitting the string.
  pos = 0
  match = capture(text, pos)
  while match != None:
    # There was some text between matches, capture it.
    if pos != match.start:
      text_out += '"s' + str(key_index_out) + '" : "' +         re.sub('"', '', text[pos:match.start]) + '", '
      key_index_out = key_index_out + 1
    text_out += '"s' + str(key_index_out) + '" : ' + match.str + ', '
    key_index_out = key_index_out + 1
    pos = match.end
    match = capture(text, pos)
  # Get any text after the last match.
  text_out += '"s' + str(key_index_out) + '" : "' +     re.sub('"', '', text[pos:]) + '", '
  text_out += '}'
  return str(text_out)

correct = ['incorrect']
messages = ['']

try:
  response_json = demjson.decode(json_encode(response))
  answer_json = demjson.decode(json_encode(answer))
  if response_json == answer_json:
    correct[0] = 'correct'
except demjson.JSONDecodeError as err:
  messages[0] = "Response failed to parse " + str(err)
]]>
</answer>
</customresponse>
</problem>
<vertical start='Nov 8 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:zKDmE590nlA'/>
</vertical>
</sequential>
<sequential graded='false' display_name='MongoDB Commands' chapter_name='week2'>
<vertical>
<video display_name='Lecture' youtube='1.0:nAfLu7vKiRI'/>
</vertical>
</sequential>
<sequential graded='false' display_name='MongoDB Commands 2' chapter_name='week2' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:0tnBD_hLUso'/>
</vertical>
<problem showanswer='attempted' url_name='MongoDB_Commands_2_problem' rerandomize='never' display_name='Quiz: MongoDB Commands 2'>
Which of these statements is true?
<choiceresponse direction='vertical'>
<checkboxgroup type='MultipleChoice'>
<choice correct='false'>
<startouttext/>MongoDB commands are for administrator use only.<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>In MongoDB collections are usually created implicitly.<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>In MongoDB indexes are usually created implicitly.<endouttext/>
</choice>
</checkboxgroup>
</choiceresponse>
</problem>
<vertical start='Nov 8 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:uktnp4cIjZk'/>
</vertical>
</sequential>
<sequential graded='true' display_name='Homework 2.1' chapter_name='week2' format='Homework' due='Nov 8 23:00 EST 2012'>
<problem showanswer='due' attempts='3' end='2012-12-25T00:00' url_name='Homework_2_1_problem' rerandomize='never' display_name='Homework: Homework 2.1'>
We will use the pcat.products collection from week 1.  So start with that; if not already set up, import it:
<pre>
mongoimport --db pcat -c products &lt; products.json
</pre>
You can find <a href="/static/handouts/products.json">products.json</a> on the education.10gen.com site. 
In the shell, if you type:
<pre>
db.products.count()
</pre>

should return 11.
<p>
Next, download <a href="/static/handouts/homework2.js">homework2.js</a> from the education.10gen.com site.  Run the shell with this script:</p>
<pre>
mongo --shell pcat homework2.js
</pre>

First, make a mini-backup of the collection before we start modifying it.  In the shell:

<pre>
b = db.products_bak; db.products.find().forEach( function(o){ b.insert(o) } )
 // check it worked: 
b.count()
// should print 11
</pre>

If you have any issues you can restore from "products_bak"; or, you can re-import with mongoimport.  (You would perhaps need in that situation to empty the collection first or drop it; see the --drop option on mongoimport --help.)

At the shell ">" prompt type: 
<pre>
homework.a()
</pre>

What is the output? (The above will check that products_bak is populated.)<customresponse>
<textbox rows='4' cols='50' correct_answer="3.05"> </textbox>
<answer type='loncapa/python'><![CDATA[
import re
correct = ['correct']
try:
  r = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  r = '0'
response = re.sub(r'\s+', '', r)
quiz_answer = '''3.05'''
quiz_answer = re.sub(r'\s+', '', quiz_answer)
if not(response == quiz_answer):
  correct[0] = 'incorrect'
  print 'response = ', response
  print 'quiz_answer = ', quiz_answer
]]>
</answer>
</customresponse>
</problem>
<vertical start='Nov 8 23:00 EST 2012' end='2012-12-25T00:00'>
<video display_name='Answer' youtube='1.0:r8szX-Q-3Hg'/>
</vertical>
</sequential>
<sequential graded='true' display_name='Homework 2.2' chapter_name='week2' format='Homework' due='Nov 8 23:00 EST 2012'>
<problem showanswer='due' attempts='3' end='2012-12-25T00:00' url_name='Homework_2_2_problem' rerandomize='never' display_name='Homework: Homework 2.2'>
Add a new product to the products collection of this form:
<pre>
{
	"_id" : "ac9",
	"name" : "AC9 Phone",
	"brand" : "ACME",
	"type" : "phone",
	"price" : 333,
	"warranty_years" : 0.25,
	"available" : true
}
</pre>
Note: in general because of the automatic line continuation in the shell, you can cut/paste in the above and shouldn't have to type it all out.  Just enclose it in the proper statement(s) to get it added.

Next, load into a shell variable the object corresponding to 
<pre>
_id : ObjectId("507d95d5719dbef170f15c00")
</pre>
<ul>
<li>Then change <code>term_years</code> to 3 for that document. (And save that to the database.)</li>
<li>Then change <code>over_rate</code> for <code>sms</code> in <code>limits</code> to 0.01 from 0.  Save that too.</li>
</ul>
At the shell prompt type: 
<pre>
homework.b()
</pre>
What is the output?<customresponse>
<textbox rows='4' cols='50' correct_answer="0.050.019031"> </textbox>
<answer type='loncapa/python'><![CDATA[
import re
correct = ['correct']
try:
  r = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  r = '0'
response = re.sub(r'\s+', '', r)
quiz_answer = '''0.050.019031'''
quiz_answer = re.sub(r'\s+', '', quiz_answer)
if not(response == quiz_answer):
  correct[0] = 'incorrect'
  print 'response = ', response
  print 'quiz_answer = ', quiz_answer
]]>
</answer>
</customresponse>
</problem>
<vertical start='Nov 8 23:00 EST 2012' end='2012-12-25T00:00'>
<video display_name='Answer' youtube='1.0:AKAxnShI8sU'/>
</vertical>
</sequential>
<sequential graded='true' display_name='Homework 2.3' chapter_name='week2' format='Homework' due='Nov 8 23:00 EST 2012'>
<problem showanswer='due' attempts='3' end='2012-12-25T00:00' url_name='Homework_2_3_problem' rerandomize='never' display_name='Homework: Homework 2.3'>
How many products have a voice limit?  (That is, have a voice field present in the limits array.)<customresponse>
<textbox rows='4' cols='50' correct_answer="3"> </textbox>
<answer type='loncapa/python'><![CDATA[
import re
correct = ['correct']
try:
  r = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  r = '0'
response = re.sub(r'\s+', '', r)
quiz_answer = '''3'''
quiz_answer = re.sub(r'\s+', '', quiz_answer)
if not(response == quiz_answer):
  correct[0] = 'incorrect'
  print 'response = ', response
  print 'quiz_answer = ', quiz_answer
]]>
</answer>
</customresponse>
</problem>
<vertical start='Nov 8 23:00 EST 2012' end='2012-12-25T00:00'>
<video display_name='Answer' youtube='1.0:qn0jv7PMO0U'/>
</vertical>
</sequential>
<sequential graded='true' display_name='Homework 2.4' chapter_name='week2' format='Homework' due='Nov 8 23:00 EST 2012'>
<problem showanswer='due' attempts='3' end='2012-12-25T00:00' url_name='Homework_2_4_problem' rerandomize='never' display_name='Homework: Homework 2.4'>
Create an index on the field <code>for</code>.

You might want to first run the following to get some context on what is present in that field in the documents of our collection:
<pre>
db.products.find({},{for:1})
</pre>

After creating the index, query products that work with an "ac3" phone; that is, "ac3" is present in the product's "for" field.
<ul>
<li>Q1: How many are there?</li>
<li>Q2: Run an explain plan on the above query.  How many records were scanned? </li>
<li>Q3: Was an index used?</li>
</ul>
<choiceresponse direction='vertical'>
<checkboxgroup type='MultipleChoice'>
<choice correct='false'>
<startouttext/>Q1: 0<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>Q1: 1<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>Q1: 3<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>Q1 : 4<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>Q2 : 1<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>Q2 : 4<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>Q2 : 5<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>Q2 : 12<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>Q3 : No<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>Q3 : Yes<endouttext/>
</choice>
</checkboxgroup>
</choiceresponse>
</problem>
<vertical start='Nov 8 23:00 EST 2012' end='2012-12-25T00:00'>
<video display_name='Answer' youtube='1.0:pIXPARhDfRE'/>
</vertical>
</sequential>
<sequential graded='true' display_name='Homework 2.5' chapter_name='week2' format='Homework' due='Nov 8 23:00 EST 2012'>
<problem showanswer='due' attempts='3' end='2012-12-25T00:00' url_name='Homework_2_5_problem' rerandomize='never' display_name='Homework: Homework 2.5'>
Referring back to 2.4 above, update those products (products that work with an "ac3" phone) and add 2 to the "price" of each of those items.

Then, at the shell prompt type: 
<pre>
homework.c()
</pre>

What is the output?<customresponse>
<textbox rows='4' cols='50' correct_answer="89.5954.5"> </textbox>
<answer type='loncapa/python'><![CDATA[
import re
correct = ['correct']
try:
  r = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  r = '0'
response = re.sub(r'\s+', '', r)
quiz_answer = '''89.5954.5'''
quiz_answer = re.sub(r'\s+', '', quiz_answer)
if not(response == quiz_answer):
  correct[0] = 'incorrect'
  print 'response = ', response
  print 'quiz_answer = ', quiz_answer
]]>
</answer>
</customresponse>
</problem>
<vertical start='Nov 8 23:00 EST 2012' end='2012-12-25T00:00'>
<video display_name='Answer' youtube='1.0:A5_7rx3GNs4'/>
</vertical>
</sequential>
</chapter>
<chapter start='2012-11-08T00:00' display_name='week3'>
<sequential graded='false' display_name='Week3 Intro' chapter_name='week3'>
<vertical>
<video display_name='Lecture' youtube='1.0:0e02eUHMSUw'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Schema Design' chapter_name='week3' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:ufiFaY995GQ'/>
</vertical>
<problem showanswer='attempted' url_name='Schema_Design_problem' rerandomize='never' display_name='Quiz: Schema Design'>
Imagine we are building an order processing system.  Customers can have many orders; each order can have many line items.  Which schema is most typical in MongoDB?<multiplechoiceresponse direction='vertical'>
<choicegroup type='MultipleChoice'>
<choice location='bottom' correct='false'><text>customers collection, orders collection, order_line_items collection</text></choice>
<choice location='bottom' correct='true'><text>customers collection, orders collection</text></choice>
<choice location='bottom' correct='false'><text>orders collection only</text></choice>
</choicegroup>
</multiplechoiceresponse>
</problem>
<vertical start='Nov 19 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:FHQszgGukfY'/>
</vertical>
</sequential>
<sequential graded='false' display_name='The Aggregation Framework 1' chapter_name='week3'>
<vertical>
<video display_name='Lecture' youtube='1.0:OOciY22Eqpc'/>
</vertical>
</sequential>
<sequential graded='false' display_name='The Aggregation Framework 2' chapter_name='week3' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:5ApeWrsjOJY'/>
</vertical>
<problem showanswer='attempted' url_name='The_Aggregation_Framework_2_problem' rerandomize='never' display_name='Quiz: The Aggregation Framework 2'>
Suppose we have postal code (zip code) data of the form:
<pre>{
  "_id": "10280",
  "city": "NEW YORK",
  "state": "NY",
  "pop": 5574,
  "loc": [
    -74.016323,
    40.710537
  ]
}</pre>
in a collection postal_codes. (In the example above '10280' is a U.S. postal code / zip code.)  We want to find which cities have the most zip codes, and print the top 5 cities.  We want to handle the fact that some states have a city with the same name as another state and treat the cities as different in that case.   We run:
<pre>
&gt; db.postal_codes.aggregate( [ 
 {$group:{_id:{state:"$state",city:"$city"},
    n:{$sum:1}}} , 
 {$sort:{n:-1}} , 
 {$limit:5} ] 
)</pre>
Which of the following is true?<choiceresponse direction='vertical'>
<checkboxgroup type='MultipleChoice'>
<choice correct='true'>
<startouttext/>In the above {city:”$city”,state:"$state"} would work fine instead of {state:"$state",city:"$city"}, albeit with a slightly different output presentation.<endouttext/>
</choice>
<choice correct='true'>
<startouttext/> {x:"$state",y:"$city"} would work fine albeit with a slightly less clear output presentation.<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>To get the states with the most postal codes we would simply do {state:"$state"} instead of {state:"$state",city:"$city"}.<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>If our documents are not consistent about the capitalization of the city names, we need to do something special AND there is a way to do that with the aggregation framework.<endouttext/>
</choice>
</checkboxgroup>
</choiceresponse>
</problem>
<vertical start='Nov 19 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:LhmYVl6VkSs'/>
</vertical>
</sequential>
<sequential graded='false' display_name='More $ operators' chapter_name='week3' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:StPhSsIkOh0'/>
</vertical>
<problem showanswer='attempted' url_name='More_operators_problem' rerandomize='never' display_name='Quiz: More $ operators'>
Suppose we have documents in a collection of the form 
<pre>{ _id : …, x : [
  { a : …, b : … },
  { a : …, b : … },
  { a : …, b : …, c : ... }, ...
] }</pre>
We want to find all the top level documents which have a subobject in their x array where for that subobject a == 1 and b == 3.  Which of these queries does that?  Check all that are correct.<choiceresponse direction='vertical'>
<checkboxgroup type='MultipleChoice'>
<choice correct='false'>
<startouttext/>db.things.find( { “x.a” : 1, “x.b” : 3 } )<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>db.things.find( { x : { a : 1, b : 3 } } )<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>db.things.find( { x : { $elemMatch : { a:1,b:3} } } )
<endouttext/>
</choice>
</checkboxgroup>
</choiceresponse>
</problem>
<vertical start='Nov 19 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:nxoM7vRQ7h8'/>
</vertical>
</sequential>
<sequential graded='false' display_name='The findAndModify Command' chapter_name='week3' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:1EQ31ggRcJc'/>
</vertical>
<problem showanswer='attempted' url_name='The_findAndModify_Command_problem' rerandomize='never' display_name='Quiz: The findAndModify Command'>
Which of these statements are true?<choiceresponse direction='vertical'>
<checkboxgroup type='MultipleChoice'>
<choice correct='true'>
<startouttext/>findAndModify is atomic.<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>Update operations which update a single document are atomic.<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>Usually, update is faster than findAndModify.<endouttext/>
</choice>
</checkboxgroup>
</choiceresponse>
</problem>
<vertical start='Nov 19 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:C5QIGAi2zFs'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Map Reduce' chapter_name='week3' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:rqcbyw8Zcvg'/>
</vertical>
<problem showanswer='attempted' url_name='Map_Reduce_problem' rerandomize='never' display_name='Quiz: Map Reduce'>
MongoDB's built-in map/reduce can do the equivalent of the aggregation framework's $unwind operator.<multiplechoiceresponse direction='vertical'>
<choicegroup type='MultipleChoice'>
<choice location='bottom' correct='true'><text>True</text></choice>
<choice location='bottom' correct='false'><text>False</text></choice>
</choicegroup>
</multiplechoiceresponse>
</problem>
<vertical start='Nov 19 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:yFBKf3VEvnk'/>
</vertical>
</sequential>
<sequential graded='true' display_name='Homework 3.1' chapter_name='week3' format='Homework' due='Nov 19 23:00 EST 2012'>
<problem showanswer='due' attempts='3' end='2012-12-25T00:00' url_name='Homework_3_1_problem' rerandomize='never' display_name='Homework: Homework 3.1'>
Download the file <a href="http://media.mongodb.org/zips.json">http://media.mongodb.org/zips.json</a>.  The example below uses "curl", but you can download with your web browser if you like.  ("Zip codes" are the U.S. version of postal codes; they are 5 digits long.)

Then import this file into a collection "zips". 
<pre>
$ curl -O http://media.mongodb.org/zips.json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2803k  100 2803k    0     0  1807k      0  0:00:01  0:00:01 --:--:-- 1848k
$
$ # this is just informational for you:
$ md5 zips.json
MD5 (zips.json) = 4854d69c2ac389f334c0abff03d96259
$
$ # now import it ...
</pre>
After import you will find 29467 documents in your zips collection -- note this is slightly different than the .json file as the zips.json file is a bit noisy in its data and has some duplicate keys (duplicate _id's):
<pre>
$ mongo
> db.zips.count()
29467
</pre>
Question: consider the state with the 4th most zip codes in it.  How many zip codes does that state have?  Use the aggregation framework to query this.<customresponse>
<textbox rows='4' cols='50' correct_answer="1458"> </textbox>
<answer type='loncapa/python'><![CDATA[
import re
correct = ['correct']
try:
  r = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  r = '0'
response = re.sub(r'\s+', '', r)
quiz_answer = '''1458'''
quiz_answer = re.sub(r'\s+', '', quiz_answer)
if not(response == quiz_answer):
  correct[0] = 'incorrect'
  print 'response = ', response
  print 'quiz_answer = ', quiz_answer
]]>
</answer>
</customresponse>
</problem>
</sequential>
<sequential graded='true' display_name='Homework 3.2' chapter_name='week3' format='Homework' due='Nov 19 23:00 EST 2012'>
<problem showanswer='due' attempts='3' end='2012-12-25T00:00' url_name='Homework_3_2_problem' rerandomize='never' display_name='Homework: Homework 3.2'>
Use the same zips collection we imported in homework 3.1.  You can confirm you have the right data (well, the count of documents at least) via:
<pre>
&gt; db.zips.count()
29467
</pre>
Now use the aggregation framework to check the data quality.  First let's check how many zip documents have a starting zip code digit of each possible character:
<pre>
&gt; db.zips.aggregate( 
...   [
...     { $project : { _id : { $substr : ["$_id",0,1] } } } , 
...     { $group : { _id : "$_id", n : {$sum:1} } }
...   ]
... )
{
	"result" : [
		{
			"_id" : "5",
			"n" : 3624
		},
		{
			"_id" : "4",
			"n" : 3387
		},
		{
			"_id" : "6",
			"n" : 3540
		},
        …
</pre>
Now check in a similar manner the "city" field for its first character.  You will find there are cities in the collection that start with a number -- which indicates either a noisy data set or zip codes that have no corresponding city.  Let's suppose we want to delete these zip code documents -- the ones that have a numeric city name -- from our data set.  Do a remove operation to delete those documents.
<p>
After removing the documents indicated above, how many documents remain in the zips collection?
</p><customresponse>
<textbox rows='4' cols='50' correct_answer="29353"> </textbox>
<answer type='loncapa/python'><![CDATA[
import re
correct = ['correct']
try:
  r = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  r = '0'
response = re.sub(r'\s+', '', r)
quiz_answer = '''29353'''
quiz_answer = re.sub(r'\s+', '', quiz_answer)
if not(response == quiz_answer):
  correct[0] = 'incorrect'
  print 'response = ', response
  print 'quiz_answer = ', quiz_answer
]]>
</answer>
</customresponse>
</problem>
</sequential>
<sequential graded='true' display_name='Homework 3.3' chapter_name='week3' format='Homework' due='Nov 19 23:00 EST 2012'>
<problem showanswer='due' attempts='3' end='2012-12-25T00:00' url_name='Homework_3_3_problem' rerandomize='never' display_name='Homework: Homework 3.3'>
Download <a href="/static/handouts/week3.js">week3.js</a> from the education.10gen.com site.

Run this file in the shell to populate in the database of your choice a few documents in collections called:
<pre>
- policies
- customers
- staffers
</pre>
(Note some of these may be used in future weeks.)
<p>
Once loaded, query each collection above and verify they are non-empty.  In particular run:
</p>
<pre>
  db.policies.find().pretty() 
</pre>
to format the output of the one policy loaded nicely.
<p>
We have been asked to find all policies that are status!="expired", with liability coverage at or above $100.  Note that current:true indicates the rate that is "current" and not historical.
</p>
<p>
The query to find those policies is which of the following?
</p><multiplechoiceresponse direction='vertical'>
<choicegroup type='MultipleChoice'>
<choice location='bottom' correct='true'><text>db.policies.find(
 { status : { $ne : "expired" }, 
   coverages : { 
     $elemMatch : {
       type : "liability", 
       rates : { 
         $elemMatch : {
           rate : { $gte : 100 }, 
           current : true 
         }
       }
     }
   }
 }
)</text></choice>
<choice location='bottom' correct='false'><text>db.policies.find(
 { status : { $ne : "expired" }, 
   coverages : { 
     $elemMatch : {
       type : "liability", 
       rates : { 
           rate : { $gte : 100 }, 
           current : true 
       }
     }
   }
 }
)</text></choice>
<choice location='bottom' correct='false'><text>db.policies.find(
 { status : { $ne : "expired" }, 
   coverages : { 
       type : "liability", 
       rates : { 
         $elemMatch : {
           rate : { $gte : 100 }, 
           current : true 
         }
       }
   }
 }
)</text></choice>
</choicegroup>
</multiplechoiceresponse>
</problem>
</sequential>
<sequential graded='true' display_name='Homework 3.4' chapter_name='week3' format='Homework' due='Nov 19 23:00 EST 2012'>
<problem showanswer='due' attempts='3' end='2012-12-25T00:00' url_name='Homework_3_4_problem' rerandomize='never' display_name='Homework: Homework 3.4'>
<p>
Use the built-in Map/Reduce functionality in MongoDB to answer the following question: of the zip codes in Pennsylvania, how many are closer to Pittsburgh and how many are closer to Philadelphia?
</p>
<p>
The map function is already written for you.  (Note it uses some lat/longs that are somewhat arbitrary and not exactly the center of those cities.)  Our map function assumes the world is flat -- for this problem assume that assumption is ok.
</p><p>
(Aside: note that MongoDB does have some geospatial features; the problem could also be solved using more than one query and a geospatial index. However here our goal is to use map / reduce and try it out. In fact the MongoDB geospatial features would handle the world-not-flat issue…)
</p><p>
The map function is provided in the aforementioned week3.js file.  Thus if you run:
</p><pre>
mongo --shell week3.js
</pre><p>
The method map_closest() will already be defined.  You can type it at the shell prompt without parameters to see its source code:
</p><pre>
> map_closest
function map_closest() {
    var pitt = [-80.064879, 40.612044];
    var phil = [-74.978052, 40.089738];

    function distance(a, b) {
        var dx = a[0] - b[0];
        var dy = a[1] - b[1];
        return Math.sqrt(dx * dx + dy * dy);
    }

    if (distance(this.loc, pitt) &lt; distance(this.loc, phil)) {
        emit("pitt", 1);
    } else {
        emit("phil", 1);
    }
}
&gt; 
</pre><p>
Question: How many zip codes in PA are closer to philadelphia than to pittsburgh (given our map function's implementation)?  Use map/reduce to find the answer.</p><customresponse>
<textbox rows='4' cols='50' correct_answer="732"> </textbox>
<answer type='loncapa/python'><![CDATA[
import re
correct = ['correct']
try:
  r = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  r = '0'
response = re.sub(r'\s+', '', r)
quiz_answer = '''732'''
quiz_answer = re.sub(r'\s+', '', quiz_answer)
if not(response == quiz_answer):
  correct[0] = 'incorrect'
  print 'response = ', response
  print 'quiz_answer = ', quiz_answer
]]>
</answer>
</customresponse>
</problem>
</sequential>
</chapter>
<chapter start='2012-11-16T00:00' display_name='week4' hide_answer_status_until_due='False'>
<sequential graded='false' display_name='Introduction to replication' chapter_name='week4' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:zlxV8h5fSN0'/>
</vertical>
<problem showanswer='attempted' url_name='Introduction_to_replication_problem' rerandomize='never' display_name='Quiz: Introduction to replication'>
Which are true?<choiceresponse direction='vertical'>
<checkboxgroup type='MultipleChoice'>
<choice correct='false'>
<startouttext/>MongoDB replication is synchronous.<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>MongoDB replication works over wide area network connections.<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>MongoDB supports multi-master replication.<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>MongoDB uses statement based replication.<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>Statement based replication is the most efficient approach.<endouttext/>
</choice>
</checkboxgroup>
</choiceresponse>
</problem>
<vertical start='Nov 26 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:UhruOnk_gd4'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Replica Sets Overview' chapter_name='week4' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:ixheNHq1qvw'/>
</vertical>
<problem showanswer='attempted' url_name='Replica_Sets_Overview_problem' rerandomize='never' display_name='Quiz: Replica Sets Overview'>
Which of the following statements are true?<choiceresponse direction='vertical'>
<checkboxgroup type='MultipleChoice'>
<choice correct='true'>
<startouttext/>Replica sets support automatic failover.<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>It's not unusual to have a set with two members.<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>Failed nodes, if they come back up, automatically recover.<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>Failed nodes, if they come back up, automatically recover.  Upon recovery any local writes which had not formerly propagated are merged into the data sets of the other members of the set.<endouttext/>
</choice>
</checkboxgroup>
</choiceresponse>
</problem>
</sequential>
<sequential graded='false' display_name='Replica Set Demo' chapter_name='week4'>
<vertical>
<video display_name='Lecture' youtube='1.0:iEnioS8n254'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Replica Set Demo (windows)' chapter_name='week4'>
<vertical>
<video display_name='Lecture' youtube='1.0:UVvU7soWXqg'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Replica Sets - the simple http admin UI' chapter_name='week4' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:vDgF0-HU9fE'/>
</vertical>
<problem showanswer='attempted' url_name='Replica_Sets_the_simple_http_admin_UI_problem' rerandomize='never' display_name='Quiz: Replica Sets - the simple http admin UI'>
Which are true statements?

Note: replica set "arbiters" do not have an oplog; disregard that for this discussion we are referring to "normal" members of the set.  Arbiters will be covered in an upcoming section.<choiceresponse direction='vertical'>
<checkboxgroup type='MultipleChoice'>
<choice correct='false'>
<startouttext/>All the members of the set always have the same oplog start time.<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>All the members of the set always have the same oplog end time.<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>For a given database write / statement, on all members of the set who have received and applied the operation, the optime time stamp i their oplogs will all be exactly the same value.<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>If --rest is on you can query the mongod via REST (via HTTP).<endouttext/>
</choice>
</checkboxgroup>
</choiceresponse>
</problem>
<vertical start='Nov 26 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:cKEqGCms17w'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Replica Set Configuration' chapter_name='week4' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:Dul7SbTVgvs'/>
</vertical>
<problem showanswer='attempted' url_name='Replica_Set_Configuration_problem' rerandomize='never' display_name='Quiz: Replica Set Configuration'>
Suppose we wish to have a three node replica set with one member in a remote disaster recovery data center.  How do we configure such that the DR node never automatically becomes primary?<multiplechoiceresponse direction='vertical'>
<choicegroup type='MultipleChoice'>
<choice location='bottom' correct='true'><text>Set priority:0 for the DR node/member in its configuration object.</text></choice>
<choice location='bottom' correct='false'><text>Set hidden:true for the DR node/member in its configuration object.</text></choice>
<choice location='bottom' correct='false'><text>When clients connect to the set, they list the preferred members only in their connect strings and not the DR node's name.</text></choice>
<choice location='bottom' correct='false'><text>This is not supported.</text></choice>
</choicegroup>
</multiplechoiceresponse>
</problem>
<vertical start='Nov 26 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:9FLuW1ycRGg'/>
</vertical>
</sequential>
<sequential graded='false' display_name='GetLasterror and cluster wide commits' chapter_name='week4' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:uCZc78FD03c'/>
</vertical>
<problem showanswer='attempted' url_name='GetLasterror_and_cluster_wide_commits_problem' rerandomize='never' display_name='Quiz: GetLasterror and cluster wide commits'>
for getLastError / WriteConcern with w=3, if you have an arbiter, it counts as one of the 3.<multiplechoiceresponse direction='vertical'>
<choicegroup type='MultipleChoice'>
<choice location='bottom' correct='false'><text>True</text></choice>
<choice location='bottom' correct='true'><text>False</text></choice>
</choicegroup>
</multiplechoiceresponse>
</problem>
<vertical start='Nov 26 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:5WE0N4tCZqY'/>
</vertical>
</sequential>
<sequential graded='false' display_name='GetLastError Quiz 2' chapter_name='week4' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:2IE2nBD5phs'/>
</vertical>
<problem showanswer='attempted' url_name='GetLastError_Quiz_2_problem' rerandomize='never' display_name='Quiz: GetLastError Quiz 2'>
You can delete members from a replica set.<multiplechoiceresponse direction='vertical'>
<choicegroup type='MultipleChoice'>
<choice location='bottom' correct='true'><text>true</text></choice>
<choice location='bottom' correct='false'><text>false</text></choice>
</choicegroup>
</multiplechoiceresponse>
</problem>
<vertical start='Nov 26 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:GIrmxlR8CA0'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Multi data center and sample configurations' chapter_name='week4'>
<vertical>
<video display_name='Lecture' youtube='1.0:TZOH92mZIN8'/>
</vertical>
</sequential>
<sequential graded='false' display_name='ReadPreference (slaveOk)' chapter_name='week4' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:IDzKHRyiE18'/>
</vertical>
<problem showanswer='attempted' url_name='ReadPreference_slaveOk__problem' rerandomize='never' display_name='Quiz: ReadPreference (slaveOk)'>
Why might one use "secondaryPreferred" instead of "secondary" Read Preference?<multiplechoiceresponse direction='vertical'>
<choicegroup type='MultipleChoice'>
<choice location='bottom' correct='false'><text>The primary may have fresher data and we would like to use it.</text></choice>
<choice location='bottom' correct='true'><text>The secondaries may be down.</text></choice>
<choice location='bottom' correct='false'><text>To reduce load on the primary.</text></choice>
</choicegroup>
</multiplechoiceresponse>
</problem>
<vertical start='Nov 26 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:9LEg_4Qc-04'/>
</vertical>
</sequential>
<sequential graded='true' display_name='Homework 4.1' chapter_name='week4' format='Homework' due='Nov 26 23:00 EST 2012'>
<problem showanswer='due' attempts='3' end='2012-12-25T00:00' url_name='Homework_4_1_problem' rerandomize='never' display_name='Homework: Homework 4.1'>
In this week’s homework we will create a replica set and add some data to it.
<br />
1. Download <a href="/static/handouts/week4.js">week4.js</a> from the education.10gen.com site.
<br />
2. We will create a three member replica set.  Pick a root working directory to work in.  Go to that directory in a console window.  
<br />
Given we will have three members in the set, and three mongod processes, create three data directories: 
<pre>
mkdir 1
mkdir 2
mkdir 3
</pre>
<br />
3. We will now start a single mongod as a standalone server.  Given we will have three mongod processes on our single test server, we will explicitly specify the port numbers (this wouldn’t be necessary if we had three real machines or three virtual machines).  We’ll also use the --smallFiles parameter and --oplogSize so the files are small given we have a lot of server processes running on our test PC.
<pre>
# starting as a standalone server for problem 1:
mongod --dbpath 1 --port 27001 --smallfiles --oplogSize 50
</pre>
Note: for all mongod startups in the homework this week, you can optionally use --logPath,  --logAppend, and --fork.  Or, since this is just an exercise on a local PC, you could simply have a separate terminal window for all and forgoe those settings.  Run “mongod --help” for more info on those.

<br />
4. In a separate terminal window (cmd.exe on Windows), run the mongo shell with the week4.js file:

<pre>
mongo --port 27001 --shell week4.js
</pre>

Then run in the shell:
<pre>
  > homework.init()
</pre>

This will load a small amount of test data into the database.

<br />
Now run 

<pre>
 > homework.a()
</pre>

and enter the result.  This will simply confirm all the above happened ok.<customresponse>
<textbox rows='4' cols='50' correct_answer="5001"> </textbox>
<answer type='loncapa/python'><![CDATA[
import re
correct = ['correct']
try:
  r = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  r = '0'
response = re.sub(r'\s+', '', r)
quiz_answer = '''5001'''
quiz_answer = re.sub(r'\s+', '', quiz_answer)
if not(response == quiz_answer):
  correct[0] = 'incorrect'
  print 'response = ', response
  print 'quiz_answer = ', quiz_answer
]]>
</answer>
</customresponse>
</problem>
<vertical start='Nov 26 23:00 EST 2012' end='2012-12-25T00:00'>
<video display_name='Answer' youtube='1.0:jI863GJQI18'/>
</vertical>
</sequential>
<sequential graded='true' display_name='Homework 4.2' chapter_name='week4' format='Homework' due='Nov 26 23:00 EST 2012'>
<problem showanswer='due' attempts='3' end='2012-12-25T00:00' url_name='Homework_4_2_problem' rerandomize='never' display_name='Homework: Homework 4.2'>
Now convert the mongod instance (the one in the problem 4.1 above, which uses “--dbpath 1”) to a single server replica set.  To do this, you’ll need to stop the mongod and restart it with “--replSet” on its command line.  Give the set any name you like.
<br />
Then go to the mongo shell.  You should see some data in the week4 database.  You can confirm with:

<pre>
> use week4
> db.foo.find()
</pre>

Once done with that run
<pre>
 > homework.b()
</pre>

in the mongo shell and enter that result.<customresponse>
<textbox rows='4' cols='50' correct_answer="5002"> </textbox>
<answer type='loncapa/python'><![CDATA[
import re
correct = ['correct']
try:
  r = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  r = '0'
response = re.sub(r'\s+', '', r)
quiz_answer = '''5002'''
quiz_answer = re.sub(r'\s+', '', quiz_answer)
if not(response == quiz_answer):
  correct[0] = 'incorrect'
  print 'response = ', response
  print 'quiz_answer = ', quiz_answer
]]>
</answer>
</customresponse>
</problem>
<vertical start='Nov 26 23:00 EST 2012' end='2012-12-25T00:00'>
<video display_name='Answer' youtube='1.0:LFKmGa7ngYs'/>
</vertical>
</sequential>
<sequential graded='true' display_name='Homework 4.3' chapter_name='week4' format='Homework' due='Nov 26 23:00 EST 2012'>
<problem showanswer='due' attempts='3' end='2012-12-25T00:00' url_name='Homework_4_3_problem' rerandomize='never' display_name='Homework: Homework 4.3'>
Now add two more members to the set.  Use the 2/ and 3/ directories we created in homework 4.1.  Run those two mongod’s on ports 27002 and 27003 respectively (the exact numbers could be different).
<p />
Connect to one of those new members from the shell for example 

<pre>
mongo --port 27002 week4.js --shell
</pre>

One the servers have sync’d with the primary and are caught up run:

<pre>
> homework.c()
</pre>

and enter the result.<customresponse>
<textbox rows='4' cols='50' correct_answer="5"> </textbox>
<answer type='loncapa/python'><![CDATA[
import re
correct = ['correct']
try:
  r = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  r = '0'
response = re.sub(r'\s+', '', r)
quiz_answer = '''5'''
quiz_answer = re.sub(r'\s+', '', quiz_answer)
if not(response == quiz_answer):
  correct[0] = 'incorrect'
  print 'response = ', response
  print 'quiz_answer = ', quiz_answer
]]>
</answer>
</customresponse>
</problem>
<vertical start='Nov 26 23:00 EST 2012' end='2012-12-25T00:00'>
<video display_name='Answer' youtube='1.0:URvE5sW0lS4'/>
</vertical>
</sequential>
<sequential graded='true' display_name='Homework 4.4' chapter_name='week4' format='Homework' due='Nov 26 23:00 EST 2012'>
<problem showanswer='due' attempts='3' end='2012-12-25T00:00' url_name='Homework_4_4_problem' rerandomize='never' display_name='Homework: Homework 4.4'>
We will now retire the first member (@ port 27001) from the set (remove it that is).
<br />

As a first step to doing this we will shut it down.  (Given the rest of the set can maintain a majority, we can still do a majority reconfiguration if it is down.)
<br />
We could simply terminate its mongod process, but if we use the replSetStepDown command, the failover may be faster.  So that is a good practice albeit not essential.  Connect to member 1 (port 27001) in the shell and run: 

<pre>
> rs.help()
> rs.stepDown(300)
</pre>

Then cleanly terminate the mongod process for member 1.
<br />

Next, go to the new primary of the set.  Run rs.status() to check that things are as you expect.  Then, reconfigure to remove member 1.  Tip: if new to javascript you may find the Array.shift() method helpful for example try this in the shell:

<pre>
> my_array = [‘a’,’b’,’c’]
> my_array.shift()
> my_array
>
</pre>

When done, run

<pre>
> homework.d()
</pre>

and enter the result.  (If you ran the shell without week4.js on the command line, run it again with that.)<customresponse>
<textbox rows='4' cols='50' correct_answer="6"> </textbox>
<answer type='loncapa/python'><![CDATA[
import re
correct = ['correct']
try:
  r = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  r = '0'
response = re.sub(r'\s+', '', r)
quiz_answer = '''6'''
quiz_answer = re.sub(r'\s+', '', quiz_answer)
if not(response == quiz_answer):
  correct[0] = 'incorrect'
  print 'response = ', response
  print 'quiz_answer = ', quiz_answer
]]>
</answer>
</customresponse>
</problem>
<vertical start='Nov 26 23:00 EST 2012' end='2012-12-25T00:00'>
<video display_name='Answer' youtube='1.0:9t5xoAljX20'/>
</vertical>
</sequential>
<sequential graded='true' display_name='Homework 4.5' chapter_name='week4' format='Homework' due='Nov 26 23:00 EST 2012'>
<problem showanswer='due' attempts='3' end='2012-12-25T00:00' url_name='Homework_4_5_problem' rerandomize='never' display_name='Homework: Homework 4.5'>
<p>Note our replica set now only has an even number of members, and that is not a best practice.  However, to keep the homework from getting too long we’ll leave it at that for now, and instead do one more exercise below involving the <i>oplog</i>.
</p>
<p>
Go to the secondary in the replica set.  Run this to verify you are on the secondary:
</p>
<pre>
> // expect to get back false as we are not primary
> db.isMaster().ismaster
false
</pre>

Switch to the local database and then look at the oplog:

<pre>
> db.oplog.rs.find()
</pre>
<p>
If you get a blank result, you are not on the right database.
</p>
<p>
Note: as the local database doesn’t replicate, it will let you query it without entering “rs.slaveOk()” first.
</p>
<p>
Note: you may wonder why the number of lines in the oplog is the number you see here.  See the homework answer video when answers are posted for more information if curious.
</p>
<p>
Next look at the stats on the oplog to get a feel for its size:
</p>
<pre>
> db.oplog.rs.stats()
</pre>

What result does this expression give when evaluated?

<pre>
db.oplog.rs.find().sort({$natural:1}).limit(1).next().o.msg[0]
</pre><multiplechoiceresponse direction='vertical'>
<choicegroup type='MultipleChoice'>
<choice location='bottom' correct='false'><text>N</text></choice>
<choice location='bottom' correct='false'><text>n</text></choice>
<choice location='bottom' correct='false'><text>Blank</text></choice>
<choice location='bottom' correct='true'><text>R</text></choice>
<choice location='bottom' correct='false'><text>r</text></choice>
<choice location='bottom' correct='false'><text>I</text></choice>
</choicegroup>
</multiplechoiceresponse>
</problem>
<vertical start='Nov 26 23:00 EST 2012' end='2012-12-25T00:00'>
<video display_name='Answer' youtube='1.0:AEW50jZISeQ'/>
</vertical>
</sequential>
</chapter>
<chapter start='2012-11-23T00:00' display_name='week5' hide_answer_status_until_due='False'>
<sequential graded='false' display_name='Week5 Intro' chapter_name='week5'>
<vertical>
<video display_name='Lecture' youtube='1.0:TjSa45rEoYw'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Indexes and Optimizing Performance' chapter_name='week5' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:a7TrHP1C6qQ'/>
</vertical>
<problem showanswer='attempted' url_name='Indexes_and_Optimizing_Performance_problem' rerandomize='never' display_name='Quiz: Indexes and Optimizing Performance'>
Quiz: a mongodb index can have keys of different types (ints, dates, string for example) in it?<multiplechoiceresponse direction='vertical'>
<choicegroup type='MultipleChoice'>
<choice location='bottom' correct='true'><text>True</text></choice>
<choice location='bottom' correct='false'><text>False</text></choice>
</choicegroup>
</multiplechoiceresponse>
</problem>
<vertical start='Dec 3 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:p-rVdi50E74'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Index Types' chapter_name='week5' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:DFfCC8Or8_U'/>
</vertical>
<problem showanswer='attempted' url_name='Index_Types_problem' rerandomize='never' display_name='Quiz: Index Types'>
Quiz: typically, what’s the best index for this query, assuming we have lots of queries compared to writes?

<pre>
db.users.find( { last:"smith", first:"john" } )
</pre>

Fill in the blank:

<pre>
db.users.ensureIndex({___})
</pre><customresponse>
<textbox rows='4' cols='50' correct_answer='{"last":1,"first":1}'> </textbox>
<answer type='loncapa/python'><![CDATA[
import demjson
correct = ['correct']
messages = ['']
try:
  r = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  r = '0'
quiz_answer = '''{"last":1,"first":1}'''
quiz_answer_json = {}
try:
  quiz_answer_json = demjson.decode(quiz_answer)
except demjson.JSONDecodeError as err:
  messages[0] = 'Quiz Answer failed to parse' + str(err)
response_json = {}
try:
  response_json = demjson.decode(r)
except demjson.JSONDecodeError as err:
  messages[0] = 'Response failed to parse' + str(err)
if not(response_json == quiz_answer_json):
  correct[0] = 'incorrect'
]]></answer>
</customresponse>
</problem>
<vertical start='Dec 3 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:KFXYaOVtyPo'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Covered Indexes' chapter_name='week5' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:boAkBnMUBnw'/>
</vertical>
<problem showanswer='attempted' url_name='Covered_Indexes_problem' rerandomize='never' display_name='Quiz: Covered Indexes'>
Suppose we run:
<pre>
db.foo.ensureIndex({a:1,b:2,c:3})
db.foo.find({a:"sports", b:{$gt:100}})
</pre>
Then<multiplechoiceresponse direction='vertical'>
<choicegroup type='MultipleChoice'>
<choice location='bottom' correct='false'><text>Only the index needs to be touched to fully execute the query (the find() statement).</text></choice>
<choice location='bottom' correct='true'><text>The index and some documents need to be touched.</text></choice>
</choicegroup>
</multiplechoiceresponse>
</problem>
<vertical start='Dec 3 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:UKOOJocN51E'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Explain and Hint' chapter_name='week5' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:oaTm0Kftit8'/>
</vertical>
<problem showanswer='attempted' url_name='Explain_and_Hint_problem' rerandomize='never' display_name='Quiz: Explain and Hint'>
Hint always works.<multiplechoiceresponse direction='vertical'>
<choicegroup type='MultipleChoice'>
<choice location='bottom' correct='false'><text>True</text></choice>
<choice location='bottom' correct='true'><text>False</text></choice>
</choicegroup>
</multiplechoiceresponse>
</problem>
<vertical start='Dec 3 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:9gqnaoe43m0'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Read vs. Write Tradeoffs' chapter_name='week5' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:USDbDotmums'/>
</vertical>
<problem showanswer='attempted' url_name='Read_vs_Write_Tradeoffs_problem' rerandomize='never' display_name='Quiz: Read vs. Write Tradeoffs'>
We choose to create an index on x with default settings: 
<pre>
	db.foo.ensureIndex({x:1})
</pre>
<multiplechoiceresponse direction='vertical'>
<choicegroup type='MultipleChoice'>
<choice location='bottom' correct='false'><text>If there are documents with no value for x, the index build will fail.</text></choice>
<choice location='bottom' correct='false'><text>If there are documents with no value for x, the index build will work and those documents aren’t in the index.
</text></choice>
<choice location='bottom' correct='true'><text>If there are documents with no value for x, the index build will work and those documents are in the index.
</text></choice>
</choicegroup>
</multiplechoiceresponse>
</problem>
<vertical start='Dec 3 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:g8AtXS5GgCw'/>
</vertical>
</sequential>
<sequential graded='false' display_name='currentOp and killOp' chapter_name='week5' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:i7XEKAtRS_M'/>
</vertical>
<problem showanswer='attempted' url_name='currentOp_and_killOp_problem' rerandomize='never' display_name='Quiz: currentOp and killOp'>
Which ops are safe to kill (without any special effort or cleanup or implications later beyond the obvious):
<choiceresponse direction='vertical'>
<checkboxgroup type='MultipleChoice'>
<choice correct='true'>
<startouttext/>a query
<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>a findAndModify on a primary

<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>a foreground create index building on a primary<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>a foreground create index building on a secondary
<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>a compact command job<endouttext/>
</choice>
</checkboxgroup>
</choiceresponse>
</problem>
<vertical start='Dec 3 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:1GBM9ghVK9Y'/>
</vertical>
</sequential>
<sequential graded='false' display_name='The Profiler' chapter_name='week5' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:MzLmI8FNB94'/>
</vertical>
<problem showanswer='attempted' url_name='The_Profiler_problem' rerandomize='never' display_name='Quiz: The Profiler'>
The profiler can tell you if updates are “moving”
<multiplechoiceresponse direction='vertical'>
<choicegroup type='MultipleChoice'>
<choice location='bottom' correct='true'><text>True</text></choice>
<choice location='bottom' correct='false'><text>False</text></choice>
</choicegroup>
</multiplechoiceresponse>
</problem>
<vertical start='Dec 3 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:qHZSuBCt7ew'/>
</vertical>
</sequential>
<sequential graded='false' display_name='mongostat and mongotop' chapter_name='week5'>
<vertical>
<video display_name='Lecture' youtube='1.0:fEgl1DT_lDA'/>
</vertical>
</sequential>
<sequential graded='true' display_name='Homework 5.1' chapter_name='week5' format='Homework' due='Dec 3 23:00 EST 2012'>
<problem showanswer='due' attempts='3' end='2012-12-25T00:00' url_name='Homework_5_1_problem' rerandomize='never' display_name='Homework: Homework 5.1'>
<p>Start a <code>mongod</code> server instance (if you still have your replica set from week 4 running, that would work too).
</p>
Next, download <a href="/static/handouts/week5.js">week5.js</a> and run:
<pre>
mongo --shell localhost/week5 week5.js
> homework.init()
>
</pre>
<p>
Assume our app development team for a project would like to speed up the following query:
</p><pre>
db.sensor_readings.find( { 
  tstamp : { 
    $gte : ISODate("2012-08-01"), 
    $lte :  ISODate("2012-09-01") 
  },
  active : true
} ).limit(3)
</pre><p>
Run the query, and explain().  Then create an index to speed up the query (with ensureIndex()).  Then run the explain() again and see that things have improved.  When you have a solution run: 
</p><pre>
homework.a()
</pre>
<p>
and enter the result below.
</p><p>
Note: if you would like to try different indexes, you can use 
<code>
db.sensor_readings.dropIndexes()</code> to drop your old index before creating a new one. (For this problem you will only need one index beyond the _id index which is present by default.)
</p>
<p>
Note: we are optimizing for the query <i>when the limit clause is present</i>.  So be use to include limit when you call explain.
</p><customresponse>
<textbox rows='4' cols='50' correct_answer="6"> </textbox>
<answer type='loncapa/python'><![CDATA[
import re
correct = ['correct']
try:
  r = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  r = '0'
response = re.sub(r'\s+', '', r)
quiz_answer = '''6'''
quiz_answer = re.sub(r'\s+', '', quiz_answer)
if not(response == quiz_answer):
  correct[0] = 'incorrect'
  print 'response = ', response
  print 'quiz_answer = ', quiz_answer
]]>
</answer>
</customresponse>
</problem>
<vertical start='Dec 3 23:00 EST 2012' end='2012-12-25T00:00'>
<video display_name='Answer' youtube='1.0:D7t_EFzWDio'/>
</vertical>
</sequential>
<sequential graded='true' display_name='Homework 5.2' chapter_name='week5' format='Homework' due='Dec 3 23:00 EST 2012'>
<problem showanswer='due' attempts='3' end='2012-12-25T00:00' url_name='Homework_5_2_problem' rerandomize='never' display_name='Homework: Homework 5.2'>
<p>
In a mongo shell run <code>homework.b()</code>.  This will run in an infinite loop printing some output as it runs various statements against the server.
</p><p>
We'll now imagine that on this system a user has complained of slowness and we suspect there is a slow operation running.  Find the slow operation and terminate it.  (Keep the other shell with homework.b() going while this is happening).  Once you have eliminated the slow operation, run
</p><pre>
homework.c()
</pre><p>
and enter the output below.  Once you have it right and are ready to move on, ctrl-c (terminate) the shell that is still running the <code>homework.b()</code> function.</p><customresponse>
<textbox rows='4' cols='50' correct_answer="12"> </textbox>
<answer type='loncapa/python'><![CDATA[
import re
correct = ['correct']
try:
  r = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  r = '0'
response = re.sub(r'\s+', '', r)
quiz_answer = '''12'''
quiz_answer = re.sub(r'\s+', '', quiz_answer)
if not(response == quiz_answer):
  correct[0] = 'incorrect'
  print 'response = ', response
  print 'quiz_answer = ', quiz_answer
]]>
</answer>
</customresponse>
</problem>
<vertical start='Dec 3 23:00 EST 2012' end='2012-12-25T00:00'>
<video display_name='Answer' youtube='1.0:mzwuJnqCOHI'/>
</vertical>
</sequential>
<sequential graded='true' display_name='Homework 5.3' chapter_name='week5' format='Homework' due='Dec 3 23:00 EST 2012'>
<problem showanswer='due' attempts='3' end='2012-12-25T00:00' url_name='Homework_5_3_problem' rerandomize='never' display_name='Homework: Homework 5.3'>
<p>Compact the <code>week5.sensor_readings</code> collection.  Then run <code>homework.d()</code> and enter the result below.
</p><p><i>Note: If you happen to bee running a replica set, just compact on the primary and run homework.d() on the primary. You may need to use the <a href="http://docs.mongodb.org/manual/reference/command/compact/">force:true</a> option to run compact on a primary.</i></p><customresponse>
<textbox rows='4' cols='50' correct_answer="21"> </textbox>
<answer type='loncapa/python'><![CDATA[
import re
correct = ['correct']
try:
  r = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  r = '0'
response = re.sub(r'\s+', '', r)
quiz_answer = '''21'''
quiz_answer = re.sub(r'\s+', '', quiz_answer)
if not(response == quiz_answer):
  correct[0] = 'incorrect'
  print 'response = ', response
  print 'quiz_answer = ', quiz_answer
]]>
</answer>
</customresponse>
</problem>
<vertical start='Dec 3 23:00 EST 2012' end='2012-12-25T00:00'>
<video display_name='Answer' youtube='1.0:CNoCFKNif7k'/>
</vertical>
</sequential>
</chapter>
<chapter start='2012-12-02T00:00' display_name='week 6' hide_answer_status_until_due='False'>
<sequential graded='false' display_name='Introduction to Sharding' chapter_name='week 6' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:9p0yPCVOl_I'/>
</vertical>
<problem showanswer='attempted' url_name='Introduction_to_Sharding_problem' rerandomize='never' display_name='Quiz: Introduction to Sharding'>
<pre>db.foo.find( 
  { name : { $gt : "Joe", $lt : "John" } } ) </pre> is efficient if the collection is sharded on <code>{ name : 1 }</code><multiplechoiceresponse direction='vertical'>
<choicegroup type='MultipleChoice'>
<choice location='bottom' correct='true'><text>True</text></choice>
<choice location='bottom' correct='false'><text>False</text></choice>
</choicegroup>
</multiplechoiceresponse>
</problem>
<vertical start='Dec 10 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:I0AL03BbvN4'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Sharding Setup Demo' chapter_name='week 6' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:5bmaomQ1piM'/>
</vertical>
<problem showanswer='attempted' url_name='Sharding_Setup_Demo_problem' rerandomize='never' display_name='Quiz: Sharding Setup Demo'>
Quiz: How many mongos processes should you run (generally)?

<multiplechoiceresponse direction='vertical'>
<choicegroup type='MultipleChoice'>
<choice location='bottom' correct='false'><text>One per config server
</text></choice>
<choice location='bottom' correct='false'><text>One per mongod process
</text></choice>
<choice location='bottom' correct='false'><text>One per shard
</text></choice>
<choice location='bottom' correct='false'><text>One</text></choice>
<choice location='bottom' correct='true'><text>However many you want but usually much more than one
</text></choice>
</choicegroup>
</multiplechoiceresponse>
</problem>
<vertical start='Dec 10 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:fI9wRpuRmGI'/>
</vertical>
</sequential>
<sequential graded='false' display_name='The config database' chapter_name='week 6' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:WVXmIVUGBm4'/>
</vertical>
<problem showanswer='attempted' url_name='The_config_database_problem' rerandomize='never' display_name='Quiz: The config database'>
Which are true?<choiceresponse direction='vertical'>
<checkboxgroup type='MultipleChoice'>
<choice correct='false'>
<startouttext/>To check via the mongo shell what's in the config database of a MongoDB cluster, you must connect it directly to one of the config servers.<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>Three config servers is typical in a MongoDB cluster with 1,000 total machines.<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>All the config servers have exactly the same data.<endouttext/>
</choice>
</checkboxgroup>
</choiceresponse>
</problem>
<vertical start='Dec 10 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:v0tZUP3coMM'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Setup Part 2 Adding the Initial Shards' chapter_name='week 6'>
<vertical>
<video display_name='Lecture' youtube='1.0:-Ijt60KkWpc'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Enabling sharding for a collection' chapter_name='week 6' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:G2GYYDTfq-M'/>
</vertical>
<problem showanswer='attempted' url_name='Enabling_sharding_for_a_collection_problem' rerandomize='never' display_name='Quiz: Enabling sharding for a collection'>
In MongoDB v2.2, by default (i.e., if shardCollection hasn't been invoked) collections are:<multiplechoiceresponse direction='vertical'>
<choicegroup type='MultipleChoice'>
<choice location='bottom' correct='true'><text>On the first shard and not sharded</text></choice>
<choice location='bottom' correct='false'><text>Not sharded but homed on different shards at random</text></choice>
<choice location='bottom' correct='false'><text>Automatically sharded on _id</text></choice>
</choicegroup>
</multiplechoiceresponse>
</problem>
<vertical start='Dec 10 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:6GIcACfPJso'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Working with a sharded collection' chapter_name='week 6'>
<vertical>
<video display_name='Lecture' youtube='1.0:m0Q5kEfBc14'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Choosing shard keys' chapter_name='week 6' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:LtY9EyTceeA'/>
<html>
<h2>Lecture Notes</h2>
<ul>
<li>monotonic</li>
<li>cardinality</li>
<li>sorting</li>
</ul></html>
</vertical>
<problem showanswer='attempted' url_name='Choosing_shard_keys_problem' rerandomize='never' display_name='Quiz: Choosing shard keys'>
Suppose we are building an email server system and have a collection:
<pre>
messages
--------
{_id:&lt;message_id&gt;, mailbox_id:___, 
 sender_id:___, subject:___, date:___, 
 body:___, read:&lt;bool&gt;, ...
}
</pre>
We have millions of users (email addresses / mailboxes) so want to shard this collection.  Of the choices listed, what's a good shard key?<multiplechoiceresponse direction='vertical'>
<choicegroup type='MultipleChoice'>
<choice location='bottom' correct='false'><text>{_id:1}</text></choice>
<choice location='bottom' correct='true'><text>{mailbox_id:1}</text></choice>
<choice location='bottom' correct='false'><text>{sender_id:1}</text></choice>
<choice location='bottom' correct='false'><text>{sender_id:1,read:1}</text></choice>
</choicegroup>
</multiplechoiceresponse>
</problem>
<vertical start='Dec 10 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:SFuXEsi4geA'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Process and Machine Layout' chapter_name='week 6' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:j43kGempShk'/>
</vertical>
<problem showanswer='attempted' url_name='Process_and_Machine_Layout_problem' rerandomize='never' display_name='Quiz: Process and Machine Layout'>
Quiz: If I have 3 shards, 3 member replica sets, 3 config servers, and 6 mongos processes running, how many machines might we use for that?<multiplechoiceresponse direction='vertical'>
<choicegroup type='MultipleChoice'>
<choice location='bottom' correct='true'><text>While we could use more if we wanted, 9 machines would be just fine.
</text></choice>
<choice location='bottom' correct='false'><text>9 machines works but is too low for best practice usage</text></choice>
</choicegroup>
</multiplechoiceresponse>
</problem>
<vertical start='Dec 10 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:64GTfgpwMD8'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Bulk inserts and pre-splitting' chapter_name='week 6' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:rYGOF-MjRQU'/>
</vertical>
<problem showanswer='attempted' url_name='Bulk_inserts_and_pre_splitting_problem' rerandomize='never' display_name='Quiz: Bulk inserts and pre-splitting'>
Generally speaking, shard keys with non-uniform key distributions are:<multiplechoiceresponse direction='vertical'>
<choicegroup type='MultipleChoice'>
<choice location='bottom' correct='true'><text>ok</text></choice>
<choice location='bottom' correct='false'><text>ok only if you "pre-split"</text></choice>
<choice location='bottom' correct='false'><text>not ok</text></choice>
</choicegroup>
</multiplechoiceresponse>
</problem>
<vertical start='Dec 10 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:YX57DMEOmUc'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Further tips and best practices' chapter_name='week 6'>
<vertical>
<video display_name='Lecture' youtube='1.0:9QXa9Uz8bjc'/>
<html>
<h2>Lecture Notes</h2>
<h4>changing config servers</h4>
<ul>
<li>only shard the big collections</li>
<li>pre-split if bulk loading</li>
<li>pick shard key with care, they aren't easily changeable</li>
<li>be cognizant of monotonically increasing shard keys</li>
<li>adding a shard is easy but takes time</li>
<li>use logical server names for config servers</li>
<li>don't directly talk to anything except mongos except for dba work</li>
<li>keep non-mongos processes off of 27017 to avoid mistakes</li>
</ul></html>
</vertical>
</sequential>
<sequential graded='true' display_name='Homework 6.1' chapter_name='week 6' format='Homework' due='Dec 10 23:00 EST 2012'>
<problem showanswer='due' attempts='3' end='2012-12-25T00:00' url_name='Homework_6_1_problem' rerandomize='never' display_name='Homework: Homework 6.1'>
<p>For this week's homework we will start with a standalone MongoDB database, turn it into a sharded cluster with two shards, and shard one of the collections.  We will create a "dev" environment on our local box: no replica sets, and only one config server.  In production you would almost always use three config servers and replica sets as part of a sharded cluster. In the final of the course we'll set up a larger cluster with replica sets and three config servers.
</p>
<p>Download <a href="/static/handouts/week6.js">week6.js</a>.</p>  
<p>
Start an initially empty <code>mongod</code> database instance. 
</p>

Connect to it with the shell and week6.js loaded:

<pre>
mongo --shell localhost/week6 week6.js
</pre>

Run <code>homework.init()</code>.  It will take some time to run as it inserts quite a few documents.  When it is done run

<pre>
db.trades.stats()
</pre>

to check the status of the collection.
<br/>
<p>
At this point we have a single mongod and would like to transform it into a sharded cluster with one shard.  (We'll use this node’s existing <code>week6.trades</code> data in the cluster.)
</p><p>
Stop the mongod process.  
Now, restart the mongod process adding the option <code>--shardsvr</code>.  If you started mongod with a <code>--dbpath</code> option, specify that as well.</p>
<pre>
mongod --shardsvr …
</pre>
<p>Note that with --shardsvr specified the default port for mongod becomes 27018.
</p>
<p>Start a mongo config server:

<pre>mongod --configsvr …
</pre>

(Note with <code>--configsvr</code> specified the default port for listening becomes 27019 and the default data directory /data/configdb.  Wherever your data directory is, it is suggested that you verify that the directory is empty before you begin.)
</p><p>
Start a mongos:

<pre>
mongos --configdb your_host_name:27019
</pre>

Connect to mongos with the shell: 

<pre>
mongo --shell localhost/week6 week6.js
</pre>

Add the first shard ("<i>your_host_name</i>:27018").
</p><p>
Verify that the week6.trades data is visible via mongos.  Note at this point the week6 database isn't "sharding enabled" but its data is still visible via mongos:

<pre>
> db.trades.find().pretty()
> db.trades.count()
> db.trades.stats()
</pre>

Run homework.a() and enter the result below.  This method will simply verify that this simple cluster is up and running and return a result key.
</p>
<customresponse>
<textbox rows='4' cols='50' correct_answer="1000001"> </textbox>
<answer type='loncapa/python'><![CDATA[
import re
correct = ['correct']
try:
  r = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  r = '0'
response = re.sub(r'\s+', '', r)
quiz_answer = '''1000001'''
quiz_answer = re.sub(r'\s+', '', quiz_answer)
if not(response == quiz_answer):
  correct[0] = 'incorrect'
  print 'response = ', response
  print 'quiz_answer = ', quiz_answer
]]>
</answer>
</customresponse>
</problem>
<vertical start='Dec 10 23:00 EST 2012' end='2012-12-25T00:00'>
<video display_name='Answer' youtube='1.0:B21ZJwdirPs'/>
</vertical>
</sequential>
<sequential graded='true' display_name='Homework 6.2' chapter_name='week 6' format='Homework' due='Dec 10 23:00 EST 2012'>
<problem showanswer='due' attempts='3' end='2012-12-25T00:00' url_name='Homework_6_2_problem' rerandomize='never' display_name='Homework: Homework 6.2'>
<p>Now enable sharding for the week6 database.  (See sh.help() for details.)
</p><p>
Then shard the trades collection on the compound shard key ticker plus time.  Note to shard a collection, you must have an index on the shard key, so you will need to create the index first:

<pre>
> db.trades.ensureIndex( { ticker:1, time:1 } )
> // can now shard the trades collection on the shard key  { ticker:1, time:1 } 
</pre>

After sharding the collection, look at the chunks which exist: 
<pre>
> use config
> db.chunks.find()
> // or:
> db.chunks.find({}, {min:1,max:1,shard:1,_id:0,ns:1})
</pre>

Run homework.b() to verify the above and enter the return value below.
</p>
<customresponse>
<textbox rows='4' cols='50' correct_answer="3"> </textbox>
<answer type='loncapa/python'><![CDATA[
import re
correct = ['correct']
try:
  r = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  r = '0'
response = re.sub(r'\s+', '', r)
quiz_answer = '''3'''
quiz_answer = re.sub(r'\s+', '', quiz_answer)
if not(response == quiz_answer):
  correct[0] = 'incorrect'
  print 'response = ', response
  print 'quiz_answer = ', quiz_answer
]]>
</answer>
</customresponse>
</problem>
<vertical start='Dec 10 23:00 EST 2012' end='2012-12-25T00:00'>
<video display_name='Answer' youtube='1.0:DKF72Qt0dwk'/>
</vertical>
</sequential>
<sequential graded='true' display_name='Homework 6.3' chapter_name='week 6' format='Homework' due='Dec 10 23:00 EST 2012'>
<problem showanswer='due' attempts='3' end='2012-12-25T00:00' url_name='Homework_6_3_problem' rerandomize='never' display_name='Homework: Homework 6.3'>
<p>Let's now add a new shard.  Run another mongod as the new shard on a new port number. Use --shardsvr.
</p><p>
Then add the shard to the cluster (see sh.help()). 
</p><p>
You can confirm the above worked by running: 
<pre>homework.check1()</pre>
Now wait for the balancer to move data among the two shards more evenly.  Periodically run: 
<pre>
> use config
> db.chunks.find( { ns:"week6.trades" }, {min:1,max:1,shard:1,_id:0} ).sort({min:1})
</pre>

and/or:

<pre>
db.chunks.aggregate( [
 { $match : { ns : "week6.trades" } } , 
 { $group : { _id : "$shard", n : { $sum : 1 } } }
] )
</pre>

When done, run homework.c() and enter the result value.
</p><p>
That completes this week's homework.  However if you want to explore more, something to try would be to try some queries and/or write operations with a single process down to see how the system behaves in such a situation.
</p><customresponse>
<textbox rows='4' cols='50' correct_answer="2"> </textbox>
<answer type='loncapa/python'><![CDATA[
import re
correct = ['correct']
try:
  r = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  r = '0'
response = re.sub(r'\s+', '', r)
quiz_answer = '''2'''
quiz_answer = re.sub(r'\s+', '', quiz_answer)
if not(response == quiz_answer):
  correct[0] = 'incorrect'
  print 'response = ', response
  print 'quiz_answer = ', quiz_answer
]]>
</answer>
</customresponse>
</problem>
<vertical start='Dec 10 23:00 EST 2012' end='2012-12-25T00:00'>
<video display_name='Answer' youtube='1.0:pHQv3fSDWys'/>
</vertical>
</sequential>
</chapter>
<chapter start='2012-12-09T00:00' display_name='week7'>
<sequential graded='false' display_name='Week7 Intro' chapter_name='week7'>
<vertical>
<video display_name='Lecture' youtube='1.0:VNNau8FRIiE'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Security Intro' chapter_name='week7' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:zIp0GZOxlB0'/>
</vertical>
<problem showanswer='attempted' url_name='Security_Intro_problem' rerandomize='never' display_name='Quiz: Security Intro'>
Which are true?<choiceresponse direction='vertical'>
<checkboxgroup type='MultipleChoice'>
<choice correct='false'>
<startouttext/>When using --keyFile with a replica set, the keyFile's content is sent over the network between mongod nodes unencrypted.<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>When using --keyFile with a replica set, database contents are sent over the network between mongod nodes unencrypted.<endouttext/>
</choice>
</checkboxgroup>
</choiceresponse>
</problem>
<vertical start='Dec 17 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:QWeHVebGRiQ'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Security and Clients' chapter_name='week7' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:4eWnyAGxJ68'/>
</vertical>
<problem showanswer='attempted' url_name='Security_and_Clients_problem' rerandomize='never' display_name='Quiz: Security and Clients'>
Which are true?<choiceresponse direction='vertical'>
<checkboxgroup type='MultipleChoice'>
<choice correct='true'>
<startouttext/>You can give a user read-only access to one database and write access to another database.<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>The user's password is sent over the network unencrypted.<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>The user's password is stored on disk unencrypted.<endouttext/>
</choice>
</checkboxgroup>
</choiceresponse>
</problem>
<vertical start='Dec 17 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:RskjIuCxhN8'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Intra-cluster Security' chapter_name='week7'>
<vertical>
<video display_name='Lecture' youtube='1.0:SI-P8wZ8hM0'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Backups' chapter_name='week7'>
<vertical>
<video display_name='Lecture' youtube='1.0:8lz9TbgGbb8'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Backup strategies' chapter_name='week7' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:7x9UPaq-i78'/>
</vertical>
<problem showanswer='attempted' url_name='Backup_strategies_problem' rerandomize='never' display_name='Quiz: Backup strategies'>
Which are true statements?<choiceresponse direction='vertical'>
<checkboxgroup type='MultipleChoice'>
<choice correct='false'>
<startouttext/>You can copy Mongo datafiles while a mongod is running.<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>It's always safe to do a true snapshot of the data volume while mongod is running.<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>mongoexport has the ability to directly access MongoDB datafiles in addition to connecting to a MongoDB cluster or server via the network.<endouttext/>
</choice>
</checkboxgroup>
</choiceresponse>
</problem>
<vertical start='Dec 17 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:DkdsoERSN58'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Geospatial Indexes' chapter_name='week7' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:67w3olDZQ_0'/>
</vertical>
<problem showanswer='attempted' url_name='Geospatial_Indexes_problem' rerandomize='never' display_name='Quiz: Geospatial Indexes'>
How many dimensions does MongoDB geospatial indexing support?<multiplechoiceresponse direction='vertical'>
<choicegroup type='MultipleChoice'>
<choice location='bottom' correct='true'><text>2</text></choice>
<choice location='bottom' correct='false'><text>3</text></choice>
<choice location='bottom' correct='false'><text>4</text></choice>
<choice location='bottom' correct='false'><text>any number</text></choice>
</choicegroup>
</multiplechoiceresponse>
</problem>
<vertical start='Dec 17 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:uVJGgl9wyDo'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Additional Features' chapter_name='week7' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:Ih8PVVwvyuU'/>
<html>
<h2>Lecture Notes</h2>
TTL and Capped Collections
GridFS
</html>
</vertical>
<problem showanswer='attempted' url_name='Additional_Features_problem' rerandomize='never' display_name='Quiz: Additional Features'>
You can store videos in MongoDB.<multiplechoiceresponse direction='vertical'>
<choicegroup type='MultipleChoice'>
<choice location='bottom' correct='true'><text>True</text></choice>
<choice location='bottom' correct='false'><text>False</text></choice>
</choicegroup>
</multiplechoiceresponse>
</problem>
<vertical start='Dec 17 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:TvsRG49Vxt0'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Hardware tips' chapter_name='week7' format='Quiz'>
<vertical>
<video display_name='Lecture' youtube='1.0:19VZJ9H3LmQ'/>
</vertical>
<problem showanswer='attempted' url_name='Hardware_tips_problem' rerandomize='never' display_name='Quiz: Hardware tips'>
If a database is 100GB in size and has an equal amount of random read and write operations occurring, which hardware configuration is likely to perform better?<multiplechoiceresponse direction='vertical'>
<choicegroup type='MultipleChoice'>
<choice location='bottom' correct='false'><text>128GB RAM and rotating disk storage</text></choice>
<choice location='bottom' correct='true'><text>32GB RAM and solid state disk storage</text></choice>
</choicegroup>
</multiplechoiceresponse>
</problem>
<vertical start='Dec 17 23:00 EST 2012'>
<video display_name='Answer' youtube='1.0:tk21Thi7csE'/>
</vertical>
</sequential>
<sequential graded='false' display_name='Additional Resources' chapter_name='week7'>
<vertical>
<video display_name='Lecture' youtube='1.0:BYegVYF7lHY'/>
<html>
<h2>Lecture Notes</h2>
jira
irc
forums
</html>
</vertical>
</sequential>
</chapter>
<chapter start='2012-12-12' display_name='Final' hide_answer_status_until_due='True'>
<sequential graded='true' display_name='Final.1' chapter_name='Final' format='Final' due='Dec 20 23:00 EST 2012'>
<problem showanswer='due' end='2012-12-25T00:00' url_name='Final_1_problem' rerandomize='never' display_name='Final: Final.1'>
<p>
Problems 1 through 3 are an exercise in running mongod's, replica sets, and an exercise in testing of replica set <i>rollbacks</i>, which can occur when a former primary rejoins a set after it has previously had a failure.
</p><p>
Get the <a href="/static/handouts/a.sh">a.sh</a> and <a href="/static/handouts/a.js">a.js</a> files from the course site.  Use <a href="/static/handouts/a.bat">a.bat</a> instead of a.sh on Windows.
</p><p>
Start a 3 member replica set (with default options for each member, all are peers).  (a.sh will start the mongod's for you if you like.)
</p>
<pre>
$ # if on unix:
$ chmod +x a.sh
$ ./a.sh
</pre>

Then run:

<pre>
$ mongo --shell --port 27003 a.js
> // ourinit() will initiate the set for us.
> // to see its source code type without the parentheses:
> ourinit
>
> // now execute it:
> ourinit()
</pre>
<p>
We will now do a test of replica set rollbacks.  This is the case where data never reaches a majority of the set.  We'll test a couple scenarios.
</p>
<p>
Take a look at the method <code>testRollback()</code> in a.js and what it does.  Then, on <code>localhost:27003</code>, with that member primary, run:
</p>
<pre>
> testRollback()
</pre>
<p>
Note: if 27003 is not primary, make it primary -- using <code>rs.stepDown()</code> (perhaps also <code>rs.freeze()</code>) for example.
</p>
<p>
At this point the mongod's on 27001 and 27002 are shut down.  See the <code>a.js</code> source code.  We now solely have our 27003 member running.  If you wait a while, it will "step down" as it does not see a majority.  Regardless, let's continue, no need to wait.  First check that only one <code>mongod</code> is running:
</p>
<pre>
$ ps -A | grep mongod
…
</pre>
<p>
Now, let's restart the two mongod's that are shut down.  If you like you can cut and paste the two relevant mongod invocations from <code>a.sh</code>.
</p><p>
Now run <code>ps</code> again and verify three are up:
</p>
<pre>
$ ps -A | grep mongod
</pre>

<p>Now, we want to see if any data that we attempted to insert isn't there.  Go into the shell to any member of the set.  Use <code>rs.status()</code> to check state.  Be sure the member is "caught up" to the latest optime (if it's a secondary).  Also on a secondary you might need to invoke <code>rs.slaveOk()</code> before doing a query.)</p>
<p>Now run:</p>

<pre>
> db.foo.find()
</pre>

to see what data is there after the set recovered from the two outages.  How many documents do you have?<customresponse>
<textbox rows='4' cols='50' correct_answer="9"> </textbox>
<answer type='loncapa/python'><![CDATA[
import re
correct = ['correct']
try:
  r = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  r = '0'
response = re.sub(r'\s+', '', r)
quiz_answer = '''9'''
quiz_answer = re.sub(r'\s+', '', quiz_answer)
if not(response == quiz_answer):
  correct[0] = 'incorrect'
  print 'response = ', response
  print 'quiz_answer = ', quiz_answer
]]>
</answer>
</customresponse>
</problem>
<vertical start='Dec 20 23:00 EST 2012' end='2012-12-25T00:00'>
<video display_name='Answer' youtube='1.0:GAgVTWRZmdQ'/>
</vertical>
</sequential>
<sequential graded='true' display_name='Final.2' chapter_name='Final' format='Final' due='Dec 20 23:00 EST 2012'>
<problem showanswer='due' end='2012-12-25T00:00' url_name='Final_2_problem' rerandomize='never' display_name='Final: Final.2'>
<p>
Let's do that again with a slightly different crash/recover scenario for each process.  With all three members (mongod's) up and running, once again:
</p>
<pre>
$ mongo --shell --port 27003 a.js
> // be sure 27003 is the primary. 
> // use rs.stepDown() elsewhere if it isn't.
> testRollback()
</pre>
<p>
Now this time, shut down the <code>mongod</code> on port 27003 (in addition to the other two members being shut down by <code>testRollback()</code> already) before doing anything else: 
</p>
<pre>
$ ps -A | grep mongod
$ # should see the 27003 one running (only)
$ killall mongod
$ # wait a little for the shutdown perhaps...then:
$ ps -A | grep mongod
$ # should get that none are present…
</pre>
<p>
Now restart just the 27001 and 27002 members.  Wait for them to get healthy -- check this with <code>rs.status()</code> in the shell.  Then query
</p>
<pre>
> db.foo.find()
</pre>

Then add another document:

<pre>
> db.foo.insert({_id:"last"})
> db.getLastErrorObj(2)
</pre>
<p>
After this (be sure the above worked first), restart the third set member (<code>mongod</code> on port 27003).  Wait for it to come online and enter a health state (secondary or primary).  
</p><p>

Run (on any member -- try multiple if you like) :
</p>
<pre>
> db.foo.find()
</pre>
<p>
You should see a difference from problem 1 in the result above.
</p>
<p>Question: which one of the following is the true statement about mongodb's operation in these scenarios?
</p><multiplechoiceresponse direction='vertical'>
<choicegroup type='MultipleChoice'>
<choice location='bottom' correct='true'><text>Mongo preserves the order of writes in a collection in its consistency model.  In this problem, 27003's oplog was effectively a "fork" and to preserve write ordering a rollback was necessary during 27003's recovery phase.</text></choice>
<choice location='bottom' correct='false'><text>The MongoDB primary does not write to its datafiles until a majority acknowledgement comes back from the rest of the cluster.  When 27003 was primary, that was never received for writes 7,8,9.</text></choice>
<choice location='bottom' correct='false'><text>When 27003 came back up, it transmitted its write ops that the other members had not seen yet to those members so that they would have them also.</text></choice>
</choicegroup>
</multiplechoiceresponse>
</problem>
<vertical start='Dec 20 23:00 EST 2012' end='2012-12-25T00:00'>
<video display_name='Answer' youtube='1.0:qUcXqarMwdg'/>
</vertical>
</sequential>
<sequential graded='true' display_name='Final.3' chapter_name='Final' format='Final' due='Dec 20 23:00 EST 2012'>
<problem showanswer='due' end='2012-12-25T00:00' url_name='Final_3_problem' rerandomize='never' display_name='Final: Final.3'>
<p>
In Final.2 the <code>mongod</code> on port 27003 does a rollback.  Go to that mongod's data directory.  Look for a <code>rollback</code> directory inside.  Find the <code>.bson</code> file there.  Run the  <a href="http://docs.mongodb.org/manual/reference/bsondump/" target="docs">bsondump</a> utility on that file.  What are its contents?
</p><multiplechoiceresponse direction='vertical'>
<choicegroup type='MultipleChoice'>
<choice location='bottom' correct='false'><text>There is no such file.</text></choice>
<choice location='bottom' correct='false'><text>The file exists but is 0 bytes long.
</text></choice>
<choice location='bottom' correct='false'><text>It contains 2 documents.
</text></choice>
<choice location='bottom' correct='true'><text>It contains 3 documents.</text></choice>
<choice location='bottom' correct='false'><text>It contains 4 documents.
</text></choice>
<choice location='bottom' correct='false'><text>It contains 8 documents.
</text></choice>
</choicegroup>
</multiplechoiceresponse>
</problem>
<vertical start='Dec 20 23:00 EST 2012' end='2012-12-25T00:00'>
<video display_name='Answer' youtube='1.0:BFX7r5euA_U'/>
</vertical>
</sequential>
<sequential graded='true' display_name='Final.4' chapter_name='Final' format='Final' due='Dec 20 23:00 EST 2012'>
<problem showanswer='due' end='2012-12-25T00:00' url_name='Final_4_problem' rerandomize='never' display_name='Final: Final.4'>
<p>
Keep the three member replica set from the above problems running.  We've had a request to make the third member never eligible to be primary.  (The member should still be visible as a secondary.) 
</p><p>
Make that change to the set -- i.e. reconfig so that the third member can *never* be primary.  Then run:
<pre>
$ mongo --shell a.js --port 27003
</pre>
And run 
<pre>
> part4()
</pre>
And enter the result in the text box below (with no spaces or line feeds just the exact value returned).
</p>
<customresponse>
<textbox rows='4' cols='50' correct_answer="233"> </textbox>
<answer type='loncapa/python'><![CDATA[
import re
correct = ['correct']
try:
  r = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  r = '0'
response = re.sub(r'\s+', '', r)
quiz_answer = '''233'''
quiz_answer = re.sub(r'\s+', '', quiz_answer)
if not(response == quiz_answer):
  correct[0] = 'incorrect'
  print 'response = ', response
  print 'quiz_answer = ', quiz_answer
]]>
</answer>
</customresponse>
</problem>
<vertical start='Dec 20 23:00 EST 2012' end='2012-12-25T00:00'>
<video display_name='Answer' youtube='1.0:Ppw8_pzQQSs'/>
</vertical>
</sequential>
<sequential graded='true' display_name='Final.5' chapter_name='Final' format='Final' due='Dec 20 23:00 EST 2012'>
<problem showanswer='due' end='2012-12-25T00:00' url_name='Final_5_problem' rerandomize='never' display_name='Final: Final.5'>
<p>
Suppose we have blog posts in a (not sharded*) postings collection, of the form: 
<pre>
{
  _id : …,
  author : 'joe',
  title : 'Too big to fail',
  text : …,
  tags : [ 'business', 'finance' ],
  when : ISODate("2008-11-03"),
  views : 23002,
  votes : 4,
  voters : ['joe', 'jane', 'bob', 'somesh'],
  comments : [
    { commenter : 'allan', 
      comment : 'Well, i don't think so…', 
      flagged:false, plus:2 },
    ...
  ]
}
</pre>
Which of these statements is true?  Note: to get a multiple answer question right in this final you must get all the components right, so even if some parts are simple, take your time.
</p><p>
*Certain restrictions apply to unique constraints on indexes when sharded, so I mentioned to be clear.
</p>
<choiceresponse direction='vertical'>
<checkboxgroup type='MultipleChoice'>
<choice correct='true'>
<startouttext/>We can create an index to make the query fast/faster:
<pre>db.postings.find( { "comments.flagged" : true } )</pre>


<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>One way to assure people vote at most once per posting is to use this form of update:
<pre>
db.postings.update( 
  { _id:… }, 
  { $inc : {votes:1}, $push : {voters:'joe'} } );
</pre>
combined with an index on { voters : 1 } which has a unique key constraint.


<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>One way to assure people vote at most once per posting is to use this form of update:
<pre>
db.postings.update( 
  { _id:… , voters:{$ne:'joe'} },
  { $inc : {votes:1}, $push : {voters:'joe'} } );
</pre>
<endouttext/>
</choice>
</checkboxgroup>
</choiceresponse>
</problem>
<vertical start='Dec 20 23:00 EST 2012' end='2012-12-25T00:00'>
<video display_name='Answer' youtube='1.0:cz8JicTQ39A'/>
</vertical>
</sequential>
<sequential graded='true' display_name='Final.6' chapter_name='Final' format='Final' due='Dec 20 23:00 EST 2012'>
<problem showanswer='due' end='2012-12-25T00:00' url_name='Final_6_problem' rerandomize='never' display_name='Final: Final.6'>
<p>Which of these statements is true? 
</p><p>
Note: to get a multiple answer question right in this final you must get all the components right, so even if some parts are simple, take your time.</p><choiceresponse direction='vertical'>
<checkboxgroup type='MultipleChoice'>
<choice correct='true'>
<startouttext/>MongoDB supports atomic operations on individual documents.
<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>MongoDB has a Date/DateTime datatype.
<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>MongoDB (v2.2) supports transactions spanning multiple documents, if those documents all reside on the same shard.<p />
<endouttext/>
</choice>
</checkboxgroup>
</choiceresponse>
</problem>
<vertical start='Dec 20 23:00 EST 2012' end='2012-12-25T00:00'>
<video display_name='Answer' youtube='1.0:7GbpkLKH_q0'/>
</vertical>
</sequential>
<sequential graded='true' display_name='Final.7' chapter_name='Final' format='Final' due='Dec 20 23:00 EST 2012'>
<problem showanswer='due' end='2012-12-25T00:00' url_name='Final_7_problem' rerandomize='never' display_name='Final: Final.7'>
Which of these statements is true? <choiceresponse direction='vertical'>
<checkboxgroup type='MultipleChoice'>
<choice correct='false'>
<startouttext/>MongoDB is "multi-master" -- you can write anywhere, anytime.
<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>MongoDB supports reads from slaves/secondaries that are in remote locations.
<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>Most MongoDB queries involve javascript execution on the database server(s).
<endouttext/>
</choice>
</checkboxgroup>
</choiceresponse>
</problem>
<vertical start='Dec 20 23:00 EST 2012' end='2012-12-25T00:00'>
<video display_name='Answer' youtube='1.0:vIiS7APVq3c'/>
</vertical>
</sequential>
<sequential graded='true' display_name='Final.8' chapter_name='Final' format='Final' due='Dec 20 23:00 EST 2012'>
<problem showanswer='due' end='2012-12-25T00:00' url_name='Final_8_problem' rerandomize='never' display_name='Final: Final.8'>
<p>
We have been asked by our users to pull some data from a previous database backup of a sharded cluster.  They'd like us to set up a temporary data mart for this purpose, in addition to answering some questions from the data.  The next few questions involve this user request.
</p><p>
First we will restore the backup.  Download <a href="/static/handouts/gene_backup.zip">gene_backup.zip</a> from the education site.  Unzip this to a temp location on your computer.
</p><p>
The original cluster that was backed up consisted of two shards, each of which was a three member replica set.  The first one named "s1" and the second "s2".  We have one mongodump (backup) for each shard, plus one of one of the config databases.  After you unzip you will see something like this:
</p>
<pre>
$ ls -la bak
total 0
drwxr-xr-x   5 dwight  staff  170 Dec 11 13:47 .
drwxr-xr-x  17 dwight  staff  578 Dec 11 13:49 ..
drwxr-xr-x   4 dwight  staff  136 Dec 11 13:45 config_server
drwxr-xr-x   5 dwight  staff  170 Dec 11 13:46 s1
drwxr-xr-x   5 dwight  staff  170 Dec 11 13:46 s2
</pre>
<p>
Our data mart will be temporary, so we won't need more than one mongod per shard, nor more than one config server (we are not worried about downtime, the mart is temporary).
</p>
<p>
As a first step, restore the config server backup and run a mongod config server instance with that restored data.  The backups were made with mongodump.  Thus you will use the mongorestore utility to restore.
</p>
<p>
Once you have the config server running, confirm the restore of the config server data by running the last  javascript line below in the mongo shell, and entering the 4 digit result it returns.
</p>
<pre>
$ mongo localhost:27019/config
configsvr> 
configsvr> db
config
configsvr> db.chunks.find().sort({_id:1}).next().lastmodEpoch.toString().substr(20,4)
</pre><customresponse>
<textbox rows='4' cols='50' correct_answer="6554"> </textbox>
<answer type='loncapa/python'><![CDATA[
import re
correct = ['correct']
try:
  r = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  r = '0'
response = re.sub(r'\s+', '', r)
quiz_answer = '''6554'''
quiz_answer = re.sub(r'\s+', '', quiz_answer)
if not(response == quiz_answer):
  correct[0] = 'incorrect'
  print 'response = ', response
  print 'quiz_answer = ', quiz_answer
]]>
</answer>
</customresponse>
</problem>
<vertical start='Dec 20 23:00 EST 2012' end='2012-12-25T00:00'>
<video display_name='Answer' youtube='1.0:uacZCQaHFWM'/>
</vertical>
</sequential>
<sequential graded='true' display_name='Final.9' chapter_name='Final' format='Final' due='Dec 20 23:00 EST 2012'>
<problem showanswer='due' end='2012-12-25T00:00' url_name='Final_9_problem' rerandomize='never' display_name='Final: Final.9'>
<p>Now that the config server from question #8 is up and running, we will restore the two shards ("s1" and "s2").
</p><p>
Use <code>mongorestore</code> to restore the data for each shard.  The shards were mongodump'd with the <code>--oplog</code> option, so you will want to use <code>--oplogReplay</code> on your <code>mongorestore</code> command line on each of these restores (note we didn't use that for the config server restore as config servers are not replica sets).
</p><p>
If we inspect our restored config db, we see this in db.shards:
</p>
<pre>
~/dba/final $ mongo localhost:27019/config
MongoDB shell version: 2.2.0
connecting to: localhost:27019/config
configsvr> db.shards.find()
{ "_id" : "s1", "host" : "s1/genome_svr1:27501,genome_svr2:27502,genome_svr2:27503" }
{ "_id" : "s2", "host" : "s2/genome_svr4:27601,genome_svr5:27602,genome_svr5:27603" }
</pre>
<p>
From this we know when we run a <code>mongos</code> for the cluster, it will expect the first shard to be a replica set named "s1", and the second to be a replica set named "s2", and also to be able to be able to resolve and connect to at least one of the seed hostnames for each shard.  If we were restoring this cluster as "itself", it would be best to assign the hostnames "genome_svr1" etc. to the appropriate IP addresses in DNS, and not change <code>config.shard</code>.  However, for this problem, our job is not to restore the cluster, but rather to create a new temporary datamart initialized with this dataset.</p>
<p>
Thus instead we will update the config.shards metadata to point to the locations of our new shard servers.  Update the config.shards collection such that your output is:</p>
<pre>
configsvr> db.shards.find()
{ "_id" : "s1", "host" : "localhost:27501" }
{ "_id" : "s2", "host" : "localhost:27601" }
configsvr> 
</pre>
<p>Be sure when you do this nothing is running except the single config server.  mongod and mongos processes cache metadata, so this is important.  After the update restart the config server itself for the same reason.</p>
<p>Now start a mongod for each shard -- one on port 27501 for shard "s1" and on port 27601 for shard "s2".  At this point if you run ps you should see three mongod's -- one for each shard, and one for our config server.  Note they need not be replica sets, but just regular mongod's, as we did not begin our <code>host</code> string in config.shards with <code><i>setname</i>/</code>.
</p><p> Now start a <code>mongos</code> for the cluster.  Connect to the mongos with a <code>mongo</code> shell.  Run this:
</p>
<pre>
> use snps
> db.elegans.aggregate([{$match:{N2:"T"}},{$group:{_id:"$N2",n:{$sum:1}}}]).result[0].n
</pre>
<p>Enter the number output for <code>n</code>.</p><customresponse>
<textbox rows='4' cols='50' correct_answer="47664"> </textbox>
<answer type='loncapa/python'><![CDATA[
import re
correct = ['correct']
try:
  r = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  r = '0'
response = re.sub(r'\s+', '', r)
quiz_answer = '''47664'''
quiz_answer = re.sub(r'\s+', '', quiz_answer)
if not(response == quiz_answer):
  correct[0] = 'incorrect'
  print 'response = ', response
  print 'quiz_answer = ', quiz_answer
]]>
</answer>
</customresponse>
</problem>
<vertical start='Dec 20 23:00 EST 2012' end='2012-12-25T00:00'>
<video display_name='Answer' youtube='1.0:_U3Xok683Y4'/>
</vertical>
</sequential>
<sequential graded='true' display_name='Final.10' chapter_name='Final' format='Final' due='Dec 20 23:00 EST 2012'>
<problem showanswer='due' end='2012-12-25T00:00' url_name='Final_10_problem' rerandomize='never' display_name='Final: Final.10'>
<p>Now, for our temporary datamart, once again from a mongo shell connected to the cluster:</p>
<br/>
1) create an index {N2:1,mutant:1} for the snps.elegans collection.
<br/>
2) now run: 
<pre>
mongos> db.elegans.find({N2:"T",mutant:"A"}).limit(5).explain()
</pre>
<p>Based on the explain output, which of the following statements below are true?</p><choiceresponse direction='vertical'>
<checkboxgroup type='MultipleChoice'>
<choice correct='false'>
<startouttext/>5 documents are scanned.<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>7 documents are scanned.<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>10 documents are scanned.<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>Thousands of documents are scanned.<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>No shards are queried.<endouttext/>
</choice>
<choice correct='false'>
<startouttext/>1 shard is queried.<endouttext/>
</choice>
<choice correct='true'>
<startouttext/>2 shards are queried.<endouttext/>
</choice>
</checkboxgroup>
</choiceresponse>
</problem>
<vertical start='Dec 20 23:00 EST 2012' end='2012-12-25T00:00'>
<video display_name='Answer' youtube='1.0:0Byk-W3n_AQ'/>
</vertical>
</sequential>
<sequential graded='true' display_name='Final.11' chapter_name='Final' format='Final' due='Dec 20 23:00 EST 2012'>
<problem showanswer='due' end='2012-12-25T00:00' url_name='Final_11_problem' rerandomize='never' display_name='Final: Final.11'>
<p>Grab <a href="/static/handouts/problem11.json">problem11.json</a> from the education site.  Import it into a mongo database (the cluster from the last problem will work fine but an empty configuration will work as well):</p>
<pre>
mongoimport -d snps -c problem11 problem11.json
</pre>
<p>Now, using the aggregation framework, answer the question: how many unique N2/mutant pairs are there in this collection?  (Assume the values as case sensitive - so 'a' and 'A' are different values for uniqueness purposes.)
</p>
<customresponse>
<textbox rows='4' cols='50' correct_answer="31"> </textbox>
<answer type='loncapa/python'><![CDATA[
import re
correct = ['correct']
try:
  r = str(submission[0])
except ValueError:
  correct[0] = 'incorrect'
  r = '0'
response = re.sub(r'\s+', '', r)
quiz_answer = '''31'''
quiz_answer = re.sub(r'\s+', '', quiz_answer)
if not(response == quiz_answer):
  correct[0] = 'incorrect'
  print 'response = ', response
  print 'quiz_answer = ', quiz_answer
]]>
</answer>
</customresponse>
</problem>
<vertical start='Dec 20 23:00 EST 2012' end='2012-12-25T00:00'>
<video display_name='Answer' youtube='1.0:2s85vRLfTc0'/>
</vertical>
</sequential>
</chapter>
</course>