Full and Incremental Extracts from Symphony

This page is to document the process for Full and Incremental extracts processed from Symphony and how they are loaded to BlackCat Server.

What is included in a Symphony extract for BlackCat?

Everything from Sirsi,

• Excluding shadowed records but not temporary records. (Most of the temporary records are shadowed.)

• Review records will be excluded by default since they don't have any items associated with them (extraction begins at the item level)

• Excluding item types:
  CARRELKEY, EBOOKREADR, EQUIP14DAY, EQUIP24FEE, EQUIP24HR, EQUIP3DAY, EQUIP4HR, EQUIP5DAY, EQUIP7DAY, ILL, LAPTOP, PALCI

• Excluding Junk tags: Generated a custom junk tag list for Blackcat (junktag_blackcat on Symphony):

  001, 029, 090, 593, 594, 595, 596, 597, 598, 599, 799, 900, 901, 920, 921, 926, 927, 930, 948, 949, 960, 961, 977, 980, 990, 994, 998, 999
  

• Bound-with extract:
  selects all the call numbers which are a bound-with child with their parent's holdings information, title and catkey of that bound-with child. The parent's title and catkey is appended and exported as a 591 field and the parent's holding information as a 949 to the child's extracted record.

• Holdings extract:
  The 949 field is used for current holdings.

• Marc Holdings extract -Summary holdings will be included in the extract. The fields 852 866 867 868 are extracted

• Records that exceeded the maximum export size to be catalog dumped

• Records that exceeded the maximum number of holdings to be catalog dumped: Split these catalog record into smaller bib records

Full Extracts

1. Full extract can start anytime after adutext finishes it's daily run of clearing out the Database Edit directory.

2. On the Symphony side, shut off the sirsi cron job that sends incremental records over to Blackcat.

   #5,35 * * * * /sirsi/Psuscripts/Scripts/move_edited_records_to_blackcat >> /sirsi/Unicorn/PSULinkages/Blackcat/cron_log 2> /sirsi/Unicorn/PSULinkages/Blackcat/cron_error_log
   

3. On blackcat QA server

   $ sudo -i
   $ cd /data/symphony_data/full_extracts
   $ mkdir <extract date>
   $ chown ansible_deploy_bot:sirsi <extract date>
   $ chmod 775 <extract date>
   

4. Extract all the Symphony MARC data and secure copy it to /data/symphony_data/full_extracts/<extract date> directory on the Blackcat server. The full extract creates the following files:

   blackcat_<date>_<#>.mrc
   blackcat_<date>_bound_<#>.mrc
   blackcat_<date>_max_exceed.mrc
   blackcat_<date>_holdings_exceed.mrc
   

5. Run the full index on the Blackcat server.

6. On the Symphony side, turn on the Sirsi cron job that sends records over to Blackcat.

   #5,35 * * * * /sirsi/Psuscripts/Scripts/move_edited_records_to_blackcat >> /sirsi/Unicorn/PSULinkages/Blackcat/cron_log 2> /sirsi/Unicorn/PSULinkages/Blackcat/cron_error_log
   

Incrementals

The incrementals are the add/updates and delete extracts from Symphony. The process is similar to full extract.

TBD

• What happens if an item changes libraries? For instance, if a BOOKFLOAT item is checked into a different library, the item library changes (ie, the library changes). Since any change to the library (item library in Workflows) forces a re-index of the catalog record by putting the catalog key into /sirsi/Unicorn/Database/Edit, the incremental extracts will pick up records for re-indexing.

Translation Maps

TBD

---

Note: These processes will be elaborated as the process is fully tested and implemented.