The Migrate module is a powerful tool for importing data into Drupal and Drupal modules. It is also an approach we have been using for large, complicated or recurring imports into CiviCRM. Some advantages of using Migrate module for imports:
- You can specify how many (and which) records you wish to import in order to do a small test.
- You can then make changes and remove / re-import the records until you are happy
- You can manipulate data on the way in using hooks
- You can automate the import using drush
- You can run the update on a regular basis via cron & new additions will be picked up (this is a technique we are using to get 3 click csv uploads into multiple Civi types
- It is easy to visualise what is happening & come back to it over a period of time
- Advantages of CiviMigrate are the ability to import related entities - eg memberships with related contributions from one source
In this blog we will look at a very basic contact import & then in the following one will look at creating memberships for those contacts based on the same csv. The csv we are using is (will be) attached.
CiviMigrate does not replace existing CiviCRM migrate methods. It is good for different things. In general you need to set up a migration step per entity so if you wish to create a contact and a related contact and a relationship between them you need three steps.
Prequisites
Your data needs to be already in a view with a unique key - here is one way to get it there from csv
migrate module installed & enabled (drush dl migrate / drush en migrate )
Note that migrate module has table wizard as a pre-requisite but will not in the future. I have mostly switched to using Data module with it now. Schema is a pre-requisite of Table Wizard. You may wish to supress schema warnings @ admin/build/schema/settings
Civimigrate module installed & enabled. Civimigrate is currently just being hosted in the Fuzion repository although after (considerable) tidy-up it will hopefully eventually reside on drupal.org. For now you can get it by
svn co https://svn.fuzion.co.nz/repos/fuzion-code/trunk/drupal/modules/custom/civimigrate
drush en civimigrate
Getting started
Start at the migrate dashboard - admin/content/migrate/dashboard or navigate via 'content' from the admin menu.
Choose to add a set. Choose your view as the source and CiviCRM Contact as the destination. I recommend you set a weight on all steps & use a numbering scheme like 10, 20, 30 to enable you to re-order steps more easily later.
Setting up a mapping
In this next step you match your views fields to CiviCRM fields. Make sure you set the Unique ID first if necessary - in this case we are using system ID. CiviMigrate will create a table which maps this ID to the civicrm contact_id which is important for us to import related items (like contributions).
CiviMigrate uses the CiviCRM API to import data and generally the fields available for you to map to are the fields from the relevant tables with some manual additions & subtractions. In this case we will choose to map:
first_name to first_name
last_name to last_name
SystemID to external_identifier
phone to phone
optional: phone_type_id - we are going to set this to 1 - see civicrm/admin/options/phone_type&group=phone_type&reset=1 for the values. (without this it will still import but phone type won't be set).
optional: source - something to identify our import
Running the Import
Once you have saved the mappings you run the import by ticking the box next to it and opening up the execute section. You can specify to only import one record.under sample size and click run.
When I run the above I find that '0' records successfully import and I can see there is 1 error. Clicking on the error does not, however, show me the error - what's happening? When the migrate ran it created two new tables
migrate_map_contacts (my migration is called contacts)
migrate_map_msgs
The code also allows for the creation of views so we can see them but before we can access them the first time we need to flush all caches (well, I'm not sure which one so blunt hammer here). Now that I can access the table in the migrate_map_messages I see
88 Contact Type not specified
88 is the system ID for the first contact it attempted and Contact Type not Specified tells me I need to go back to my migrate map & enter 'Individual' against contact_type.
I re-do my import with another '1' record and the next record updates fine. My first record won't be touched until I choose the update existing option - but read the section on that below before you go there.
Looking at what has happened
Go to views and filter on 'tw' - you will see the migrate_map table for your import. It has a sourceid - your unique ID and a destid - the CiviCRM contact ID, and a 'needs_update' field. It can be very helpful to edit the destid field to output as link - use this link and it will link to your CiviCRM Contact
civicrm/contact/view?reset=1&cid=[destid]
When you next run an import it will look for any records in your source view that are not in your migrate_map or that are in your migrate map & have needs_update=1 against them and run the update command on them.
dupe matching
Civimigrate will always treat an external identifier match as a match - whether it's in your dedupe rule or not.
If your contact matches an existing CiviCRM contact based on your default de-dupe rule Civimigrate currently has 2 possible behaviours (only because that's all we've needed so far as we have mostly used an external_identifier for our contact import matches)
By default CiviMigrate will treat your imported contact as a match for the CiviCRM contact with the lowest ID and create any that don't exist.
If you choose 'Only update (don't create) and only if ONLY one match is found' CiviCRM will only update matches and not create any new contacts (existing ID is used). To choose this put 1 in the default for this field. This option is often valid when you don't want to import contacts at all - just contributions but you need to map up the contact_id first - see below for that.
Not hugely useful but you can tell CiviCRM to consider the logged in users permissions when determining a match by choosing "Do you want the current users's permissions used to limit dupe matching?(default = False)"
Changing your import - deletes
You can use 'clear' to delete your records if you wish to re-do them. Most probably if you are using a 3.2 version of CiviCRM this will cause the contact to be soft-deleted but 3.1 & 3.3 should cause hard deletes (desired in this case). Be careful if using an existing data set as contacts that have been matched by de-dupe will be deleted!
Updates & mapping contact ids
If you wish to do an update there is a very important step to do first (this is less important for contacts where matches can be done based on dedupe & external identifiers than it is for other civimigrate entities but let's do it here as good practice and prevents 'odd' things happening).
A word first about table wizard & data. Table Wizard is deprecated but I have been unable to create the joins described here using Data module with Migrate 1. I believe it is because Table Wizard manages the migrate_map tables and you need Data to manage them in order to use Data
Table Wizard Relationship
Go to the Table Wizard dashboard admin/content/tw - choose Add existing tables and add your source table. Then go to 'Relationships' admin/content/tw/relationships. Add a relationship from (the top field / base table) the unique ID of your source table to (bottom field) the equivalent of migrate_map_contacts.sourceid. The Automatic setting is fine.
Each time you accidentally connect to the migrate_msg_contacts instead log onto CiviCRM & make a small donation - this will save you time in the long run.
Once the relationship is added sort out your view
Now go back to your source view (the one based on your data table) stopping only to clear your views cache. Now try to add fields. You should now see the fields from the migrate_map_contacts table. Remember, your destid is your CiviCRM contact_id. Add migrate_map_contacts.destid to the view and label it contact_id (and probably link it to civicrm/contact/view?reset=1&cid=[destid] )
Update your migrate map
Finally, go back to your migrate job map and map this new contact_id field to the contact_id field. When you update your records now there will be no danger of a new contact being created. For contributions mapping the id field is absolutely essential
Updating records
Once you have your contact ID mapped up you can change your mapping and then click 'update existing' to update previously migrated records with your new settings. This is especially useful if you have changed your mind about where to map a particular value to.
Caution selecting 'update previously-imported content' sets the value needs_update to 1 for ALL records in the migrate_map table. Even if you have set your import to update 1 record next time you run it it will try to update all the others as needs_update is set to 1
Where to?
These were the basics of importing a contact using CiviMigrate but the real power is related transactions. The next blog will explain how to import our next 2 fields 'amount' & date as a membership & a related contribution.