A major challenge for CiviCRM implementers is migrating data from legacy systems into CiviCRM. For large datasets the front end import mechanisms time-out and don't allow for the complexity of the data. For the last year I have been using Migrate module in conjunction with the CiviCRM api. I have recently switched to using migrate module 2 (drupal 6 & drupal 7 compatible) with CiviCRM api v3. To do this I have written a CiviMigrate module which is a lightweight bridge between the two.
In order to use CiviMigrate for Migrate 2 you will need to write your own module. However, there is very little real code in it. In order to write a repeatable import for contacts and their contributions from MySQL tables I have used 34 lines of code.
Migrate module is a very powerful module which allows you to grab data from a variety of 'sources' such as csv, xml, oracle and mysql and map them to a variety of destinations such as node, user, commerce items, comments and ....the CiviCRM api. Some of the nice features of Migrate is that it keeps track of the mappings between the ids of your source data and your CiviCRM destination IDs and allows you to rollback your imports and try again or tweak them and run them as an update (e.g. you could add a field mapping and then re-run the import after setting 'needs_update'. Migrate module gives you several opportunities to manipulate data.
There is a lot of good documentation about migrate module and it ships will a detailed example module so I'm going to write a fairly basic explanation of using Migrate module with the CiviCRM api' as a destination. In my example I am using mysql as the source. I recommend you use this with the version of the API that ships with 4.1 (it's fairly easy to back port to 3.4/ 4.0 although you do need to change one file outside the API folder). The reason for this is that CiviCRM 'advertises' different field names in 4.1 - for example it 'advertises' 'source' rather than 'contribution_source' on the contribution entity (although it accepts both we now encourage you to use 'source').
I have written a small civimigrate example module & you can enable it along with CiviMigrate. (I stole the sample data from the beer migration example so it's a bit obscure :-). The example module imports 3 contacts and then attempts to import 4 contributions (one fails as I filtered out the contact it relates to as they are inactive). It demonstrates the fairly important technique of importing data against a contact (or other entity) created in a previous import.
For a large number of the imports I was working with I wanted to get all the fields from a base table (and then possibly join on other tables / add conditions / concatenate fields or whatever). One of the advantages of using Migrate is that it helps you to explore your data and categorise it in the code itself in a way that can be seen through the web so exposing all fields of the base table was generally useful. Since I was doing this a lot I made it pretty simple to do. Once you have your migration set up you can easily see what fields you have and haven't mapped.
The destination is a CiviCRM entity within the CiviCRM UI - you define it by saying
protected $entity = 'contact'; // this is the default - it refers to the CiviCRM entity
When you click on 'destinations' you can see all the API fields that are advertised for that entity and what, if anything, you have mapped to them
The lines to designate a base table as the source are
protected $base_table = 'civimigrate_example_people_names';
protected $base_table_id = 'aid'; // name of id field
protected $base_table_alias = 'names';
And then add these lines to generate the query - I have added a condition in here to only include active 'people'
$query = $this->getQuery($this->base_table,$this->base_table_alias,$this->base_table_id)
->condition('status', 1); // only include those with status = 1
$this->source = new MigrateSourceSQL($query);
As with the destination data you can see which fields you mapped
The three mappings in the first job are
$this->addFieldMapping('first_name', 'names_name');
$this->addFieldMapping('contact_type')->defaultValue('Individual');
$this->addFieldMapping(null, 'names_posted')
->description('Not sure what this field is for - ideas?')
->issueGroup('To discuss'); The first 2 can be seen on the 'Mapped tab'
Whereas the last has been categorised 'To discuss' and shows on that tab with the description posted
The second job, the contributions, relies on a neat trick of using the results from one migration in the next migration
$this->dependencies = array('people');
/*
* The line below says 'take the value of the person_id field and look up what it was mapped to
* during the previous peopleMigration import and use that for the contact_id
*/
$this->addFieldMapping('contact_id','payments_person_id')
->sourceMigration('people');
You can see one error on the dashboard. That's because I only imported 3 of the 4 contacts in the table (using the condition 'is_active') so one of the contributions will not import as the contact doesn't exist
This results in an error message and we would probably exclude it from our import.
Note that if you do a Drupal User import after your CiviCRM contact import using the Migrate module it will link the new drupal user accounts & the CiviCRM contacts as long as you have imported emails against both