Delta changes not migrated #316
Comments
Did you make a comparison what exactly records are missing? for example in customer_entity table of M1 and M2 stores |
|
The m1 db has higher number of records and m2 db has lower number of records. MariaDB> use m1; MariaDB> use m2; Also all of our m2_cl tables became empty on M1 DB, though there was no delta migration happened. |
|
Usually for every new record in customer_entity of M1 new record to m2_cl_customer_entity should be added by MySQL triggers. Check this table |
|
Please see below what we did:
Could you please confirm whether this approach is correct? Thanks, |
|
Not really correct. Normally, the tool must connect to Live M1. As soon as you run Now, to fix delta you can fill m2_cl_* tables (find the list in deltalog.xml.dist) with missing records and run
|
|
@victor-v-rad - Thanks a lot for your suggestion. I believe your solution will fix our issue. |
|
@victor-v-rad I've applied the changes you suggested, and I still getting the same errors perhaps seems the delta migration is working more on something. Just to be sure, I filled the m2_cl tables with the INTs of column marked as PRIMARY and as "operation" I used "INSERT" then "processed" as 0 e.g. After that I run the delta migration, the records are disappearing from the m2_cl_ tables, I guess because processed, but the rest of the DB seems the same and in the Admin area the latest customer and order is the same there was before the delta migration. |
|
@victor-v-rad moreover I repopulated the |
|
@victor-v-rad I guess it was a matter of cache, now all the new data are showing in the Admin area. 👍 |
|
@victor-v-rad Even though the delta migration is successful, we got the warnings "Mismatch in last increment id" [2017-06-13 14:03:09][INFO][mode: delta][stage: delta delivering][step: SalesIncrement Step]: started We checked both M1 and M2 tables and found that the Below is the full delta migration log.
[2017-06-13 14:02:53][INFO][mode: delta][stage: delta delivering][step: Custom Customer Attributes Step]: started Thanks, |
|
The SalesIncrement delta step takes data from eav_entity_store table of M1 and migrate last increment ids into sequence_order, sequence_invoice, sequence_creditmemo, sequence_shipment, sequence_rma_item tables. Make sure you filled m2_cl_eav_entity_store table with appropriate data. |
|
@croxarens @rg-srini Hi I have the same problem (I made a copy of the live database and made a migration on that so delta is now not working). And since you two have seem to solve the problem? I am wondering if there is any possibility to share the sql script made to repopulate the m2_cl tables (I am guessing you didn't do it manually) so I don't have to write it from scrach. I would really appreciate it. |
|
@victor-v-rad - The suggested solution worked fine now we see the delta data on the admin. Thanks a lot. 👍 |
|
Glad it worked out ) |
|
@croxarens @rg-srini asking you both again if you could share the sql-s created to fill the missing m2_cl tables data I made the same mistake of transfering data from a dump of the original database and would now like to do the delta migration directly to original database and any existing sql would really help me speed the process.. If you are willing to share.. |
|
@MoMannn - See below the steps what we did.
|
|
@rg-srini I made a dump of m1 database and migrated to m2. Now, after a month or so I would like to migrate the changes that where made to m1 in the mean time. I ran the delta migration directly on m1 so that the tables m2_* and triggers are created but the delta doesn't run trough because of the id problem. So I figured I need to populate the m2_* tables with the missings ids from the live m1 database so I could establish a delta connection. But I'm not sure your answer does that since you are dropping triggers and importing the latest dump? or am I missing something? |
|
@MoMannn Then you can run the delta migration. |
|
Ok. |
|
All the m2_* tables should be empty, so when you run the delta migration the records there are processed accordingly and removed from the m2_* table. The m2_* tables have been created during the data migration |
|
ok, so I should first empty all the data inside the m2_* tables Is there any document that shows which m2_* tables points to which magento2 tables? So I can check the last id in the magento2 corresponding table. And I am guessing I should make updates to all the customers aswell since I don't know which where changed. Thank you in advance for the info. |
|
The m2_* tables where all empty after the data migration, so I populated them accordingly. e.g. |
|
My m2_* tables have some data inside for some reason.. Maybe because I tried delta migration even tho it failed? Ah stupid me I still have the original dump I made for the data migration so I can just use that to compare with the live version. Thanks. |
|
I created a simple script that fixed this problem for me by connection to the magento 1 dump and magento 1 live database and checked for changes than added them to the m2_cl_* tables as insert statements. If anyone has the same problem they can check the script here: https://github.com/MoMannn/delta_migration_script It is poorly written and slow but does the job. |
|
@MoMannn |
|
@MoMannn I was trying to use your delta_migration_Script but i don't know how to use it. I have a live site, a staging site(M v.1.9) and a dev site for M2.
++ I have a question here : I'm a quite newbie to run script file.
I will wait for your reply. |
|
This is what connect.php includes: function conm1() } function conm2() } You can put the script anywhere on the server you just need to make sure it can connect to both databases. I would advice you to connect directly M1 database and migrate via default migration tool since that makes things much simpler. |
|
Hi, Momann
Thank you so much for your reply.
As a matter of fact, i migrated M1 db direclty and it works.
And i was really frustrated about the document about migration is not quite
in detail.
But when i received your email, that feeling disappear although i'm still
struggled with migration work. It's about time difference issue after
migration tho.
Once again, I want to say my appreciation to you.
Have fun in Magento!
2017-10-03 11:19 GMT-04:00 MoMannn <[email protected]>:
… This is what connect.php includes:
function conm1()
{
ob_start();
$host = "host";
$upImeBaza = "username";
$gesloBaza = "password";
$imeBaze = "databaseName";
$conn = mysqli_connect($host, $upImeBaza, $gesloBaza, $imeBaze);
mysqli_set_charset($conn, 'utf8');
ob_end_flush();
return $conn;
}
function conm2()
{
ob_start();
$host = "host";
$upImeBaza = "username";
$gesloBaza = "password";
$imeBaze = "databaseName";
$conn = mysqli_connect($host, $upImeBaza, $gesloBaza, $imeBaze);
mysqli_set_charset($conn, 'utf8');
ob_end_flush();
return $conn;
}
You can put the script anywhere on the server you just need to make sure
it can connect to both databases.
I would advice you to connect directly M1 database and migrate via default
migration tool since that makes things much simpler.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#316 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ARwIwTOeVsnNPE1WJ1A04vKwNPhnRlotks5solB7gaJpZM4Nz5R9>
.
--
*Best regards,*
Mijin Kim
김 미진
+1 929 341 9285
[email protected]
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon>
Virus-free.
www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
|
|
Hi Momannn, Would you please help me? I am having exactly the same problem with yours and figured out how it works after the fact... Anyway, I used your script and this is what I got: Warning: mysqli_fetch_assoc() expects parameter 1 to be mysqli_result, null given in /srv/www/mydomain.com/delta.php on line 321 Can you please tell me what I did wrong? Thanks so much for your help! Don |
|
I created a magerun command to retro-actively populate the m2_cl tables in the M1 installation; https://github.com/peterjaap/magerun-addons#create-changelog-entries-for-magento-2-data-migration-tool-delta-migration Here are the queries to create the m2_cl tables in M1 db (not the triggers though); https://gist.github.com/peterjaap/97431fddad0e2015fccbfd961f15d239 |
|
@peterjaap i'm trying your magerun addons to populate the m2_cl tables in my M1 database but when i run the n98-magerun.phar m2-migration:delta-update-changelog command i get the "There are no commands defined in the "m2-migration" namespace" error. And this go for all the other commands. I guess it is a misconfiguration problem can you get me on the right way pls? I installed your module following the read.me instruction in the .n98-magerun/modules/ directory, i'm using the Magento1 magerun version. What i'm missing? Thank you |
|
Came looking for a solution to the same problem. |
|
Can you confirm delta command working fine in migrate product information M1 into M2? |
|
I have not tried the script above |
|
Hi, |
I have tried your script but I think it is not working in my case. Let me explain what we did: we have freshly installed 2.3.5-p1 and we have connected it to our live M1 database and performed first data migration and all the data and settings were migrated successfully. Approximately 2 -3 weeks after doing first data migration we have placed 3 test orders in the M2 website and additionally client created some products in the M2. after that we tried to perform a delta migration but we got an integrity check error. So we have deleted new products which we suspect was created by the client in M2 and attempted delta migration again. the delta migration executed with so many warnings but without any errors. We thought the delta migration was successful but new data was not migrated into M2 at all. So, we have created two databases, in one DB we have dumped the DB of live M1 site and in another one, we have dumped DB of M1 website which we have backed up after first data migration and before delta migration and I executed your script between these two databases. but when I check the databases I noticed that no changes were moved to any of the databases. what could be happened in this situation? |
|
@MoMannn Your script is working well for us, thank you. (Need to do few changes in code but it's working) |
|
@Vishrootways I only added those that I needed for my project. Feel free to update and modify the script to your needs. |
|
@Ckaneriya Well this is not really the situation for which the script was created but I believe you could modify to suite your purpose. |
|
@MoMannn Got it, thank you. Do you know how can we skip duplicate entiry error while running delta migration second time? |
|
@Vishrootways To be honest I haven't dealt with magento in 3 years so I am completely out of it. Sry I can't be of more help. |
|
Hello - this issue was reported 3 years ago. Or is there a better way to run Delta on Development Database from non-prod DB? Thank you. |
|
I have updated the script of @MoMannn to make it compatible with latest magento 2.4.4 version. I have added all tables and fix some bugs. Use updated Script |
|
Hi @MoMannn ! I need to put the script inside m1 or m2 server? Do I need to run something after that? |
We completed the migration successfully. When we try to do the delta changes we get the below warnings. And the delta changes were not at all carried out for the entities (orders, invoices, shipments, returns, credits, customers, transactions, reviews).
[2017-06-08 10:29:26][INFO][mode: delta][stage: delta delivering][step: Custom Customer Attributes Step]: started
[2017-06-08 10:29:26][INFO][mode: delta][stage: volume check][step: Custom Customer Attributes Step]: started
[2017-06-08 10:29:26][WARNING]: Mismatch of entities in the document: magento_customercustomattributes_sales_flat_order
[2017-06-08 10:29:26][WARNING]: Mismatch of entities in the document: magento_customercustomattributes_sales_flat_order_address
[2017-06-08 10:29:26][WARNING]: Mismatch of entities in the document: magento_customercustomattributes_sales_flat_quote
[2017-06-08 10:29:26][WARNING]: Mismatch of entities in the document: magento_customercustomattributes_sales_flat_quote_address
[2017-06-08 10:29:26][WARNING]: Volume Check failed
[2017-06-08 10:29:26][INFO][mode: delta][stage: delta delivering][step: Customer Attributes Step]: started
[2017-06-08 10:29:26][INFO][mode: delta][stage: volume check][step: Customer Attributes Step]: started
[2017-06-08 10:29:26][WARNING]: Mismatch of entities in the document: customer_entity
[2017-06-08 10:29:26][WARNING]: Mismatch of entities in the document: customer_entity_int
[2017-06-08 10:29:26][WARNING]: Mismatch of entities in the document: customer_entity_text
[2017-06-08 10:29:26][WARNING]: Mismatch of entities in the document: customer_entity_varchar
[2017-06-08 10:29:26][WARNING]: Mismatch of entities in the document: customer_address_entity
[2017-06-08 10:29:26][WARNING]: Mismatch of entities in the document: customer_address_entity_int
[2017-06-08 10:29:26][WARNING]: Volume Check failed
[2017-06-08 10:29:26][INFO][mode: delta][stage: delta delivering][step: Map Step]: started
[2017-06-08 10:29:26][INFO][mode: delta][stage: delta delivering][step: Log Step]: started
[2017-06-08 10:29:26][INFO][mode: delta][stage: volume check][step: Log Step]: started
[2017-06-08 10:29:26][WARNING]: Mismatch of entities in the document: customer_visitor
[2017-06-08 10:29:26][WARNING]: Volume Check failed
[2017-06-08 10:29:26][INFO][mode: delta][stage: delta delivering][step: SalesOrder Step]: started
[2017-06-08 10:29:26][INFO][mode: delta][stage: volume check][step: SalesOrder Step]: started
[2017-06-08 10:29:26][WARNING]: Mismatch of entities in the document: sales_order
[2017-06-08 10:29:26][WARNING]: Volume Check failed
[2017-06-08 10:29:26][INFO][mode: delta][stage: delta delivering][step: OrderGrids Step]: started
[2017-06-08 10:29:26][INFO][mode: delta][stage: volume check][step: OrderGrids Step]: started
[2017-06-08 10:29:26][WARNING]: Mismatch of entities in the document: sales_order_grid
[2017-06-08 10:29:26][WARNING]: Mismatch of entities in the document: sales_invoice_grid
[2017-06-08 10:29:26][WARNING]: Mismatch of entities in the document: sales_shipment_grid
[2017-06-08 10:29:26][WARNING]: Mismatch of entities in the document: sales_creditmemo_grid
[2017-06-08 10:29:26][WARNING]: Volume Check failed
[2017-06-08 10:29:26][INFO][mode: delta][stage: delta delivering][step: SalesIncrement Step]: started
[2017-06-08 10:29:26][INFO][mode: delta][stage: volume check][step: SalesIncrement Step]: started
[2017-06-08 10:29:26][WARNING]: Mismatch in last increment id of order entity
[2017-06-08 10:29:26][WARNING]: Mismatch in last increment id of invoice entity
[2017-06-08 10:29:26][WARNING]: Mismatch in last increment id of creditmemo entity
[2017-06-08 10:29:26][WARNING]: Mismatch in last increment id of shipment entity
[2017-06-08 10:29:26][WARNING]: Mismatch in last increment id of rma_item entity
[2017-06-08 10:29:26][WARNING]: Volume Check failed
[2017-06-08 10:29:26][INFO][mode: delta][stage: volume check][step: SalesIncrement Step]: Migration completed successfully
[2017-06-08 10:29:26][INFO][mode: delta][stage: volume check][step: SalesIncrement Step]: Automatic restart in 5 sec. Use CTRL-C to abort
[2017-06-08 10:29:31][INFO][mode: delta][stage: delta delivering][step: Custom Customer Attributes Step]: started
[2017-06-08 10:29:31][INFO][mode: delta][stage: volume check][step: Custom Customer Attributes Step]: started
[2017-06-08 10:29:31][WARNING]: Mismatch of entities in the document: magento_customercustomattributes_sales_flat_order
[2017-06-08 10:29:31][WARNING]: Mismatch of entities in the document: magento_customercustomattributes_sales_flat_order_address
[2017-06-08 10:29:31][WARNING]: Mismatch of entities in the document: magento_customercustomattributes_sales_flat_quote
[2017-06-08 10:29:31][WARNING]: Mismatch of entities in the document: magento_customercustomattributes_sales_flat_quote_address
[2017-06-08 10:29:31][WARNING]: Volume Check failed
[2017-06-08 10:29:31][INFO][mode: delta][stage: delta delivering][step: Customer Attributes Step]: started
[2017-06-08 10:29:31][INFO][mode: delta][stage: volume check][step: Customer Attributes Step]: started
[2017-06-08 10:29:31][WARNING]: Mismatch of entities in the document: customer_entity
[2017-06-08 10:29:31][WARNING]: Mismatch of entities in the document: customer_entity_int
[2017-06-08 10:29:31][WARNING]: Mismatch of entities in the document: customer_entity_text
[2017-06-08 10:29:31][WARNING]: Mismatch of entities in the document: customer_entity_varchar
[2017-06-08 10:29:31][WARNING]: Mismatch of entities in the document: customer_address_entity
[2017-06-08 10:29:31][WARNING]: Mismatch of entities in the document: customer_address_entity_int
[2017-06-08 10:29:31][WARNING]: Volume Check failed
[2017-06-08 10:29:31][INFO][mode: delta][stage: delta delivering][step: Map Step]: started
[2017-06-08 10:29:31][INFO][mode: delta][stage: delta delivering][step: Log Step]: started
[2017-06-08 10:29:31][INFO][mode: delta][stage: volume check][step: Log Step]: started
[2017-06-08 10:29:31][WARNING]: Mismatch of entities in the document: customer_visitor
[2017-06-08 10:29:31][WARNING]: Volume Check failed
[2017-06-08 10:29:31][INFO][mode: delta][stage: delta delivering][step: SalesOrder Step]: started
[2017-06-08 10:29:31][INFO][mode: delta][stage: volume check][step: SalesOrder Step]: started
[2017-06-08 10:29:31][WARNING]: Mismatch of entities in the document: sales_order
[2017-06-08 10:29:31][WARNING]: Volume Check failed
[2017-06-08 10:29:31][INFO][mode: delta][stage: delta delivering][step: OrderGrids Step]: started
[2017-06-08 10:29:31][INFO][mode: delta][stage: volume check][step: OrderGrids Step]: started
[2017-06-08 10:29:31][WARNING]: Mismatch of entities in the document: sales_order_grid
[2017-06-08 10:29:31][WARNING]: Mismatch of entities in the document: sales_invoice_grid
[2017-06-08 10:29:31][WARNING]: Mismatch of entities in the document: sales_shipment_grid
[2017-06-08 10:29:31][WARNING]: Mismatch of entities in the document: sales_creditmemo_grid
[2017-06-08 10:29:31][WARNING]: Volume Check failed
[2017-06-08 10:29:31][INFO][mode: delta][stage: delta delivering][step: SalesIncrement Step]: started
[2017-06-08 10:29:31][INFO][mode: delta][stage: volume check][step: SalesIncrement Step]: started
[2017-06-08 10:29:32][WARNING]: Mismatch in last increment id of order entity
[2017-06-08 10:29:32][WARNING]: Mismatch in last increment id of invoice entity
[2017-06-08 10:29:32][WARNING]: Mismatch in last increment id of creditmemo entity
[2017-06-08 10:29:32][WARNING]: Mismatch in last increment id of shipment entity
[2017-06-08 10:29:32][WARNING]: Mismatch in last increment id of rma_item entity
[2017-06-08 10:29:32][WARNING]: Volume Check failed
[2017-06-08 10:29:32][INFO][mode: delta][stage: volume check][step: SalesIncrement Step]: Migration completed successfully
Note: we have not changed anything on the M2 DB post data migration other than adding few custom customer attributes. We tried to initiate the delta migration after a week time.
Please help. We highly rely on this tool.
The text was updated successfully, but these errors were encountered: