Custom import adapter data validation issue

[jaimin-ktpl]

Preconditions (*)

1. Magento 2.3
2. PHP 7.1.22

Steps to reproduce (*)

1. Create a custom import adapter extending Magento\ImportExport\Model\Import\Entity\AbstractEntity
2. Add valid columns in getValidColumnNames() function.
   e.g. ['column1', 'column2', 'column3']
3. Create CSV file with columns stated above and add data.
4. Go to System > Import
5. Select your adapter from "Entity Type" dropdown
6. Select your CSV file in "Select File to Import" field.
7. Click on "Check Data" button

Expected result (*)

1. CSV file should get validated if column and data are available

Actual result (*)

1. Displays error Notice: Undefined index: sku in /var/www/html/hamtc/vendor/magento/module-import-export/Model/Import/Entity/AbstractEntity.php on line 411

Other

I have checked and found that function Magento\ImportExport\Model\Import\Entity\AbstractEntity::_saveValidatedBunches() checks for row data validation and on line 411 $skuSet[$rowData['sku']] = true; code is added.
It checks for sku column in CSV data which is not always there in every adapter.
If I add dummy sku column and dummy sku data in my CSV, the file gets validated without errors.

[magento-engcom-team]

Hi @jaimin-ktpl. Thank you for your report.
To help us process this issue please make sure that you provided the following information:

• Summary of the issue
• Information on your environment
• Steps to reproduce
• Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento-engcom-team give me $VERSION instance

where $VERSION is version tags (starting from 2.2.0+) or develop branches (for example: 2.3-develop).
For more details, please, review the Magento Contributor Assistant documentation.

@jaimin-ktpl do you confirm that you was able to reproduce the issue on vanilla Magento instance following steps to reproduce?

• yes
• no

[magento-engcom-team]

Hi @engcom-backlog-nazar. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: 👇

• 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).

  Details

  If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

• 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

• 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• 4. Verify that the issue is reproducible on 2.3-develop branch

  Details

  - Add the comment @magento-engcom-team give me 2.3-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.3-develop branch, please, add the label Reproduced on 2.3.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

• 5. Verify that the issue is reproducible on 2.2-develop branch.

  Details

  - Add the comment @magento-engcom-team give me 2.2-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.2-develop branch, please add the label Reproduced on 2.2.x

• 6. Add label Issue: Confirmed once verification is complete.

• 7. Make sure that automatic system confirms that report has been added to the backlog.

[magento-engcom-team]

@engcom-backlog-nazar Thank you for verifying the issue. Based on the provided information internal tickets MAGETWO-97146, MAGETWO-97147 were created

[meandor-ua]

@jaimin-ktpl current fixes still don't provide complete backward compatibility with code designed for Magento 2.0.X, 2.1.X, 2.2.X.
IMO it is not good at all to have any variables related to SKU in abstract entity code. In the best case new class must be implemented and extended from abstract entity.
In case it is impossible to achieve that, then it would be probably better to introduce one more variable to the constructor. It could be something like

//...
ProcessingErrorAggregatorInterface $errorAggregator,
$uniqueField = null
) {
        $this->jsonHelper = $jsonHelper;
        $this->_importExportData = $importExportData;
        $this->_resourceHelper = $resourceHelper;
        $this->string = $string;
        $this->errorAggregator = $errorAggregator;

        foreach ($this->errorMessageTemplates as $errorCode => $message) {
            $this->getErrorAggregator()->addErrorMessageTemplate($errorCode, $message);
        }

        $entityType = $config->getEntityType($this->getEntityTypeCode());

        $this->_entityTypeId = $entityType->getEntityTypeId();
        $this->_dataSourceModel = $importData;
        $this->_connection = $resource->getConnection();
        $this->uniqueField = $uniqueField;

Then 'sku' could be injected for all core import classes via di.xml.

In _saveValidatedBunches() we could rename $skuSet into $uniqueFieldSet.

        $bunchSize = $this->_importExportData->getBunchSize();
        $uniqueFieldSet = [];
//...
                try {
                    $rowData = $source->current();
                    if ($this->uniqueField !== null) {
                        $uniqueFieldSet[$rowData[$this->uniqueField]] = true;
                    }
                } catch (\InvalidArgumentException $e) {
                    $this->addRowError($e->getMessage(), $this->_processedRowsCount);
                    $this->_processedRowsCount++;
                    $source->next();
                    continue;
                }
//...
if ($this->uniqueField !== null) {
    $this->_processedEntitiesCount = count($uniqueFieldSet);
}

In case it is impossible to go with null as default value for $uniqueField, then we could set 'sku' there,

//...
ProcessingErrorAggregatorInterface $errorAggregator,
$uniqueField = 'sku'
) {

but then in custom code all developers would have to override that 'uniqueField' in di.xml to make custom import code operational again.

Current fix
$this->_processedEntitiesCount = (count($skuSet)) ? : $this->_processedRowsCount;
can also break custom import. At least it breaks my import, which was written basing on Magento\CatalogImportExport\Model\Import\Product where $this->_processedEntitiesCount is calculated within validateRow() method. validateRow() is triggered from _saveValidatedBunches(), which means that
$this->_processedEntitiesCount = (count($skuSet)) ? : $this->_processedRowsCount;
would override everything after all validateRow(). Similar as it is in M 2.3 for Magento\CatalogImportExport\Model\Import\Product. The only difference that 'sku' is present in CSV for product import and
$this->_processedEntitiesCount = (count($skuSet)) ? : $this->_processedRowsCount;
is valid in this case. But for custom imports written earlier $this->_processedEntitiesCount had to be calculated in the custom import model extended from Magento\ImportExport\Model\Import\Entity\AbstractEntity .

Currently to make my code compatible with M 2.3 I had to move code of _saveValidatedBunches() from Magento 2.2 to my custom import model. With fixes, which we have now the problem is still not solved.

[magento-engcom-team]

Hi @jaimin-ktpl. Thank you for your report.
The issue has been fixed in #19765 by @jaimin-ktpl in 2.3-develop branch
Related commit(s):

• 6b9aa47cc9bb2574360858802444e546b091dbf9
• 939b665d3b2b8e0f717c61a8c746b6a602156724

The fix will be available with the upcoming 2.3.2 release.

[magento-engcom-team]

Hi @jaimin-ktpl. Thank you for your report.
The issue has been fixed in #22180 by @amol2jcommerce in 2.2-develop branch
Related commit(s):

• d23dd1ed5df664dcfbe79106058049e7e0b1e7fc
• 60034d5540bb959c5bf724b0f18c9fb962dc49e4

The fix will be available with the upcoming 2.2.10 release.