Everything about Hot Folder in Hybris

1. Overview

The hot folder is one of the best ways to import and feed data into Hybris, it is based on Spring integration framework.

Some reasons to use hot folder :

• Easy to configure and to extend.
• Use the power of Spring integration.
• It’s very fast than the regular impex import (uses multi-threading imports).

Basically the main idea behind hot folders is to transform csv files giving in input to impexes using a pre-configured ImpexConverters and import them using ImportService (see diagram bellow).

This is a more complete version of how the hot folder works.

 

Let’s explain this diagram step by step 🙂

• Inbound Channel Adapter : is a Spring integration component that allows us to create a watcher over csv files on a specific directory.
• File Order Comparator : the Inbound Channel Adapter uses a file order comparator to treat files by order of priority, this comparator compares files based on their file names.
• Header Setup Task : in this step of the flow a BatchHeader is created/initialized with the file, catalog, and some other information, this BatchHeader will be used throughout the flow as a reference for file and some other information.
• Header Init Task : this is just another initialization step, in this step we extract the sequenceId and the language then we add them to the BatchHeader for later use.
  For example for this file name product-fr-2313213672186.csv the sequenceId is 2313213672186 and the language is fr.
• Impex Transformer Task : this is one of the important steps in the flow, basically here where the original file (csv) is converted to an impex with the help of a pre-configured ImpexConverter.
• Impex Runner Task : imports the transformed file (impex) using ImportService.importData() method.
• Cleanup Task : deletes the transformed file (impex) and archive the original file (csv).

The majority of the components that constitute the hot folder flow can be found at : …\hybris\bin\ext-accelerator\acceleratorservices\resources\acceleratorservices\integration\hot-folder-spring.xml

2. Implementation

Fortunately, the accelerator comes armed with an initial and complete configuration of the hot folder.

The ImpexConverters for prices, products, medias, stocks, customers…, are already there and good to go.

However in this article, we will try to extend the hot folder functionalities to be able to import an other item type UnitModel.

You need three macro steps to configure the hot folder :

• Define a base directory where the csv files will be put.
• Initiate your flow with the catalog and base directory…
• Create an ImpexConverter and associated with a MappingConverter.

2.1. Preparation

1. Create an xml file hot-folder-store-training-spring.xml inside the …\hybris\bin\custom\training\trainingcore\resources\trainingcore\integration

<!-- ...\hybris\bin\custom\training\trainingcore\resources\trainingcore\integration\hot-folder-store-training-spring.xml -->

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
	   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	   xmlns:int="http://www.springframework.org/schema/integration"
	   xmlns:file="http://www.springframework.org/schema/integration/file"
	   xmlns:p="http://www.springframework.org/schema/p"
	   xmlns:context="http://www.springframework.org/schema/context"
	   xsi:schemaLocation="http://www.springframework.org/schema/integration
	http://www.springframework.org/schema/integration/spring-integration.xsd
	http://www.springframework.org/schema/integration/file
	http://www.springframework.org/schema/integration/file/spring-integration-file.xsd
	http://www.springframework.org/schema/beans
	http://www.springframework.org/schema/beans/spring-beans.xsd
	http://www.springframework.org/schema/context
    http://www.springframework.org/schema/context/spring-context.xsd">

    <context:annotation-config/>


</beans>

2. Import the hot-folder-store-training-spring.xml file to the trainingcore-spring.xml.

<!-- ...\hybris\bin\custom\training\trainingcore\resources\trainingcore-spring.xml -->

<!-- Spring Integration -->
<import resource="classpath:/trainingcore/integration/hot-folder-store-training-spring.xml"/>
<import resource="classpath:/trainingcore/integration/hot-folder-store-electronics-spring.xml"/>
<import resource="classpath:/trainingcore/integration/hot-folder-store-apparel-spring.xml"/>
<import resource="classpath:/trainingcore/integration/hot-folder-common-spring.xml"/>

2.2. Initial Config

1. Inside the hot-folder-store-training-spring.xml add a base directory and an inbound-channel-adapter to watch over the base directory.

<!-- ...\hybris\bin\custom\training\trainingcore\resources\trainingcore\integration\hot-folder-store-training-spring.xml -->

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns... >

    <context:annotation-config/>

	<!-- Config a base directory -->
	<bean id="baseDirectoryTraining" class="java.lang.String">
		<constructor-arg value="#{baseDirectory}/${tenantId}/training" />
	</bean>

	<!-- Scan for files inside the base directory with names matches the pattern ^(.*)-(\d+)\.csv -->
	<file:inbound-channel-adapter id="batchFilesTraining" directory="#{baseDirectoryTraining}"
								  filename-regex="^(.*)-(\d+)\.csv"
								  comparator="fileOrderComparator">
		<!--  Periodic trigger in milliseconds -->
		<int:poller fixed-rate="1000" />
	</file:inbound-channel-adapter>
</beans>

The #{baseDirectory} by default is ${HYBRIS_DATA_DIR}/acceleratorservices/import, you can re-define it with the property : acceleratorservices.batch.impex.basefolder

2. Add an outbound-gateway to move the received file to processing and invoke the first step of the flow.

<!-- ...\hybris\bin\custom\training\trainingcore\resources\trainingcore\integration\hot-folder-store-training-spring.xml -->

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns... >

    <context:annotation-config/>
	
	...
	
	<!-- Move the file to processing and start the flow -->
	<file:outbound-gateway request-channel="batchFilesTraining" reply-channel="batchFilesTrainingProc"
						   directory="#{baseDirectoryTraining}/processing"
						   delete-source-files="true" />

</beans>

3. Create a service-activator which is the first step of the flow, it feeds the flow with the catalog and other relevant information.

<!-- ...\hybris\bin\custom\training\trainingcore\resources\trainingcore\integration\hot-folder-store-training-spring.xml -->

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns... >

    <context:annotation-config/>
	
	...
	
	<!-- Initialize the batch header with relevant information -->
	<int:service-activator input-channel="batchFilesTrainingProc" output-channel="batchFilesHeaderInit"
						   ref="trainingHeaderSetupTask"
						   method="execute" />

	<bean id="trainingHeaderSetupTask" class="de.hybris.platform.acceleratorservices.dataimport.batch.task.HeaderSetupTask">
		<property name="catalog" value="trainingProductCatalog" />
		<property name="net" value="false" />
		<property name="storeBaseDirectory" ref="baseDirectoryTraining" />
	</bean>
</beans>

2.3. Create ImpexConverter and ConverterMapping

1. Create an ImpexConvert for the UnitModel.

The ImpexConverter has two properties a header and an impexRow :

• header : will be the header of the generated impex.
• impexRow : defines the mapping between the column of the csv file and the impex file.

Inside the hot-folder-store-training-spring.xml, create a Spring bean from DefaultImpexConverter, with :

• header : INSERT_UPDATE Unit ;unitType[unique=true] ;code[unique=true] ;name[lang=$lang] ;conversion
• impexRow :  ;{+0}  ;{+1}  ;{2}  ;{3}

<!-- ...\hybris\bin\custom\training\trainingcore\resources\trainingcore\integration\hot-folder-store-training-spring.xml -->

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns... >

    <context:annotation-config/>
	
	...
	
	<!-- UnitModel impex converter -->
	<bean id="unitConverter" class="de.hybris.platform.acceleratorservices.dataimport.batch.converter.impl.DefaultImpexConverter">
		<property name="header">
			<value>
				#{defaultImpexProductHeader}
				INSERT_UPDATE Unit;unitType[unique=true];code[unique=true];name[lang=$lang];conversion
			</value>
		</property>
		<property name="impexRow">
			<value>;{+0};{+1};{2};{3}</value>
		</property>
	</bean>
</beans>

This means that :

• {+0} means that column 0 will be mapped to unitType[unique=true]
• {+1} means that column 1 will be mapped to code[unique=true]
• {2} means that column 2 will be mapped to name[lang=$lang]
• {3} means that column 3 will be mapped to conversion

2. Map the unitConverter to a file name prefix using DefaultConverterMapping.

<!-- ...\hybris\bin\custom\training\trainingcore\resources\trainingcore\integration\hot-folder-store-training-spring.xml -->

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns... >

    <context:annotation-config/>
	
	...
	
	<!-- UnitModel impex converter mapping to unit prefix -->
	<bean id="unitConverterMapping"
		  class="de.hybris.platform.acceleratorservices.dataimport.batch.converter.mapping.impl.DefaultConverterMapping"
		  p:mapping="unit"
		  p:converter-ref="unitConverter"/>
</beans>

This means that all the csv files start with unit will be treated using the unitConverter, example unit-fr-1234567.csv.

3. Run an ant all command then restart the server and you are good to go.

3. Hot Folder in Action

To test what you have accomplished so far, put a csv file that respect the impexRow format and the p:mapping="unit" inside the : ${HYBRIS_DATA_DIR}\acceleratorservices\import\master\training

Example : unit-en-123456789.csv

toto,	toto,	toto,	1

If everything goes well a Unit instance with code toto and name toto should be created.

Find a complete example on GitHub.