Everything about Hot Folder in Hybris

Hybris Logo

1. Overview

The hot folder is one of the best ways to import and feed data into Hybris, it is based on Spring integration framework.

Some reasons to use hot folder :

  • Easy to configure and to extend.
  • Use the power of Spring integration.
  • It’s very fast than the regular impex import (uses multi-threading imports).

Basically the main idea behind hot folders is to transform csv files giving in input to impexes using a pre-configured ImpexConverters and import them using ImportService (see diagram bellow).

Simplified overview about hot folder in Hybris

This is a more complete version of how the hot folder works.

Overview of the hot folder in Hybris

 

Let’s explain this diagram step by step ūüôā

  • Inbound Channel Adapter : is a Spring integration component that allows us to create a watcher over csv files on a specific directory.
  • File Order Comparator : the Inbound Channel Adapter uses a file order comparator to¬†treat files¬†by order of priority, this comparator compares files based on their file names.
  • Header Setup Task :¬†in this step of the flow a BatchHeader is created/initialized with the file, catalog, and some other information, this BatchHeader will be used throughout the flow as a reference for file and some other information.
  • Header Init Task : this is just another initialization step, in this step we extract the sequenceId and the language then we add them to the¬†BatchHeader for later use.
    For example for this file name product-fr-2313213672186.csv the sequenceId is 2313213672186 and the language is fr.
  • Impex Transformer Task :¬†this is one of the important steps in the flow, basically here where the original file (csv) is converted to an impex with the help of a pre-configured¬†ImpexConverter.
  • Impex Runner Task :¬†imports the transformed file (impex) using¬†ImportService.importData() method.
  • Cleanup Task : deletes the transformed file (impex) and archive the original file (csv).

The majority of the components that constitute the hot folder flow can be found at :¬†…\hybris\bin\ext-accelerator\acceleratorservices\resources\acceleratorservices\integration\hot-folder-spring.xml

2. Implementation

Fortunately, the accelerator comes armed with an initial and complete configuration of the hot folder.

The ImpexConverters¬†for prices, products, medias, stocks, customers…, are already there and good to go.

However in this article, we will try to extend the hot folder functionalities to be able to import an other item type UnitModel.

You need three macro steps to configure the hot folder :

  • Define a base directory¬†where the csv files will be put.
  • Initiate your flow with the catalog and base directory
  • Create an ImpexConverter and associated with a MappingConverter.

2.1. Preparation

1. Create an xml file¬†hot-folder-store-training-spring.xml inside the¬†…\hybris\bin\custom\training\trainingcore\resources\trainingcore\integration

<!-- ...\hybris\bin\custom\training\trainingcore\resources\trainingcore\integration\hot-folder-store-training-spring.xml -->

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
	   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	   xmlns:int="http://www.springframework.org/schema/integration"
	   xmlns:file="http://www.springframework.org/schema/integration/file"
	   xmlns:p="http://www.springframework.org/schema/p"
	   xmlns:context="http://www.springframework.org/schema/context"
	   xsi:schemaLocation="http://www.springframework.org/schema/integration
	http://www.springframework.org/schema/integration/spring-integration.xsd
	http://www.springframework.org/schema/integration/file
	http://www.springframework.org/schema/integration/file/spring-integration-file.xsd
	http://www.springframework.org/schema/beans
	http://www.springframework.org/schema/beans/spring-beans.xsd
	http://www.springframework.org/schema/context
    http://www.springframework.org/schema/context/spring-context.xsd">

    <context:annotation-config/>


</beans>

2. Import the hot-folder-store-training-spring.xml file to the trainingcore-spring.xml.

<!-- ...\hybris\bin\custom\training\trainingcore\resources\trainingcore-spring.xml -->

<!-- Spring Integration -->
<import resource="classpath:/trainingcore/integration/hot-folder-store-training-spring.xml"/>
<import resource="classpath:/trainingcore/integration/hot-folder-store-electronics-spring.xml"/>
<import resource="classpath:/trainingcore/integration/hot-folder-store-apparel-spring.xml"/>
<import resource="classpath:/trainingcore/integration/hot-folder-common-spring.xml"/>

2.2. Initial Config

1. Inside the hot-folder-store-training-spring.xml add a base directory and an inbound-channel-adapter to watch over the base directory.

<!-- ...\hybris\bin\custom\training\trainingcore\resources\trainingcore\integration\hot-folder-store-training-spring.xml -->

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns... >

    <context:annotation-config/>

	<!-- Config a base directory -->
	<bean id="baseDirectoryTraining" class="java.lang.String">
		<constructor-arg value="#{baseDirectory}/${tenantId}/training" />
	</bean>

	<!-- Scan for files inside the base directory with names matches the pattern ^(.*)-(\d+)\.csv -->
	<file:inbound-channel-adapter id="batchFilesTraining" directory="#{baseDirectoryTraining}"
								  filename-regex="^(.*)-(\d+)\.csv"
								  comparator="fileOrderComparator">
		<!--  Periodic trigger in milliseconds -->
		<int:poller fixed-rate="1000" />
	</file:inbound-channel-adapter>
</beans>

The #{baseDirectory} by default is ${HYBRIS_DATA_DIR}/acceleratorservices/import, you can re-define it with the property : acceleratorservices.batch.impex.basefolder

2. Add an outbound-gateway to move the received file to processing and invoke the first step of the flow.

<!-- ...\hybris\bin\custom\training\trainingcore\resources\trainingcore\integration\hot-folder-store-training-spring.xml -->

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns... >

    <context:annotation-config/>
	
	...
	
	<!-- Move the file to processing and start the flow -->
	<file:outbound-gateway request-channel="batchFilesTraining" reply-channel="batchFilesTrainingProc"
						   directory="#{baseDirectoryTraining}/processing"
						   delete-source-files="true" />

</beans>

3. Create a service-activator which is the first step of the flow, it feeds the flow with the catalog and other relevant information.

<!-- ...\hybris\bin\custom\training\trainingcore\resources\trainingcore\integration\hot-folder-store-training-spring.xml -->

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns... >

    <context:annotation-config/>
	
	...
	
	<!-- Initialize the batch header with relevant information -->
	<int:service-activator input-channel="batchFilesTrainingProc" output-channel="batchFilesHeaderInit"
						   ref="trainingHeaderSetupTask"
						   method="execute" />

	<bean id="trainingHeaderSetupTask" class="de.hybris.platform.acceleratorservices.dataimport.batch.task.HeaderSetupTask">
		<property name="catalog" value="trainingProductCatalog" />
		<property name="net" value="false" />
		<property name="storeBaseDirectory" ref="baseDirectoryTraining" />
	</bean>
</beans>

2.3. Create ImpexConverter and ConverterMapping

1. Create an ImpexConvert for the UnitModel.

The ImpexConverter has two properties a header and an impexRow :

  • header : will be the header of the generated impex.
  • impexRow : defines the mapping between the column of the csv file and the impex file.

Inside the hot-folder-store-training-spring.xml, create a Spring bean from DefaultImpexConverter, with :

  • header : INSERT_UPDATE Unit ;unitType[unique=true] ;code[unique=true] ;name[lang=$lang] ;conversion
  • impexRow :¬† ;{+0}¬† ;{+1}¬† ;{2}¬† ;{3}
<!-- ...\hybris\bin\custom\training\trainingcore\resources\trainingcore\integration\hot-folder-store-training-spring.xml -->

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns... >

    <context:annotation-config/>
	
	...
	
	<!-- UnitModel impex converter -->
	<bean id="unitConverter" class="de.hybris.platform.acceleratorservices.dataimport.batch.converter.impl.DefaultImpexConverter">
		<property name="header">
			<value>
				#{defaultImpexProductHeader}
				INSERT_UPDATE Unit;unitType[unique=true];code[unique=true];name[lang=$lang];conversion
			</value>
		</property>
		<property name="impexRow">
			<value>;{+0};{+1};{2};{3}</value>
		</property>
	</bean>
</beans>

This means that :

  • {+0} means that column 0¬†will be mapped to¬†unitType[unique=true]
  • {+1}¬†means that column 1 will be mapped to¬†code[unique=true]
  • {2} means that column 2¬†will be mapped to¬†name[lang=$lang]
  • {3}¬†means that column 3 will be mapped to conversion

2. Map the unitConverter to a file name prefix using DefaultConverterMapping.

<!-- ...\hybris\bin\custom\training\trainingcore\resources\trainingcore\integration\hot-folder-store-training-spring.xml -->

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns... >

    <context:annotation-config/>
	
	...
	
	<!-- UnitModel impex converter mapping to unit prefix -->
	<bean id="unitConverterMapping"
		  class="de.hybris.platform.acceleratorservices.dataimport.batch.converter.mapping.impl.DefaultConverterMapping"
		  p:mapping="unit"
		  p:converter-ref="unitConverter"/>
</beans>

This means that all the csv files start with unit will be treated using the unitConverter, example unit-fr-1234567.csv.

3. Run an ant all command then restart the server and you are good to go.

3. Hot Folder in Action

To test what you have accomplished so far, put a csv file that respect the impexRow format and the p:mapping="unit" inside the : ${HYBRIS_DATA_DIR}\acceleratorservices\import\master\training

Example : unit-en-123456789.csv

toto,	toto,	toto,	1

If everything goes well a Unit instance with code toto and name toto should be created.

Find a complete example on GitHub.

 

28
Leave a Reply

avatar
15 Comment threads
13 Thread replies
17 Followers
 
Most reacted comment
Hottest comment thread
19 Comment authors
Nupur GuptaSachi Nayakihyaten rachidMehulSuresh Recent comment authors

This site uses Akismet to reduce spam. Learn how your comment data is processed.

  Subscribe  
newest oldest most voted
Notify of
Sarah
Guest
Sarah

Great post easy to understand.
A question please what does the + means for {+0} ?

Anto
Guest
Anto

Is there any way to hardcode values while configuring hotfolder?

shiva
Guest
shiva

I want to send an email with error message if anything wrong while running hot folder (import error). can you please help me.

Keerthi
Guest
Keerthi

Easy to understand.
but, where does these values comes from #{defaultImpexProductHeader}.

Manoj
Guest
Manoj

Hi Bro,
The content and the way of explanation is very good. But the text color is too light and a bit hard to read. Thanks for clear explanation.
Thanks,
Manoj

Saran
Guest
Saran

Good post, can you also please let us know how to have bean configuration in xml config file so that service activator can call the particular bean and in the class i could have the file read and manipulate it.

Lokesh
Guest
Lokesh

Can we do the deletion operation using hot folder like “Delete from {table_name}”

Maaz Ahmed
Guest
Maaz Ahmed

Hi! Can you please provide some information about how to pass ¬ę¬†&¬†¬Ľ with the impex.

INSERT_UPDATE PointOfService;name[unique=true];type(code);address(&addrID);latitude;
I am getting this error ¬†¬Ľ The reference to entity ¬ę¬†addrID¬†¬Ľ must end with the ‚Äė;‚Äô delimiter¬†¬Ľ

Maaz Ahmed
Guest
Maaz Ahmed

Actually I want to add it through hot-folder in a csv file.

fmarquez
Guest
fmarquez

Hi Mouad thanks for the informarion I have a question.How to create the ‚Äútrainingcore‚ÄĚ directory and its internal directories?

Antonio
Guest
Antonio

The character √ß at the end of the filename unit-en-12345678√ß.csv is INCORRECT. It cause file isn’t loaded.

Suresh S
Guest
Suresh S

Please make a video on this session, starting from feeding CSV file as input To storing it to DB.
This will help us so much for understanding,

Suresh
Guest
Suresh

Hi Fakir, nice post for easy understanding.
Please post a detail configuration setup on V2 Hot-Folder (Cloud) from Hybris system to Microsoft Azure Storage Explorer.

Mehul
Guest
Mehul

Hi Mouad,

I have implemented Hot Folder Functionality seeing your example. In Our case we have ; as delimeter for csv file and it works fine if all coulmns have singular value.

I am facing issue for columns who are of list or collection types. I tried like ex. “A,B,C” for same but it also takes , as seperator,

Any suggestion would be helpful.

Sachi Nayak
Guest
Sachi Nayak

A very basic and excellent blog of explanation which can help alot to a biginner.

Nupur Gupta
Guest
Nupur Gupta

Hi,

How can we convert json data getting from restful api to csv files to implement hot batch folder concept