Everything about Hot Folder in Hybris

1. Overview
The hot folder is one of the best ways to import and feed data into Hybris, it is based on Spring integration framework.
Some reasons to use hot folder :
- Easy to configure and to extend.
- Use the power of Spring integration.
- It’s very fast than the regular impex import (uses multi-threading imports).
Basically the main idea behind hot folders is to transform csv files giving in input to impexes using a pre-configured ImpexConverters
and import them using ImportService
(see diagram bellow).
This is a more complete version of how the hot folder works.
Let’s explain this diagram step by step 🙂
- Inbound Channel Adapter : is a Spring integration component that allows us to create a watcher over csv files on a specific directory.
- File Order Comparator : the Inbound Channel Adapter uses a file order comparator to treat files by order of priority, this comparator compares files based on their file names.
- Header Setup Task : in this step of the flow a
BatchHeader
is created/initialized with the file, catalog, and some other information, thisBatchHeader
will be used throughout the flow as a reference for file and some other information. - Header Init Task : this is just another initialization step, in this step we extract the
sequenceId
and thelanguage
then we add them to theBatchHeader
for later use.
For example for this file name product-fr-2313213672186.csv thesequenceId
is 2313213672186 and thelanguage
is fr. - Impex Transformer Task : this is one of the important steps in the flow, basically here where the original file (csv) is converted to an impex with the help of a pre-configured
ImpexConverter
. - Impex Runner Task : imports the transformed file (impex) using
ImportService.importData()
method. - Cleanup Task : deletes the transformed file (impex) and archive the original file (csv).
The majority of the components that constitute the hot folder flow can be found at : …\hybris\bin\ext-accelerator\acceleratorservices\resources\acceleratorservices\integration\hot-folder-spring.xml
2. Implementation
Fortunately, the accelerator comes armed with an initial and complete configuration of the hot folder.
The ImpexConverters
for prices, products, medias, stocks, customers…, are already there and good to go.
However in this article, we will try to extend the hot folder functionalities to be able to import an other item type UnitModel
.
You need three macro steps to configure the hot folder :
- Define a base directory where the csv files will be put.
- Initiate your flow with the catalog and base directory…
- Create an
ImpexConverter
and associated with aMappingConverter
.
2.1. Preparation
1. Create an xml file hot-folder-store-training-spring.xml inside the …\hybris\bin\custom\training\trainingcore\resources\trainingcore\integration
<!-- ...\hybris\bin\custom\training\trainingcore\resources\trainingcore\integration\hot-folder-store-training-spring.xml -->
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:int="http://www.springframework.org/schema/integration"
xmlns:file="http://www.springframework.org/schema/integration/file"
xmlns:p="http://www.springframework.org/schema/p"
xmlns:context="http://www.springframework.org/schema/context"
xsi:schemaLocation="http://www.springframework.org/schema/integration
http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/integration/file
http://www.springframework.org/schema/integration/file/spring-integration-file.xsd
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/context
http://www.springframework.org/schema/context/spring-context.xsd">
<context:annotation-config/>
</beans>
2. Import the hot-folder-store-training-spring.xml file to the trainingcore-spring.xml.
<!-- ...\hybris\bin\custom\training\trainingcore\resources\trainingcore-spring.xml -->
<!-- Spring Integration -->
<import resource="classpath:/trainingcore/integration/hot-folder-store-training-spring.xml"/>
<import resource="classpath:/trainingcore/integration/hot-folder-store-electronics-spring.xml"/>
<import resource="classpath:/trainingcore/integration/hot-folder-store-apparel-spring.xml"/>
<import resource="classpath:/trainingcore/integration/hot-folder-common-spring.xml"/>
2.2. Initial Config
1. Inside the hot-folder-store-training-spring.xml add a base directory and an inbound-channel-adapter
to watch over the base directory.
<!-- ...\hybris\bin\custom\training\trainingcore\resources\trainingcore\integration\hot-folder-store-training-spring.xml -->
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns... >
<context:annotation-config/>
<!-- Config a base directory -->
<bean id="baseDirectoryTraining" class="java.lang.String">
<constructor-arg value="#{baseDirectory}/${tenantId}/training" />
</bean>
<!-- Scan for files inside the base directory with names matches the pattern ^(.*)-(\d+)\.csv -->
<file:inbound-channel-adapter id="batchFilesTraining" directory="#{baseDirectoryTraining}"
filename-regex="^(.*)-(\d+)\.csv"
comparator="fileOrderComparator">
<!-- Periodic trigger in milliseconds -->
<int:poller fixed-rate="1000" />
</file:inbound-channel-adapter>
</beans>
The
#{baseDirectory}
by default is ${HYBRIS_DATA_DIR}/acceleratorservices/import, you can re-define it with the property :acceleratorservices.batch.impex.basefolder
2. Add an outbound-gateway
to move the received file to processing and invoke the first step of the flow.
<!-- ...\hybris\bin\custom\training\trainingcore\resources\trainingcore\integration\hot-folder-store-training-spring.xml -->
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns... >
<context:annotation-config/>
...
<!-- Move the file to processing and start the flow -->
<file:outbound-gateway request-channel="batchFilesTraining" reply-channel="batchFilesTrainingProc"
directory="#{baseDirectoryTraining}/processing"
delete-source-files="true" />
</beans>
3. Create a service-activator
which is the first step of the flow, it feeds the flow with the catalog and other relevant information.
<!-- ...\hybris\bin\custom\training\trainingcore\resources\trainingcore\integration\hot-folder-store-training-spring.xml -->
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns... >
<context:annotation-config/>
...
<!-- Initialize the batch header with relevant information -->
<int:service-activator input-channel="batchFilesTrainingProc" output-channel="batchFilesHeaderInit"
ref="trainingHeaderSetupTask"
method="execute" />
<bean id="trainingHeaderSetupTask" class="de.hybris.platform.acceleratorservices.dataimport.batch.task.HeaderSetupTask">
<property name="catalog" value="trainingProductCatalog" />
<property name="net" value="false" />
<property name="storeBaseDirectory" ref="baseDirectoryTraining" />
</bean>
</beans>
2.3. Create ImpexConverter
and ConverterMapping
1. Create an ImpexConvert
for the UnitModel
.
The ImpexConverter
has two properties a header
and an impexRow
:
- header : will be the header of the generated impex.
- impexRow : defines the mapping between the column of the csv file and the impex file.
Inside the hot-folder-store-training-spring.xml, create a Spring bean from DefaultImpexConverter
, with :
- header :
INSERT_UPDATE Unit ;unitType[unique=true] ;code[unique=true] ;name[lang=$lang] ;conversion
- impexRow :
;{+0} ;{+1} ;{2} ;{3}
<!-- ...\hybris\bin\custom\training\trainingcore\resources\trainingcore\integration\hot-folder-store-training-spring.xml -->
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns... >
<context:annotation-config/>
...
<!-- UnitModel impex converter -->
<bean id="unitConverter" class="de.hybris.platform.acceleratorservices.dataimport.batch.converter.impl.DefaultImpexConverter">
<property name="header">
<value>
#{defaultImpexProductHeader}
INSERT_UPDATE Unit;unitType[unique=true];code[unique=true];name[lang=$lang];conversion
</value>
</property>
<property name="impexRow">
<value>;{+0};{+1};{2};{3}</value>
</property>
</bean>
</beans>
This means that :
{+0}
means that column 0 will be mapped tounitType[unique=true]
{+1}
means that column 1 will be mapped tocode[unique=true]
{2}
means that column 2 will be mapped toname[lang=$lang]
{3}
means that column 3 will be mapped toconversion
2. Map the unitConverter
to a file name prefix using DefaultConverterMapping
.
<!-- ...\hybris\bin\custom\training\trainingcore\resources\trainingcore\integration\hot-folder-store-training-spring.xml -->
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns... >
<context:annotation-config/>
...
<!-- UnitModel impex converter mapping to unit prefix -->
<bean id="unitConverterMapping"
class="de.hybris.platform.acceleratorservices.dataimport.batch.converter.mapping.impl.DefaultConverterMapping"
p:mapping="unit"
p:converter-ref="unitConverter"/>
</beans>
This means that all the csv files start with unit will be treated using the unitConverter
, example unit-fr-1234567.csv.
3. Run an ant all
command then restart the server and you are good to go.
3. Hot Folder in Action
To test what you have accomplished so far, put a csv file that respect the impexRow
format and the p:mapping="unit"
inside the : ${HYBRIS_DATA_DIR}\acceleratorservices\import\master\training
Example : unit-en-123456789.csv
toto, toto, toto, 1
If everything goes well a Unit instance with code toto and name toto should be created.
Find a complete example on GitHub.

Software Craftsmanship, Stackextend author and Full Stack developer with 6+ years of experience in Java/Kotlin, Java EE, Angular and Hybris…
I’m Passionate about Microservice architectures, Hexagonal architecture, Event Driven architecture, Event Sourcing and Domain Driven design (DDD)…
Huge fan of Clean Code school, SOLID, GRASP principles, Design Patterns, TDD and BDD.
Great post easy to understand.
A question please what does the
+
means for{+0}
?The
+
means that this row is mandatory and cannot be empty.Is there any way to hardcode values while configuring hotfolder?
I don’t understand your question, sorry!
I want to send an email with error message if anything wrong while running hot folder (import error). can you please help me.
You have to override the
de.hybris.platform.acceleratorservices.dataimport.batch.task.ErrorHandler
and do what ever you need inside thepublic void onError(...)
method.Do we have mechanism to rollback all record if any point we face error during hot-folder import.
no impex is not transactionnal, all lines rejected are stored in a log file you can correct them and re-import
Easy to understand.
but, where does these values comes from #{defaultImpexProductHeader}.
You can find it here : ${HYBRIS_BIN_DIR}\bin\ext-accelerator\acceleratorservices\resources\acceleratorservices\integration\hot-folder-spring.xml
<bean id="defaultImpexProductHeader" class="java.lang.String">
<constructor-arg>
<value># ImpEx for importing data into $CATALOG$
$catalogVersion=catalogversion(catalog(id[default=$CATALOG$]),version[default='Staged'])[unique=true]
$approved=approvalstatus(code)[default='check']
$lang=$LANGUAGE$</value>
</constructor-arg>
</bean>
May I know how would it resolve value to a particular catalog like catalog = powertoolsProductCatalog?
I think I got my answer. It is in service activator part.
Hi Bro,
The content and the way of explanation is very good. But the text color is too light and a bit hard to read. Thanks for clear explanation.
Thanks,
Manoj
Thanks for your feedback!
Good post, can you also please let us know how to have bean configuration in xml config file so that service activator can call the particular bean and in the class i could have the file read and manipulate it.
Re-check the part where I created the
trainingHeaderSetupTask
bean.Can we do the deletion operation using hot folder like “Delete from {table_name}”
Yes, in the
header
of theDefaultImpexConverter
, addREMOVE [YourItemType]; code[unique=true];...
Hi! Can you please provide some information about how to pass « & » with the impex.
INSERT_UPDATE PointOfService;name[unique=true];type(code);address(&addrID);latitude;
I am getting this error » The reference to entity « addrID » must end with the ‘;’ delimiter »
Actually I want to add it through hot-folder in a csv file.
Hi Mouad thanks for the informarion I have a question.How to create the “trainingcore” directory and its internal directories?
The character ç at the end of the filename unit-en-12345678ç.csv is INCORRECT. It cause file isn’t loaded.
Oops! my bad, I fixed it.
Thanks Antonio 🙂
Please make a video on this session, starting from feeding CSV file as input To storing it to DB.
This will help us so much for understanding,
Hi Fakir, nice post for easy understanding.
Please post a detail configuration setup on V2 Hot-Folder (Cloud) from Hybris system to Microsoft Azure Storage Explorer.
Hi Mouad,
I have implemented Hot Folder Functionality seeing your example. In Our case we have ; as delimeter for csv file and it works fine if all coulmns have singular value.
I am facing issue for columns who are of list or collection types. I tried like ex. “A,B,C” for same but it also takes , as seperator,
Any suggestion would be helpful.
A very basic and excellent blog of explanation which can help alot to a biginner.
Hi,
How can we convert json data getting from restful api to csv files to implement hot batch folder concept