Everything about Hot Folder in Hybris

Hybris Logo

1. Overview

The hot folder is one of the best ways to import and feed data into Hybris, it is based on Spring integration framework.

Some reasons to use hot folder :

  • Easy to configure and to extend.
  • Use the power of Spring integration.
  • It’s very fast than the regular impex import (uses multi-threading imports).

Basically the main idea behind hot folders is to transform csv files giving in input to impexes using a pre-configured ImpexConverters and import them using ImportService (see diagram bellow).

Simplified overview about hot folder in Hybris

This is a more complete version of how the hot folder works.

Overview of the hot folder in Hybris

 

Let’s explain this diagram step by step ūüôā

  • Inbound Channel Adapter : is a Spring integration component that allows us to create a watcher over csv files on a specific directory.
  • File Order Comparator : the Inbound Channel Adapter uses a file order comparator to¬†treat files¬†by order of priority, this comparator compares files based on their file names.
  • Header Setup Task :¬†in this step of the flow a BatchHeader is created/initialized with the file, catalog, and some other information, this BatchHeader will be used throughout the flow as a reference for file and some other information.
  • Header Init Task : this is just another initialization step, in this step we extract the sequenceId and the language then we add them to the¬†BatchHeader for later use.
    For example for this file name product-fr-2313213672186.csv the sequenceId is 2313213672186 and the language is fr.
  • Impex Transformer Task :¬†this is one of the important steps in the flow, basically here where the original file (csv) is converted to an impex with the help of a pre-configured¬†ImpexConverter.
  • Impex Runner Task :¬†imports the transformed file (impex) using¬†ImportService.importData() method.
  • Cleanup Task : deletes the transformed file (impex) and archive the original file (csv).

The majority of the components that constitute the hot folder flow can be found at :¬†…\hybris\bin\ext-accelerator\acceleratorservices\resources\acceleratorservices\integration\hot-folder-spring.xml

2. Implementation

Fortunately, the accelerator comes armed with an initial and complete configuration of the hot folder.

The ImpexConverters¬†for prices, products, medias, stocks, customers…, are already there and good to go.

However in this article, we will try to extend the hot folder functionalities to be able to import an other item type UnitModel.

You need three macro steps to configure the hot folder :

  • Define a base directory¬†where the csv files will be put.
  • Initiate your flow with the catalog and base directory
  • Create an ImpexConverter and associated with a MappingConverter.

2.1. Preparation

1. Create an xml file¬†hot-folder-store-training-spring.xml inside the¬†…\hybris\bin\custom\training\trainingcore\resources\trainingcore\integration

<!-- ...\hybris\bin\custom\training\trainingcore\resources\trainingcore\integration\hot-folder-store-training-spring.xml -->

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
	   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	   xmlns:int="http://www.springframework.org/schema/integration"
	   xmlns:file="http://www.springframework.org/schema/integration/file"
	   xmlns:p="http://www.springframework.org/schema/p"
	   xmlns:context="http://www.springframework.org/schema/context"
	   xsi:schemaLocation="http://www.springframework.org/schema/integration
	http://www.springframework.org/schema/integration/spring-integration.xsd
	http://www.springframework.org/schema/integration/file
	http://www.springframework.org/schema/integration/file/spring-integration-file.xsd
	http://www.springframework.org/schema/beans
	http://www.springframework.org/schema/beans/spring-beans.xsd
	http://www.springframework.org/schema/context
    http://www.springframework.org/schema/context/spring-context.xsd">

    <context:annotation-config/>


</beans>

2. Import the hot-folder-store-training-spring.xml file to the trainingcore-spring.xml.

<!-- ...\hybris\bin\custom\training\trainingcore\resources\trainingcore-spring.xml -->

<!-- Spring Integration -->
<import resource="classpath:/trainingcore/integration/hot-folder-store-training-spring.xml"/>
<import resource="classpath:/trainingcore/integration/hot-folder-store-electronics-spring.xml"/>
<import resource="classpath:/trainingcore/integration/hot-folder-store-apparel-spring.xml"/>
<import resource="classpath:/trainingcore/integration/hot-folder-common-spring.xml"/>

2.2. Initial Config

1. Inside the hot-folder-store-training-spring.xml add a base directory and an inbound-channel-adapter to watch over the base directory.

<!-- ...\hybris\bin\custom\training\trainingcore\resources\trainingcore\integration\hot-folder-store-training-spring.xml -->

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns... >

    <context:annotation-config/>

	<!-- Config a base directory -->
	<bean id="baseDirectoryTraining" class="java.lang.String">
		<constructor-arg value="#{baseDirectory}/${tenantId}/training" />
	</bean>

	<!-- Scan for files inside the base directory with names matches the pattern ^(.*)-(\d+)\.csv -->
	<file:inbound-channel-adapter id="batchFilesTraining" directory="#{baseDirectoryTraining}"
								  filename-regex="^(.*)-(\d+)\.csv"
								  comparator="fileOrderComparator">
		<!--  Periodic trigger in milliseconds -->
		<int:poller fixed-rate="1000" />
	</file:inbound-channel-adapter>
</beans>

The #{baseDirectory} by default is ${HYBRIS_DATA_DIR}/acceleratorservices/import, you can re-define it with the property : acceleratorservices.batch.impex.basefolder

2. Add an outbound-gateway to move the received file to processing and invoke the first step of the flow.

<!-- ...\hybris\bin\custom\training\trainingcore\resources\trainingcore\integration\hot-folder-store-training-spring.xml -->

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns... >

    <context:annotation-config/>
	
	...
	
	<!-- Move the file to processing and start the flow -->
	<file:outbound-gateway request-channel="batchFilesTraining" reply-channel="batchFilesTrainingProc"
						   directory="#{baseDirectoryTraining}/processing"
						   delete-source-files="true" />

</beans>

3. Create a service-activator which is the first step of the flow, it feeds the flow with the catalog and other relevant information.

<!-- ...\hybris\bin\custom\training\trainingcore\resources\trainingcore\integration\hot-folder-store-training-spring.xml -->

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns... >

    <context:annotation-config/>
	
	...
	
	<!-- Initialize the batch header with relevant information -->
	<int:service-activator input-channel="batchFilesTrainingProc" output-channel="batchFilesHeaderInit"
						   ref="trainingHeaderSetupTask"
						   method="execute" />

	<bean id="trainingHeaderSetupTask" class="de.hybris.platform.acceleratorservices.dataimport.batch.task.HeaderSetupTask">
		<property name="catalog" value="trainingProductCatalog" />
		<property name="net" value="false" />
		<property name="storeBaseDirectory" ref="baseDirectoryTraining" />
	</bean>
</beans>

2.3. Create ImpexConverter and ConverterMapping

1. Create an ImpexConvert for the UnitModel.

The ImpexConverter has two properties a header and an impexRow :

  • header : will be the header of the generated impex.
  • impexRow : defines the mapping between the column of the csv file and the impex file.

Inside the hot-folder-store-training-spring.xml, create a Spring bean from DefaultImpexConverter, with :

  • header : INSERT_UPDATE Unit ;unitType[unique=true] ;code[unique=true] ;name[lang=$lang] ;conversion
  • impexRow :¬† ;{+0}¬† ;{+1}¬† ;{2}¬† ;{3}
<!-- ...\hybris\bin\custom\training\trainingcore\resources\trainingcore\integration\hot-folder-store-training-spring.xml -->

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns... >

    <context:annotation-config/>
	
	...
	
	<!-- UnitModel impex converter -->
	<bean id="unitConverter" class="de.hybris.platform.acceleratorservices.dataimport.batch.converter.impl.DefaultImpexConverter">
		<property name="header">
			<value>
				#{defaultImpexProductHeader}
				INSERT_UPDATE Unit;unitType[unique=true];code[unique=true];name[lang=$lang];conversion
			</value>
		</property>
		<property name="impexRow">
			<value>;{+0};{+1};{2};{3}</value>
		</property>
	</bean>
</beans>

This means that :

  • {+0} means that column 0¬†will be mapped to¬†unitType[unique=true]
  • {+1}¬†means that column 1 will be mapped to¬†code[unique=true]
  • {2} means that column 2¬†will be mapped to¬†name[lang=$lang]
  • {3}¬†means that column 3 will be mapped to conversion

2. Map the unitConverter to a file name prefix using DefaultConverterMapping.

<!-- ...\hybris\bin\custom\training\trainingcore\resources\trainingcore\integration\hot-folder-store-training-spring.xml -->

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns... >

    <context:annotation-config/>
	
	...
	
	<!-- UnitModel impex converter mapping to unit prefix -->
	<bean id="unitConverterMapping"
		  class="de.hybris.platform.acceleratorservices.dataimport.batch.converter.mapping.impl.DefaultConverterMapping"
		  p:mapping="unit"
		  p:converter-ref="unitConverter"/>
</beans>

This means that all the csv files start with unit will be treated using the unitConverter, example unit-fr-1234567.csv.

3. Run an ant all command then restart the server and you are good to go.

3. Hot Folder in Action

To test what you have accomplished so far, put a csv file that respect the impexRow format and the p:mapping="unit" inside the : ${HYBRIS_DATA_DIR}\acceleratorservices\import\master\training

Example : unit-en-123456789.csv

toto,	toto,	toto,	1

If everything goes well a Unit instance with code toto and name toto should be created.

Find a complete example on GitHub.

 

44
Leave a Reply

avatar
19 Comment threads
25 Thread replies
26 Followers
 
Most reacted comment
Hottest comment thread
27 Comment authors
Sven de BieRohanmuratSoufianeVishal Recent comment authors

This site uses Akismet to reduce spam. Learn how your comment data is processed.

  Subscribe  
newest oldest most voted
Notify of
Sarah
Guest
Sarah

Great post easy to understand.
A question please what does the + means for {+0} ?

Anto
Guest
Anto

Is there any way to hardcode values while configuring hotfolder?

soufiane
Guest
soufiane

I think you could achieve that by modifying the header of Impex. e.g setting the language of the session for all CronJobs to english (I add default=en so if row do not specify a language for session it will have en by default)

INSERT_UPDATE CronJob ; code[unique=true]; job(code);sessionLanguage(isoCode)[default=en]

shiva
Guest
shiva

I want to send an email with error message if anything wrong while running hot folder (import error). can you please help me.

aarshi
Guest
aarshi

hi shiva did u implement this email with error message while running hot folder import errorr

Keerthi
Guest
Keerthi

Easy to understand.
but, where does these values comes from #{defaultImpexProductHeader}.

Manoj
Guest
Manoj

Hi Bro,
The content and the way of explanation is very good. But the text color is too light and a bit hard to read. Thanks for clear explanation.
Thanks,
Manoj

Saran
Guest
Saran

Good post, can you also please let us know how to have bean configuration in xml config file so that service activator can call the particular bean and in the class i could have the file read and manipulate it.

Lokesh
Guest
Lokesh

Can we do the deletion operation using hot folder like “Delete from {table_name}”

Maaz Ahmed
Guest
Maaz Ahmed

Hi! Can you please provide some information about how to pass ¬ę¬†&¬†¬Ľ with the impex.

INSERT_UPDATE PointOfService;name[unique=true];type(code);address(&addrID);latitude;
I am getting this error ¬†¬Ľ The reference to entity ¬ę¬†addrID¬†¬Ľ must end with the ‚Äė;‚Äô delimiter¬†¬Ľ

Maaz Ahmed
Guest
Maaz Ahmed

Actually I want to add it through hot-folder in a csv file.

fmarquez
Guest
fmarquez

Hi Mouad thanks for the informarion I have a question.How to create the ‚Äútrainingcore‚ÄĚ directory and its internal directories?

Antonio
Guest
Antonio

The character √ß at the end of the filename unit-en-12345678√ß.csv is INCORRECT. It cause file isn’t loaded.

Suresh S
Guest
Suresh S

Please make a video on this session, starting from feeding CSV file as input To storing it to DB.
This will help us so much for understanding,

Suresh
Guest
Suresh

Hi Fakir, nice post for easy understanding.
Please post a detail configuration setup on V2 Hot-Folder (Cloud) from Hybris system to Microsoft Azure Storage Explorer.

Mehul
Guest
Mehul

Hi Mouad,

I have implemented Hot Folder Functionality seeing your example. In Our case we have ; as delimeter for csv file and it works fine if all coulmns have singular value.

I am facing issue for columns who are of list or collection types. I tried like ex. “A,B,C” for same but it also takes , as seperator,

Any suggestion would be helpful.

Mohamed BELMAHI
Admin

You have to use this syntax on header -> CollectionAttribute[separator = :]
And mak a value like A:B:C separated with : instead of , since it’s used for your csv

Sachi Nayak
Guest
Sachi Nayak

A very basic and excellent blog of explanation which can help alot to a biginner.

Nupur Gupta
Guest
Nupur Gupta

Hi,

How can we convert json data getting from restful api to csv files to implement hot batch folder concept

Mohamed BELMAHI
Admin

You can use a job that call the api and get data then convert it to a csv file (there’s many lib to do that) then send the result file to hotfolder.

Vishal
Guest
Vishal

Hi,
I’m getting below error in error folder
Missing value for 1: kw1;kws1;;

I/P file contain
;kw;kws;Kilowatt;

Thanks in advance

Mohamed BELMAHI
Admin

Could you provide the configuration for this particular import ?

Vishal
Guest
Vishal

I have use ; in CSV as a Field separator
But default is , comma

Resolved my issue now.

Thanks

Mohamed BELMAHI
Admin

Good
Thanks for your feedback!

Haris
Guest
Haris

Hi @Mouad EL Fakir, Very well explained ! I don’t have any questions regarding this owesome guide but I just want to know what methodology you used to get a great grip on hybris ? do you have any tutorials or any content which can help others to put their efferots on the right track to get expertise as you have ?

Soufiane
Guest
Soufiane

Great article! thanks.

Rohan
Guest
Rohan

Hi,
Nice article!
Can you also please let us know how we can access S3 bucket folder via this
Config a base directory code :
bean id=”baseDirectoryTraining” class=”java.lang.String”
constructor-arg value=”#{baseDirectory}/${tenantId}/training”

Mohamed BELMAHI
Admin

The simple way is to use a custom job that connect to S3 bucket and get files then put them inside
#{baseDirectory}/${tenantId}/training

Otherwise you have to change a lot of things inside spring integration configuration which is not recommended due to performance issues that you will get during read/write on S3 Bucket.

Sven de Bie
Guest
Sven de Bie

Great post! One additional question, is there a way to use the hot folders to import predefined impexes? So just copy final impexes into the hotfolder and let the hotfolder import it automatically after each other?