HCM Data Loader automation examples

Posted by

22

Overview

Today I would like to continue discussion about HCM Data Loader. In my previous post I briefly explained HDL approach and steps needed to load huge amount of data into Fusion HCM.
As you have already known HCM Data Loader is a utility for loading business objects in bulk from any source to Oracle Fusion HCM. You use HCM Data Loader to:

  • Perform a one-time migration of legacy data to HCM.
  • Maintain your Oracle Fusion HCM data for ongoing updates.

In my opinion integration part is the most important and difficult part of any implementation project especially in the Cloud. And there are a lot of customers in the world who imagine an Oracle HCM Cloud as a black box without any ability being incorporated into customer’s IT system landscape. I will try to disrupt such opinion :-).
In this article I would like to show a couple of examples how to provide automated synchronization between third party applications and Fusion HCM Cloud with HCM Data Loader.

To provide the ability to automate bulk loading of data into HCM you should use the following approach:

  • You need to deliver a zip data file to the WebCenter Content server for consumption by HCM Data Loader.
  • You need to invoke the web service to consume and process the staged zip file.

Tools

Oracle provides tools that help to achieve these goals. In my examples I will use:

  • The WebCenter Content Document Transfer Utility for Oracle Fusion Applications
  • HCM Data Loader Integration Service

You can get more detailed information about the tools mentioned above in the MOS Note: Automating HCM Data Loader.

The WebCenter Content Document Transfer Utility for Oracle Fusion Applications is a feature-set Java library that provides content export and import capabilities. You can use it to load ZIP file to Fusion HCM UCM from command line:

java -classpath “oracle.ucm.fa_client_11.1.1.jar” oracle.ucm.client.UploadTool  –url=https://{host}/cs/idcplg
–username=<provide_user_name> –password=<provide_password> –primaryFile=”<file_path_with_filename>” –dDocTitle=”<provide_Zip_Filename>” –dDocAccount=hcm/dataloader/import

This tool returns UCM Content ID (ex. dDocName=UCMFA009506). Content ID will be needed for the web service call.

The second tool is HCM Data Loader Integration Service. HCM Data Loader implements a web service that can be used to initiate the import and load of data files staged on the WebCenter Content server. Once you have placed the zip file containing the .dat files in the WebCenter Content, secured with account hcm/dataloader/import, the web service HCMDataLoader can be invoked to perform an Import or Import and Load of data to Fusion.

SOAP message looks like:

http://Host/hcmCommonDataLoader/HCMDataLoader
<soap:Envelope xmlns:soap=”http://schemas.xmlsoap.org/soap/envelope⁄″&gt;
<soap:Body>
<ns1:importAndLoadDataAsync
xmlns:ns1=”http://xmlns.oracle.com/apps/hcm/common/dataLoader/core/dataLoaderIntegrationService/types/”&gt;
<ns1:ContentId>UCMFA009506</ns1:ContentId>
<ns1:Parameters></ns1:Parameters>
</ns1:importAndLoadDataAsync>
</soap:Body>
</soap:Envelope>

Examples

Let’s imagine that we use Oracle HCM Cloud as a Talent Management system to provide performance assessment and we have HRIS system to store personnel records. And we need to update personnel records from HRIS system to Oracle HCM Cloud daily. So You have to generate data files from your external application and populate them into Fusion. This logic can be incorporated directly in HRIS application or can be provided by external tools.

PHP

More than 13 years ago I bought book about PHP Web Development. PHP technology was getting popular at that time and I enjoyed reading that book. And I would like to show sample PHP source code that provides loading HDL zip file to Fusion HCM content server and then invokes HDL web service to start batch import. Please be not very strict to this code because it’s only for demonstration purposes and, in addition, I haven’t written any line of code on PHP for at least 7 years :-(.

Let me show simple sample Worker DAT file:

HDL-PHP-DAT-file

In the next screenshot you can see the part of PHP source code which uploads DAT file and then invokes HCM Data Loader Integration Service.

HDL-PHP-code

In the following screenshot you can see a sample PHP page I have created to provide ability to load and import files from simple UI.

HDL-PHP-page

In the following picture you can see Fusion HCM Data Exchange UI and make sure that HDL batch import has been started.

HDL-PHP-Data-Exchange

After all you can make sure that employee record has been successfully loaded.

HDL-PHP-Person

Unix shell script

In the second example I would like to show approach that let you use only Unix/Linux environment without any additional IT tools(except JRE) to complete the steps described above (loading file and import batch). So to achieve our goal we only need to create simple unix shell script that can be launched on a daily basis. As the scheduler you can use Unix Cron utility. See more: https://en.wikipedia.org/wiki/Cron

I have recorded a short video to demonstrate this approach. Please note that script has been created from scratch only to demonstrate features and of course it should be improved by adding exception handling, etc. Here is direct link to YouTube video: https://www.youtube.com/watch?feature=player_detailpage&v=4s0luXEaGbk

I hope this will help. Good luck and remember:

“When one door closes, another opens; but we often look so long and so regretfully upon the closed door that we do not see the one which has opened for us.” ~ Alexander Graham Bell

SY,

Volodymyr.

22 thoughts on “HCM Data Loader automation examples”

  1. Hi Volodymyr.,

    thank you so much that mu made this very simple., and it is very helpful. thank you 🙂 I really appreciate your knowledge and work.

    but I have one more Question:

    I want to place Encrypted file in WebCenter Content Server (WCC/UCM) hcm/dataloader/import and then I want Import and Load program to decrypt before importing and loading.,

    is this possible.? if Yes any Idea how this works.?

    thanks
    Kranthi

    Reply

    • Hi Kranthi,

      As stated in documentation, you have to raise SR to achieve you encryption requirements. See below:

      Setting up Encryption for File Transfer: Procedure

      You use encryption keys to transfer files securely between Oracle HCM Cloud and your own servers via the Oracle WebCenter Content server. This topic describes how to set up encryption keys for secure file transfer. After you perform this setup, you can encrypt and decrypt files and transfer them between your servers and Oracle HCM Cloud using HCM File-Based Loader, HCM Data Loader, payroll batch loader, and HCM Extracts.

      Creating a Service Request
      Create a service request (SR) with the following settings:
      1. Select Hosting Services in the Component field.
      2. Enter Key Exchange for Encryption and Decryption in the Application Subcomponent field.
      3. Include the following information:
      Encryption type Pretty Good Privacy (PGP)
      Encryption public key
      Encryption key size (1024 or 2048)

      Regards,
      Volodymyr

      Reply

    • Hi Volodymyr,
      Thank you so much for sharing your work on this.
      I have a question on calling the java -jar command ,can we call this from Java Program?

      If so ,could you please tell me how can we do it?

      Thanks,
      Priyanka

      Reply

      • Dear Priyanka,

        I would like to notice that reading documentation sometimes may be useful.
        If you open MOS document “Automating HCM Data Loader(ID:1664133.1)” mentioned in my article and read section about “Remote Intradoc Client (RIDC)” you will get to know that it’s possible. Document also provides sample java code that shows how to incorporate RIDC communication API into your Java program.

        Hope it helps.

        Regards,
        Volodymyr

  2. Hi Volodimyr,

    Brilliant Article Mate, Appreciate it.
    and I have small clarification here,

    Is it possible to advise the HDL to Pick the ZIP automatically from UCM in recurring basis, Like once in a Day?

    Chandru A

    Reply

  3. Thanks for your brilliant work.

    I need your help in getting the right .jar file to communicate with Java.
    When I tried to download, I could not find “oracle.ucm.fa_client_11.1.1.jar” instead I found the “fmw_12.2.1.1.0_wccontent.jar”.

    Kindly let me know if this .jar is a valid one, if not, give me the correct source where I can find the correct .jar file.

    Reply

  4. Hi volodymyr,

    I need your sample Shell script, Also it will be handy if you explain how can we get content ID to variable, so we can pass it to web service calling API.

    Regards
    Zubair

    Reply

  5. Hi Volodymyr,

    Thanks for such a wonderful post.
    I have tried to replicate the thing you did for UNIX shell script. I am goo with everything and cURL do not give any error message but my files are not loaded. (However, my files are uploaded and I do recieve a CONTENTID)

    Would you please advise on what should be taken care to resolve this.

    Thank you so very much for your time in looking into this.

    Regards,
    Sowdha

    Reply

  6. Hi Volodymyr,
    Brilliant article. Thanks for sharing the UNIX code.
    Can we call the RIDC document utility from the oracle DBCS Linux access.

    we had Oracle Saas service. Our requirement is to automate the HCM load. we have Oracle DBCS from the SSH i have transfer the “150521-REL10-oracle.ucm.fa_client_11.1.1.zip” which is the RIDC document utility file ““oracle.ucm.fa_client_11.1.1.jar””. under the /tmp folder i have unzip the RIDC.zip file. then call the
    command:
    ========
    java -jar ridc/oracle.ucm.fa_client_11.1.1.jar DownloadTool –ping –url=https://XXXX.oraclecloud.com/cs/idcplg –username=username –password=xxxx

    received the below error message:
    ==============================

    Oracle WebCenter Content Document Transfer Utility
    Oracle Fusion Applications
    Copyright (c) 2013-2014, Oracle. All rights reserved.
    Performing connection test (PING_SERVER) …
    java.net.SocketTimeoutException: connect timed out
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:579)
    at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:625)
    at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
    at sun.net.www.protocol.https.HttpsClient.(HttpsClient.java:275)

    Please do guide me to automate the process via UNIX.
    Thanks in Advance,
    Chandra

    Reply

  7. Hi Volodymyr,
    Very nice article. The article gives in depth guidance on HDL process.
    I have a requirement of uploading the zip files and calling the HDL. For this i’m idcclient ServiceResponse response = idcClient.sendRequest(userContext,
    request);

    After this i’m calling the hcmDataUpload
    HCMDataLoaderServiceSoapHttpPortClient.hcmDataUpload(serviceURL,username,
    password, keystoreTyp, keystorePath, keystorePass,
    encryptkeyAlias, encryptyKeyPass, recptKeyAlias
    , contentId)

    which calls the hcmLoaderService.importAndLoadData(contentID,parameters).

    All this is working fine when run from machine which is not proxy to connect HCM content server. But when the same is run from a machine which has proxy enabled, this fails giving error as
    2016-12-01 15:24:44 INFO – I/O exception (java.net.ConnectException) caught whe
    n processing request: Connection timed out: connect
    at oracle.stellent.ridc.protocol.http.IdcHttpProtocol.writeRequest(IdcHt
    tpProtocol.java:216)
    at oracle.stellent.ridc.IdcClient.sendRequest(IdcClient.java:181)
    at com.kotak.hcm.dataupload.util.HCMDataUploadUtil.checkin(HCMDataUpload
    Util.java:101)
    at com.kotak.hcm.dataupload.util.HCMDataUploadUtil.HCMDataUpload(HCMData
    UploadUtil.java:41)

    Have you tried this where proxy is used to connect to HCM content server.

    Thanks & Regards,
    Gauri

    Reply

  8. Pingback: HCM Data Loader Automation – Part II | HCM Cloud Hub

  9. Hi Volodymyr,

    Excellent article and very helpful.

    We’re using Oracle Fusion Financials R11 – a very simple question because I can’t seem to get the answer from Oracle SAAS Support as they are saying the developers recommend SoapUI. Can we use your method using HCM Data Loader to upload data from a third party on-premise site to Fusion Financials Cloud?

    I have tried to mimic your steps however with the java string I’m getting the following error:

    java -cp “oracle.ucm.fa_client_11.1.1.jar” oracle.ucm.client.UploadTool –url=”h
    ttps://{host}/cs/idcplg” –username=”user” –password=”password” –primaryFile=”/u01/WCUpload/aruploadcustomers.
    zip” –dDocTitle=”Department Load File” –dSecurityGroup=”FAFusionImportExport”
    –dDocAccount=”hcm/dataloader/import”

    WARNING: Unable to locate authentication handler for Content Server response: Not Found

    Error – Http status: Not Found

    IdcClientException Caught …
    oracle.stellent.ridc.protocol.http.HttpProtocolException: Http status: Not Found

    I can’t find any hits on this error – could it be due to the user privileges? I’m hoping you can help here because I’m not getting any joy from Oracle SAAS support.

    Thanks & Kind Regards,
    Faaiza

    Reply

  10. Hi Volodymyr,
    I was trying to do the same which you mentioned but i got the below exception.
    WARNING: Unable to locate authentication handler for Content Server response: Not Found
    Error – Http status: Not Found
    IdcClientException Caught …
    oracle.stellent.ridc.protocol.http.HttpProtocolException: Http status: Not Found
    at oracle.stellent.ridc.protocol.http.IdcHttpProtocol.writeRequest(IdcHttpProtocol.java:398)
    at oracle.stellent.ridc.IdcClient.sendRequest(IdcClient.java:179)
    at oracle.ucm.client.AbstractTool.sendRequest(AbstractTool.java:1088)
    at oracle.ucm.client.UploadTool.run(UploadTool.java:475)
    at oracle.ucm.client.UploadTool.main(UploadTool.java:191)
    Can you please help me out with this?
    Thanks in advance.
    Regards,
    DivyaKrishn

    Reply

  11. Dear Vladimir, im getting the following error:
    Error – UCMCLIENT_REQUIRED_OPTION_ERROR: Program option “dDocAccount” must be provided.
    java.lang.IllegalStateException: UCMCLIENT_REQUIRED_OPTION_ERROR: Program option “dDocAccount” must be provided.
    at oracle.ucm.client.UploadTool.getDocAccount(UploadTool.java:928)
    at oracle.ucm.client.UploadTool.processLocalFileIfSpecified(UploadTool.java:745)
    at oracle.ucm.client.UploadTool.run(UploadTool.java:280)
    at oracle.ucm.client.UploadTool.main(UploadTool.java:191)
    at oracle.ucm.client.JarEntryClass.main(JarEntryClass.java:84)
    Please Advise

    Reply

  13. Hi, very nicely explained,
    I received the content don’t and file is placed in /import folder, I am facing problem in invoking the service to load the .dat files in fusion.
    Please help on this.
    Regards,
    Anand

    Reply

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.