[+]
[-]
 [-]
   
   
   
   
   
   
 [+]
 [+]
Updated on 9/27/2019
Commissions Data Loader Publication
CDL Architecture and Data Transfer Process
Direct link to topic in this publication:

PREVIOUS  |  NEXT

CDL Architecture and Data Transfer Process


The following figure illustrates the CDL architecture and describes the data transfer process.


The CDL data transfer infrastructure uses secure FTP (SFTP) protocols and related toolsets to optimize the security and maintenance of the file transfer accounts and processes. The SFTP server is configured to allow only secure FTP, exclusively from the customer’s external IP. All Shell access or other protocols are blocked by the firewall policy.

In the case of inbound transfers, once the files are dropped off in the Inbound folder (CDL Secure Box) using SFTP, the File Management solution checks if the uploaded files are encrypted and encrypts the files if not encrypted. The File Management solution then performs various checks for template compliance and routes the files to the Inbound NFS Mount folder for pick up. 

CDL creates a job for every file transfer, and in conjunction with Kafka, it processes and validates the data to check for data types, data formats, duplications, and so on. See Data Transfer Validations for more information.

Once validations are complete, data is loaded into the corresponding staging tables in the HANA repository in Commissions and further into the Commissions database after various functional validations are performed through Relational Import. When the file transfer process is completed, CDL checks the status of the jobs and notifies the user who initiated the transfer, if the data has been successfully transferred. Details about the transfer, including time, event, status (success or failure) and error logs are refreshed and accessible through CDL during the entire process.

Note: CDL creates a job for each file that is being transferred and runs the data transfer process for each file. In the case of paired files, CDL identifies the dependent files and processes the files once all the dependent files are dropped. Pipeline Import is triggered only for parent file types. 


In outbound transfers, files are routed to the outbound NFS Mount folder and then dropped off into the outbound folder (CDL Secure Box). The extracted files can be downloaded from the customer’s FTP site. Email notifications are sent out when the data transfer is complete.

BusinessObjects

BusinessObjects is a component of CDL which maintains user group data. CDL interacts with BO and Commissions to sync user groups in the BO repository when the following BO templates are dropped for data transfer:
  • BOTRIG (Inbound transfer): Once the empty file is dropped, CDL identifies the BOTRIG file and communicates with Commissions (Participant Object), and obtains the user ID and label which has the group data, and syncs the details in BO.
  • BOUPDATE (Inbound transfer): CDL connects to BO and associates or disassociates users (as required) to the group and updates the BO repository. Data is picked up from the updated BO repository when reports are generated in Commissions.
  • BOEXTRACT (Outbound transfer): CDL extracts user group data and populates the flat files.


For more details, refer Commissions and BusinessObjects User Group Association.