The Client

A leading growth accelerator for online marketplace retailers.

Project Requirement & Challenges

The Merchants were selling their products thru Amazon. They were supposed to maintain the inventory based on the demand and purchases. The transaction details, inventory, purchase order details etc., were available only on Amazon Market Place as a Report to be downloaded by the merchants.

This was a manual process and time consuming too since the reports will be available only after a day of actual transaction. There have been times when due to delay in downloading reports, merchants have missed big orders and also delayed the delivery of products to the customers. The Client came up with the requirement of automating the whole process of downloading the reports and pushing the details to their data warehouse.

Our Solution

As part of Technology team, we came up with the below ideas:

• Created a screen scraping tool using Python to login to the Seller Central and downloaded all the required reported based on data & time range for each merchant.
• All the reports were in different formats like JSON, PFD, excel. Before moving the reports to the staging server, we converted all the reports in same format – excel.
• Automated the process of Data Ingestion by pushing the downloaded reports to Azure Data Lake.
• Setup the data pipeline by creating a Data Factory to push the data from Data Lake to Data Warehouse.
• Power BI reports to pick the data from Data warehouse and display the required details for each Merchants.

Output

This approach has enabled us to completely automate the Data Collection, Ingestion and Transformation process. The Client was able to overcome the delays due to manual intervention of downloading reports and processing and pushing the data-to-data warehouse.