Contact Us 214.891.7417 Let's Chat

Serverless Data Lakes on Amazon Web Services (AWS)

    Loan Marketplace Harnesses Their Data Using Serverless Data Lakes

    WCI Data Solutions was able to help the largest small business loan marketplace in the United States harness their data by designing and developing a serverless data lake service on Amazon Web Services (AWS) with Amazon S3 as the primary storage platform. 

     

    Background

    Our customer is the largest small business loan marketplace in the United States. Small business owners utilize their free online service to find financing by browsing multiple loan products from a network of more than 75 lenders. The platform reviews metrics including the business’s financial projections, use of funds, industry, and monthly revenue to find the right loan option for borrowers, helping traditionally underserved groups like women and minority-owned small businesses, mom and pop shops, and seasonal businesses. Their matching engine allows small business borrowers the opportunity to comparison shop across a broad range of lenders and loan products, decreasing their time and effort, and optimizing which lenders to send applications to.

     

    Challenge

    Our customer came to us with an interest in the development and deployment of a Data Lake solution on Amazon Web Services. They wanted a system capable of supporting both analysis and reporting across the organization but also designed to accommodate future consumers, query methods, and new data sources. They wanted to begin this development with an internally developed system we’ll refer to as their “CRM”. The CRM application is also deployed in AWS and uses Aurora RDS as a backend. This will be the first data source used by the WCI developed solution, with other sources to follow. The data will be consumed by the organization using both Domo and Power BI.

     

    Solution

    The engagement required utilizing a broad array of services on AWS, including:

    Amazon Elastic Compute Cloud (Amazon EC2), Amazon Relational Database Service (Amazon RDS), Amazon Glue, Amazon Athena, Amazon CloudWatch, Amazon Simple Queue Service (Amazon SQS), AWS Lambda, Amazon Virtual Private Cloud Peering (VPC-Peering), AWS Identity and Access Management (IAM) and Amazon Simple Storage Service (Amazon S3).

    Serverless Data Lake Architecture on AWS

    Serverless Data Lake Architecture on AWS

     

    WCI assisted the customer in solving this problem by designing and building a serverless data lake service on AWS with Amazon S3 as the primary storage platform. Utilizing an Amazon S3-based data lake architecture allows our customer the means to have a centralized, secure, and highly durable cloud-based storage platform. AWS makes it easy to store data in any format, both securely and at a massive scale with Amazon S3. In an effort to reduce both complexity and cost of development, WCI decided to utilize Amazon Athena as a query service to produce views that could then be used, in conjunction with Domo, to provide valuable business metrics and insights from the CRM data.

     

    With the use of WCI’s solution, our customer now has the ability to scale their data analytics platform as they continue to add data from new sources, regardless of whether it be structured or unstructured data. The new cost-efficient and cloud-based architecture provides business end-users a data analytics process that is both expedient and lends itself to being future proof. Using AWS native services like Athena, Glue and Lambda enables the ability to continue improving data analysis without the burden of maintaining infrastructure or complex ETL processes, and with WCI’s assistance, our customer now has a secured end-to-end framework for their data and analytics practice on AWS.

     

    Benefits

    Our customer now has the ability to harness increased data from various sources and options for future integrations and for use in other business-critical functions. This gives them the capability to analyze their data using different analytics tools and services, which allows them to derive more insights and provide more value for their customers, suppliers, and partners. WCI developed solution allows them the ability to gain deeper insights into their data in ways that traditional data silos cannot, which in turn helps them continue to support their customers better than any other lending platform.

    Not sure how to begin your journey adopting AWS? Let WCI be your guide.

    Explore AWS Solutions

    About WCI Data Solutions

    WCI is a consulting services company focused on the application of data through the use of Business Intelligence and Data Warehouse technologies that bring increased business performance for our clients. WCI has serviced well over 300 companies through our knowledge and expertise of how to make data valuable to decision-makers.