Automated Data Management Using AWS Serverless Framework
National Consumer Panel Automates Data Management using a Serverless Framework on AWS
WCI Data Solutions helped the National Consumer Panel (NCP) automate the management of consumer panel data by designing and developing a serverless framework using Amazon Web Services.
Background
The National Consumer Panel (NCP) is a joint venture between Nielsen and IRi – the two leading consumer insight providers in the United States. They are a market research panel that focuses on measuring consumer attitudes and behavior. Since 1987, NCP has been gathering panel member’s opinions and shopping data to help manufacturers and retailers make informed decisions on what products to develop, improve or sell in the market.
Challenge
NCP came to WCI with the challenge of handling the processing, validation and auditing of consumer panel data, feedback, receipts and reward payment. Each consumer panel member is invited to participate in special programs and surveys and in return gain rewards. The goal of the WCI solution was to simplify and automate the entire process.
Solution
The first step in the process was selecting Amazon Web Services (AWS) which allowed NCP to quickly scale to demand without expensive hardware purchases. Utilizing a serverless architecture & native AWS services proved to be the most efficient way of preparing for a workload that could be highly unpredictable.
Architecture Creation & Validation
- WCI started by building API Gateway endpoint to receive receipt requests captured in the existing NCPMobile app by panelists.
- With the use of S3, Lambda and SQS, we created tasks for Turk workers responsible for transcribing panelist receipts.
- Build Lambda function triggered by API gateway endpoint to create HIT tasks for transcribing receipts.
- For each received receipt create Mechanical Turk HIT for which input screen was built and manually tested. In order to audit the process, Lambda was used to suspend the autopay service
- For each step of the process a persisted record was created and updated with the use of DynamoDB.
- Data is streamed from the DynamoDB tables to an Aurora instance using Kinesis Streams.
- The data from the Aurora instance is used to populate a Quicksight dashboard for analysis on the number of transcriptions being done and the number of unique workers working on those tickets.
The engagement with NCP required utilizing a broad array of services on AWS, including:
Amazon Elastic Compute Cloud (Amazon EC2)Amazon Relational Database Service (Amazon RDS), Amazon Glue, Amazon Athena, Kinesis Streams, Amazon CloudWatch, Amazon Simple Queue Service (Amazon SQS), AWS Lambda, Amazon Virtual Private Cloud Peering (VPC-Peering), Amazon QuickSight, AWS Identity and Access Management (IAM) and Amazon Simple Storage Service (Amazon S3).
Benefits
NCP was able to automate the management of consumer panel data using a serverless framework, which enabled them to save reduce labor costs and avoid significant investments in hardware.
Not sure how to begin your journey adopting AWS? Let WCI be your guide.
About WCI Data Solutions
WCI is a consulting services company focused on the application of data through the use of Business Intelligence and Data Warehouse technologies that bring increased business performance for our clients. WCI has serviced well over 300 companies through our knowledge and expertise of how to make data valuable to decision-makers.
- Founded in 1998
- Full-Time Consultants all based in the U.S. with an average tenure of 7 years.
- Because there is a significant spectrum across the 300 customers we have serviced, we pride ourselves on our flexibility and the types of services we provide necessary to help our customers succeed no matter the platform, environment or objective.
- Forbes | America’s Best Consulting Firms | Big Data & Analytics – 2016, 2017, 2018 & 2019
