Automated Data Management Using AWS Serverless Framework

Automated Data Management Using AWS Serverless Framework

National Consumer Panel Automates Data Management using a Serverless Framework on AWS

WCI Data Solutions helped the National Consumer Panel (NCP) automate the management of consumer panel data by designing and developing a serverless framework using Amazon Web Services.


The National Consumer Panel (NCP) is a joint venture between Nielsen and IRi – the two leading consumer insight providers in the United States. They are a market research panel that focuses on measuring consumer attitudes and behavior. Since 1987, NCP has been gathering panel member’s opinions and shopping data to help manufacturers and retailers make informed decisions on what products to develop, improve or sell in the market.


NCP came to WCI with the challenge of handling the processing, validation and auditing of consumer panel data, feedback, receipts and reward payment.  Each consumer panel member is invited to participate in special programs and surveys and in return gain rewards.  The goal of the WCI solution was to simplify and automate the entire process.


The first step in the process was selecting Amazon Web Services (AWS) which allowed NCP to quickly scale to demand without expensive hardware purchases. Utilizing a serverless architecture & native AWS services proved to be the most efficient way of preparing for a workload that could be highly unpredictable.

Architecture Creation & Validation

  • WCI started by building API Gateway endpoint to receive receipt requests captured in the existing NCPMobile app by panelists.
  • With the use of S3, Lambda and SQS, we created tasks for Turk workers responsible for transcribing panelist receipts.
  • Build Lambda function triggered by API gateway endpoint to create HIT tasks for transcribing receipts.
  • For each received receipt create Mechanical Turk HIT for which input screen was built and manually tested. In order to audit the process, Lambda was used to suspend the autopay service
  • For each step of the process a persisted record was created and updated with the use of DynamoDB.
  • Data is streamed from the DynamoDB tables to an Aurora instance using Kinesis Streams.
  • The data from the Aurora instance is used to populate a Quicksight dashboard for analysis on the number of transcriptions being done and the number of unique workers working on those tickets.

The engagement with NCP required utilizing a broad array of services on AWS, including:

Amazon Elastic Compute Cloud (Amazon EC2)Amazon Relational Database Service (Amazon RDS), Amazon Glue, Amazon Athena, Kinesis Streams, Amazon CloudWatch, Amazon Simple Queue Service (Amazon SQS), AWS Lambda, Amazon Virtual Private Cloud Peering (VPC-Peering), Amazon QuickSight, AWS Identity and Access Management (IAM) and Amazon Simple Storage Service (Amazon S3).


NCP was able to automate the management of consumer panel data using a serverless framework, which enabled them to save reduce labor costs and avoid significant investments in hardware.

Not sure how to begin your journey adopting AWS? Let WCI be your guide.

Explore AWS Solutions

About WCI Data Solutions

WCI is a consulting services company focused on the application of data through the use of Business Intelligence and Data Warehouse technologies that bring increased business performance for our clients. WCI has serviced well over 300 companies through our knowledge and expertise of how to make data valuable to decision-makers.

Get Started

Want a FREE on-site discovery session with your team?  Receive a FREE data management evaluation session with one of our veteran data architects. Reap the benefits of an expert’s outlook on taking control of your business insights and data. Stop missing out on data-driven opportunities,  and start making smarter, more profitable decisions today.