Customer Feedback Sentiment Analysis at Scale – Enterprise Software Case Study

LXT ran a sustained, human-powered feedback sentiment analysis program that turned 1,000+ weekly free-text entries into actionable product intelligence — and ultimately into the training foundation for the client’s own AI classifier.

The Challenge

A leading German provider of professional accounting and tax software receives more than 1,000 pieces of free-text user feedback every week through its product interface. The feedback arrives unstructured, written by a range of users — from accountants and tax advisors to business owners — and covers everything from bugs and missing features to praise for specific functionality. To be useful, this stream of raw text needed to be reliably mapped to a consistent taxonomy of product themes through systematic customer feedback sentiment analysis.

Specifically, the program had to:

• Process a continuous, high-volume stream of German-language free-text feedback on an ongoing basis, with new input arriving every few weeks.
• Classify each piece of customer feedback against a 10-category codebook covering both negative dimensions (poor usability, bugs, performance issues, missing features) and positive dimensions (good usability, efficiency, stability, feature satisfaction, regulatory currency).
• Handle multi-label sentiment analysis correctly: a single comment could validly belong to several categories simultaneously, including opposing sentiment categories within the same text.
• Adapt to an evolving requirements specification, as the client revised the codebook over time to reflect new product areas and refinements to category definitions.
• Deliver structured output in a specific Excel format that plugged directly into the client’s internal reporting workflow.

Automated classification and off-the-shelf AI tools were not reliable enough to interpret the nuance and domain-specific language of professional software feedback. The client required human judgment to correctly apply a detailed codebook — particularly for ambiguous texts, mixed-sentiment comments, and edge cases such as service complaints and security mentions that the taxonomy explicitly excluded from categorization.

The Solution

LXT assembled a qualified German-speaking crowd and built a structured annotation workflow around the client’s codebook, delivering categorized feedback sentiment analysis on a rolling basis throughout the engagement.

Qualified Crowd, by Design
Contributors were required to pass Gold Data qualification tests before accessing production tasks. Gold Data testing exposed annotators to pre-labeled examples and measured their ability to apply the codebook correctly, ensuring only those who demonstrated sufficient understanding of the category definitions worked on client data.

Structured Multi-Label Sentiment Categorization
The annotation task required contributors to assess each piece of free-text against all ten sentiment categories independently, selecting yes or no for each. The design explicitly required annotators to mark multiple categories for a single comment where applicable — including cases where a text expressed both positive and negative sentiment simultaneously. This preserved the full informational content of each response rather than forcing a single primary label.

Codebook Discipline and Edge-Case Handling
Task guidelines provided clear rules for ambiguous situations. Comments referencing general customer service quality, rather than the software itself, were categorized as none of the ten categories. Security comments were handled the same way unless they explicitly referenced legal or regulatory compliance, in which case the legal currency category applied. These rules were documented and enforced consistently across the contributor pool.

Adaptive Delivery Across a Sustained Engagement
The client revised category definitions and codebook rules on several occasions as the product evolved. Each time, LXT updated contributor instructions and retrained the active crowd accordingly, maintaining consistency of output through each transition. Delivery followed a recurring cycle: the client provided a new batch of feedback texts, LXT categorized them and returned structured Excel files in the client’s specified format.

Project Specifications

The exact numbers and rules behind the approach above, scannable at a glance.

• Task Type: Multi-label free-text classification / feedback sentiment analysis
• Sentiment Categories: 10 (5 negative, 4 positive, 1 regulatory)
• Language: German
• Crowd Type: German-speaking contributors only
• Qualification: Gold Data tests required before production access
• Feedback Volume: 1,000+ items per week from client’s product interface
• Total Items Categorized: 117k+ across the full engagement
• Delivery Model: Recurring batches; new input every few weeks, output returned as structured Excel files by email
• Requirements Changes: Codebook revised multiple times; crowd retrained at each revision
• Edge Cases: Service comments and general security mentions: no category assigned; regulatory security references: category 10 (legally current) applied

Contributor Task Flow

1. Pass Gold Data qualification test to demonstrate correct application of the 10-category sentiment codebook before accessing production tasks.
2. Read the German-language free-text comment and the accompanying satisfaction rating (scale from −2 very dissatisfied to +2 very satisfied).
3. Assess the comment against each of the 10 sentiment categories independently, applying codebook definitions and signal-word guidance.
4. Select yes or no for each category; mark all applicable categories including simultaneous positive and negative sentiment labels within the same text.
5. Apply edge-case rules: mark all categories no for general service comments or security mentions without a regulatory reference.
6. Review all selections before submitting, checking that no applicable category was missed.

Outcomes

• Delivered structured, multi-label customer feedback sentiment analysis across more than 117,000 pieces of German-language product feedback over the course of the engagement.
• Enabled the client to convert an unstructured weekly feedback stream into consistent, reportable data that fed directly into product management workflows.
• Maintained delivery continuity through multiple codebook revisions, keeping output quality stable as requirements evolved.
• Provided a human-powered feedback sentiment analysis alternative at a point when AI-based classification was not sufficiently reliable for the nuance and domain specificity of professional software feedback.
• The client ultimately automated the categorization process using AI, with the LXT-labeled dataset serving as the training foundation for their own classifier.

Why LXT

• German-speaking contributor pool with domain-appropriate qualification via Gold Data testing — essential for accurate sentiment analysis for customer feedback in professional software contexts.
• Sustained output quality: proven capacity to maintain consistent results across a high-cadence engagement with requirements that evolved over time.
• Flexible delivery model: structured Excel output matched directly to the client’s internal reporting format, with no integration overhead.
• Codebook governance capability: updated contributor guidance and crowd retraining at each requirements revision with no disruption to delivery.