Last year, I had the opportunity to lead a Market Research and Benchmarking selection project for two associations. These associations survey their members each year about key industry metrics and current hot topics—and then after the survey closes, provide a system for respondents to benchmark themselves against their peers and to compare themselves against national trends.
Members of the associations can view or export the data and print reports and charts that analyze the results. These surveys have been running for many years, so they provide the associations and members with a great source of longitudinal and market research data.
In this article, I share some of the important considerations that factored into my evaluation for this client and that will likely be relevant to other organizations approaching similar evaluation processes.
Replacing Outdated Technology to Provide a Better Experience for the Associations and their Members
The primary challenge my clients faced was that their benchmarking system was outdated. The end user experience was suffering because the incumbent system’s interface and navigation were not intuitive, its reporting and visualization capabilities were not modern, and its survey distribution and administration tools had significant limitations.
Consequently, two main goals of the project were to improve and modernize the experience for the end users and to increase the efficiency of staff to administer the survey so that they could maximize the value of the tool.
Paving the Way for Gathering and Managing High-Quality Data
Before discussing benchmarking tools though, the first issue to address was—and should be for all organizations in this situation—the quality of the data. The data, after all, is what matters most—it is the new gold. A benchmarking tool without good data is worse than none at all, as it might provide confusing or irrelevant information that results in misleading analysis.
Consequently, it was important for the associations to work with a market research firm that understood data gathering methodologies, could design valid surveys, and knew how to ask clear and valid quantitative and qualitative questions that would provide statistically significant results. They needed to understand the target audiences, how to ensure their sampling was representative of the larger population, the context of the survey, and the parameters that would guide the respondents. Since the data for the associations’ benchmarking solutions was almost always gathered through online surveys, it was important not to seek information best gathered through focus groups or in-person interviews.
These types of decisions are best made by people with deep research experience and formal training in data analytics—and many nonprofits and associations do not have these capabilities in-house. If such an organization is considering creating and launching a benchmarking tool, the first step may be finding the right market research partner to ensure the data used in the benchmarking is of sufficient quality.
If this is your first time comprehensively surveying your stakeholders/constituents, you may want to work with a research firm that, if possible, has deep experience in your industry. If you have your own research department, you may still be well served by finding an excellent research firm outside of your industry that brings a fresh perspective, asks great questions, and helps you see your data in a new way.
In addition, if your organization is considering creating and launching a benchmarking tool for the first time, you might want to consider running a pilot where the data is analyzed by the research firm and returned to the respondents for feedback. Once you confirm what is important and valuable to your audience, then you can proceed to create a benchmarking tool that provides self-service features and ad hoc reporting and analysis.
Selecting a research partner and a benchmarking partner does not necessarily mean hiring two separate vendors, as some software firms that develop benchmarking tools have in-house research and data experts, and some consulting firms have proprietary benchmarking systems. But do your due diligence, as a software vendor may have less expertise in the research side of the equation, or vice versa—and it is very important to make a careful evaluation to see if the needs can be met by a single vendor, or if two vendors are needed to provide an excellent solution.
Categories of Benchmarking Solutions Available in the Market
In terms of the benchmarking solutions, the options can be divided into five main categories (although some firms cross boundaries, in part due to consolidation in the industry).
- Standalone, off-the-shelf benchmarking systems created for this specific purpose, provided under the SaaS model (software as a service) They come with a survey creation tool, a survey distribution mechanism, some respondent management functionality (i.e., payment processing if there is a fee to participate), and a pre-built benchmarking portal that allows respondents to analyze the results. Systems in this category are purposely built for benchmarking, so they understand the annual survey benchmarking cycle, and sometimes even provide end-user support for people filling out the surveys. Systems in this category tend to be among the most affordable and least complex options. On the other hand, their survey and benchmarking tools also tend to be less robust or flexible than some of the other options, and they usually do not have pre-built connectors with outside databases, because they are built for one main purpose—to benchmark data gathered through the tool.
- Business Intelligence companies that understand big data and can create beautiful dashboards and infographics They can be particularly good at combining data from multiple sources and transforming data as necessary. However, most of them do not have built in survey distribution and respondent management features, so they must partner with third party survey tools, and often must create custom “CRM-lite” functionality to manage users. Consequently, this option involves more complexity, which leads to higher costs—but the end result can be more visually appealing than off-the-shelf systems.
- Professional services consulting firms that provide market research and data analytics services and who also have proprietary benchmarking tools This option tends to be concentrated within the largest consulting firms. They will have deep market research experience and may bring a lot of benchmarking data with them. You will benefit from their knowledge of the field and their market data that has already been vetted and tested so that your survey design phase can be guided by tried-and-true methodologies. On the other hand, their benchmarking tools tend to be less flexible because they are built for specific vertical markets, and they can be costly to maintain because its proprietary software and ongoing consulting is built into the price.
- Survey software companies that have advanced survey creation tools and can also provide dashboards and analytics Their survey tools are the most powerful and flexible in this comparison. However, they are not usually in the business of creating benchmarking portals, and their respondent management tools do not track respondent history across surveys and years by default—so it requires a major custom development project to create the dashboard interface. This custom work can be costly, and there may he higher ongoing maintenance fees. However, some of these vendors are building benchmarking tools for the market, so the costs will likely be driven downward.
- Market research & data analytics companies that also build custom benchmarking tools for their clients These firms have deep expertise in data analytics and build benchmarking systems for their clients. As with the standalone, off-the-shelf benchmarking tools (first in this list, above), this option provides a “one-stop shop”—with all needed functionality integrated together. The difference is that each custom tool is heavily modified to match the requirements of the customer. Accordingly, the costs tend to be higher than the standalone products, but they also usually provide a more flexible design, integration with multiple data sources, a better custom fit, and usually have richer reporting features.
Where My Clients Landed
My association clients were looking for a vendor that could manage the end-to-end process, from survey distribution all the way through to creating the benchmarking tools to supporting the user of those tools through the fulfillment process. But they also knew it is rare for a vendor to excel in every single area across that process. Because they designed and ran their own surveys for many years, my clients felt comfortable with the quality of their data and the expertise they brought to the table in that area, and consequently they partnered with a data analytics firm that built custom benchmarking tools that would provide the best end-user experience.
Clarify Your Needs and Understand the Market Before Selecting a Vendor
Ultimately, looking for good benchmarking vendors and software can be a confusing process, because there are so many companies with different approaches, technologies, and services that provide different paths to meeting your needs. So, the most important thing to do before embarking on your selection effort is to clarify your own needs—and how they align with the categories of solutions available in the market—prior to engaging in a selection process. Doing so is much more likely to result in a successful project that meets both staff and survey respondent expectations.
Need More Expertise?
Ready to have a conversation about your software selection process? Or perhaps you are looking for an assessment and roadmap to ensure your organization is considering software investments with long term strategic value. Whatever your nonprofit technology consulting needs, Build is here to help.
