New Evaluation Framework

We used a new Evaluation Framework for our latest Product Evaluation Report, which is about Salesforce Service Cloud. We introduced the new Framework to make our reports shorter and more easily actionable. Shorter for sure, our previous report on Service Cloud was 57 pages including illustrations. This one is 22 pages including illustrations, shorter by more than 60 percent!

We don’t yet know whether the Report is more easily actionable. It was just published. But, our approach to its writing was to minimize descriptions and to bring to the front our most salient analyses, conclusions, and recommendations.


Our Product Evaluation Reports had become increasingly valuable but to fewer readers. Business analysts facing a product selection decision, analysts for bankers and venture capitalists considering an investment decision, and suppliers’ competitive intelligence staff keeping up with the industry have always appreciated the reports, especially their depth and detail.

However, suppliers, whose products were the subjects of the reports, complained about their length and depth. Requests for more time to review the reports have become the norm, extending our publishing cycle. Then, when we finally get their responses, we’d see heavy commenting at the beginning of the reports but light commenting and no commenting at the end, as if they lost interest. Our editors have made the same complaints.

More significantly, readership, actually reading in general, is way down. Fewer people read…anything. These days, people want information in very small bites. Getting personal, for example, I loved Ron Chernow’s 800-page Hamilton, but I have spoken to so many who told me that it was too long. They couldn’t get through it and put it down unfinished, or, more typically, they wouldn’t even start it. I’m by no means comparing my Product Evaluation Reports to this masterpiece about American history. I’m just trying to emphasize the point.

Shorter Reports, No Less Research

While the Product Evaluation Report on Salesforce Service Cloud was 60 percent shorter, our research to write it was the same as our research for those previous, much longer Product Evaluation Reports. Our approach to research still has these elements, listed in order of increasing importance:

  • Supplier presentations and demonstrations
  • Supplier web content: web site, user and developer communities
  • Supplier SEC filings, especially Forms 10Q and 10K
  • Patent documentation, if appropriate
  • Product documentation, the manuals for administrators, users, and developers
  • Product trial

Product documentation and product trial are the most important research elements and we spend most of our research time in these two areas. Product documentation, the “manuals” for administrators, users, and developers provides complete, actual, accurate, and spin-less descriptions of how to setup and configure a product, of what a product does—its services and data, and of how it works. Product trials give us the opportunity to put our hands on a product and try it out for customer service tasks.

What’s In?

The new Framework has these four top-level evaluation criteria:

  • Customer Service Apps list and identify the key capabilities of the apps included in or, via features and/or add-ons, added to a customer service software product.
  • Channels, Devices, Languages list supported assisted-service and self-service channels, devices attachable to those channels, and languages that agents and customers may use to access the customer service apps on those devices.
  • Reporting examines the facilities to measure and present information about a product’s usage, performance, effectiveness, and efficiency. Analysts use this information continually to refine their customer service product deployments.
  • Product, Supplier, Offer. Product examines the history, release cycle, development plans, and customer base for a customer service product. They’re the factors that determine product viability. Supplier examines the factors that determine the supplier’s viability. Offer examines the supplier’s markets for the product and the product’s packaging and pricing.

This is the information that we use to evaluate a customer service product.

What’s Missing?

Technology descriptions and their finely granular analyses are out. For example, the new reports do not include tables listing and describing the attributes/fields of the data models for key customer service objects/records like cases and knowledge items or listing and describing the services that products provide for operating on those data models to perform customer service tasks. The new reports do not present analyses of individual data model attributes or individual services, either. Rather, the reports present a coarsely granular analysis of data models and services with a focus on strengths, limitations, and differentiators. We explain why data models might be rich and flexible or we identify important, missing types, attributes, and relationships then summarize the details that support our analysis.

“Customer Service Technologies” comprised more than half the evaluation criteria of the previous Framework and two thirds of the content of our previous Framework-based reports. These criteria described and analyzed case management, knowledge management, findability, integration, and reporting and analysis. For example, within case management, we examined case model, case management service, case sources, and case management tools. They’re out in the new version and they’re the reason the reports are shorter. But, they’re they basis of our analysis of the Customer Service Apps criterion. If a product has a rich case model and a large set of case management services, then rich case model and large set of case management services will be listed among the case management apps key capabilities in our Customer Services Apps Table and we’ll explain why we listed them in the analysis following the Table. On the other hand, if a product’s case model is limited, then case model will be absent from the Table’s list of key capabilities and we’ll call out the limitations in our analysis. Just a reminder, our bases for the evaluation of the Customer Service Apps criteria, the subcriteria of Technologies for the old Framework are shown in the Table below:

Slide1Table 1. We present the bases for the evaluation of the Customer Service App criteria in this Table.

Trustworthy Analysis

We had always felt that we had to demonstrate that we understood a technology to justify our analysis of that technology. We had also felt that you wanted and needed our analysis of all of that technology at the detailed level of every individual data attribute and service. You have taught us that you’d prefer higher-level analyses and low-level detail only to understand the most salient strengths, limitations, and differentiators.

The lesson that we’ve learned from you can be found in a new generation of Product Evaluation Reports. Take a look at our latest Report, our evaluation of Salesforce Service Cloud and let us know if we’ve truly learned that lesson.

Remember, though, if you need more detail, then ask us for it. We’ve done the research.


Microsoft Dynamics 365 for Customer Service

Serious Customer Service Capabilities

In our more than 10 years of customer service research, publishing, and consulting, we’d never before published a report about a Microsoft offering. It’s not because Microsoft hasn’t had a customer service offering or that the company hasn’t had success in business applications. Since 2003, its CRM suite has always included a customer service app. And, its Dynamics CRM brand has built a customer base of tens of thousands of accounts and millions of users. But, Dynamics CRM had always been more about its sales app and that app’s integration with Office and Outlook. Customer service capabilities have been a bit limited. No longer.

Beginning in November 2015, the improvements in two new releases—CRM 2016 and CRM 2016 Update 1—and, in November 2016, the introduction of the new Dynamics 365 brand have strengthened, even transformed, Microsoft’s customer service app and have made Microsoft a player to consider in the high end of the customer service space.

Our Product Evaluation Report on Microsoft Dynamics 365 for Customer Service, published December 1, 2016, will help that consideration. These are the new and/or significantly improved customer service components:

  • Knowledge management
  • Search
  • Customer service UI
  • Web self-service and communities
  • Social customer service

Let’s take a closer but brief look at each of them.

Knowledge Management

Knowledge Management is the name of a new customer service component. Introduced with CRM 2016, it’s a comprehensive knowledge management system with a rich and flexible knowledge model, a large set of useful knowledge management services, and an easy to learn and easy to use toolset. The best features of Knowledge Management are:

  • Visual tools of Interactive Service Hub, the customer service UI
  • Knowledge lifecycle and business processes that implement and support the lifecycle
  • Language support and translation
  • Version control
  • Roles for knowledge authors, owners, and managers

For example, Knowledge Management comes with a predefined but configurable knowledge lifecycle with Author, Review, Publish, and Expire phases. The screen shot in Figure 1 shows the steps in the Author phase.

ish-knowledge-author-stage-stepsFigure 1. This screen shot shows the steps in the Author phase of the knowledge management process.

Note that Knowledge Management is based on technology from Parature, a Reston, VA-based supplier with a customer service offering of the same name that Microsoft acquired in 2014. Beginning with the introduction of Dynamics 365, Microsoft no longer offers the Parature customer service product.


Search is not a strength of Dynamics 365. Search sources are limited. Search query syntax is simple. There are few search analyses and few facilities for search results management. However, with the Dynamics 365 rebranding Microsoft has made improvements. Categorized Search, the new name of the search facility in Dynamics 365, retrieves database records with fields that begin with the words in search queries and lets administrators and seekers facet (Categorize) search results. The new Relevance Search adds relevance and stemming analyses. Microsoft still has work to do, but faceting, stemming, and relevance are a start to address limitations.

Customer Service UI – Interactive Service Hub

Interactive Service Hub (ISH) provides several useful and very attractive capabilities in Dynamics 365. It’s the UI for Knowledge Management, one of two UIs for case management, and a facility for creating and presenting dashboards. For the case management and knowledge management UIs, ISH provides visual tools that are easy to learn and easy to use. The tools let agents perform every case management task and let authors and editors perform every knowledge management function. For example, Figure 2 shows a screen shot of ISH’s presentation of an existing Case—the Name of the Case at the top left, the Case information to display “SUMMARY | DETAILS | CASE RELATIONSHIPS | SLA” under the Name, the phases of the deployment’s case management process “IDENTIFY QUALIFY RESEARCH RESOLVE” within a ribbon near the top of the screen, and the (SUMMARY) Case information in the center.

ish-existing-caseFigure 2. This screen shot shows the Interactive Service Hub display of an existing Case.

In addition to tools for building dashboards, ISH also packages useful predefined dashboards, two for case management and two for knowledge management. The four help customer service managers and agents and knowledge management authors and editors manage their work. Figure 3 shows an example of the My Knowledge Dashboard. It presents information useful to authors and editors very visually and interactively.

my-knowledge-dashboardFigure 3. This screen shot shows an example of the My Knowledge Dashboard.

Web Self-service and Communities

We were quite surprised to learn that, prior to the May 2016 introduction of CRM 2016 Update 1, Dynamics 365 for Customer Service and all of its predecessor products did not include facilities for building and deploying web self-service or communities sites. This limitation was addressed in Update 1 with the then named CRM Portal service, renamed the Portal service in Dynamics 365. Portal service is a template-based toolkit for developing (web development skills are required) and deploying browser-based web self-service and communities/forums sites. It’s based on technology from Adxstudio, which Microsoft acquired in September 2015 and it packages templates for a Customer Service Portal and a Community Portal. Note that Dynamics 365 for Customer Service licenses include one million page views per month for runtime usage of sites built on the Portal service (licenses may be extended with additional page views per month).

Social Customer Service

Microsoft Social Engagement is a separately packaged and separately priced social customer service offering that Microsoft introduced early in 2015. Social Engagement provides facilities that listen for social posts across a wide range of social sources (Instagram, Tumblr, WordPress, and YouTube as well as Facebook and Twitter), that analyze the content and sentiment of those posts, and that interact with social posters. In addition, Social Engagement integrates with Dynamics 365 for Customer Service. Through this integration, the automated or manual analysis of social posts can result in creating and managing customer service Cases. It’s a strong social customer service offering. What’s new is Microsoft bundles Social Engagement with Dynamics 365 for Customer Service. That’s a very big value add.

All This and More

We’ve discussed the most significant new and improved capabilities of Dynamics 365 for Customer Service. Knowledge Management, Interactive Service Hub, improved Search, the Portal service, and bundled Social Engagement certainly strengthen the offering. Although not quite as significant, Microsoft added and improved many other capabilities, too. For example, there are language support improvements, improvements to integration with external apps, new Customer Survey and “Voice of the Customer” feedback capabilities, and the use of Azure ML (Machine Learning) to suggest Knowledge Management Articles as Case resolutions automatically based on Case attribute values. Bottom line, Microsoft Dynamics 365 for Customer Service deserves serious consideration as the key customer service app for large businesses and public sector organizations, especially those that are already Microsoft shops.

Framework for Evaluating Customer Service Products

This week’s report is a new version of our Framework for Evaluating Customer Service Software Products. We had two goals for its design. First, we wanted your evaluation, comparison, and selection processes to be simpler and faster. Second, we wanted shorter and more actionable Product Review Reports. The new Framework eliminates evaluation criteria that do not differentiate. For example, we no longer analyze and evaluate web content management for a product’s self-service and assisted-service UIs. These UIs have become a bit static. They’re configurable and localizable, but they’re no longer as customizable and manageable as they had been. The new Framework also decreases the number of factors (sub-criteria) that we consider within an evaluation criterion. For example, the Knowledge Management criterion now has two factors: Knowledge Model, and Knowledge Management Services. The previous version of the Framework examined these and six others.

We also added a criterion—Case Management. When we began evaluating customer service products back in 1993, we felt that case management, while a critical customer service process, was well understood, did not differentiate, and was not really customer-centric. We’ve changed our point of view. We still believe that the purpose for customer service is answering customers’ questions and solving customers’ problems. However, we also recognize that at the point in time that a customer asks a question or poses a problem you might not have an answer or solution available. You create a case to represent that question or problem, your process to resolve the case is a process to find or develop an answer or solution, and its resolution is, itself, the answer or solution. Our evaluation of case management considers four factors that focus on a product’s packaged services and tools for performing the tasks of the case management process. The process includes finding and using case resolutions in communities and social networks.

Customer Service Best Fit and Customer Service Technologies are the Framework’s two top-level evaluation criteria. Customer Service Best Fit presents information and analysis that classifies and describes customer service software products. Customer Service Technologies examines the implementation of a product’s customer service applications. The graphic below shows the Framework, its top-level criteria, and their sub-criteria.


We plan to use the Framework to evaluate every type of customer service product within our current research—case management, knowledge management, virtual assistant, and social network monitoring, analysis, and interaction. The Customer Service Best Fit criterion applies very nicely to any product. The application of the Customer Service Technologies criterion is product-type dependent. Look for our Product Review Report on Salesforce Service Cloud. It will be the first against the new Framework. Based on the draft of that report, the Framework works very nicely.