4th Annual Internet MLS Customer Satisfaction Survey

The report is also available for download in Adobe Acrobat format (PDF).
You must have Adobe's Free Acrobat Reader software to view the document.

Introduction

In January through March of 2005, Clareity Consulting conducted its fourth annual survey of America’s leading Multiple Listing Services. The purpose of the study was to learn more about MLS customer satisfaction, system performance, and vendor responsiveness. This survey was designed as a follow up to Clareity’s previous Internet MLS Customer Satisfaction surveys and summarizes how well MLS Executives rated their vendor’s performance during 2004. This year, a record 169 MLSs participated, representing 759,971 subscribers. Several MLS vendors encouraged their customers to respond to this year’s survey and we thank those vendors for their cooperation. Clareity also thanks each of the MLSs that participated.

Executive Summary

The following MLS software vendors had enough responses where they are the primary MLS vendor to be included in the primary vendor survey results.

* Stratus has two large MLS customers where it is the primary vendor and both responded, so Stratus has been included in this year’s survey.

** Technology Concepts (average MLS response size = 373) and FBS (average MLS response size = 548) were included in the head-to-head comparisons with the larger vendors; however it should be noted that larger MLS customers generally require more customization and a higher level of service. FBS has been installed in two larger MLSs.

The following MLS vendors received only one response each as primary MLS vendors, and so are not included in the parts of the survey pertaining to primary MLS vendors: ARIS, eProperty Data, Quest Technologies, and Tarasoft. The following MLS vendors received no responses: Advanced Marketing Services, Coherent, dynaConnections, Hillside Software, RealGo, Superlative, and Systems Engineering Inc. (SEI). FBS originally only had one customer respond when the survey results were first analyzed, but several FBS customers requested the opportunity to participate, so Clareity has updated the report to include FBS and its additional customer responses. This updated version of the report also includes one additional larger Rapattoni MLS customer that requested to be included.

As one can see in the chart above, some vendors serve primarily large markets, some only serve small and medium markets, and some service a wide variety of market sizes. Certain vendors, such as Stratus and MarketLinx provide custom programming and special services for their customers, and some vendors do not. There are many factors to consider in selecting a MLS vendor. Every year Clareity leads a number of MLSs though a structured selection process, involving the:

• determination of member needs via surveys and focus groups,
• a request for proposal (RFP) and competitive bid,
• an analysis of the RFP where vendor capabilities are measured against member needs,
• system demonstrations,
• hands-on testing,
• and other processes designed to ensure the best system and vendor selection is made by each MLS.

Clareity’s MLS satisfaction survey is only one means of evaluating a vendor or the capabilities of their system, and that said, we hope you find this report valuable.

Customer Satisfaction

Always of the most interest to our survey readers, the following chart represents ‘end user satisfaction today’:

* MLS Owned responses are displayed separately from vendors throughout this report.

While some individual MLS accounts experienced end user satisfaction issues, all of the vendors scored well this year. EVERY MLS vendor received a total of 85%   ‘Good’ or ‘Excellent’ by its customers in this overall rating. This is an improvement from the previous year, where that figure was only 75%  . Also, for the first year, no vendor received a ‘Poor’ end user rating, so it is a pleasure to report that the vendors as a group are improving.

Three years ago when Clareity first performed this survey, the group of “new” MLS companies, led by MarketLinx, Stratus, and Rapattoni, outscored the established MLS vendors at implementation, end-user satisfaction, and vendor responsiveness. Last year, they were joined by FBS as a customer service leader.

This year:

• Stratus maintained industry-leading service of its two large-scale MLS customers
• FBS also maintained its above-average customer service record
• Technology Concepts, a vendor that primarily services smaller (sub-1000) member MLSs, made a strong initial showing in the survey.
• Solid Earth, Offutt Systems, Rapattoni, and Interealty followed in a very close grouping. Solid Earth and Offutt generally service small and medium size MLSs, while Rapattoni and Interealty have proven they can service MLSs of various sizes.

The following chart represents the trend in ‘end user satisfaction today’. Of note is the dramatic increase in satisfaction with Fidelity MLS, the substantial increase for Offutt Systems, as well as some disillusionment with MLSs owning and operating their own MLS and the decreases seen for some other MLS vendors.

Numeric score calculation:((3 x Excellent %  ) + Good %   - Fair %   - (3 x Poor %  ))

Clareity asked MLSs whether they measure satisfaction via surveys, and found that 44%   do so. Clareity asked this question to determine if the use of member surveys would correlate to how MLS executives answered the end-user satisfaction related question. The following cross-tab analysis shows that similar percentages of MLSs rated overall end-user satisfaction along a spectrum of Excellent to Poor, regardless of whether surveys were performed or not - so it appears most MLS executives have a high level understanding of their end-user’s satisfaction.

Surveys are an important part of keeping MLS leadership in alignment with subscriber needs and satisfaction levels. Clareity performs a number of surveys for MLSs every year, creating benchmarks for various aspects of MLS service and regularly measuring against that benchmark so an MLS can quantitatively evaluate and monitor its own performance and customer satisfaction of its various services.

When asked whether the MLS was hosted by the MLS vendor or locally hosted, 82%   responded that the system was hosted by the vendor and the remaining 18%   hosted the system locally. Clareity analyzed these results to determine if there was any correlation between hosting and end-user overall satisfaction and found there was no significant relationship.

There was, however, a relationship between hosting and satisfaction with system performance and uptime, where there was an eight point increase in the ‘Excellent’ rating when the system was locally hosted and the only ‘Acceptable’ rating was given to a vendor-hosted system.

Also, as seen in the chart below, there was clearly a relationship between hosting and system response time during peak periods, with the strong advantage again going to locally hosted systems:

76%   of locally hosted systems were rated as having ‘Excellent’ system response time during peak periods, compared with only 60%   of vendor-hosted systems. 7%   rated the response time of locally hosted systems ‘Acceptable’, while vendor-hosted systems were rated almost twice that percentage ‘Acceptable’ and ‘Poor’.

The following chart represents ‘Overall staff satisfaction’ with the systems:

As in previous years, in the chart above we see some differences from the end user satisfaction ratings – but the systems that satisfied the end users the most were given the best ratings by MLS staff, too. There are other aspects of the system and service that affect the staff and end-user ratings and those will be illustrated in the sections below.

Both overall ratings tied closely to how respondents answered the question, “If you could do it over again, would you select this vendor again?” the results of which are illustrated in the chart below:

Last year, when this question was asked, 30%   of FNIS (now Fidelity MLS) customers answered the question negatively, and this year that percentage is reduced to 9%   - a definite improvement.

There was only one case where the previous question was answered negatively, but the question, “If you could do it over again, would you select this Internet MLS system again?” was answered positively. In that case, the vendor experience is positive, but the respondent seems to desire one of the other systems offered by the vendor and Clareity assumes they may elect to migrate to another system rather than to another vendor.

System Features, Performance and Reliability

As MLSs continue to expand in functionality, maintaining a simple, easy-to-use user interface can be difficult. Ease of use has an impact on the amount of training and support a system requires, and is an important component of end user satisfaction. Systems were rated on ‘User interface (i.e. ease of use, minimal keystrokes to accomplish tasks)’ as follows:

Generally, the MLSs are satisfied with the user interface of the system that they selected, but Clareity has been called on during the past year to help identify and resolve usability issues, as spurred by end user complaints to the leadership of regional MLSs.

The following chart illustrates ‘Response times during peak periods’. Some of the systems were well rated, but over the past few years, several systems are receiving lower ratings in this area. Either customer’s expectations have grown greater, or the systems are not performing as well. Clareity suggests that the vendors may wish to track system performance during peak times and make improvements or, if applicable, gauge expectations and figure out a way to reverse this customer perception.

All vendors were rated exceptionally well for ‘Reliability and system up-time’. The trend in this area has continued to improve over the past few years, and this year all vendors scored 96%   or higher as 'Excellent' and 'Good'.

System Features

The MLS vendors all work hard to make core features such as Search and Listing Maintenance robust and conform to customer needs, but there are a number of other functional areas where they differentiate. The following are the ratings for each of these features:

Public records usability and integration

Mapping

Reports (CMAs, Statistics, and other print reports)

IDX Solution - Implementation and Data Conversion

IDX Solution – Overall Satisfaction

RETS

Stratus was not rated by its customers on RETS. It is interesting to note that the MLS Owned and Operated systems that responded are apparently not having an easy time with RETS.

Clareity asked respondents to rate their MLS vendor on the “Pace of software development and enhancements,” and the ratings were as follows:

When asked if their MLS vendor “Keeps pace with evolving technology (e.g. RETS, Mobile Devices)” the ratings were as follows:

Clareity wonders if Marketlinx was penalized in the previous two ratings for the time taken to develop its new 4.0 release, during which time it may have decreased the development pace of its previous version. Likewise, Fidelity MLS has focused much of its resources on its flagship Paragon product. Otherwise, vendors were rated well with regard to their development pace. The MLSs that own and operate their own systems rated themselves well in terms of keeping pace, but this is inconsistent with some of their other self-ratings, especially regarding RETS.

If a vendor does not provide all the features required by an MLS, sometimes the MLS core functionality can be supplemented by using 3^rd party software and information providers. The following chart illustrates how vendors were ranked on their ability and willingness to “Cooperate with other information and technology vendors”.

Vendor Responsiveness and Customer Service

The following chart illustrates how vendors were rated on “responsiveness to system change requests”.

The ratings above are similar to, but distinct from, the respondents rating of ‘Vendor's willingness to customize the system to your local needs’, which has more to do with initial customization of the system to meet specific local market needs. It can be difficult to make a transition to a new MLS system if one is unable to specify distinctive market requirements and have them implemented. Vendors rated as follows:

One can also see a clear correlation between the customization and staff satisfaction. 40%   of those respondents rating a vendor as ‘Poor’ on customization rated their staff satisfaction as ‘Acceptable’ and the remaining 60%   as ‘Poor’.

MLS vendors were rated on end-user support as follows:

MLS vendors were rated on staff support. Offutt and Fidelity MLS showed improvement over previous years, while some other vendors did not rate as well as in the past. However, it is important to note that every vendor received 85%   or higher combined ‘Excellent’ and ‘Good’ ratings.

The respondents also rated the vendors on ‘Timely and comprehensive communications regarding system and support issues’.

Value

Clareity asked respondents to ‘Rate the value of your MLS system based on what the vendor for the price you are paying’. This may be the most difficult question we ask of our respondents and requires a complex judgment call, balancing the software and service against the price paid for the system. Last year, almost every system exceeded a 90%   rating of ‘Excellent’ and ‘Good’ and results were close enough that the rank was not significant, but this year there was a lot more variation in rating. The top five value-rated vendors were as follows:

1. MLS Owned
2. Stratus
3. Technology Concepts
4. FBS
5. Offutt Systems

System Conversions

This year, Clareity asked respondents to answer the conversion-related questions only if their organization had undergone a system conversion in 2004. Stratus customers did not convert this year, and so they are not rated in this section.

“End-user satisfaction initially (first 30 days)” is a key rating, since issues from the earliest days of the system going live can affect the perceptions of the system for several years. Ratings are as follows:

Clareity asked respondents, “How many months did you run in parallel before transitioning to standalone?” and the answers were as follows:

There was no clear correlation between parallel period length and initial end-user satisfaction.

Respondents rated “Vendor's implementation planning and preparation” and the top five rated MLS vendors were as follows:

• FBS
• Solid Earth
• Interealty
• Technology Concepts
• Rapattoni

Respondents rated “Vendor's data conversion accuracy” and the top five rated MLS vendors were as follows:

• FBS
• Interealty
• Solid Earth
• Technology Concepts
• Fidelity MLS

Respondents rated “Vendor's ability to hit deadlines and deliver system on time” and the top five rated MLS vendors were as follows:

• Technology Concepts
• Interealty
• FBS
• Solid Earth
• Rapattoni

Some vendors had significant issues with on-time delivery this past year.

Satisfaction with implementations can sometimes be traced to the ability to deliver on promises made during the initial sales presentations. Clareity asked the MLSs to rate the vendors on “Truthfulness in representing their software and capabilities during the initial sales presentations; compared to what was delivered” The following chart shows the responses:

Parallel Systems

Respondents were asked to rate their “Overall satisfaction with parallel system vendor”. dynaConnections, Technology Concepts, Offutt Systems, and Fidelity MLS each received single “Excellent” ratings. Tarasoft received two “Excellent” ratings. Rapattoni received a singular “Good” rating. The other vendors that provide parallel system service received mixed ratings, including some that were “Fair” and “Poor”.

Recommendations from MLS Executives

Respondents were asked to share the top three things they would do differently in the system/vendor selection, contract negotiations, conversion planning, user training, or member communications. Clareity received hundreds of excellent recommendations.

As usual, the most common advice was not to skimp on training and communication - both before and after cutover. Training is always more effective in smaller groups with handouts, and this was reflected in the comments. Having a vendor presence on-site was also recommended several times. More than three respondents indicated that a short parallel period is preferable, which was interesting given the survey results showed more variation.

Here were some other great comments:

Selection

• Perform More research into various MLS systems and features that each would offer.
• Talk directly with current users both staff and members.
• Ensure you meet customer needs and not what the MLS "thinks" are customer needs.
• Recognize the stability and quality of the vendor and how it runs its business is significantly more important than the nuances of the MLS system itself.
• Investigate existing systems developed/built by MLSs
• Don't get caught up in technology fever - sizzle may sell but few use it.
• The three most important things to consider in an MLS system are reliability...reliability and RELIABILITY.
• Check references
• Discuss future product vision
• Request Data security measures be included in proposals.

Of course, Clareity likes seeing a comment like this one - "Use an independent consultant for the entire process, not just for one portion."

Contract

• Plan for a longer period of contract negotiations, i.e. 6-9 months in lieu of 3-4 months.
• In negotiating the contract, ensure there are system performance, tech support and issue remediation standards and penalties.
• Everything is negotiable. Negotiate, negotiate, negotiate!
• Clear remedy for failure to perform by specified dates
• A kinder, fairer negotiating position - both parties win!
• Try to limit agreements to a maximum of three years.
• Identify more specific terms of what happens to licensing terms and source code if vendor goes out of business or is purchased by another entity. Would have tried to get better documentation on existing source code.
• Ensure system feature specifics are included in the contract, including provisions for future enhancements
• Equate diminished response time (performance) with downtime.
• Tie payment to delivery of service more rigorously in contract
• Implement more enforceable controls in the design and build timeline
• Don't accept a blanket "we will give you everything you currently have" promise. It never happens - so get what you want in writing

Implementation

• Demand a minimum of 7 - 8 months of conversion planning time. We were given less than 4 and we have paid the price with poor reports and large number of change requests.
• Don't try to make any data changes during conversion.
• More extensive user training communication and office visits.
• Steering Committee with MLS & Broker Representatives helping guide system evolution.

Parallel Period

• More time for conversion.
• At least one month parallel operation and debugging.
• Insist on more testing prior to cutover
• Anyone changing MLS systems take responsibility for final testing and not cutover until they were satisfied the new system was fully functional for their particular needs.
• Have an extended period of parallel operations, minimum of 60 days to identify bugs and give vendor adequate time to correct.

Training and Communications

• Keep it in house as much as possible for higher quality training
• Don't let the vendor tell you when user training should begin -- set dates for user training when YOU think the system is ready.
• Be up front with your members; let them know as many details as possible.
• Can never have enough training!
• Mandatory training
• Have all user training sessions within two weeks prior to cutover date. Some of our members trained 8 weeks prior to cutover and forgot much of what they learned.
• Provide a written document which lists common MLS-related tasks from the prior MLS system and how to perform the equivalent function in the new system.
• Initially offer 3 levels of agent training: Beginner, Competent, and Expert instead of one size fits all.

Cutover

• Perform a site readiness review of every office before install and more site visits AFTER install
• Request on site support from my vendor during conversion for agents as well as staff.

Other

• Create a partnership relationship with your MLS provider not a vendor relationship.
• Ensure you have a complete buy-in and understanding of your elected officials (don't just accept their bobble-head nods at the table, drill into everything that's involved the migration so they really understand and appreciate the task at hand).
• Install a link on every listing that auto creates and addresses an email to the MLS to report a problem/issue with that listing. Make it easy, simple, and anonymous to report mistakes and violations.

Clareity was encouraged to see comments like: "For the planning, training and communications, a consultant, serving as a facilitator, would probably be worth the investment, especially in a quality-control capacity (checking for bugs in new system). It is difficult to get a new system up and running while keeping pace with the ongoing, every-day operations of an association and an MLS." and "Would have utilized a consultant in developing an RFP and negotiating contract terms."

Thank You!

Thank you for reading our 2005 report! And thanks again to the 169 MLS organizations representing nearly 760,000 subscribers that participated in this year’s survey.

The Clareity Team

About Clareity

Founded in 1996, Clareity continually strives to provide our clients an independent and unique perspective. Due to our extensive involvement and interaction across the entire Real Estate industry, we have a finger on the pulse of the industry. Clareity has successfully executed a vast array of consulting projects for our clients, related to:

• IT Security Audit and business continuity assessment
• Development and analysis of RFPs for MLS systems, public records, broker systems, transaction management systems and IP telephone systems
• Mergers and acquisitions and strategic alliances
• Strategic planning

• New product marketing and business plans
• Product integration specifications
• Public speaking and presentations
• Conference planning and content development
• Competitive analysis
• Contract negotiation
• Executive recruitment
• Project management and implementation assistance
• Market research including agent, broker, and staff electronic and telephone surveys as well as onsite focus groups

For more information please contact:

Gregg Larson

Clareity Consulting

(480) 368-8100 x201

www.callclareity.com

Home Page  |  About Clareity  |  Services  |  Clients  |  Publications  |  Events  |  Contact