I attended a presentation from Dale Ross and Marty Frame to a variety of MLS and public records software vendors last Sunday at the NAR Convention in San Diego. Here are a few points I took away from the Q&A session:
There were questions from other vendors as well about the APIs and the business model, but the above were the highlights for me. I walked away from the meeting thinking they were missing a lot of the potential for how a system like RPR could help MLSs and their broker and agent members. Helping MLSs improve data quality, data standards, and data sharing are all key benefits not being addressed by RPR. Hopefully this will change over the coming months as MLSs negotiate licenses with RPR and require that these issues be part of the deal.
——————————————————————–
Related and recent posts RPR by others:
Brian Larson — Report of RPRs Birth Is An Exaggeration
Rob Hahn — No More Drama and Hype: Known Facts on RPR
——————————————————————–
P.S. Can I please request that all NAR conventions from here on out be in San Diego? What a perfect location, with a great convention center, great hotels nearby, awesome restaurants, entertainment, and, of course, the views and weather!
I wrote yesterday why NAR’s RPR won’t mean the death of the MLS. Today, I want to ask some more important questions regarding the terms on which MLSs will allow NAR’s RPR to use (license) their MLS data. I suspect NAR will be presenting a lot of MLSs with some sort of terms of use over the next few weeks and months, promising their members “free” access to the RPR in exchange for their MLS data. There is some basic information on a fact sheet NAR posted about RPR (thanks, Kevin McQueen!) but the fact sheet doesn’t provide many details.
If I were negotiating this terms of use, I’d want to know answers to at least the following questions:
I understand a press release will be coming later today, but my guess is that the press release will be more hype than details. The real facts will come in the proposed license agreements NAR tries to get MLSs to sign. I think a system like RPR has a ton of opportunity for MLSs to make their members more efficient, but the key to leveraging the RPR opportunity will be in the license agreement for the MLS data. RPR won’t amount to much without the MLS data and so that value should be recognized by the brokers and agents in the MLSs providing the data.
Update: Here’s NAR’s press release for November 9. Not much different from the fact sheet except that CAR is going to partner with RPR with their ZipForms and Relay transaction management systems. I’ll probably post more about that later.
There’s been quite a dustup over the decision reportedly made by the Indianapolis Metropolitan Board of REALTORS® (MIBOR) that their MLS IDX rules against “scraping” also prohibit Google from indexing an agent’s site showing IDX listings.
For a bit of background, indexing is what Google does — it crawls the web and creates indexes of as much of it as it can so that when people search on Google it can return relevant results quickly. Here’s what Wikipedia has to say about scraping (with some emphases from me added):
Web scraping (or Web harvesting, Web data extraction) is a computer software technique of extracting information from websites. Usually, such software programs simulate human exploration of the Web by either implementing low-level Hypertext Transfer Protocol (HTTP), or embedding certain full-fledged Web browsers, such as the Internet Explorer (IE) and the Mozilla Web browser. Web scraping is closely related to Web indexing, which indexes Web content using a bot and is a universal technique adopted by most search engines. In contrast, Web scraping focuses more on the transformation of unstructured Web content, typically in HTML format, into structured data that can be stored and analyzed in a central local database or spreadsheet. Web scraping is also related to Web automation, which simulates human Web browsing using computer software.
The highlighted sentence is where the confusion begins on this issue. Scraping and indexing are closely related. That they are different, however, is emphasized by the important words “in contrast” that follow the “closely related” sentence. Put together, indexing is “closely related” to scraping but it is “in contrast” to it in what I think are important ways, namely the resulting use of the data. I’ll expound on this more below, but, for now, back to the controversy at hand.
In responding to the post on Agent Genius, Hilary Marsh from NAR said:
. . . questions have arisen about the scope of the requirement that IDX site operators protect the listings of other participants displayed on their IDX sites from “scraping”. Specifically, whether the policy distinguishes between “malicious” scraping and what might be considered “good” or “benign” scraping. Also, whether “indexing” is a type of scraping. The Center for REALTOR® Technology (”CRT”) advised that while the intent of “scrapers” may be malicious, and the intent of “indexers” good, the two practices from the Web server’s view appear to be the same. Consequently, NAR staff responded to questioners that the requirement to prevent scraping includes indexing.
So, the rub of the issue is that MIBOR punted the ball back to NAR, which asked CRT, and CRT (as a technical body) said, technically, there’s no difference between scraping and indexing. Of course, as is clear from the above Wikipedia definition, CRT is right — there really is no distinction from the perspective of the computer activity between scraping and indexing. Both processes read the web site and do stuff with the data.
However, focusing on the technical process here is wrong. Instead, the important distinction is between the results of the activity. Here is perhaps a compelling explanation of how these two are different. When you go to visit a web site, your web browser reads the web site and displays the information back to you. In fact, most web browsers store a copy of that site on your computer so that it can display it back to you faster if you look at it again later. From a technical perspective, your visit to the web site and your browser caching the content locally on your computer is not very different from what a scraper does.
However, nobody is going to argue that web visitors are scrapers. Why? Because of their intent and what they are doing with the data. A consumer looking at content is a good thing. So, too, I would suggest is Google indexing the web and real estate content. Google is not (at least today) taking the content and presenting it as their own creation. Instead, they are linking back to the source of the data, which provides a critically important service to the web site being indexed. This is what the web is all about and so interpreting indexing and scraping as the same thing results in the leap backward the commenters on the Agent Genius post decry. It’s an undoing of the web for IDX sites, which have become critically important to agents and brokers today.
Before concluding this post, however, I also want to point out that not every one agrees that Google’s indexes are positive or even benign. In Belgium, a court has ruled that Google’s News service violates certain newspapers copyrights. In hailing the opinion, the winner of the case is quoted by the New York Times as saying:
”Today we celebrate a victory for content producers,” said Margaret Boribon, secretary-general of Copiepresse. ”We showed that Google cannot make profit for free from the credibility of our newspaper brands, hard work of our journalists and skill of our photographers.”
Could a similar argument be made by MLSs or listing agents about Google indexing listing data? Possibly. However, I think getting a similar ruling from a US court is unlikely. (Any lawyers out there who know the law on this, please comment to clarify, because I’m definitely no expert here.)
More importantly, our industry has accepted the web as its friend and Google is accepted as a critical part of the web. To many, in fact, Google is the web. What’s wrong with the MIBOR decision and CRT’s narrow, technical interpretation that led to MIBOR’s decision, is that it goes against the many decisions that have already been made that the web is the real estate industry’s friend. That decision cannot be unmade. It’s done. Rule interpretations like that provided by CRT, however, do result in NAR members not being able to compete. As many on Agent Genius have commented, Trulia, Zillow and Realtor.com are not hamstrung by this same interpretation of the IDX policy, which only hinders and restricts NAR’s members. That’s wrong.
Fortunately, we live in a web world and, for many, that means we know each other personally. Most of those commenting over at Agent Genius have met, know and greatly respect Chris McKeever (@crtweet on Twitter), who now heads up CRT. My hope is that Chris can join the conversation and clarify CRT’s interpretation or let us know why the current interpretation is best. I’m asking for this conversation with the greatest respect for Chris and everyone at CRT. MIBOR put them on the hot seat but perhaps there’s a possibility the conversation can result in greater understanding for everyone, and hopefully a quick clarification on this critically important matter for MLS organizations that haven’t yet interpreted the policy on this issue.
The RETS community has been busy for the last year or so trying to prepare MLSs for June 2009, when the NAR MLS Committee has required all MLSs to be RETS compliant. This is a great step forward for national RETS compliance, but I have a more incentive-based approach for promoting RETS adoption.
NRDS is the National REALTORS Database System or, more simply, the directory for all NAR members. All the NAR-affiliated Associations of REALTORS are required to keep this system up to date. For many smaller Associations (the ones NAR is trying to get to adopt RETS), keeping NRDS up to date means double-entering the data into both the MLS system and the NRDS system. So, I’m thinking NAR could save everyone a lot of time and money by making NRDS RETS-compliant. Associations would win, NAR would win, MLS vendors would win, and NAR would win. That’s a lot of winning, from what could/should be a relatively straight-forward project for NAR.
The controversy over NAR’s attempt to cover up their failure to trademark MLS with claims of ethical violations rages on over a case in Sarasota, Florida. Inman News and the Bloodhound Blog report on it, and the issue has elicited huge commentary at Real Estate Webmasters. I’ve long had the position that the NAR won’t be able to put this genie back in the bottle, and trying to do so will just anger their customers. I also suggested a few times that the NAR would be better off spending their money creating a new brand to promote IDX authorized sites:
Once data standards are established, promising a strong and comprehensive data product from brokers and agents to consumers, NAR should create and heavily promote a branding and logo program for legitimate IDX sites so consumers can have confidence they are seeing everything.
With the recent VOW litigation settlement, this idea is one step nearer fruition. Now NAR just needs to shift its focus away from annoying its members and instead use their funds to help their members with a strong brand that will enhance the authority of their individual web sites. Imagine what NAR could do with their TV and other ad dollars targeted at promoting a simple brand certifying web sites that have all the data available for consumers?
Alternatively, perhaps a company like Roost or someone else (e.g., FBS with its flexmls or other brand?) will step up to the plate with such a brand. NAR can lead or follow.
About a year ago, I wrote about my repeated failures at predicting agent declines over the last several years. This afternoon, I was reading a post on Read/Write Web called How Decoupled Is The Innovation Economy From The Rest Of The Economy and it got me to thinking about agent counts again. You might think those sorts of numbers are top of mind for me in running FBS, but the numbers tell why they aren’t. Since January, we’ve only seen a .32% decline in membership in existing accounts. Certainly, some of our 100+ MLSs have had some significant decline but, on average, it’s been completely flat. (We’ve also been fortunate to see lots of growth in new accounts.)
Some of this flatness could be from some MLSs using end of year billing but that isn’t likely to be the case for over 100 different MLSs, many of which are sure to have had billing cycles since January. Rather, my belief is there was a big enough decline in 2007 that left 2008 pretty stable. The big question is what 2009 holds in store. Your thoughts?
I’ve had my first pass over the terms of the settlement of the NAR/DOJ litigation over VOWs (virtual office web sites). Though I was surprised by the announcement of the settlement, I’m not surprised by the fact the litigation finally settled. In fact, during a couple of different meetings in D.C. a week or so ago, I mentioned to people that I felt the litigation was, for practical purposes, irrelevant. As such, I don’t see the settlement as huge news (it won’t likely be a game changer), though it’s definitely interesting news.
The action on the web has shifted over the last few years from VOWs (authenticated access) to syndication (advertising oriented, non-authenticated sites), with everyone racing to get listings nearly everywhere instead of trying to figure out how to control where the listings are going to be shown. If anything, I think the settlement puts an arrow back in the quiver for brokers, MLSs, their vendors, and others to develop hybrid systems bridging the gap between syndication sites and client service oriented sites. In this regard, I definitely see the settlement as a win for everyone, as it clears away the last vestiges of this issue.
Though I think the settlement is a “good thing” and don’t believe the NAR gave up much of importance given the current reality of listing distribution on the web, the DOJ did get pretty much everything they wanted in the settlement. The primary issue in the VOW litigation was whether a participant could keep some VOWs from displaying their listings while allowing others. This was known in the litigation as the “selective opt-out” provision because it allowed participants to select which web sites could display their listings in a VOW and which could not. On that front, the DOJ clearly prevailed as there is no selective opt out in the new policy. In fact, the only opt-out available now is by the seller:
An MLS shall, if requested by a Participant, provide basic ‘downloading’ of all MLS non-confidential listing data . . . . Confidential data includes only that which Participants are prohibited from providing to customers orally and by all other delivery mechanisms. . . . If an MLS provides a VOW-specific feed, that feed must include all of the non-confidential data included in the feed described . . . above excpet for listings or property addresses of sellers who have elected not to have their listings or addresses displayed on the Internet.
Settlement Agreement Sections III(2) and III(4) (emphasis added).
Earlier, Section II(5) of the Agreement specifies that, in order to have their listing excluded from the VOW feed, a seller must “affirmatively direct their listings brokers to withhold their listing or property address from display on the Internent” by signing a document that says “I understand and acknowledge that . . . consumers who conduct searches for listings on the Internet will not see infomration about the listing property in response to their search.”
In other words, this seller exception is no back door way of getting a selective opt out. First, it’s not selective, it’s all or nothing. Second, very few sellers will want their listings completely off the web. The end result is that a VOW will be able to display all listings from all participants, with no blanket opt-outs allowed except by the seller. This is a clear departure from the selective opt-out fought for by NAR and even the IDX blanket opt-out.
I think this last point is important. Way back when, the impetus to VOWs was to get around the limitation in some IDX policies allowing participants to opt out or even requiring them to opt in to include their listings in the IDX feeds from the MLS. There were and are several markets where participants with significant market share were not participating in IDX, making that program ineffective, and so those who wanted to display all the MLS listings on the web argued that a VOW should be able to display all the listings just like they could show all the listings if a consumer came into their physical office. That’s where everything fell apart, because there were participants who didn’t want to be told what they had to with their listings. With this new VOW settlement, however, no opt outs are possible and everyone is opted in, which means the DOJ won this point decisively.
The DOJ also appears to have won the other major issue in the litigation: referral fees. Section III(11) of the Agreement says, “An MLS may not prohibit, restrict, or impede a Participant from referring Registrants to any person or from obtaining a fee for such referral.” This was the big argument raised by RE/MAX, Realogy and others, namely that they didn’t want to provide their data to a web company only to have it sold back to them in the form of a referral fee. As Dave Liniger is often quoted as saying, this is “like the guy who shows up at a pot-luck dinner bringing only a fork.” RE/MAX even went so far as to say that they’d advocate withdrawing from MLSs if selective opt-out was not allowed, and that ultimately gave rise to the ill-fated ILD policy and the DOJ litigation.
I suspect the reason the litgation was able to be settled was because no one really cares about this issue any more. The listings are flowing everywhere already, referral fees and other models are developing regardless and the main issue now is effectiveness and cost and not who is providing what service. Perhaps the litigation bought some time for a variety of competitors but I think the more likely conclusion is that the market just found the solutions around the litigation, making it irrelevant enough that a settlement could be crafted.
One of the more interesting provisions from an enforcement perspective is what I’ll call the “no outsourced call center” provision, which appears to be a requirement NAR sought to balance the referral fee provision won by DOJ:
A Participant’s VOW must prominently display an e-mail address, telephone number, or specific identification of another mode of communication (e.g., live chat) by which a consumer can contact the Participant to ask questions, or get more information, about properties displayed on the VOW. The Participant, or a non-principal broker or sales licensee licensed with the Participant, must be willing and able to respond knowledgeably to inquiries from Registrants about properties within the market area served by that Participant and displayed on the VOW.
Enforcing this provision is going to be interesting. What does knowledgeable mean in this context? What is the “market area” served? The provision definitely has the potential to thwart abuses of the referral fee provision by trying to require that VOWs be operated by “real” agents but it seems to open a huge can of worms for MLSs and others trying to figure out who the real agent is or isn’t.
There are all sorts of other interesting provisions that are different from the earlier ILD/VOW policies, such as the indirect recognition of Zillow and Trulia in Section II(5)(c) by allowing the seller to require removal of automated value estimates (zestimates) and comments or other discussion about the listings. I’ll likely cover these and other provisions in a separate post, as I want to get this posted first to see what others are finding before delving into more of the minutiae.
The main conclusion I have from the settlement is that it’s about time and doesn’t really change the game in a big way, but it clears the field for innovation in the space between the MLS system and listing portals.
Coverage elsewhere:
Press releases from the NAR and DOJ announce the settlement. The actual terms of the settlement are the key and I’m reading them now and will be posting my thoughts soon.
Inman has yet another article today on NAR’s recent rules trying to control use of the term MLS by their members in promoting their web sites. Here’s a not too hypothetical question: What if FBS were to create an IDX site that uses our domain and registered trademark flexmls® in the URL where agents or brokers could have domains like brokername.idx.flexmls.com? Would that violate NAR’s rules? I’m pretty sure it wouldn’t or shouldn’t but would love to hear what others think.
Follow-up: Even less hypothetical: How about portal.flexmls.com/agentname?
Matt Cohen from Clareity Consulting has produced a white paper “to generate discussion on possible MLS system future features by providing a big picture view of the changing relationship of real estate professionals with each other and with consumers, the changing relationship of local and regional MLSs with each other, and to illustrate, at least at a high level, how these changes may be either enabled or reflected technically in the MLS system of the future.”
Of course, this is right up our alley here at the FBS Blog, so I’m psyched I finally feel like I have something of substance to write about again. I’m going to focus on a couple of the ideas floated by Matt, because I think they are related and pose some of the most interesting possibilities. (I’m definitely stretching the ideas Matt proposed to my own needs, so don’t blame him for my crazy ideas. )
Widgets-Broker Tools-RETS
Three of the ideas Matt has put forward are widgets, broker tools and expanding use of RETS. I’m going to put my own spin on these ideas and try to relate them together as my contribution to the discussion.
I love the idea of MLS widgets and Matt’s are great examples. What I most like about widgets is that they often rely upon APIs (application programming interfaces) that allow for other developers to modify the tools or even create their own. For example, at the core of many widgets is the use of some sort of syndication (RSS/Atom) feed. The data is made available through the syndication feed and the widget re-purposes or figures out a clever way to display the data.
This type of creativity relates directly to developing better broker tools. If brokers (or their developers) had access to easy to use MLS APIs, they could develop all sorts of cool things. RETS, of course, is an API to the MLS system but it’s not terribly easy to use and is what Robbie Paplin would say is pretty close to the metal. In other words, RETS provides access to the data and images from the MLS system but pretty much everything else is up to the developer.
What I think is the future are RESTful MLS systems with excellent APIs that allow for all kinds of new ideas and developments by brokers to allow them to differentiate themselves. Brokers could then develop widgets and all sorts of other cool stuff. I think this also is the right vision for the NAR’s Gateway/TREC/[new name coming soon]. From Mark Lesswing’s comment on my last post about the Gateway, NAR can’t develop another public-facing search engine outside of Realtor.com, but I don’t think that would prohibit [New Name] from developing an API that would allow brokers to do that. Then imagine an API that not only does listing searches but also exposes through simple requests all sorts of statistics, graphs, heat maps and what otherwise would be complex data-warehousing type queries. Bloggers with a bit of coding skills would be in heaven, creating all sorts of market analyses for their customers and the public.
The key to all of this, however, is developing the API with an excellent permission model. The MLS aggregation is built on cooperation among competitors and the type of creative freedom fostered by a more open API needs to work within that model of cooperation. However, I’m convinced it can be done and that such tools would re-empower brokers to compete at an even higher level. I’m also convinced that fostering such competition is the role of the MLS and that cooperation is a necessary part of that competition.
