DISQUS

louisgray.com: louisgray.com: Twitter's Search Engine Is Very, Very, Broken

  • Jordan · 6 months ago
    I've spoken to somebody at Twitter and this is a temporary sacrifice until they fix their scalability issues. It seems to go back 35 days. (for the Twitter Search part).
  • Louis Gray · 6 months ago
    Jordan, glad you could get hold of someone at Twitter. It seemed from the searches I tried that 35 days was even well beyond many of the accounts - with 10 days being a more standard ceiling. They must clearly be under duress. Of note, in our previous e-mail threads with Twitter on issues like this, they have been less than responsive. The fact we had to find these issues instead of them alerting the community to this type of problem is frustrating as well.
  • Sheamus · 6 months ago
    Just out of curiosity, I typed 'twitter oprah century' into Google and her opening tweet was the first result. Just about says it all.
  • Sheamus · 6 months ago
    I think one of the biggest issues with Twitter right now is they continue to operate in a cloud of secrecy and mystery, only becoming accountable after they’ve made some sweeping change. I hope Jordan is right and they are making improvements behind the scenes, but how difficult is it to let the userbase know before they start to tinker?

    Biz could write a few paragraphs explaining what they’re doing and why it’s important, and put it on Twitter’s blog. Then, @Twitter could tweet this out to its million followers, and re-tweets would mean probably the entire network would receive the news. It’s common sense. It’s common decency.

    Instead, they make huge adjustments to things like @replies, tell us *after* they’ve done it, and then sit back and soak up the howls of protest. They tell us they’re paying attention to feedback, but then announce another ludicrous change that nobody asked for.

    In a week we’ll probably be told you can now only search the tweets of celebrities. We’ll all complain, so they’ll change it to the suggested user list. But hey, they’re listening.

    Search is their single biggest deal. There’s nothing wrong with making it better, or moving other things around so it’s more efficient, but it’s simple common courtesy to announce things like this before you start pulling out the wires. Just a couple of months ago Twitter was riding the wave of praise, while Facebook was getting slated by all and sundry. While they haven’t quite swapped places, I think Twitter needs to be very, very careful the next few weeks. Things can go downhill very fast.
  • Louis Gray · 6 months ago
    I agree with your comments on Twitter's cloak of secrecy being a huge issue. Were they hoping nobody would notice their search engine had been deprecated? A quick note saying "Hey! We're more popular than ever and struggling to keep up, so we're limiting searches to the last two weeks" would be absolutely acceptable, and the grumbling would be lighter.
  • ballance · 6 months ago
    Twitter itself has been a case study in scalability issues since their beginning.
  • sofarsoShawn · 6 months ago
    Amongst other things.
  • Johnny Schroepfer · 6 months ago
    I couldn't even "Find People" today due to bugs.
  • Vinko · 6 months ago
    Louis I was noticing this exact same thing this week. I was trying to help structure a campaign for a company and noticed the previous (March 13 - March 30) campaign for this company no longer exists on Twitter.

    Strangely Twitter only show my tweets dating back to Jan. 2009 when I had been tweeting since 2006.

    Like you I went straight to Friendfeed for the information I was looking for, but only found some but not all the tweets. Could this be because not everyone's timeline are on Friendfeed?
  • Scobleizer · 6 months ago
    Yup Vinko, friendfeed only has about 1/3rd of the people I'm following on Twitter. The number is going way up every day, but lots of people still haven't signed into friendfeed and even some of those who have don't bring their tweets in there (which is VERY stupid, your readers can turn off Tweets, let THEM decide whether your tweet stream is too noisy or not).
  • Vinko · 6 months ago
    How can we as owners of these content fix this?

    Like Jesse Stay says we need to be able to use these tools (Twitter, Friendfeed, etc.) the way we want, to a certain extent, less of what the owners of these sites believe are correct.
  • Louis Gray · 6 months ago
    I have often said to let us determine the best way to use your products, and not tell us how to do it. Those rules are true whether we're talking about Facebook, Twitter or Microsoft.
  • Jesse Stay · 6 months ago
    I can't remember, but I think it was Louis that said that first. :-) I've said it multiple times as well though.

    However, let them create these sites how they want, that's fine with me. The minute they stop listening to their users though (especially those that have made them) is the minute their users start to leave.
  • Vinko · 6 months ago
    Let the principle of Default behaviour prevail http://tr.im/m6p6
  • Louis Gray · 6 months ago
    Not everybody's timelines are on FriendFeed because not all Twitter users are on FriendFeed. (Yet)

    And while the purpose of this post was not so much to highlight the strength of FriendFeed's search over that of Twitter, it did end up having some of that result. At least the data is kept safe somewhere.
  • Daniel J. Pritchett · 6 months ago
    I'm glad you did show the superior FF search as a counterpoint to Twitter's latest folly. That kept me nodding along throughout the article. FF search is extremely powerful and I still find myself using it several times a day - mostly as a supplement to my hazy memory of interesting things people have said in the past.
  • Byron McCollum · 6 months ago
    When the earthquake was happing a few days ago, search.twitter.com was all messed up. The most recent tweet shown was days old, but the ajax banner would still display saying there were 3000 some odd new results since my last search. Clicking the link to refresh would result in nothing.
  • Louis Gray · 6 months ago
    That's certainly odd behavior. I would anticipate that under times of heavy load and duress, there would be greater tolerance for error, but these issues seem to be across the board.
  • Michael Gracie · 6 months ago
    Very difficult to find tweets regarding certain keywords or subject matter, even from your own profile. Further, Google search produced mobile caches for me, with no linkage back to the original tweet.
  • Steve Chou · 6 months ago
    The Twitter search function is very unreliable,if Twitter wants to grow up,it must fix the problem of his own first.
  • Louis Gray · 6 months ago
    Twitter does a few things very well and we applaud that. It seems that everything on the perimeter is touch and go, and they do an extremely poor job of keeping us updated. If they would just come clean on this, we would be a lot more sympathetic.
  • guruvan · 6 months ago
    I have completely given up on twitter's search. I find that it's much more effective to search at FriendFeed and/or Google. search.twitter.com can't find tweets that I sent out an hour ago.

    I have noticed this for months, but always figured "well, nothing else there is reliable, why should search be?" But that seems out of line with what they think they're going to be able to sell, and it's unfortunate that it's not, because it could be a valuable source of data. The only real hope for Twitter's data (IMO) is to have more companies attach "firehoses" like the one that FriendFeed has.

    Twitter is still a valuable source of data - Lots of people post tons of useful tidbits there. Surely the links to these blog articles go there, mine do. And FriendFeed is a huge resource for finding that data. But we will find flaws in their systems too. Just not as huge. And even if FriendFeed would be flawless, they can't index everything the right way for everyone to find everything they want.

    Hopefully Twitter doesn't see their way to being "greedy" with their data, and in the process lose it, and its value. The way to prevent this loss is to hook up more hoses that receive the entire Twitter stream. That, in my opinion, is how Twitter should be monetizing. Not building tools to use against the data that they can't keep straight. I think that there will be no lack of companies to buy hoses either, people want tweets.

    Data is only of any use if it's shared.
  • JesseNewhart · 6 months ago
    With all the hype surrounding the value of Twitter search they should definitely invest some of that venture capital into it. I've hit these same walls myself. The original reason I joined friendfeed was as a place to store my tweets for safe keeping (of course I've since seen the unique value of friendfeed). Come on Twitter! Get on it!
  • Louis Gray · 6 months ago
    In Twitter's defense, I often hear comments on the amount of money they've received from VCs being bandied about. Many other companies have raised much more, and been expected to do less. I hope they solve their scaling issues soon, or have the integrity to eventually cry uncle and ask for help.
  • PeteWright · 6 months ago
    I suppose that's a defensible argument. :)

    The problem, however, doesn't appear to be explicitly about money. Rather, it's the scale of the engineering challenges they're up against. I find myself wishing I could be a fly on the wall at Twitter -- this is a team that simply must be pulling their hair out to rectify the fruits of early engineering shortcuts regarding integration (search from Summize) and scaling. I've been commenting (apparently) negatively about Twitter lately -- I absolutely adore the service were it not for two things. First, it's unreliable. Second, I want more control to reduce noise. The perfect fix for both of these issues for me has been FriendFeed of late, but I'm still on Twitter and support what their doing. Really looking forward to substance coming out of Twitter HQ in the coming weeks.
  • JamesFuller · 6 months ago
    Currently, from a daily hash tag I was using up until roughly a week an a half ago they only have 18 days of backdating for searches now. Down from 4-5 months back in January. It's truly a sad repercussion that has occurred from the explosion in growth over the last few months.
  • holdenpage · 6 months ago
    I am usually an advocate for Twitter, but that is just unacceptable *sighs*
  • Louis Gray · 6 months ago
    Holden, absolutely agree with you. I am an advocate for what Twitter could be, absolutely. But we keep finding limitations in a seemingly flexible service. I kept wondering if it was just me, but I found issues at practically every turn.
  • venkat · 6 months ago
    I never used twitter search that much often
  • kevinwmurray · 6 months ago
    I think twitter jumped the shark with @oprah. So noisy, less quality conversations (and they were hard enough to track before) I find myself hiding in my little country club FFCC. (Friend Feed Country Club).
  • reechard · 6 months ago
    Hmm, I'm getting 15 pages of my own tweets going back 17 days. Perhaps I should import my tweets to a "room" so I can get ff search w/o twit-terrifying my ff :)
  • JamesFuller · 6 months ago
    I did a little inquiring into the distance of search and it appears to be just over 18 days. http://ff.im/38nNQ
  • Tony Hollingsworth · 6 months ago
    My sentiments exactly - some respite can be found with hastags though. You can head on over to http://hashtags.org and find some more history there. It's not perfect either but it goes beyond Twitter search to uncover older entries. Of course provided the tweet has a hashtag.

    Looking forward to improved search asap.

    Cheers
    Tony
  • Darren · 6 months ago
    The thing I hate about twitter search is the fact it does not go back far enough.

    Also from a developer point of view its a pain that the ids used in the search results do not match the id of tweets. It on the fix list but its been there for ages.
  • David Berrebi · 6 months ago
    I agree with you Louis. Also, Twitter Search is so slow comparing to Friendfeed....
    But I guess Twitter team is working on it and should release a new version pretty soon ;)
  • Hank · 6 months ago
    The biggest failure isn't that they don't index all stuff (Google doesn't either).

    Twitter's biggest failure is that they can't remove indexed tweets on request.
  • Louis Gray · 6 months ago
    The "removing tweets on request" issue is a tough one. The tendency would be to blame the user for some ill-sent tweets. Just like Google won't remove a page you wish would go away, I think the current usage pattern is fine.
  • piers · 6 months ago
    Thank you! I couldn't agree more, and found this argument annoying - the way-back machine is watching you no matter what you do anyways.
  • aerobroken · 6 months ago
    Nice work, I like twitter and it's become quite valuable and the search does need to be fixed, but right now they are so busy trying to keep up with the scalability and huge growth. Honestly, I would have never started using friendfeed which I really enjoy and like if it wasn't for Scobleizer and all his preaching about it many months back on twitter. I respect his opinion and I use friendfeed daily now.
  • ZuDfunck · 6 months ago
    I don't get the problem Louis
    Why Is it important to find Tweets
    Particularly when you have that vast
    warehouse of info at Friendfeed.

    Take it one layer deeper
    Why do I need to search how many times
    I tweeted Big Lebowski quotes?
    Or how many Tweets are about Lighthouses...

    Ok I need more coffee
    Waiter!
  • Rebecca · 6 months ago
    I think it really does depend on what your goals are for using twitter. If you are simply there to chit chat with friends and go on to links then who cares if it can accurately aggregate data. But if you are basing your justification for using it on its ability to do that then it is a real problem
  • Rebecca · 6 months ago
    Very interesting and informative. I would hope that the suitor isn't apple. I don't think they have search sorted out for themselves. Google or the like would likely be much more successful.. My 2cents
  • JesseNewhart · 6 months ago
    Perhaps they're due for another round of fund raising then. The VC's would be fools not to give it to them. One thing is for sure; it's exciting to watch these nascent real time technologies develop. We've only scratched the surface. Hive mind, here we come :)
  • Louis Gray · 6 months ago
    If Twitter were going back to the well for more money, I would hope they could prove that "this time" they could scale and meet growth requirements. If they cannot, they need to come clean on what we can expect from the service, and what we should not expect.
  • BarbaraKB · 6 months ago
    As you noted, Louis, Twitter's search has *never* been stable even when it was Summize.com. BUT, that doesn't diminish its usefulness as a search engine. And while I *love* FriendFeed and its community, Twitter will continue to charm and cajole bloggers, journalists and media producers because of its simple and instant streaming human feed.
  • Louis Gray · 6 months ago
    Barbara, the fact that it is not stable, and that I couldn't find basic things I was looking for absolutely diminishes its usefulness as a search engine. How can you say that falsely declaring no results or flat-out failures doesn't diminish it? What we know is that it is good for "right now" responses, but anything deeper, and you're dead.

    As for FriendFeed vs. Twitter, that's not this article. I've already written that one a few times. The good news is that FriendFeed has a viable, true, searchable index.
  • BarbaraKB · 6 months ago
    Good for "right now" responses is Twitter's beauty. And maybe that's all Twitter search will ever be, although I hold out hope for more, like you.

    Louis, you're the one who keeps bringing up FriendFeed whenever critiquing Twitter, not me. I see them as two complementary, but vastly different, social tools. FriendFeed competes w/ Facebook, not Twitter.

    Peace!
  • Danny Burkes · 6 months ago
    The search API documentation at http://apiwiki.twitter.com/Twitter-Search-API-M... does state that search is limited to "a max of roughly 1500 results"- this translates to a variable number of days, depending on the popularity of the topic you are searching for. Your research, however, indicates that the limit is much more severe than that.

    One other thing worth noting is that the search database at Twitter is completely separate from the tweet database- since the time that Twitter bought Summize, they have never consolidated the the databases. This results in things like, when you delete a tweet, it still shows up in the search results for some time.

    All in all, they've got a ways to go.
  • Jerome Paradis · 6 months ago
    Even if it worked well, there is some flexibility missing in their search.

    For example, on twitter.com, we cannot search profile information. Only user names, first name, last name. As a user, I was trying to find out someone I had followed but forget the name but could not do any search based on the information I remembered.

    I see Twitter search like a proof of concept. It has lot of potential but it can't be fully exploited yet.
  • kosso · 6 months ago
    True, but I think they've just been busy integrating the results in to the main interface.

    But one thing that really bugs me is that they still keep deleted tweets in the results. When I delete a tweet, for whatever reason, I want it deleted. (Rather like the stories around yesterday about photos not being deleted) - It just seems they don't re-index things much, if at all. They just sit across 'the firehose', suck it all in and that's it.
  • Ari Herzog · 6 months ago
    Have you seen the review of other search engines I wrote at http://mashable.com/2009/04/22/twitter-search-s... yet?
  • Daniel Tunkelang · 6 months ago
    I haven't had the kindest words for Twitter Search: http://thenoisychannel.com/?s=twitter+search

    But I think some of the comments here are overkill. I use Twitter search regularly as an alerting service for a broad vanity query. It works reasonably well, and I've learned to live with the glitches as part of what comes along with a free service. Could they do a lot better? For sure, and I've blogged about that. But what they have now is nonetheless useful.
  • piers · 6 months ago
    Well put. It's a free service, and you get what you pay for. It does some things well, some things poorly. And a catch 22 - Twitter could offer premium service, but could they support it?
  • Jose Sandoval · 6 months ago
    I think we are forgetting that implementing a good search engine takes quite lot of effort. Twitter will need more than an handful of PHP/Scala (or whatever they use) developers to come even close to a fraction of what google can do. A deal would make sense at this point. Of course, if Twitter has the money, they should grow it inside. But Twitter doesn't have that much time to be messing around.
  • Jesse Stay · 6 months ago
    Ari, great article - you should have added Google to that list as well. I usually just use Google to do my Twitter search, using "site:Twitter.com" - it has been much more effective than search.twitter.com
  • Josh Haley · 6 months ago
    That's pretty egregious stuff.
  • Paul · 6 months ago
    Remember that email was first available in 1974, and it took a couple of decades before it became the tool we all know and use, today. Twitter needs to be around for a bit longer before it is given parity with email, regardless of Moore's Law and its effect on new technology adoption.

    That being said, I definitely agree that Twitter needs an angel with experience in scaling. It probably (should have) realized this pretty early on, when the explosion in users was first detected and projections for the future were being developed. It's really just a matter of money ... and that's probably what's slowing things down. I mean, when will Twitter become profitable? For that matter, HOW will they become profitable? At least Google, Microsoft and Apple (by far the least experienced of the three) have deep pockets and well-monetized public channels.
  • Mark Bean · 6 months ago
    @Paul - you can't give Twitter any slack when comparing it to email because the retards that built Twitter should have learned how to architect a system to handle any amount of traffic. It's not that hard. As Scoble and many others point out - FriendFeed can handle much more complex data problems. #TwitterSucks
  • Mark Evans · 6 months ago
    Louis,

    What I find interesting is that despite search potentially being a key asset for Twitter, the better and more useful search applications are being created by third-parties such as Twazzup and Twingly.
  • Jim Connolly · 6 months ago
    Interesting post Louis.

    The element that's bugging me right now, is how useless Twitter's 'trending topics' feature has become; thanks to spam.
  • Jay Bryant · 6 months ago
    No kidding Twitter search is broken. None of my tweets over the last nine days (http://www.twitter.com/jaysbryant) show up in the search. For some reason my account has been excluded from the twitter search. No explanation yet from Twitter.
  • Pascal · 6 months ago
    Do you know that we are closed to the 2 billionth tweet ?

    See http://www.tweespeed.com which gives the "instant speed of twitter" and try to estimate the reaching date. Amazing.
  • Marco · 6 months ago
    There is an offiziell Twitter issue.. now.

    http://twitter.zendesk.com/forums/31935/entries...