meyerweb.com

Skip to: site navigation/presentation
Skip to: Thoughts From Eric

Announcing Followerlap

Last week, I got an interesting inquiry from Velda Christensen:

@meyerweb *wondering just how many of your followers follow @zeldman and vice-versa*

I had no idea.  Furthermore, I didn’t know of a tool that could tell me.  So I built one: Followerlap.

As it turned out, the Twitter API made answering the specific question pretty ridiculously easy, thanks to followers/ids.  All it takes is two API requests, one for each username.  The same would be true of friends/ids, on top of which I suspect I’ll fairly shortly build a tool quite similar to Followerlap.

Since I announced Followerlap’s existence on (where else?) Twitter, I’ve had a few repeated (and not unexpected) bits of feedback.

  • Why not list the common followers?  Because followers/ids returns a list of numeric IDs.  Resolving those IDs as usernames would require one API hit per ID.  If there are 15 followers in common, that’s not such a big deal, but if there are 1,500, well, I’ll run out of my hourly allotment of API requests very quickly.  Maybe there’s a better way; if so, I’d love to hear about it, because that would be a great addition.

  • Why can’t I find out how many people follow both Stephen Fry and Shaquille O’Neal?  Past a certain number of followers, somewhere in the 200,000–250,000 range, the API just dies.  You can’t even count on getting a consistent error message back.  There are ways around this, but I didn’t want to stress the API that way, so it just fails.  Sorry.

  • How can I link to a specific comparison?  At the moment, you can’t.  I hope to make that happen soon, but I decided that a tool this simple should have a similarly simple launch.  Ship early, ship often, right?  Anyway, it’s on the list of things to add soon.  Use the new “Livelink to this result” link underneath a result.  (See update below for more.)

So that’s Followerlap.  Any other questions?  I’ll do my best to answer them in the comments, though for a number of reasons I may be slow to respond.

Update 6 Jul 09: as noted in the edited point above, livelinking of comparison results is now, um, live.  So now you can pass around results like the number of people who follow both God and the Devil (thanks to Paul M. Watson for coming up with that one!).  I call this “livelinking” because hitting a result URL will get you the very latest results for that particular comparison.  “Permalinking” to me implied that it would link to a specific result at a specific time, which the tool doesn’t do and very likely never, ever will.

29 Responses»

    • #1
    • Comment
    • Mon 6 Jul 2009
    • 1335
    carlos wrote in to say...

    Hm, you could build a cache of requested IDs, and a histogram of how frequently they come up, then use you spare allotted API requests to build up your local database of ID – username equivalents, starting with the more popular ones, then they can come from your cache while you keep building your dossier. Yep, eventually you could end up duplicating the whole catalog of Twitter users, which might be against their TOS though. Just an idea that may have legs.

    • #2
    • Comment
    • Mon 6 Jul 2009
    • 1404
    Jason Beaird wrote in to say...

    That’s really a bummer about not being able to list the common followers. Finding new people to follow would be the primary benefit of such a tool, IMO. I guess the only way to do this would be to use statuses/followers instead of followers/id. You only get 100 results per call though, so if both people are following 1,500 people, that’s 30 calls for what should take 2. Ugh.

    Also <cough>Followinglap</cough>?

    • #3
    • Comment
    • Mon 6 Jul 2009
    • 1414
    Joan Vermette wrote in to say...

    Thanks for this; I’ve been wanting a utility that would do this for me for a long while, now.

    Re: the comment above, I would love to see “Followinglap” too. If someone isn’t well known, but says smart stuff and has good taste in who they follow, I’ll give them a follow as well.

    • #4
    • Comment
    • Mon 6 Jul 2009
    • 1417
    Eric Meyer wrote in to say...

    Carlos: I really don’t want to recreate Twitter’s user catalog– that’s millions upon millions of people. I’ll give it some thought to see if there’s a lower-impact way to make things happen.

    Jason: Yeah, if the method returned usernames instead of numeric IDs, we’d be golden. I might not do the icons but I could list out, with links, all of the accounts in common. (And the ones that aren’t in common, for that matter.) I did mention the idea of a “Followinglap” in the post (the bit about friends/ids) and hope to create and launch that soon. Shouldn’t be too much effort to adapt the code.

    • #5
    • Comment
    • Mon 6 Jul 2009
    • 1419
    Erik Vorhes wrote in to say...

    I love the name.

    @Jason, it’s not “Followinglap,” but “Followerlap.” And then think about what happens if you pronounce the w as a v (as in German). Maybe an overlapping logo would help? (If you want one, Eric, I’d be happy to make one, gratis.)

    • #6
    • Comment
    • Mon 6 Jul 2009
    • 1433
    Damon wrote in to say...

    Hey Eric,

    I built something similar to this a few week’s back that does list the followers in common, as well as their friends. It’s not very pretty, but it gets the job done.

    I’m kind of cheating by using App Engine and a whitelisted account to distribute the load/API calls for me, but it does work for most situations. The site was originally built to allow programmatic access to Twitter Avatars in cases like this.

    You can see @meyerweb and @zeldman’s commonality here – be forewarned though that my browser wasn’t really fond of loading 7,000+ images, heh.

    • #7
    • Comment
    • Mon 6 Jul 2009
    • 1447
    Jonathan Knoll wrote in to say...

    As some of your replies mentioned, I built something similar last year at: twitter.infinityplusone.com. The tool in mind would be TwInCommon. It’s done completely on the front-end, which makes it a bit of a hog — have a back-end version I never finished implementing because I got too busy. Consequently, I put a threshold of 1,500 or so in just-in-case (back then, the Twitter API tended to fail over that anyway).

    TwInCommon also distinguishes between follower and following. This is a pretty important distinction, I’d think, because many people are not simply comparing themselves to others; rather, they are looking for new people to follow, or wish to see if a new follower is an unknown member of their community, or just a troll/spam. (It’s a bit buggy now due to API changes, so I should prolly give it a tune-up one of these days…)

    Also have a few other Twitter API hack-tools in there I built for friends. Nothing special, but fun to do at the time.

    ~ @yoni

    • #8
    • Comment
    • Mon 6 Jul 2009
    • 1455
    Vinay wrote in to say...

    One way to list common followers would be by using the statuses/followers, statuses/friends method (a quick dirty script I wrote few months ago http://wg.vinayraikar.com/apps/commontweeps/). Best thing about followers/ids is that it is light weight but to list followers, you will need to cache user details and that will lead to data consistency issues (Something like http://twivatar.org for usernames would be wonderful :D ).

    • #9
    • Comment
    • Mon 6 Jul 2009
    • 1802
    Erik Vorhes wrote in to say...

    And now, I feel stupid for misunderstanding the request for “Followinglap.” That’d be neat to see, though a more descriptive name might be called for.

    • #10
    • Comment
    • Mon 6 Jul 2009
    • 2051
    Neal G wrote in to say...

    Here’s a cool site I just heard of. Looks like Zeldman and you are trending at the same amount – http://twittercounter.com/compare/zeldman/meyerweb/all/

    • #11
    • Comment
    • Mon 6 Jul 2009
    • 2132
    Eric Meyer wrote in to say...

    Erik, no worries. And if you’ve the time to whip up a spiffy logo as previously mentioned, I’d be only too glad to see it!

    • #12
    • Comment
    • Mon 6 Jul 2009
    • 2143
    Eric Meyer wrote in to say...

    I must say I’m not too surprised to see that other people had done similar things in the past. Part of me is sorry I didn’t know they existed, but a bigger part of me is glad—because had I known of them, I wouldn’t have had an excuse to do a little programming and learning!

    • #13
    • Comment
    • Tue 7 Jul 2009
    • 0031
    Damon wrote in to say...

    Hehe – as a guy that’s developed a bunch of Twitter apps, there’s not one that I’ve thought of that somebody else hasn’t done already. It’s really quite amazing.

    • #14
    • Comment
    • Tue 7 Jul 2009
    • 1116
    Dougal Campbell wrote in to say...

    I had/have similar issues with Twitual. You can only fetch follower details in batches of 100. Which is fine for most accounts. When when you start analyzing users with thousands of followers, it begins to get icky. You have to loop through, building a list, handling sporadic errors along the way, and it’s not very fast making all those HTTP requests.

    Getting an application registered helps with the API request limit. But that’s a minor problem compared to the speed and error handling issues. I’ve been considering caching ID/name mappings server-side, but like you say, attempting to basically mirror all of Twitter’s info is not a very fun prospect.

    • #15
    • Comment
    • Tue 7 Jul 2009
    • 1143
    Eric Meyer wrote in to say...

    Dougal: For what I’m doing, I haven’t had to register because the API limit isn’t really an issue: the default hourly limit for an IP address seems to be 100,000 hits, and since each comparison requires two hits that’s 50,000 comparisons an hour, or 13.889 comparisons per second. No problem. Even if I set up a tool to show (or integrate into Followerlap) friend-overlap statistics, that will double my API hits and halve my upper bound to 6.944 comparisons per second. I think I can handle that.

    The key there, of course, is to just present the numbers and not try to build a list of details. What I really wish is that the API had followers/usernames and friends/usernames methods. Then I could at least list and link to them all, even if I didn’t have avatars or bios or whatever. That’d be pretty cool.

    • #16
    • Comment
    • Tue 7 Jul 2009
    • 1201
    Brian Deterling wrote in to say...

    Tweepdiff (http://tweepdiff.com) does essentially the same thing. It shows three lists: one for in common and one for unique to each user being compared. It also lets you compare more than 2 users and you can mix and match followers/friends. As mentioned in previous comments, the API tends to fall over when you get above a certain number of friends/followers but I doubt people really want an accurate comparison of 200k followers anyway. I use memcache and a combination of the followers and statuses APIs which works ok with pagination. And yes, I too wrote the tool without checking to see if someone else had already done it, and was happy I got the experience.

    Brian

    • #17
    • Comment
    • Tue 7 Jul 2009
    • 1207
    Damon wrote in to say...

    Eric: Did Twitter bump it up for you or something? On my whitelisted boxes, the hourly rate limit is only 20,000 hits. Without whitelisting it’s 150.

    • #18
    • Comment
    • Tue 7 Jul 2009
    • 1319
    Eric Meyer wrote in to say...

    Damon: nope. account/rate_limit_Status returns an hourly-limit value of 100000 when I run the tool in “debug” mode (which, among other things, shows me a var_dump of account/rate_limit_Status). I’m not signing in as me, either, so it must be IP-bound.

    • #19
    • Comment
    • Tue 7 Jul 2009
    • 1337
    Dougal Campbell wrote in to say...

    There have been many calls for API methods to return all the follower/friend names in one fell swoop. The problem is that the way that the Twitter datastore is structured, it’s a very expensive operation on the Twitter server-side. So it ain’t going to happen any time soon, according to what their API guys have said.

    • #20
    • Comment
    • Tue 7 Jul 2009
    • 1345
    Jonathan Knoll wrote in to say...

    Eric,

    I know that, by the rules, Damon is correct about the API limits: http://apiwiki.twitter.com/Rate-limiting. Have you outputted the rate-limit when *not* in debug mode? In fact, where did you find out about debug mode? And are you sure the TOS allow you to use it like that?

    ~ @yoni

    • #21
    • Comment
    • Tue 7 Jul 2009
    • 1358
    Eric Meyer wrote in to say...

    Apparently I was unclear, Jonathan. What I meant is that Followerlap has a “debug mode”, which is invoked with a URL parameter. It’s the same tool doing the same stuff, just outputting values for me to see (or triggering failure states based on values given to said parameter). Here: compare God and the Devil, with the details.

    • #22
    • Comment
    • Tue 7 Jul 2009
    • 1406
    saleiva wrote in to say...

    Hey Eric,

    I’m really like your idea on followerlap.
    I’ve some design ideas for it. If you would be interested in it, please let me know!

    Congrats and regards.

    • #23
    • Comment
    • Tue 7 Jul 2009
    • 1546
    Jonathan Knoll wrote in to say...

    Eric,

    That’s actually what I suspected you meant, except that I was trying to find an explanation for why the rate-limit is so high. This is very weird. I just went to: http://twitter.com/account/rate_limit_status.xml, and it gave me 20000 (I’m whitelisted). My wife tried at her office, and it gave her 150. Could your app, perhaps, be hosted on a domain that has “uber”-whitelisting? (Not something mentioned in the API dox. :p)

    Dunno. Lucky you.

    ~ @yoni

    • #24
    • Comment
    • Sat 25 Jul 2009
    • 1415
    Mark wrote in to say...

    What does the @god & @devil comparison tell us? and then again what does it tell us when we compare these?

    • #25
    • Comment
    • Sat 25 Jul 2009
    • 1614
    Aklap wrote in to say...

    Periods are legal characters in Twitter names, but your site chokes on them.

    • #26
    • Comment
    • Sat 25 Jul 2009
    • 2132
    Eric Meyer wrote in to say...

    According to the Twitter signup form, periods aren’t allowed, Aklap; only letters, numbers, and underscores. Can you point me to a username containing a period?

    • #27
    • Comment
    • Fri 31 Jul 2009
    • 1528
    Bruno Bergher wrote in to say...

    Eric, perhaps you could leave the option of cross referencing the two lists via JavaScript.

    I mean: you run the core of it on the server, then print a hidden list of the overlapping user IDs. Then, if the browser supports JS, add a button which runs the queries based on the IDs on the client side, so they don’t use up your API limit.

    Maybe there’s something to it I’m not seeing, but would probably be a pretty simple solution to actually SEEING who’s in the overlap.

    • #28
    • Comment
    • Wed 26 Aug 2009
    • 1646
    web page designer wrote in to say...

    I had to do this in the past too for a twitter app I created, except I was doing the opposite. If I remember correctly, I think the twitter API returns an array of IDs for all the users… then you can just put those into your own two arrays, and do some array math for whatever you’re trying to accomplish. Should only end up being a couple lines of code after the API requests are made.

    Cool little app, I love simple but useful sites like that :)

    • #29
    • Comment
    • Mon 5 Oct 2009
    • 1137
    Terence Eden wrote in to say...

    I did something similar earlier this year – Worth Retweeting. Enter your name and the name of the person you’re retweeting and you’ll see how many of your followers you’ll annoy.

    I wrote it up in May this year with code examples if anyone wants them – http://shkspr.mobi/blog/index.php/2009/05/worth-retweeting/

Leave a Comment

Line and paragraph breaks automatic, e-mail address required but never displayed, HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>



Remember to encode character entities if you're posting markup examples! Management reserves the right to edit or remove any comment—especially those that are abusive, irrelevant to the topic at hand, or made by anonymous posters—although honestly, most edits are a matter of fixing mangled markup. Thus the note about encoding your entities. If you're satisfied with what you've written, then go ahead...


July 2009
SMTWTFS
May September
 1234
567891011
12131415161718
19202122232425
262728293031  

Sidestep

Feeds

Extras