Archive for the 'devblog' Category

We’ve been using CentOS here at Meebo since we started. Sandy chose it because it was “basically Red Hat Linux… but free.” Over the years, we’ve upgraded from CentOS 4 to CentOS 5, and through each minor version up to 5.5.

It has served us well, but the distro is starting to show its age. On a daily basis we try to do something in Python that doesn’t exist in 2.4. libpurple will stop supporting our version of glib in the next month or three. And having to build our own packages for cgicc, django, gnutls, gsasl, a plethora of Python libraries, etc. is time consuming.

If we’re going to upgrade, we might as well look at all our options, right?

The Contenders

• Fedora – Released every 6 months. Only supported for 13 months. This is only a valid option for us if we’re willing to do “rolling updates.” By rolling updates, I mean upgrading a server to the next major release without reinstalling from scratch.
• Red Hat Enterprise Linux (“RHEL”) – Released every 2 to 3.5 years. Supported for 7 years, which is crazy long. But it seems expensive ($350 per server at the cheapest? Yikes!). To be fair, we haven’t asked them about site licensing.
• CentOS 6 – Based on RHEL, so has similar release and support life cycle. Over the years it seems as if the vibrancy of the CentOS project has faded. We’re concerned that the project isn’t as active as it once was. For example, RHEL6 has been out for 5 months already and one of the core CentOS maintainers thinks it will be 2-3 more weeks before CentOS 6 is out. There is also some concern on one of their mailing lists that the project could be more transparent to the public—which isn’t horrible, but also not an encouraging sign.
• Scientific Linux 6 – Based on RHEL, so has similar release and support life cycle. They seem very open/transparent. Their website lists all the changes they’ve made to Red Hat’s packages and the reason for each change, which is really cool.
• Debian Stable – Released every 2 years. Usually supported for 1 year after a new major version is released. Packages only get updated if there are major security or usability fixes. This support life cycle is scary… we would be obliged to upgrade all our servers within a year’s time. Only a valid option for us if we’re willing to do rolling updates.
• Debian Testing – Packages that will become the next stable release of Ubuntu. Packages have only been lightly tested… probably not a good idea when stability is a requirement.
• Ubuntu Server – Released every 6 months. Supported for 1.5 years. Only a valid option for us if we’re willing to do rolling updates.
• Ubuntu Server LTS – Released every 2 years. Supported for 5 years (some packages, anyway—other packages are only supported for the normal 1.5 years).

And just to be clear, we’re not upset with any of the above distros for having a slow release cycle, or for not supporting old versions long enough. We very much appreciate the work of open sources developers around the world. But we must chose the distro that we feel will work best for us at Meebo.

The Decision
Historically, we haven’t done rolling updates at Meebo. If you’re ok with rolling updates, or if you’re ok with re-installing all your servers from scratch once a year (we don’t even restart our servers that often!), then Fedora, Debian, and Ubuntu Server (non-LTS) are valid options. If you’re not willing to commit to this amount of work then the support lifecycle is too short for these distros to be considered.

That leaves two options:

• Scientific Linux
• Ubuntu Server LTS

They’re both great!

One inconvenient trait of Scientific Linux is that major releases are infrequent. The next one might not be until 2014. Some packages might be quite stale by then. For example, we could still be using Python 2.6 three years from now :-(

Some of us prefer dpkg, apt, and Debian/Ubuntu’s packages and package repositories over rpm, yum, and Fedora/Red Hat/EPEL’s packages and package repositories. We feel like we’ve had fewer annoying dependency problems with Debian/Ubuntu.

Open questions:

• When packaging our own software, is it easier for us to use rpm or dpkg?
• What kind of updates does Ubuntu provide to Ubuntu Server LTS? Does it only provide security and critical bug fixes? Or does it provide updates for new features, too?

We’re going to test both Scientific Linux and Ubuntu Server 10.04 LTS and see how it goes. We’re curious to hear from other companies about this issue. What distro do you use? Have you thought about switching? Do you find yourself building a lot of your own packages?

— Mark

[ comments ] [ past blogs ]
Follow us on Twitter

As the Internet evolves, so do the technologies that drive it, and here at Meebo we constantly aim to innovate by evaluating new technologies and products that will enable us to build better products for our awesome users.

A little over 10 years ago, IPv6 was developed as the Next Generation IP protocol to replace IPv4, which is what is used across the world (Internet) today. IPv6 adoption has been slow until very recently. IT and Engineering managers couldn’t justify the business case to make the switch, others just didn’t believe we would run out of IPv4 addresses so quickly, and as with any new technologies, others were just hesitant to introduce change if nothing was breaking.

A few weeks ago, the Internet Society announced World IPv6 Day on June 8th, 2011. On this day, major web and technology companies would enable IPv6 on their main web sites for a 24 hour period as a trial run. This will help us work together, and prepare for the use of the new protocol on a much larger scale. Users will not have to do anything to participate, just continue to use the site as you normally would.

Meebo is excited to be joining Google, Facebook, Limelight Networks and other leading companies for World IPv6 Day.

For more information, comments, or to report problems on World IPv6 Day, please email us at: ipv6 at meebo.com.

- CK

I’m happy to announce that we’ve open sourced the Meebo Bar embed code! It is the result of more than two years of iteration and careful consideration for the best way to embed a third party webapp on your website. I’ll get into the technical details below, but first let’s take a quick trip down memory lane and see how the embed code got to where it is today :)

The first commit to the Meebo Bar codebase was made by Paul S. on the 4th of June, 2008. Historic weather reports say that it was a beautiful Wednesday in Mountain View CA, with the sun shining and temperatures in the low 70s. Eleven days later, on June 25th, Martin H. made his first commit to the Meebo Bar – he would go on to be the key engineer for the first 7 versions of the bar. As the Meebo Bar gained traction, more and more of our developers started working on the bar codebase.

Today, roughly two and a half years later, the Meebo Bar sits on thousands of websites and reaches roughly a third of the US population every month. There are many reasons for the bar’s success (the greatest reason is the Meebo team, which you can be a part of too!), but today I want to highlight the embed code.

We knew when we set out to create the Meebo Bar that we could not let it block the hosting page from loading or rendering. Even if the Meebo servers were to respond slowly, we had to ensure that the rest of the page would not be adversely affected! In addition, we did not want our Javascript to leave a footprint on the hosting page (called “namespace pollution” in the javascript community), and we wanted to avoid having our code break if the Javascript environment of the hosting page changed (e.g. if the webpage made changes to the Javascript prototype objects).

The solution that addressed all of these issues was to create an iframe and then load all of the Meebo code into it. It sounds simple, but there are evil edge cases and arcane tricks that you need to discover in order to make a truly non-blocking third party webapp embed code. We are happy to share our discoveries, and hope you will find it useful!

The first couple of iterations of the bar embed code required the hosting site to host an html file on their domain. Our embed code created an iframe that loaded that html file, and once the html file had loaded completely it in turn would load all of the bar’s Javascript and CSS. Since the html file was hosted on the site’s domain, the state of Meebo’s servers would not affect the iframe’s loading time. In addition, the Javascript that was loaded into the iframe would have access to the hosting page to draw the Meebo Bar on it and to integrate with the contents of the page, since the iframe was not “cross-domain”.

The major shortcoming of this first approach was that the hosting site had to put up a custom html file on their servers. Not only was it a technical and error-prone hassle that would slow down launching the bar on new sites, but some sites did not have the capability to put up a custom html file. Obviously, something had to be done!

The solution was to still create an iframe, but rather than set its src attribute and load an external html file, Martin came up with a great hack that allowed for us to inject Javascript into the iframe right from the embed code. The resulting context of the iframe is the same domain as the hosting page, so the javascript that executes inside of the iframe still has access to the DOM of the hosting page (i.e. it’s not “cross-domain”).

With a working solution in hand the solution can often seem obvious. However, I have a genuine appreciation for the two and a half years of discovery that lead us to the current version of the embed code. I’ll let you read the commented source of the embed code by yourself below, and simply leave off with a list of features of the embed code that I find particularly awesome:

Line 46: Create the Meebo function, which is the page’s asynchronous API for Meebo functionality. It can be used right away, even before the Meebo Bar code has finished loading. When the Meebo bar code has finished loading, it grabs all the argument objects already pushed onto the Meebo._ array and executes those API calls – after that the code reassigns Meebo._.push to a function that immediately executes the API call. It works brilliantly!

Line 69: We want to know how the embed code behaves in the real world, so we track the time it takes to execute the embed code and then load the Meebo bar code. This way we can ensure that the bar doesn’t slow down as we add new features and keep innovating.

Line 83: Every byte counts! Since the embed code will be included inline on every webpage of your site, we do everything we can to make it tiny. By creating named variables for all the common DOM operations, the closure compiler can compress them all into single-letter variables.

Line 148: In IE, if the hosting page has set document.domain then the created iframe without the src attribute set is considered “cross-domain” and reaching into its contentWindow will throw an exception. Luckily, we can try/catch this case and in the case of an exception set the iframe src to a javascript: URL. In IE, this will have the javascript in the URL execute in the context of the iframe – that javascript then sets the document.domain of the iframe contentWindow, after which we the iframe is no longer “cross-domain”, and can access the iframe’s contentWindow again.

Line 159: This function creates the HTML that will be injected into the iframe. The generated HTML does two things: 1) creates an onload function for the contentWindow of the iframe, and 2) inside of the onload function, creates a script tag that points to a meebo server and appends that script tag to the head of the contentWindow of the iframe.

Line 188: A webpage’s onload event will fire only after the onload of all its children iframes fire. When we’ve injected the HTML that will load the Meebo Bar JS into the iframe, we call document.close on the iframe contentWindow – this is what causes the iframe’s onload event to fire. Now, with the iframe’s onload event fired, we’re free to load the Meebo Bar JS into the iframe without blocking the onload event or holding up the loading of assets of the parent page.

Cheers!
Marcus

ps. This is the raw source of the Meebo embed code. It’s not straight javascript, as we have a pre-processor that we run over the raw embed code to strip out comments and macros in order to produce the compressed “production” versions of the code. This raw version with its detailed comments is perfect for educational purposes!

It’s been a while since we’ve had a developer post! Since joining Meebo, I’ve been getting familiar with the so-called “NoSQL” trend in data storage, where people are moving away from relational databases toward more exotic and scalable data storage systems. Meebo started out using only MySQL and as our storage needs have changed, so have the technologies we’ve used. I’d like to share some stuff I’ve learned, and I’m interested in hearing about your experiences too.

Many cool new open source non-relational databases have sprung up recently. The quality of the conversation comparing these projects can sometimes be disappointing because there isn’t much recognition of the more subtle and complicated trade-offs that need to be made when choosing among these databases. I hope to help this conversation just a little bit by describing one of the trade-offs you’ll need to make if you’re choosing a distributed data storage system.

Specifically, I’m going to talk about why a system based on sharding with master/slave failover might not be the right solution for some of your problems. By “master/slave failover” I mean any scheme where all writes for a database (or shard) are sent to a single node. I’m looking in the general direction of mongo, membase, redis and similar systems.

This post is intended for all developers, so if you’re not familiar with databases, no worries. I’m not going to assume you know the CAP theorem, but you really should glance at it. Smarter people than me have thought hard and written a lot about these topics. In CAP terms, this entire post can be summarized as “the master has total consistency, therefore we have to choose one of availability or partition tolerance.”

What’s good about master/slave failover

It’s simple to understand. Since the master makes the decisions, there are no complicated protocols for maintaining agreement among multiple hosts.

It’s simple to implement. You can just bolt on an asynchronous replication protocol to get slaves. Persistent hash table + replication = NoSQL win. Let’s have an IPO.

It’s simple to get ACID transaction semantics on the master, and transactions are extremely useful. Operations that depend on a consistent view of the data (like counting) are easy. Statistics are easy to collect.

It’s fast. If you have low latency to the master, operations are fast and simple, as long as it’s not overloaded.

Some reads can go to slaves, allowing distribution across high-latency links in many cases where stale data is OK.

Since simplicity is a hugely desirable characteristic of distributed systems, there are many good things to be said about master/slave failover.

The Problem

As Prof. David Cheriton told my class one day, it doesn’t matter whether your distributed system works on a sunny day going downhill with a tailwind, but how things work on a rainy day going uphill with a headwind and wolves attacking. The point is that any good distributed system gracefully handles component failures that are reasonably likely to occur, and as things scale up, failures become a certainty. So let’s talk about fault tolerance.

If one of these things happens:

Node failure

Network failure

One of these two bad effects will result, depending on your system and the type of failure:

Loss of availability: clients cannot write

Loss of consistency: there will be divergent or stale copies of your database, and writes that occurred right before the failure may be lost.

Presumably there is some node (or group of nodes) that is monitoring the master and waiting to elect a new master if the old master fails. This would be fine if it were actually possible. The unfortunate truth is that it’s impossible to tell the difference between a node failure and a network failure. The slaves only know that they are not receiving packets from the master.

A decision must be made: should a new master be promoted?

If no, you lose, because the original master may have actually crashed, so there’s no master, and no one can write. Users get angry and write tweets about how much you suck.

If yes, you lose, because the original master may still actually be running and accepting writes and there’s just a network issue. Now you have two masters, and two divergent copies of the database. Chaos ensues. Also, writes that recently committed on the old failed master may not have replicated to the new master before the failure. [Footnote 1]

In brief: you lose unless you can do perfect failure detection, which you can’t. In even more brief: you lose.

More problems

If clients are not geographically near the master, writes and high-consistency reads will be slow.

Scaling a database or shard to multiple data centers is problematic. There are some interesting tricks that are possible here. For example, Yahoo’s PNUTS system migrates a user’s data between data centers to follow them around geographically. You can think of a user’s data as a special case of shard (since it can be a horizontal partition of the data).

Is there anything we can do?

If you’re willing to give up on total consistency and you’re willing to deal with a bit more complexity, you can get great performance, latency and fault tolerance. Amazon’s Dynamo system has some really interesting ideas, some of which have been implemented in the open source projects Cassandra and Riak.

Conclusion

Go forth and make smart trade-offs when choosing a database for your next cool scalable backend. Please feel free to leave comments or questions :)

- Dave

Footnotes:

1: You could design a database system that allows multiple masters to be cleanly merged into a single master. This is fine, but it makes ACID transactions impossible and takes away the nice consistency properties that made master/slave a good idea to start with.

On the back-end of Meebo’s infrastructure, we use a lot of different storage solutions, varying from MySQL to CouchDB. Each one has its perks and pitfalls, which is to say there is never one perfect solution. Since the early days of Meebo, we’ve used MySQL a fair bit (it was something we were all pretty familiar with and pre-no-sqldays). It’s served us well but one of my biggest gripes about it is replication and restoring data.

No one likes to wait. Whether it’s in line for coffee, stuck in traffic, or on-line waiting for a web page to load, waiting does not make for a great experience. In fact, performance is such an important ingredient to a user’s experience on a website that the Google User Experience team named “fast” as one of their 10 fundamental design principles for a great experience.

At Meebo we are constantly thinking about performance. We know that our Meebo Bar adds value to any publisher’s web site (sharing, notifications, custom applications, instant messaging, and an engaging advertising platform). But we also know that any third-party plugin will affect page performance.

In the upcoming version 10 of the Meebo Bar we took a step back and looked at our underlying architecture. First, we rebuilt the bar out of smaller components. This led to a simpler, more compartmentalized model. In addition to improving performance, our new architecture ensures additional privacy for our users. Next we went to work applying industry best practices to ensure the best performance possible. In fact along the way we even led the charge on some new emerging standards around third-party plugin performance.

That’s why Martin Hunt and I were recently invited by Steve Souders (Performance Lead at Google) to give a detailed talk on our approach to performance at the Velocity 2010 Web Performance and Operations Conference.

Plugin Performance

So what makes a plugin fast? The performance of a third-party plugin is directly related to how it handles these four concerns:

• Payload size. The amount of code that needs to be downloaded & processed.
• Network requests. The number of additional network requests required to load the plugin.
• Page blocking. Whether the plugin blocks the page from rendering.
• Plugin speed. How fast the plugin can render itself and be ready for activity.

We made version 10 of the Meebo Bar significantly faster by addressing each of the above concerns, implementing five specific performance improvements.

1. Just-in-time code delivery

One secret to making third-party plugins faster is to download the smallest amount of code needed at just the right time. A smaller initial payload means the bar can start up much faster. Deferred loading of functionality means the loading cost of each feature is paid incrementally instead of all at once. As a consequence, the initial JavaScript payload is now just 1/3 of its previous size. Smaller payload size and network requests made just when you need them leads to an overall faster bar.

2. True persistent caching

What if we could take the code that gets loaded for the Meebo Bar during an initial site visit and store all of that code locally? And on subsequent visits to sites with the bar remember the local copy instead of re-downloading it from the network? This is exactly the approach we have taken with our true caching mechanism. Simply put: near zero payload size and fewer network requests results in a faster bar.

3. Smarter image loading

The number of requests a plugin makes to load and render itself is an important factor in its overall performance. Some plugins have small download sizes but then require dozens of additional requests to fetch the necessary images to render themselves. Lots of network requests can really affect page rendering time.

In this version, all of our images (e.g., for buttons) are bundled within our CSS stylesheet as embedded data. Because image download piggybacks on an existing request, the total number of requests is kept low regardless of the number of images required. The end product is a faster overall page experience.

Martin wrote about embedded images back in March, and some of you asked what tools we use to accomplish this. Paul has been working to prepare an open source version of our CSS parser and pre-processor. Make sure to check back for more news about this later!

4. Non-blocking code

Normally when you add JavaScript code to a page the browser will block the rest of the page from rendering while it downloads and processes the recently downloaded JavaScript.

To solve this problem, Martin went to work and pioneered a technique that ensures the browser will never block while loading the Meebo Bar. This approach has garnered interest in the engineering community, leading us to share the mechanism with the rest of the industry. Not blocking the page from rendering is fundamental to performant third party plugins.

5. Faster, more flexible rendering

In previous versions of the bar we took the standard approach of using images to create gradients, drop shadows and rounded corners. Images are static, inflexible and relatively large in payload size.

In the new version of the bar we take a different approach. By rendering a large portion of our graphics using vector drawing technologies (like SVG and VML) we can draw the bar faster without additional HTTP requests and with greater flexibility to customize the look and feel of the bar.

The Result

Shortly we will be releasing version 10 of the Meebo Bar. It is a key step forward in terms of the quality it delivers to the user’s site experience. The good news is it delivers all of the previous Meebo Bar functionality with even greater security and with a significant boost in speed. While we can’t speed up the coffee line or change that traffic light, we can at least reduce the time you spend waiting when using a website on which we’re featured.

Marcus

[ comments ] [ past devblog posts ]
Follow us on Twitter

While building the Meebo Bar we have identified a number of problems that plague third party plugins on the web. We’ve experimented with various solutions that involve:

• cross-browser JavaScript loading mechanisms that do not block rendering or loading of the parent page
• replacing images with cross-browser vector graphics
• generating browser-specific CSS with embedded images
• caching JavaScript in client-side storage
• loading private user data in a secure fashion on third-party sites

All of these techniques come together in the next version of the Meebo Bar, making it the best performing and most technically advanced version of the bar yet.

This Thursday, Martin and I will be giving a talk at the Velocity conference in Santa Clara, outlining and describing in depth all of the techniques that went into optimizing the Meebo Bar. If you are interested in embedding your JavaScript on other people pages, you don’t want to miss this :)

We hope to see you at Velocity in Santa Clara on Thursday!

Marcus

[ comments ] [ past devblog posts ]
Follow us on Twitter

Hi Everybody!

We’ve open sourced the code for XAuth and it’s available now on github. If you’re unfamiliar with XAuth, check out Chris’s previous post to learn more about the problems we’re trying to solve as the web becomes more social.

For the curious developer (you!), XAuth is built in JavaScript and utilizes capabilities only available in modern (HTML5) browsers: localStorage (persistent client-side storage), postMessage (cross domain communication) and JSON (object serialization). If any of these technologies pique your interest, head on over to github and take a look at how we’re using them. There’s a high level “how does XAuth work” that you’ll want to read before diving into the code itself.

If you’d like to get involved, jump into the discussion at the XAuth Group. Chris and I will also be giving a quick overview of the technology at Mozilla Labs Night (Tonight! Apr 29th, 6pm @MozillaHQ) if you’re in the Mountain View area.

We’re excited to share this with the community and hope that this will kick off further debate and innovation around the challenges that are ahead for the social web.

-Jian

[ comments ] [ past devblog posts ]

The web certainly is a lot more social than it was a year ago! Today we are announcing a new open platform called XAuth that is going to help make the social web smarter and more relevant. The XAuth initiative has also been joined by major internet and social services like Google, Microsoft, MySpace, Yahoo!, Gigya, DISQUS, and Janrain.

The goal of XAuth is to create a more relevant social experience across the web, and it is truly exciting that all of our partners are united on this front.

By including the light-weight XAuth JavaScript library, service providers like Google, Microsoft, MySpace, Yahoo! will be able to extend authenticated user services across the web to developers like Meebo, Gigya, DISQUS, and Janrain.

A few weeks ago, I was invited to share at the OpenID Technology Summit and gave a brief presentation on some of the problems that Meebo faced distributing third party identities and services. I wanted to share these problems here today so that you can get a taste of some of the interesting challenges that XAuth will help solve.

Challenge #1: Relevance

Have you ever been confronted by too many choices while signing in or sharing? Sometimes these user interfaces include services you may never have even heard of.

Challenge: Are there too many choices? Where’s Yahoo!?

Meebo faces this problem as demand increases for more services to appear in our Meebo Bar. XAuth will help solve this problem by enabling developers to focus on the services that a user is currently signed in to. When partners like Google or MySpace begin using XAuth, developers can efficiently detect if a user has a session and create a more relevant user experience. Users are more likely to engage in simpler and more relevant experiences.

Challenge #2: Efficiency

As sites become more social, they are including JavaScript libraries or widgets for multiple services. Many of these services may attempt to make a connection back to their services to see if you are authenticated so that they can appear more relevant. Meanwhile the site is also trying to load the rest of its own page. The consequence is that the publisher’s site may take just a bit longer to load, resulting in a possibly relevant but slower user experience.

XAuth enables developers to check if a user is authenticated with a specific set of services with at most a single HTTP request.

XAuth can help reduce the number of requests needed when a site uses multiple social services

Challenge #3: User Controls

Most service providers take privacy very seriously and have their own privacy policy governing their services. As a user, sometimes I don’t feel like I have any control over what information these service providers are sharing about me. While XAuth enables service providers to extend authentication across sites more efficiently, it also gives users the ability to see and manage what is being extended. Users can visit xauth.org to see what services are using XAuth for.

Users can see and manage XAuth at xauth.org

What’s Next?

There’s even more you can do with XAuth and we’re looking forward to bringing you some exciting additional examples of how you can use this new platform.

XAuth is available as a preview right now for service providers and developers. Meebo, Google and Microsoft have put together a few demo pages to give developers a preview of what will be possible. To check out how we envision this working with the Meebo Bar start here.

You can head on over to xauth.org/info to learn more about XAuth. In the coming weeks, we plan to continue to move forward on XAuth and collaborate further with additional partners, the Open Identity Exchange (OIX) and the OpenID Foundation to facilitate an open consortium to host the XAuth technology.

Stay tuned!

- Chris

[ comments ] [ past blogs ]
Follow us on Twitter