How to get all the results available from your Google AJAX Search API application

If you haven't heard of it yet, the Google AJAX Search API is a pretty nifty little doo-dad. Basically, it allows you to add Google search functionality without having to divert visitors from your site, hack the Google infrastructure, or even deviate from your own formatting rules. It works like this: you build a control of some sort that will process user input; form a request URL; and get information, including web, local, news, video, image, blog, book, and even patent search results via JSONP. Pretty slick, if you ask me, but one of the questions I often receive regarding the API goes something like this: "It's a cool concept, but how can I get more results?" You see, using it, you can currently only get up to 64 results (less for local and blog searches), and you can only retrieve those in blocks of 8.

So for today's adventure, I thought we could explore getting as many results as we possibly can from the Google AJAX Search API. But right off the bat, there is a disclaimer: I'm not going to tell you how to get around the limit of total results you can retrieve. The AJAX Search API is designed to provide rudimentary search functionality for the users of your website or other application, not for SEO, data mining, or even deep searching. And it does that intended task admirably. To try to bypass the total results limit is a violation of the service's Terms of Use and, I believe, generally unnecessary when using the API as intended.

Rather, what we're going to talk about is getting all the results that they will let us get in one fell swoop.

So, how do we do it? Well, first things first; we set up our searcher as normal. Because the default google.search.SearchControl is obfuscated, we're going to have to use a RAW searcher. If you don't know how to do this, you'll want to check out the documentation at http://code.google.com/apis/ajaxsearch/documentation/.

Once that's done, there are two methods of the searcher object that we're going to take advantage of: .setSearchCompleteCallback() and .gotoPage(). Details for both of these methods are provided in the class reference.

Essentially, the process, broken down into its component steps, is going to go like this.

1. We set up the searcher's completion callback
2. We execute a search query
3. The query returns, executes the completion callback
4. The completion callback checks to see if there are more results
5. If there are more results to get, calls gotoPage(...);
6. Otherwise, finish processing the results

So, are you ready for this? Here's the code, commented up so you can see what's going on.

// Initialize the searcher object, in this case a WebSearch.
window.gs = new google.search.WebSearch();
gs.setResultSetSize(google.search.Search.LARGE_RESULTSET);

// Set the search complete callback. Notice that we're using the searcher itself as the context. This is going to allow us to use this in the callback to refer to the searcher.
gs.setSearchCompleteCallback(gs,function(){

// Set a handle to the cursor object that we're going to use repeatedly
var cursor = this.cursor;

// Create a new property on the searcher that we can stash results into so they don't disappear when we go to the next page.
if(!this.allResults || cursor.currentPageIndex==0){this.allResults = [];}

// Add the new results to the other results
this.allResults = this.allResults.concat(this.results);

// Check to see if the searcher actually has a cursor object and, if so, if we're on the last page of results. If not...
if (cursor && cursor.pages.length>cursor.currentPageIndex+1){

// Go to the next page.
this.gotoPage(cursor.currentPageIndex+1);

// Else, if there is no cursor object or we're on the last page...
} else {

// Loop through the results and...
for(var i=0; i<this.allResults.length; i++){
var result = this.allResults[i];


// Plug them into the document where we want them.
document.body.appendChild(result.html.cloneNode(1));
}
}
});


Check it out in action here.

And there you have it! All 64 results from a Google AJAX Search API WebSearch, in one shot.

What I learned on today's adventure:

1. I can use the Google AJAX Search API to get up to 64 search results for use on my site or in my application.
2. How to use searcher.setSearchCompleteCallback and searcher.gotoPage to get all available results in one fell swoop.