Programming

How to (Nearly) Sidestep API Limits

Let’s say that you’re trying to grab a lot of data from a 3rd-party API. In fact, you’re grabbing so much data that you hit the API’s hourly request limit in a mere 3 minutes. What can you do? Well, if the API is open to the public, then your options are:

  • You can take a 57 minute coffee break.
  • You can beat the system.

I’ve already had my coffee today, so let’s beat the system.

Since the API is open to the public, we can just obtain another API key. This will allow us to grab data for another 3 minutes before we hit the API limit again. Afterwards, we can grab yet another API key and then repeat the process.

By the end of the hour, we’ll have 20 API keys. Additionally, we will be able to reuse our first API key because we’ve already waited for an hour. By the time we finish reusing the first API key, the 2nd API key will have been renewed as well. As you can see, each time we finish using an API key, the next one will become available again. Therefore, we no longer need to obtain any more API keys. Hooray!

Of course, swapping out these API keys every 3 minutes will be pretty inconvenient. Wouldn’t it be nice if there was a way to do this automatically? As a matter of fact, there is a way to do that: we can just create a fancy proxy server that performs an API key rotation for us. This would look something like this:

API Proxy

Now, instead of calling the API directly, we’ll call our API proxy. The API proxy will attach a non-exhausted API key to the request before forwarding the request to the API. Brilliant!

To help you get started, here’s a very simple proof-of-concept using node-http-proxy:

Our app.js file is pretty straightforward. We’re just intercepting all requests to /proxy/*, appending an API key, and then forwarding the request to the 3rd-party API. In production code, we’d want to avoid manipulating the URL via string concatenation, but it gets the job done for now. So, let’s write the code for our nextKey() function. For our proof-of-concept, we’ll just store all of the keys in memory rather than persisting them to a database:

Have fun beating the system!

Leave a Reply

Your email address will not be published. Required fields are marked *