From Zero to Real-time Bidding (RTB)

A story

Posted by Alejandro Peralta 2 years, 3 months ago Comments

Prologue

In the beginning there was happiness.

Then, someone came along with a bunch of tubes and invented the Internet. Through the tubes flow an incredible amount of advertising that is displayed on the pages we fetch to get happier.

One of the ways an ad is attached to a page is through a technique called RTB. When our browsers requests a page through the tubes to a site using this technique, behind the scenes, a request is made to an intermediary (called exchanges) which starts offering the ad space in that page to machines that want to buy (on behalf of people, no skynet, yet) that space.

Through a bidding mechanisms one of the buyers of the space will be notified that it won, and it will forward that offer “you can’t refuse”. All this is thoroughly explained else where...

But, this particular story is about how to get one of them machines to buy ad space.

The story...

... or article, begins when, we, the developers, were thinking about something completely unrelated to advertising. We were probably procrastinating, excuse us, deploying one of our current projects... can’t say compiling, we use Python.

We were asked if we knew about OpenRTB. We quickly googled and figured out that it was something about selling ads using HTTP as transport. “Great, it’s HTTP”, we thought, we are web developers, so the answers was, “sure, we can learn!”.

We were asked to implement “one of those machines to buy ads space”. To be more specific a program which bids on impressions (an instance of a web advertisement being seen) called a bidder.

As we read through the specification of OpenRTB, we found out that we had to deliver the ads in about 100ms. The “RT” part stands for “real-time”. We also figured out that OpenRTB is an effort to standardized the RTB technique. And... that we should be prepared to handle gazillion of requests! (Remember, remember, that you are bidding on impressions, that you could be bidding for many requests from different sites!)

Was there anything already done that we could use?

Lo and behold! Enter RTBkit. RTBkit is a framework that (from the FAQ):

”...takes much of the hard engineering work out of creating a Real-Time Bidder for online advertising. Its open, service-oriented architecture can be used to assemble a bidder as simple or complex as desired.”

So, TL;DR, with RTBkit you can build bidders. RTBkit handles the communication, filtering (segmenting the campaign) requests, augmenting the requests with information. One has to provide the bidding logic.

RTBkit, built by Datacratic, has great documentation on getting started. We used a virtual machine, installed Ubuntu 12.04.5 and followed the installation procedure in the documentation.

We tried out our installation following the Demo-Stack guide from the wiki. We made the mistake of not installing at first Graphite. We learned later it’s usefulness for debugging and checking the health and status of the system.

We were happy, but that too, passed. How do we bid? How do we create a campaign?

If unlike us, your C++ is not rusty, you would follow the guide on How to write a bidding agent. If you are more like us you would start with this guide how to set up an agent with python.

Oh. Did we mention that Agents handle the campaigns? That’s were you have to write the bidding logic.

Our understanding of an agent is an implementation of a campaign. An agent can be a service, which listens directly to RTBkit or behind another service (like an HTTP server through the HTTPBidderInterface, see below) for the bid requests and sends RTBkit the bid response for each impression that suits the campaign.

We won’t explain the whole architecture of RTBkit. We will concentrate on the router, the agents, the ad server and the banker.

On how to talk to an exchange, please go see the documentation, maybe there will be a sequel to this story :-)

Router

The router receives a stream of bid requests from several exchanges. Requests will be filtered by the router depending on your agents configuration. The agents’ configuration are registered in the Agent Configuration Server. It’s through the ACS which you communicate to the RTBkit system what agents you have and it’s configurations.

The router will forward the request to the agents those request that pass all the filters. The router communicates with the agents through an interface.

There are two interfaces:

  • AgentsBidderInterface (We will not talk about, ‘cause C++).
  • HTTPBidderInterface

Also there is MultiBidderInterface which allows us to multiplex interfaces. You can configure several interfaces through the MultiBidderInterface. There are good docs on the subject.

To create our bidder we used the HTTPBidderInterface, that is well described in the documentation.

We used it mainly because we didn’t have time to learn C++ to be effective in the short period of time we had (yes, you know, deadlines).

Agents

One question that we had was, how can we instantiate campaigns dynamically, that is how to instantiate agents and let RTBkit know we have a new campaign?

Well to answer the later part of the question, we found out that sending the ACS the configuration of the agent and the router will start sending bid request to the agent.

The configuration lasts until a DELETE (HTTP) is issued to the ACS informing it that a specific agents is to be stopped. If the router stops the agents are also lost and you have to reconfigured them.

You can use the HTTPBidderInterface to handle multiple agents. You don’t have to configure an HTTPBidderInterface per agent! When using this interface the bid requests are augmented with a list of allowed creatives per agent. With this information one can forward the request to the agents who have those creatives and let each agent handle the bid request.

The bid request comes with a field "ext" that includes a list of "external-ids" that identify each agent and a list of the ids of the creatives ("creative-ids") that were configure per agent. The external-ids matched with those set on the property "externalId" per agent in the configuration.

The following is an example of the value of the "ext" field in a bidding request.

"ext": {
    "external-ids": [0, 1],
    "creative-ids": {"0": [10], "1": [11]}
}

What this means is that the following bid request can be answered with the creative with id 10 of agent with "externalId" equal to 0 and with the creative with id 11 of the agent with "externalId" equal to 1.

To answer the first part of the question (how to instantiate agents) we figured we could have a service that listens to commands that would register a new agent to the ACS and create new service to listen to the requests with a particular "externalId".

In older versions of RTBkit "creative-ids" was known as "creative-indexes" and held the indexes of the creatives.

A note about MultiBiddierInterface: You cannot instantiate interfaces dynamically. The configuration is static and it’s given to the router when it starts. So you can’t use it to dynamically add agents.

Ad Server

How do our agents know if they won the bid? RTBkit provides ad server events to communicate different events, such as clicks on impressions, winning bids on impressions, etc.

We were confused when we had to configure these services, as there are three different components that are called Ad Sever:

  • The ad server events, as we said the way RTBkit communicates the agents that it won a bid.
  • Actual ad servers, for example like Revive whose purpose is to serve impressions and can be used to notify RTBkit.
  • The ad server connector which is part of the RTBkit architecture. It’s purpose is to receive event notifications (such as wins, clicks, etc) from the actual ad server. It will parse the event and notify the PAL (Post-Auction Loop) which will then match the win, click, etc. to the bid made. The PAL then notifies agents.

Banking

So we had our agents running. We test it with mock_exchange_runner. We had no wins.

Agents spend money and RTBKit provides a way to manage the accounting of your agents through it’s banking service

We stumbled a bit when we configured several agents. We learned that the most important thing you have to remember about banking is:

Each agent have their own account tree. That is each agent has to have and account where you can change the budget.

From the example in the documentation there is "hello:world". "hello" is the root of the tree and "world" is the child account. Each agent should have it’s "hello".

The other most important thing is to never forget to add a budget to the agent’s account. (You won’t be able to bid).

And lastly the other, other most important thing is to never forget to add balance to the account the agent is using. :-)

Lastly after adding some money to the accounts our bidder starting receiving WINS!

Some notes

  • The creatives (the markup for the ads) that your agents is offering should be on the agent configuration. You can’t just add creatives to the agent, you have reconfigure the agent.
  • How does the exchange get the add you ask? Well each creative should have an id. With this id and the providerConfig of the agent configuration, the router builds the nurl (see the OpenRTB spec) for the exchange. This attribute holds the url for the creative in your adserver (the id is a parameter of the url).

Further Information

Thanks

To octal_ at #rtbkit at freenode who we bugged constantly and the RTBkit community (in no particular order Jeremy, Eric, Mathieu who answered my emails in the list and Nicolas who answered questions regarding his agent_gateway )

Epilogue

The idea of this post was to tell our experience and stumbling blocks we found, and to serve as a guide into RTB, RTBkit when setting up a bidder. Hopefully we also gave you enough references for you to read and get informed.

We’ve built a tool (“bidderd”) after some initial interest from the community and some clients in the industry to get you started. It’s written in Go as it was an opportunity to try it out.

There is a lot still to talk and learn about.

We are sorry for the narrative techniques used in this article. Most of our literary background is Douglas Adams and quotes from Monty Python.

No developers were harmed in the making of this story. Well they suffered a little, but at the end no serious damage was done.


Previous / Next posts


Comments