I had been working in the advertising industry, specifically in programmatic advertising, since 2013; and exited in November of 2019, my total period of operation was 6 years, and five of those years had been spent as a developer where I built programmatic advertising platforms for other clients, usually under contract, Supply-Side and Demand-Side. This article consists mostly of my experiences in my final year when I decided to give a shot at running my own Demand-Side Platform after 5 years of making them for other people. I don’t regret my time working with OpenRTB; it was interesting, and although my DSP made a small amount of profit I was happy to have made a profit at all. I didn’t close the business resentfully, I just felt like it was too hard to operate such a business based on the kind of profit I was making, and I was not willing to bend the rules to increase my margins or turn a blind eye to clients who might have been.
Even with all of the fraud checks we could possibly do, some of which I still will not reveal in this document as they are well-guarded industry secrets, making a return on investment from advertising was very hard.
As my company was a Demand-Side Platform (DSP) the theory was that I didn’t really need to worry about running ad-campaigns that made an ROI, rather I only had to concern myself with providing a decent service for clients to run their campaigns on and maintaining healthy relationships with the traffic suppliers who provided traffic to my platform. The latter was the easy part, and the way the industry worked at the time is that a DSP would take anything between 7–30% of a advertisers spent budget as payment, this percentage was public and often negotiated between the client and the service provider. Clients with a lower monthly spend would have a higher percentage and clients with a large spend would often have a much lower percentage fee. However the former was the tricky part as if your client broke any rules with the advert they were running, you would be penalised by your traffic suppliers for not moderating/screening your clients correctly. For bigger company's that can afford to splash a bit of extra cash, they would often use the services of The Media Trust (TMT) which would automatically screen clients advertisements before the platform took them live. For smaller startups like mine that was not an option, we had to rely on manual moderation, and often traffic suppliers were warier of taking on clients who did not use TMT, as it was an industry-standard at that time.
Another problem is that many clients would want to use NET30 or NET365 (that one is an industry joke) what this basically means is that if a client would like to work on a NET30 basis with you that they would be paying you at the end of a 30-day interval, the long story short is that 99% of the time they are just looking to rip you off, they will rack up enough debt with you as they can and then cut and run depending on how much ROI they made, but there were so many DSP’s to choose from in this saturated market that there many clients who would just cut and run on every DSP to make some money on the side, only paying the bills to larger SSP’s who provided the best traffic for the lowest price. So you can see why NET365 was an industry joke, pay at the end of the year, good luck getting that “haha”.
So running a DSP was a tricky place to hedge yourself, in order to justify the percentage that you were charging on a clients advertising budget, you had to have good quality traffic suppliers which were hard to obtain without a minimum entry spend of 10–30 thousand USD and you additionally needed a platform that was feature-rich. I had the platform and my plan was to work my way up the ladder in terms of traffic suppliers, unfortunately this was much harder than I had hoped and ultimately never materialised having to close the business one year later making a net profit of around 7,000 GBP, after paying one employee on a percentage basis which worked out to around 300–800 GBP a month as well as various operating costs.
The problem with running DSP is that the majority of the clients that end up approaching you are dodgy, and I mean, really, dodgy. But also clever, very clever and very cunning.
One such example was a particular client who approached me via LinkedIN, he went by the identity of a recent graduate from Cambridge University and looking up the identity it seemed very legitimate — there was no reason for me to suspect this person was dodgy in any way what so ever, however it would later transpire he was in-fact not who he claimed, as with any new client I would ask other connections in the industry that I worked with or bought traffic from if they knew this particular person or had worked with them, and yes it turns out this man from Cambridge was actually an Iranian and he was running fake tech call scams (I think he was harvesting phone numbers and email addresses via fake competition entries or something of that nature to then re-sell to dodgy actors who would use that information to perform fake tech call scams), it took us all the best part of almost two months to fully uncover what he was doing and luckily I managed to dodge that bullet but some of my industry contacts did end up providing him with some traffic. The thing is, it’s a pretty desperate world out there in advertising, it’s hard to get clients, and often when one comes along flashing cash you have to take them on if you want to pay your bills for that month — and dodgy clients like the Iranian were never short of cash, boasting an advertising budget in the tens of thousands. For this reason, many people like to live in ignorant bliss, but the truth is, if you’re running a business as a middle-man like a DSP you have to demonstrate at-least some due-diligence with the clients you take on if you wish to remain in business for the long run. The problem is, they are so well-versed in deception, that often you’d be too late to realise, a really well versed conman can hide their dodgy payload, be it tech scams or whatever they are ‘pushing’ in those adverts, from even you running the platform and often they are pushing a wide variety of ‘baddies’, they will use IP analytics to ensure their payload is only delivered to very specific users and that if you try to look for their payload it would never be served to you because you don’t match that very specific nice that they are targeting — so when you look at their advert you just see something normal like Xero but a user from Spain on Lowi telecom from the city of Vigo could get something completely different, something malicious in nature perhaps.
So why, you may ask, are the majority of clients who use DSP platforms dodgy? Well, it’s like this, legitimate advertisers rather than run the risk of being ripped off prefer to advertise with well-known options such as Google via the Google Adwords service. If a client can’t advertise with Google or get an account with a bigger, more reputable DSP they will start hunting the smaller guys, like myself. Now if a client can’t get an account with Google you already know they’re probably doing something dodgy, towards the end of my business year I had a lot of new clients approach me via LinkedIN because Google had just changed their terms of service and as a result purged a large number of advertisers they had using their platform — of-course all of these purged advertisers came looking for the DSP’s in order to continue their business. It was at this point I knew I would not be able to remain in business, either I was going to go down the road of getting NET30 scammed, or being forced to take on the ‘lesser-dodgy’ advertisers to just about make ends meet each month, and I really mean just about for the entire year I had 0 savings and was living 100% on the knife-edge so to speak.
On the more dodgy end of the spectrum, they could be using exploit kits to automatically install browser toolbars which will make them money on a Pay Per Install (PPI) basis with the company that makes the toolbars, via some kind of affiliate program, the toolbar company then will make their money back by hijacking your search engine with their own custom version which displays their adverts and paid affiliate links as search results.
Then on the really dodgy end of the spectrum they could be infecting you with a banking trojan or ransomware such as the infamous CryptoLocker.
The problem is there’s really no way to know, often until it is too late and one of the publisher websites that is targeted by the ‘malvertising’ realises what is going on, often from a google alert or penalty being issued to their domain as it’s often google that notices these things before anyone, then the publisher will complain to the SSP and then the SSP will complain to you the DSP, or just outright terminate your account. That’s the thing, these malvertising campaigns are often so well hidden that only Google really has the power to detect them with their vast network of website crawlers, when one of them visits the target site displaying the malicious advert it will then issue a warning to the domain owner of that website via Google Webmaster Tools (if the site owner is even registered there, which more often than not they are, being Google is one of the biggest search engine providers in existence). Although it’s worth nothing that TMT, Risk IQ and Geo Edge also perform ad-tag scanning from a range of IP addresses and can be just as formidable as Google crawlers.
Not only did OpenRTB provide malverisers a better way to hide their malicious payloads but also it was too hard to implement, many developers got it wrong, even big networks were plagued by outages, corrupted packets, and just generally bad implementations. Many developers would implement their OpenRTB services in Node.js and PHP which would max out at around 6,000 QPS (Queries Per Second) which was just too little for most advertisers expectations which was more like 60,000 QPS and as such developers would need to run large expensive clusters of nodes to supply that demand. Almost no one was running a C or C++ implementation, and the use of TCP in the OpenRTB spec was in my opinion a huge mistake.
The problem with TCP is that in a real-time bidding environment the order of packets doesn’t matter at all, and also to be honest neither does the reliability of them reaching their destination — you are bidding on each bid request you receive against thousands of other people around the globe, it’s already a game of odd’s if you win a request or not, if 10% of your responses are lost, it’s not a big deal particularly if you can make 50% or more responses in that time. With TCP you’d hit a limit on a basic VPS server at around 6,000 – 9,000 QPS, but using UDP you’d be able to hit 100,000’s of QPS. That’s ~10–16x more bids that you could have bided on, with some slightly higher, but negligible in comparison, percentage of loss in the communication protocol.
OpenRTB is plagued by fraudulent traffic and dodgy malvertisers so both ends of the ecosystem are poisoned, you’re fighting a losing battle, particularly with monopolies like Google and Amazon taking up the best clients in the business leaving you with the dregs they don’t want.
I luckily managed to avoid many suspicious clients in my single year running a DSP, since then I didn’t even go back to working as a developer for other businesses that use the OpenRTB protocol; I just never looked back, I saw what a horrendous mess it all was first hand and washed my hands of it forever, the fact I even made a profit was a small miracle in its own right.
I have since open-sourced all of the technology I had written for my OpenRTB business (apart from the C/C++ bidder, which I’m sitting on in the very very rare case that the advertising industry turns a new leaf in the future), it is all available here it’s a fully-fledged Demand-Side Platform with customer accounts, campaign management, etc, it supports everything a customer needs. Platforms like that, made new, start at 10,000 USD and go up from there. So if you need a platform to connect to an SSP for internal use, by all means, use the one I made for free to avoid paying a middle-man fee to a DSP like AdKernel or similar.