Back to all posts

How Accurate is Our Slippage Model: Comparing Real and Simulated Transaction Costs

At Quantopian, our backtester's primary purpose is to accurately simulate trading strategies. If an algorithm would have performed well historically with real money, we also want our simulation to show positive performance — ideally matching almost exactly with what would have happened in the real world. In many ways, we believe we do an excellent job at this simulation; recently, however, we've been wanting to take a closer look. Using data from trading algorithms licensed from our community and run in our proprietary Interactive Brokers account, we dug into different sources of discrepancy between our backtester and the real world. One of those sources of discrepancy is transaction costs, or more specifically, commission fees and slippage.

If you're unfamiliar with what transaction costs are, as the name implies, they're costs associated with every transaction. This cost has two components: commission fees and slippage. Commission fees are what we pay to the broker for their work in processing and completing our order. A commission fee is applied every time an order is placed, but different brokers will have different fee structures and will charge different amounts. Slippage, on the other hand, is the penalty we incur when we place an order and it doesn't fill at the original price we wanted. For example, if we place an order for a share of some stock when its price is $25.00, but the order eventually completes with a price of $25.01, then we have paid an effective $0.01 in slippage. Slippage tends to be positive, especially for larger orders or when buying illiquid stocks, because our own order will alter the price of the asset against us.

We're going to need a way to measure these transaction costs so we can compare real-world performance with simulated performance. One way we might think to do this is just to look at the plain dollar amounts we pay in commission and slippage for each order. But this creates a problem: since we tend to pay more slippage and commission for larger orders, and our orders are for different amounts, then we may get different conclusions depending on what size orders we are looking at. Therefore, what we really want is a cost we're paying relative to how much money we're trading. So, for example, if we make the same order from before that completes at $25.01, which is $0.01 higher than intended, then we can say $0.01/$25.01 = 0.04% of our total cost was slippage. If we paid $0.08 in commission, then we can say we paid $0.08/$25.01 = 0.32% more due to commission. Now, these fractions tend to be very small, so instead of working with percentages we work instead with basis points. A basis point is just one-hundredth of a percentage point. So using those examples from before, we paid 0.04% = 4 basis points (or bps) in slippage, and 0.32% = 32 bps in commission for that particular trade. We can then compute these costs for each trade and analyze the results to see how closely our backtester and the real world align.

Commission

Let's start off by looking at some data on commission fees. This is a sample of some real orders sent to Interactive Brokers, and those same orders simulated under our backtester's default commission setting. Each blue dot represents an order on Interactive Brokers (IB), and has associated with it a number of shares and a commission fee in dollars. Each red dot represents an order in our backtester:

nccex4wiaraaaaaelftksuqmcc

At first, it looks like we are charged far more under IB than we simulate because the blue dots seem generally above the red dots. However, there are a huge amount of dots right on the $1.00 line that are difficult to depict: in about 50% of the IB orders, the commission fee is $1.00. For larger orders, we also tend to simulate a larger fee than we actually receive.

For real IB orders, our median commission is 4.76 basis points, while our simulated median for those same orders is 4.65 basis points. This difference of 0.09 basis points adds up to $0.90 for every $100,000 transacted — not much. Of course, in an ideal world, these numbers and the plot above would match exactly if we wanted to perfectly replicate IB's behavior. If you're curious about IB's commission fee schedule, you can read about it here.

Slippage

Now let's take a look at slippage. To dig into this, we can look at the distribution plots for slippage on IB and slippage on our own backtester using the default slippage setting (both with market orders). The blue histogram depicts slippage on Interactive Brokers while the green histogram depicts slippage on Quantopian's backtester:

h1lelefjeuvgaaaaaelftksuqmcc

Clearly things are different between the two cases. The green distribution, our backtester's slippage, seems centered around zero and relatively flat. The blue distribution, however, has a distinct crest around two basis points — this indicates than on IB, we are often getting our orders filled at 0.02% higher of a price than we originally wanted. Additionally, under IB, we see a huge spike at zero basis points. This means we often get exactly the price we want, and that the price doesn't change from the time we order to the time our order gets filled. This is an effect of the fact that our order sometimes fills so quickly the price hasn't yet drifted from the price at the time of our order.

Taking a look at the numbers, the median slippage on IB is 1.64 basis points, while the median slippage for the same orders in our backtester is only 0.02 basis points. This difference of 1.62 basis points (or 0.0162%) means that for every $100,000 we transact, our backtester overestimates profit by $16.20 according to this sample of orders. Although the difference isn't large enough to decide the fate of a strategy in most cases, it does add up over time, especially for algorithms that trade frequently.

We're currently working on implementing a more accurate, state-of-the-art slippage model that we hope will shift the green distribution in the plot above to nearly match the blue one. In the meantime, if you are concerned about the accuracy of the slippage model, you can set a custom model for your algorithm.

Fill Timing

The final source of discrepancy we're going to look at here is fill timing. Quantopian's backtester is currently based around minute bars — the backtester will jump from one minute to the next, and just fill orders with prices at the end of every bar. Here's how the timing of placing an order and that order filling might look in our simulation:

screen-shot-2016-11-02-at-4-15-58-pm

Although other pricing data is available in the backtester, orders will always transact at the close of each minute. If we place a market order at the end of the 9:33 bar, the earliest it can get filled in the simulator is at the end of the 9:34 bar. In the real world, with real money trading on Quantopian, the market order can get filled at any time after it's placed:

screen-shot-2016-11-02-at-4-20-58-pm

So, there is often a difference in fill timing between the simulator and what would have happened in the real world. The spot price at the close of the bar, which is what we use now, is not fully indicative of the price over the whole bar. This often doesn't bias the price in one direction or another — it's more just small random noise added to our simulation that we don't want. There are a few different ways to tackle this problem. One way we may decide to reduce this noise is to fill with a price that better represents the bar as a whole; possibly some sort of average price like the average of the high and the low within the bar. The goal will be to match real world fill prices as accurately as possible, aligning the simulation's distribution closer to the real-world distribution.

Although the impacts of these three effects we looked at are small in many cases, they reduce the accuracy of our backtester and can add up over time. We model commissions well, but there is still room for improvement, especially in catching the subtleties in different commission fee schedules. Although we don't model slippage well, we are currently working to improve this, and hope to have a state-of-the-art model on the platform soon. With fill timing, we plan on updating the fill price to reduce the amount of discrepancy in the form of noise in our simulation. Going forward, we'll continue to monitor and reduce discrepancies between the Quantopian backtester and the real world. We'll keep you updated as we uncover more and make updates to the backtester.

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

D Gill

1. Can I assume that if I leave a limit order there is no slippage? It's only hinted at here. How do you calculate the fill, if the order price is exceeded?
2. Is the commission rate Fixed or Tiered? (IB's terms)
3. Why IB in the first place? IB is retail. There are direct to market ECN's with API's that are providing much better commission structures, such as rebates for liquidity enhancing limit orders (meaning you can earn $). Steve Cohen should be aware of this, so for a $250 million allocation you would think he would set you guys up.

Quantopian

Thanks for the questions.

1. We simulate the fill based on the as-traded bar data, for both limit orders and market orders. We know that we can do more work at simulating limit order fills better, but it hasn't been a priority yet.

2. Fixed.

3. Quantopian doesn't want to be a broker itself. When we first provided live trading for our community members in 2013, IB was the broker that was in the highest demand. We surveyed our community about what brokerage they were using, and IB was the clear leader.

We are currently building an integration with a prime broker. When we manage Steve Cohen's money, it will be through that prime broker. Unfortunately we will not be able to make that brokerage an option for our community.

- Dan Dunn

Yoram Aviram

In the real world, when trading large quantities you sometimes have to break the order into a few partial orders. In that case, measuring slipage simply in relation to each orders market price, doesn't show the real market effect of our order (underestimating it). I believe, the real slippage should be measured in relation to the first (partial) order's market price.
I believe that's a major cause of difference between backtested and real world results.

Gus Gordon

Hi Yoram, you're right in that the way we measure slippage should be consistent across backtests and real-world trading. However, in this case we are consistent — we are measuring both backtester slippage and real world slippage relative to the same reference price.

Comments are closed.