A reinforcement learning approach to improve the performance of the Avellaneda-Stoikov market-making algorithm PLOS ONE

In its beginner mode, the user will be asked to enter min and max spread limits, and it’s aversion to inventory risk scaled from 0 to 1 . Additionally, sensitivity to volatility changes will be included with a particular parameter vol_to_spread_multiplier, to modify spreads in big volatility scenarios. While the other parameters are fixed to those in Table1, we see that there are more buy market orders arriving, thus the optimal filled sell spreads are larger for all inventory levels comparing to the case when the arrival of market orders is symmetric. Recently, there have been crucial developments in quantitative financial strategies to execute the orders driven in markets by computer programs with a very high speed . In particular, high-frequency trading is one of the major topics that attracts the attention excessively due to its benefits on market microstructure, being an interdisciplinary field including the hot topics, such as stochastic optimization, finance, economics and statistics.

We mention neuroevolution to train the neural network using genetic algorithms and adversarial networks to improve the robustness of the market making algorithm. Reinforcement learning algorithms have been shown to be well-suited for use in high frequency trading contexts [16, 24–26, 37, 45, 46], which require low latency in placing orders together with a dynamic logic that is able to adapt to a rapidly changing environment. In the literature, reinforcement learning approaches to market making typically employ models that act directly on the agent’s order prices, without taking advantage of knowledge we may have of market behaviour or indeed findings in market-making theory. These models, therefore, must learn everything about the problem at hand, and the learning curve is steeper and slower to surmount than if relevant available knowledge were to be leveraged to guide them. Figure3 depicts one simulation of the profit and loss function of the market maker at any time t during the trading session in the left panel. The profit and loss performance of the trading is displayed by the cash level histogram in the left panel.

Maximum drawdown

But as its value increases, the distance between the mid-price and the reservation price will increase when the trader inventory is different from his target. The value of q on the formula measures how many LTC units the market maker inventory is from the desired target. To start this override feature, users must input the parameters manually in the strategy config file they intend to use. This parameter denoted by the letter kappa is directly proportional to the order book’s liquidity, hence the probability of an order being filled. The Volatility Sensibility will recalculate gamma, kappa, and eta after the value of volatility sensibility threshold in percentage is achieved. For example, when the parameter is set to 0, it will recalculate gamma, kappa, and eta each time an order is created.

For more developments in avellaneda & stoikov market making literature, we refer the reader to Guéant , Ahuja et al. , Cartea et al. , Guéant and Lehalle , Nyström and Guéant et al. . High-frequency trading is a popular form of algorithmic trading that leverages electronic trading tools and high-frequency financial data. A typical HFT algorithm is based on limit order book data (Baldauf and Mollner, 2020, Brogaard et al., 2014, Kirilenko et al., 2017). 1 illustrates the bid and ask prices and their 5-level queues for a stock at two consecutive time points .

Trading signals in VIX futures

In order to see the time evolution of the process for larger inventory bounds. This part intends to show the numerical experiments and the behaviour of the market maker under the results given in Sect. Similar to the proof of Proposition2, the optimal spreads can be found by the first order optimality conditions.

Data transmission is achieved through radio packet transfer, thus it is prone to various attacks such as eavesdropping, spoofing, and etc. Monitoring the communication links by secure points is an essential precaution against these attacks. Also, deploying monitors provides a virtual backbone for multi-hop data transmission.

To this , more specifically one based on deep reinforcement learning, we turn to next. In this paper we present a limit order placement strategy based on a well-known reinforcement learning algorithm. We use the RL algorithm to modify the risk aversion parameter and to skew the AS quotes based on a characterization of the latest steps of market activity. Another distinctive feature of our work is the use of a genetic algorithm to determine the parameters of the AS formulas, which we use as a benchmark, to offer a fairer performance comparison to our RL algorithm.

Login or register to access this feature

We also consider the case of the market impact occuring by the jumps in volatility dynamics. We derive the closed-form solutions for the optimal quotes and solve the corresponding nonlinear HJB equations using the finite difference discretization method which enables us to evaluate the spread values and derive the various simulation analyzes. Furthermore, we explore the risk and normality testings of the models depending on their strategies. Lastly, we compare the models that we have derived in this paper with existing optimal market making models in the literature under both quadratic and exponential utility functions. Data normalization for features and labeling for signals are required for classification. Instead of simply labeling the mid-price movement as in Kercheval and Zhang and Tsantekidis et al. , we consider the direct trading actions, including long, short, and none.

The results obtained from these tests are discussed in Section 6. The concluding Section 7 summarises the approach and findings, and outlines ideas for model improvement. Gen-AS performs better than the baseline models, as expected from a model that is designed to place bid and ask prices that minimize inventory risk optimally given a set of parameter values that are themselves optimized periodically from market data using a genetic algorithm.

The imprecise Dirichlet model provides workaround, by replacing point probability estimates with interval-valued ones. This paper investigates a new tree aggregation method based on the theory of belief functions to combine such probability intervals, resulting in a cautious random forest classifier. In particular, we propose a strategy for computing tree weights based on the minimization of a convex cost function, which takes both determinacy and accuracy into account and makes it possible to adjust the level of cautiousness of the model. The proposed model is evaluated on 25 UCI datasets and is demonstrated to be more adaptive to the noise in training data and to achieve a better compromise between informativeness and cautiousness. Together, a) and b) result in a set of 2×10d contiguous buckets of width 10−d, ranging from −1 to 1, for each of the features defined in relative terms. Approximately 80% of their values lie in the interval [−0.1, 0.1], while roughly 10% lie outside the [−1, 1] interval.

Market indicators, consisting of features describing the state of the environment. Thus, the DQN approximates a Q-learning function by outputting for each input state, s, a vector of Q-values, which is equivalent to checking the row for s in a Qs,a matrix to obtain the Q-value for each action from that state. A discount factor (γ) by which future rewards are given less weight than more immediate ones when estimating the value of an action (an action’s value is its relative worth in terms of the maximization of the cumulative reward at termination time). To maximize trade profitability, spreads should be enlarged such that the expected future value of the account is maximized. For asymptotic expansions when T is large you should read the paper by Guéant, Lehalle, and Fernandez-Tapia here or the book of Guéant The financial mathematics of market-liquidity.

A Metaheuristic Algorithm for Vertex Cover based Link Monitoring and Backbone Formation in Wireless Ad hoc Networks

Inventory management is therefore central to market GALA making strategies , and particularly important in high-frequency algorithmic trading. In an influential paper , Avellaneda and Stoikov expounded a strategy addressing market maker inventory risk. The optimal bid and ask quotes are obtained from a set of formulas built around these parameters.

• The half-second required by the system is put to good use in practice.
• This will set “boundaries” to the calculated optimal spread, so hummingbot will never create your orders with a spread smaller than the minimum nor bigger than the maximum.
• The two most important features for all three methods are the latest bid and ask quantities in the orderbook , followed closely by the bid and ask quantities immediately prior to the latest orderbook update and the latest best ask and bid prices .
• Based on the market state and the agent’s private indicators (i.e., its latest inventory levels and rewards), a prediction neural network outputs an action to take.

The market microstructure, which can be stated as the research on the strong trading mechanisms managed for the financial securities, has been equipped with the contributions by the books Hasbrouck and O’Hara . The question of the truncation of the interval of possible state feature values remains open, or there seems to be some misunderstanding between the authors and the reviewer. For instance, how are market prices (or actually differences to the mid-price) truncated to the interval [-1,1]? Are they scaled by some scaling parameter beforehand – and what data is this parameter estimated from ? If not, how much data is lost by only using the price differences with absolute values smaller than 1? Also, if the market candle features are “divided by the open mid-price for the candle”, does this mean that all of those higher than the mid-price would be would be truncated to 1?

At each training step the parameters of the prediction DQN are updated using gradient descent. An early stopping strategy is followed on 25% of the training sets to avoid overfitting. The architecture of the target DQN is identical to that of the prediction DQN, the parameters of the former being copied from the latter every 8 hours.

For instance, Avellaneda and Stoikov (ibid.) illustrate their method using a power law to model market order size distribution and a logarithmic law to model the market impact of orders. Furthermore, as already mentioned, the agent’s risk aversion (γ) is modelled as constant in the AS formulas. Finally, as noted above, implementations of the AS procedure typically use the reservation price as an approximation for both the bid and ask indifference prices.

Therefore, the trader will have the same risk as if he was using the symmetrical price strategy. These are additional parameters that you can reconfigure and use to customize the behavior of your strategy further. To change its settings, run the command config followed by the parameter name, e.g. config max_order_age.

Moreover, Yang et al. have improved the existing models with Heston stochastic volatility model, to characterize the volatility of the stock price with price impact and, implemented an approximation method to solve the nonlinear HJB equation. They have considered a constant price impact using the same counting processes for both arrival and filled limit orders. More recently, Baldacci et al. have studied the optimal control problem for an option market maker with Heston model in an underlying asset using the vega approximation for the portfolio.

• Another distinctive feature of our work is the use of a genetic algorithm to determine the parameters of the AS formulas, which we use as a benchmark, to offer a fairer performance comparison to our RL algorithm.
• It also leaves sufficient time to submit and execute orders before the next tick-report.
• The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions.
• Alternatively, experimenting with further layers to learn such policies autonomously may ultimately yield greater benefits, as indeed may simply altering the number of layers and neurons, or the loss functions, in the current architecture.

https://www.beaxy.com/ that are very large can have a disproportionately strong influence on the statistical normalisation of all values prior to being inputted to the neural networks. By trimming the values to the [−1, 1] interval we limit the influence of this minority of values. The price to pay is a diminished nuance in the learning from very large values, while retaining a higher sensitivity for the majority, which are much smaller.

For solving the combinatorial problem, RAGE generates a customizable number of random solutions, computes the objective function for each solution, and then scores each candidate element in terms of the value returned by the objective function. After that, RAGE removes a customizable number of candidate elements presenting the smallest score when considering all solutions generated. The heuristic loops performing iterations until there are left the exact number of candidates that we are looking for. In order to evaluate the efficiency of RAGE, we perform experiments showing how RAGE behaves when we change the number of random solutions generated per round, and the number of candidate elements removed per round. Finally, we apply RAGE for solving an NP-Hard problem related to the allocation of infrastructure for vehicular communication. The original Avellaneda-Stoikov model was chosen as a starting point for our research.

There are various methods to achieve this, a particularly common one being gradient descent. The models underlying the AS procedure, as well as its implementations in practice, rely on certain assumptions. Statistical assumptions are made in deriving the formulas that solve the P&L maximization problem.