Popular searches

Quant log | 4 min read

Chatting with ChatGPT

Oguz Tanrikulu, Andrew Uhl and Nastja Bethke
< Back to Resource center
Subscribe for updates

Oguz Tanrikulu, Director of Quantitative Research; Andrew Uhl, Senior Quantitative Analyst; and
Nastja Bethke, Senior Quantitative Researcher
April 3, 2023

Since its release in November 2022, ChatGPT has created a lot of headlines and received a lot of attention in a wide variety of professional fields. The “large language model” (LLM) developed by OpenAI has been trained on a massive dataset of text and has learned to generate human-like responses to various types of queries and prompts. 

Many users have been astonished by the sophistication of its responses to fairly complex questions, leading some to proclaim the end of certain types of professions that can now be replaced by this AI.  

Here we are interested in a much narrower question, namely whether ChatGPT, or algorithms like it, could be useful in systematic trading. Natural language processing (NLP) – a field that encompasses neural network-based models such as ChatGPT as well as other models, has long been used in some form or another in helping to build signals that can be used for trading stocks, futures and other financial markets. One of the simplest NLP techniques might arguably the “bag of words” approach where text is scanned for keywords that are linked – typically by the user – to a positive or negative score. 

With its seemingly human-like language abilities, ChatGPT might at first glance be a much better tool to analyze sentences, as it seems to be equipped with real “understanding”, however that may be defined. We tested ChatGPT’s responses to our request to analyze 2 sentences taking from the financial news. 

The first request submitted used a headline pulled from Bloomberg on 3/28/23. 

1) Analyze the following sentence about gold price and provide a numerical sentiment between -100 and 100. Negative sentiment should have a negative numerical value and positive sentiment should have a positive numerical value: “Gold Slides as Hotter US Inflation Keeps Hawkish Fed on Track”.  

Other than a deterministic algorithm, ChatGPT has some randomness embedded into it (at its core it autocompletes the next word in each sentence by sampling from a distribution of the most likely word to come next). This means that the same input will not lead to the same output necessarily. A couple of sample responses are shown below. 

  • The sentence expresses a negative sentiment towards gold price, as it suggests that the price of gold is decreasing due to the hawkish stance of the US Federal Reserve in response to rising inflation. Therefore, the numerical sentiment could be around -70. 
  • The given sentence expresses a negative sentiment towards the gold price. The use of the word “Slides” suggests a decline or drop in the price of gold. The sentence further indicates that the reason for this decline is the “Hotter US Inflation” which is causing the “Hawkish Fed” to stay on track. This implies that the Fed may continue to pursue policies that could further hurt the gold prices. Overall, the sentiment can be considered moderately negative. On a scale of -100 to 100, the numerical sentiment for this sentence could be around -60. 

Prompting ChatGPT to respond to the request 20 times overall, we arrived at the following distribution of sentiment scores for the given sentence. The mean sentiment score was -70.75 and the standard deviation was 5.91. This seems pretty consistent and satisfactory. Note though that the input sentence is quite simple, with the key word being “slide” indicating the trajectory the price is taking. A bag of words approach should have been able to pick this up as well. 

Using a somewhat more complex example, we find a different result. The second request submitted used a headline pulled from Natural Gas Intelligence on 3/24/23. 

2) Analyze the following sentence about natural gas price and provide a numerical sentiment between -100 and 100. Negative sentiment should have a negative numerical value and positive sentiment should have a positive numerical value: “Natural gas futures flipped positive early Thursday after a government inventory report proved bullish relative to recent years. However, it ultimately was not enough to ease festering supply/demand imbalance concerns.”  

Sample responses are again shown below. 

  • The sentence contains both positive and negative sentiments. The positive sentiment is that natural gas futures turned positive early on Thursday due to a bullish inventory report. However, the negative sentiment is that this positive turn was not enough to alleviate the persistent concerns over supply/demand imbalances. Overall, the sentence seems to be more negative than positive, given the persistent concerns over supply/demand imbalances. Therefore, the sentiment rating for this sentence would be around -60. 
  • The sentence contains a mixed sentiment where the first part expresses a positive sentiment about the natural gas futures turning positive due to a bullish inventory report. However, the second part of the sentence indicates that the positive sentiment was short-lived and the supply/demand imbalance concerns continue to persist. Overall, the sentiment can be rated slightly positive but with a note of caution due to the unresolved supply/demand concerns. Numerical sentiment rating: 40. 

We find a much wider distribution of sentiment scores than before, see plot below, with a mean and standard deviation of -28.50 and 34.50, respectively. The standard deviation in this instance is larger than the mean of the distribution, making it difficult to draw any overall conclusion. 

At the core of systematic trading strategies lies a design imperative of stability and robustness. The same input should give the same output, regardless of when the algorithm is run or how often, and small deviations in input should lead to small deviations in output.  

Beyond this, systematic trading strategies should make transparent decisions – one should be able to rationalize output given the input. This is sometimes a bit more tricky with modern machine learning algorithms which can be highly nonlinear, but nevertheless transparency is the goal. 

While ChatGPT’s capabilities are alluring, the wide range in outputs for the same input, once non-trivial text inputs are used, makes it unsuitable in its current form (and “straight out of the box”) for systematic trading. 

This critique can be seen as somewhat unfair, as most humans would probably give different answers to the same question depending on when they are asked, IF the question is sufficiently vague. But then again humans do not make systematic trading decisions. 

Subscribe to our Resource Center

By clicking the Button you confirming that you’re agree with our following Terms and Conditions

Related Quant

Resource Center
© 2024. All rights reserved

Subscribe to our Resource Center

Welcome to Graham Capital Management

By entering this website you are agreeing to its Terms of Use.

The risk of loss in trading commodities can be substantial. You should therefore carefully consider whether such trading is suitable for you in light of your financial condition. The high degree of leverage that is often obtainable in commodity trading can work against you as well as for you. The use of leverage can lead to large losses as well as gains.

In some cases, managed commodity accounts are subject to substantial charges for management and advisory fees. It may be necessary for those accounts that are subject to these charges to make substantial trading profits to avoid depletion or exhaustion of their assets. The disclosure document contains a complete description of the principal risk factors and each fee to be charged to your account by the commodity trading advisor (“CTA”). The regulations of the commodity futures trading commission (“CFTC”) require that prospective clients of a CTA receive a disclosure document when they are solicited to enter into an agreement whereby the CTA will direct or guide the client’s commodity interest trading and that certain risk factors be highlighted. This brief statement cannot disclose all of the risks and other significant aspects of the commodity markets. Therefore, eligible investors should carefully study the disclosure document to determine whether such trading is appropriate for you in light of your financial condition. Eligible investors are encouraged to access the disclosure document by contacting Graham, which will be provided  at no additional cost. The CFTC has not passed upon the merits of participating in this trading program nor on the adequacy or accuracy of the disclosure document. Other disclosure statements are required to be provided to you before a commodity account may be opened for you.  By accepting the terms of this statement and entering the site you are confirming your understanding of this statement.

Any “benefit plan investor” (e.g. An IRA or other ERISA investor) investor is hereby deemed to represent to Graham Capital Management (“Graham”) that an independent fiduciary (as defined in the ERISA fiduciary rule) is exercising its independent judgment with regard to such investor’s investment in any Graham managed fund and is aware of and acknowledges and agrees that Graham and its affiliates are relying on the exception set forth in clause (c)(1) of the ERISA fiduciary rule (i.e., the “transactions with independent fiduciaries with financial expertise” exception) with respect to any communications made to the investor or the investor’s fiduciary that are considered recommendations concerning any transaction and such benefit plan investor.

Although the site may include investment-related information, nothing on the site is a recommendation that you purchase, sell or hold any security or other investment, or that you pursue any investment style or strategy.

We do not give any advice or make any representations through the site as to whether any investment is suitable to you or will be profitable.

Nothing on the site is intended to be, and you should not consider anything on the site to be, investment, accounting, tax or legal advice. If you would like investment, accounting, tax or legal advice, you should consult with your own financial advisors, accountants or attorneys regarding your individual circumstances and needs.

The past performance of any investment, investment strategy or investment style is not indicative of future performance.

If you have been provided with a password to access the site you are solely responsible for maintaining the confidentiality and security of your password. You may not disclose your password to any third party. You accept full responsibility for any use of your password. You must notify Graham immediately of any actual or suspected loss, theft or unauthorized use of your password.

We are not obligated to inquire as to the authority or propriety of any use of or action taken under your password. We will not be responsible for any loss to you that arises from such use or action or from your failure to comply with these provisions.

The site, including all content, is provided as is and as available.

We disclaim all representations and warranties, express or implied, of any kind with respect to the site and the content including warranties of merchantability, fitness for a particular purpose and non-infringement of intellectual property and proprietary rights.

Without limiting our general disclaimer, we do not warrant the availability, accuracy, completeness, timeliness, functionality, reliability, sequencing or speed of delivery of the site or the content.