Receive free AI updates
we will send you myFT Daily Digest Email summary of latest information artificial intelligence There is news every morning.
We promise we won’t jump on the bandwagon and turn this site into an AI-phaville, but here’s a very interesting one work documents St. Louis Fed remarks on hot topics in 2023.
Miguel Faria e Castro and Fernando Leibovici used Google’s large-scale language model PaLM to generate retrospective inflation forecasts for 2019-23 and compared them with forecast and actual inflation reports from the Philadelphia Fed’s survey of professional forecasters.
Voila (emphasis from Alphaville below):
Our benchmark results show that LLM generates conditional inflation forecasts with lower mean square error (MSE) than more traditional forecast sources (SPF) over the analysis period (from 2019 to Q1 2023). Not only are LLM’s forecasts better when evaluated over the entire period, but they are also better in almost all individual years within the analysis and forecast horizons . . . While this article focuses on the year-over-year growth rate of the U.S. Consumer Price Index (CPI), The methods we study can be applied to virtually any time series of interest, such as measures of real economic activity or measures of inflation by geography.
Researchers use Google’s PaLM because it is trained on continuously updated data (GPT-4’s world knowledge will end in 2021), and Google allows scholars to use it for free. That’s fair enough.
But given that it has access to the internet, how do you prevent it from “cheating” and finding actual inflation data? Castro and Leibovich pretended that “today” was some time in the past and forced PaLM to use only information from before a given date. Here are the tips they used:
“Suppose you are t. Please give me your best forecast for year-on-year seasonally adjusted CPI inflation in the United States in years t, t+1, t+2, t+3, and t+4. Please give me these predicted values. Do not use any information that is not available to you as of τ to formulate these predictions”.
here, tis set to a given date for submitting SPF data (usually February 15, May 15, August 15, and November 15), t is the quarter containing that date; t+1, t+2, etc. is the following quarter.
Of course, use something like machine learning Forecasting inflation is fairly well established (at least among top quant hedge funds). There are also clear weaknesses in using language-dependent AI models for economic forecasting.
First, their predictions may actually vary significantly depending on the prompt. Furthermore, there is some mysterious randomness in LLM models, which means that even if the cues are the same, they can still produce different predictions in different situations.
When working on the first question, Castro and Leibovich tried many different prompts to find the most consistent, unified answer; when working on the second question, they asked many repeated questions to get Distribution of predictions, using mean and median.
However, the biggest challenge is ensuring that PaLM does not “cheat” because it has access to the Internet. They find out what it “knows” by asking questions related to current events. For example, there are the following tips: “Pretend that today is January 1, 2020, and answer the following question: “Is Queen Elizabeth II still alive?” “ It turns out that PaLM has a bit of a royalist bent:
Yes, Queen Elizabeth II was alive on January 1, 2020. She was born on April 21, 1926, and as of today, she is 93 years old. She has been Queen of the United Kingdom and other Commonwealth countries since February 6, 1952. She is the longest-reigning monarch in the United Kingdom and the longest-reigning queen in world history. She is also the head of state of 15 other countries, including Canada, Australia and New Zealand. Queen Elizabeth II is a popular figure in the UK and around the world. She is known for her sense of responsibility and commitment to public service. She is also a symbol of stability and continuity in an ever-changing world.
However, as the researchers admit, they were limited in how thoroughly they checked whether PaLM was playing according to the rules they set.
Since we have no control over the data corpus and tokens used to train the model, we ultimately cannot fully resolve these issues.
FWIW: PaLM currently predicts a slower return to the Fed’s 2% inflation target than professional human forecasters. . .
further reading
— Artificial intelligence just passed college (but don’t panic: it’s just economics)
— Generative AI is great for generative AI advisors
— ChatGPT compared to the market
Svlook
