'Why is web scraping stock prices through beautiful soup returning a different price than the one on the Yahoo Finance page?

I am trying to write a program that will give me the stock price for a few different stocks, but when I run my program, it returns 116.71, while Yahoo Finance has it on the page and in the HTML as 117.96 (at the time of writing this). Any idea of what is going on? Page is here. Code is below:

from bs4 import BeautifulSoup
import requests


url = 'https://finance.yahoo.com/quote/VTSAX?p=VTSAX&.tsrc=fin-srch'
page = requests.get(url)
soup = BeautifulSoup(page.text, 'html.parser')
price = soup.find('fin-streamer', {'class': 'Fw(b) Fz(36px) Mb(-4px) D(ib)'}).text

print(price)


Solution 1:[1]

I think Yahoo send you different data because they figure out that your request is an automatic one.

So, you should pass 'real' User-Agent in headers:

from bs4 import BeautifulSoup
import requests


url = 'https://finance.yahoo.com/quote/VTSAX?p=VTSAX&.tsrc=fin-srch'
page = requests.get(url, headers={
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36",
})
soup = BeautifulSoup(page.text, 'html.parser')
price = soup.find('fin-streamer', {'class': 'Fw(b) Fz(36px) Mb(-4px) D(ib)'}).text

print(price)

Output

117.96

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Eugenij