How to crawl question and answer of Google People Also Ask with Selenium and Python?

Question

I used this code for crawl question and anwser of Google People Also Ask. I want use that for create idea for writer.

But I can't get exactly that element, in this time I was try with full Xpath, but with some keyword it will change Full Xpath and it will have error.

Variable order is quality of question and number I need crawl.

def get_question(order):
    try:
        question = driver.find_element(By.XPATH, '/html/body/div[7]/div/div[9]/div[1]/div/div[2]/div[2]/div/div/div[2]/div/div/div[1]/div[' + str(order + 1) + ']/div[2]/div/div[1]/div[2]/span').text
    except:
        question = ''
        print('Xpath question wrong')
    return question

def get_answer(order):
    try:
        answer = driver.find_element(By.XPATH, '/html/body/div[7]/div/div[9]/div[1]/div/div[2]/div[2]/div/div/div[2]/div/div/div[1]/div[' + str(order + 1) + ']/div[2]').get_attribute('outerHTML')
    except:
        answer = ''
        print('Xpath answer wrong')

And after that I will use loop for crawl one by one question and answer, like this:

if __name__ == '__main__':
    quality_question = 5
    order = 0
    while order <= quality_question:
        question = get_question(order)
        answer = get_answer(order)

Have any idea to get exactly question and answer in all cases. Full Xpath will change when structure of result different. Thanks for your support. Have any idea to do that?

This is URL Google I need crawl you can try with search query "How to make bakery"

Link

Answer 1

You should first construct an xpath

//span[text()='People also ask']/../following-sibling::div/descendant::div[@data-hveid and @class and @jsname and @data-ved]

to just figure out how many questions are present.

Once you do that, the next job is to click on it and then retrieve the answer.

Code:

driver = webdriver.Chrome(driver_path)
driver.maximize_window()
driver.implicitly_wait(30)
wait = WebDriverWait(driver, 30)

driver.get("https://www.google.com/search?q=How%20to%20make%20bakery%3F&source=hp&ei=j0aZYYjRAvja2roPrcWcyAU&iflsig=ALs-wAMAAAAAYZlUn4NMUPjfIpQmrXSmjIDnaWjJXWIJ&ved=0ahUKEwjI1JDn0Kf0AhV4rVYBHa0iB1kQ4dUDCAc&uact=5&oq=How%20to%20make%20bakery%3F&gs_lcp=Cgdnd3Mtd2l6EAMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBNQAFgAYJMDaABwAHgAgAF-iAF-kgEDMC4xmAEAoAECoAEB&sclient=gws-wiz")

all_questions = driver.find_elements(By.XPATH, "//span[text()='People also ask']/../following-sibling::div/descendant::div[@data-hveid and @class and @jsname and @data-ved]")
print(len(all_questions))

j = 1
for question in all_questions:
    time.sleep(1)
    ele = driver.find_element(By.XPATH, f"(//span[text()='People also ask']/../following-sibling::div/descendant::div[@data-hveid and @class and @jsname and @data-ved])[{j}]")
    j = j + 2
    ele.click()
    time.sleep(1)
    answer = ele.find_element(By.XPATH, ".//../following-sibling::div").get_attribute('innerText')
    print(answer)
    print('--------------')

Imports:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Output:

6
The most profitable bakeries have a gross profit margin of 9%, while the average is much lower at 4%. The growth of profitable bakeries can be as high as 20% year over year. While a large number of bakeries never reach the break-even, a handful of them can even have a net profit margin as high as 12%.06-Jul-2020

How Much Do Bakery Owners Make? | Restaurant Accounting
https://restaurantaccounting.net › how-much-do-bakery-o...
Search for: Do bakeries make money?
--------------
Home Bakery Business - How to Start
Decide on the goods to bake. ...
Plan your kitchen space. ...
Get a permit. ...
Talk to a tax agent. ...
Set appropriate prices. ...
Start baking and selling.
16-Feb-2015

Home Bakery Business - How to Start | BakeCalc
http://www.bakecalc.com › blog › how-to-start-a-home-b...
Search for: How do I start a small baking business from home?
--------------
Follow the below-mentioned steps to open a successful bakery business in India in 2021:
Create A Bakery Business Plan. ...
Choose A Location For Your Bakery Business. ...
Get All Licenses Required To Open A Bakery Business In India. ...
Get Manpower Required To Open A Bakery. ...
Buy Equipment Needed To Start A Bakery Business.
More items...

A Detailed Guide On How To Start A Bakery Business In India
https://www.posist.com › Home › Resources
Search for: How do I start my own bakery?
--------------
Baking is a profitable business. ... And so long as you exercise good business practices and maintain the quality of your products, the bakery is sure to give you a good return. Like all business ventures, however, a bakery business requires that you prepare well for it.11-Nov-2015

6 key ingredients to start a bakery business
https://business.inquirer.net › 6-key-ingredients-to-start-a-...
Search for: Is bakery a good business?
--------------
Whatever your reason, investing in a small bakery can be a benefit for a community and a boon for your wallet. Bakeries are booming and if you can get in on the ground floor of a good one, the opportunity can be very profitable.

Investing in a Small Bakery Business
https://smallbusiness.chron.com › investing-small-bakery-...
Search for: Is a bakery a good investment?
--------------
When respondents were asked what are the top bakery items they produce, cookies rank first at 89 percent, followed by cakes at 79 percent, cupcakes 73 percent, muffins/scones 68 percent, cinnamon rolls 65 percent, and bread 57 percent.06-Sep-2017

The Most Profitable Products at Bakeries - Bake Magazine
https://www.bakemag.com › articles › 5668-the-most-prof...
Search for: What are the most popular bakery items?
--------------

Process finished with exit code 0