'Render column data lists as table with pandas

I need to scrape a website which has a 'table' like paragraph and I want to put it as a DataFrame on python.

I need to get the Name, Price and the description of the page and put it all in a DataFrame format. The problem is that I can scrape all of it individually, but I can't get them to a proper DataFrame.

Here is what I have done so far:

I get the product links first because I need to scrape multiple pages:
baseURL = 'https://www.civivi.com'
product_links = []
for x in range (1,3):
    HTML = requests.get(f'https://www.civivi.com/collections/all-products/price-range_-70?page={x}',HEADER)
    #HTML.status_code
    Booti= soup(HTML.content, "lxml")
    knife_items = Booti.find_all('div',class_= "product-list product-list--collection product-list--with-sidebar")
    
    for items in knife_items:
        for links in items.findAll('a', attrs = {'class' : 'product-item__image-wrapper product-item__image-wrapper--with-secondary'}, href = True):
            product_links.append(baseURL + links['href'])

And then I scrape the individual web pages here:

Name = []
Price = []
Specific = []
for links in product_links:
#testlinks = "https://www.civivi.com/collections/all-products/products/civivi-rustic-gent-lockback-knife-c914-g10-d2"
    HTML2 = requests.get(links, HEADER)
    Booti2 = soup(HTML2.content,"html.parser") 
    try:
        for N in Booti2.findAll('h1',{'class': "product-meta__title heading h1" }):
            Name.append(N.text.replace('\n', '').strip())
        for P in Booti2.findAll('span',{'class': "price" }):
            Price.append(P.text.replace('\n', '').strip())
        Contents = Booti2.find('div',class_= "rte text--pull")
        for S in Contents.find_all('span'):
            Specific.append(S.text)

    except:
        continue 

So I need to get all the information in this format:

         Name.     | | Price          || Model Number  Model Name. Overall Length
|------------------| |----------------||-------------| ---------||----------------|
| Product Name 1   | |  $$            ||  XXXX       |  ABC.    ||   XX"/XXcm.    |  
| Product Name 2   | |  $$            ||  XXXX       |  ABC.    ||   XX"/XXcm.    |
| Product Name 3   | |  $$            ||  XXXX       |  ABC.    ||   XX"/XXcm.    | 
| Product Name 4   | |  $$            ||  XXXX       |  ABC.    ||   XX"/XXcm.    |

...and so on with rest of the columns from the web pages. Any help would be appreciated!! Thank you so much!!



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source