BS4 Iterate over table elements nextsibling or parent?

by wiedhas   Last Updated May 16, 2019 00:26 AM

I am trying to pull all of the text and links by date in a table and so far can only get one entry (but not correctly as the link is not named correctly). I think nextsibling might work here but perhaps that's not the right solution.

Here's the html:

<ul class="indented">

                        <strong>May 15, 2019</strong>
             Sign up for more insight into FERC with our monthly news email, The FERC insight
<a href="/media/insight.asp">Read More</a> 


                       <strong>May 15, 2019</strong>
            FERC To Convene a Technical Conference regarding Columbia Gas Transmission, LLC on July 10, 2019
<a href="/CalendarFiles/20190515104556-RP19-763-000%20TC.pdf">Notice</a> <img src="/images/icon_pdf.gif" alt="PDF">  

| <a href="/EventCalendar/EventDetails.aspx?ID=13414&amp;CalType=%20&amp;CalendarID=116&amp;Date=07/10/2019&amp;View=Listview">Event Details</a>   


Here's my code:

import requests
from bs4 import BeautifulSoup

url1 = ('')
r = requests.get(url1)
# Create a BeautifulSoup object
soup = BeautifulSoup(r.content, 'lxml')

# Pull headline  text from the ul class indented

headlines = soup.find_all("ul", class_="indented")
headline = headlines[0]

date  = headline.select_one('strong').text.strip()


headline_text = headline.select_one('ul').text.strip()

headline_link = headline.select_one('ul a')["href"]
headline_link = '' + headline_link 

I get the first date, text and link because I'm using select_one. I need to get all of the links and name them properly for each date. Would findnext work here or findnextsibling?

Answers 1

I believe this is what you are looking for; it gets the date, announcement and related links:

[start same as your code; thru soup declaration]

dates = soup.find_all("strong")
for date in dates:
    if '2019' in date.text:        
        for ref in date.nextSibling.nextSibling.find_all('a'):
            new_link = "" + ref['href']

Random part of the output:

May 15, 2019

            FERC To Convene a Technical Conference regarding Columbia Gas Transmission, LLC on July 10, 2019

| Event Details
Jack Fleeting
Jack Fleeting
May 16, 2019 00:25 AM

Related Questions

Updated February 19, 2017 23:26 PM

Updated February 22, 2018 19:26 PM

Updated February 24, 2017 18:26 PM

Updated June 07, 2017 00:26 AM

Updated July 29, 2017 12:26 PM