Find div aria label starting with certain text and then extract

ghz 14hours ago ⋅ 4 views

I'm trying to locate the button element "Follow" because it contains the text of the user that you're going to follow.

Element:

<div aria-label="Follow @Visiongeo" role="button" tabindex="0" class="css-18t94o4 css-1dbjc4n r-42olwf r-sdzlij r-1phboty r-rs99b7 r-15ysp7h r-4wgw6l r-1ny4l3l r-ymttw5 r-o7ynqc r-6416eg r-lrvibr" data-testid="1676648952-follow" style="background-color: rgb(239, 243, 244);"><div dir="auto" class="css-901oao r-1awozwy r-6koalj r-18u37iz r-16y2uox r-37j5jr r-a023e6 r-b88u0q r-1777fci r-rjixqe r-bcqeeo r-q4m81j r-qvutc0" style="color: rgb(15, 20, 25);"><span class="css-901oao css-16my406 css-bfa6kz r-poiln3 r-1b43r93 r-1cwl3u0 r-bcqeeo r-qvutc0"><span class="css-901oao css-16my406 r-poiln3 r-bcqeeo r-qvutc0">Follow</span></span></div></div>

But the thing is, the text after Follow, which is @Visiongeo changes, so I don't want to particularly locate the element with specific text.

Instead, I want to locate this element, and then get the text that starts with "@" and write it to a text file.

My code:

for followers in wait.until(EC.element_to_be_clickable((By.XPATH, "//div[starts-with(@aria-label, 'Follow')]"))):
    file = open("testfile.txt", "a")
    file.write(followers)
    file.write("\n")
    file.close()

I get TimeoutException message when I use the above code. I think I'm way off from what I intend to do.

Answer

You’re on the right track, but the code has a few issues:

  1. The TimeoutException error usually occurs because the XPath expression is not finding the element, or the element isn't clickable in the expected time frame. You might want to verify that the element is present and correctly identified by the XPath.

  2. The code logic to extract the text: When you extract the followers from the XPath expression, you're grabbing the entire WebElement object. You need to access the text or attributes of the element, not the element itself.

  3. Writing to a file: It’s inefficient to open the file and close it on every iteration. Instead, you can open the file once, write all the followers, and then close the file.

Let’s address the issues step by step.

Solution

  1. XPath Issue: Your XPath expression should find the div element where the aria-label starts with "Follow". However, you need to extract the text that starts with "@" (e.g., @Visiongeo). The aria-label contains the full string "Follow @Visiongeo", and you can extract the username part by splitting it or using regex.

  2. Extracting the username: Use .text to get the inner text of the span element that contains the username.

  3. Improving file handling: Open the file once and use the write method inside the loop. This prevents opening and closing the file repeatedly.

Here’s how you can modify your code:

import re
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# Initialize your WebDriver (e.g., ChromeDriver)
driver = webdriver.Chrome()
driver.get('YOUR_URL_HERE')  # Replace with your URL

# Wait for the elements to be clickable
wait = WebDriverWait(driver, 10)

# Open the file in append mode before the loop
with open("testfile.txt", "a") as file:
    try:
        # Locate all "Follow" buttons
        followers_elements = wait.until(EC.presence_of_all_elements_located(
            (By.XPATH, "//div[starts-with(@aria-label, 'Follow')]")
        ))

        for element in followers_elements:
            # Extract the aria-label attribute and match the username using regex
            aria_label = element.get_attribute('aria-label')
            match = re.search(r"@([A-Za-z0-9_]+)", aria_label)

            if match:
                username = match.group(0)  # This will return "@Visiongeo" or any other username
                file.write(username + "\n")
    
    except Exception as e:
        print("Error:", e)

# Don't forget to close the driver at the end of the script
driver.quit()

Key Changes and Explanation:

  1. XPath for Multiple Elements:

    • I replaced EC.element_to_be_clickable() with EC.presence_of_all_elements_located() since you're working with multiple elements (Follow buttons). You want to locate all of them, not just one.

  2. Regular Expression (re.search):

    • The aria-label attribute contains the full string "Follow @Visiongeo", so I used a regular expression (r"@([A-Za-z0-9_]+)") to extract the username that starts with @.

  3. File Handling:

    • Open the file once before the loop (with open(...)), which will automatically close the file once the block finishes executing. This ensures that the file is not opened and closed repeatedly, which is inefficient.

  4. Error Handling:

    • A try-except block catches any exceptions, so you can debug or handle errors gracefully (e.g., if an element isn't found).

Output:

The script will write usernames (e.g., @Visiongeo) into the testfile.txt file, one per line.

This should work without the TimeoutException, assuming the Follow buttons are present on the page and are clickable. If the element is not clickable in the specified wait time, you might need to adjust the wait time or check if the page content is fully loaded.

#py