How to find price from udemy website with web scraping?

ghz 13hours ago ⋅ 5 views

I am using beautiful soup package of python to find the price of courses. Using beautiful soup I am getting price in dollars, and when I convert it in rupees it is different.

price in udemy website : 700
price by beautiful soup : 13.99$

I have tried to find logic by calculating different courses ratio but it did not work. Here is my code :

from bs4 import BeautifulSoup
import requests
page = requests.get('https://www.udemy.com/course/python-data-science-machine-learning-bootcamp/')
soup = BeautifulSoup(page.content, 'html.parser')
for sp in soup.find_all('span',class_='price-text__current'):
   print(sp)

I am getting this output :

<span class="price-text__current" data-purpose="discount-price-text">
<span class="sr-only">Current price:</span> $13.99
</span>
</span>

Answer

It looks like the price you are getting from the website ($13.99) is the discounted price, whereas the original price you mentioned (700) might be in a different currency (likely Indian Rupees, INR). This is common on websites like Udemy, where courses are listed with a discounted price in USD, but the original price might be displayed in another currency (INR, in this case).

Issues to Consider:

  1. Currency Discrepancy: The price you are scraping is in USD ($13.99), but you are expecting it to be in INR (₹700). This discrepancy is likely due to the different currencies being used for the original and discounted prices.

  2. Exchange Rate: If you want to convert the scraped USD price to INR, you need to account for the current exchange rate. Exchange rates fluctuate, and scraping data without converting it will result in incorrect comparisons.

Steps to Correct the Issue:

  1. Check for Original Price: Try to check if the original price is also available on the page, and if so, scrape that as well.

  2. Use an Exchange Rate API: To convert USD to INR, you'll need a current exchange rate. You can use an API like CurrencyLayer, Open Exchange Rates, or Forex-python.

Suggested Solution:

  1. Scrape the Price: First, make sure you are scraping both the discounted and the original price (if available).

  2. Convert USD to INR using a currency converter API.

Here's a revised version of your code to handle this:

from bs4 import BeautifulSoup
import requests

# Fetch the page content
page = requests.get('https://www.udemy.com/course/python-data-science-machine-learning-bootcamp/')
soup = BeautifulSoup(page.content, 'html.parser')

# Find the discounted price
discounted_price = None
original_price = None

# Find the discounted price in USD
for sp in soup.find_all('span', class_='price-text__current'):
    discounted_price = sp.get_text(strip=True)
    print("Discounted price:", discounted_price)

# Find the original price (if available)
for sp in soup.find_all('span', class_='price-text__original'):
    original_price = sp.get_text(strip=True)
    print("Original price:", original_price)

# If no original price is found, fallback to some logic
if not original_price:
    print("Original price not found, assuming it is in INR.")

# Now, let's convert the discounted price from USD to INR.
if discounted_price:
    discounted_price = float(discounted_price.replace('$', '').strip())

    # Using a currency exchange rate API (for example, Forex-python library or any API).
    from forex_python.converter import CurrencyRates
    cr = CurrencyRates()

    # Get the exchange rate from USD to INR
    exchange_rate = cr.get_rate('USD', 'INR')

    # Convert the price from USD to INR
    price_in_inr = discounted_price * exchange_rate
    print(f"Price in INR: {price_in_inr:.2f}")

Explanation of the Code:

  1. Scraping Prices: The code first tries to scrape the discounted price and, if available, the original price. The original price might be displayed in a different element with the class price-text__original or similar.

  2. Currency Conversion: The forex-python library is used to get the current exchange rate for USD to INR and convert the scraped price from USD to INR.

Install the Required Library:

To use the forex-python library for currency conversion, install it using pip:

pip install forex-python

Output:

  • You'll get the price in INR after converting it from USD.
  • If the original price is also available on the page, you can compare the two (discounted price vs. original price).

Notes:

  1. Price Variability: Be aware that prices on websites like Udemy may fluctuate based on the region, discounts, or even time of day. Ensure that your scraping logic can handle such changes.

  2. Rate Limits: Be mindful of rate limits on currency conversion APIs. You may need an API key or may be limited to a certain number of requests per minute or day depending on the service.

  3. Error Handling: It’s a good practice to add error handling for situations where the price is not available or the API request fails.

#py