Google Trends is a great tool for checking relative search volumes trends for keywords or topics, and to compare them with each other. The tool allows you to download the main trend graph points in a CSV file.
Some time ago I was practicing calculating linear regression with Excel on Google Trends sets of 2-year data. I chose a time-range of 2 years to account for possible seasonality. I wondered if I could automate this process with a script and output a single metric (the slope value), as an indication of the search trend of the past 2 years. A negative number would mean that searches are decreasing, while a positive number means that the trend is positive.
After a few searches I became aware of Scipy, a Python open-source library for scientific and technical computing that includes mathematical functions, including regressions.
I started this project by analyzing how Google Trends constructs its web requests in the browser with the Google Chrome “Network” tool, and then I transferred the learnings in a Scrapy script (my favorite scraping library).
The result is the script shared with you at the end of this article; it can be called server-side with the following command:
scrapy runspider -L WARNING filename.pyIn order for the script to work, both Scrapy and Scipy libraries should be installed on your server, and imported at the beginning of the script, along with the “datetime” and “json” modules. The code is ready to be used, just make sure to substitute the “keyword” value with the term you want to query trends for. The default data points range is set in weeks, and the slope is calculated by pairing the numbers from 1 to 104 (the numbers of weeks) with the value for each week extracted.
I hope you will make good use of the script for your purposes, and please don’t hesitate to ask me if you have any doubts about its functions.
import scrapy
import json
from datetime import date, timedelta
from scipy.stats import linregress
# Calculate the current date and the same day of 2 years before, in the format YYYY-MM-DD
today = str(date.today())
mo = date.today().strftime('%m')
da = date.today().strftime('%d')
ye = date.today().year
old = str(ye-2)+'-'+mo+'-'+da
today = str(ye)+'-'+mo+'-'+da
# We will query 2 years of data, therefore 104 weeks
weeks = list(range(1,105))
# Customize this with the keyword you want to query trends for
keyword = 'bitcoin'
# URLs constructors
url1 = 'https://trends.google.com/trends/api/explore?hl=en-US&tz=-60&req={"comparisonItem":[{"keyword":"'
url2 = '","geo":"IE","time":"'+old+' '+today+'"}],"category":0,"property":""}&tz=-60'
url3 = ('https://trends.google.com/trends/api/widgetdata/multiline?hl=en-US&tz=-60&req={"time":"'+old+' '+today+
'","resolution":"WEEK","locale":"en-US","comparisonItem":[{"geo":{"country":"IE"},"complexKeywordsRestriction":'+
'{"keyword":[{"type":"BROAD","value":"')
url4 = '"}]}}],"requestOptions":{"property":"","backend":"IZG","category":0}}&token='
url5 = '&tz=-60'
# Scrapy scraper
class TrendsSpider(scrapy.Spider):
name = "trends2"
handle_httpstatus_list = [429, 500]
def start_requests(self):
# First we crawl the Google Trends root domain, in order to receive a valid session cookie
url = 'https://trends.google.com/'
yield scrapy.Request(url, callback=self.parse,headers={'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.79 Safari/537.36'})
def parse(self, response):
# The request that will return a JSON file with the tokens for each widget
newurl = url1+keyword+url2
return scrapy.Request(newurl,callback=self.parse_1)
def parse_1(self, response):
# We have to delete the first 4 characters of the response as not relevant
jsonresponse = json.loads(response.text[5:])
# We parse the JSON response, capture the token for the "Interest over time" widget and construct the new URL with it
kwd = jsonresponse['widgets'][0]['request']['comparisonItem'][0]['complexKeywordsRestriction']['keyword'][0]['value']
tok= jsonresponse['widgets'][0]['token']
finalurl = (url3 + kwd + url4 + tok + url5)
# We pass the keyword to the next parser function
return scrapy.Request(finalurl,callback=self.parse_2,meta={'keyword':kwd})
def parse_2(self, response):
jsonresponse2 = json.loads(response.text[6:])
# We insert the weekly values in an array
values = []
for i in jsonresponse2['default']['timelineData']:
values.extend(i['value'])
if len(values)==0:
slope = 0
else:
# The following function returns the slope of the calculated linear regression by combining the weeks numbers (1-104) with the values
slope = "%.2f" % linregress(weeks, values).slope
# Print the keyword with the slope value
print(response.meta['keyword']+': '+slope)