Auto-complete search terms

by RS admin@robinsnyder.com : 1024 x 640

The Google API (and others) can be used to get suggestions to auto-complete a typed query (in progress).

It is intended for JavaScript usage from a web page but can be automated using Python.

Try the following URL in your browser.

http://suggestqueries.google.com/complete/search?client=firefox&q=corovirus

A browser such as Chrome or Firefox will allow the text to be saved as text - but it is a JSON list.

Here is the returned JSON text, formatted to be easier to read.

["corovirus", [ "coronavirus", "coronavirus map", "coronavirus us", "coronavirus in usa", "coronavirus update", "coronavirus tips", "coronavirus vaccine", "coronavirus china", "coronavirus symptoms", "coronavirus italy" ] ]

This manual operation can be automated using, say, Python.

The code shows how a date and time stamp might be created in order to save the results on a periodic basis to a file in a given folder.

Here is the Python code [#1]

import requests
import json
import datetime
import time
dashDateFormat1 = "%Y-%m-%d"
dashTimeFormat1 = "%H-%M"
now1 = datetime.datetime.now()
dashDate1 = now1.strftime(dashDateFormat1)
dashTime1 = now1.strftime(dashTimeFormat1)
timeStamp1 = "{0:s}-{1:s}".format(dashDate1,dashTime1)
termList1 = [
	"coranovirus",
	"covid-19",
	"pandemic",
	]
headersDict1 = {
	"User-agent" : "Mozilla/5.0",
	}
urlBase1 = "http://suggestqueries.google.com/complete/search?client=firefox&q="
print("Suggestions: {0:s}".format(timeStamp1))
for pos1,term1 in enumerate(termList1):
	url1 = "{0:s}{1:s}".format(urlBase1,term1)
	print("")
	print("{0:d}. {1:s} (term)".format(pos1+1,term1))
	print("")
	response1 = requests.get(url1, headers=headersDict1)
	resultList1 = json.loads(response1.content.decode("utf-8"))
	suggestList2 = resultList1[1]
	for pos2,suggest2 in enumerate(suggestList2):
		print("\t{0:d}. \"{1:s}\"".format(pos2+1,suggest2))

Here is the output of the Python code.

Suggestions: 2021-03-22-09-54
1. coranovirus (term)
	1. "coronavirus"
	2. "coronavirus symptoms"
	3. "coronavirus pa"
	4. "coronavirus update"
	5. "coronavirus update pennsylvania"
	6. "coronavirus update lancaster"
	7. "coronavirus vaccine"
	8. "coronavirus us"
	9. "coronavirus cases"
	10. "coronavirus stats"
2. covid-19 (term)
	1. "covid-19"
	2. "covid-19 vaccine"
	3. "covid-19 symptoms"
	4. "covid-19 vaccine near me"
	5. "covid-19 vaccine pennsylvania"
	6. "covid-19 vaccine lancaster"
	7. "covid-19 usa"
	8. "covid-19 cases"
	9. "covid-19 vaccine side effects"
	10. "covid-19 florida"
3. pandemic (term)
	1. "pandemic unemployment"
	2. "pandemic"
	3. "pandemic unemployment assistance"
	4. "pandemic unemployment assistance pa"
	5. "pandemic unemployment pa"
	6. "pandemic definition"
	7. "pandemic unemployment assistance (pua)"
	8. "pandemic ebt pa"
	9. "pandemic emergency unemployment compensation"
	10. "pandemic unemployment portal"

In the result, the first element in the list is the search term provided. The second item is a list of suggestions which, of course, will always start with the search term provided since the suggestions are intended to help complete what is typed for a search (of the Google site).

One way to save results as JSON (or other format) is as follows.

Create a date and time stamp such as 2020-04-01-12 (assuming once per hour, suffix 00 to 23) of the general form yyyy-mm-dd-hh to be used as a file name.
If all the files are in one folder, there is no need for a file name prefix to distinguish files.

A few years ago, I set up a Raspberry Pi to periodically get the suggested search term completion suggestions for a set of search terms. The idea was to wake up every few hours (say 1 to 24 times a day, using a cron job), and check the suggestions.

The analysis then was to determine, over time, what search terms and orderings had changed.

by RS admin@robinsnyder.com : 1024 x 640