ネタ的な内容ですが、UA を grep で引っ掛けて、ユニークにするだけの簡易的な算出。
こうやってみると、結構色々な種類の Bot 様が活動しているのだなぁ。
$ cat /var/log/nginx/xxxxxx-access.log | cut -d " " -f12- | cut -d "\"" -f1 | egrep "bot|Bot|BOT" | sort | uniq -c | sort -nr
443051 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
96301 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
83442 (compatible; Jooblebot/2.0; Windows NT 6.1; WOW64; +http://jooble.org/jooble-bot)
51714 (compatible; Mappy/1.0; Warning:UserAgent will be changed by Feb 2020; +http://mappydata.net/bot/)
41229 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)
38992 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
26122 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)
20704 (compatible; Jooblebot/2.0; Windows NT 6.1; WOW64; +http://jooble.org/jooble-bot) Mobile
19407 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)
15036 (compatible; Mappy/1.0; +http://mappydata.net/bot/)
11181 (+http://www.google.com/adsbot.html)
4807 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)
4629 (compatible; YandexBot/3.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106
4028 (compatible; YandexBot/3.0; +http://yandex.com/bots)
2538 (compatible; Jooblebot/2.0; Windows NT 6.1; WOW64; +http://jooble.org/jooble-bot) AppleWebKit/537.36 (KHTML, like Gecko) Safari/537.36
2101 (compatible; LinkpadBot/2.3; +http://linkpad.org/robot/)
1411 (+http://search.msn.com/msnbot.htm)
1164 (zoominfobot at zoominfo dot com)
783 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0
595 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)
339 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
263 (compatible; YaK/1.0; http://linkfluence.com/; bot@linkfluence.com)
199 (compatible; Onespot-ScraperBot/1.0; +https://www.onespot.com/identifying-traffic.html)
190 (Windows NT 6.1; Win64; x64; +http://www.komodia.com/newwiki/index.php/URL_server_crawler) KomodiaBot/1.0
181 (compatible; Clarabot/1.4; +http://www.clarabot.info/bots)
138 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36
132 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
128 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
91 (compatible; coccocbot-web/1.0; +http://help.coccoc.com/searchengine)
86 (https://domainsbot.com/pandalytics/)
79 (+https://api.slack.com/robots)
78 (compatible; tracemyfile/1.0; +bot@tracemyfile.com)
73 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html)
66 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.74 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
61 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
42 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/78.0.3904.74 Safari/537.36
40 (compatible; YandexImages/3.0; +http://yandex.com/bots)
32 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com)
30 (compatible; oBot/2.3.1; +http://www.xforce-security.com/crawler/)
30 (compatible; coccocbot-image/1.0; +http://help.coccoc.com/searchengine)
27 1.0 (+https://api.slack.com/robots)
24 (compatible; oBot/2.3.1; http://www.xforce-security.com/crawler/)
24 (compatible; MixrankBot; crawler@mixrank.com)
20 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36 (compatible; SMTBot/1.0; http://www.similartech.com/smtbot)
16 (compatible; special_archiver/3.1.1 +http://www.archive.org/details/archive.org_bot)
15 AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview Analytics) Chrome/41.0.2272.118 Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
14 (Linux; Android 5.0; SM-G920A) AppleWebKit (KHTML, like Gecko) Chrome Mobile Safari (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)
10 (compatible; TOBBOT; +http://tobbot.com/)
10 (compatible; Cliqzbot/3.0; +http://cliqz.com/company/cliqzbot)
9 (+https://awario.com/bots.html; bots@awario.com)
9 (+http://www.google.com/bot.html)
8 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.120 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
7 adbeat_bot
6 (compatible) SemanticScholarBot (+https://www.semanticscholar.org/crawler)
5 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
5 (Linux; Android 7.0; CUBOT X18) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.116 Mobile Safari/537.36
4 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot)
4 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/)
3 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/79.0.3945.120 Safari/537.36
3 (https://turnitin.com/robot/crawlerinfo.html)
3 (compatible; SemrushBot/6~bl; +http://www.semrush.com/bot.html)
3 (compatible; SemrushBot-BM/1.0; +http://www.semrush.com/bot.html)
3 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com)
2 (compatible; Qwantify/Bleriot/1.1; +https://help.qwant.com/bot)
2 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)
1 Bot
1 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
1 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html; Linux; Android 4; iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A523 Safari/8536.25
1 (compatible; SemrushBot/1.0~bm; +http://www.semrush.com/bot.html)
1 (compatible; SOLOFIELD/1.0 +http://solofield.net/bot.html)
1 (compatible; PlurkBot/1.0; +https://www.plurk.com/) Firefox/61.0
1 (advanced backlink tracking bot; curl/7.58.0; http://serpstatbot.com/; abuse@serpstatbot.com)
