近日オープン
Google Sitemaps を利用しているページとそうでないページで、クロールやインデックスに違いがあるのか実験する。
現状では、Google Sitemaps を利用しているページの方がインデックスされやすいという感触である。
Site Map に全ページを記載しても、全てがインデックスされるとは限らないということは、他のサイトでの実験で明らかである。Site Map の限界を知りたいということもあるが、コンテンツを手打ちしている都合上、いつになったら結果がでるかは見当もつかない。インデックスされるためには、個性的なコンテンツとコンテンツの量が必要であろうが、取り合えず、これぐらいの文章量から始める。
2007/07/09 実験開始
2007/07/14,08:24:02,crawl-66-249-73-197.googlebot.com,"","-",GET,"/smstest4.html","","1.1",200,885,"-","","Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2007/07/14,15:02:45,lj511356.crawl.yahoo.net,"","-",GET,"/smstest4.html","","1.0",200,1388,"-","","Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
2007/07/21,17:47:06,lj511356.crawl.yahoo.net,"","-",GET,"/smstest4.html","","1.0",200,1772,"-","","Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
2007/08/04,07:49:54,crawl-66-249-73-197.googlebot.com,"","-",GET,"/smstest4.html","","1.1",200,1103,"-","","Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2007/08/06,07:21:44,crawl-66-249-73-197.googlebot.com,"","-",GET,"/smstest4.html","","1.1",200,1102,"-","","Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2007/08/09,12:32:36,crawl-66-249-73-197.googlebot.com,"","-",GET,"/smstest4.html","","1.1",200,1140,"-","","Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2,3日に1回のペースで来てます。
2007/08/10 10:30 Googleのインデックスを確認(2007年7月13日 23:24:02 GMT)。"site:"で確認。"inurl:"では表示されないので、例の”補足結果”相当か。
なお、Yahooにはインデックスされていない。
2007/08/11,18:30:53,lj511356.crawl.yahoo.net,"","-",GET,"/smstest4.html","","1.0",200,2760,"-","","Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
3週間ぶりのアクセス。
2007/08/13,07:19:15,crawl-66-249-73-197.googlebot.com,"","-",GET,"/smstest4.html","","1.1",200,1295,"-","","Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2007/08/14,12:28:17,crawl-66-249-73-197.googlebot.com,"","-",GET,"/smstest4.html","","1.1",200,1295,"-","","Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2007/08/17,00:49:52,crawl-66-249-73-197.googlebot.com,"","-",GET,"/smstest4.html","","1.1",200,1293,"-","","Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2007/08/18,07:16:37,crawl-66-249-73-197.googlebot.com,"","-",GET,"/smstest4.html","","1.1",200,1364,"-","","Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2007/08/18,18:32:39,lj511356.crawl.yahoo.net,"","-",GET,"/smstest4.html","","1.0",200,3354,"-","","Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
1週間ぶりのアクセス。
2007/08/19,08:35:14,crawl-66-249-73-197.googlebot.com,"","-",GET,"/smstest4.html","","1.1",200,1364,"-","","Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
いいペースになってきました。
2007/08/21,07:15:01,crawl-66-249-73-197.googlebot.com,"","-",GET,"/smstest4.html","","1.1",200,1454,"-","","Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
このままコンスタントに来てくれればいいのですが。
2007/08/22 22:20 Yahooでインデックスを確認。キャッシュは2007年8月11日。どうやら更新があったようなので、その影響か。
2007/08/24,07:17:41,crawl-66-249-73-197.googlebot.com,"","-",GET,"/smstest4.html","","1.1",200,1574,"-","","Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2007/08/25,07:25:35,crawl-66-249-73-197.googlebot.com,"","-",GET,"/smstest4.html","","1.1",200,1576,"-","","Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2007/08/25,18:36:01,lj511356.crawl.yahoo.net,"","-",GET,"/smstest4.html","","1.0",200,4549,"-","","Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
2007/08/28,08:56:02,crawl-66-249-73-197.googlebot.com,"","-",GET,"/smstest4.html","","1.1",200,1575,"-","","Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2007/08/28 10:07 Yahooでインデックスをチェック。”site:”では検索されないが、”http://yourmall.../smstest4.html”で検索するとヒットする。キャッシュは2007年7月22日。
2007/08/30 10:31 Yahooでインデックスをチェック。”site:”では検索するとヒット。キャッシュは2007年8月25日。
2007/08/30,08:13:50,crawl-66-249-73-197.googlebot.com,"","-",GET,"/smstest4.html","","1.1",200,1715,"-","","Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
未だに inurl: ではヒットしない。
2007/09/01,20:37:24,lj511356.crawl.yahoo.net,"","-",GET,"/smstest4.html","","1.0",200,5739,"-","","Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
2007/09/03,07:20:53,crawl-66-249-73-197.googlebot.com,"","-",GET,"/smstest4.html","","1.1",200,1760,"-","","Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2007/09/05,01:47:03,crawl-66-249-73-197.googlebot.com,"","-",GET,"/smstest4.html","","1.1",200,1792,"-","","Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2007/09/06,07:44:40,crawl-66-249-73-197.googlebot.com,"","-",GET,"/smstest4.html","","1.1",200,1791,"-","","Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2007/09/06 11:14 Googleにて。"site:"では確認できるが、"inurl:"では表示されない状態のまま。キャッシュの日付は 2007年8月27日 23:56:02 GMT なので 2007/08/28,08:56:02 のクロール時のもの。
2007/09/08,20:43:03,lj511356.crawl.yahoo.net,"","-",GET,"/smstest4.html","","1.0",200,6780,"-","","Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
2007/09/09,07:25:16,crawl-66-249-73-197.googlebot.com,"","-",GET,"/smstest4.html","","1.1",200,1889,"-","","Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2007/09/10,08:13:57,crawl-66-249-73-197.googlebot.com,"","-",GET,"/smstest4.html","","1.1",200,1898,"-","","Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2007/09/12,07:31:49,crawl-66-249-73-197.googlebot.com,"","-",GET,"/smstest4.html","","1.1",200,1947,"-","","Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2007/09/14,07:28:42,crawl-66-249-73-197.googlebot.com,"","-",GET,"/smstest4.html","","1.1",200,1955,"-","","Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2007/09/16,00:53:45,lj511356.crawl.yahoo.net,"","-",GET,"/smstest4.html","","1.0",200,7536,"-","","Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
2007/09/16,07:25:40,crawl-66-249-73-197.googlebot.com,"","-",GET,"/smstest4.html","","1.1",200,1954,"-","","Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2007/09/18,07:21:27,crawl-66-249-73-197.googlebot.com,"","-",GET,"/smstest4.html","","1.1",200,1955,"-","","Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2007/09/19 19:42 Googleにて、site:yourmall.fitss.com で検索。7件ヒットするが、当ページはヒットしない。"inurl:"では6件ヒットで当ページはヒットしない。しかし、"cashe:yourmall.fitss.com/smstest4.html"ではキャッシュを確認できる(2007年9月4日 16:47:03 GMT - Wednesday, 05-Sep-2007 01:47:03 JST)。
2007/09/19 19:54 実験から2ヶ月と10日になる。Google Sitemapsに記載しないページと比較すると、Googleのクローラーのアクセス頻度は断然高い傾向にある。ここで、他のページ(ysttest1.html)もGoogle Sitemapsに記載し、その経過を観察する。
2007/10/16,03:43:45,lj511356.crawl.yahoo.net,"","-",GET,"/smstest4.html","","1.0",200,8835,"-","","Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
2007/10/23,03:45:53,lj511356.crawl.yahoo.net,"","-",GET,"/smstest4.html","","1.0",200,9026,"-","","Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
2007/10/25,18:39:57,lj511356.crawl.yahoo.net,"","-",GET,"/smstest4.html","","1.0",200,9220,"-","","Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
2007/10/26,17:45:26,crawl-66-249-73-197.googlebot.com,"","-",GET,"/smstest4.html","","1.1",200,2275,"-","","Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2007/11/01,20:50:10,lj511356.crawl.yahoo.net,"","-",GET,"/smstest4.html","","1.0",200,9601,"-","","Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
Monday, 02-Feb-2026 21:11:36 JST
©2005 YourMall