by the formula of spider page retention time = spider total residence time / total amount to grab, know the page loading speed and server response speed about what the situation is not to optimize the web page code and optimize the hardware facilities. We all know that the page loading speed but also on the website rankings.
by the formula spider to grab every page = total number to volume / spiders, which know spiders crawl depth, is not the website structure is too deep, grasping depth is not high, will cause a lot of deep level pages will not be captured; and resulting in a small number of pages are repeatedly in the crawl, waste the crawler time. In this way, many websites to get included became a problem in the fall in love with the sea.
also tell you a secret, some spiders to visit the web site is not User-agent, like Kangxi is incognito, what is the purpose of doing so, you want to do, but how to judge the IP is the spider, you can know what is the tracertIP address of a spider.
can access the web log records of visitors and spiders very good, through the web log can be a very good understanding of some of the site, which is why a lot of Shanghai dragon will go to the web log analysis, web log analysis but people don’t fully understand the web log, below I will discuss the Web log the secret.
analysis of Web log requires Web log analyzer, of course, many people now use the free web log analyzer, but the web log analyzer of things is very limited, so a lot of website information will be hidden, here I will take that pay for the web log analyzer to explain.
everyone through the ordinary log analyzer is generally not to see the spider, what spider, access time, access to which page, access times and return a status code, as Figure 1, and more professional log analyzer can know the number of spiders to, also know the total residence time, total spiders crawl, such as figure 2,
well, today is the end of it, I hope you can help, this paper prepared by the 贵族宝贝hzht007贵族宝贝 station, please keep, don’t like yesterday’s article "promotion by statistical graph from the web traffic of thinking" has been reprinted more than N, and change the connection, be sure to please respect copyright.