If you’re in government or intel, and your Intel9.us bot/crawler/scraper has been blocked, please read.

Web traffic to any website includes a large proportion of bots and crawlers- automated engines, some of which are good, others malevolent. There’s also quite a bit of hacking. Since Intel9 has limited bandwidth, bots noticeably slowed the site.

It was decided to enable filters to block most of the bots that are not part of public search engines, with exceptions made for bots that are light on bandwidth. If you are an official organization, and your bot was blocked, my apologies. There is no way for me to tell it is you from a nameless IP address without a corresponding DNS record.

Every government, most especially our European allies, and including Russia, is welcome to scrape Intel9 with bots. The caveat is that if a bot scrapes the site from top to bottom frequently, and I have no way of determining its benign intent, it will be blocked.

If it is practical for you, as it may be for some  official organs outside the intelligence communities, drop me a line, and I’ll make sure not to block you.