Options

Discussion

IP Address Ranges used by General-Files crawler bots 2014-04-12 23:24:30
    • File-Guide.com
    • offline
    • 49
    Recently I've been blocking IP network ranges to deny various malicious scrape bots any access to my web server. I am nearly positive that in this process I have blocked IP addresses used by the General-Files bots as I have seen that my site isn't getting indexed as it had in the past.

    At present I have white-listed the IP network ranges used by legitimate bots such as Google, Bing, Yahoo, etc and I'd like to do the same for General-Files.

    I have emailed twice regarding this issue but never received any response.

    I do understand if General-Files would prefer to not publicly release this information for security purposes. I can confirm ownership of my website domain and will agree to keep any IP addresses used by General-Files' bots strictly confidential. If confidentiality of these IP addresses is important to General-Files then I'd be more than happy to exchange this information privately via E-Mail.

    Thank you!
    • Owl
    • offline
    • 245
    • 2014-04-21 11:12:16
    Thank you for contacting us, we received your messages. If it will be necessary we'll fix the issue from our side.
    BTW we recommend you to add the RSS feed on the website for more frequent updates
    • Guest
    • 2014-04-22 04:40:15
    I apologize but I don't believe you understand what I've said...

    If you attempt to "fix the issue from our side" that means you'll likely just use a new proxy or set of IP addresses which my systems will simply detect immediately and block those out as well. I need the IP addresses that your scraper bots use so that I can whitelist them on my webservers so that you guys stop getting blocked.

    We do not have an RSS feed for security purposes, but we could create one and punch a hole in our firewall for your IP addresses to receive it.
    • File-Guide.com
    • offline
    • 49
    • 2014-04-22 04:40:50
    I apologize but I don't believe you understand what I've said...

    If you attempt to "fix the issue from our side" that means you'll likely just use a new proxy or set of IP addresses which my systems will simply detect immediately and block those out as well. I need the IP addresses that your scraper bots use so that I can whitelist them on my webservers so that you guys stop getting blocked.

    We do not have an RSS feed for security purposes, but we could create one and punch a hole in our firewall for your IP addresses to receive it.
    • bulvinkl
    • offline
    • 431
    • 2014-04-24 14:41:27
    We doesn't have some fixed IP's. But we can send to you some HTTP headers. It's good idea?

Add message

Login via services

Site Stats

Files total: 440 214 239

Invalid files deleted yesterday:6 328
Added by users in total:19 955 113
Added by users yesterday:40
Indexed by Crawler in total:420 259 126
Indexed by Crawler yesterday:2

General Downloader

Downloads:454 234
Downloads yesterday:1

G-Points

for users: 25 513 423

Users total: 202 775

registered total:202 775
registered yesterday:1
online now:0

Commentstotal: 139 571

by Users total:139 571
by Users yesterday:6
Follow us on Twitter