Options

Discussion

IP Address Ranges used by General-Files crawler bots 2014-04-12 23:24:30
    • File-Guide.com
    • offline
    • 49
    Recently I've been blocking IP network ranges to deny various malicious scrape bots any access to my web server. I am nearly positive that in this process I have blocked IP addresses used by the General-Files bots as I have seen that my site isn't getting indexed as it had in the past.

    At present I have white-listed the IP network ranges used by legitimate bots such as Google, Bing, Yahoo, etc and I'd like to do the same for General-Files.

    I have emailed twice regarding this issue but never received any response.

    I do understand if General-Files would prefer to not publicly release this information for security purposes. I can confirm ownership of my website domain and will agree to keep any IP addresses used by General-Files' bots strictly confidential. If confidentiality of these IP addresses is important to General-Files then I'd be more than happy to exchange this information privately via E-Mail.

    Thank you!
    • Owl
    • offline
    • 245
    • 2014-04-21 11:12:16
    Thank you for contacting us, we received your messages. If it will be necessary we'll fix the issue from our side.
    BTW we recommend you to add the RSS feed on the website for more frequent updates
    • Guest
    • 2014-04-22 04:40:15
    I apologize but I don't believe you understand what I've said...

    If you attempt to "fix the issue from our side" that means you'll likely just use a new proxy or set of IP addresses which my systems will simply detect immediately and block those out as well. I need the IP addresses that your scraper bots use so that I can whitelist them on my webservers so that you guys stop getting blocked.

    We do not have an RSS feed for security purposes, but we could create one and punch a hole in our firewall for your IP addresses to receive it.
    • File-Guide.com
    • offline
    • 49
    • 2014-04-22 04:40:50
    I apologize but I don't believe you understand what I've said...

    If you attempt to "fix the issue from our side" that means you'll likely just use a new proxy or set of IP addresses which my systems will simply detect immediately and block those out as well. I need the IP addresses that your scraper bots use so that I can whitelist them on my webservers so that you guys stop getting blocked.

    We do not have an RSS feed for security purposes, but we could create one and punch a hole in our firewall for your IP addresses to receive it.
    • bulvinkl
    • offline
    • 431
    • 2014-04-24 14:41:27
    We doesn't have some fixed IP's. But we can send to you some HTTP headers. It's good idea?

Add message

Login via services

Site Stats

Files total: 437 893 458

Invalid files deleted yesterday:2 973
Added by users in total:19 945 982
Added by users yesterday:94
Indexed by Crawler in total:417 947 476
Indexed by Crawler yesterday:12 763

General Downloader

Downloads:454 201
Downloads yesterday:2

G-Points

for users: 25 491 949

Users total: 202 695

registered total:202 695
registered yesterday:4
online now:2

Commentstotal: 138 344

by Users total:138 344
by Users yesterday:11
Follow us on Twitter