About / FAQ 3 October 2022
I like the idea of building crawlers. This website is more of an exercise in intellectual curiosity for me at the moment then it is something that I think will take off. I want to figure out how to effecitively find and crawl directories and then index that data in a usable way and then present it in a useful way.
What are Open Directories? "Open Directory" is the monicor given to basically any computer that will allow a browser to remotely view and access files. These files could be shared purposefully, maliciously, or unintentionally. Most of Open Directories are just websevers configured to allow their web directories to be viewable. When you go to a website that has images, the images aren't normally stored in the same folder as the page you are viewing. They are stored in a seperate folder. If you look at the code for the site you may see something like "img src="example.com/img/example.png"" used to display an image on the page. This would indicate that the website has a /img/ folder. You could copy the URL, delete the example.png from the URL, and paste it into a browser. If the webserver allows it you can now see a list of all the images in the /img/ folder. This would be an open directory. Sometimes this is a misconfiguation and not intentional, other times this is competely on puporse.
What can be found in Open Directories? Literally anything. Any file you could possible imagine existing. It is all potentially out there open for you to look at. You just have to know what your looking for and where to or how to look for it. Movies, songs, audiobooks... again literally anything.
Do you host any files yourself? No, I don't host or vouch for any of the files on that you may find on this site. I simply indexed their existance and let you search my index. I have no idea if any of the files you download are legitimate or clean of viruses. I normally advise sticking to files that can be viewed in the brower as oppose to files you have to download. If you are looking for a video, for example, you could add mp4 at the end of you search. MP4 files are playable in your browser. MKV which is also a popular file type for video files is not streamable. You have to download it all first before you can look at it.
I clicked on a link and no file was downloaded or I got an unauthorized error, what's up? Well, these links are only valid while the server is still actively hosting the files at the same paths. If the server re-arranges their files or if they remove permission to the view the files then the links may become inoperable. This is why I include a timestamp. Older files in the index are more likely to not work. Newer files are more likely to work.
Do servers hosting the files track you when you download files? I'd say... most likely. If not actively then at least passively through webserver access logs. Doesn't mean they are looking for it or even know it happened.
Should I use a VPN or TOR to look at these files? I suppose that is up to you and your individual risk profile. Personally, and to Cox Cables determent sometimes, I just raw dog the internet with no VPN. I've scanned millions of files and millions of open ports from my home internet. With that said I do keep getting letters to behave from Cox. So maybe don't do things the way I do.
Can I request that files be removed from the index? Sure, search the index for the files you want removed, export the CSV of the search, then send me a message on one of the socials medias that I have linked. Provide me the CSV, the details of the rows you want removed, and the justification you have for their removal. I'm not gonna waste the time to remove any links without good cause. Again, I don't host any files myself so this will not take the file off the internet. You would need to contact the server owners for that.