I use Duckduckgo, but I realised these big(ish) search engines give me all the commercialised results. Duckduckgo has been going down the slope for years, but not at such a rate as Google or Bing has.

I want to have a search engine that gives me all the small blogs and personal sites.

Does something like this exist?

top 50 comments

sorted by: hot top controversial new old
[–] 52 points 2 years ago (3 children)
  • [–] 14 points 2 years ago* (last edited 2 years ago) (1 child)

    I'm intrigued. The search results are more akin to how they used to be 25 years ago on the internet that I loved
    Https://Search.marginalia.nu is definitely something I'll be exploring going forward!

  • source
  • parent
  • hideshow 2 child comments
  • [–] 8 points 2 years ago (1 child)

    Replying under the top comment but this really applies to all of these, how do these search engines determine what counts as a personal site? For example I had procrastinated for years on finally spinning up a static, barren HTML blog. The infamous Lucidity AI post introduced me to Mataroa and I got over the hump and started writing. Would that get indexed? Etc

    Does it just crawl through webrings?

  • source
  • parent
  • hideshow 2 child comments
  • [–] 37 points 2 years ago (5 children)

    Teclis - Includes search results from Marginalia, free to use at the moment. This search index has been in the past closed down due to abuse.

    Kagi, whose creation Teclis is, is a paid search engine (metasearch engine to be more precise) also incorporates these search results in their normal searches. I warmly recommend giving Kagi a try, it's great, I've been enjoying it a lot.

    --

    Other options I can recommend; You could always try to host your own search engine if you have list of small-web sites in mind or don't mind spending some effort collecting such list. I personally host Yacy [github link] (and Searxng to interface with yacy and several other self-hosted indexes/search engines such as kiwix wiki's.). Indexing and crawling your own search results surprisingly is not resource heavy at all, and can be run on your personal machine in the background.

  • source
  • hideshow 7 child comments
  • [–] 13 points 2 years ago (1 child)

    Not just a meta search engine though - they do have their own index as well.

    https://help.kagi.com/kagi/search-details/search-sources.html

  • source
  • parent
  • hideshow 2 child comments
  • [–] 6 points 2 years ago

    Yes, I mentioned Kagi because of the Teclis search index is hosted by them.

    However, most of the search results in Kagi are aggregated from dedicated search engines. (such as, but not limited to: Yandex, Brave, Google, Bing, etc.)

  • source
  • parent
  • load more comments (3 replies)
    [–] 35 points 2 years ago (2 children)

    Try this engine

    https://search.marginalia.nu/

    Or a SearXNG instance

    https://search.disroot.org/search

    You may also be interested in the Indie Web movement. This site is a great resource for it, with yet more links to indie sites and blogs.

    Finally, not quite what you asked but here's a freebie, in case you didn't know about it:

    https://wiby.me/

    It's an old web search engine. It only indexes pages from the 00s and earlier.

  • source
  • hideshow 4 child comments
  • [–] 6 points 2 years ago (1 child)

    Ah Marginalia is absolutely awesome! I feel like modern search is almost an extension of website names now, so if I want to find netflix but don't know it's website, I might search for "netflix". Marginalia is actually a cool way to find new stuff- like you can search "bike maintenance" and find cool blog posts about that topic.

    I honestly can't remember if that's something google and the like used to do, but doesn't now, or if they never did. Either way, I love it!

  • source
  • parent
  • hideshow 2 child comments
  • [–] 28 points 2 years ago (3 children)

    This is a great question, in that it made me wonder why the Fediverse hasn't come up with a distributed search engine yet. I can see the general shape of a system, and it'd require some novel solutions to keep it scalable while still allowing reasonably complex queries. The biggest problems with search engines is that they're all scanning the entire internet and generating a huge percent of all internet traffic; they're all creating their own indexes, which is computationally expensive; their indexes are huge, which is space-expensive; and quality query results require a fair amount of computing resources.

    A distributed search engine, with something like a DHT for the index, with partitioning and replication, and a moderation system to control bad actors and trojan nodes. DDG and SearX are sort of front ends for a system like this, except that they just hand off the queries to one (or two) of the big monolithic engines.

  • source
  • hideshow 6 child comments
  • [–] 8 points 2 years ago* (last edited 2 years ago) (2 children)

    We'd love to build a distributed search engine, but it would be too slow I think. When you send us a query we go and search 8 billion+ pages, and bring back the top 10, 20....up to 1,000 results. For a good service we need to do that in 200ms, and thus one needs to centralise the index. It took years, several iterations and our carefully designed algos & architecture to make something so fast. No doubt Google, Bing, Yandex & Baidu went through similar hoops. Maybe, I'm wrong and/or someone can make it work with our API.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 9 points 2 years ago (1 child)

    I think 200ms is an expectation of big tech. I know people have very little patience these days, but if you provided better quality searches in 5 seconds people would probably prefer that over a .2 second response of the crap we’re currently getting from the big guys. Even better if you can make the wait a little fun with some animations, public domain art, or quotes to read while waiting.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 2 points 2 years ago

    if you provided better quality searches in 5 seconds people would probably prefer that over a .2 second response of the crap we're currently getting from the big guys

    This is precisely what made me switch to ChatGPT as my primary "search engine". Even DDG is fucking useless these days if you need anything more complex than a list of popular sites that contain a couple of keywords.

  • source
  • parent
  • load more comments (1 reply)
  • [–] 22 points 2 years ago (1 child)

    You're looking for Kagi.com

    Not only does it give better search results quality wise on "the big web" - you can select to search specific parts, like blogs.

    Best part - it's completely ad and spam free. You pay for it with actual money instead of with your data.

  • source
  • hideshow 2 child comments
  • [–] 18 points 2 years ago* (3 children)

    Why not run an SearXNG instance and help everyone instead? Y'know, Kagi is pretty expensive and they are also getting into AI shit.

  • source
  • parent
  • hideshow 6 child comments
  • [–] 11 points 2 years ago (3 children)

    I'm hoping just as Proton do good free stuff using money I pay them (Visionary account) Kagi does/will do the same. The Internet as a whole needs to stop being ad-supported.

  • source
  • parent
  • hideshow 4 child comments
  • [–] 4 points 2 years ago* (4 children)

    I refuse to believe Proton when they do advertisements lol. They also are being pretty suspicious with ignoring XMR support since years of people requesting it. If they ever even considered it a bit, their new shit Proton Wallet wouldn't allow you to store (or only store) bitcoin, which we all know has nothing that protects your privacy.

  • source
  • parent
  • hideshow 4 child comments
  • load more comments (4 replies)
  • load more comments (2 replies)
  • [–] 3 points 2 years ago (1 child)

    Can you expand on how running your own SearXNG helps others? Does it contribute to some shared index or something?

  • source
  • parent
  • hideshow 2 child comments
  • [–] 20 points 2 years ago (1 child)

    Google are the ones who have really gone down the toilet in recent years. They ditched cached pages, soured search results with paid ads and even their image search is as bad as Tineye for reverse image searching these days. Literally the only thing Alphabet really have going for them anymore is Android and YouTube.

    It's baffling that a company which was once so dominant in the web search space that their name was literally used as a verb for looking things up for decades have now enshittified their flagship product so much that they're making rivals like Bing, Lycos, Duckduckgo, etc look like viable alternatives.

  • source
  • hideshow 2 child comments
  • [–] 19 points 2 years ago* (1 child)

    Maybe try Mojeek, it uses completely independent indexing system.

  • source
  • hideshow 2 child comments
  • [–] 4 points 2 years ago (1 child)

    Mojeek

    Thanks for the rec, I'll give Mojeek a try for a while. So far the results seem better than Brave (which I didn't seriously consider using regularly anyway) but I miss the bang options (!w, !yt, etc.) that DDG has.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 16 points 2 years ago

    Offtopic but ddg is a bing frontend so they should share the same results.

  • source
  • [–] 12 points 2 years ago

    I'm building my own. Keep you posted.

  • source
  • [–] 10 points 2 years ago (3 children)

    The more obscure a web page is, the more likely it is to be indexed only by the large search engines (i.e. Google). There are search queries that return 0 results on DDG, but quite a few (relatively) obscure websites on Google. This is simply because the more money a search engine operator has, the more websites it will index.

    So what you want is kind of contradictory.

  • source
  • hideshow 3 child comments
  • load more comments (3 replies)
    [–] 9 points 2 years ago
    [+] 8 points 2 years ago* (last edited 1 year ago) (2 children)
  • [–] 7 points 2 years ago (3 children)

    If you want blogs, I recommend you use gemini: https://en.wikipedia.org/wiki/Gemini_(protocol)

    Download Lagrange and begin browsing. It's basically a small-web of personal blogs.

  • source
  • hideshow 3 child comments
  • load more comments (3 replies)
    [–] 3 points 2 years ago (1 child)

    Before google existed I used https://www.metacrawler.com it appears to still be around. I have not used it in a long time, so I know nothing about it any longer.

  • source
  • hideshow 2 child comments
  • [–] 4 points 2 years ago (1 child)

    https://system1.com/ adtech company syndicating Bing and/or Google

  • source
  • parent
  • hideshow 2 child comments
  • load more comments
    view more: next ›