Lobsters still got a pulse, largely.
It's also not open access. You have to run an invisible tightrope so its algorithms won't automatically shadowban you and then you have to run a barely visible tightrope to get enough karma to not have your posts automatically removed from every subreddit.
As in: I just don’t understand how IRC works…
Then again, maybe they like the ability to critique your posts without you being able to rebut their opinions. It's an effective tactic in online echo chambers.
This feels antithetical/dystopian/elitist compared to what online communities are, for me. I think it's ok for some communities to be like this, but I don't think it's the future of online communities.
I think that was one of the big mistakes/oversight that came with the Internet..
I am not so thankful. But it is coming from my view of the current world.
I think open Internet for business is good. But open Internet for communication was bad. I cannot think of one good thing that have resulted from an open internet for communication/discussions. It has turned out to result in a collective, perpetual state of seizure for all humanity.
I am someone who had to learn stuff from random encounters with books in the 90s. Even then, I would not have it any other way...
Deciding who is in or out is the fundamental basis of any community, that's freedom of association. That's what defines a community. A place anyone can enter is a train station bathroom and that's what the logical terminus of the open internet seems to be.
Although the statement "invite-only only is the future of online communities" isn't super well defined (what does it mean to "be the future"?), so I think the comments are debating past each other.
I joined their IRC channel last week and it's been rather chill.
I do like the invite-only nature of the site though, keeps quality high...while making some folks tyrannical.
My intuition says it's to do with Status Signaling theory .
AI posts are usually the most commented on. Linus Torvalds’ comments saying that LLMs are actually useful is still on the front page, tagged as vibecoding, and has a lot of comments from people mostly disagreeing with him.
Lobste.rs is more of a monoculture than Hacker News. If you get downvoted enough they stick a banner on the top of the page inviting you to delete your account. It has a nasty side effect of driving away good contributors who don’t align perfectly with the hive mind. It’s a site where you learn to keep your mouth shut if your input doesn’t agree with what the core users want to hear.
edit: my response isn't about lobste.rs. I don't have an account there.
If you got into my group of death metal people, and suddenly started talking about Oasis... we would invite you to get out haha.
Same with patriots.win . They may be a group of right wing biggots, but hey, they did their platform and are happy talking there.
Some serious cognitive dissonance here.
I think a lot of people like this idea of a close-knit clique of like minded people, but the communities tend to pick up toxic qualities where membership becomes a game in itself. Anyone who doesn’t fall in line with the established in-crowd gets pushed out via the Byzantine rules that the in-crowd uses to maintain their position.
I still visit and participate, but I’m realistic about what it is. There is some value there, but you have to navigate it like a small subreddit where the moderator keeps tight control over what can be discussed and indirectly or directly influences the allowable opinions on the site. I really liked their original ideas about having an open moderation log and clear rules, but I’ve seen more than enough questionable moderation decisions and selective application of the rules to accept that it’s not as pure as I thought it was going to be.
That said, I don't believe it's as much of a hivemind as you've portrayed. I've seen dissenting opinions voiced there which weren't downvoted (or at least not noticeably), while the same opinion in a similar thread here on HN would be downvoted and flagged. So I don't think that getting downvoted enough so that you're invited to delete your account is a common occurrence with productive members.
It is! It’s a rite of passage on the site if you dare comment anything that someone might disagree with.
There have been several high profile cases of valued members getting fed up with it and leaving the site. One example where even the community was starting to question if this is really what they want: https://lobste.rs/s/zp4ofg
0: deleting comments of people who aren't interesting from my view and highlighting ones who often say things that are interesting, the user I'm replying to is one of the latter
There are some interesting comments there from some people who don’t spend time on other sites, but your assessment is similar to mine
My comment about being anti-AI above may not have been clear. They actually talk about AI a lot and AI threads attract a high amount of activity. The difference is that insightful or honest comments don’t have much of a chance because there’s a clear popular opinion on the site and any information that doesn’t fit isn’t welcome.
The giant soupy mess of opinions on HN can be tiresome, but it’s nice to get a wider range. HN is its own little bubble but at least there’s some diversity of experiences to be found.
Sometimes (maybe 5% or less) the request won't render at all, and you get a browser error page.
Today they ran into this bug, lost a bunch of voting data, and went into read-only mode for several hours:
https://github.com/rails/rails/pull/57128
I wonder how much of this is usual bugs which crop up during major database migrations, and how much is caused by choice of SQLite.
Maybe it's improved since I last used it, but to my knowledge SQLite essentially forces all writes to be serialised, at risk of data corruption otherwise.
There are tricks for improving the performance such as WALs, but that is merely a performance boost rather than genuine concurrency with things like row-level locks that you might find in other databases.
I guess if the whole thing is architected with a write-through cache that handles concurrent writes and deals with serialising all the writes, then it can be a single writer streaming changes through to the database, but then you still have a point of serialisation, it just will manifest itself slightly differently.
And SQLite is something that will give you constraints you will always have to consider.
Whereas running mariadb or postgres on the same machine would deliver similar benefits without a risky migration.
If your DB is small enough to run as a SQLite database, then it probably ought to have never been on a different machine in the first place.
There is a very happy medium between SQLite and a database on a different machine, one I am continually surprised to see people ignore.
For example, one of the biggest optimisations that Hacker news does is that it serves logged out users from a cached copy of the front-page.
Logged out users don't care/notice about comment counts, they don't notice that it doesn't update as often, they can't be hiding articles so you can serve the same front-page to the millions of anonymous users and bots, and update that cached copy once and on a slower cadence than every request.
Can you qualify "many"? SQLite easily handles 100k+ writes per second and it's not hard to have app layer code serialize and batch writes to take advantage of that speed. Concurrent writes require a ton of overhead and your system and code can be quite a lot faster and simpler if you just skip the idea of them altogether.
They could have one SQLite instance per user and then have a single sweeper that goes through all last writes and then replicates it to the main instance - eventual consistency fanned out across files
Eventual consistency on users would make it difficult to do things like "WHERE !user.is_banned" when getting the stories, so you need to keep your users database tightly synchronised. Sure, you could pro-actively delete or mark comments when deleting users, but now you're risking having the ban itself fail, and have also now added a much longer write operation, as you have to mark all those comments deleted. And long running writes is the one thing you desperately need to avoid in SQLite.
And why go to all that effort when you could avoid all that effort by running a database which allows concurrent writes on the machine local to the web application?
All the benefits of machine-local latency, and all the benefits of concurrent writers and transaction isolation.
If you can do that, you can have infinity instances.
As an aside, on replication, eventual consistency is not a panacea:
If I make this comment, it's absolutely fine if you don't see it for a minute.
If I make this comment, it's absolutely broken if I then don't see it on my subsequent requests. I'll think the site has broken, and try to resubmit my comment.
You can relatively easily work around that one by pinning people to instances, but that's still yet another thing to consider, and those kinds of considerations add up when you're dealing with distributed systems.
However, this a single instance, it's way too early to talk about replication.
Given that just hitting their front-page is taking 6+ seconds, they've got a performance problem that needs fixing, and my hunch is that they have some kind of "last accessed" database entry, which turns every request into a write.
Citation needed. I love postgres as much as the next person, but it does have more overhead than Sqlite which is in an in-process db linked through compiled C code, it doesn't run as a separate server. Very significant difference that when you use Sqlite db, there is typically no db process other than your application logic, unless you implement the server yourself. If you don't want your application to have multiple processes (say, as a toy example) then it totally makes sense to prefer Sqlite just for this reason. Sqlite and postgres are different tools, they serve different purposes.
---
I've heard "why not PostgreSQL?" a few times this week. It was even our original plan in #539! Well, it was a pragmatic choice in two different ways:
The person who volunteered to do the work used SQLite.
I don't want to use solutions that are bigger and more complex than our likely needs. Postgresql is my default for projects, but it does have the added complexity of being a separate service to run, tune, and maintain.
That is a totally valid reason.
Though if you can run it from SQLite there is not much tuning on Postgres that would be needed. I would expect migration to Postgres from MariaDB would be easier.
More: <https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...>
K1 acquired the commercial entity behind MariaDB Enterprise, but that's separate from the non-profit MariaDB Foundation. And there's literally zero risk of the MariaDB server suddenly going closed-source; as a fork of MySQL (which is GPL), this is not even legally possible!
[1] https://github.com/lobsters/lobsters/issues/539#issuecomment...
If traffic grows they will find problems, then limitations, then move to postgres. And if not it's fine. Story old as time.
And in any case, https://github.com/MariaDB/ links to mariadb.org which is run by the MariaDB Foundation, not the commercial enterprise owned by K1.
- you have to use locks to prevent mangled write transactions, instead of the db handling it. Jepsen report is scathing
- its pretty hard to set it up in semi-sync replication mode, a.k.a. "only return from a transaction commit when the transaction is present on at least one other replica". Once you've set it up, you have to deal with insane things like transactions being visible on replicas before leaders, or even worse transactions visible on replicas briefly but never visible on leaders or other replicas
- every knob and feature contains subtle bugs, sometimes ones as scary as index corruption, that you only discover after it has caused you pain
I don't know why you would choose it unless you were already running it
Vitess. Also I often wonder if these system are on latest mySQL? I have read plenty of mySQL opinions that still based their experience during 5.x era.
The locking anomalies are a well-known and well-documented side effect of InnoDB REPEATABLE READ differing from the standard in certain pathological cases. The real-world relevance is minor, especially since companies using external write-through caches are already making much worse inherent trade-offs in that area. Regardless, MariaDB fixed it, while providing an option to revert to the old behavior for MySQL compatibility. And fwiw this only comes up because MySQL/Maria defaults to REPEATABLE READ; yet meanwhile the overwhelming majority of Postgres users are totally fine with the much weaker guarantees of READ COMMITTED, because that's their default isolation level.
Semi-sync has some odd behaviors indeed; there are conceptual trade-offs that must be made if you want synchronous-like replication without the full performance penalty! Do you have some novel academic solution for this that you can propose?
Your last bullet is lacking in details so I cannot comment.
Sometimes there are valid technical motivations to move away completely, but a nonsensical fear of a massively-used GPL database somehow becoming closed-source (with no surviving forks or continuity) is not one of them in my book!
That's not necessarily a bad thing, but honestly, as a foreigner, I have limited English vocabulary, so I end up getting stressed out over being judged as AI. At least here, the admin DEN kindly understood my situation and let me participate after getting permission. But when I look at subreddits and other communities, the stronger the monocultural management, the more fan-like sentiment tends to grow, making it harder for minority voices to speak up.
I'm in favor of LLMs (since being for or against LLMs inevitably depends on regional context—I personally find LLM translation incredibly convenient and helpful), but there are moments when I find it hard to argue against blanket opposition or certain topics. I thought about why that is, and I think it's because Lobste.rs has high-quality posts, but since it's inviet only, the cultural barrier to entry is high, and judgment itself is part of the community identity. It's more about signals like 'are you the same kind of person as us' than logical counterarguments. The community is small, so the same core users appear across many topics, and the reputation built there follows you around, making it closed off to minority opinions.
HN, on the other hand, isn't really a single community but a mix of many groups—startup founders, big-tech engineers, and independent developers like me. Depending on who happens to be reading at that time, the comment sentiment often shifts. And since there are so many readers, even if I say something on one thread, it's unlikely anyone will recognize me in the next.
So Lobste.rs feels like a hard place for an outsider to fit into.
(Is this a typo?)
I'd like one too if someone wishes to invite me as well, Thanks and have a nice day!
You are awesome mate!, have a good day :-D
1. Stats are used by "Cost-Based Optimizers."
I think this is the most appropriate wiki:
https://en.wikipedia.org/wiki/Query_optimization
On my Oracle databases, I run the dbms_stats.gather_database_stats package procedure once a week. SQL Server also has stats for sure, and I would be surprised by a relational database that did not.
SQLite will quietly gather stats with each database connection. Running an explicit analyze devotes full attention to an otherwise piecemeal group effort.
2. I don't think rows are physically removed when deleted, instead they are marked. Removal will definitely happen at vacuum.
3. There is a backup command in the CLI that will make a transaction consistent copy of the database file, and there are lower-level C functions behind it that can be used with object code.
4. It is important to remember the many restrictions of WAL mode, including the impact on ATTACHing multiple database files: ACID consistency is lost.
https://sqlite.org/lang_attach.html
'Transactions involving multiple attached databases are atomic, assuming that the main database is not ":memory:" and the journal_mode is not WAL.'
In any case I will take an invite if one has one because the community is solid.
I do have to say that I've found lobste.rs generally a pretty slow site to render and navigate and I wonder whether that's the Rails backend, or underpowered hardware, or lack of caching. It's more feature-ful than a HN or Reddit though with much lower usage so I wonder what it is.
DHH has made a big deal about using SQLite in rails
It's in SQLite's documentation, and almost no one switching to Sqlite seems to be aware of it because no one discusses it in blog posts like these. I guess most projects switching to SQLite have very low traffic and no malicious users (yet)
I'm absolutely no expert, I'm just reading about it now, but from the SQLite WAL documentation [0]:
> 7. There is the extra operation of checkpointing which, though automatic by default, is still something that application developers need to be mindful of.
and from their "checkpointing" documentation [1]:
> By default, SQLite does a checkpoint automatically when the WAL file reaches a threshold size of 1000 pages.
I'm only skimming but I see no mention of waiting for an idle time window.
Have you been successful in DoS'ing lobste.rs?
>However, if a database has many concurrent overlapping readers and there is always at least one active reader, then no checkpoints will be able to complete and hence the WAL file will grow without bound.
>This scenario can be avoided by ensuring that there are "reader gaps": times when no processes are reading from the database and that checkpoints are attempted during those times.
Dunno, maybe Rails has a built-in workaround for this.
My workaround was to run a separate thread that monitored the WAL size on disk every second. If it went above the target size of 8 MB, my framework would enter "slow down" mode, where all reads and writes were artificially delayed by calling "sleep()", starting at 16 ms and gradually increasing the sleep time based on a few heuristics.
This allowed the application to have short gaps with no reads or writes, so the checkpointer could actually proceed.
Seems to indicate a collection cron with a manual query to truncate can fix it. This is fun little things I like to learn. SRE Easter eggs.
This seems like a huge DoS security hole.
I encourage you to write a blog post about this. It seems especially relevant.
Lobsters is invite only. That helps. :)
... and is installed and available:
# sqlite3_rsync -vv ./test.db user@rsync.net:test.db
https://use.expensify.com/blog/scaling-sqlite-to-4m-qps-on-a...
https://lobste.rs/s/ko1ji1/lobste_rs_is_now_running_on_sqlit...
I hope they get it sorted soon, I'm rooting for sqlite.
I'm not interested in a rehash of Hacker News but I am interested in more tech news.
Anyhow on topic: I find it puzzling the whole "use sqlite in production!" thing.
- you're stuck on a single machine now.
- you don't have remote database access.
- you have to live with the inflexibility of sqlite's schema change capability.
- all writes go through a single writer.
- fewer data types
- fewer backup/security/high availability options
- foreign keys and strict disabled by default.
- you're single core in a multi core world.
It's not a terrible decision but when you can choose Postgres, why would you go with sqlite?
make sure you're not doing a bunch of migrations. that's the SQLite weak point that I know of & have experienced.
use SQLite - as if you're using a KV or better yet maybe a CQRS approach.