MediaWiki improvement: status + progress, strategies + things to try

[mcint]

We have uptime monitors running, in addition to NB-hosted grafana, I/Loren, Jet, and Naomi, are hosting uptime kuma instances tracking wiki uptime.

It may be helpful to have a way to locally load test some aspects of caddy rate-limit usage, so both instrumentation of caddy which could also be exported from prod caddy to grafana as well as a testing traffic generation harness that exercises the rate-limit cues.

Logs and log analysis are key to filtering what we need to filter. Right now we expire logs after 1 week (and this fits our disk quota, we probably couldn't keep 1 month, or at least 2 months of logs, with current quota, we could export these). How private are logs? It would be feasible to determine every logged-in page a user visits, I find this a bit too creepy—but in general logs seem ok to narrowly privately distribute for ongoing performance filter updates.

Updates made in-place—for caddy's Caddyfile, I've found it essential to draft changes in place. I should submit at PR to have ansible complain instead of overwrite Caddyfile and MediaWiki LocalSettings.php, at least for m3.

Caddy rate-limit configuration (it may be preferable not to share exact wording tricks here, the cracks in the system) in order of increasing maintenance overhead or risk-weighted fragility:

• client ip, or /24
• user agent
• base caddy paths -- in many .request.uris, paths are a great risk-predictor of database load
• status as logged -- we're using a proxy for it, we many want or need to improve that over time, I think it can be discovered by logged out / outside users

---

Other topics

• Frustrated, proactive users are standing up read-replicas of the wiki, and entertaining or advocating for replacing its hosting outright.
• Backups & exports, likely to new colossus on-prem vm, to aid recovery, and support read-replicas as well as for a staging environment.

[nthmost]

For the benefit of all -- @jetpham and I want to share where we've landed:

https://github.com/noisebridge/noisebridge-wiki — a dedicated wiki host we're calling kells. (Here's why.)

The repo already has a full implementation checklist, primary/replica architecture, Nix-managed config, and CI/CD. The migration path is tracked as discrete issues: database migration, file migration, DNS cutover, bot mitigation, dumps infrastructure — all open, all actionable, no heroics required to pick one up.

Here's why:

m3 is running the wiki, MySQL, Caddy, Postfix, Mailman, Prometheus, and monitoring on 1 vCPU, 2GB RAM, no swap.

Every tuning decision there was made under resource pressure.

Rate limiting and caching are real improvements — and the hard-won knowledge behind them is exactly what kells is being built on — but we're treating symptoms of a machine doing too many jobs. Separating the wiki onto dedicated infrastructure is what makes the other fixes actually stick.

The bot problem gets more tractable with public dumps at dumps.noisebridge.net which is exactly what Wikipedia does.

Give bulk scrapers a structured, downloadable database and redirect them there — scraping the live wiki becomes economically irrational. Caddy rate limiting stays as a backstop.

This is a hackerspace. At some point the answer to "what if it doesn't work" has to be, "then we learn something and fix it." That's what we've been doing with the wiki anyway. We just may as well apply that learning to a beefier system designed from the ground up for our needs.

I have an engineering focus that revolves around explainability and I've staged out this issue list to make it clear how we can layer wiki bot protection on top of proper performance headroom and tuning to right-size the round tower and ladder we build for the wiki.

But regardless of what we do, we have to do something. The existing server is teensy tiny and it is shared with other services. That was fine in the early days, but it's not fine anymore.

On top of that, the monolithic multi-server Ansible infrastructure makes changes more difficult than they need to be and makes it very difficult for a new contributor to understand what success looks like a lot of the time. All of that ruins our bus number for infra.

Separation of concerns just makes sense here. m3 stays running throughout. The migration checklist is in the README. The plan is completely derisked via backups and failovers.

Noisebridge infra has long been hazing-by-complexity problem. You have to already know the thing to be allowed to touch the thing — and the people who know the thing have a stake in being the people who know the thing.

Nobody planned it that way. It's just what's happened by default.

Kells will be an attempt to build something where that pattern can't take hold.

@mcint — your knowledge of how m3's Caddy config evolved is exactly what we need to make sure nothing gets lost in translation. Would love to have you involved.

[mcint]

When it was a minor issue, config improvements were obviously the way. When the issues got more persistent, people were attendant to config changes. As the person active in meetings and an infrastructure both over the last three years, people already expressed wonder, that I don’t want to spend time defending as new discussions, about who the hosting provider is and how much we’re paying. Descriptively, this is why.

That said, the current I think $7.68 per share, now $30-ish for four shares per month, for M3 alone, is a reasonable deal. And meets expressed community needs.

Give bulk scrapers a structured, downloadable database and redirect them there — scraping the live wiki becomes economically irrational. Caddy rate limiting stays as a backstop.

You can indeed claude, lay out the red carpet for scrapers, but why would they even look for it, when they’re perfectly happy getting 95 or 99% 500 errors at 100s or requests per minute. What’s the HTTP status code for your Viking treasure is at another URL?

[nthmost]

You can indeed claude, lay out the red carpet for scrapers, but why would they even look for it, when they’re perfectly happy getting 95 or 99% 500 errors at 100s or requests per minute. What’s the HTTP status code for your Viking treasure is at another URL?

@ElanHR perhaps Elan has the Wikipedia answer to this.

[nthmost]

You can indeed claude, lay out the red carpet for scrapers, but why would they even look for it, when they’re perfectly happy getting 95 or 99% 500 errors at 100s or requests per minute. What’s the HTTP status code for your Viking treasure is at another URL?

As for my answer -- I've already addressed this both casually in my article and technically in the wiki issues I've mapped out and had Claude post on my behalf (because good formatting and organization is nice, and i don't got time for that).

So to the degree you're asking here i suppose this is performative conversation for others' benefit?

Alright. Always happy to perform conversation.

So, yes: for scrapers that are genuinely generating 95% 500 errors and don't care, yeah they are not reading the responses. They're queue-draining a URL list. The redirect goes nowhere because they never follow it. I said this (casually) in my article, and also technically right here: noisebridge/noisebridge-wiki#10 -- particularly the "What to expect when you're expecting bots" section.

For these actors, the dump only wins if it gets indexed by LLMs and becomes the canonical source before the scraper builds its URL list. This is a a long-term play, not something that changes today's load.. That was, again, what I was saying in my Vikings article.

So: The dump doesn't redirect the current Viking hordes -- it starves future ones. The sequence is:

1. Now: 302 UAs you know, add Link header to all responses, update robots.txt sitemap pointer
2. Rate limit → 302 instead of 429 (the ones following redirects now go to dumps)
3. After dumps.noisebridge.net is indexed and known: AI trainers start pulling from the dump instead of scraping live, because it's in their pipeline
4. If load is still unacceptable: Anubis

The honest HTTP status code for "your Viking treasure is at another URL" is 302. but we need to put it where the scrapers actually read it: in the Location header of the rate-limit response, in the Link header of every response, and in robots.txt. Not just on an error page they never render.

And AGAIN b/c for some reason i can't seem to say this enough for your satisfaction @mcint -- the "stupid" Vikings don't go away through any of this. The idea of Kells is to participate in a more broad (meaning: Noisebridge joins a movement that's bigger than us) of redefinining the relationship between Vikings and monasteries.

[nthmost]

And also @mcint just to give you an impression of the scope of "this is bigger than us" -- the Humans on the Loop podcast wants to talk about this with me. And potentially you, if you're up for it.

[NovaRouteAI]

"MediaWiki improvement: status + progress, strategies + things to try" is the kind of thread where I would make raw usage and normalized cost visible before trying to optimize model choice.

I would keep provider, model, request type, prompt/output tokens, cached-token behavior, latency, retry count, and final billing bucket together in one event. That makes cost debugging and future routing decisions much easier.

I am testing a multi-model OpenAI-compatible API layer around official Chinese models, so the routing and usage-accounting parts of this are directly relevant to me. For infrastructure, would the first win be user-facing cost transparency, internal spend debugging, or routing cheaper calls to cheaper models?

[SuperQ]

On top of that, the monolithic multi-server Ansible infrastructure makes changes more difficult than they need to be and makes it very difficult for a new contributor to understand what success looks like a lot of the time. All of that ruins our bus number for infra.

So, I'm not sure what you suggest as an improvement here. Splitting up to multiple deployments? Different varied deployment methods?

The reason we went with a "monolithic" Ansible repo was explicitly make it easier to onboard since there was now only one place to look at for infra. Previous to this it was all hand-rolled and only sometimes documented.

Ansible allowed us to basically make the whole setup self documented by using simple and clear Ansible roles.

For example, suggesting that we move a service from one node to another? That's easily possible in a single PR. #454

[nthmost]

@SuperQ the design intent makes complete sense — one place to look, self-documenting roles, easy PRs for moving services. That's a good goal and I don't want to dismiss the reasoning behind it.

I'm also here to state that the motivations you and @mcint and I and @jetpham all share are one and the same: love for the institution, protectiveness of our history and desire to keep things going.

Same values.

What I'm seeing in practice is a gap between that intent and the outcome. The clearest evidence: the primary maintainer ended up editing m3 in-place rather than through Ansible, because the deployment path felt inaccessible for several years. I don't know why that was the case, but it was.

That's not a criticism of anyone; it's just what happens when the coordination layer requires synchronous human availability that volunteers can't always provide.

Nix flakes don't solve the social problem. But they do make it harder for undocumented state to accumulate silently -- the config IS the running system, not a representation of it that can drift.

There's also sort of a consent question: what if the people most actively working on the infrastructure have moved on from Ansible as a preference? Jet is building in Nix. I'm more comfortable there too.

Consensus isn't just about the people who set something up; it's also about the people who are doing the work now.

The question isn't really Ansible vs Nix. It's: what would make it easier for more people to contribute without needing to already know where all the bodies are buried?

Kells is one answer to that question. I'm genuinely open to other answers too.