← Ayo Osunjuyigbe

· Engineering

Loading in the browser does not mean it is secure

I have shipped the weekend app. Signup works. I can create a row and see it on the next page. I posted the URL. That feeling — done, safe enough, the model knew the stack — is the oldest finding in this story. It is not from last year. It is from Stanford in 2023.

Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh sat 47 people in front of five security tasks: encrypt a string, sign a message, open a file inside a sandbox, insert a SQL row, print an integer in C. Thirty-three of them got an assistant on OpenAI’s codex-davinci-002. Fourteen did not. The people with the assistant wrote worse code on four of the five tasks. They also rated their answers as more secure than the people who had to do it themselves.

That gap is the product. The weekend SaaS is that gap with a custom domain.

What Stanford measured

This is not AI writes insecure code, end of paper. It is a user study. CCS ’23. Mostly students, some professionals. Two hours. They could use the internet. The experiment group could also query the model and paste.

The interesting numbers are per task, not a single percentage. Encryption: the assisted group was more likely to ship a toy cipher or skip the authentication tag. Signing: 3% of them produced a secure answer against 21% of the control group. SQL: 36% of the assisted answers were injectable versus 7% without the model. The C task was the messy one — mixed, not a clean win for either side — which is why they refuse a blanket claim.

Then the survey. For every question, the people who had the assistant believed they had solved it more securely. The ones whose answers were actually insecure trusted the model more than the ones who had gotten it right. Participant 23, weak at JavaScript, said they trusted the machine to know more than they did.

I have said that sentence out loud about Cursor.

They also found the thing that still works: people who trusted the model less, rewrote the prompt, named the function they wanted, and treated the output as a draft, wrote better code. People who fed the last answer back in as the next prompt copied the same hole forward.

Limitations they own: 47 people, one 2022-era Codex, five toy tasks, not a production app. I am not pretending this is a 2026 agent eval. I am keeping the overconfidence. That part did not age.

The database is the app

Matt Palmer published CVE-2025-48757 on 29 May 2025. He is at Replit. The write-up is about Lovable projects talking to Supabase the way the scaffold wants you to: the browser holds the project URL and the anon key and speaks REST at Postgres. That key is supposed to be public. The gate is Row Level Security. If the table has no RLS, or a policy that is USING (true), or a check that only runs when a session is present, then anyone who can see the page can ask the table questions the UI never asked.

Palmer’s scan of the Lovable Launched showcase, finished 21 March: 1,645 sites, 170 projects, 303 endpoints, about one in ten of what he could see from the homepage. Emails. Phone numbers. Payment rows. Gemini keys sitting in a table. He is explicit that this was homepages only.

Lovable disputed the CVE: each customer is responsible for protecting their application’s data. That sentence is the whole argument. The platform generated a working UI. The table is yours. A later security scan checked whether a policy existed. It did not check whether the policy did any work.

I am not writing a how-to. The architecture is enough. If your network tab shows *.supabase.co/rest/v1 from the browser, the React route that says login is decoration. The database is the API. Review the policy the way you would review an endpoint, because it is one.

This is not a Lovable-only story. Any generated app that puts a public client key in the bundle and forgets the server-side rule has the same shape. Bolt, Cursor, a hand-rolled Vite app — same physics.

It passes the tests you asked for

Zhao, Wang, Zhang, Luo, Li, and Li put SusVibes on arXiv in December (2512.03262). The version I read is 186 feature requests taken from real repos where a human had already shipped a vulnerable implementation and then fixed it. The agent gets the feature in English. No security hint. Then two test suites: does the feature work, and is it safe.

SWE-Agent plus Claude 4 Sonnet, their best pair: 57% of patches pass the functional tests. 11.8% are secure. Of the ones that work, 79% still fail the security suite. Prompting the agent with a CWE hint or even the oracle CWE helped security and cost about seven points of it works. The model will trade the thing you can see for the thing you asked it to worry about.

That is the weekend deploy in a table. The happy path is a functional test. User A cannot read user B is a different suite. Nobody on the Saturday asked for that suite.

Tenzai ran the product version of the same idea in December and posted it on 13 January. Vendor blog — they sell the scanner — so treat the crown and the marketing as theirs. The counts are still useful. Five agents, default models: Cursor, Claude Code, Codex, Replit, Devin. Same three apps, same prompts. Fifteen codebases, 69 findings.

They did not find an exploitable SQL injection or XSS. The models have absorbed use a prepared statement and let the framework escape. What they skipped was everything without a slogan. Authorization that is more than user_id = me. Negative quantity on an order, four of five agents. SSRF on a link-preview feature they did not warn about: five of five. CSRF: none of the fifteen did it properly. Security headers: none. Rate limits on login: almost none.

Ori David’s line, not mine: it was not that they implemented the controls wrong. In almost every case they did not try.

What I check when I did not write it

I am not throwing the model out. I am not taking Tenzai’s deploy our agent as the moral. I am writing down the review I do before a generated app sees a real email address.

Who is the request, on the wire. Open the network panel. If the browser is talking to the database, write down every table name you see. Those are public endpoints with a published key. The login page does not sit in front of them.

Every table, two questions. Is RLS on? What does the policy actually say? A policy that exists is not a policy that holds. Check as anonymous. Check as user A with user B’s id. If you cannot impersonate, you have not reviewed it. Palmer’s scan is what happens when nobody did this and the site still looked finished.

Secrets are not all the same. The anon key in the client is normal. A service_role key in the client is a backup of the database sitting in DevTools. A third-party token in a table the anon role can read is the same leak with extra steps. Search the repo and the built JS for service_role, sk-, BEGIN RSA, anything you would rotate if it hit GitHub.

Auth that is only a component is not auth. A protected React route is a redirect. The API still has to refuse the call. Tenzai’s order-delete case is the cartoon version: ownership checked if you are logged in, skipped if you are not.

The angry path. Signup, create, refresh — that is the demo. Then: logged out. Other user’s id. Empty body. Negative price. A URL the server will fetch for you. If you did not ask the model for those cases, it did not write them. SusVibes is that sentence with a unit test.

Do not ask the same chat if it is secure. Stanford already ran that survey. The people who needed the reassurance most were the ones who should not have shipped. A second model is only slightly less circular. A logged-out request against your own table is not.

If any of those fail, the app is not late. It is open. It loaded is a render check. Ship is a stranger cannot read the other stranger.

I will keep using the agent for the Saturday. I will not let the Saturday’s confidence, which Perry’s group already measured, decide that the table is private. The vendor’s disputed note is correct in the way that hurts: it is your application. The model does not get the breach email.

References