Not Everything Should Be Open Source
Especially When AI Influences Human Choice
I keep coming back to one thought.
I’m building an AI system that analyzes skin condition and helps people choose skincare. In one form or another, I’m regularly asked the same question: why not open source? It comes from investors. From potential partners. From technical people for whom openness has become almost a universal sign of trust.
Every time, I give the same answer.
There are things that should not be open. And this is not a contradiction of the AI era. It’s a part of it that almost no one is talking about yet.
Open Source Has Become Trust Orthodoxy. But It’s an Oversimplification
For the past two years, the market has been repeating one formula. Want to be trusted? Open the code. Want to become a standard? Open the weights. Want enterprise adoption? Allow self-hosting.
This works for one type of system and works poorly for another. But no one is drawing this line clearly.
Foundation models. Llama, DeepSeek, Mistral, Qwen. Universal tools. Weights opened, the market builds on top, responsibility for application sits with whoever builds the product. A foundation model doesn’t decide anything for a person by itself. It generates tokens. The meaning is given by the next layer.
Applied decision layers. Systems that influence user choice in domains where users lack the expertise to evaluate options themselves. AI recommending a procedure. AI helping select between options. AI guiding a person through a space they don’t understand.
Between these two types of systems runs a boundary that doesn’t exist in the public conversation. A foundation model is a tool. A decision layer is a verdict. Opening a tool distributes capability. Opening a verdict distributes responsibility in such a way that no one holds it anymore.
A Neutral Layer Loses Its Power the Moment It’s Opened
The system I’m building is designed as a neutral AI layer between brand, retailer, and customer. Its value sits in a single property: no party can influence the recommendation logic. Brands don’t buy placement. Retailers don’t shift the weights. Customers receive a decision that wasn’t optimized for someone else’s margin.
This only works while the architecture is closed.
Open it, and neutrality disappears within a month. Brands reverse-engineer the algorithm and optimize their products against it. Retailers demand customizations. Competitors fork the system as “the same thing, but with our partners prioritized.” Ten versions of it live on the market, and users can no longer distinguish the neutral one from the tuned one.
Neutrality is not a property of code. It’s a property of a closed architecture operating in a field where every participant has an interest in influencing the result.
The open source community works in a different situation. Everyone reading the code is on the same side, wanting it to become better. A neutral layer lives with three sides whose interests conflict. Open neutrality is an oxymoron. The moment the architecture becomes visible, it stops being neutral, because it becomes a surface for pressure.
Trust toward such a layer isn’t built through “look inside.” It’s built through “no one, including the founder, can change the output to serve someone’s interest.” This is a different model of trust. Trust through equidistance, not through transparency.
Openness Assumes Expertise. Decision Layers Operate Where There Is None
There’s another argument that often gets missed.
Open source builds trust through an idea: anyone can look inside, verify, find a flaw, propose an improvement. This works when the user or developer has the expertise to evaluate what the system does.
A decision layer exists precisely because the user lacks that expertise. A person comes to it not to verify the algorithm. They come to trust a choice they cannot make themselves.
Open code is useful to an engineer capable of reading it. To a customer choosing skincare, open code offers nothing. They won’t read the repository before buying. They won’t inspect the model weights. They cannot distinguish an honest implementation from a fork where priorities have been quietly reordered to serve commercial logic.
In a decision layer, trust cannot be built through code access. It can only be built through two things: architectural constraints that cannot be bypassed, and the presence of a specific person or entity responsible for what the system does.
The Medical-Adjacent Layer Is a Separate Category, and Here It Gets Stricter
I have a medical education. Sechenov University. I understand what a protocol is. I understand where the line runs between cosmetology and the zone that requires a physician.
An AI system working with external skin features and skincare recommendations operates adjacent to a zone where errors can have physical consequences. This is not yet medicine. But it’s no longer ordinary e-commerce.
Between an everyday skin condition and a condition that requires a specialist, there is no visual boundary an algorithm can reliably draw. A recommendation for a procedure where a consultation is needed isn’t a code bug. It’s a displacement of the user into a zone the system wasn’t designed for.
The system I’m building was constructed with a safety layer much larger than the recommendation logic itself. Restricted phrasings that exclude clinical authority. Red-flag triggers. Hard refusal to diagnose. Handoff to a specialist in zones of uncertainty.
This isn’t lines of code that can be copied. These are years of work on where the system must stop.
And here is the core of the argument this entire essay was written for.
In ordinary software, a fork is a technical copy. In health-adjacent AI, a fork without a safety layer is a removed safety catch.
Open the architecture as open source, and within a month a fork appears on GitHub without those restrictions. “For aggressive sales.” “With a diagnosis function.” “Without excessive caution.” Someone deploys it in a clinic that doesn’t know the safety layer wasn’t an optional feature, it was the essence of the product.
Open source ideology assumes the community will improve the product. In medical-adjacent zones, the community is equally likely to simplify it. Remove restrictions that hurt commercial output. Change phrasings that seemed overly cautious. Expand the range of cases where the system is willing to respond.
This isn’t a theoretical risk. This is the baseline dynamic of any open system under commercial pressure.
The EU AI Act Is Already Moving the Market in This Direction
The European AI Act doesn’t say that every wellness or beauty AI is automatically high-risk. It introduces a risk-based logic: the more strongly a system influences health, safety, human rights, or regulated decisions, the more it requires a specific provider, documentation, update control, post-market monitoring, and accountability.
Article 6 and Annex III define the classification of high-risk systems. For such systems, the regulation requires a provider, a specific legal entity responsible for every aspect of the product’s operation.
Open source itself does not eliminate the provider requirement. The Linux Foundation has explicitly stated that the open-source exemption in the AI Act does not extend to high-risk systems: if an open system qualifies as high-risk, provider obligations arise regardless.
But open source makes the provider question harder, not easier. Every fork, every modification, every deployment in a specific location requires its own responsible party. If a system starts living in dozens of derivative versions, accountability disperses among them so quickly that effectively no one holds it anymore.
In a closed architecture, there is one responsible party. In an open architecture entering sensitive zones, responsibility has to be reassembled for each fork. The industry doesn’t yet have the tools to do that.
This isn’t a philosophical position. It’s a structural problem that regulators will have to solve, and the solution will almost certainly move toward strengthening requirements, not loosening them.
Closed Implementation Does Not Equal Closed Company
Here is the fork in reasoning that gets missed most often.
Closed code does not mean a non-transparent company. The system I’m building publishes the architecture of its safety layer, the logic of refusing to diagnose, the principles of neutrality, the protocols for data handling. Any partner receives full documentation on how the system behaves in edge cases. Any regulator can audit it.
What’s closed is the specific implementation. The weights. The ranking algorithms. The internal logic that makes neutrality neutral.
Transparency of principles and closedness of implementation are different axes. Open source ideology fuses them, as if one cannot exist without the other. This isn’t true. You can be entirely transparent about how you think and entirely closed about how exactly you did it.
In sensitive zones, this is the only correct configuration.
What This Means for Founders Building in Sensitive Domains
If you’re building a product where AI influences user choice in a zone touching the body, health, money, access, or safety, you need to consciously decide which side of this boundary you’re on.
Openness will give you fast growth, developer community trust, distribution through GitHub, easier entry into technical conversations.
It will also strip you of control over how your architecture gets applied. In sensitive zones, that is not an acceptable price.
I built this system closed not because there’s something technically precious hidden inside. I built it closed because I can see what happens to the safety layer the moment the architecture becomes public. And I’m responsible not only for what the system does today in one location or another. I’m responsible for what happens to its derivatives two years from now, when forks start living their own lives.
In ordinary SaaS, this risk doesn’t exist. In AI, it does. In medical-adjacent AI, it’s central.
Openness Should Become Deliberate
The main shift I see isn’t that one camp will defeat the other.
Foundation models will continue moving toward openness, because for them it works. Universal tools need community, distributed development, local adaptation.
Applied decision layers will move in the opposite direction. Their defense isn’t code, it’s architectural decisions that cannot be reproduced by copying. Neutrality, safety, regulatory alignment, relationships with regulators and partners.
These aren’t two competing approaches. They are two different layers of the stack, each with its own hygiene.
When the market sees this distinction, the conversation about open source will stop being moral and become engineering.
Openness is appropriate at the layers where errors don’t harm.
Closedness is required where responsibility cannot be distributed among everyone reading the code.
The era when “open” automatically meant “good” is ending. Not because openness got worse. Because AI got more serious. When a system generates a meme, opening it is an act of freedom. When a system recommends a procedure on a person’s skin, opening it is an act of irresponsibility.
And this difference shouldn’t have to be explained by a regulator. It should be seen by whoever is building.
Ekaterina Shalel is a product architect working on neutral AI infrastructure for beauty retail. She writes about AI decision layers, regulatory architecture, and the design of trust in sensitive systems.