As global AI vendors move closer to public health infrastructure, India must ensure that the systems shaping care remain accountable to the people they serve
Image Source: Getty Images
The modern hospital is becoming a place where care is being reorganised around code. In England, that shift has acquired a political edge through Palantir’s role in NHS England’s Federated Data Platform (FDP), a system meant to connect hospital data, improve planning, reduce waiting lists, and make the health service more responsive. The National Health Service (NHS), England’s publicly funded health system, says that patient data remains under public control. However, the controversy has persisted because the question is larger than data protection. A public health system depends on trust, and that trust becomes fragile when the software at its centre is supplied by a company whose broader reputation has been shaped by defence, intelligence, and surveillance-linked work.
India should pay close attention. Finance Minister Nirmala Sitharaman’s recent warning about Anthropic’s Artificial Intelligence (AI) model 'Mythos' concerned cyber risk and the vulnerability of critical infrastructure, a framework through which health systems should now be viewed. Hospitals, insurance platforms, disease surveillance networks, and electronic health records (EHRs) are no longer ordinary service-delivery tools once they become dependent on AI systems built elsewhere.
Google DeepMind has already announced partnerships with Indian government bodies and local institutions in science and medicine, showing how frontier AI firms are moving closer to sensitive public domains. The issue now at the heart of the hospital state is whether the political character of technology can be separated from the people behind AI companies and the states whose strategic interests they will inevitably inhabit. If hospital systems become part of the nervous system of governance, India must ask who is writing the operating system. AI sovereignty begins with that question because a health system cannot be fully public if its core logic is shaped beyond public control.
The NHS controversy begins with the scale of the institution itself. A separate NHS England Secure Data Environment project, Foresight AI, gives a sense of the scale now being organised within this ecosystem, with de-identified records from more than 57 million people, over 10 billion healthcare events, and more than 40,000 event types across admissions, diagnoses, procedures, medications, and vaccinations. The NHS FDP is not the same project, but it exists within the same broader transformation of health records into operational intelligence. The Palantir-led consortium won the FDP contract in 2023, valued at up to £330 million over seven years, with potential coverage across 240 NHS organisations.
Once hospitals plan care delivery through a vendor’s architecture, that architecture begins to shape institutional judgement. It determines what appears urgent and what becomes administratively inconvenient.
NHS England has tried to draw a clear boundary around the arrangement. Palantir is described as a data processor. Patient data is to remain under NHS control, processing must take place in the United Kingdom (UK), and the company is not permitted to commercialise NHS data or use it to train its own models. These are meaningful safeguards, but their weakness lies in what they cannot fully measure. Public systems can retain legal ownership of data while losing practical control over the routines and dependencies created by a platform. Once hospitals plan care delivery through a vendor’s architecture, that architecture begins to shape institutional judgement. It determines what appears urgent and what becomes administratively inconvenient.
Palantir’s reputation sharpens this concern. The company’s history in the intelligence sector makes its entry into healthcare politically charged, even when the contract terms appear restrictive. The recent controversy over its manifesto-style language, described in Westminster as the “ramblings of a supervillain”, became potent because it captured a broader discomfort about importing the instincts of hard-power technology into institutions built around care. Australia’s calls for audits or limits on Palantir’s government contracts show that this anxiety is spreading across democracies.
When the NHS first published the Palantir contract, 417 of its 586 pages were completely redacted, prompting the Good Law Project to launch legal action over the scale of the redactions. The NHS later agreed to republish the contract with fewer redactions, including material on the protection of personal data. For a platform that depends on public trust, this was a damaging way to begin.
The protest around Palantir is more coherent than a generic objection to private technology. Campaigners and civil society groups argue that the NHS is creating a data architecture whose social licence has not been properly earned. The “No Palantir in our NHS” campaign has urged NHS Trusts and Integrated Care Boards to refuse Palantir’s technology because of the company’s broader military and security work. Medact notes that the British Medical Association (BMA) opposed the FDP rollout at its 2025 annual meeting, while a public petition to Parliament has called for consultation and a clear national opt-out before further expansion. The government’s response states that the National Data Opt-Out does not apply where processing is tied to “direct care, is legally required or is anonymised”. A patient may accept that their information must move across the NHS so that doctors can treat them safely. It does not follow that the same patient has accepted a national data architecture built with a company associated with immigration enforcement and military targeting.
A patient may accept that their information must move across the NHS so that doctors can treat them safely. It does not follow that the same patient has accepted a national data architecture built with a company associated with immigration enforcement and military targeting.
Other health-AI failures show how fragile this terrain is. DeepMind’s Royal Free episode involved access to 1.6 million patient records and later drew criticism from the UK Information Commissioner over transparency and legal basis. The Epic Sepsis Model, used in United States (US) hospitals, performed poorly when tested independently, recording a hospitalisation-level AUC (Area Under the Curve) of 0.63 in a validation study covering 38,455 hospitalisations. Cigna’s PxDx system exposed another risk, with reports that more than 300,000 claims were rejected over two months at an average of 1.2 seconds per case. The lesson for India is that AI in the healthcare domain must also be assessed through the power it concentrates within public systems, the dependencies it creates over time, and the ability of citizens to understand and challenge decisions made in their name.
India does not have the luxury of treating AI in health as a future regulatory issue. The market is already moving faster towards the patient’s first moment of uncertainty than the state. A Boston Consulting Group (BCG) survey across 15 countries found that nearly 60 percent of internet-connected consumers were using AI tools for health, while a report on the same study placed India much higher, at 85 percent. Patients are already taking medical uncertainty to software before taking it to clinics.
Large technology firms are also moving closer to that encounter. OpenAI introduced ChatGPT Health, citing more than 230 million health-related queries each week and offering users the ability to connect their medical records and wellness apps. Google’s MedGemma is already working with the All India Institute of Medical Sciences (AIIMS), New Delhi, for outpatient triage and dermatology screening, while Google is working with the National Health Authority (NHA) to convert fragmented medical records into Fast Healthcare Interoperability Resources (FHIR), a standard that enables health information systems to exchange data more easily. Microsoft’s Artificial Intelligence Diagnostic Orchestrator (MAI-DxO) reported an 85.5 percent accuracy rate on complex New England Journal of Medicine (NEJM) benchmark cases when paired with OpenAI’s o3 model, compared with 20 percent for physicians in the same test setting. These figures show that a new layer is forming between patient anxiety and medical judgement. Whoever controls that layer will influence how people understand risk and the need for care.
No model should enter large-scale use on the strength of a company demonstration, a foreign benchmark, or a small pilot that has not been independently replicated in Indian conditions.
India has begun building safeguards, but they must now become enforceable. The Strategy for Artificial Intelligence in Healthcare for India (SAHI) offers a policy framework, while the Benchmarking Open Data Platform for Health AI (BODH), launched with the Indian Institute of Technology Kanpur and the National Health Authority (NHA), creates a route for testing models on diverse anonymised real-world health datasets. That route should be made mandatory before any AI system is used at a population scale in public healthcare. No model should enter large-scale use on the strength of a company demonstration, a foreign benchmark, or a small pilot that has not been independently replicated in Indian conditions.
Procurement is where caution has to become enforceable. A foreign vendor should not enter a sensitive health system until the state can see through the company, its incentives, and the route by which data will move. The contract should give the state real audit powers and make any unauthorised reuse of data punishable. It should also make exit realistic, with money and technical preparation set aside before deployment, so that hospitals can move to another system without disrupting care. Patient-facing AI needs a higher threshold because it reaches people before a clinician can correct it. Such systems must be tested in the languages Indians actually use, with a simple route for patients to report harm.
Sovereignty in health AI should not mean shutting the door on foreign technology. India will need outside expertise, but a public system should not become dependent on a tool whose workings it cannot examine or whose vendor it cannot leave without disrupting care. Regulators must be able to understand how a health platform works, and patients must have a meaningful way to challenge harm when automated systems affect them. Without this discipline, dependence will resemble innovation until its costs become too large to ignore.
K.S. Uplabdh Gopal is an Associate Fellow with the Health Initiative at the Observer Research Foundation.
The views expressed above belong to the author(s). ORF research and analyses now available on Telegram! Click here to access our curated content — blogs, longforms and interviews.
Dr. K. S. Uplabdh Gopal is an Associate Fellow with the Health Initiative at the Observer Research Foundation. He writes and researches on how India’s ...
Read More +
PREV
