Accessibility-First AI: Building Glucolily for the Eli Lilly Hackathon
A diabetes companion that listens, reads photos of plates, and never assumes the user can see the screen clearly, and why designing for the edge is the commercial play.
Glucolily started as a 48-hour build and ended as an Eli Lilly Code for All semi-finalist. The brief I gave myself: design as if the primary user has low vision, shaky hands, and is logging lunch one-handed at a noisy restaurant, then make the AI keep up. Most health apps assume the user is healthy. The whole premise of Glucolily is that they are not.
Multi-modal input, by necessity
Text, audio, photo. A user logging a meal should be able to snap the plate, dictate the carbs, or type, whichever is easiest in the moment. Each modality flows into the same structured log:
{
'meal_id': 'm_4f2a',
'logged_at': '2025-05-12T13:04:00Z',
'modality': 'photo' | 'voice' | 'text',
'items': [{'name': 'basmati rice', 'grams': 180, 'carbs_g': 49}],
'confidence': 0.84,
'needs_review': false
}The `confidence` field is what makes the multi-modal pipeline honest. A photo of a half-eaten plate at low light is not 0.99 confident, and pretending otherwise is how you get a bad bolus calculation.
The pipeline, end to end
Speech-to-text: Whisper-class model with a diabetes-vocabulary biasing list
Vision: a portion-estimation model on plate photos, with a reference-object hint when available
Agent: a tool-using LLM that resolves dish names against a nutrition database and asks one clarifying question max
Output: a structured log entry plus a plain-English confirmation read back to the user
Latency is an accessibility feature
If the round-trip is longer than four seconds, users abandon mid-log and don't come back. The budget broke down to ≤300 ms STT, ≤800 ms vision, ≤1500 ms agent reasoning, ≤400 ms TTS, with the UI streaming partial results so the user always sees something happening. Streaming is not a polish item; it is what keeps the user inside the flow.
On-device vs cloud
STT on-device for privacy and offline use; vision in the cloud because the models are too large for a phone today; agent in the cloud with a strict tool allowlist. The split is a moving target, every six months something migrates on-device, and the architecture deliberately keeps each stage swappable behind a thin interface.
Accessibility is a feature set, not a checkbox
Adaptive font sizing tied to system preferences, not in-app sliders
Contrast modes designed for diabetic retinopathy, validated against WCAG 2.2 AA at minimum
Voice-first flows that work without ever opening the visual UI
One-tap doctor report generator producing a PDF a clinician can actually use
Emergency alerts that bypass the chat UI entirely and go straight to a pre-set contact
PII, HIPAA, GDPR, and the MHRA
Health data is the most regulated payload you can move. The architectural commitments, minimise data collected, encrypt at rest and in transit, regionalise storage, give users a one-tap export and a one-tap delete, are not optional. In the UK, a diabetes-management tool that gives bolus advice would be a Class IIa device under UK MDR; we drew the product boundary deliberately at *logging and pattern surfacing* to stay in the wellness category for the hackathon prototype, with a clearly mapped path to regulated features later. Knowing where that line sits is a product decision as much as a regulatory one.
Why accessibility expands the market, not shrinks it
The standard pitch error is treating accessibility as a niche. ~10% of UK adults have some form of disability registered; closer to 25% report a condition that affects daily tech use. Add the long tail, users with temporary impairments (post-surgery, broken wrist, tired eyes at 11pm), and the addressable market for a low-friction, voice-first health log is far larger than the chronic-disease cohort alone. Designing for the edge usually makes the centre happier. The clinician filling in a report between appointments wants the same one-tap export the visually-impaired user does.
The semi-final as a signal
Hackathon wins are not revenue. They are validation signals, from a pharma incumbent, in front of a clinical audience, that the problem framing is correct and the team can ship under constraint. That signal is what unlocks the next conversation, whether it's a research partnership, a clinical pilot, or a funding round. Building a product that demos well *and* survives a regulatory question in the same hour is the actual deliverable.
Designing for the edge of the user base usually makes the centre of it happier too, and the edge is where the regulators, the journalists, and the long-term customers are all already paying attention.