ESE5160 Final Submission | Team 29

Mind2Matter

Internet-connected smart glasses for voice assistance, visual context, reminders, fall-event support, cloud AI, and over-the-air firmware updates.

Final Demo Public Source Repository

Mind2Matter final integrated smart-glasses prototype — Final integrated smart-glasses prototype with electronics and casework.

Project Team Team 29

Built By Yibo Wang

Built By Zicong Zhang

Submission A11G Final

Platform SiWx917 / BRD2708A

Cloud MQTT + Node-RED + HTTP

Validated Voice, Vision, OTA

Form Factor Wearable Glasses

1. Video Presentation

Final demo video

The final video shows the complete wearable prototype, mechanical mounting, sensor data on Node-RED, cloud-to-device controls, voice/vision interaction, speaker playback, and OTA firmware update.

2. Project Summary

A wearable embedded/cloud assistant.

Device Description

Mind2Matter is an Internet-connected smart-glasses prototype that combines embedded sensing, voice interaction, fall/reminder support, cloud AI, and OTA firmware updates into a wearable assistive device.

The project was inspired by the need for lightweight assistive technology that can provide contextual help without requiring the user to hold a phone or laptop. Glasses are naturally positioned near the user's eyes and ears, making them useful for visual context, voice input, and spoken feedback.

Internet connectivity augments the MCU by moving heavy computation and coordination to the cloud. MQTT carries small status/control messages, HTTP carries media and firmware payloads, Node-RED provides the dashboard, and the cloud worker handles speech-to-text, text/vision reasoning, text-to-speech, and file hosting.

Device Functionality

Voice QA: short press records audio, uploads WAV, receives a cloud reply, downloads TTS audio, and plays it.
Vision request: long press records audio and captures/uploads image context before speaking the cloud answer.
Fall/reminders: IMU events and cloud reminders trigger local prompts and dashboard-visible state.
OTA: Node-RED triggers firmware update; MCU downloads `.rps`, reboots, and reports the new version.

Mind2Matter wearable, firmware, cloud, and dashboard block diagram — System-level block diagram for the wearable prototype, MCU firmware, cloud stack, and dashboard.

Challenges

The hardest issues came from combining Simplicity Studio configuration, constrained MCU memory, Wi-Fi/MQTT/HTTP reliability, camera UART handshakes, microphone/I2S timing, cloud orchestration, custom PCBA flash timeout risk, and rushed casework.

Prototype Learnings

We learned how tightly coupled hardware and software are in embedded systems: sampling rate, buffering strategy, timing, driver lifecycle, and communication protocols all changed real system behavior.

Next Steps

The next version should improve state machines, reconnection logic, memory ownership, timeout/retry behavior, custom PCBA bring-up, and casework fit/clearance testing.

Node-RED Dashboard Cloud API Health Altium 365 PCBA GitHub Pages Website

3. Hardware & Software Requirements

Final validation snapshot

The final validation reflects the integrated wearable prototype, the custom PCBA design target, and the SiLabs BRD2708A firmware fallback used to keep the full cloud and OTA workflow demonstrable despite PCBA flash timeout risk.

IMU Loop20 ms

Meets 50 Hz target.

PTT Start120-180 ms

Under 300 ms target.

Image Upload3.8 s median

Under 10 s target.

Vision TTS7.6 s median

Under 20 s target.

Hardware requirements

ID	Requirement	Validation Result	Status
HRS-01	Use SIWG917Y121MGABA as MCU/wireless IC.	Firmware ran on BRD2708A / SiWx917 and reached MQTT broker.	Dev-board substitution
HRS-02	Include 3-axis accel and 3-axis gyro IMU.	I2C IMU sampled every 20 ms with decoded accel/gyro reads.	Met
HRS-03	Include at least 640x480 still camera.	Camera path worked for cloud upload; UART reliability still needs hardening.	Met
HRS-04	Include user button for PTT and alert acknowledgement.	Short press QA, long press vision, and fall cancel all worked.	Met
HRS-05	Include audio actuator for prompts and alerts.	Local tones, reminders, and downloaded WAV replies were audible.	Met
HRS-06	Use single-cell Li-ion battery.	Custom PCBA targets single-cell Li-ion power.	Met
HRS-07	Include USB-C or USB-5V charging/protection.	Charging/protection circuitry is included in the board design target.	Met
HRS-08	Provide required regulated rails.	PCBA + battery powered MCU, IMU, camera path, and audio path during integrated demo.	Met
HRS-09	Include microphone sampling at >= 16 kHz.	Firmware captured 16 kHz, 16-bit mono WAV audio.	Met
HRS-10	Wearable eyeglasses form factor.	Prototype was mounted into eyeglasses-style casework for final demo.	Met

Software requirements

ID	Requirement	Validation Result	Status
SRS-01	Sample IMU at >= 50 Hz.	20 ms task period, about 50 Hz observed over 5-minute run.	Met
SRS-02	Fall trigger within <= 2 s.	8/10 simulated traces triggered; median 1.18 s, max success 1.62 s.	Met
SRS-03	Prompt on fall and alert after 30 s if not cancelled.	Prompt immediate; no-cancel trial escalated after about 30.2 s.	Met
SRS-04	Apply reminders/settings within <= 10 s.	20 MQTT commands applied with 0.42 s median and 1.8 s max.	Met
SRS-05	Start PTT capture within <= 300 ms.	Typical start latency was 120-180 ms after debounced press.	Met
SRS-06	Reminder audio within <= 10 s.	10/10 reminder tests triggered within 1.3 s.	Met
SRS-07	Acknowledge/cancel within <= 5 s.	Local ack under 0.2 s; MQTT ack usually visible under 1 s.	Met
SRS-08	Image capture/upload within <= 10 s.	6 normal trials completed in 3.8 s median and 6.6 s max.	Met
SRS-09	Vision TTS result within <= 20 s.	Controlled trials: 7.6 s median and 11.9 s max; Wi-Fi instability produced outliers.	Function met
SRS-10	Internet Q&A spoken answer within <= 15 s.	Mock/fast path completed under 1 s; full AI path usually completed near 2-3 s.	Met

4. Project Photos & Screenshots

Gallery

Final project photos, PCBA images, thermal captures, Altium screenshots, the public dashboard, Node-RED flow screenshots, the system block diagram, and the MCAD model are collected below.