跳到正文
DEV Community· Sohaib·· 7 小時前AI 評分38

認識 BetterZoom:具備智能分析功能的自我診斷應用程式

Meet BetterZoom: the only self diagnostic app with smart rowdy intelligence needed

AI 導讀

BetterZoom 是為 Presage SmartSpectra SDK Challenge 打造的應用程式,提供會議模式及個人練習模式,協助用戶留意情緒與身體訊號。會議模式每秒更新一次參與者狀態、每 25 秒向主持人提供數據摘要;個人模式每 30 秒把數據交予 Gemini,提供即時語音指導。分享須經參與者同意,AI 不會接收姓名或影片。

正文

Sohaib

This is a submission for the Presage SmartSpectra SDK Challenge
 

What I Built

For the SmartSpectra SDK Challenge, I built an app that might make Zoom, google meet obsolete forever considering the office meetings are highly unproductive due to managers fail to keep tabs on their juniors/team members & ultimately failing to track their emotional/mental state; results in dead productivity. My app features 2 modes: self mode which is uniquly built to train/regulate your ownself for tough situations. The next feature is main highlight (Meetings) which would allow a host to combine the power of SDK monitoring with use Artifical Intelligence to be much more aware about their team mates.

Meeting Mode (main highlight)

The problem

  • Meetings eat up a huge share of the working day, and most people think they don't work. In a Harvard Business School survey of 182 senior managers, 71% said meetings are unproductive and inefficient, and 65% said meetings keep them from finishing their own work.
  • In Atlassian's 2024 State of Teams survey of 5,000 knowledge workers, 78% said they struggle to get their work done because of how many meetings they have.
  • A big reason: the host can't watch everyone at once. With 5 other faces on screen, it's easy to miss who has switched off, who is nervous to speak, or who is overwhelmed. The meeting slowly loses its purpose.

What BetterZoom does about it

  • Every participant who agrees is measured on their own laptop: pulse, breathing, HRV and face signals (looking at the screen, talking, blinking, expressions).
  • The host sees a live one-word state next to each person's video, updated every second: Calm, Steady, Tense or Anxious. No charts, no clutter, unless the host wants them.
  • Every 25 seconds, a small summary of numbers (never video) goes to the host. It covers attention, talking time, expressions and heart rhythm.
  • The host taps the bot on someone's card, and Smokey writes a 3–4 line summary in seconds: how that person seems now, what changed over the meeting, and a gentle suggestion like "invite their thoughts" or "give them a moment." He can read it out loud.
  • One tap for the full picture: a report window with trend charts for pulse, breathing and HRV, attention signals and expressions.
  • Before leaving, the host can download a .md report card for every participant, or a CSV of all readings.

Built on consent, not surveillance

  • Nobody is measured unless they say yes. Everyone sees a "Reporting to host" badge on anyone who is sharing, and anyone can stop with one click.
  • The AI only ever sees "P1, P2…", never names or video. Everything is wiped when the host leaves the meeting.

In numbers: up to 6 people per meeting · live state every 1 s · reports every 25 s · AI summary on demand in 3–4 lines · 0 videos uploaded or stored.

Self Mode

  • Built for emotional and mental regulation for tense situation preparation: Job interviews, Public speaking, or Startup Pitches. Even managers can use this feature to prep before hoping on a call or briefing about something important to their bosses.

  • Monitors your facial expression, registers your emotional state and uses Smart Intelligence system (Smokey Bot referrence in the video) to help you navigate through the situation.

  • Smokey turns your signals into coaching. Every 30 seconds he gets a snapshot of your pulse, breathing rate, HRV, calm level and expressions, compared with your own normal. He also knows what you're practising for.

  • He reads the patterns. Fast breathing and a rising pulse usually mean nerves, which is common in a job interview. He spots that and gives you a quick fix, like "inhale for four seconds, exhale for six" or "drop your shoulders and pause before answering." Calm numbers? He tells you to keep that pace.

  • A real example: in one of my runs my breathing jumped to about 30 breaths a minute while I talked. Smokey caught it within half a minute and walked me through slowing it down.

Demo

SDK fetching readings

Readings on the screen
 

Code

Github Repo: https://github.com/Msohaib-maker/SneakMeet

Note: My Github repo includes all the necessary intructions to install & use the Windows App.

How It Responds to Human Signals

We have SDK integrated in both of our app modules: self mode and Group Mode. The SDK provides us the following readings:

Reading What it means
Pulse Compared with your own resting pulse, not an average person's. "↑ a bit higher than usual" means something.
Breathing rate The strongest sign of nerves in my testing. Fast breathing pushes the calm meter towards Tense.
HRV (plus average beat gap, SDNN and stress index) Shown once you've sat still for about a minute. Higher variability usually means more relaxed.
Breathing waveforms (chest and belly) Drawn live, and used to tell whether you breathe mostly from your chest or your belly.
Pulse waveform Drawn live, one peak per heartbeat.
Expressions, blinking, talking Your main expression appears as a meme-style sticker on your video. Talking tells the app to ignore breathing for a moment, because speech throws breathing readings off.

How the experience reacts in real time

Self mode (practising for an interview, pitch or talk):

  1. For the first ~30 seconds the app quietly learns your normal pulse and breathing.
  2. After that, a calm meter (Calm → Steady → Tense → Anxious) moves as your body reacts, with the reason shown underneath: "Pulse 86, 8 above your baseline."
  3. Every 30 seconds the numbers go to Gemini, and Smokey (my AI host) answers in character and out loud. In one of my runs my breathing climbed to around 30 breaths a minute while I talked. Smokey noticed and told me: "Slow your breathing right down by inhaling for four seconds and exhaling for six."
  4. At the end you get a debrief: how your calm level changed over the session, strengths, things to work on, and a downloadable report.

Meeting mode:

  1. Participants who agree to share get a live one-word state next to their video: Calm, Steady, Tense or Anxious. Only the host sees it.
  2. Every 25 seconds a small summary (numbers only) goes to the host.
  3. The host taps the little bot on someone's card, and Gemini writes a 3–4 line summary of how that person seems and what changed. Smokey can read it out.

How I integrated the SDK

  • BetterZoom is an Electron desktop app using @smartspectra/node-sdk. The same camera stream feeds both the video call and the SDK (useMediaStream), so nobody needs two cameras.
  • I request three groups of readings: breathing (rate plus chest/belly traces), heart (pulse, pressure waveform, HRV) and face (blinks, talking, expressions). The SDK sends roughly 28 updates a second, which I decode with decodeMetrics.
  • I only trust readings the SDK is confident about: below 45% for pulse or 40% for breathing, the app shows the number faded with a "weak signal" label instead of guessing.
  • The SDK's camera checks (face too low, chest not visible, frame rate too low) appear on screen as plain tips, so people can fix their setup themselves.
  • The calm meter is my own logic on top of the SDK. It compares pulse and breathing with your personal baseline, smooths them out, and ignores breathing while you talk.

My honest experience

After about 90 credits of testing:

  • Expressions and breathing came through well. Most of the time the SDK reported around 68–73% confidence for me, and the readings matched how I actually felt.
  • Pulse usually appears after 15–25 seconds. I'm not a medical expert, so I can't judge how accurate it is, but it stayed in a believable range and reacted when I got nervous.
  • Expressions are not perfect. The SDK sometimes read me as neutral or sad when I was actually tense.
  • Breathing needs your chest in view, and HRV needs about a minute of sitting still. That's rare while you're talking, so HRV often didn't show up during practice.
  • Frame rate is the big one. The SDK needs 25+ fps and stops itself if the camera can't keep up. At first my app was the problem: redrawing the screen on every update and animating Smokey starved the SDK, and it stopped after about 15 seconds. Throttling the UI, freezing animations while measuring, and sending lighter video in meetings took my runs from stopping early to 5 minutes without a break. Dim rooms and power-saving laptops can still cause it, because webcams lower their frame rate in bad light.
  • Some readings listed in the SDK (breathing amplitude, apnea, inhale/exhale ratio) aren't available yet. Requesting them makes the SDK refuse to start, so I left them out.

The 68–73% figure and the expression observations are yours, so I've kept them as your experience. Everything else comes from the logs of our test runs.

Prize Categories

I am going to say I would personally put my app in Thoughful Category with a bit of creativity. Office meetings as I mentioned earlier woul have been the best use case for my application given the API has high amount of credits. Given SDK can be used to monitor particpants' expressions and state of comfort would help the team managers/HRs to be better decision makers. We might be looking forward to see many people never mad to be part of meetings because of their state of comfort or we could be looking to companies adapting to new meet policies which ultimately get them better meet profits.

Consent and Data Handling

Camera consent

  • BetterZoom only uses your camera and microphone after Windows asks for your permission. No other page or website can use them.
  • Readings are off until you start them yourself: by pressing Practice solo, Measure vitals or Start scan.
  • In meetings, nothing is shared by default. When a meeting has a host, every participant sees a consent popup first. It names the host and explains what will be measured, what is sent, how often, and that an AI may summarise it. You can say no and still stay in the meeting.
  • While someone is sharing, everyone in the meeting sees a "Reporting to host" badge on their video. They can stop at any time with one click. There is no hidden or background monitoring.

What is measured and where it goes

  • All readings (pulse, breathing, HRV, waveforms, face signals and expressions) are calculated on your own computer by the SmartSpectra SDK. Video frames are never uploaded or recorded.
  • Self mode: your readings stay on your laptop. Only numbers go to the AI (see below).
  • Meetings: video and audio go directly between participants (peer-to-peer, encrypted by WebRTC). A participant's readings go only to the host, and only if they agreed: a short update every second, plus a summary every 25 seconds. These are numbers only, never images.
  • Our server (on Fly.io) helps people connect and passes AI requests on to Gemini. It never sees video and stores nothing.

AI (Google Gemini)

  • Smokey's coaching and summaries are written by Google Gemini. The app sends only numbers (pulse, breathing, calm level, expression scores), never video.
  • In meetings, participants are sent as P1, P2…, never by name. The real name is added back on the host's computer.
  • On Gemini's free tier, Google may use this data to improve its products. The consent popup says so.
  • Smokey's voice uses the voices built into Windows, which work offline. No audio is sent anywhere.

Storage, retention and deletion

  • Nothing is stored on a server. In meetings, reports and AI summaries live only in the host's app memory and are deleted when the host leaves the meeting or closes the app.
  • Files (the Self mode debrief, report cards, CSV) are saved only if you click download. They are then your files to keep or delete.
  • The app keeps only your settings on your computer. Uninstalling the app removes them.

SDK telemetry

Not medical

  • All values are wellness estimates. BetterZoom is not a medical device, makes no diagnosis, and must not be used for medical decisions. Labels like "Calm" or "Tense" are estimates from camera readings, not facts about how someone feels.

Looking forwars to the results for SmartSpectra SDK challenge.

來源:DEV Community · dev.to