Beyond Object Detection: Building SpotQuest with Jev’s Vision API
TL;DR: I built an online scavenger-hunt game called SpotQuest, that leverages TypeSafe's Jev model behind the scenes for visual inferences. This blog dissects the app and explores the design choices made.
A few weeks back, when TypeSafe first introduced Jev, I was amazed by the possibilities that it could hold when it comes to natural language-based classification decisions at scale. It was a text-only model; meaning that while it could go through textual content to make certain calls, multimodal support was not available. Recently, I noticed that the Jev API has support for images in what TypeSafe today calls 'dreamer mode' (read: non-production). With the base model already having impressed me, I wanted to deep-dive into what we could unlock with the new images feature.
That prompted me to build a virtual scavenger-hunt game: SpotQuest. SpotQuest is an app where you are presented a task to show something that could be near you: an object, a pattern or a reference. You simply point your camera feed at it and click 'analyze': once the model finds credible evidence of the artefact, it scores you and you move on to the next mission. There are a total of ten challenges: at the end, you have your own performance postcard to share. It's as simple as that!
Foundations: Getting in shape
The idea was to build an interactive application. It is built with React for the frontend and Express as the backend. The backend was necessary as we had to read the environment secret for the API key for Jev.
I usually prefer to use Claude Design to generate the wireframe and then proceed for the rest of the implementation. It gives a good overview without having everything tied together; and IMHO it's a lot cleaner than generic frontend applications that are one-shotted by the frontier models today. So I began, generating the first prototype: entirely local, using an in-memory API key obtained from the player.
Once the design was ready, I exported the design files and began the implementation phase. The idea was to have:
A challenge bank: Graded across Easy/Medium/Hard, the challenge bank would contain items that the player would have to find. The level would be set by the user at the start, there is also a 'Progressive' mode where the user moves across Easy, Medium and Hard challenges.
An analyzer: The central 'decision maker' for the app; this module would analyze whether the captured feed was in line with the requirements of the challenge presented. For example, if the challenge was to find a dry twig, and the image had a coffee mug in it, the challenge would be considered unsuccessful. It has to be smart enough to adapt to the varied requirements per the difficulty levels.
Scorer: For each mission conquered, a scorer would aggregate 'points', and ultimately present them at the end. The user would then have an option to share a score 'postcard', a digitally-generated banner representing the user's progress and conferring a game title on the player.
Building the Analyzer with Jev Images
A request builder module structures the API call to Jev. It interpolates the prompt by tying together the challenge with the rest of the request body that the System-1 model expects. Given below is a snippet of the same in action, for one random challenge. Jev returns a probability that visible evidence satisfies the challenge (noul), a 'box' to determine the locale of the detected item that fits the bill, and the score awards challenge base points plus match-band and bounding-box-confidence bonuses.
// The challenge supplies a description of what the player must find.
const prompt = challenge.localizationPrompt;
const imageRef = `vision-${crypto.randomUUID()}`;
const request = {
model: "jev-preview",
state: { scene: imageRef },
assets: {
[imageRef]: { type: "image", part: "vision-frame" },
},
questions: {
presence: {
type: "noul",
instructions: `Does the visible evidence in \`scene\` satisfy: ${prompt}?`,
criteria: {
true: "Visible evidence clearly satisfies the challenge.",
false: "No visible evidence satisfies the challenge.",
},
},
target: {
type: "box",
instructions: `Locate the strongest visible match in \`scene\` for: ${prompt}`,
},
match: {
type: "score",
instructions: `How well does the visible evidence in \`scene\` satisfy: ${prompt}?`,
criteria: [
"No match.",
"Weak or unclear match.",
"Reasonable match.",
"Strong match.",
"Unmistakable match.",
],
},
category: {
type: "choice",
instructions: `Classify the strongest match in \`scene\` for: ${prompt}`,
criteria: {
direct_match: "Directly satisfies the challenge.",
functional_match: "Performs the required function.",
contextual_match: "Fits because of the visible context.",
ambiguous_match: "Evidence is unclear.",
no_suitable_match: "Nothing suitable is visible.",
},
},
},
};
This request is subsequently processed by Jev and a structured JSON is returned, which is parsed by the React frontend. If the noul returns a confidence threshold higher than a 'gate' score, the player moves on for the next mission and a score is assigned. Else, the player has the option to skip the mission altogether and look for the next item.
Building the Gameflow
Before play, SpotQuest explains that submitted images pass through its server to TypeSafe for analysis. TypeSafe’s handling of those images is governed by its policies; players should avoid submitting sensitive information or other people’s images without consent.
The game begins by selecting a difficulty mode. There are four options available: Easy, Medium, Hard and Progressive. The easy tier focuses on objects alone, the medium tier around indirect object references that the user has to understand, and the hard tier revolves around abstract patterns or arrangements. The progressive tier blends the modes as the user proceeds.
Mode choice: Easy, Medium, Hard or Progressive?
Under the hood, each of the modes correspond to their set of challenges in the common 'bank', there are a total of ten challenges that the player will be presented with. These are randomized by default and no challenge repeats itself in the set of ten picked. Once a mode is determined, the game begins.
The user is presented with a choice, and must either use the camera feed (recommended), or upload an image (I trust you folks: no cheating!). Once they click 'Investigate', the image data along with the composed request body shown earlier is sent for processing, and the response JSON is broken down into frontend components for the user to perceive. Attached below is a snippet of a demo run.
Notice how the analyzer correctly 'boxes' the expected output (a door handle), determines the match relevance and scores the user for the task at hand. All of this happens in near real time, with sub-second latency for the API call. Each investigation sends a single still image rather than a video stream. Camera captures are JPEG-encoded, and uploads are limited to JPG or PNG files up to 12 MiB.
As the game proceeds, and the challenges are attended to one after the other, there comes a point where the challenges end. After ten tasks are presented to the user, a postcard is instantaneously generated that contains the progress made. This can be digitally shared of course with native share integrations built into the application.
Thoughts on efficiency, economics and alternatives
I've been around a fair while to know that object detection is nothing new to cheer about. A conventional vision pipeline using OpenCV or a detector such as YOLO could handle many object and geometry tasks. Jev lets me express a wider range of semantic challenges in natural language without building task-specific detection and rule pipelines.
Why did I go with Jev for this use-case? Two reasons, primarily.
Complex challenges that go beyond one single object: think of a task like 'Find an item that is out of place in a work desk', requires intelligence to determine what fits and what doesn't fit in the relevant context without having to explicitly define options in the backend to match against. That would be limitless as the challenge back scales otherwise! Or, cases like 'arrange four pencils in a square' is objectively hard to determine with traditional CV models that a modern SOTA decision model like Jev.
Cost! Jev is extremely cost efficient for decision making. Clocking $0.042/M tokens (official rates), it provides a much faster inference and better unit economics per individual request. While YOLO can be batched on the same container, it would translate to a higher CPU and memory usage, leading to higher operational costs on a pay-as-you-go basis.
Given below is a snapshot of the usage dashboard for Jev, totalling $0.008 for around 213,000 tokens (~0.038/1M, close to the published rate).
Next steps and further improvements
SpotQuest is now a standalone application that anybody can play on any compatible, camera-enabled device. One of the ideas I've had is to personalize this further by letting the user describe their environment first: an LLM then formulates ten challenges, which is then adjudged by Jev as usual. This would provide for a more personalized experience while also boosting success rates.
The other idea is to make the progress shareable, but then that would necessitate user-level personas to be maintained and authentication to be enabled, which would complicate flows. I may pursue this if there's some demand.
The app is currently living on the Azure App Service with CI/CD enabled from GitHub, where the code stays. It's also externally exposed at spotquest.idipankan.com (managed via Azure DNS). I've also published it as an image on the Docker Hub, so one could pull it off the internet and run it locally should they wish too (with their own Jev keys, of course!). Here's the Docker hub link.
It was fun building this not just for the sake of discovering Jev's vision capabilities, but also a fun de-stressing game that can be quite competitive!