#Model Strategy

Midscene's model strategy serves two goals. First, it uses pure vision to understand the interface that users actually see, keeping automation independent of the rendering stack. Second, it supports flexible single-model and multi-model configurations so users can choose a suitable combination for their business requirements.

#A pure-vision approach based on the visible interface

AI-powered UI automation requires task planning and element localization. The industry mainly uses two localization approaches: combining DOM data with annotated screenshots, or using screenshots directly for pure-vision localization. Midscene uses pure vision, with the model analyzing the UI screenshot and locating target elements directly. UI actions and element localization do not depend on DOM data or extra annotations.

This choice makes the visible interface the source of truth for automation and gives Midscene several advantages:

• It uses a consistent approach across browser Canvas, Android, iOS, desktop applications, and other types of interfaces.
• It can validate what users actually see, including colors, highlighted states, and layout.
• It is independent of the UI rendering stack and does not require selectors or extra UI annotations.
• Token consumption depends only on page resolution and task complexity, and does not inflate as the page structure (such as the number of DOM elements) grows.

Pure vision is not a local optimization for one platform. It is the shared foundation of Midscene's cross-platform interaction capabilities. The same task descriptions and interaction patterns can extend across interfaces and devices while staying close to how real users operate software.

The vision-based approach also has clear limitations. Pure-vision localization requires models with visual understanding capabilities — only designated models that are stable for GUI operations can be used, not any arbitrary LLM. Midscene accepts higher model capability requirements in exchange for cross-platform consistency and lower UI maintenance costs.

Data extraction and page-understanding workloads can still include DOM data when needed. See the API Reference for the relevant options.

#Single-model and multi-model configurations

Midscene always requires a Default model. It can independently handle task planning, element localization, page understanding, and the rest of the automation workload, or serve as the foundation of a multi-model combination. Depending on their business requirements, users can use only the Default model or add Planning and Insight models as needed. In a multi-model combination, the three model roles can work together in one automation task:

A multi-model combination can increase task latency and token usage.

#Next steps

This page focuses on Midscene's model philosophy and selection principles. For setup instructions, see Optional: configure multiple models. If task quality is unstable, use Model debugging and observability to identify the problem.