• English

      • HarmonyOS

      Midscene connects to HarmonyOS NEXT devices through HarmonyOS Device Connector (HDC) to automate apps and system interfaces.

      This guide covers device connection, model configuration, Playground, and JavaScript SDK integration with @midscene/harmony.

      See it in action

      Prompt: Open Settings, find About phone, and view the device information.

      View the full report, or explore more Midscene showcases.

      Get started

      Prepare your HarmonyOS device

      Before writing scripts, verify that HDC can connect to your device and the device trusts the current computer.

      Install HDC

      HDC (HarmonyOS Device Connector) is a command-line tool provided by HarmonyOS for communicating with devices. Installation options:

      Verify HDC is installed:

      hdc version

      A version number in the output confirms successful installation.

      Configuring HDC Path

      If hdc is not in your system PATH, set the HDC_HOME environment variable to the directory containing HDC:

      export HDC_HOME=/path/to/hdc/directory

      Enable Developer Mode and verify the device

      In your HarmonyOS device settings, go to Developer Options and enable USB Debugging, then connect via USB cable.

      Verify the connection:

      hdc list targets

      A device ID in the output confirms a successful connection:

      0123456789ABCDEF

      Launch Playground

      Playground is the fastest way to validate the connection and try core capabilities such as aiAct, aiQuery, and aiAssert without writing code. It shares the same core as @midscene/harmony, so anything that works here will behave the same once scripted.

      1. Launch the Playground CLI:
      npx --yes @midscene/harmony-playground
      1. Click the gear button in the Playground window and paste your API Key configuration. See Supported models and setup if you still need a model configuration.

      Use the JavaScript SDK

      Once Playground runs successfully, you can switch to reusable JavaScript scripts.

      Configure the model

      Set the model configuration through environment variables. For supported models and copyable setup examples, see Supported models and setup.

      export MIDSCENE_MODEL_BASE_URL="https://replace-with-your-model-service-url/v1"
export MIDSCENE_MODEL_API_KEY="replace-with-your-api-key"
export MIDSCENE_MODEL_NAME="replace-with-your-model-name"
export MIDSCENE_MODEL_FAMILY="replace-with-your-model-family"

      For all configuration options, see Model configuration.

      Install dependencies

      npm
      yarn
      pnpm
      bun
      deno
      npm install @midscene/harmony dotenv --save-dev

      Write a script

      The following example opens the Settings app on the device and performs scrolling operations.

      ./demo.ts
      import 'dotenv/config'; // load Midscene environment variables from .env if present
import {
  HarmonyAgent,
  HarmonyDevice,
  getConnectedDevices,
} from '@midscene/harmony';

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
Promise.resolve(
  (async () => {
    const devices = await getConnectedDevices();
    const device = new HarmonyDevice(devices[0].deviceId, {});

    const agent = new HarmonyAgent(device, {
      aiActionContext:
        'This is a HarmonyOS device. The system language is Chinese. If any popup appears, dismiss or agree to it.',
    });
    await device.connect();

    // Open Settings app
    await agent.launch('com.huawei.hmos.settings');
    await sleep(2000);

    // Scroll down
    await agent.aiAct('scroll down one screen');

    // Query page content
    const items = await agent.aiQuery(
      'string[], list all visible setting item names',
    );
    console.log('Settings items', items);

    // Assert
    await agent.aiAssert('There is a settings item list on the page');
  })(),
);

      Run the script

      npx tsx demo.ts

      After the script finishes, you should see Midscene - report file updated: /path/to/report/some_id.html in the console. Open the generated HTML file in a browser to replay every interaction, query, and assertion.

      Advanced

      Use this section to customize device behavior, integrate Midscene into a standalone framework, or troubleshoot HDC issues. See the HarmonyOS section of the API reference for more constructor parameters.

      Extending Midscene on HarmonyOS

      Use defineAction() to define custom actions. When constructing HarmonyDevice, pass these actions through customActions. Midscene appends these actions to the planner so the Agent can call the domain-specific actions you define.

      import { getMidsceneLocationSchema, z } from '@midscene/core';
import { defineAction } from '@midscene/core/device';
import { HarmonyAgent, HarmonyDevice, getConnectedDevices } from '@midscene/harmony';

const ContinuousClick = defineAction({
  name: 'continuousClick',
  description: 'Click the same target repeatedly',
  paramSchema: z.object({
    locate: getMidsceneLocationSchema(),
    count: z.number().int().positive().describe('How many times to click'),
  }),
  async call(param) {
    const { locate, count } = param;
    console.log('click target center', locate.center);
    console.log('click count', count);
  },
});

const devices = await getConnectedDevices();
const device = new HarmonyDevice(devices[0].deviceId, {
  customActions: [ContinuousClick],
});
await device.connect();

const agent = new HarmonyAgent(device);

await agent.aiAct('click the red button five times');

      For more details on custom actions and action schemas, see Integrate with Any Interface.

      FAQ

      Keyboard is not dismissed or the page goes back after typing

      Midscene automatically dismisses the keyboard after entering text. By default, HarmonyOS uses the ESC key so the current page is less likely to navigate back. If ESC does not close the keyboard in your app, switch to Back first:

      const device = new HarmonyDevice('device-id', {
  keyboardDismissStrategy: 'back-first',
});

      If your input field listens for Back and clears or closes in response, disable auto keyboard dismiss:

      const device = new HarmonyDevice('device-id', {
  autoDismissKeyboard: false,
});

      With auto dismiss disabled, the keyboard will remain visible. You can use aiAct to manually dismiss it, e.g. await agent.aiAct('dismiss the keyboard').

      How to use a custom HDC path?

      Set the HDC_HOME environment variable to point to the HDC directory:

      export HDC_HOME=/path/to/hdc/directory

      Or pass it via the constructor:

      const device = new HarmonyDevice('0123456789ABCDEF', {
  hdcPath: '/path/to/hdc',
});

      More