What are app actions?

 

App Actions overviewbookmark_border

App Actions let users launch and control Android apps with their voice, using Google Assistant. Extend your app to Assistant with App Actions and enable your users to open your app by saying, "Hey Google, open ExampleApp." Assistant is able to open your apps with no development work required from you.

You use App Actions to offer deeper voice control of your apps to users by enabling them to use their voice to perform specific tasks in your app. If a user has your app installed, they can simply state their intent using a phrase that includes your app name, such as "Hey Google, order a pizza on ExampleApp." App Actions supports dozens of built-in intents (BII) that model the common ways users express tasks they want to accomplish or information they seek, like:

• Ordering food, booking rides, and other category specific actions
• Opening a feature of your app
• Querying for products or content, using in-app search

With App Actions, Assistant can proactively suggest your voice capabilities as shortcuts to users, based on the user’s context. This functionality enables users to easily discover and replay your App Actions. You may also suggest these shortcuts in your app, with the App Actions in-app promo SDK.

You enable support for App Actions by declaring <capability> tags in shortcuts.xml. Capabilities tell Google how your in-app functionality can be semantically accessed using BII, and enables voice support for your features. Assistant fulfills user intents by launching your app to the specified content or action. For some use cases, you can specify an Android widget to display within Assistant to fulfill the user query.

Beta: This page describes App Actions functionality you can add to your app today using the Android Shortcuts framework. This framework integration is in the Beta release stage. If you are maintaining a production implementation of App Actions, refer to the Actions.xml documentation.

App Actions are supported on Android 5 (API level 21) and higher. Users can only access App Actions on Android phones, and Assistant on Android Go does not support App Actions.

Note: App Actions, like all Actions that support built-in intents, trigger on a number of factors, including quality and relevancy to the user's request. Based on those factors, Google may exercise discretion in surfacing your Action in response to user requests.

Use Cases

Here are a few common reasons to use App Actions to extend your apps to Assistant:

• Display app information on Google surfaces. Provide Android widgets for Assistant to display, offering inline answers, simple confirmations and brief interactions to users without changing context.
• Launch features from Assistant. Connect your app's capabilities to user queries that match predefined semantic patterns (BII).
• Suggest voice shortcuts from Assistant. Use Assistant to proactively suggest tasks for users to discover or replay, in the right context.

How App Actions work

App Actions extend your in-app functionality to Assistant, enabling users to access your app's features by voice. When a user invokes an App Action, Assistant matches the query to a BII declared in your shortcuts.xml, launching your app at the requested screen or displaying an Android widget. You declare BIIs in your app using Android capability elements.

Note: The Built-in intents reference offers BIIs across several functional categories, or "verticals," making it simple for developers to enable voice control of their app by matching their app shortcuts to supported voice intents.

For example, you might provide a capability for ordering food in your app. When a user says, “Hey Google, order a pizza from ExampleApp,”, the following steps occur:

• Assistant performs natural language analysis on the query, matching the semantics of the request to the predefined pattern of a BII. In this case, the actions.intent.ORDER_MENU_ITEM BII is matched to the query.
• Assistant scans your shortcuts.xml for a capability declared for the ORDER_MENU_ITEM BII.
• Assistant generates an Android intent to launch the in-app destination of the request using information you provide in the <capability>. Assistant extracts the parameters of the query and passes them as extras in a generated Android intent.
• Assistant fulfills the user request by launching the generated Android intent. You configure the intent to launch a screen in your app, or to display a widget within Assistant.

Figure 1. Example App Actions user query flow.

Note: App Actions generate intents that are used to launch in-app destinations to fulfill user requests. An intent can specify either an explicit Android Activity, or an Android deep link that launches an Activity. Apps with navigation built around Android deep links may prefer to use these links to fulfill an action. For more information, see Create Shortcuts.xml.

After a user completes a task, you use the Google Shortcuts Integration Library to push a dynamic shortcut of the action and its parameters to Google, enabling Assistant to suggest the shortcut to the user at contextually relevant times. Using this library makes your shortcuts eligible for discovery and replay on Google surfaces, like Assistant. For instance, you might push a shortcut to Google for each destination a user requests in your ride sharing app for quick replay later as a shortcut suggestion.

Building App Actions

App Actions build on top of existing functionality in your Android app, and the process is similar for each App Action you implement. App Actions take users directly to specific content or features in your app using capability elements you specify in shortcuts.xml.

When you build an App Action, the first step is to identify the activity you want to allow users to access from Assistant. Then, using that information, find the closest matching BII from the App Actions BII reference.

BIIs model some of the common ways that users express tasks they want to do using an app or information they seek. For example, BIIs exist for actions like ordering a meal, booking a ride, and checking an account balance. BIIs are the best way to start with App Actions as they model common variations of user queries in multiple languages, making it easy for you to quickly voice enable your app.

Once you identify the in-app functionality and BII to implement, you add or update the shortcuts.xml resource file in your Android app that maps the BII to your app functionality. App Actions defined as capability elements in shortcuts.xml describe how each BII resolves its fulfillment, as well as which parameters are extracted and provided to your app.

Note: For cases where there isn't a BII for your app functionality, you can use a custom intent to extend your app with App Actions.

A significant portion of developing App Actions is mapping BII parameters into your defined fulfillment. This process commonly takes the form of mapping the expected inputs of your in-app functionality to a BII's semantic parameters.

Note: If your app exports its activities by allowing other on-device apps to invoke the exported activity, Assistant may automatically use these intents to bootstrap your app's support for Assistant queries. You can override this behavior by implementing your own App Actions as covered in this section or opt-out using the App Actions support form.

Testing App Actions

During development and testing, you use the Google Assistant plugin for Android Studio to create a preview of your App Actions in Assistant for your Google account. This plugin helps you test how your App Action handles various parameters prior to submitting it for deployment. Once you generate a preview of your App Action in the test tool, you can trigger an App Action on your test device directly from the test tool window.

Next steps

Follow the App Actions codelab to build an App Action using our sample Android app. Then, you can continue on to our guides to build App Actions for your own app. You can also explore these additional resources for building App Actions:

• Download and explore our sample fitness Android app on GitHub.
• r/GoogleAssistantDev: The official Reddit community for developers working with Google Assistant.
• If you have a programming question about App Actions, submit a post to Stack Overflow using the "android" and "app-actions" tags. Before posting, ensure your question is on topic and that you've read the guidance for how to ask a good question.
• Report bugs and general issues with App Actions features in our public issue tracker.