Modalities integration

From start to finish, set up a chat-based application with custom modalities with NLX

What's a Modality?

A modality allows for information to be transmitted to a user that goes beyond a standard message. Images, audio, structured text, haptic signals, videos, and more can be modalities you may want to provide users during conversation.

Checklist

You'll complete the following to successfully use a custom modality:

You may add the following to an existing chat application in your workspace, if desired.

Pre-setup: Integrations

Est. time to complete: ~5 minutes

A one-time integration of a Natural Language Processing (NLP) engine must be completed in your workspace:

Step 1: Set up a modality

Est. time to complete: ~1 minute

  • Select Resources from workspace menu > Choose Modalities

  • Select + Add modality

  • Enter "Video" as the name name

  • Select Schema and change the Type to Object

  • Select Properties and then + Add property

  • Enter "src" as the Field name

  • Click Create modality

Step 2: Link to a flow node

Est. time to complete: ~1 minute

You'll now enable the video modality you created on an end node (user-type node or node at the end of a flow), and provide it with a video URL.

  • Select Intents from workspace menu > Create or choose an existing intent

  • Create a new flow card or select one with the API channel set as channel > Click on any end node

  • Select Add functionality > Choose Modalities

  • Toggle the Enable on

  • Enter dQw4w9WgXcQ as the value for "src"

  • Click Save

Repeat for any other intent flows where you would like to enable your modality.

🧠 Looking for more? See Construct flows

Step 3: Set up & deploy application

Est. time to complete: ~10 minutes

  • Select Applications from workspace menu > Choose New application

  • Enter a descriptive name > Click Save

  • Click Intents tab of application > Select + Add intents > Attach one or more intents you've created to make available to your application

  • Select Default behaviors tab of application > Assign any attached intents to the application's behaviors > Click Save

  • Select Channels tab of application > Expand API option > Click + Create channel

  • Click + Add domain under Whitelisted domains > Add your webpage's URL and https://developers.nlx.ai

  • Click Create channel

  • Click Deployment tab of application > Select Create or Review & build

  • Wait for validation to complete > Select Create build*

  • When satisfied with a successful build, click Deploy

*After a build status appears as 🟢 Built, you may use the Test feature to test the conversation with your application using the latest build.

🔧 Need to troubleshoot your build? See how to troubleshoot build errors

Step 4: Set up a webpage

Est. time to complete: ~10 minutes

Using the NLX Chat SDK, you can quickly build an HTML page to test out your video modality. You'll need to copy the application URL and API key from the application's Deployment tab.

  • Resuming from Step 3, click on the Details link next to the Deployed status

  • Under Setup instructions, expand the API row

  • Copy the Application URL and API key

  • Copy the HTML file below and replace REPLACE_WITH_APPLICATION_URL and REPLACE_WITH_API_KEY with the values you just copied

  • Save the HTML file you just edited and serve the HTML file with your preferred command line tool such as serve

<html lang="en">
  <head>
    <title>NLX Widget Sample HTML</title>
    <meta name="viewport" content="width=device-width, initial-scale=1">
  </head>
  <body>
    <script defer src="https://unpkg.com/@nlxai/chat-widget/lib/index.umd.js"></script>
    <script defer src="https://cdnjs.cloudflare.com/ajax/libs/htm/3.1.1/htm.js" integrity="sha512-RilD4H0wcNNxG2GvB+L1LRXCntT0zgRvRLnmGu+e9wWaLKGkPifz3Ozb6+WPsyEkTBLw6zWCwwEjs9JLL1KIHg==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>
    <script>
      window.addEventListener("DOMContentLoaded", () => {
        // EMBEDDABLE COMPONENT SETUP
        // Destructure dependencies
        const React = nlxai.chatWidget.React;
        const useConversationHandler = nlxai.chatWidget.useConversationHandler;

        // Use the https://github.com/developit/htm package as a JSX alternative
        const html = htm.bind(React.createElement);
 
        Video = ({ data }) => {
          const videoSrc = `https://www.youtube.com/embed/${data.src}`;
          return html`
            <iframe
              width="264"
              height="160"
              src=${videoSrc}
              allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"
              allowFullScreen
            ></iframe>`;
        };

        const widget = nlxai.chatWidget.create({
          config: {
            applicationUrl: "REPLACE_WITH_APPLICATION_URL",
            headers: {
              "nlx-api-key": "REPLACE_WITH_API_KEY"
            },
            languageCode: "en-US"
          },
          titleBar: {
            "title": "Modalities"
          },
          // CUSTOM BEHAVIOR SNIPPET
          onExpand: (conversationHandler) => {
            const checkMessages = (messages) => {
              if (messages.length === 0) {
                conversationHandler.sendWelcomeIntent();
              }
              conversationHandler.unsubscribe(checkMessages);
            };
            conversationHandler.subscribe(checkMessages);
          },
          customModalities: { Video },
          // CUSTOM BEHAVIOR SNIPPET END
          theme: {
            "primaryColor": "#2663da",
            "darkMessageColor": "#2663da",
            "lightMessageColor": "#EFEFEF",
            "white": "#FFFFFF",
            "fontFamily": "-apple-system, BlinkMacSystemFont, \"Segoe UI\", Roboto, Oxygen-Sans, Ubuntu, Cantarell, \"Helvetica Neue\", sans-serif",
            "spacing": 12,
            "borderRadius": 8,
            "chatWindowMaxHeight": 640
          }
        });
      });
    </script>
  </body>
</html>

Step 5: Test the modality

To test the modality, either host your HTML file or serve it locally using a command line tool like serve.

Once you can access the page, click on the chat widget at the bottom right of the page and converse with your application until you reach the node in the conversation where you set the modality.

Last updated