» Be assumption that we will add custom clientlibs in helloworld component. The structure can be designed as below:

» If we create clientlib-author-dialog in CRXDE | Lite dashboard, we will get .content.xml was generated. But, if we create from IntelliJ or any IDE tool. We will need to add .content.xml as manually.

<?xml version="1.0" encoding="UTF-8"?>
<jcr:root xmlns:cq="http://www.day.com/jcr/cq/1.0" xmlns:jcr="http://www.jcp.org/jcr/1.0"
    jcr:primaryType="cq:ClientLibraryFolder"
    categories="[cq.authoring.dialog.all]"/>

Note: You need to define properly configuration cq.authoring.dialog.all when create folder named clientlib-author-dialog.

In while, if you want to apply javascript for only the component. We will replace prefix .all by .<id-aem-component>. For example, here is cq.authoring.dialog.helloworld.

» Here, we will create file Javascript named mycustom.js and add into js folder and also update in js.txt as well.

» Content of js.txt as below:

#base=js
mycustom.js

» To get started javascript in cq:dialog, we will need to use granite:class to define custom class and add into cq.dialog. Thus, our dialog looks like as below:

<?xml version="1.0" encoding="UTF-8"?>
<jcr:root xmlns:sling="http://sling.apache.org/jcr/sling/1.0"
          xmlns:cq="http://www.day.com/jcr/cq/1.0"
          xmlns:jcr="http://www.jcp.org/jcr/1.0"
          xmlns:nt="http://www.jcp.org/jcr/nt/1.0"
          xmlns:granite="http://www.adobe.com/jcr/granite/1.0"
    jcr:primaryType="nt:unstructured"
    jcr:title="Properties"
    sling:resourceType="cq/gui/components/authoring/dialog">
    <content
        jcr:primaryType="nt:unstructured"
        granite:class="cq-dialog-helloworld-comp"
        sling:resourceType="granite/ui/components/coral/foundation/fixedcolumns">
        <items jcr:primaryType="nt:unstructured">
            <column
                jcr:primaryType="nt:unstructured"
                sling:resourceType="granite/ui/components/coral/foundation/container">
                <items jcr:primaryType="nt:unstructured">
                    <text
                        jcr:primaryType="nt:unstructured"
                        sling:resourceType="granite/ui/components/coral/foundation/form/textfield"
                        fieldLabel="Text"
                        name="./text"/>
                </items>
            </column>
        </items>
    </content>
</jcr:root>

» That is mycustom.js

$(document).on("foundation-contentloaded", function(e) {
   console.log($('.cq-dialog-helloworld-comp'));
});

» Open the console in Chrome, use this keyboard shortcut: "Cmd + Option + J" (on a Mac) or "Ctrl +Shift +J" (on Windows)

Note: You need to pay attention created class named cq-dialog-helloworld-comp. We will look at in Inspect Element window. 

» How can we get value from Text field

(function(document, $) {
    "use strict";

    $(document).on("foundation-contentloaded", function(e) {
       let textField = $('.cq-dialog-helloworld-comp').find('._coral-Textfield');
       console.log(textField.val());
    });

})(document, Granite.$);

Note: foundation-contentloaded will be triggered upon dialog is opened. In terms of dialog off, the value return null or undefined. Hence, you can improve code as below:

You can refer another way to approach clientlib in specific AEM component. Here is link official document.