So my idea was the following:<\/p>\n\n\n\n
- \n
- Train my own model based on all the support chat answers we\u2019ve given manually over the last 3 years<\/li>\n<\/ul>\n\n\n\n
- \n
- Don\u2019t reply automatically to customer automatically but rather give highly accurate suggestions<\/li>\n\n\n\n
- Ideally this will reduce support workload and make it easier to onboard new support agents<\/li>\n<\/ul>\n\n\n\n
So I started to dig into it\u2026<\/p>\n\n\n\n
Requirements<\/h1>\n\n\n\n
- \n
- Have a big chat history in Intercom: <\/strong>In order to fine tune a ChatGPT model you need data, a lot of data\u2026 the more the better. So if you\u2019re reading this article with the plan to implement this yourself you have to have a big data set. <\/li>\n\n\n\n
- A paid OpenAI account:<\/strong> Since we\u2019ll be using OpenAIs ChatGPT you\u2019ll need a paid account. A quick heads up: fine tuning a model isn\u2019t cheap, I\u2019ve spent about $6-10 per fine tuning job.<\/li>\n<\/ol>\n\n\n\n
Exporting your Intercom Chat History<\/h1>\n\n\n\n
Unfortunately there\u2019s no easy way to export and download your Intercom chat history. I was really hoping it would be as easy as signing into the admin panel and hitting a download button. <\/p>\n\n\n\n
After looking into this for a while I found a few GitHub repositories. Some of them came close to what I had in mind:<\/p>\n\n\n\n
- \n
- https:\/\/github.com\/TheArtling\/intercom-conversations-export<\/a>\n
- \n
- A Python script that exports your Intercom conversations as txt files<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- https:\/\/github.com\/toch\/intercom-export<\/a>\n
- \n
- A Ruby script to export your Intercom data<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n
I decided to give the Python script a try first. After fiddled around with it for a while I managed to export our existing conversations as txt files. Great!<\/p>\n\n\n\n
There was only a small problem: In order to fine tune the ChatGPT model later, we need to have the data as a JSONL file (yes, you read correctly, not JSON, but JSONL, see here: https:\/\/platform.openai.com\/docs\/guides\/fine-tuning<\/a> and https:\/\/jsonlines.org\/<\/a>). Additionally the data has to be in a prompt\/completion format. Meaning that we have to prepare the data further and potentially cut out subsequent conversations parts after the initial customer question was answered.<\/p>\n\n\n\n
Here\u2019s an example of the structure that we need to fine tune the model later:<\/p>\n\n\n\n
{\"prompt\": \"Do you offer a free trial?\", \"completion\": \"No, but we have a 30 day money back guarantee, so you can go ahead and purchase one of our plans and always get your money back afterwards if you\u2019re not happy.\"}\n\n{\"prompt\": \"Do you share any data with 3rd parties?\", \"completion\": \"No, we don\u2019t share any data with any 3rd party. Our terms and conditions state that by using our service you solely grant us the right to access the data for the purpose of the conversion and nothing else.\"}<\/code><\/pre>\n\n\n\nGiven that we need a different output format and also potentially have to clean the exported Intercom data, I decided to write my own JavaScript script to export all conversations from Intercom.<\/p>\n\n\n\n
I\u2019ve uploaded the Intercom export script here: https:\/\/github.com\/kgoedecke\/intercom-exporter-node<\/a><\/p>\n\n\n\n
Credit to @TheArtling, because my codebase is heavily influenced by their project.<\/p>\n\n\n\n
A few words about what this script does, this is important since the data we need is not directly what comes back from the API:<\/p>\n\n\n\n- \n
- It retrieves all conversations from your Intercom account<\/li>\n\n\n\n
- It then filters for only conversations that are customer initiated<\/li>\n\n\n\n
- It then removes all replies except for the VERY FIRST one, this is critical since we don\u2019t want the whole conversation<\/li>\n\n\n\n
- It then filters out empty replies and only keeps replies which were written by an admin\/support agent<\/li>\n\n\n\n
- Last step: It removes all HTML tags from the reply<\/li>\n<\/ol>\n\n\n\n
Let\u2019s go ahead and see what this nifty script can do for us in practice.<\/p>\n\n\n\n
In order to get started you need to get an Intercom API access token. Head over to the Intercom developer hub and get your token (see: https:\/\/developers.intercom.com\/building-apps\/docs\/authentication-types#access-tokens<\/a>).<\/p>\n\n\n\n
Once you\u2019ve got it, simply clone the Intercom Conversation Export repository from GitHub:<\/p>\n\n\n\n
git clone https:\/\/github.com\/kgoedecke\/intercom-exporter-node<\/code><\/pre>\n\n\n\nAfterwards copy the
.env.example<\/code> file to.env<\/code> and change the values in it.<\/p>\n\n\n\nFor example your .env file should look like this:<\/p>\n\n\n\n
INTERCOM_TOKEN = 2LDJwyV9S7HsKxAxLx7b2Yv3H0nSeuh='\n\nOUTPUT_PATH = '\/Users\/kevin\/Downloads'<\/code><\/pre>\n\n\n\nInstall all dependencies with
yarn<\/code> and then you can go ahead and simply run `yarn start` to get the conversation export process started. This should print out something like the following<\/p>\n\n\n\nyarn start \n\nyarn run v1.22.19\n\n$ node src\/index.js\n\nNext page URL: https:\/\/api.intercom.io\/conversations?starting_after=WzE2ODE5MjY5NjkwMDAsMTYyNTAzODAwMjMzMzg4LDJd\n\nRetrieving 2 of 485\n\nNext page URL: https:\/\/api.intercom.io\/conversations?starting_after=WzE2ODE4NDM3MjIwMDAsMTYyNTAzODAwMjMzNDc1LDNd\n\nRetrieving 3 of 485<\/code><\/pre>\n\n\n\nOnce it has finished you\u2019ll find a file called
output.jsonl<\/code> in the directory you\u2019ve specified in your.env<\/code> file.<\/p>\n\n\n\nGo ahead and open it to check what\u2019s in it.<\/p>\n\n\n\n
Here are a few lines of what our file looks like:<\/p>\n\n\n\n
{\"prompt\":\"Hey wondering if your program works for Keynote to Figma too?\",\"completion\":\"Hi there,\\n\\nThanks for reaching out. Unfortunately, we don't support Keynote to Figma conversion. Nonetheless, you can try our Powerpoint to Figma converter, since Keynote offers an option to export files as PowerPoint.\\n\\nI hope this information is helpful. Please don't hesitate to contact us if you have any further queries or concerns.\\n\\nBest regards,\\nLokman\"}\n\n{\"prompt\":\"Hi, the canva to figma converter appears to still be down. Could someone please convert this manually for me? Thanks\",\"completion\":\"Hi there,\\n\\nSorry for the trouble. Yes, unfortunately, our Canva to Figma production server is down. We will convert your file locally and send you the file as soon as possible.\\n\\nBest regards,\\nLokman\"}\n\n{\"prompt\":\"Hi Support,I recently purchased a subscription at Magicul and already have two issues. Maybe you can assist.1. How do I download the converted file?I took the following steps:I uploaded a Canva file, then paid for the subscription, and logged in.Clicked Files.I see the copy (Download your converted files here) and see the converted file below.I see the option to Convert (again) but not download.2. It seems as though the same file was converted twice, so now I have one less credit. Thanks for your help.\",\"completion\":\"Hi there,\\n\\nSorry for the inconvenience. Would you please send me the link to your Canva design? It seems the file did not convert due to a bug.\\n\\nLet me know if you have any questions.\\n\\nRegards,\\nLokman\"}\n\n{\"prompt\":\"can i get a trial of one file one artboard to see how well magicul converts from XD to Figma?\",\"completion\":\"Hi there, \\n\\nYour converted file is ready. Please click here to download your file.\\n\\nLet us know if there's any issue.\\n\\nRegards,\\nLokman\"}<\/code><\/pre>\n\n\n\nCleaning your Chat History Data<\/h1>\n\n\n\n
Just by looking at the data that we\u2019ve exported from Intercom it\u2019s already quite obvious that we\u2019ll have to do some cleaning here. <\/p>\n\n\n\n
Here are some of the issues with the raw data:<\/p>\n\n\n\n
- \n
- Data may contain sensitive information that should not be feed into ChatGPT<\/li>\n\n\n\n
- Incomplete answers should be deleted<\/li>\n\n\n\n
- For now our export only contains the very first reply, if the support agent just replied with \u201cHello, how can I help\u201d then it\u2019s not really helpful<\/li>\n<\/ul>\n\n\n\n
All those things can be potentially implemented in the Intercom conversation export script. To keep this tutorial short I\u2019ve manually cleaned our dataset by removing sensitive data and useless replies. <\/p>\n\n\n\nFine Tuning the ChatGPT model<\/h1>\n\n\n\n
So let\u2019s fine tune our ChatGPT model with the data we\u2019ve exported from Intercom.<\/p>\n\n\n\n
In order to do so we\u2019ll be using the OpenAI Command Line Interface (CLI).<\/p>\n\n\n\n
I would recommend you follow my tutorial on how to get the OpenAI CLI set up on your computer: https:\/\/kevingoedecke.com\/blog\/2023\/04\/14\/how-to-install-openai-cli-on-macos\/<\/a><\/p>\n\n\n\n
To verify that you\u2019ve got it properly setup, simply run the following command:<\/p>\n\n\n\n
openai --help<\/code><\/pre>\n\n\n\nThis should give you back a set of available command from the OpenAI CLI.<\/p>\n\n\n\n
The first thing we\u2019ll have to do is prepare the JSONL data. The OpenAI CLI has a command that takes care of that.<\/p>\n\n\n\n
openai tools fine_tunes.prepare_data -f \/Users\/kevin\/Downloads\/output.jsonl<\/code><\/pre>\n\n\n\nIn my case this prompted me with the following:<\/p>\n\n\n\n
openai tools fine_tunes.prepare_data -f \/Users\/kevin\/Downloads\/output.jsonl \n\nAnalyzing...\n\n- Your file contains 1648 prompt-completion pairs\n\n- Your data does not contain a common separator at the end of your prompts. Having a separator string appended to the end of the prompt makes it clearer to the fine-tuned model where the completion should begin. See https:\/\/platform.openai.com\/docs\/guides\/fine-tuning\/preparing-your-dataset for more detail and examples. If you intend to do open-ended generation, then you should leave the prompts empty\n\n- Your data does not contain a common ending at the end of your completions. Having a common ending string appended to the end of the completion makes it clearer to the fine-tuned model where the completion should end. See https:\/\/platform.openai.com\/docs\/guides\/fine-tuning\/preparing-your-dataset for more detail and examples.\n\n- The completion should start with a whitespace character (` `). This tends to produce better results due to the tokenization we use. See https:\/\/platform.openai.com\/docs\/guides\/fine-tuning\/preparing-your-dataset for more details\n\nBased on the analysis we will perform the following actions:\n\n- [Recommended] Add a suffix separator `\\n\\n###\\n\\n` to all prompts [Y\/n]: Y\n\n- [Recommended] Add a suffix ending ` END` to all completions [Y\/n]: Y\n\n- [Recommended] Add a whitespace character to the beginning of the completion [Y\/n]: Y<\/code><\/pre>\n\n\n\nSay yes (type \u201cY\u201d) for all optimization suggestions.<\/p>\n\n\n\n
Afterwards we\u2019re ready to properly train our model with our Intercom exported chat history.<\/p>\n\n\n\n
Execute the following (Note:<\/strong> make sure to use the cleaned jsonl file from the step before):<\/p>\n\n\n\n
openai api fine_tunes.create -t \"\/Users\/kevin\/Downloads\/output_prepared (1).jsonl\"<\/code><\/pre>\n\n\n\nThis might take a while depending on how big your data set is.<\/p>\n\n\n\n
After a bit of waiting around I got prompted with this:<\/p>\n\n\n\n
$ openai api fine_tunes.create -t\n\n\/Users\/kevin\/Downloads\/output_prepared (1).jsonl -m davinci\n\nUpload progress: 100%\n\nUploaded file from \/tmp\/output. jsonl: file-XXX\n\nCreated fine-tune: ft-XXX\n\nStreaming events until fine-tuning is complete...\n\n(Ctrl-C will interrupt the stream, but not cancel the fine-tune)\n\n[2023-04-18 14:58:17] Created fine-tune: ft-XXX\n\n[2023-04-18 14:59:16] Fine-tune costs $6.94<\/code><\/pre>\n\n\n\nAs you can see it cost me $6.94 to fine tune my ChatGPT model with my existing Intercom records. This is obviously not cheap. The price highly depends on how many records you feed into the fine tuning and also which underlying model you use. I used `davinci`, but according to other posts out there you might be able to achieve similar results using a cheaper model like `ada`.<\/p>\n\n\n\n
Alright so now that we\u2019ve finally managed to fine tune our model let\u2019s give it a spin \ud83e\udd41<\/p>\n\n\n\n
Results – How well does it work?<\/h1>\n\n\n\n
You can now go ahead and try your fine tuned model out by shooting some prompts to the ChatGPT API via the CLI. To my surprise I was also able to see my fine tune model even in the ChatGPT playground (https:\/\/platform.openai.com\/playground<\/a>).<\/p>\n\n\n\n
<\/a><\/figure>\n\n\n\n
Let\u2019s feed it some questions, shall we\u2026<\/p>\n\n\n\n
Most of our customer service requests are one of the following:<\/p>\n\n\n\n
- \n
- Someone wants to know if we offer a free trial<\/li>\n\n\n\n
- Someone has questions about one of our plans<\/li>\n\n\n\n
- Someone has a problem with a file conversion (we offer a design file converter)<\/li>\n<\/ul>\n\n\n\n
Here are some of the results are training ChatGPT.<\/p>\n\n\n\n<\/a><\/figure>\n\n\n\n
It was interesting to see that ChatGPT tried to become one of our customer support agents (Lokman) and it was even more funny to see it giving away a 50% discount on the first payment \ud83d\udcb8HAHA! Am I really ok with this \ud83d\ude02<\/p>\n\n\n\n
Here\u2019s another very impressive one:<\/p>\n\n\n\n
<\/a><\/figure>\n\n\n\n
It seems to have managed to pick up which exact file formats our design file converter supports and in which plan it is included. Kudos ChatGPT, not bad at all!<\/p>\n\n\n\n
<\/a><\/figure>\n\n\n\n
This one is really impressive too, since it actually picked up very specific things about our business. <\/p>\n\n\n\n
Generally speaking I was very impressed with the results. That being said it also gave some very bad answers. I think this is partly because of the input data on which we fine tuned the model. <\/p>\n\n\n\n
- A Ruby script to export your Intercom data<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n
- A paid OpenAI account:<\/strong> Since we\u2019ll be using OpenAIs ChatGPT you\u2019ll need a paid account. A quick heads up: fine tuning a model isn\u2019t cheap, I\u2019ve spent about $6-10 per fine tuning job.<\/li>\n<\/ol>\n\n\n\n
- Have a big chat history in Intercom: <\/strong>In order to fine tune a ChatGPT model you need data, a lot of data\u2026 the more the better. So if you\u2019re reading this article with the plan to implement this yourself you have to have a big data set. <\/li>\n\n\n\n
![\"\"\/](\"https:\/\/lh4.googleusercontent.com\/xhHU8-D-KoF07nu9Q6Hqp6HwSrJzQSl8M3ou8ogvwhIRPDScsntQ0QzHYdL2X1fmwL2b_FehwSjTdbOlJHR70vTM0TZQm5vKhkgOk8UzoXoMKjGXTo_bS7KHJyILUWghh1KgHPBHD5pDwHrnVsK0yA\")
<\/a><\/figure>\n\n\n\n