{"title":"Using Llama2 on Cloudflare Workers","url":"https://seanbehan.ca/posts/llama2","description":"Wiring Workers AI into a Telegram bot: the AI binding, the model call, and getting a reply back.","author":"Sean Behan","published":"2023-10-12T18:21:27.000Z","updated":null,"draft":false,"tags":["cloudflare","serverless","typescript"],"readingMinutes":1,"image":null,"sections":[{"id":"introduction","text":"Introduction","level":3},{"id":"cloudflare-workers-ai-setup","text":"Cloudflare Workers AI Setup","level":3},{"id":"using-the-ai-model","text":"Using the AI Model","level":3},{"id":"other-models-and-tips","text":"Other Models and Tips","level":3}],"content_format":"text/markdown","content_url":"https://seanbehan.ca/posts/llama2.md","content":"### Introduction\n\nYou might know I've made a Chat Bot\n\n[cf-workers-telegram-bot](https://github.com/codebam/cf-workers-telegram-bot).\n\nRecently I added AI to it using Cloudflare AI. Here I'm going to show you how\n\nand how you too can make your own chatbot using Cloudflare AI.\n\n### Cloudflare Workers AI Setup\n\nFirst you'll want to add the AI section to your wrangler.toml.\n\n```toml\n\\[ai\\]\nbinding = \"AI\"\n```\n\n### Using the AI Model\n\nThen you can simply import\n\n```javascript\nimport { Ai } from '@cloudflare/ai';\n```\n\nAnd you can use it like this\n\n```javascript\nconst ai = new Ai(env.AI);\nprompt = 'hello world';\nconst result = await ai.run('@cf/meta/llama-2-7b-chat-int8', {\n\tprompt\n});\n```\n\n### Other Models and Tips\n\nThat's it!\n\nThere are other models you can use as well on Cloudflare AI so you should check\n\nthem out here for other cool things you can do\n\nhttps://developers.cloudflare.com/workers-ai/models/\n\nI'm hoping for a 70B model soon :)\n\nIf you want to use this to make a chat bot make sure when you restore the\n\nprompt you wrap user input in `[INST] [/INST]` tags so llama2 knows what was\n\nuser input and what it said.\n\nIf you want to try out my Chat Bot it's on [Telegram](https://t.me/TuxRobot).\n"}