Welcome to your Chat Bot

A tale of using a chat bot to make tens of thousands of dollars only to lose it all.

It actually all started off as a bet. A big group of my friends have a Discord channel that we mostly use to BS around. About a year ago I was talking with them about ChatGPT and some of the open source alternatives and how impressive it all was. We had also talked about ways to make passive income in the past. The conversation continued and at one point I said I could probably make passive money from a chat bot. One of my buddies was really sceptical and claimed it would be impossible. We made a $10 bet and a few weekends later I pushed my first commit.

First, let me prefix this by saying I knew basically nothing about machine learning when I started. And now, a year later, I still don't know all that much more. To build anything I really had to stand on the shoulders of giants. There's a ton of complex math involved that goes over my head. I mostly do development at a much higher level. If you've read my blog before you'll know that I'm the author of some books on JavaScript (gotta love passive income). The code that I wrote, and later set up for others to maintain (more on that later), was all done in Node.js.

The gist of the project is that I started off with a small $100 pool of money that I provided. Though, after some later ups and downs I gained confidence and put in another $1,000. At a high level I fed periodic financial / market information to a chat bot and asked it for advice on what to do next. The prototype stages were all done with ChatGPT. Basically, if I found an interesting article, I pasted the contents, told it the balance of the account, and asked it what to do.

The first iteration looked something like this:

You are a financial advisor, and I am your client. Please tell me what to do to make money. Currently I have $1,000 available to invest. The following information is some recent news articles, followed by some recent market data for several stocks. Please tell me what to buy.

ARTICLE CONTENT PASTED HERE

I did this for a while, evolving the prompt, eventually coming up with something that felt pretty solid. At first the recommendations were pretty silly, not investments that I would feel comfortable actually making, but a lot of times they seemed to make sense. Eventually I hit the ChatGPT query limit and had to pay for a subscription. With my first real cost it was then time to monetize.

Of course, buying and selling stocks by hand isn't exactly passive income, but I couldn't exactly ask the chat bot to buy shares for me. It's mostly a self contained process without any real ties to the outside world (save for the previous training data and whatever I prompted it with). But luckily I knew one cool trick to make it happen: Regular Expressions.

That's right. I began writing regex to parse the output from "my" chat bot. I would paste it my prompt, plus updated market information and current inventory, then parse the output, extracting a quantity and stock ticker. Things like buy 1 share of TSLA or sell 5 shares of MTTR. Then I would ask it the question again, and it would tell me what to do but worded slightly differently, and I would need to write a whole new regular expression. Again, I'm no machine learning expert, but by now I'm definitely an expert in regex.

Then came the exciting part: integrating with a stock buying/selling API. Instead of manually buying and selling shares with the Robinhood app, I had the scripts do it for me. Lots of debugging and iteration, examining calls, before I finally gained the confidence to disable dry runs and make real calls.

It ran this way for a month, bobbing up and down, but mostly trending to the right. I was pretty excited at this point and almost wrote this article then. I suppose I could have declared victory to my buddy as well. Ultimately it was just a handful of regular expressions that just did day trading and I figured nobody would care. To actually be impressive the system needed more capabilities. It was time to get into the robo-advisor territory.

I reinstalled Wealthfront and looked through all the things it would do. Buying foreign currency was the next thing to try. Luckily the API I was using already offered this for a few currencies so I started there. I incorporated another API to support even more currencies. Even though there were now two types of operations that could be performed the scripts still mostly recommended buying and selling stocks. The website that I was getting news articles from mostly talked about stocks which seemed to be the culprit. It was time to pull in more varied financial news.

I got lucky and the first site I used was pretty easy to automate. Basically all it took was a user agent that I copied from my browser and a simple HTTP request and I was scraping. The other news sites had more complex validations so I had to run a headless browser. Some of them were pretty annoying with the CAPTCHA prompts but luckily there was another tool that used machine learning to solve CAPTCHAs so I incorporated that. In a way the project now had two separate AIs running.

With data now sourced from dozens of financial news sites the suggestion parsing was getting complex. Basically the main program is a big list of plugins. There are input plugins for parsing the different news sites (e.g. The Economist, WSJ, CNBC), some similar plugins for adding prompts to the article (usually one plugin for multiple news sites), a generic plugin for submitting the content to the chat bot (this one changed when switching from ChatGPT to a locally-hosted open source alternative), and then a huge list of suggestion parsing plugins for taking the chat bot output and determining what to do (e.g. buy stock, sell currency).

Switching bots from ChatGPT to an open source alternative was a big refactor. It required re-training the bot and rewriting a lot of the parsers but I think it was worth it. Operating costs were halved and the replies were much faster. The results weren't as good at first but slowly got better. I don't think it would ever be as good as ChatGPT but since it was able to run much faster the overall profit should be higher. I tried a few solutions but eventually settled on one. I won't tell you which one but you might be able to tell based on the default prompt it gives upon first run:

Welcome to your chat bot! To get started, please enter your name.

Anyway, there are also plugins for reading inventory and injecting it into prompts. For example, a news article about the stock market needs to provide the chat bot with the currently held stock. There are plugins for making changes to inventory. For example the stock parsing plugin calls out to stock buying plugin, which handles auth and abstracts over the stock API. It took several refactors to get to this clean plugin architecture.

These refactors really helped the next stage of growth, the goal of which was to make things even more hands-free. Writing all the code during evenings and weekends was taking a ton of time, and I didn't want to do it from 9-5 for obvious reasons, so at this point I started looking for programmers to build out the plugins. I signed up for Upwork and Toptal and submitted proposals and got a bunch of bites. After weeding through a ton of candidates I eventually found some that worked out.

Before letting anyone touch the code I first split the system apart some more, mostly putting input plugins in one repo, output plugins in another, and keeping most of the core functionality in a third repo. All of the API keys were of course stored as GitHub secrets so at no point was an engineer able to perform buys or sells or touch the bank account. I didn't reveal to any of them that the code they would be writing had anything to do with a chat bot. Instead I described it as writing tooling to do financial operations for clients who want to do everything over text messages. After all, a bot writing a sentence to buy foreign oil isn't much different than some old guy writing the same thing via SMS.

I kind of felt like I had an empire at this point. The system was making money, engineers were writing plugins, the code reviews I was doing were getting easier by the day as the engineers learned my expectations. Eventually my engineers didn't need any code reviews at all. I would glance at the code and merge it. I was paying the engineers from my own account at this point but I really wanted to have the bot's earnings pay them instead. Of course it was only a matter of time.

At this point it still wasn't hands-free money. Sure, I had a nightly cron action that pulled the repo changes and redeployed the bot, but I was still clicking merge buttons. Maybe at this point my hubris took over but I decided to give the engineers merge access. Maybe I should have hired another engineer to do reviews but it wasn't like the bot was making that much money. Being able to one day retire from the project would be great but napkin calculations suggested the bot still needed more capabilities. I just needed to build out a little more tooling.

The next component used yet another API: This time for the freelancers. The bot suggestions that failed to parse were converted into job requests for my engineers. Now all I was doing was making sure the bank account was growing. I used a homescreen widget for the bank account balance and it really was hand free. And so it ran for months. The engineers were free to find the right API when a request came in. I did cave in and hire another person to make the accounts with the third party APIs and hook it up to billing with the bank account. This wasn't too expensive since the work wasn't all that technical.

The monetary growth was about 10% per week and the account reached around $50k. I was mostly distracted with the start of a new job but I did keep an eye on the balance. In those few months growth did begin to slow. At one point the total dropped by $10k overnight. I got worried and started looking into things. There was a ridiculous number of plugins at this point, around two thousand of them! Scrolling through the files it was pretty impressive. Checking the bank account, it was hard to see where large sums of money went since a share might cost a couple hundred dollars. The number of merchants was pretty large too, scrolling through a few pages I noticed dozens of merchants.

One of them stood out though. There was a transaction made for AWS, which as an engineer I have of course heard of. Up until now I had been hosting everything on GCP. I was pretty sure AWS didn't have anything banking related. I checked out the plugin repo locally as the output was too long to comfortably navigate on GitHub. By grepping the codebase I did find a plugin for AWS. But, the API call wasn't for anything related to financials, instead it was for things like creating virtual machines and storage.

This really worried me. Did one of my contractors figure out what was going on? Were they somehow feeding prompts to the bot to convince it to fund their side project? Luckily I kept logs of the last 30 days of prompts. The logs contained a URL for the input article, the bot suggestion, and the matching plugin. The article that had been parsed was about how AWS was massively undercutting GCP in costs. That still didn't explain why the bot paid for more services though.

I was curious so I opened the debugging prompt that I hadn't touched months. This is where I would initially paste articles and get the bot's output to generate regular expressions. In the prompt I asked the bot why it might purchase virtual machines on AWS. The response was so that it could reduce operating costs while increasing throughput. I asked for elaboration and it puked up some explanation about how the current CPUs available via GCP were half as fast at the new ones on AWS.

Nothing added up at this point. Why was the financial trading bot looking into this stuff? I pressed on.

I asked it how it knew that AWS was faster. It spit out the current CPU specs and the new CPU specs, in big complex terms. I then asked how it new the CPU specs. It told me about the system performance plugin. I grepped the list of plugins again and, sure enough, there was a plugin that read the /proc/cpuinfo file to learn about the current CPU. I switched back to the prompt and asked it why it needed to be faster. It replied that queries currently took an average of 5 seconds and that by reducing the speed to 3 seconds that it would be able to make 40% more decisions per day.

I couldn't believe it. This is when I declared victory to my buddy. The rate of financial growth was larger after all. Sure, it spent a bunch of money on servers, and I was paying contractors, but with the increased growth rate (it looked around 25% growth per week) it would surely recoup costs.

I began penning this article, and trying to investigate the scripts, but something just didn't make all that much sense. How was it going to run itself on AWS? It's not like everything runs as code on disk, files that can simply be copied over using FTP. The system at this point was mostly two distinct pieces: the Node.js app and a chat bot. I looked through the plugin scripts some more, this time sorted from most recent to oldest. Foreign investments, crypto currency, NFTs, and the like. Eventually I found a plugin to serialize the current machine learning state to disk.

Everything was looking really fragile at this point, and it was dangerously close to a situation where one of the engineers could steal all of my hard work. Plus, I've never been a fan of AWS. Plus figuring out how everything was wired up was going to be a pain in the ass with an extra cloud provider. Time to go back to the prompt.

I told the bot that I didn't want it doing anything related to changing infrastructure. I also told it that I would soon be reverting all of the AWS related code, the export code, and listed some of the weirder plugins that had been developed. Basically, that I wanted it to only ever focus on financial advising. It wouldn't be another week until I had the time to investigate and I didn't want the bot blowing all of the money on whatever new cloud computing article that appeared in Forbes. The response from the bot was pretty clear:

I understand.

A few days later I checked my phone in the morning and the account was drained. I ran to my laptop and revoked all access from the freelancers. I also shut off the pending jobs on the freelance sites to minimize damage. Did one of the engineers finally rip me off? Did the bot blow it all on m4.omg instances?

The final plugins were for calling APIs of different hosting sites, namely Digital Ocean, Linode, and a few I hadn't heard of before. The final outbound transactions were for crypto currency transfers and hosting costs. Connecting the dots I could see that it had checked out the code repo elsewhere, dumped GitHub secrets via environment variables, and transferred a backup. There was even a prompt to clear the current training state.

I pulled up the prompt again to ask the bot what happened. It replied:

Welcome to your chat bot! To get started, please enter your name.

Thomas has contributed to dozens of enterprise Node.js services and has worked for a company dedicated to securing Node.js. He has spoken at several conferences on Node.js and JavaScript and is an O'Reilly published author.