Restart vector if it stops receiving logs from sources

[jsierles]

This PR adds a small monitoring service that queries vector at a configurable interval. It checks whether vector has processed any new events. If it hasn't we can assume that its NATS subscription is 'stuck' and needs to be restarted.

[cayblood]

It seems like my log shipper app is getting stuck and this might be a way to fix it. @jsierles is it possible for me to deploy this update before it gets merged? Is there a docker image I can refer to now, or would I need to build it myself?

[jsierles]

You'd need to build it yourself at the moment.

[punkpeye]

This is very silly, but it took me a while to realize that this log shipper is simply connecting to nats and therefore can be easily implemented in user land. I struggled with setting up this repo, but I got it it work with basic TypeScript. Here is a good start:

import { connect } from 'nats';

const connectToNATS = async () => {
  console.log('Connecting to NATS');

  try {
    const nc = await connect({
      name: 'Fly logs stream',
      pass: process.env.ACCESS_TOKEN,
      servers: [`nats://[fdaa:0:0::3]:4223`],
      user: process.env.ORG,
    });

    console.log('Connected to NATS');
    return nc;
  } catch (error) {
    console.error('Failed to connect to NATS:', error);
    throw error;
  }
};

const main = async () => {
  const nc = await connectToNATS();

  const subscription = nc.subscribe('logs.>');

  for await (const message of subscription) {
    console.log(message);
  }
};

await main();