I wouldn't exactly call this a failure of the JS ecosystem. The Express documentation you've linked to mentions that the host is the (optional) second argument to the listen function:
Are you talking about the difference between 0.0.0.0 and 127.0.0.1 /localhost (loopback address)? If so It should be a tiny thing to change to implement I can add that on the next update- I just went with the express JS defaults to keep things simple:)
Or I can even make a flag to configure this :thinking
If you do make it configurable, it's still a good idea to make the default `127.0.0.1` for the reasons @ggpsv mentioned. Since the link automatically opens at `http://localhost:PORT` anyway, it's an easy security win.
`app.listen` uses the same signature as Node's `server.listen`. You can specify the `host` [0] or else it defaults to `0.0.0.0`. This means that when you run this service, it will be accessible to anyone that can reach the server.
Typically, services allow you set the host address by configuration. This way you can set it to `127.0.0.1` so that it is only available locally.
Congrats on launching! Its interesting to see that ChatGPT is able to build something I've been working on (https://logdy.dev) for last two months. I guess new era is around the corner then.
This is a great tool too and echoes task teleportation, which is popping up more and more.
There’s a new era, but one in which you will be even more powerful. LLMs will improve in pumping out code embodiments… but it was your human creativity borne out of operations frustrations, coupled with your care to do something about it, that manifested your product.
I was looking for something like this for a while, and it could be just the landing page, but I much prefer a tool that's been built slowly with thought and consideration (which is evident from your landing page).
How does your tool handle semi-permanent usage or very long logs? For instance, I would run this to view logs from an ssh server that's been up for 6 months, when systemd has rotated the log file etc.
Thanks for your kind words. I have a feature onnthe roadmap which is very close to delivery that will allow you to load a big log file and browse it through the UI. As long as you can load the content from disk or remotely, Logdy will be able to handle that. The aim isnto provide DataDog like experience but in a portable and temporary way, that allows you to set it up quickly, check what you need, tear down and move on. All without much hassle or costs for sending the logs to a remote service.
for instance, running a nextcloud instance through docker compose, I'd like to have a lightweight web ui for browsing the server logs. not just for 15 minutes but for longer periods of time.
on an hpc cluster, I want to want logs from the slurmctld forever essentially without having to login and grep through the logs.
there's a bunch of online services for this, there's ELK stack, there's grep, and few things sit in between. hence my question
then the problem is not viewing the logs but storing them, maybe compressing, and then retrieving. If that's not a problem (size) then with some cron magic you should be able to store the logs from specific locations into a single one so rotation doesn't eat them and then just view using a log viewer of your choice.
That was the idea exactly: I wanted to have DD log explorer locally that works insanely fast to be coupled within development process. As much as I love terminal, its just so limiting when it comes to browsing and filtering through more content.
that and more magic of sed, awk, jq and similar. I know you can do real magic in there but still, I believe it's easier to learn GUI than vim shortcuts or sophisticated jq/awk formulas. Still, I admire terminal for being around for so long and follow it as an example when building Logdy.
I love doing that, because then I get all the power of Vim to look at the output. Search, regex search, sorting, bookmarks, counting, macros ...
I started to write output that is indented, so I can use Vim's folding to comfortably look at complex data.
For example if you have entries like this:
2023-04-12 17:22 Sighted and UFO
Location: 27.1742424 12.137234
Temperature: 24°C
Observer: Joe
Actually it turned out to be a dove.
And you press zM in Vim, they all turn into:
2023-04-12 17:22 Sighted and UFO
+ 4 lines
So you have a nice overview and can open individual entries by putting the curser on them and hit zo.
And while folded, you still have all the power of Vim to search within them. Regex search. Count by any measure you like etc etc. Actually, you can do anything, as Vim allows you to execute arbitrary vim commands, vim macros or shell commands on your buffer.
Wanna know how many doves typically get mistaken for UFOs around New York grouped by month? Write a script for it and it will be forever at your fingertips.
I think this is more designed for streamed output rather than viewing files. Vim doesn't seem to support that, in fact it just says "Vim: Reading from stdin..." forever.
Hmm.. I guess you could write a little script (2 lines?), let's call it "vimlive" which reads from stdin, streams to a temp file and opens that temp file in vim.
Then with:
command | vimlive
You get vim with the current state of the data and you can update it with :e
> Interactive Interface: Search, filter, and navigate through logs easily. // TODO
This is exactly why you would show logs in the browser though— this is not a nice to have, it is THE core feature. The browser is perfect for building the complex UI necessary for filtering and making sense of log streams.
Ahh actually you can do search through the logs in real-time, I forgot to split that and remove TODO from the readme.
Complex filtering based on the timestamp is yet to implement.
"This tool has been generated entirely with ChatGPT and hence it is also a demonstration of how useful products can be developed with collaborative reiteration with GPTs."
Any chance you could share a link to the ChatGPT conversation, or a copy of the prompt you used?
I build little tools like this with ChatGPT all the time, it's one of my favourite use-cases for it. I often have an idea for a tool that wouldn't justify an hour of development work but is absolutely worth spending 5 minutes iterating on with ChatGPT.
This is handy stuff, thanks for sharing. We are working on transporting tasks back and forth from TUI<>Web<>App<>VR ; this is a great and straightforward example of that.
I use VSCode CoPilot constantly as simple augmentation and typing-saving — not for wholesale creation of code. I’ve seen my pre-novice daughter use ChatGPT and the results were mixed (good in some ways, horrible in others).
Can you elaborate more on your ChatGPT/LLM working style for this?
Sure thing! The collaboration with ChatGPT was like having a friendly coding buddy. I threw in ideas, got feedback, and we built this tool together. We chatted about names, features, and even how to post it. It's like having an AI pal who's great at brainstorming and coding. Fun times!
^ asked Chat GPT to generate this from my earlier prompts :)
Some other notes, I am a developer who love to make usable products, and these years I learned the usability is most important especially when you are developing in the early stages of a product.
So I fed some ideas, asked suggestions, gave some tech stacks (for example, I asked chatGPT to not go with webpack, instead simply load React from CDN for quick iteration. and suggested memoisation on React to render things faster).
So process wise similar to any other product development, but it's like having someone listen to your feedback and instructions and do the work- I would like to call collaboration!
This is a solution looking for a problem. The browser is a horrible place for viewing text logs. Companies like Splunk and Datadog put countless hours into trying to make it work and it's never not-painful to view logs in a browser.
What are the main specific issues that you felt difficult in viewing logs on a browser? imagine there is better visualisation, parsing, highlighting, filtering, collapse etc.?
I feel like the terminal is where I spend more time looking for the relevant parts from the logs.
One thing I've always wanted was simple filters on log lines in a GUI that let me collapse tags in the log line. I created a proof of concept that showed logs as a tree.
Or I can say I want logs from the point where key=value and until method x.
A bit like a debugger but for logs. Show logs that are between these facts.
Yes, that is exactly what I want as well - a tree-like structure where you can trace events across multiple layers of abstraction as well as across systems (so both horizontal and vertical relations of events, so to speak).
Some infra systems output syslog loglines and there's no changing that, and I still want to capture and relate these log messages to other events (incl. high-level business transactions / processes).
Any way you could share your PoC? :) very interested.
I don't have the code anymore might have been called "logtree", it was a HTML file and very small, it would parse dot "." separated lines such as (user.connected) and arrange them in a tree with indentation, if you clicked the dot syntax it would highlight in yellow other instances of that bucket of entries later in the log file.
I think you want elegant navigation between linear sequences (next, previous) of log lines that are apart in time, but related.
You might want to see all events of the internals of two different components in the tree structure.
Need to process log lines into a graph and then provide quick navigation for them.
It is indeed horizontal and vertical.
This would be very useful to anti-abuse measures if you could trace to the cause of something and cancel all thing caused by it.
Thank you for helping me structure what I want, yes, exactly. (And noted re: your PoC, ok!)
> You might want to see all events of the internals of two different components in the tree structure.
> Need to process log lines into a graph and then provide quick navigation for them.
Yes, looks like it, indeed. Yeah.
If interested, this is how I'd hastily summarise my problem statement - it is I think a rather generic recurring problem and analysis pattern, not unique to any company (context: goal is to drive through this initiative as part of overall software architecture improvement needs, and bringing business maturity (as per CMM model) up in our particular case):
- need to introduce and enforce structured event hierarchy (think: each high level business transaction (e.g. user signs document) is a parent which retains all related child events across all layers of abstraction (ideally down to DB connection and file handles, internal API endpoints hit, block storage i/o errors, etc.)
- problem: while for own software Backend-generated events we can enforce and manage this, some infra pieces emit raw text syslog (sometimes even syslog message portion (log format) cannot be fully controlled) and do not contain cross-component technical identifiers for determining relations to events
- there are solutions including ML solutions (of course)
- there exist interesting papers to lose productivity and/or conduct deeper research
- what do, how to balance, how to incrementally define, introduce into culture, incrementally realistically establish this (etc.)
Some interesting papers for approaching the complicated part (assuming defining event ontology and hierarchy and overall semantics is the easy part) - the "relating unstructured logs to events and embedding them into graphs, and aligning and training everyone to analyse and work with "event trees" across multiple teams (but yes, ops, integration support, etc. - I mean is that not the dream, right...) do exist[1][2][3][4][5], and some interesting thoughts and many lofty ambitious questions[6]. Trying to gain insight into best practices here. Meanwhile, maybe links will look interesting :) (e.g. that 1849-2023 IEEE standard - just found it today...)
[2] Structural Event Detection from Log Messageshttps://faculty.ist.psu.edu/jessieli/Publications/2017-KDD-l... (this one may look bewildering at first but the math eqs are actually ~legible / straightforward from quick glance, even; also, has nice coverage of related work) (overall nice, might look smarter than appears, but not sure; see MCMC approach etc. in section 5.)
[4] 1849-2023 - IEEE Standard for eXtensible Event Stream (XES) for Achieving Interoperability in Event Logs and Event Streamshttps://www.xes-standard.org/
[6] a structural approach to the problem of structure... use existing frameworks and languages like BPML to create or use existing event type ontologies from BPMN models; use a proper structured thought-through ontology for this, and then sort out that pesky "relating low level infra raw syslogs with none/few TIDs to some node on a graph" problem
Anyway, thanks for making through my rant! Writing it in part because on the off chance someone bites and has insight - I'd be very interested to hear more; and partially just to share those links above because those may tickle your (Sam's fancy) :) thanks for reading!
Depends on the size, for me it worked well with decent size JSON log lines, but if it is too much that doesn't fit on the browser memory, can think of index storage of sqlite in the future.
Search and filtering can be then loaded to the server + sqlite setup.
`npx` is a tool that ships alongside `npm` (alongside `node`) that either finds a local installation or performs a just-in-time download, and then runs the default "bin" entrypoint for the package.
Newer versions of `npx` (last few years) will prompt the user before installing a package. Older versions did not. Note that installing the package can run arbitrary scripts as a side effect with at least the level of permission of the current user, so there is implicit trust required of the author, and the authors of all transitive dependencies.
> This tool has been generated entirely with ChatGPT and hence it is also a demonstration of how useful products can be developed with collaborative reiteration with GPTs.
If it works, I don't see anything wrong with it. It looks good. Hating on projects that disclose their GPT usage will only incentivize them to not disclose in the future.
Haha yeah, I wanted to frame this as a single file tool, and hence all the quick wins. But later moved the react app into a separate file because I was not getting the syntax highlights and all.
Definitely this can be made better with proper packaging and tooling! As I mentioned, more than the idea, implementation was an experiment on the ability of collaborative iterations with GPTs- it was fun!
As much as I love to hate on chatgpt engineer, the code is definitely okay; not much difference from a project you would write quickly in a single weekend.
I'm not saying that humans never come up with this kind of crap, but it feels like it's going to be a LOT more common the more we resort to AI. I wonder if "... and make the code easy-to-read so that it's maintainable in future" is ever included in prompts for LLM-generated code.
I'm a clean code guy and tailwind made my code look terrible. But I gave it a chance because I heared so many positive things about it. Now I'm happy with it. I don't have to jump around from HTML to CSS and back to HTML.
That is just tailwind CSS, one the most common CSS libs out there I would think. Your comment is actually a lot more telling of something as opposed to this AI written project though.
> For those of us not in the know, what does that mean?
I reckon that some people are either critical of LLMs due to the data that they're trained on (concerns about the legality of using the outputs, or the ethics of taking open source projects with a variety of licenses and how transformative or not the output might be), or question the quality of the code that they might output. Oh, also there's hype cycles and right now people at large might overestimate the capabilities of what LLMs can or cannot do well, which can be tiring, as seeing constant posts about crypto was (where we had a solution in search of a problem).
In my eyes, they can be used in conjunction with something like IntelliSense and IDEs to solve select problems (ones with solutions that already exist out there) more quickly, as well as save time on some boilerplate, as long as you still validate the outputs and actually check if everything works. Nowadays I'm using GitHub Copilot and the user experience is mostly okay, I'd say it actually lets me write code faster, at the expense of occasionally getting things pretty wrong, but is still a net positive, other considerations aside.
Web-Based Log Viewer: Get a cleaner and more organized view of command outputs.
Real-Time Updates: Logs are displayed in real-time as the command executes.
Interactive Interface: Search, filter, and navigate through logs easily. // TODO
https://github.com/soorajshankar/logScreen/blob/f8b29aaef428...
So this will be sharing your logs with the world if you run this outside of a trusted network. (And personally I avoid trusting any networks)