@foone oh my lord this is such a glorious idea - since apparently we will never solve criminally bad UI/UX :'(
the thing is I'd still use it even if it took me way longer to configure and debug than the time it would have taken to do by hand! apparently that is more acceptable to my adhd's frustration limits
@abrasive yeah exactly! ADHD just makes this worse because it's infinitely better to spend hours and hours building automation than to spend minutes doing it manually
@munin fun fact: the software I'm currently subtooting? I manage it using two custom keyboards I designed and a python script doing a lot of screen scraping.
@foone@munin The lead provisioning engineer at an MSSP I worked at in the aughts automated most of the device provisioning for the security devices we managed for our customers with a metric crapton of AutoHotKey scripts. I found it simultaneously glorious and horrifying.
does anyone make a laptop which has USB-device support and an HDMI capture port? I could just put my laptop down in front of the computer I'm supposed to use, and it could click on things for me and look at the screen for me
This clearly has been misinterpreted by everyone suggesting this is AI or virtualization because frankly I have been feeling like this since 1995 when none of that was on the radar yet
also, the root cause is that I'm having to use software that doesn't act in predictable user-friendly ways, right? so I want a meta layer that insulates me from that.
AI is not the solution to this problem. AI is one of the types of problems this is trying to solve!
Computers are great at waiting! They can just sit there in a loop waiting for a response, periodically trying again if needed.
Computers are great at waiting, but I, a (presumed) human with ADHD, am very shit at it. That's why I write software to wait for me, so I can do other stuff while the cloud is being slow
To be clear, the context here is that I am a professional computer automation engineer: my last two jobs have primarily focused on controlling GUI applications for testing or performance reasons. I've been releasing software to do this very thing for like 20 years: this is not just someone complaining that software is "too hard" and needing to be told about AutoHotkey.
@foone oh wow. I vibe with this. I recently spent 2 days trying to click a button in a windows app. I remember being able to do this 20 years ago with a few lines of a WSH script. The most reasonable powershell equivalent I could find was 90 lines of C#. WHY IS IT SO COMPLICATED?
@foone so kinda like if you could use AppleScript on every computing system that has ever existed and could perform every function possible through the UI?
@foone@ZiggyTheHamster applescript is the only bit of apple special sauce that i’m mad about having missed out on. It seems so useful and magical and I think it’s a huge failure that we don’t have a good equivalent in any free OS or desktop environment
(it would be so cool if this summoned a ton of people telling me exactly how i’m wrong)
@foone
Ooh! Accessibility APIs might help you here; most modern OS situations (not Linux-Wayland) offer a screenreader API and/or braille terminal access. brltty especially might be the interface you want, then you don't need the HDMI capture hardware?
Then you can write your "wait on my behalf while I do something better" scripts to speak braille-tty or screen reader or whatever.
@silvermoon82 It's a good start but frankly I consider Accessibility APIs as part of the enemy I am fighting. I don't want permission to be able to automate: I am going to automate, no matter what the software developer chooses to do
@foone It's one of the many things that has gotten way worse about using a computer over the last 30 years.
Basically, you want AppleScript from the early 90's. It worked pretty well for automating GUI apps for like ten minutes, then everything was web apps and half-baked cross platform toolkits and Apple stopped caring about automation and moved to focus on walled gardens.
@foone kinda sounds like you want a KVM that works over the network.
You got the screen as a digital image, and can send key and mouse events (with a bit of hacking)
@foone I'll say! Waiting is the wooorrrrsssttt. I hate it to my bones. Maybe bcs ADHD but still it winds me up until I explode when t happens too often
@foone I am not saying "AI" is the solution; the quotes are load-bearing. :P
I am saying "what if we automated this human interface" is a problem that really requires "make a virtual human" to solve, and ends getting Selenium, or a very bad hackjob reimplementation of Selenium, or a bunch of regexps around a python script pretending to be Firefox.
Adding more layers insteading of fixing the existing ones is how you get everything Google touch, and nobody wants that.
but what if "fixing the existing one" is not an option. What do you do? Do you fix it? or do you give up on ever using that program ever again, even if it's required for your job?
@foone Based on what I’ve seen, I don’t think there is one with a native HDMI captive port, and the HDMI capture USB (3.0) dongles are all pretty crappy (elgato is supposedly “ok”, but I don’t think it’s that good.) Probably better building a SFF PC and using a low profile PCIe capture card.
With USB C you could in theory build all this into a cable with a small box in the middle but unfortunately nobody has that I know of. All the individual pieces exist, it just needs 6 or 7 cables. :(
@foone I hear you but what happens if the window moves or changes dimensions? Any sufficiently advanced visual recognizer is either AI or indistinguishable from AI.
let’s consider instead Microsoft’s position on apps and Powershell. To paraphrase “In Unix, everything is a file, in Windows everything is an API.” So most of the administrative and mgmt features of windows are (or have) APIs that care hooked directly or indirectly through powershell.
@foone i remember i kinda did that with an old Win7 laptop to make Papa's Pizzeria Bad Apple because I don't know a reasonable way to directly make Flash think that I'm using the mouse a certain way, so I just fudged it with AHK and ran it on that machine for, a few weeks or something, however long it took to do every frame
@foone I’ve been saying for years that I’d love to write a sufficiently complex series of scripts and utilities to do an entire day’s work for me automatically and have it fired off each morning by me pressing a huge button with JOB written on it while I eat breakfast
@foone tfw when you use macro programs to click on GUI elements then need to hunt down a program to automatically align windows in the precise location so the event clicks land on the right ui buttons
@foone Nobody knows when to write a library, when to write a CLI, when to write a GUI, when to provide a network service, or when to provide a website. It is a goddamn mystery, and no matter what is chosen, what is written will always suck.
@foone I had this abandoned project to build a simple GUI toolkit for my toy OS where everything user can somehow interact with has public API so that also script or program can interact with it in sane way without external tools.
@foone That was the vision behind AppleScript/AppleEvents. You’d write GUI software using an AppleEvents-based framework, and then it would automatically be 100% scriptable and even have the actions be recordable for people who didn’t want to code.
Of course, nobody wanted to rewrite their software from scratch, and “early 90’s object oriented frameworks” were a massive PITA to use, so in the end it was just another “what the coder could be bothered to allow” macro language