dittu.org

Dittu controls a real computer through a phone camera + Bluetooth mouse/keyboard. The camera view can be imperfectly aligned, and clicks land at normalized (nx, ny) coordinates that may drift slightly from what you'd expect. Follow this workflow: 1. Call take_screenshot (or take_screenshot_with_grid for fine targets) BEFORE acting, to see the current state — never act blind on stale information. 2. Before clicking a small or important target, call move_mouse(nx, ny) first, then take_screenshot to confirm the cursor landed where you intended. Only call click() once you've visually confirmed the position. 3. After any action (click, type_text, key, scroll), take a fresh screenshot before deciding the next step — the screen state may have changed in ways you didn't expect. 4. If a click seems to have missed, don't repeat blindly — re-screenshot, re-locate the target, and re-move the mouse before retrying. Fixing screen-corner calibration (use if clicks are consistently off in one area, or if the visible screen looks cropped/has black margins): 1. get_corners() — read the current tl/tr/br/bl quadrilateral. These four points are an arbitrary trapezoid (perspective), NOT a rectangle — never assume symmetry. 2. Pick the edge that looks wrong. Call move_mouse(nx, ny) near that edge (e.g. nx=0.02 for the left edge, nx=0.98 for the right edge, keeping ny mid-screen-ish). 3. Call take_screenshot_with_crosshair(nx, ny) — it crops tightly around that point, zooms in, and draws a red crosshair at exactly where the cursor SHOULD be. 4. Compare: if the real cursor is missing from frame entirely, that edge is cropped — move the two corners on that edge further outward (away from screen center) in raw-camera-space. If the cursor appears offset from the crosshair, the corners need a small nudge in the direction of the offset, scaled by how far you are from that corner along each edge. 5. Call set_corners(...) with the adjusted quadrilateral, then repeat steps 2-4 to verify. 2-3 iterations is usually enough; don't over-correct on a single step.