9 comments

  • _pdp_ 3 minutes ago
    This is the plasma TV phase of local AI.

    Early, heavy and priced premium. I estimate it will take ~5 years for most practical AI to run on our own machines, with a few super brains left in the cloud.

    I have been putting some effort into making local AI as consumer friendly as possible on macOS with this project https://github.com/pdparchitect/noodle/ but I will be the first to admit that the hardware you need is beyond most people's budgets. Hell, perhaps beyond most enterprises comfort levels too.

    For now, subscriptions are the most practical way in.

  • thangalin 5 minutes ago
    Since Spring, I've been working on a locally hosted, containerized web application to narrate my audiobook using a full cast of characters (skip to 9m47s for a reading of the demo book):

    https://www.youtube.com/watch?v=Q9PKT6bdKnc

    All weights are either permissively licensed or free until fantasy-level monthly recurring revenue sets in. The system runs entirely on the following hardware:

        CPU: AMD Ryzen 5 7600 (Zen 4), 6 cores / 12 threads, AVX-512
        GPU: NVIDIA T1000, 8 GB GDDR6 (Turing, 896 CUDA cores, 50 W, no BF16)
        RAM: 96 GB DDR5
    
    Text analysis uses Qwen3.6-35B-A3B MTP (Q4_K_XL GGUF) for speaker attribution, voice descriptions (not voice audio), emotive directions, and grammar. Both Muse Glimmer and GRMR cannot compete with Qwen3.6 for grammar checking. I haven't tried Gemma4, but I suspect Qwen3.6 remains the champ. My hope is that Inkling is more accurate for grammar than Qwen3.6, which is being tested at time of writing.

    Voice design and cloning uses FireRedTTS3, OmniVoice, Qwen3-TTS, VoxCPM2, and Ming-3.0-tts-tiny. Most are approximately equivalent. For narration, nothing comes close to Drama 3 Expressive Medium fine-tune in terms of quality, consistency, or pacing.

    Unsloth's GGUFs have been absolutely instrumental in making the system a reality. Highly recommend using them.

  • snappr021 11 minutes ago
    Doesn’t sound to far off the simplexity chatted to ChatGPT and Claude yesterday.

    When entire operating systems and software stacks to run end to end business processes, including engineering is a reality, Bill’s word “optimise” is close to the ultimate fulfillment.

    There is too much compute in data centres, when multiple machines can be replaced with a functionally intelligent pi.

  • junga 16 minutes ago
    Set this up this weekend after AMD NPU support was announced for Unsloth. My ThinkPad x13 with the Ryzen AI 350 and 32 GB RAM is now serving my older GhostBSD ThinkPad running crush. Using Qwen3.5-9B (npu2) the quality of course is not comparable to the latest frontier models. Basically no result can be trusted. And 12 tokens per second are quite relaxing but it somehow feels 'right'.
  • angry_octet 25 minutes ago
    So pretty soon 10,000+ people could be running decent models on their home PCs with unsecured LAN access, connected to their residential modem? We won't just have residential proxy networks, we'll have residential ARTEX swarm clouds.
    • geraneum 17 minutes ago
      What do exactly expect to happen and how is it worse running other people’s agents on our machines?
  • IsTom 39 minutes ago
    Saying that 32GB+ VRAM is needed for Qwen3.6-35B@q4 is certainly an interesting statement.
    • steeleduncan 37 minutes ago
      That leaves 10GB(+) for context and KV cache, which seems about right
      • IsTom 36 minutes ago
        It's MoE and it runs on 3060 just fine.
    • baoooooooooooo 24 minutes ago
      Yeah, was this written before strata?
  • amelius 22 minutes ago
    What do the local-llmers do to get an experience closer to Claude?

    Do you reuse Claude's plugins/skill files?

    Do you use a special harness or system prompts?

    • segmondy 18 minutes ago
      You can't get experience like clause but you can get result like Claude for 90% of tasks with more effort.
    • comprev 19 minutes ago
      OpenWebUI, OpenTerminal and OpenClaw are useful tools for using local models. There are dozens more tools available depending on your needs.
  • bashhq_r 42 minutes ago
    how much vram do you need before qwen stops feeling sluggish on a normal laptop
    • yread 12 minutes ago
      I run Basalt with Qwen 3.8 flash next (default Basalt NVFP4 quant) on 5070ti (16GB VRAM) with 96GB RAM. Prefill ~2000tok/s, generation ~100 tok/s. Definitely doesn't feel sluggish but it's not a laptop
    • Retell15 39 minutes ago
      You need more processor power.
    • steeleduncan 39 minutes ago
      which qwen model?
  • poulpy123 28 minutes ago
    Easy, you just need a 10 000€ graphic card !