Another YC company was acquihired today by OpenAI
15 by liurenju | 6 comments on Hacker News.
Wednesday, September 3, 2025
New top story on Hacker News: Show HN: Entropy-Guided Loop – How to make small models reason
Show HN: Entropy-Guided Loop – How to make small models reason
11 by andrewmonostate | 0 comments on Hacker News.
TLDR: A small, vendor-agnostic inference loop that turns token logprobs/perplexity/entropy into an extra pass and reasoning for LLMs. - Captures logprobs/top-k during generation, computes perplexity and token-level entropy. - Triggers at most one refine when simple thresholds fire; passes a compact “uncertainty report” (uncertain tokens + top-k alts + local context) back to the model. - In our tests on technical Q&A / math / code, a small model recovered much of “reasoning” quality at ~⅓ the cost while refining ~⅓ of outputs. I kept seeing “reasoning” models behave like expensive black boxes. Meanwhile, standard inference already computes useful signals both before softmax normalization and after it(logprobs), which we usually throw away. This loop tries the simplest thing that you could think of: use those signals to decide when (and where) to think again. GitHub (notebook + minimal code): https://ift.tt/v2pLKoZ Paper (short & engineer made): https://ift.tt/ru7BVew Blog (more context): https://ift.tt/iPx1kMp Requirements: Python, API that exposes logprobs (tested with OpenAI non reasoning 4.1). OPENAI_API_KEY and WEAVE for observability. Run the notebook; it prints metrics and shows which tokens triggered refinement. - Python, simple loop (no retraining). - Uses Responses API logprobs/top-k; metrics: perplexity, max token entropy, low-confidence counts. - Weave for lightweight logging/observability (optional). - Passing alternatives (not just “this looks uncertain”) prevents over-correction. - A simple OR rule (ppl / max-entropy / low-confidence count) catches complementary failure modes. - Numbers drift across vendors; keeping the method vendor-agnostic is better than chasing fragile pairings. - Needs APIs that expose logprobs/top-k. - Results are indicative—not a leaderboard; focus is on within-model gains (single-pass vs +loop). - Thresholds might need light tuning per domain. - One pass only; not a chain-of-thought replacement. - Run it on your models and ideas (e.g., 4o-mini, v3, Llama variants with logprobs) and share logs in a PR for our README in GitHub if you'd like, PRs welcome - I’ll credit and link. Overall let me know if you find making small models reason like this useful!
11 by andrewmonostate | 0 comments on Hacker News.
TLDR: A small, vendor-agnostic inference loop that turns token logprobs/perplexity/entropy into an extra pass and reasoning for LLMs. - Captures logprobs/top-k during generation, computes perplexity and token-level entropy. - Triggers at most one refine when simple thresholds fire; passes a compact “uncertainty report” (uncertain tokens + top-k alts + local context) back to the model. - In our tests on technical Q&A / math / code, a small model recovered much of “reasoning” quality at ~⅓ the cost while refining ~⅓ of outputs. I kept seeing “reasoning” models behave like expensive black boxes. Meanwhile, standard inference already computes useful signals both before softmax normalization and after it(logprobs), which we usually throw away. This loop tries the simplest thing that you could think of: use those signals to decide when (and where) to think again. GitHub (notebook + minimal code): https://ift.tt/v2pLKoZ Paper (short & engineer made): https://ift.tt/ru7BVew Blog (more context): https://ift.tt/iPx1kMp Requirements: Python, API that exposes logprobs (tested with OpenAI non reasoning 4.1). OPENAI_API_KEY and WEAVE for observability. Run the notebook; it prints metrics and shows which tokens triggered refinement. - Python, simple loop (no retraining). - Uses Responses API logprobs/top-k; metrics: perplexity, max token entropy, low-confidence counts. - Weave for lightweight logging/observability (optional). - Passing alternatives (not just “this looks uncertain”) prevents over-correction. - A simple OR rule (ppl / max-entropy / low-confidence count) catches complementary failure modes. - Numbers drift across vendors; keeping the method vendor-agnostic is better than chasing fragile pairings. - Needs APIs that expose logprobs/top-k. - Results are indicative—not a leaderboard; focus is on within-model gains (single-pass vs +loop). - Thresholds might need light tuning per domain. - One pass only; not a chain-of-thought replacement. - Run it on your models and ideas (e.g., 4o-mini, v3, Llama variants with logprobs) and share logs in a PR for our README in GitHub if you'd like, PRs welcome - I’ll credit and link. Overall let me know if you find making small models reason like this useful!
New top story on Hacker News: Vector search on our codebase transformed our SDLC automation
Vector search on our codebase transformed our SDLC automation
17 by antonybrahin | 2 comments on Hacker News.
Hey HN, In software development, the process of turning a user story into detailed documentation and actionable tasks is critical. However, this manual process can often be a source of inconsistency and a significant time investment. I was driven to see if I could streamline and elevate it. I know this is a hot space, with big players like GitHub and Atlassian building integrated AI, and startups offering specialized platforms. My goal wasn't to compete with them, but to see what was possible by building a custom, "glass box" solution using the best tools for each part of the job, without being locked into a single ecosystem. What makes this approach different is the flexibility and full control. Instead of a pre-packaged product, this is a resilient workflow built on Power Automate, which acts as the orchestrator for a sequence of API calls: Five calls to the Gemini API for the core generation steps (requirements, tech spec, test strategy, etc.). One call to an Azure OpenAI model to create vector embeddings of our codebase. One call to Azure AI Search to perform the Retrieval-Augmented Generation (RAG). This was the key to getting context-aware, non-generic outputs. It reads our actual code to inform the technical spec and tasks. A bunch of direct calls to the Azure DevOps REST API (using a PAT) to create the wiki pages and work items, since the standard connectors were a bit limited. The biggest challenge was moving beyond simple prompts and engineering a resilient system. Forcing the final output into a rigid JSON schema instead of parsing text was a game-changer for reliability. The result is a system that saves us hours on every story and produces remarkably consistent, high-quality documentation and tasks. The full write-up with all the challenges, final prompts, and screenshots is in the linked blog post. I’m here to answer any questions. Would love to hear your feedback and ideas!
17 by antonybrahin | 2 comments on Hacker News.
Hey HN, In software development, the process of turning a user story into detailed documentation and actionable tasks is critical. However, this manual process can often be a source of inconsistency and a significant time investment. I was driven to see if I could streamline and elevate it. I know this is a hot space, with big players like GitHub and Atlassian building integrated AI, and startups offering specialized platforms. My goal wasn't to compete with them, but to see what was possible by building a custom, "glass box" solution using the best tools for each part of the job, without being locked into a single ecosystem. What makes this approach different is the flexibility and full control. Instead of a pre-packaged product, this is a resilient workflow built on Power Automate, which acts as the orchestrator for a sequence of API calls: Five calls to the Gemini API for the core generation steps (requirements, tech spec, test strategy, etc.). One call to an Azure OpenAI model to create vector embeddings of our codebase. One call to Azure AI Search to perform the Retrieval-Augmented Generation (RAG). This was the key to getting context-aware, non-generic outputs. It reads our actual code to inform the technical spec and tasks. A bunch of direct calls to the Azure DevOps REST API (using a PAT) to create the wiki pages and work items, since the standard connectors were a bit limited. The biggest challenge was moving beyond simple prompts and engineering a resilient system. Forcing the final output into a rigid JSON schema instead of parsing text was a game-changer for reliability. The result is a system that saves us hours on every story and produces remarkably consistent, high-quality documentation and tasks. The full write-up with all the challenges, final prompts, and screenshots is in the linked blog post. I’m here to answer any questions. Would love to hear your feedback and ideas!
Tuesday, September 2, 2025
Monday, September 1, 2025
Sunday, August 31, 2025
Subscribe to:
Posts (Atom)