A Markdown-based test suite
57 points by zdw 4 days ago | 13 comments
  • nibblecid a day ago |
    Amazing Post! Got intrigued when I read markdown because I’m currently working on a project related to Markdown and I need to understand the agents thing!
  • abusaidm a day ago |
    Interesting approach, I can see the benefits of such approach where the md is acting like a Jupyter notebook of some sort. You place the content with documentation and it becomes a self standing unit of test.

    I wonder if this can be replicated to other notions where the docs/data needed for test add more value to live with the test

  • riffraff a day ago |
    The "test anything protocol" was a text based system for writing tests, I think perl might still use it

    I remember using it to implement the test suite for a Shakespeare language interpreter. Fun times.

    https://en.wikipedia.org/wiki/Test_Anything_Protocol

    • catketch a day ago |
      TAP is not really about writing tests, it's more about making sure that a system under test emits plain text logs that can easily be consumed/validated

      The markdown based tests described is more of an analog to gerken, robotframework, etc.

      there's a number of similar efforts. turning english-like specs for a test into executable tests has seens lots of variants over the years but LLMs have reaaly stepped up the possibilities

  • dhruv3006 a day ago |
    I think you will like https://voiden.md/ - some what similar to what you did here.

    PS: I am currently working on Voiden.

    • rubyn00bie a day ago |
      You’re developing that so it would be nice to have the disclosure.

      Additionally, I cannot find any obvious examples of what it does, or how it possible relates, and none of the sections seem to offer anything even resembling an example. The entire site seems like a marketing template.

      FWIW— if it is similar, or there are examples, you should absolutely put them to the forefront. I literally have no idea what any of the statements/propositions in the pages actually mean.

      • dhruv3006 a day ago |
        yep editing my comment !
  • eichin a day ago |
    Anybody remember "cram"? From about 10 years ago, https://bitheap.org/cram/ basically a markdown syntax (making heavy use of code-blocks) for documenting and writing tests as "shell commands and expected output" (with a bunch of the sharp edges filed off, like line endings and partial matches.) Was particularly good for easy-to-write, easy-to-review tests of unix utilities. (It's the kind of thing that you only stumble on if you've been working with doctests but they don't really fit well for shell/unix stuff...)
  • thangalin a day ago |
    Speaking of test suites, LLMs cannot quite curl straight quotes correctly. My Markdown editor uses the following suite:

    https://repo.autonoma.ca/repo/keenquotes/tree/HEAD/src/test/...

    Am curious whether SOTA LLMs can curl the ambiguous cases.

    • echoangle a day ago |
      Is curl a synonym of grok?
      • luplex a day ago |
        no, this is just about the shape of the quote.

        There are straight quotes: ' or "

        and there are curly quotes: ‘ and ’ or “ and ”

        "curling" then just means "figuring out whether it's an opening or closing quote"

  • pyrolistical a day ago |
    You can also write Gherkin in markdown and have test fixtures for each given, when, then