This is part of our commitment to R in production. As R applications scale – more users, more processes, more machines – you need tools to understand what’s happening across your entire system. That’s what OpenTelemetry is built for, and it’s now available for R.

What is OpenTelemetry?

OpenTelemetry (OTel) is a vendor-neutral, open source observability framework backed by the Cloud Native Computing Foundation . It has broad industry support across languages and platforms, and is already the standard in the Python, Java, JavaScript, and Go ecosystems. Now it’s available for R.

OpenTelemetry defines a standard for collecting telemetry data:

• Traces follow an operation as it moves through your system, showing exactly which functions ran, in what order, and how long each took.
• Metrics capture numerical measurements over time – things like request counts, response latencies, or memory usage.
• Logs record detailed events as they happen, providing context when you need to investigate a specific moment.

The instrumented packages described in this post all use the otel package under the hood, and focus on traces, which provide the most immediate value for understanding production behavior. You can also use otel directly to add your own metrics and logs to your application code.

Why observability matters for R in production

When you’re developing interactively in RStudio or Positron, debugging is straightforward – you can step through code, inspect objects, and add print statements. But when your R code runs in production – a Shiny app serving hundreds of users, a plumber2 API handling thousands of requests, a batch pipeline running across a cluster – the picture changes.

The core concept in OpenTelemetry is a trace: the full path of a request through your system. Each trace is made up of spans – individual units of work with a name and a duration. Spans nest inside each other, so you can follow how each operation triggered the next:

This structure gives you four things that are hard to get any other way:

• Performance: Which part of a request is slow? Span durations pinpoint where time is spent – and nesting reveals unnecessary overhead.
• Errors: In development and testing, you know where errors are – you wrote the test, you control the inputs. In production, errors surface far from their root cause, across process boundaries and async operations, triggered by conditions you never anticipated. Traces show you the full chain of real operations that led to each failure, in the context where it actually happened.
• Centralized view: When your application extends across multiple R processes or machines – a Shiny app with mirai workers, or a plumber2 API behind a load balancer – traces are aggregated into a single view across all of them.
• Real-time monitoring: OTel is designed to be left on in production, not just enabled during testing or staging. With low overhead, it runs continuously – so you see what’s happening as it happens, and dashboards and alerts catch problems before users report them.

Instrumented packages

We’ve worked across teams to add OpenTelemetry instrumentation to the R packages where it matters most:

Together, these packages cover the most common production R workloads: web applications, APIs, parallel computing, HTTP clients, AI/LLM tools, report rendering pipelines, CI test runs, and database access. Because the instrumentation is built into the packages themselves, you benefit from it automatically – no wrapper functions, nor modifications to existing code. If you already have a Shiny app or plumber2 API, your app will generate traces as soon as you enable OpenTelemetry. Your application code stays exactly as it is.

Seeing it in action

To make this concrete, let’s look at a Shiny chat app built with shinychat and ellmer that fetches weather forecasts. It uses mirai for async execution and httr2 for weather API requests.

A user asks about the weather in Atlanta and Newcastle:

With OpenTelemetry enabled, every step is captured automatically. Here are the traces from those two queries:

The trace reveals the full chain of operations from user input through to the HTTP request, across process boundaries, with no manual logging. The nesting shows how each step triggered the next, and the durations show where time was spent.

The second query failed. Without tracing, you’d see an error in your logs and start investigating. Here, the failure is immediately visible – the red span pinpoints where it occurred and the surrounding context shows why. In a production system with many concurrent users, that’s the difference between minutes and seconds of debugging.

Getting started

Getting started requires the otelsdk package and a few environment variables. No changes to your application code.

Here’s how the pieces fit together: the instrumented R packages generate telemetry data as they run. The otelsdk package collects and exports this data over HTTP to a backend – a service that stores your traces and provides a web dashboard where you can search, filter, and visualize them (like the trace screenshots above).

Step 1: Install otelsdk

install.packages("otelsdk")

Step 2: Choose a backend

OpenTelemetry is vendor-neutral, so you can send your data to any compatible backend:

• Cloud services: Logfire , Grafana Cloud , Langfuse
• Self-hosted: Jaeger , Zipkin , Prometheus

During development, you can also run an OpenTelemetry Collector on your own machine and inspect traces directly, without sending data anywhere else.

Each backend will give you an endpoint URL and an authentication token. The otelsdk collecting telemetry data guide has examples for some common backends.

Step 3: Set environment variables

Add these to your .Renviron file (use usethis::edit_r_environ() to open it), replacing the endpoint and token with the values from your chosen backend. This example uses Logfire , which offers a free tier to get started:

OTEL_TRACES_EXPORTER="http"
OTEL_EXPORTER_OTLP_ENDPOINT="https://logfire-eu.pydantic.dev"
OTEL_EXPORTER_OTLP_HEADERS="Authorization=<YOUR-WRITE-TOKEN>"

If you’re deploying content to Posit Connect, refer to how to set environment variables on Posit Connect .

Step 4: Run your app

That’s it. Restart R, then run your Shiny app, plumber2 API, or any code that uses the instrumented packages. Traces will flow to your backend automatically. Open your backend’s web dashboard to see them – you’ll see a view like the trace screenshots shown above, with each span representing an operation in your application.

You can verify that tracing is active at any time:

otel::is_tracing_enabled()
#> [1] TRUE

OpenTelemetry is designed to be safe in production. If anything goes wrong in the telemetry code itself, it will never crash your application – errors are silently suppressed so your app keeps running.

Zero-code instrumentation

Beyond the packages that ship with built-in instrumentation, otel supports zero-code instrumentation for any R package. Set the OTEL_R_INSTRUMENT_PKGS environment variable to a comma-separated list of package names, and otel will automatically create spans for their exported functions:

OTEL_R_INSTRUMENT_PKGS=dplyr,tidyr

You can also fine-tune which functions are instrumented using include and exclude filters:

OTEL_R_INSTRUMENT_PKGS_DPLYR_INCLUDE=mutate,filter,select

This is useful for adding visibility to any package in your stack, even those without built-in OTel support. See the otel documentation for full details.

Configuration options

The otelsdk package is configured entirely through environment variables, following OpenTelemetry conventions:

See the otelsdk environment variables reference for the complete list.

Looking ahead

With instrumentation built into the packages that R users already rely on, observability becomes something you can turn on, not something you have to build.

We’re also integrating an OpenTelemetry collector into Posit Connect , giving you an end-to-end observability solution you can simply turn on or off.

OTel support across the ecosystem continues to expand. If you’d like to learn more:

• otel package documentation – the instrumentation API
• otelsdk package documentation – the SDK for collecting and exporting telemetry
• Shiny 1.12 OTel blog post – deep dive into Shiny’s OpenTelemetry support
• OpenTelemetry project – the upstream standard

We’re excited about what this opens up for the R community. Whether you’re running a Shiny dashboard for a small team, a plumber2 API serving thousands of requests, or a data pipeline distributed across a cluster – you now have the tools to see what’s happening, in real time, with no code changes required.

Welcome to our newsletter, posit::glimpse()!

If you’re currently reading this on our blog, consider subscribing to Product Updates - Open Source on our subscription page to receive this newsletter directly in your inbox.

posit::glimpse() is our roundup of the most important open-source news for Posit’s community! We’ve moved to monthly editions, and we still have so much to share.

Registration for posit::conf(2026) is now open!

Check out the keynote speakers , workshops , and agenda for our upcoming conference, happening in Houston and online. It’s sure to be incredible! Register here .

Key product updates and new releases

ggsql alpha release

The alpha release of ggsql brings the grammar of graphics to SQL, enabling powerful data visualization directly within SQL queries using declarative clauses. Built by the ggplot2 team with 18 years of experience, ggsql executes visualization computations as optimized SQL queries on backends, works without R or Python runtimes, and is designed for integration with Quarto, Jupyter, Positron, and AI agents.

• Learn more in the ggsql alpha release blog post.
• We love seeing folks in the community be early adopters! Harry Snart shares how he explored ggsql with Oracle Database , and Ansgar Wolsing tried it out for the #30DayChartChallenge .

RAG with raghilda

LLMs are great at reasoning and generating text, but their knowledge is frozen at training time. Retrieval-augmented generation (RAG) solves this by giving the model access to relevant information at query time, without needing to retrain it.

The new raghilda package simplifies building RAG systems in Python. It supports multiple storage backends (DuckDB, ChromaDB, OpenAI Vector Stores) with a consistent API, enabling teams to start locally and scale to hosted solutions without code rewrites.

• Read more in the RAG with raghilda blog post.

Quarto 1.9 and Quarto 2

The Quarto team has announced plans for Quarto 2, a complete rewrite in Rust with a major focus on collaborative editing! Key features include a collaborative editor for web and command-line, a visual editor that works seamlessly alongside source editing without disrupting code, and enhanced error detection across projects. The public release is expected in at least 6 months.

• Learn more in the What’s next: Quarto 2 announcement .

Meanwhile, be sure to take a look at all the exciting updates from v1.9, including:

• Chrome Headless Shell in Quarto
• Typst Books, Article Layout, and typst-gather
• PDF Accessibility and Standards

All v1.9 updates can be found in the roundup blog post .

What’s new in Positron

Our new data science IDE, Positron , has been significantly updated with numerous improvements, including Positron Server for Academic Use via JupyterHub, AI enhancements , telemetry updates and R improvements (like Addins!).

• Read more in the April 2026 Newsletter .
• For continuous updates on Positron, follow the new Positron LinkedIn page and subscribe to the Positron newsletter .

Shiny for Python 1.6

Shiny for Python 1.6 is now available. The release introduces toolbar components designed for tight spaces such as card headers, input labels, and AI chat interfaces.

The second major addition is built-in OpenTelemetry support, enabling zero-configuration observability by automatically tracing session lifecycles, reactive updates, and individual reactive expressions. Set SHINY_OTEL_COLLECT=reactivity and send traces to any OTLP-compatible backend like Pydantic Logfire, Jaeger, or Grafana Cloud.

• Read more in the Shiny for Python 1.6 brings toolbars and OpenTelemetry blog post.

Pointblank 0.24.0

Pointblank is a package for data validation. With the release of Pointblank 0.24.0, there is now OpenTelemetry integration for bridging data validation and production observability. After running checks on a table, you can push pass/fail metrics, per-validation-step trace spans, and structured threshold-breach logs to your OTel-compatible backend (Grafana, Datadog, New Relic, etc.). For those that run pipelines in Airflow, Prefect, or Dagster, validation spans slot into existing distributed traces automatically.

• Read the full OTel integration guide for setup instructions and examples.

Tidymodels

The tidymodels team is on a roll! There are new updates available for dial, parsnip, yardstick, tune, and tidymodels, as well as two new cheatsheets.

• Read the New tidymodels Releases for April 2026 blog post and tidymodels Cheatsheet blog posts.

The group has also been developing a set of skill files for machine learning with tidymodels as well as developer focused skills. You can find the current versions at skills.tidymodels.org . Give them a try; the group would love some feedback.

mori

mori is a new R package for shared memory across processes. ​​Parallel R no longer has to mean duplicating your dataset in every worker’s RAM. mori places it in OS-level shared memory once, and every worker maps the same physical pages via R’s ALTREP framework. Works with any parallel backend that uses R serialization, including mirai, parallel, and callr.

• Learn more in the mori 0.1.0 blog post.
• Tyler Morgan-Wall has already implemented a fork to the targets package to incorporate mori! Read the discussion here . It’s fantastic to see the community adopt and improve the ecosystem with open source.

tabpfn 0.1.0

The new tabpfn v0.1.0 package provides an R interface to TabPFN, a pretrained deep learning model for tabular data that delivers strong predictive performance without requiring model training. The package integrates with tidymodels syntax and future updates will add parsnip model types and additional interpretability tools.

• Learn more in the tabpfn 0.1.0 blog post.

nanoparquet 0.5.1

nanoparquet is a small, self-sufficient R package for reading and writing Parquet files. Version 0.5.1 introduces list columns, bit64::integer64 and blob::blob support, writing Parquet to the standard output.

• Read more in the nanoparquet 0.5.1 blog post.

torch Ecosystem Updates

The team has expanded torch ecosystem support to include cudatoolkit packages, torchvision datasets, and advanced model architectures and transformations for computer vision.

• Read more in the torch Ecosystem Updates blog post.

roxygen2 8.0.0

roxygen2 uses specially formatted comments in your R code to generate .Rd files. This version offers new support for S7, a raft of improvements to R6 documentation, a more natural way to configure roxygen2 in your DESCRIPTION, the changes to rendered .Rd files you’re most likely to see, and some other minor improvements, and a bunch of new vignettes.

• Learn more in the roxygen2 8.0.0 blog post.

Great Docs

Last month, we introduced Great Docs for beautiful documentation for Python packages. Author Rich Iannone shares more details in the Great Docs introductory blog post .

Learning and community

Posit website relaunch

The Posit website has a new look! Check out the refreshed pages, in particular the wonderful demo gallery with examples of workflows using Posit tools.

Showcases from the community

There are so many community examples to share, here is just a small sample():

Dianyi Yang , DPhil candidate in Politics at the University of Oxford (DPIR), shares practical guide to structuring reproducible academic research projects using Git, renv, Quarto, and GitHub, from data cleaning to manuscript preparation.

Read the blog post here !

Sophia Zilber shared a public-facing dashboard for the Cure Mito Foundation Leigh syndrome patient registry using Shiny for R.

See the dashboard here .

Tom Geens and his team used Quarto to generate HTML and LaTeX reports on occupational accidents in Belgium.

See the report here .

Gen Re uses Posit Connect to automate their underwriting workflow, ingesting broker submissions every minute and routing them through AI services that extract key information and generate an early risk assessment. What used to take 30 minutes per submission now takes just 5, saving the team roughly 600 hours of cumulative processing time each day.

Check out their story .

What’s next?

You can join us every Tuesday at the Data Science Lab and every Thursday at the Data Science Hangout !

• On May 12, Nicola Rennie will live code a TidyTuesday visualization from end-to-end and share the secrets of her craft!
• On May 21, our DSH will be a Data Career Panel with Gabriela de Queiroz, Dan Boisvert, and Makarand Malu. Bring your career questions about the field of data, hiring, asking for promotions, and more!

I’m a real person, and I would love to hear any feedback on the newsletter! Find me on LinkedIn and Bluesky , or email me at isabella [dot] velasquez [at] posit.co.

Install

Install once:

uvx skillsmd add has2k1/plotnine-skill

or

npx skills add has2k1/plotnine-skill

With either option, you will be prompted to choose the specific agents you wish to configure.

For versions of plotnine >= 0.15.4, the skill is built-in and you can install it using library-skills .

# Install into .agents
uvx library-skills install -s plotnine

# Install for Claude (.claude)
uvx library-skills install --claude -s plotnine

Try it

Once installed, simply ask your Agent to use plotnine and create a plot. For example:

Use plotnine to create a scatter plot of bill_length_mm vs bill_depth_mm from the penguins dataset, colored by species, with a colorblind-safe palette and a smoothed trend line per species.

Compare the results for the above prompt with skill and without skill.

• With Skill
• No Skill

from plotnine import ggplot, aes, geom_point, geom_smooth, scale_color_brewer, labs
from plotnine.data import penguins

(
    ggplot(
        penguins.dropna(),
        aes(x="bill_length_mm", y="bill_depth_mm", color="species"),
    )
    + geom_point(size=2, alpha=0.7)
    + geom_smooth(method="lm", se=True, alpha=0.2)
    + scale_color_brewer(type="qual", palette="Set2")
    + labs(
        x="Bill Length (mm)",
        y="Bill Depth (mm)",
        title="Penguin Bill Dimensions by Species",
        color="Species",
    )
)

from plotnine import aes, geom_point, geom_smooth, ggplot, scale_color_brewer
from plotnine.data import penguins

p = (
    ggplot(penguins, aes(x="bill_length_mm", y="bill_depth_mm", color="species"))
    + geom_point(alpha=0.7)
    + geom_smooth(method="lm", se=True)
    + scale_color_brewer(type="qual", palette="Dark2")
)
                       
p.save("scatter_penguins.png", width=8, height=6, dpi=100)

Note that even without a skill, the Agent (Claude Code Opus 4.6) generates readable and idiomatic code. The differences then map directly to some of the aspects that the skill enforces. In this case they are:

1. Accessibility — axis labels with units (where possible) and a real title
2. Data Completeness — penguins.dropna() means that the smoother isn’t quietly skipping rows.

As Agents are prone to doing, without a skill the Agent was presumptuous and generated code to save the plot to a file. The skill makes that experience tighter with fewer surprises, fewer fixes after the fact.

What is next?

We want this skill to rock, hard, and this is only the beginning. The vibes may seem good but, with intent, we want to build a sense of where the skill reliably helps and where it does not, so that performance and improvements are grounded in more formal evidence rather than vibes.

Your feedback can help. If you try it, tell us how it performs; the prompts it handled poorly, those were it was surprisingly exceptional, and even those where it was good but it can be exceptional. The show is still going on, and we will be glad to see what you show us at the repo .

install.packages("roxygen2")

This release has been a long time in the making and includes well over a hundred bug fixes and improvements. This post covers the highlights: new support for S7, a raft of improvements to R6 documentation, a more natural way to configure roxygen2 in your DESCRIPTION, the changes to rendered .Rd files you’re most likely to see, and some other minor improvements, and a bunch of new vignettes. You can read the full list of changes in the release notes .

library(roxygen2)

Documenting S7

The headline feature is initial support for S7 . S7 is a new object-oriented programming system built to be a successor to S3 and S4. It’s been designed and implemented collaboratively by the R Consortium Object-Oriented Programming Working Group, which includes representatives from R-Core, Bioconductor, tidyverse/Posit, and the wider R community. S7 is still a work in progress, but it is useful today, and adding roxygen2 support makes it easier to use in packages.

roxygen2 now supports documenting S7 generics, classes, and methods:

• Generics are documented just like regular functions.
• Classes are documented like regular functions (because S7 constructors are functions), but you can also use @prop to document properties that aren’t constructor parameters. If you document several related classes on the same page, @prop ClassName@prop_name description lets you group properties by class.
• Methods registered with method(generic, class) <- fn are picked up automatically, and roxygen2 generates the right usage and aliases for you.

Read vignette("rd-S7") for a full rundown, and please let us know if you discover any issues or have suggestions for improvement!

More ways to document R6

R6 got the biggest pile of improvements in this release, driven by years (!!) of accumulated feedback. The most important change is that you no longer need to document all methods inside the class definition. If you add methods with $set(), you can now document them directly:

Counter <- R6::R6Class(
  "Counter",
  public = list(
    count = 0
  )
)

#' Increment the counter by `by`.
#' @param by Number to add.
Counter$set("public", "increment", function(by = 1) {
  self$count <- self$count + by
  invisible(self)
})

For more exotic cases where methods get attached through code that roxygen2 can’t follow, there’s now @R6method Class$method, which lets you document a method from anywhere in your package.

You can also now opt out of documenting pieces of a class. Writing @noRd above an R6 method excludes it from the docs, and @field name NULL does the same for fields and active bindings.

Finally, inheritance got a significant upgrade. When a method overrides one from a superclass, it now automatically inherits that parameter documentation — so common parameters only have to be documented once, on the base class. The same is true for inherited fields and active bindings. This should cut down on a lot of duplication if you have a deep class hierarchy,

A cleaner home for configuration

roxygen2 has historically been configured through a Roxygen: field in DESCRIPTION, with a separate RoxygenNote field recording the version. This release introduces a more natural home using the Config/roxygen2/ namespace that packages like testthat and lifecycle have adopted:

# before
Roxygen: list(markdown = TRUE)
RoxygenNote: 7.3.2

# after
Config/roxygen2/markdown: TRUE
Config/roxygen2/version: 8.0.0

The old fields still work, and roxygen2 will quietly migrate RoxygenNote the next time you run devtools::document(). Over time, devtools and usethis will switch to the new form by default.

Rendering tweaks

When you re-document your package for the first time with roxygen2 8.0.0, you might notice some small differences in the rendered output. The most commonly encountered changes are:

• All generated cross-reference links now go through the same code path and share a single style, e.g. \code{\link[=compose]{compose()}}.

• Links to external packages now use the topic alias rather than the .Rd filename. This brings roxygen2 into alignment with current CRAN best practices.

• Tags that typically expect a single line of input now warn if you spread them across multiple lines. This catches a common class of silent mistakes, e.g. a @name or @rdname with no content that quietly included the next line of the block. This will create warnings for some existing legitimate uses, but I think the payoff (eliminating problems that are otherwise very hard to spot) is worth it.

• People with both "aut" and "cre" roles now appear in both the Authors and Maintainer sections of package documentation.

Other important changes

roxygen2 now requires R 4.1 and no longer depends on purrr, stringr, or stringi. Those last two are the big win because it means that stringi is now gone from the complete devtools dependency graph, making it easier to install for folks on constrained Linux images.

There’s a new helper, needs_roxygenize(), that tells you whether your .Rd files are out of date by comparing modification times against their sources. It’s much cheaper than running roxygenize() just to find out nothing has changed, which makes it a good fit for pre-commit hooks or CI checks.

Finally, parameter inheritance has gained a small but very useful feature: @inheritParams now supports filtering, just like @inheritDotParams. If you only want to inherit a subset of arguments from another function, you can now list them explicitly:

#' @inheritParams other_fn x y

Or exclude the ones you don’t want:

#' @inheritParams other_fn -z

Speaking of @inheritDotParams, it now also works more like @inheritParams: it inherits documented parameters rather than formal arguments. This may introduce new false positives (replacing the old approach’s false negatives), which you can prevent by explicitly listing the argument names to inherit.

New vignettes

roxygen2’s own documentation has had a significant tidy-up. The old vignette("rd-other") has been broken into focused vignettes — vignette("rd-datasets"), vignette("rd-packages"), vignette("rd-S3"), vignette("rd-S4"), and vignette("rd-R6") — with the new vignette("rd-S7") joining them. The main vignette("rd") has been renamed to vignette("rd-functions"), and the “getting started” content has moved to vignette("roxygen2").

vignette("rd-S3") has also been rewritten with clearer guidance for documenting S3 generics, classes, and methods, including how to use the new doclisting package to automatically list methods for a generic — a long-standing pain point for generics with methods in multiple packages. You can also use doclisting with S4 and S7 generics.

Acknowledgements

A big thank you to everyone who has contributed issues, pull requests, and discussion since the last release! @achubaty , @adithya604 , @akersting , @alandipert , @alannearme , @alecw , @alexgenin , @AlexisDerumigny , @AliSajid , @alisonmosky , @aljabadi , @allenzhuaz , @andrew-schulman , @andrewmarx , @aphalo , @apreshill , @arilamstein , @arnaudgallou , @ashbythorpe , @ateucher , @b-niu , @bahadzie , @balthasars , @BartJanvanRossum , @bastistician , @batpigandme , @beginb , @BenEngbers , @BenWiseman , @bgctw , @BGWKlein , @bhagwataditya , @billdenney , @Bisaloo , @bluewomble , @bobjansen , @boshek , @brendanf , @brodieG , @BroVic , @brpetrucci , @brry , @bryanhanson , @bwiernik , @cbielow , @cboettig , @cderv , @CGMossa , @chlebowa , @chrarnold , @ChristopherEeles , @chrk623 , @chuxinyuan , @cjyetman , @coatless , @ColinFay , @courtiol , @cthombor , @d-morrison , @d-sci , @daattali , @DanChaltiel , @DanielHermosilla , @danielvartan , @DarioS , @davidrubinger , @DavisVaughan , @daynefiler , @dfrankow , @dgkf , @dieghernan , @dipterix , @dmurdoch , @dpprdan , @drag05 , @dragosmg , @dsweber2 , @dvg-p4 , @dwachsmuth , @eddelbuettel , @eitsupi , @ejosymart , @elcortegano , @ElsLommelen , @espinielli , @FelixErnst , @florisvdh , @flrd , @gaborcsardi , @GABurns , @galachad , @gavinsimpson , @genomaths , @ggrothendieck , @ghost , @goldingn , @gowerc , @gregorgorjanc , @gustavdelius , @gwd666 , @hadley , @harrelfe , @hdarjus , @HenningLorenzen-ext-bayer , @HenrikBengtsson , @hongooi73 , @hughjonesd , @iferres , @IndrajeetPatil , @J-Moravec , @jakubnowicki , @jameslamb , @jan-abel-inwt , @JanaJarecki , @JanMarvin , @JantekM , @jcubic , @JDenn0514 , @jdprimus , @jeffcraggy , @jennybc , @jensmassberg , @jeroen , @jeroenjanssens , @JesseAlderliesten , @jgellar , @jgutman , @JiaxiangBU , @Jiefei-Wang , @jimhester , @jmbarbone , @jmpanfil , @johanneswerner , @johnbaums , @JohnCoene , @JonathanUrbach , @jonkeane , @jonocarroll , @jonthegeek , @JosiahParry , @jranke , @JulieBlasquiz , @jwijffels , @kamapu , @karchjd , @karoliskoncevicius , @kathi-munk , @kellijohnson-NOAA , @kevinushey , @kingaa , @klmr , @Klorator , @kongdd , @kortschak , @kostrzewa , @kpagacz , @krivit , @krlmlr , @kurt-o-sys , @kylebutts , @lgatto , @lindeloev , @LiNk-NY , @lionel- , @llrs , @llrs-roche , @lorenzwalthert , @LouisLeNezet , @LukasWallrich , @lukasz-bednarz-reddeersystems , @m-muecke , @maelle , @malcolmbarrett , @mamueller , @math-mcshane , @maxheld83 , @MaximilianPi , @mbojan , @mccarthy-m-g , @mccroweyclinton-EPA , @mcol , @mcsage , @MichaelChirico , @michaelquinn32 , @mikemahoney218 , @mikkmart , @mikmart , @MilesMcBain , @mine-cetinkaya-rundel , @MislavSag , @mjsteinbaugh , @mkoohafkan , @MLopez-Ibanez , @mnazarov , @mnneely , @monkeywithacupcake , @moodymudskipper , @mpadge , @mrchypark , @ms609 , @msaltieri , @msberends , @mschilli87 , @multimeric , @muschellij2 , @musvaage , @naikymen , @nathaneastwood , @nbenn , @nealrichardson , @Nelson-Gon , @neshvig10 , @netique , @ngreifer , @nick-robo , @NikNakk , @nlneas1 , @nouvrita , @nr0cinu , @nteetor , @ntguardian , @oharar , @okhoma , @olivroy , @orgadish , @p-carter , @p00ya , @pakjiddat , @pat-s , @PauloJhonny , @PavelBal , @pbreheny , @phargarten2 , @pnacht , @pvanlaake , @ralmond , @ramiromagno , @RaphaelS1 , @retostauffer , @RiboRings , @richelbilderbeek , @rjake , @RMHogervorst , @robchallen , @Robinlovelace , @romainfrancois , @rorynolan , @rossellhayes , @rsbivand , @russHyde , @rvernica , @s-fleck , @saipenikalapati , @Salatbesteck , @salim-b , @sbgraves237 , @sboehringer , @schloerke , @schradj , @sckott , @sebffischer , @setgree , @ShixiangWang , @simonpcouch , @simonsays1980 , @simpar1471 , @slager , @smilberg , @stefanfritsch , @stefanoborini , @stellathecat , @stemangiola , @stla , @strazto , @strboul , @sven-stodtmann , @swnydick , @t-kalinowski , @TanguyBarthelemy , @tappek , @tau31 , @tdhock , @ThierryO , @tjebo , @TomKellyGenetics , @tommarshall2 , @trusch139 , @turgeonmaxime , @tzakharko , @uhkeller , @unDocUMeantIt , @vertesy , @VPetukhov , @wch , @wibeasley , @wilcoxa , @wurli , @wviechtb , @yogat3ch , @Yunuuuu , @yutannihilation , @zachary-foster , @zeehio , @zettlchen , and @zkamvar .

We’re very chuffed to announce the release of nanoparquet 0.5.1 (and 0.5.0). nanoparquet is a small, self-sufficient R package for reading and writing Parquet files.

You can install it from CRAN with:

install.packages("nanoparquet")

This blog post will go over some of the improvements in nanoparquet 0.5.0 and 0.5.1.

You can see a full list of changes in the release notes here and here .

library(nanoparquet)

List columns

Parquet has a LIST type for columns whose values are variable-length sequences of scalars. nanoparquet 0.5.0 adds support for reading and writing such columns.

Note: for now nanoparquet supports one level of nesting: each element of a list column must be an atomic vector of a single type, not a list of lists. All elements in a column must have the same scalar type.

To write a list column, put a regular R list into your data frame. Each element must be an atomic vector (integer, double, or character), NULL for a missing list, or an empty vector for an empty list. NA values inside an element vector encode missing elements.

df <- data.frame(id = 1:4)
df$scores <- list(c(80L, 95L, 70L), c(100L), NULL, integer(0))
write_parquet(df, tmp <- tempfile(fileext = ".parquet"))

read_parquet_schema(tmp)

# A data frame: 5 × 14
  file_name  r_col name  r_type type  type_length repetition_type converted_type
  <chr>      <int> <chr> <chr>  <chr>       <int> <chr>           <chr>         
1 /var/fold…    NA sche… <NA>   <NA>           NA <NA>            <NA>          
2 /var/fold…     1 id    integ… INT32          NA REQUIRED        INT_32        
3 /var/fold…     2 scor… list(… <NA>           NA OPTIONAL        LIST          
4 /var/fold…     2 list  <NA>   <NA>           NA REPEATED        <NA>          
5 /var/fold…     2 elem… <NA>   INT32          NA OPTIONAL        INT_32        
# ℹ 6 more variables: logical_type <I<list>>, num_children <int>, scale <int>,
#   precision <int>, field_id <int>, children <list>

read_parquet() reads LIST columns back as R list columns:

as.data.frame(read_parquet(tmp))

  id     scores
1  1 80, 95, 70
2  2        100
3  3       NULL
4  4           

infer_parquet_schema() shows how nanoparquet maps each column to a Parquet type. For list columns, the r_type shows e.g. list(integer):

infer_parquet_schema(df)[2:7]

# A data frame: 4 × 6
  r_col name    r_type        type  type_length repetition_type
  <int> <chr>   <chr>         <chr>       <int> <chr>          
1     1 id      integer       INT32          NA REQUIRED       
2     2 scores  list(integer) <NA>           NA OPTIONAL       
3     2 list    <NA>          <NA>           NA REPEATED       
4     2 element <NA>          INT32          NA OPTIONAL       

A LIST column occupies three rows in the schema: the outer list node, a repeated group node, and the leaf element node.

When you need to specify the element type explicitly, you can use parquet_schema():

schema <- parquet_schema(
  id     = "INT32",
  scores = list("LIST", element = "INT32")
)
write_parquet(df, tmp2 <- tempfile(fileext = ".parquet"), schema = schema)

read_parquet_schema(tmp2)

# A data frame: 5 × 14
  file_name  r_col name  r_type type  type_length repetition_type converted_type
  <chr>      <int> <chr> <chr>  <chr>       <int> <chr>           <chr>         
1 /var/fold…    NA sche… <NA>   <NA>           NA <NA>            <NA>          
2 /var/fold…     1 id    integ… INT32          NA REQUIRED        <NA>          
3 /var/fold…     2 scor… list(… <NA>           NA OPTIONAL        LIST          
4 /var/fold…     2 list  <NA>   <NA>           NA REPEATED        <NA>          
5 /var/fold…     2 elem… <NA>   INT32          NA OPTIONAL        INT_32        
# ℹ 6 more variables: logical_type <I<list>>, num_children <int>, scale <int>,
#   precision <int>, field_id <int>, children <list>

New types

bit64::integer64

Parquet’s INT64 type holds 64-bit integers. R’s native integer is only 32 bits, so nanoparquet has mapped INT64 to double by default. nanoparquet 0.5.1 adds support for bit64::integer64, which gives you true 64-bit integer arithmetic in R.

write_parquet() now writes bit64::integer64 columns as INT64:

library(bit64)
df2 <- data.frame(id = as.integer64(c(1e18, 2e18, 3e18)))
write_parquet(df2, tmp3 <- tempfile(fileext = ".parquet"))
read_parquet_schema(tmp3)

# A data frame: 2 × 14
  file_name  r_col name  r_type type  type_length repetition_type converted_type
  <chr>      <int> <chr> <chr>  <chr>       <int> <chr>           <chr>         
1 /var/fold…    NA sche… <NA>   <NA>           NA <NA>            <NA>          
2 /var/fold…     1 id    double INT64          NA REQUIRED        INT_64        
# ℹ 6 more variables: logical_type <I<list>>, num_children <int>, scale <int>,
#   precision <int>, field_id <int>, children <list>

To read INT64 columns back as bit64::integer64 instead of the default double, use the read_int64_type option. The bit64 package must be installed; if it isn’t, nanoparquet throws a clear error.

read_parquet(tmp3, options = parquet_options(read_int64_type = "integer64"))

# A data frame: 3 × 1
       id
  <int64>
1    0e18
2    2e18
3    3e18

blob::blob

read_parquet() previously returned raw BYTE_ARRAY and FIXED_LEN_BYTE_ARRAY columns (i.e. those without a string, UUID, or decimal annotation) as plain lists of raw vectors. They are now returned as blob::blob objects, which print more neatly and come with the full set of blob helpers. write_parquet() now also accepts blob::blob columns, so round-tripping binary data is straightforward:

library(blob)
df3 <- data.frame(id = 1:3)
df3$payload <- blob::blob(
  charToRaw("hello"),
  charToRaw("world"),
  charToRaw("!")
)
write_parquet(df3, tmp4 <- tempfile(fileext = ".parquet"))
as.data.frame(read_parquet(tmp4))

  id   payload
1  1 blob[5 B]
2  2 blob[5 B]
3  3 blob[1 B]

nanoparquet as a filter

In Unix, a filter is a program that reads from standard input and writes to standard output, making it a composable building block in shell pipelines. write_parquet() now supports writing to standard output via file = ":stdout:":

write_parquet(mtcars, ":stdout:")

The most common use case is from the command line:

Rscript --quiet -e 'nanoparquet::write_parquet(mtcars, ":stdout:")' > mtcars.parquet

You can build this into a data pipeline. For example, to convert a CSV to Parquet, and then process Parquet with another tool in one shot, without an intermediate .parquet file on the disk, you can do:

cat data.csv |
  Rscript --quiet -e '
    df <- read.csv(file("stdin"))
    nanoparquet::write_parquet(df, ":stdout:")
  ' | another-parquet-tool

Since nanoparquet 0.4.0, read_parquet() can also read from an R connection, so you can pipe Parquet data in as well:

url <- "https://raw.githubusercontent.com/r-lib/nanoparquet/main/inst/extdata/userdata1.parquet"
con <- pipe(paste("curl --silent", url))
df <- read_parquet(con)

Acknowledgements

We thank all contributors to nanoparquet so far, for opening issues, submitting pull requests, and providing feedback: @Aariq , @alvarocombo , @apalacio9502 , @atsyplenkov , @cboettig , @ChandlerLutz , @cmrnp , @D3SL , @damonbayer , @DavideMessinaARS , @eitsupi , @gksmyth , @hadley , @jack-davison , @jeroenjanssens , @lbm364dl , @lschneiderbauer , @mrcaseb , @pmarks , @PMassicotte , @r2evans , @RealTYPICAL , @tanho63 , @thisisnic , @torfason , @TurnaevEvgeny , @Upipa , @vankesteren , @vincentarelbundock , @wlandau , @YipengUva , and @yutannihilation .

torch v0.17.0

The most exciting experimental new feature is support for the cudatoolkit packages. With this, you no longer need a global CUDA toolkit installation in order to use torch on the GPU.

You can now do:

install.packages(
  "cuda12.8", 
  repos = c("https://mlverse.r-universe.dev", "https://cloud.r-project.org")
)
install.packages("torch")

The {cuda12.8} package bundles all the CUDA runtime libraries and torch can find it and use it by default. See more details in the installation docs .

We also highlight the update to LibTorch v2.8.0 led by Troy Hernandez ( #1419 ).

Additionally, this release includes many small bug fixes and small additions to the API. See the full release notes in the changelog .

torchvision v0.9.0

torchvision provides datasets, model architectures, and image transformations for computer vision. This is a big release with new models, datasets, and many improvements — largely driven by community contributors.

New models:

• model_maskrcnn_resnet50_fpn() and model_maskrcnn_resnet50_fpn_v2() for instance segmentation.
• model_convnext_*_detection() for object detection (tiny/small/base).
• model_convnext_*_fcn() and model_convnext_*_upernet() for semantic segmentation (tiny/small/base).

New datasets and features:

• vggface2_dataset() for loading the VGGFace2 dataset.
• New coco_segmentation_dataset(), split from coco_detection_dataset(), reducing memory usage by ~50%.
• Collection dataset catalog with search_collection(), get_collection_catalog(), and list_collection_datasets() for discovering and exploring datasets.
• New visualization utilities draw_segmentation_masks() and vision_make_grid().

See the full release notes in the changelog .

A huge thank you to the community contributors who made this release possible: @cregouby , @ANAMASGARD , @Chandraveersingh1717 , @DerrickUnleashed , and @srishtiii28 .

Other releases

Most of the other packages don’t have significant changes, and the releases add minimal improvements to docs, CI infrastructure and CRAN related updates.

• luz v0.5.2 — Higher-level API for torch with a Keras-like interface for training neural networks.
• hfhub v0.1.2 — Download and cache files from Hugging Face Hub repositories, making it easy to use pretrained models and datasets from R.
• tok v0.2.2 — Fast tokenizers for R, powered by the Hugging Face Tokenizers library written in Rust. Supports BPE, WordPiece, and other tokenization algorithms.
• torchdatasets v0.3.2 — Extra ready-to-use datasets for torch, complementing the built-in datasets in torchvision.
• safetensors v0.2.1 — Read and write the Safetensors file format, a safe and fast format for storing and loading tensors.
• tfevents v0.0.5 — Write event files compatible with TensorBoard from R for experiment tracking and visualization.
• wav v0.2.0 — Read and write WAV files in R.

New maintainer

We’re excited to welcome Tomasz Kalinowski as the new maintainer of torch and the broader mlverse ecosystem.

Both cheatsheets have a dedicated HTML version on the Posit Open Source site, so you can browse and search them without opening a PDF. In this post we’ll walk through what each one covers, starting with the newest.

Create Models with parsnip

The cheatsheet is organized into three main parts: an introduction to parsnip’s basics, a catalog of all models available through the package, and a hands-on operations reference for fitting and inspecting models. The basics section introduces how parsnip provides a single, unified interface for defining and fitting models, regardless of the underlying package powering them.

Model catalog

The largest section of the cheatsheet catalogs all models available through parsnip, grouped by use case:

• Classification only: models for binary and multiclass prediction. It also includes probability-based classification using Bayes’ theorem and models for ordinal responses.
• Regression only: models for predicting continuous numeric outcomes, from standard linear regression to generalized linear models for count data.
• General use: a versatile mix of model types that work for both classification and regression, including decision trees, nearest neighbors, neural networks, and spline-based approaches.
• Discriminant analysis: models that estimate the distribution of predictors separately for each class and use Bayes’ theorem to assign probabilities, available in linear, quadratic, flexible, and regularized variants.
• Ensemble methods: models that combine many individual learners into a stronger prediction, including random forests, gradient boosting, bagged trees, and Bayesian additive regression trees.
• Support Vector Machines: models that find an optimal boundary between classes, or fit a robust regression, using linear, polynomial, or radial kernel functions.
• Feature rules: models that extract simple, human-readable rules from tree ensembles and use them as the basis for prediction.
• Survival models: models for time-to-event data, covering both proportional hazards and fully parametric approaches.

One design choice in particular makes this section much easier to navigate: pills. Each model’s compatible engines and supported modes are shown as small, visually distinct tags, so you can see at a glance which mode a given engine supports, without having to read through the description text. Each mode is encoded in the pill with a number: Classification (1), Regression (2), Censored Regression (3), and Quantile Regression (4). A legend mapping each number to its mode is available at the top of page one.

And true to the R cheatsheet tradition, individual models or groups of related models are paired with small illustrations, thoughtfully designed for visual impact to aid recall. Each one attempts to accurately represent the function or functions it accompanies, making them a genuine navigation aid rather than decoration, especially when you have a vague memory of “that tree-based ensemble that used Bayesian analysis” and need to scan quickly.

Operations

The last section covers the practical workflow of fitting and using a model. Each function is paired with a quick runnable example, and the examples build on each other starting from the two lines of code right below the section title, making it easy to follow the full workflow from model specification to results.

Preprocessing Data with recipes

After a quick Basics section covering the core workflow, the vast majority of the cheatsheet is dedicated to step_*() functions, the building blocks of any recipe, before finishing with role and type management.

Step catalog

The steps are organized into groups based on what they do, each listed with its arguments and a short description:

• Filters: steps for removing variables that are sparse, zero-variance, linearly dependent, highly correlated, or missing too many values
• In-place Transformations: basis functions (splines, polynomials), discretization, and normalization steps
• Imputation: steps for filling in missing values, ranging from simple statistical substitution to model-based approaches
• Encodings: type converters (e.g. factor to string, numeric to factor), value converters, and other factor-handling steps
• Dummy Variables: one-hot and binary encoding, text pattern matching, and conversion helpers
• Multivariate Transformations: signal extraction (PCA, ICA, PLS, and friends) and centroid-based distance measures
• Date & Time: steps for converting date and datetime columns into usable numeric or factor features
• Row operations: sampling, shuffling, slicing, and removing rows with missing values
• Other: interaction terms, renaming, rolling window statistics, geographic distances, and ratios

As with the parsnip cheatsheet, each group of steps is paired with small, thoughtfully designed illustrations to help you visually locate a step family when scanning.

Role & type

The last section focuses on the selection and management of variable roles and types within the recipe. The selection side covers ways to target variables by their role (outcome, predictor, or any custom role) as well as by their type (numeric, factor, logical, and so on), including a handy set of convenience selectors for the most common combinations. The management side shows how to add, update, and remove roles, showing you how to gain fine-grained control over how each variable participates in the recipe.

Need them on the go? Print them!

A lot of care went into ensuring both cheatsheets hold up when printed, particularly in black and white. We know that many folks print cheatsheets to keep at their desk for quick reference, and we wanted to make sure they remain fully usable in that medium. That meant making sure font sizes and weights stay legible on paper, that the illustrations remain perceptible without color, and that contrast levels are strong enough that no text ends up too pale to read or too heavy to parse. Accessibility in print mattered to us just as much as clarity on screen.

# tidymodels installs all of the new versions
require(pak)
pak::pak("tidymodels")

Here are links to the NEWS files for each package:

• dials
• parsnip
• tune
• yardstick
• finetune
• tidymodels

Let’s first talk about the two biggest updates enabled by this group of releases, then we’ll cover some of the other changes for each package.

Ordered Outcomes

parsnip has a new model type, ordinal_reg(), analogous to multinom_reg(), for fitting various generalized linear models with ordered class levels.

The ordered package by Cory Brunson is now on CRAN. This contains the specific engine code for these models, including:

• ordinal_reg(): three engines: "polr", "ordinalNet", and "vglm".
• gen_additive_mod(): "vgam"
• decision_tree(): "rpartScore"
• rand_forest(): "ordinalForest"

These models can be fitted, tuned, and evaluated with tidymodels. For the evaluation, we’ve added a specific performance metric for ordered categories: the ranked probability score (RPS). The function ranked_prob_score() is in the new yardstick release and requires an ordered factor for the outcome.

Quantile Regression

We previously reported that parsnip supports quantile regression models. With the latest set of releases, new boosting and neural network engines are available, and these models can now be tuned and evaluated using a relevant metric. yardstick now includes the weighted interval score ( Bracher et al (2021) ) to evaluate the quality of the quantile predictions.

Here’s a simple one-dimensional example using the Ames data; we’ll predict the sale price as a function of latitude. To start, let’s make a training/test split, generate some resamples, and plot the training data.

library(tidymodels)
# We'll also need the qrnn package for the neural network engine

set.seed(1215)
ames_split <-
  ames |>
  select(Latitude, Sale_Price) |>
  initial_split(strata = Sale_Price)
ames_train <- training(ames_split)
ames_test <- testing(ames_split)
ames_rs <- vfold_cv(ames_train, strata = Sale_Price)

ames_train |>
  ggplot(aes(Latitude, Sale_Price)) +
  geom_point(alpha = 1 / 5) +
  geom_smooth(se = FALSE) +
  labs(x = "Latitude", y = "Sale Price (USD)")

Note that we almost always model these data with a log transformation on the outcome due to its inherent skewness. That helps us avoid making negative predictions, be more robust to overly influential points (i.e., locations with very large sale prices), and stabilize the variance. However, we don’t necessarily have to do that with quantile regression. The objective functions used to estimate parameters do not impose requirements on the normality of the data or heterogeneity of residuals. For this analysis, let’s stick with the original units of the outcome (USD).

There are a few engines for quantile regression, and we’ll use a neural network model. To get started, the quantiles to be predicted need to be specified. We make a model specification with a few additions:

# Pre-defined quantiles of interest
qnt_lvls <- c(0.05, 0.25, 0.5, 0.75, 0.95)

nnet_spec <-
  mlp(hidden_units = tune(), penalty = tune(), epochs = 10) |>
  # Set the quantile levels with the mode:
  set_mode("quantile regression", quantile_levels = qnt_lvls) |>
  # A new engine for quantile regression with neural networks via the
  # qrnn package. We'll add an engine argument to specify the
  # optimization method for training the model:
  set_engine("qrnn", method = "adam")

# Scale the single predictor to help the model initialize its
# parameters.
nnet_rec <- recipe(Sale_Price ~ ., data = ames_train) |>
  step_normalize(all_predictors())

nnet_wflow <- workflow(nnet_rec, nnet_spec)

From there, we can use any of our tuning functions to optimize the number of hidden units and the amount of weight decay. By default, the weighted interval score is used for this particular mode.

We’ll consider 25 tuning parameter candidates to optimize model performance.

set.seed(971)
nnet_res <-
  nnet_wflow |>
  tune_grid(
    resamples = ames_rs,
    grid = 25,
    control = control_grid(save_workflow = TRUE)
  )

We can get the performance metric and visualize which tuning parameter combinations have the smallest weighted interval score:

nnet_mtr <- collect_metrics(nnet_res)

nnet_mtr |>
  ggplot(aes(penalty, hidden_units, size = mean)) +
  geom_point() +
  scale_x_log10() +
  coord_fixed(ratio = 1) +
  labs(x = "Penalty", y = "# Hidden Units", size = "WIS")

select_best(nnet_res, metric = "weighted_interval_score")

# A tibble: 1 × 3
  hidden_units     penalty .config         
         <int>       <dbl> <chr>           
1           10 0.000000215 pre0_mod24_post0

The model appears to prefer a smaller penalty and more hidden units.

It’s hard to conceptualize how well the model functions with just these numbers. To show that the metric does select good models, let’s fit the best, median, and worst models and see how they look on the test set.

set.seed(8281)
best_model <- fit_best(nnet_res)

set.seed(8281)
worst_model <-
  nnet_mtr |>
  slice_max(mean, n = 1) |>
  select(hidden_units, penalty) |>
  finalize_workflow(nnet_wflow, parameters = _) |>
  fit(ames_train)

set.seed(8281)
mid_model <-
  nnet_mtr |>
  # Since we have an odd number of grid points:
  filter(mean == median(mean)) |>
  select(hidden_units, penalty) |>
  finalize_workflow(nnet_wflow, parameters = _) |>
  fit(ames_train)

Now let’s plot the results. We’ll color the predicted quantiles: black indicates the predicted median sale price, orange lines indicate the inner quartiles, and smoky periwinkle lines indicate the 0.05 and 0.95 quantiles (which could serve as 90% prediction intervals).

bind_rows(
  best_model |> augment(ames_test) |> mutate(Model = "Best Results"),
  mid_model |> augment(ames_test) |> mutate(Model = "Meh Results"),
  worst_model |> augment(ames_test) |> mutate(Model = "Worst Results")
) |>
  mutate(
    .pred_quantile = map(.pred_quantile, ~ as_tibble(.x))
  ) |>
  unnest(.pred_quantile) |>
  arrange(Latitude) |>
  ggplot(aes(Latitude)) +
  geom_point(aes(y = Sale_Price), alpha = 1 / 30, cex = 3 / 4) +
  geom_path(
    aes(
      y = .pred_quantile,
      group = .quantile_levels,
      col = factor(.quantile_levels)
    ),
    show.legend = FALSE,
    linewidth = 1
  ) +
  scale_color_manual(
    values = c("#8785B2FF", "#D95F30FF", "black", "#D95F30FF", "#8785B2FF")
  ) +
  facet_wrap(~Model)

These plots show that configurations with very large score values have poor fits (linear in this case). The “meh” model is nonlinear but not responsive enough to the datas’ ups and downs. The best model, with more hidden units and a low penalty, appears to be flexible enough to model the data well.

We’ll have more metrics in yardstick that can use quantile predictions in the future. For example, we can extend the ones that we have, such as rmse() or rsq(), to use a predicted value from the center of the predictive distribution, such as the 0.5 quantile.

Now we’ll describe various other improvements in the recently released versions.

dials

The latest dials release contains several new parameters for new-ish models in parsnip: For the ordinal_reg() models, dials now contains ordinal_link() and odds_link(). For the tab_pfn(), dials contains num_estimators(), softmax_temperature(), balance_probabilities(), average_before_softmax(), and training_set_limit().

The other user-facing changes were related to input checking and related error messages. The most prominent example is that parameters() and the grid_*() functions now give more information in the error message when non-parameter objects are passed in: which inputs aren’t a parameter object and what they are instead.

@corybrunson , @daltonkw , @hfrick , @jeroenjanssens , @topepo , and @vmikk contributed to the package since the last release.

yardstick

Beyond the two new metrics ranked_prob_score() and weighted_interval_score() described above, this release adds a further 8 metrics.

Three new regression metrics:

• mse() — mean squared error (the squared counterpart to the existing rmse()).
• rmse_relative() — root mean squared error normalized by the observed value range.
• gini_coef() — normalized Gini coefficient.

Three new classification metrics:

• fall_out() — false positive rate (1 − specificity).
• miss_rate() — false negative rate (1 − sensitivity).
• markedness() — predictive power of a classifier, computed as PPV + NPV − 1.

Two new probability-based classification metrics:

• roc_dist() — Euclidean distance from the perfect-classifier corner of ROC space.
• sedi() — Symmetric Extremal Dependence Index.

In addition to these new metrics, we have also updated the documention of all metrics. Now each metric shows the formula used to calculate it, as well as the valid values it can produce.

We also have pages that list all metrics of the same type. These can be found with ?class-metrics , ?numeric-metrics or linked within each metric documentation.

We are thankful to the developers who contributed to this version: @abichat , @astamm , @corybrunson , @DarioS , @EmilHvitfeldt , @FvD , @hfrick , @JavOrraca , @jeroenjanssens , @jkylearmstrong-temple , @mle2718 , @nathant181 , @SimonDedman , @topepo , and @tripartio

parsnip

Version 1.5.0 of parsnip had a variety of changes. Besides the additions for the two new model types shown above:

• We enabled case weight usage for the "nnet" engines of mlp() and bag_mlp() as well as for the "dbarts" engine of bart().

Many of the other changes are most likely to be noticed by developers:

• The interface for declaring tunable parameters has been simplified and is the same for main arguments as well as engine parameters. Also, these values can now be set inside extension packages.

• We now export the generics for predict_quantile(), predict_class(), predict_classprob(), and predict_hazard().

• format_predictions() is a new unified function for formatting prediction outputs, consolidating the logic from the individual format_*() functions. The individual functions format_num(), format_class(), format_classprobs(), format_time(), format_survival(), format_linear_pred(), and format_hazard() are now deprecated.

Thanks to those who contributed to parsnip since the last release: @CeresBarros , @corybrunson , @EmilHvitfeldt , @hfrick , @iamYannC , @jack-davison , @jameslamb , @martinju , and @topepo .

tune

The core functionality of tune is to do all the model fitting (including pre- and postprocessing) and performance evaluation across various resamples and tuning parameter combinations. For grid search, we could take the full parameter grid, splice one parameter combination into the workflow at a time, and run with it. That can be pretty inefficient though. So what actually happens in tune are a few optimizations in how we do all that fitting and evaluating: For preprocessing, we do it once for a resample (per preprocessing parameter combination) and then evaluate all model candidates on it. This lets us avoid unnecessarily repeating the same preprocessing multiple times. For model fitting, we make use of what Max calls “the submodel trick” : For certain models, like a boosted tree, you can use a submodel to make predictions without having to refit the model. A boosted tree ensemble fitted with 20 trees can be used to make predictions for any number of trees up to the 20 used for fitting. That allows us to evaluate different tuning parameter candidates for, here, the number of trees, without having to refit the model. When we added postprocessing, we temporarily disabled this (to ensure we got the integration right) - now we’ve brought it back. We make use of this speedup for both the main model as well as the calibration model.

Another big update is that the Gaussian process model package was changed from GPfit to GauPro because the former is no longer actively maintained. There are some differences:

• Fit diagnostics are computed and reported. If the fit quality is poor, an “uncertainty sample” that is furthest away from the existing data is used as the new candidate.

• The GP no longer uses binary indicators for qualitative predictors. Instead, a “categorical kernel” is used for those parameter columns. Fewer starting values are required with this change.

• For numeric predictors, the Matern 3/2 kernel is always used.

Some other changes of note:

• When calculating resampling estimates, we can now use a weighted mean based on the number of rows in the assessment set thanks to Tyler Burch. You can opt-in to this using the new add_resample_weights() function. See ?calculate_resample_weights

• The warning threshold when check the size of a workflow is now a parameter to the control functions and has a new default of 100MB.

Some bug fixes:

• Models with submodel parameters would train all calibration models on predictions from a single submodel value instead of the correct value for each submodel. We sorted this out.

• We fixed a bug for cases where we tune a grid without a model parameter but with a postprocessing parameter.

• Another bug was fixed for augment() when using last_fit() objects

Thanks to the following contributors: @edgararuiz , @EmilHvitfeldt , @hfrick , @jeroenjanssens , @jjcurtin , @mikewolfe , @mthulin , @ncalliencsu , @rvalieris , @StevenWallaert , @tjburch , and @topepo

finetune

This release was mostly focused on internal changes to support the new version of tune.

tidymodels

A basic release that updates the version numbers to require the latest releases of the core packages.

The AI newsletter has moved! From this edition forward, we’ll be publishing here, on the Open Source website. You can read past editions on the main Posit blog .

External news

AI and cybersecurity

Anthropic recently announced Project Glasswing , an effort to bolster critical software against the threat of AI systems that are increasingly capable of finding and exploiting security vulnerabilities. Through the project, a number of organizations that maintain critical software will be allotted credits to Claude Mythos, Anthropic’s yet-to-be-publicly-released model that reportedly found high-severity security issues in “every major operating system and web browser” . The goal is to give these organizations a head start in finding and patching these vulnerabilities before bad actors have access to the same technology.

A step change for local coding agents

One of the most common questions we receive when releasing LLM-enabled software is “Can I use this with a local model?” Up until this point, we’ve largely discouraged using our products with LLMs small enough that you could run them on a laptop. Local models are not there (yet) showed that, at the time, models small enough to run on a laptop failed a simple refactoring task every time, while the frontier, cloud-hosted models were largely successful.

However, the last couple weeks have brought two important releases for the local model space— Gemma 4 and Qwen 3.6 . Both of these new models successfully completed the refactoring task in 9 of 10 attempts .

Posit news

Posit recently announced ggsql , an implementation of the grammar of graphics for SQL. One of the motivations for ggsql is safer AI agent tooling for visualization. With ggsql, rather than giving a visualization agent a full R or Python runtime (where the agent could execute arbitrary code, write files, or make network calls), you can provide it with a SQL runtime instead. SQL limits the agent to database operations, and you can restrict even those by connecting read-only, reducing the risk that the agent will do something undesirable.

Terms

A dense model is one where every parameter is used to process every token. For example, if a model has 14 billion parameters, in a dense model, all 14 billion are involved in generating each token.

Dense models are in contrast to sparse models, a subset of which use an architecture called Mixture of Experts (MoE). In MoE models, only a fraction of the model’s parameters are involved at any given time. The model contains many parallel sub-networks called “experts,” and a small router decides which ones to activate for each token. This means a model only needs to pay the computational cost of the small active subset per token.

This is the architecture of the most recent local models, and partially explains the jump in ability we saw. Gemma 4 26B-A4B, for example, has 26 billion total parameters but only activates 4 billion per token (the “A” is for “active parameters”). The models we tested from the previous generation were dense, requiring you to choose between a model small enough to run on a laptop or one that was good enough to be useful. The MoE architecture helps break that tradeoff.

Learn more

• Anthropic released Opus 4.7 , an incremental improvement over Opus 4.6. OpenAI released GPT-5.4-Cyber , a fine-tune of GPT 5.4 for cyberdefense.
• A new paper shows that vision language models excel at many image benchmarks even when the actual images aren’t provided, because the answers are implicit in the questions.
• Posit’s Jeremy Allen has been keeping a log of the ways AI is transforming work .

Until now, parallel R has meant serializing your data to every worker and duplicating it in each worker’s RAM. Eight workers × 1 GB is 8 GB, plus the serialization, transfer, and deserialization cost to get it there. R processes don’t share memory — each has its own heap, so data crosses between them through a serialization pipe.

mori changes that. It places an R object once into OS-level shared memory and lets every process on the machine read the same physical pages directly — with no data copying between processes.

install.packages("mori")

mori is built on R’s ALTREP (Alternative Representation) framework, which lets a package expose a custom vector backend that reads its data from somewhere other than ordinary R memory — a memory map, a database, a compressed store, or in this case, OS shared memory.

How it looks

The entry point is share(). You pass it an R object, you get back a shared version that you can use in the same way as the original:

library(mori)

set.seed(42)
x <- share(rnorm(1e6))
mean(x)

[1] 0.0005737398

share() works on atomic vector types, lists, and data frames — it writes them directly into shared memory with attributes preserved. In practice that also covers tibbles, data.tables, factors, dates, and matrices, since they’re built on those types. Environments, functions, S4 objects, and external pointers are returned unchanged, since their state can’t be meaningfully exposed as raw bytes in shared memory.

The returned object is an ALTREP view into shared memory — it costs no additional RAM beyond the original region. It also serializes compactly: instead of sending the full 8 MB payload, mori’s ALTREP hooks serialize shared objects as their shared-memory name, just over 100 bytes on the wire.

x |> serialize(NULL) |> length()

[1] 125

That compact serialization is what makes the rest of the picture work.

Parallel workers, one copy

mori pairs naturally with mirai . When you send a shared object to a local daemon, only the shared-memory name crosses the wire; the daemon maps the same physical pages and sees the full data with no deserialization cost. The same is true for any other parallel backend that uses R serialization.

Here’s the motivating case — a bootstrap across eight workers on a 200 MB data frame:

library(mirai)
library(purrr)

daemons(8)

df <- as.data.frame(matrix(rnorm(25e6), ncol = 5))
shared_df <- share(df)

boot <- \(i) colMeans(data[sample(nrow(data), replace = TRUE), ])

# Without mori — each daemon deserializes its own copy
system.time(
  map(1:8, in_parallel(\(i) boot(i), boot = boot, data = df))
)

   user  system elapsed 
  0.445   3.338  10.454 

# With mori — each daemon maps the same shared memory
system.time(
  map(1:8, in_parallel(\(i) boot(i), boot = boot, data = shared_df))
)

   user  system elapsed 
  0.001   0.000   4.959 

daemons(0)

The payload each daemon receives is ~300 bytes instead of 200 MB — roughly 700,000× smaller. There’s a ~2× wall-clock saving on this run, and eight workers now share a single 200 MB copy in memory instead of materializing one each. The function call is the same; the savings come from not copying data that’s already in RAM.

share() itself is paid once upfront — roughly the cost of one serialization to write into shared memory. Daemons don’t pay a deserialize cost on the other end, since they read the same physical memory directly — so even a single send is a net win, and the savings compound with every additional daemon.

The wall-clock gap depends on how much of the run is data transfer versus compute. On cheap per-task work — bootstrap, cross-validation, parameter sweeps — serialization dominates and the wall-clock win is largest; the ratio shrinks as each task starts to involve more substantial compute, although we always get the memory saving.

Lists and data frames travel element-wise too: sending a single column of a shared data frame transmits only that element’s reference, not the whole data frame.

daemons(3)

x <- share(list(a = rnorm(1e6), b = rnorm(1e6), c = rnorm(1e6)))

mirai_map(x, \(v) lobstr::obj_size(v) |> format())[.flat]

      a       b       c 
"840 B" "840 B" "840 B" 

daemons(0)

What this unlocks

Anywhere parallel R workers process the same large dataset, share() removes the per-worker copy:

• A Shiny dashboard where every worker process reads from one shared reference dataset instead of loading its own.
• A tidymodels tune_grid() sweep — or a targets pipeline branching over model variants — where every fit reads the same training data without copying it.
• Bootstrap, Monte Carlo, or permutation work dispatched across mirai or crew , where thousands of iterations all read from one shared dataset.

The pattern is the same in each case: call share() on your dataset once, then pass the result wherever you’d normally pass the data. Parallel dispatches that hit the serialization path transmit only the reference, not the data.

Access and lifetime

A shared data frame lives in a single shared region, but ALTREP columns are only materialized when touched. A task that reads three columns out of one hundred pays for three — character vectors are lazier still, with per-element access. Workers only pay for the data they actually touch.

Shared memory is tied to R’s garbage collector. As long as the shared object (or anything extracted from it) is live in R, the data stays available; when the last reference is dropped, it’s freed automatically with no manual cleanup. The process that called share() needs to hold its reference until a consumer has mapped a view — from that point on, the view itself keeps the shared memory alive.

Mutations go through R’s normal copy-on-write: editing a value inside a shared vector produces a private copy of that one vector, leaving the rest of the shared region untouched.

x <- share(rnorm(1e6))
lobstr::obj_size(x)

960 B

x[1] <- 0  # local mutation materializes a private copy
lobstr::obj_size(x)

8.00 MB

Sharing by name

If you want to access a shared region from another process without going through serialization at all, you can pass its name directly:

x <- share(1:1e6)

nm <- shared_name(x)
nm

[1] "/mori_113fe_4"

# Works from another process; same session here to demonstrate
y <- map_shared(nm)
identical(x[], y[])

[1] TRUE

Handy when the consumer needs to attach by name rather than receive the shared object through R’s serialization path.

How mori fits

R has had partial answers to cross-process data sharing before. bigmemory offers shared big.matrix objects — effective, but limited to numeric matrices. SharedObject on Bioconductor targets a similar goal with its own memory-sharing machinery, oriented around BiocParallel workflows. Arrow ’s memory-mapped Parquet gives zero-copy columnar reads across processes, though the data lives on disk. On Unix, parallel::mclapply gets shared memory via fork copy-on-write (until a worker writes to a page), with the usual fork caveats (unsafe in GUI sessions, with open DB connections, or alongside multithreaded libraries), and with no equivalent on Windows.

mori is usable across any backend that plugs into R’s standard serialization — mirai, future, parallel, foreach, callr — with no special cooperation required. Atomic vectors, lists, and character vectors are all supported, with lazy per-element access preserved in every process. Lifetimes are managed by R’s garbage collector: shared regions are freed automatically when the last reference drops. And mori itself is pure C — POSIX shared memory on Linux and macOS, Win32 file mapping on Windows, nothing beyond the package to install.

Try it

mori is available from CRAN . The package website has a walkthrough of the mirai integration and full reference documentation. mori is designed to slot quietly into existing parallel-R workflows — anywhere a worker currently receives a big dataset, share() it first and you’re done. It complements mirai : mirai handles async evaluation and daemon coordination, mori handles shared access to the data those daemons work on.

The package is in the experimental lifecycle stage while the API settles, so feedback and issue reports on GitHub are very welcome.

Today, we are super excited to announce the alpha-release of ggsql . As the name suggests, ggsql is an implementation of the grammar of graphics based on SQL syntax, bringing rich, structured visualization support to SQL. It is ready for use in Quarto, Jupyter notebooks, Positron and VS Code among others.

In this post we will go over some of the motivations that lead us to develop this tool, as well as give you ample examples of its use; so you can hopefully get as excited about it as we are.

Meet ggsql

Before we discuss the why, let’s see what ggsql is all about with some examples.

The first plot

To get our feet wet, lets start with the hello-world of visualizations: A scatterplot, using the built-in penguins dataset:

VISUALIZE bill_len AS x, bill_dep AS y FROM ggsql:penguins
DRAW point

That wasn’t too bad. Sure, it has the verbosity of SQL, but that also means that you can speak your plot code out loud and understand what it does. We can break down what is going on here line-by-line:

1. We initiate the visual query with VISUALIZE and provide a mapping from the built-in penguins dataset, relating x to the data in the bill_len column, and y in the bill_dep column.
2. We draw a point layer that, by default, uses the mapping we defined at the top.

With this in place, we can begin to add to the visualization:

VISUALIZE bill_len AS x, bill_dep AS y, species AS color FROM ggsql:penguins
DRAW point

We see that a single addition to the mappings adds colored categories to the plot. This gradual evolution of plot code is one of the biggest strengths of the grammar of graphics. There are no predefined plot types, only modular parts that can be combined, added, and removed. To further emphasize this, let’s add a smooth regression line to the plot:

VISUALIZE bill_len AS x, bill_dep AS y, species AS color FROM ggsql:penguins
DRAW point
DRAW smooth

We add a new layer on top of the point layer. This layer also borrows the same mapping as the point layer. Since we color by species, the smooth line is split into one for each species.

We can continue doing this, adding more mappings, adding or swapping layers, controlling how scales are applied etc until we arrive at the plot we need, however simple or complicated it may be. In the above example we may well end up deciding we are more interested in looking at the distribution of species across the three islands the data was collected from:

VISUALIZE island AS x, species AS color FROM ggsql:penguins
DRAW bar

While a completely different plot, you can see how much of the code from the previous plot carries over.

A complete example

With our first couple of plots under the belt, let’s move on to a complete example. It will contains parts we have not seen before, but don’t worry, we will go through it below, even the parts we’ve already seen before. The example is an adaptation of a visualization created by Jack Davison for TidyTuesday.

WITH astronauts AS (
  SELECT * FROM 'astronauts.parquet'
  QUALIFY ROW_NUMBER() OVER (
    PARTITION BY name
    ORDER BY mission_number DESC
  ) = 1
)
SELECT                                          
  *, 
  year_of_selection - year_of_birth AS age,
  'Age at selection' AS category
FROM astronauts
UNION ALL
SELECT 
  *, 
  year_of_mission - year_of_birth AS age,
  'Age at mission' AS category
FROM astronauts

VISUALIZE age AS x, category AS fill
DRAW histogram
  SETTING binwidth => 1, position => 'identity'
PLACE rule
  SETTING x => (34, 44), linetype => 'dotted'
PLACE text
  SETTING
    x => (34, 44, 60),
    y => (66, 49, 20),
    label => (
      'Mean age at selection = 34',
      'Mean age at mission = 44',
      'John Glenn was 77\non his last mission -\nthe oldest person to\ntravel in space!'
    ),
    hjust => 'left',
    vjust => 'top',
    offset => (10, 0)
SCALE fill TO accent
LABEL
  title => 'How old are astronauts on their most recent mission?',
  subtitle => 'Age of astronauts when they were selected and when they were sent on their mission',
  x => 'Age of astronaut (years)',
  fill => null

That was a lot of code, but on the flip-side we have now covered a lot of the most important aspects of the syntax with one example.

At the topmost level there are two parts to this query: The SQL query, and the visualization query. The SQL query is anything from the beginning to the VISUALIZE clause. It is your standard SQL, and it accepts anything your backend accepts (in this blog post we use a DuckDB backend). The result of the query is funnelled directly into the visualization rather than being returned as a table like you’d normally expect.

Since the point of this post is not to teach you SQL we won’t spend much more time discussing the SQL query part. The main take away is that everything before the VISUALIZE clause is pure SQL, any resulting table is automatically used by your visualization, and any table or CTE created there is available for referencing in the visualization query.

As we saw in the first examples, the SQL query part is optional. If your data is already in the right shape for plotting you can skip it and instead name the source directly in the VISUALIZE clause:

VISUALIZE year_of_selection AS x, year_of_mission AS y FROM 'astronauts.parquet'

Now, let’s look at the visual query — everything from VISUALIZE and onwards. VISUALIZE marks the end of the SQL query and the beginning of the visualization query (or VISUALISE for those who prefer UK spelling). It can stand on its own or, as we do here, have one or more mappings which will become defaults for every subsequent layer. Mappings are purely for relating data to abstract visual properties. A mapping is like a SELECT where you alias columns to a visual properties (called aesthetics in the grammar of graphics). In the visualization above we say that the age column holds the values used for x (position along the x axis) and the category column holds the values used for fill (the fill color of the entity). We do not say anything about how to draw it yet.

Following the VISUALIZE query we have a DRAW clause. DRAW is how we add layers to our visualization. There is a large selection of different layer types in ggsql. Some are straightforward: e.g. point for drawing a scatterplot. Some are more involved: histogram (which we use here) requires calculation of derived statistics like binned count. A visualization can have any number of layers and layers will be rendered in the sequence they are defined. DRAW has a sibling clause called PLACE. It is used for annotation and works like DRAW except it doesn’t get data from a table but rather as provided literal values. It follows that our visualization above contains three layers: A histogram layer showing data from our table, a rule annotation layer showing precomputed mean values for each category, and a text annotation layer adding context to the visualization. It is worth mentioning that a layer does not correspond to a single graphical entity. Like with the text layer above, each layer can render multiple separate entities of its type so there is no need to have e.g. 3 line layers to render line plots for 3 different categories.

After the DRAW and PLACE clauses we have a SCALE clause. This clause controls how data values are translated into values that are meaningful for the aesthetic. In our case, the category column holds the strings “Age at mission” and “Age at selection” which doesn’t in itself translate to a color value. The clause SCALE fill TO accent tells ggsql to use the “accent” color palette when converting the values mapped to fill to actual colors. Scales can be used for much more, like applying transformations to continuous data, defining break points, and setting specific scale types (like ordinal or binned).

The last clause in our visual query is LABEL which allows us to add or modify various text labels like title, subtitle, and axis and legend titles.

Stepping back

That was a mouthful. But there are two very silvery linings to it all:

1. You now know the most important aspects of the syntax (there are more, of course, but you can grow into that)
2. Many visualization queries will be much simpler than the one above

We have already seen examples of shorter visual queries above but let’s continue with a boxplot of astronaut birth year split by sex:

VISUALIZE sex AS x, year_of_birth AS y FROM 'astronauts.parquet'
DRAW boxplot

That’s much shorter than the last plot code but still, if you are coming from a different plotting system you may even think this is overly verbose (e.g. compared to something like boxplot(astronauts.sex, astronauts.year_of_birth)). Yes, it is longer, but it is also more structured, composable, and self-descriptive. These features (which are a direct result of its grammar of graphics lineage) means that both you and your future LLM coding buddy will have an easier time internalizing the workings of all types of plots that can be made. The 18 years of dominance of ggplot2 (which shares these features) in the R ecosystem is a testament to this.

As an example, let’s change the above plot to instead show the same relationship as a jittered scatterplot.

VISUALIZE sex AS x, year_of_birth AS y FROM 'astronauts.parquet'
DRAW point
  SETTING position => 'jitter'

Or perhaps the jitter follows the distribution of the data so it doubles as a violin plot:

VISUALIZE sex AS x, year_of_birth AS y FROM 'astronauts.parquet'
DRAW point
  SETTING position => 'jitter', distribution => 'density'

As you can see the syntax and composable nature makes visualization iteration very ergonomic, something that is extremely valuable in both explorative analyses and visualization design.

Why ggsql?

Writing a new visualization library from scratch is a big task and you might wonder why we’re doing it again. Some of the reasons are:

• We want to engage with and help data analysts and data scientists that predominantly work in SQL
• SQL and the grammar of graphics fit together extremely well
• We want to create an extremely powerful, code-based, visualization tool that doesn’t require an entire programming language (like R or python)
• LLMs speak SQL very well and also presents a new interface to data visualization creation
• We have learned so much from 18 years of ggplot2 development that we’re excited to apply to a blank canvas

Let’s discuss each in turn.

Hello, SQL user!

While first R and then Python captured all the attention of the data science revolution, SQL chugged along as the reliable and powerful workhorse beneath it all. There are many people who work with data that do so only or predominantly in SQL. The choice they have for visualizing their data are often suboptimal in our view:

• Export the data and use R or Python which may not be within their comfort zone
• Use a GUI-based BI tool with poor support for reproducibility
• Rely on one of the few tools that exist for creating visualizations directly within the query that we feel are not powerful or ergonomic enough

Our goal when designing ggsql was that the syntax should immediately make sense to SQL users, tapping into their expectation of composable, declarative clauses.

Apart from offering a better way to visualize their data, ggsql is also a way to invite SQL users into our rich ecosystem of code based report generation and sharing build on top of Quarto .

Declarative wrangling, declarative visualization

If you are reading this with no prior knowledge of SQL, here’s a very brief recap: SQL is a domain specific language for manipulating relational data stored in one or more tables. The syntax is based on the concept of relational algebra which is a structured way to think about data manipulation operations. The semantics defines a set of modular operations that are declarative rather than functional, allowing the user to compose very powerful and custom manipulations using a well-defined set of operations.

If you are reading this with no prior knowledge of the grammar of graphics, here’s a very brief recap: The grammar of graphics is a theoretical deconstruction of the concepts of data visualization into its modular parts. While purely theoretical, tools such as ggplot2 have implemented the idea in practice. The semantics defines a set of modular operations that are declarative rather than functional, allowing the user to compose very powerful and custom visualizations using a well-defined set of operations.

From the above, slightly hyperbolic, overview it is clear that both SQL and the grammar of graphics have a lot of commonality in their approach to their respective domains. Together they can offer a very powerful and natural solution to the full pipeline from raw data to final visualization.

No runtime, no problem

Why does it matter that ggplot2 and plotnine requires R and Python installed respectively? There are clear benefits to a single, focused executable to handle data visualization:

• Embedding a small executable in other tools is much easier than bundling R/Python (or requiring them to be installed)
• A smaller scope makes it easier to sandbox and prevent malicious code execution (either deliberately or in error)

Both of the above points make ggsql a much more compelling option for integrating into tools such as AI agents assisting you in data analysis, or code based reporting tools that may execute code in different environments.

You may think we have had to swallow some bitter pills by moving away from an interpreted language, but it has also given us a lot. Most importantly, the rigid structure means that we can execute the whole data pipeline as a single SQL query per layer on the backend. This means that if you want to create a bar plot of 10 billion transactions you only ever fetch the count values for each bar from your data warehouse, not the 10 billion rows of data. The same is true for more complicated layer types such as boxplots and density plots. This is in stark contrast to most visualization tools which must first materialize the complete data, then perform the necessary computations on it, then plot it.

SELECT pod_door FROM bay WHERE closed

LLMs have proven very effective at translating natural language into SQL, and we’re bullish that they can be just as effective with ggsql. We’ve already seen evidence of this in querychat , where you can now visually explore data using natural language via ggsql. And, since ggsql is a much safer and lighter runtime than R or Python, you can much more confidently ship coding agents into a production environment.

We are now wise beyond our years

18 years of ggplot2 development and maintenance also means 18 years of thinking about data visualization syntax, use, and design. While not trying to be boastful we do believe that gives us some expert knowledge on the subject matter. However, not all of this knowledge can be poured back into ggplot2. There are decisions and expectations established many years ago that we have to honor, or at least only challenge very gradually (which we do on occasions).

ggsql is a blank slate. Not only in the sense that we are building it from the ground up, but also in that it is built for an environment with no established expectations for a visualization tool. I cannot stress how liberating and invigorating this has felt, and I am positive that this shines through in how ggsql feels to the user.

The future

We are nearing the end of a rather long announcement — thanks for sticking with us. In the very first line we called this an alpha-release which implies that we are not done yet. To get you as excited about the future as you hopefully are about the present state of ggsql, here is a non-exhaustive list of things we want to add.

• New high-performance writer, written from the ground up in Rust
• Theming infrastructure
• Interactivity
• End-to-end deployment flow from Posit Workbench/Positron to Connect
• Fully fledged ggsql language server and code formatter
• Support for spatial data

What does this mean for ggplot2 development

If you are a current ggplot2 user you may have read this with a mix of fear and excitement (or maybe just one of them). Does this mean that we are leaving ggplot2 behind at Posit to focus on our new shiny toy? Not at all! ggplot2 is very mature and stable at this point but we will continue to support and build it out. We also hope that ggsql can pay back all the experience from ggplot2 that went into its development by informing new features in ggplot2.

Want more?

If you can’t wait to learn more about ggsql and begin to use it you can head to the Getting started section of the ggsql website for installation instructions, and a tutorial, or head straight to the documentation to discover everything ggsql is capable of. We can’t wait for you to try it out and hear about your experiences with it.

While building documentation sites for projects like Great Tables and Pointblank , we learned just how much effort goes into making a site that looks distinctive and matches the character of each project: custom themes, tailored layouts, interactive features, and countless small design decisions. That experience taught me what a great documentation site needs, and I wanted to distill all of those learnings into a tool that gives every Python package a polished site from the start. That led me to build Great Docs , a documentation generator that produces beautiful sites out of the box (but with simple options to customize the look and make it yours).

Great Docs is now part of the Posit open-source ecosystem, available on PyPI , and at v0.7 with seven releases since its initial soft launch.

What Is Great Docs?

Great Docs is a documentation site generator for Python packages. You point it at your project and it produces a documentation site with API reference, CLI reference, user guides, changelog, and landing page. It auto-discovers your package’s public API, generates Quarto -based pages, and renders the result into a static site.

The entire workflow involves just a few commands:

great-docs init       # one-time: auto-detect your package, write config
great-docs build      # build (or rebuild) the site
great-docs preview    # open it in your browser

There is no boilerplate to write, no templates to configure, and no theme to choose as the defaults produce a good-looking site on their own. If you want to go further, the great-docs.yml configuration file offers extensive customization: navbar gradients, content styles, announcement banners, author metadata, custom sections, and much more.

Why Another Documentation Generator?

There is no shortage of documentation tooling in the Python ecosystem, but after years of building and maintaining documentation sites for my own packages, a few pain points kept surfacing:

Discovery is manual. Most tools require you to explicitly list every class, function, and method you want documented, which becomes tedious and error-prone as your package grows.

The output looks dated. Many popular generators produce sites that feel like they belong to an earlier era of the web. Mobile responsiveness, dark mode, and modern typography are afterthoughts, if they are supported at all.

LLMs cannot easily consume the output. Developers routinely paste documentation into AI assistants, and if your docs are not structured for machine consumption, the answers those assistants produce will be worse.

Deployment is a separate project. Getting from a built site to a live GitHub Pages deployment often involves manually writing CI workflows.

No quality tools. Most generators stop at rendering. If you want to check for broken links, catch spelling and grammar issues, or understand how your API has changed across versions, you are on your own with third-party tools and custom scripts.

Great Docs addresses all of these, and the rest of this post walks through the key features.

Auto-Discovery: Your API, Documented Automatically

When you run great-docs init, the tool inspects your package and discovers its public API automatically. It finds classes, functions, dataclasses, protocols, enumerations, exceptions, type aliases, and more, using a combination of runtime introspection and static analysis via griffe . It detects your docstring style (NumPy, Google, or Sphinx) and writes a great-docs.yml configuration file that captures the full structure of your API.

The result is 13 distinct object-type categories: classes with many methods get their own dedicated sections, overloaded signatures are displayed correctly, and long signatures are formatted across multiple lines for readability (all without you having to enumerate a single export).

If you later add new functions or classes to your package, just run great-docs init again (or great-docs build with the --scan option) and the configuration updates to reflect your current API.

Powered by Quarto and quartodoc

Great Docs is an evolution of quartodoc , which pioneered the idea of generating Quarto-based API reference pages for Python packages. All of the excellent work quartodoc does for API documentation (docstring rendering, cross-references between symbols, etc.) carries forward into Great Docs. quartodoc will continue to be maintained as a standalone tool for projects that need focused API reference generation, while Great Docs builds on that foundation to add site-wide features like styling, interactive widgets, CLI reference, quality tools, and the LLM-friendly output described below.

Under the hood, Great Docs generates Quarto .qmd files and renders them into a static site. Quarto is a scientific and technical publishing system that supports executable code cells, rich cross-references, callout blocks, tabsets, and many other features useful for documentation, and by building on it Great Docs inherits all of these capabilities. User guides can include live code examples with rendered output, API reference pages get syntax highlighting via Pygments, and the entire site benefits from Quarto’s rendering pipeline.

Styling and Interactive Features

Great Docs ships with a set of styles and interactive features that cover most of what you would want from a documentation site:

• Dark mode toggle with system-preference detection and persistence across sessions
• Animated navbar gradients with eight preset themes (sky, peach, prism, lilac, mint, ocean, sunset, forest)
• Subtle content gradients that add visual warmth without distraction
• GitHub widget showing live star and fork counts
• Sidebar search for quickly filtering long API reference lists
• Smart sidebar wrapping that handles long class and method names gracefully
• Responsive design that works well on phones and tablets
• Copy Page widget for one-click copy-as-Markdown on reference pages
• Back-to-top button that appears after scrolling, with smooth animation and dark mode support
• Keyboard navigation with shortcuts for search (s), page browsing ([/]), dark mode (d), and a help overlay (?)
• Social cards with automatic Open Graph and Twitter Card meta tags for rich link previews
• Page tags for categorizing pages via YAML frontmatter, rendered as pill-shaped links above the title with an auto-generated tags index page
• Page status badges marking pages as new, beta, deprecated, or experimental, with color-coded badges below the title and compact icons in the sidebar
• Inline icons via the {{< icon name >}} shortcode, giving access to 1,900+ Lucide icons that scale with text and inherit color
• Navigation icons via Lucide for navbar and sidebar labels
• Internationalization with support for 23 languages via a single site.language config option
• JS Tooltips replacing native browser tooltips with styled, theme-aware popovers
• License feature badges showing permissions, conditions, and limitations as color-coded badge groups

Dark mode and sidebar filtering are expected in any modern documentation site, keyboard shortcuts let you navigate without reaching for the mouse, and the navbar gradients give each project a distinct visual identity without requiring any design work from the package author.

LLM-Friendly by Default

Great Docs automatically generates llms.txt and llms-full.txt files alongside your documentation site: structured, plain-text representations of your entire documentation designed for consumption by large language models.

Every reference page also gets a parallel Markdown version, and a “Copy Page” widget lets users (or AI assistants) grab the content of any page in Markdown format with a single click. A “View as Markdown” option renders the plain-text version directly in the browser.

Great Docs also supports the Agent Skills specification. If your project contains a SKILL.md file, Great Docs automatically discovers it, publishes the skill at /.well-known/skills/, and serves it for agent discovery without any configuration. If you do not have a hand-written skill, Great Docs generates one for you. Either way, coding agents like Claude Code, GitHub Copilot, Cursor, and Codex can find and install your package’s skill with a single command:

npx skills add https://your-org.github.io/your-package/

You can also author a comprehensive, hand-written SKILL.md in your project and Great Docs will automatically discover and publish it. A detailed skill can include core concepts, step-by-step workflows, common error patterns with fixes, reference files for configuration and CLI commands, setup and build scripts, and a description of what the agent can and cannot do autonomously. Great Docs publishes the entire skill directory structure and renders it on a dedicated Skills page in your documentation site, with a visual layout of the skill’s file tree, expandable sections for each companion file (references, scripts, assets), and install instructions for every major agent platform.

The practical effect is that when a developer asks an AI assistant “how do I use the build() method?”, the quality of the answer depends on the quality of the documentation the model has access to. And when a coding agent needs to configure, build, or troubleshoot your package’s docs, a structured skill file gives it enough context to act autonomously. These features mean the documentation works for human readers, LLM chat contexts, and agentic workflows.

CLI Documentation

If your package has a Click -based command-line interface, Great Docs can generate reference pages for every command automatically. It discovers your CLI commands, renders each one with its arguments, options, and help text, and adds them to the site with a dedicated sidebar. A reference switcher dropdown lets users toggle between the API Reference and CLI Reference views.

All you have to do is specify the CLI module in your configuration:

cli:
  enabled: true
  module: my_package.cli
  name: cli

Click is the starting point, but the roadmap includes support for Typer, argparse, and Fire, with a plugin interface for additional frameworks. The plan is also to move beyond raw --help output toward richer, API-reference-style rendering: styled parameter tables with type annotations and defaults, auto-generated usage examples, cross-references between subcommands and parent groups, environment variable documentation, and full search integration so CLI commands are indexed alongside API symbols.

User Guides, Custom Sections, and Blogs

API reference alone is rarely sufficient. Developers need narrative documentation that explains concepts, walks through workflows, and provides context that docstrings cannot.

Great Docs supports a user_guide/ directory where you place .qmd or .md files. You can also add hand-written HTML pages that are auto-discovered and integrated into the site with minimal transformation, which is useful for product landing pages, interactive demos, or any content that does not fit the .qmd workflow. These are automatically discovered and added to the site with their own navigation section. You can use numeric prefixes (e.g., 01-installation.qmd, 02-quickstart.qmd) to control ordering in the directory; Great Docs strips the prefixes from the generated URLs, so readers see clean paths like /user_guide/installation.html. Subdirectories are supported for hierarchical organization.

Beyond user guides, you can define arbitrary custom sections for recipes, tutorials, examples, or any other content grouping. And if you want a blog, Great Docs supports blog-type sections with chronological listings and Quarto’s blog features built in.

sections:
  - title: Recipes
    dir: recipes
    navbar_after: User Guide

Source Code Links

Every documented class, method, and function gets an automatic link back to its source code on GitHub, with precise line-number ranges. This is powered by griffe’s static analysis, so the links point to exactly the right lines, not just the file. The placement is configurable (in the usage section or next to the title), and you can specify the branch or tag to link against.

One-Command Deployment

When your documentation is ready, deploying to GitHub Pages is a single command:

great-docs setup-github-pages

This generates a GitHub Actions workflow file that builds your documentation and deploys it automatically on every push. The workflow is configurable (branch, Python version, docs directory), and once it is in place, your documentation stays up to date with every commit.

Quality Tools Built In

Great Docs includes tooling for keeping your documentation healthy:

• great-docs check audits your configuration against your actual exports, showing what is documented and what is missing
• great-docs check-links finds broken links across your documentation, with configurable timeouts and ignore patterns
• great-docs lint analyzes your public API for missing docstrings, broken cross-references, style mismatches, and unknown directives
• great-docs proofread runs local grammar and spelling checks powered by Harper , with a built-in technical dictionary and multiple output formats

All of these produce machine-readable JSON output, making them suitable for CI integration.

Configuration

The great-docs.yml configuration file is the single source of truth for your documentation site. The defaults are sensible enough that most projects need very little customization, but when you do want to adjust things the options are extensive: display names, author metadata with ORCID identifiers, funding and copyright information, announcement banners, homepage modes, theme settings, sidebar behavior, and more.

Here is what a configuration might look like for a well-established project:

display_name: My Package

announcement:
  content: "Version 2.0 is here! Check out the changelog."
  type: info
  style: lilac
  dismissable: true

navbar_style: sky
content_style: lilac

authors:
  - name: Mara Rosario
    role: Maintainer
    github: mbrosario
    orcid: 0000-0002-8471-3056

sections:
  - title: Tutorials
    dir: tutorials
    navbar_after: User Guide

cli:
  enabled: true
  module: my_package.cli
  name: cli

reference:
  sections:
    - title: Core
      contents: [MyApp, Config]
    - title: Utilities
      contents: [parse, validate, format_output]

The Iterative Workflow

In practice, using Great Docs looks like this:

1. great-docs init: Run once to scaffold your configuration. Review the generated great-docs.yml and make any adjustments (reorder sections, add authors, enable CLI docs, etc.).

2. Edit great-docs.yml: Customize display name, add announcement banners, configure sections, set navbar gradients. This file is committed to your repository.

3. great-docs build: Rebuild whenever you want to see changes. This is fast and idempotent.

4. great-docs preview: Open the built site locally to review.

5. Iterate: Adjust configuration, write user guide pages, add recipes. Rebuild and preview.

6. great-docs setup-github-pages: When you are ready to go live, set up automated deployment. From this point on, every push to your main branch publishes updated documentation.

The great-docs.yml file and your user_guide/ directory are the only things you commit. The entire great-docs/ build directory is ephemeral and gitignored.

What’s Next

Great Docs is under active development and has shipped seven releases (v0.1 through v0.7) since its initial launch. The roadmap is quite ambitious. Near-term priorities include author attribution with GitHub-style avatars, reading time estimates, breadcrumb navigation, and an enhanced CLI reference supporting Typer, argparse, and Fire alongside Click. Further out, the roadmap covers multi-version documentation with a version selector, multi-language API documentation spanning Python, R, Rust, and JavaScript, interactive examples powered by Pyodide or JupyterLite, notebook galleries, instant SPA-like page navigation, and a plugin system for third-party extensions.

Get Started

Great Docs is open source under the MIT license and available on PyPI . To get started:

pip install great-docs
cd your-python-project
great-docs init
great-docs build
great-docs preview

If you have feedback, feature ideas, or run into issues, please open an issue on GitHub .

Starting with Quarto 1.9, quarto install chromium is deprecated. The recommended replacement is Chrome Headless Shell , a lightweight, headless-only browser from Google’s Chrome for Testing infrastructure.

Why the switch?

Puppeteer-bundled Chromium served Quarto well, but it had limitations that kept coming up:

• System dependencies in containers: The Puppeteer Chromium binary requires system libraries that aren’t always present in minimal Docker images or WSL environments. This led to cryptic errors that were hard to debug.
• No arm64 Linux support: Puppeteer didn’t distribute Chromium builds for arm64 Linux, leaving users without an easy install path.
• Large download size: The full Chromium bundle is significantly larger than what Quarto actually needs for headless rendering.

Chrome Headless Shell solves all three. It’s purpose-built for headless automation, has fewer system dependencies, ships arm64 Linux builds, and is smaller to download.

Installing Chrome Headless Shell

If you don’t already have Chrome or Edge installed on your system, install Chrome Headless Shell:

quarto install chrome-headless-shell

Quarto will automatically detect and use any existing Chrome or Edge browser on your system. Chrome Headless Shell is the recommended fallback for environments where a full browser isn’t available — CI servers, Docker containers, and headless VMs.

Migrating from Chromium

If you previously installed Chromium via quarto install chromium, the migration is straightforward:

quarto uninstall chromium
quarto install chrome-headless-shell

Running quarto check install will warn you if a legacy Chromium installation is detected and suggest the migration.

CI and automation

If your CI pipeline uses quarto install chromium --no-prompt, it will continue to work — the command still installs a working headless browser, but now shows a deprecation warning. Updating your scripts to quarto install chrome-headless-shell --no-prompt avoids the warning and uses the new tool directly. In Quarto 1.10, quarto install chromium will transparently redirect to Chrome Headless Shell, so either command will produce the same result.

What’s next

The transition away from Puppeteer Chromium is happening gradually. In Quarto 1.9, quarto install chromium shows a deprecation warning and quarto check install flags any legacy Chromium installation. In the upcoming Quarto 1.10, quarto install chromium will transparently redirect to Chrome Headless Shell, and installing Chrome Headless Shell will auto-remove any legacy Chromium.

If you’re still using the old Chromium install, now is a good time to switch.

Learn more on the Chrome Install documentation page.

The Chromium icon in the listing and social card image for this post is by Jeremiah via icon-icons.com. License: CC BY 4.0

RAG is a simple concept that comes up anytime you want to retrieve content for an LLM to improve or augment the generated output.

LLMs are great at reasoning and generating text, but their knowledge is frozen at training time. They can’t access private documents, recent information, or anything that wasn’t in the training data. When asked about these topics, they either refuse to answer or — worse — hallucinate a confident-sounding response. RAG solves this by giving the model access to relevant information at query time, without needing to retrain it.

In practice, most tools built on top of LLMs already do this. ChatGPT uses web search to include recent news in its answers. Claude Code reads the codebase using tools like grep, list_files, and symbol_search before generating code. RAG is what makes LLMs useful for tasks that require specific, up-to-date, or private knowledge.

Modern LLMs support context windows of 100K tokens or more, which might seem like it makes RAG unnecessary — just paste everything into the prompt. But this doesn’t work well in practice. LLMs suffer from “lost in the middle” (sometimes called “context rot”): they pay less attention to information buried in the middle of a very long prompt, so relevant content gets missed. On top of that, sending your entire knowledge base with every query is expensive, slow, and most real-world document collections are too large to fit in a single context window anyway. Long context and RAG are complementary — a larger window lets you include more retrieved chunks, but retrieval is what gives you precision: the model sees 5 relevant paragraphs instead of 500 irrelevant pages.

raghilda

Building a good retrieval system is the hard part of RAG. raghilda is a Python framework designed to handle it. You give it URLs or file paths, and it takes care of the rest. The defaults are opinionated but transparent — every step is exposed and replaceable. You can swap the chunker, change the embedding provider, or write a custom ingestion function without fighting the framework.

• Document processing. Converting HTML pages, PDFs, and DOCX files into clean text is surprisingly messy. raghilda handles this automatically, converting documents to Markdown. For websites, find_links() crawls and discovers pages so you don’t have to list them by hand.

• Smart chunking. Naive fixed-size chunking can split a code block or paragraph in half. raghilda’s Markdown chunker splits text at semantic boundaries — headings, paragraphs, sentences — and preserves the heading hierarchy so each chunk retains context about where it came from.

• Multiple storage backends. raghilda supports DuckDB (local, zero-config), ChromaDB, and OpenAI Vector Stores. The API is the same across backends, so you can start with a local DuckDB file and move to a hosted solution later without rewriting your code.

• Hybrid retrieval. Pure vector search finds semantically similar content but misses exact keyword matches. raghilda combines semantic search, BM25 keyword matching, and attribute filters — so you can search by meaning, by keywords, and by metadata (e.g. source URL, document type, or any custom attribute) all at once.

How it works

A retrieval system has two phases: ingestion — turning your documents into a searchable store — and retrieval — finding the right chunks given a query. raghilda exposes both phases clearly, with each step exposed as an individual call you can customize or replace.

Let’s walk through a minimal example using a Wikipedia article about Princess Ragnhild of Norway.

First, you create a store with an embedding provider. The store is where your chunks and their vector embeddings will live:

from raghilda.store import DuckDBStore
from raghilda.embedding import EmbeddingOpenAI

store = DuckDBStore.create(
    location="ragnhild.db",
    embed=EmbeddingOpenAI(),
    overwrite=True,
)

Then you read the document and chunk it. read_as_markdown() converts the URL to Markdown, and MarkdownChunker splits it into overlapping chunks at semantic boundaries:

from raghilda.read import read_as_markdown
from raghilda.chunker import MarkdownChunker

doc = read_as_markdown(
    "https://en.wikipedia.org/wiki/Princess_Ragnhild,_Mrs._Lorentzen"
)

# We intentionally use a small chunk size for display purposes.
# In practice, chunk sizes of ~1600 characters are a good
# compromise on size versus retrieval quality.
chunker = MarkdownChunker(chunk_size=200, target_overlap=0.25)
chunked = chunker.chunk(doc)
print(f"{len(chunked.chunks)} chunks")

185 chunks

Finally, you upsert the chunked document into the store and build the search indexes. Embedding is handled by the store itself — since the embedding provider is configured at creation time, all chunks in a store are guaranteed to use consistent embeddings:

store.upsert(chunked)
store.build_index()

During retrieval, you query the store and get back the most relevant chunks. raghilda runs semantic search and BM25 keyword search, then merges the results:

chunks = store.retrieve("Did she move to Brazil?", top_k=2)
for chunk in chunks:
    print(chunk.text)
    print("---")

Lorentzen"), a member of the Lorentzen family of shipping magnates.
In the same year, they moved to Brazil, where her husband was an
industrialist and a main owner of Aracruz Celulose. She lived in
Brazil until her death 59 years later.
---
to Rio de Janeiro, where her husband had substantial business
holdings. Their residence in Brazil was originally temporary, but
they
---

Tip: In practice, you’ll usually work with many documents at once — just loop over your URLs or file paths and call upsert() for each one. See the Getting Started guide for a full walkthrough.

Using with an LLM

A retrieval system on its own just returns text chunks. To get actual answers, you connect it to an LLM. The simplest way to do this is to register a search function as a tool that the LLM can call when it needs information.

Here’s an example using chatlas :

from chatlas import ChatOpenAI
import json

def search_ragnhild(query: str) -> str:
    """Search for information about Princess Ragnhild."""
    chunks = store.retrieve(query, top_k=10)
    return json.dumps(
        [{"text": c.text, "context": c.context} for c in chunks]
    )

chat = ChatOpenAI(
    model="gpt-4.1-mini",
    system_prompt="Answer questions about Princess Ragnhild "
    "using the search tool. Always search before answering.",
)
chat.register_tool(search_ragnhild)
_ = chat.chat("Which year did she move to Brazil?", echo="text")

Princess Ragnhild moved to Brazil in the same year she got married,
1953. Her marriage and the move to Brazil were connected, as her
husband was an industrialist and owner of business holdings in Brazil.
They settled in Rio de Janeiro and lived there until her death in 2012.

Compare this with the same question without the search tool:

chat_no_rag = ChatOpenAI(
    model="gpt-4.1-mini",
    system_prompt="Answer questions about Princess Ragnhild.",
)
_ = chat_no_rag.chat(
    "Which year did she move to Brazil?",
    echo="text",
)

Princess Ragnhild moved to Brazil in 1960.

With the search tool, the LLM retrieves the relevant chunks and grounds its answer in the actual document. Without it, the model has to rely on its training data — and may hallucinate or give a vague response.

Learn more

• Getting Started — full walkthrough building a store from a documentation site
• Examples — complete scripts showing RAG workflows with chatlas, ChromaDB, and more
• GitHub repository — source code, issues, and contributions

Yet preparing replication materials can be daunting, especially for researchers new to data science. It is not uncommon for scholars to struggle to reproduce even their own results, due to issues such as disorganized code and data, software version mismatches, missing random seeds, or differences in operating systems and platforms. While some of these challenges are complex, many reproducibility problems stem from preventable organizational issues. Structuring code and data in a clear, consistent, and tool-friendly manner can significantly reduce these difficulties — and brings additional benefits at virtually no cost, including smoother collaboration and easier integration with AI tools.

In this post, I’ll walk through the project structure I use in my own research. It has evolved over time and reflects what I currently consider best practice for reproducible academic work. Adopting a clear structure from the outset makes it significantly easier to reproduce results, onboard collaborators, and maintain projects over the long term.

The structure covers the full research lifecycle: from data cleaning and analysis to manuscript preparation and presentation. The template is built around R and Quarto (the successor to R Markdown), but the underlying principles translate easily to Python and other publishing workflows, including LaTeX. For convenience, I’ve created a ready-to-use template on GitHub that you can clone and adapt for your own projects.

The project structure looks like this:

I’ll now break down each component in turn, explaining the purpose of the key folders and files, and the reasoning behind the design.

Git and GitHub

One major threat to reproducibility is a file named latest_final_v3_definitive.R. While this naming convention feels natural during exploratory analysis—especially under deadline pressure—it quickly becomes impossible to reconstruct what actually changed, when, and why. Future-you (and your collaborators) will not be grateful.

Version control systems like Git solve this problem by recording a structured history of changes through commits. Instead of creating new files for every iteration, you preserve a single evolving project with a transparent timeline. This makes it easy to revisit earlier versions, understand how results evolved, and collaborate without overwriting each other’s work.

GitHub builds on this by providing a shared platform for hosting repositories, reviewing changes, managing issues, and coordinating collaboration. It also makes it very clear who introduced a particular change—an accountability feature that tends to concentrate the mind when committing code.

In the example project structure, several files and folders are Git-related:

.git/ (invisible folder created by Git)
.gitignore
*.gitattributes
*/
└── .gitignore

The .git/ folder is created automatically when you initialize a repository. It contains the full version history and metadata of the project. You generally do not need to interact with it directly—just avoid modifying or deleting it.

The .gitignore file specifies which files and folders should be tracked or ignored by Git. Since all changes to tracked files are recorded in the project history, a useful rule of thumb is to track only what is necessary to reproduce your results. Files that can be regenerated from code—such as intermediate data, plots, tables, or compiled PDFs—are often better ignored to keep the repository clean and lightweight.

The .gitattributes file defines additional rules for how Git should handle specific file types. Compared with .gitignore, it appears less often in many repositories, but it is especially useful when you need file-type-specific behavior. In particular, large raw data files (e.g., large .csv files) and binary formats (such as .rds, .RData, or images) can significantly increase repository size if tracked directly. In these cases, Git Large File Storage (Git LFS) can be used to store the actual files separately, while keeping lightweight pointer files in the main repository history.

Data directories

I use two separate folders to store raw (data_raw/) and cleaned data (data_processed/). Keeping these distinct makes the workflow more transparent: the original data remains untouched, while processed data can be saved in a format that is fast to load during analysis.

This structure also aligns with common journal expectations that replication code should be able to reproduce results starting from the raw data. By preserving raw inputs and separating preprocessing steps, you make the transformation pipeline explicit rather than implicit.

data_raw/
└── *.csv (tracked through Git LFS)

data_processed/
├── *.rds (not tracked in Git)
└── .gitignore

In the template, the example raw data file (data_raw/Brexit.csv) is tracked using Git LFS. While the file itself is neither large nor binary (unlike formats such as .dta or .rds), LFS is used here for demonstration purposes. In real-world research projects, raw datasets are often substantial in size, and setting up LFS early helps avoid repository bloat later on.

The data_processed/ folder contains the cleaned dataset in .rds format, along with a .gitignore file that excludes all files in this folder (except the .gitignore file itself) from version control. This design encourages regeneration of cleaned data from raw inputs via code, rather than relying on previously saved intermediate objects. It also helps keep the repository lightweight.

The .rds format is used for several reasons. First, it is a native R format that preserves data types and attributes faithfully. Second, it encourages the use of RDS over .RData. Unlike .RData, which loads all stored objects into the global environment under their original names, .rds files require explicit assignment when loaded. This reduces the risk of unintentionally overwriting existing objects and promotes more transparent workflows.

Finally, the .gitignore file inside data_processed/ serves an additional purpose: it ensures that the folder itself is tracked by Git. Since Git does not record empty directories, this placeholder guarantees that the directory exists when the project is cloned. This is important because saving cleaned data to a non-existent directory will otherwise result in an error in R.

For researchers working with confidential or restricted data, it is often helpful to separate sensitive materials from those that can be shared publicly. In some cases, this may mean maintaining a private repository during the research phase and preparing a separate public-facing repository upon publication.

Importantly, simply deleting confidential files before making a repository public is not sufficient. Git preserves the full commit history, meaning sensitive data may still be accessible in earlier revisions. Creating a fresh repository that contains only the materials intended for public release—such as replication code and non-sensitive data—helps prevent unintended data leakage.

This approach also makes it easier to curate a clean, well-documented version of the project specifically designed for replication and reuse.

Code organization and outputs

code/
├── 1_data_cleaning.R
├── 2_descriptive.R
└── 3_main_analysis.R

outputs/
├── *.rds (not tracked in Git)
└── .gitignore

The code/ folder contains all scripts related to data processing and analysis. In the template, these are written in R, but the same structure works equally well for Python scripts (.py), notebooks (.ipynb), or other programming languages.

A key principle is to prefix scripts with numbers and descriptive names that reflect the research workflow: data cleaning, descriptive analysis, and then inferential analysis. If the logical order of steps is unclear, following the sequence in which results appear in the manuscript is often a reliable guide. This makes the analytical pipeline explicit and allows others (and future-you) to reproduce results step-by-step.

Each script should include a header section with basic metadata about its role in the project. This may include the paper title, authors, purpose of the script, input files, and output files. Making inputs and outputs explicit helps clarify dependencies and encourages scripts that transform data rather than rely on objects lingering in the global environment.

Below is an example of an analysis script (code/3_main_analysis.R) I used in the template that follows these principles:

######## INFO ########

# PROJECT
## Paper: YOUR PAPER TITLE
## Authors: YOUR NAME

# R Script
## Purpose: This script performs linear regression analysis.
## Inputs: data_processed/Brexit.rds
## Outputs: outputs/regression_table.rds

# Setup ----

library(tidyverse)
library(here)
library(modelsummary)

i_am("code/3_main_analysis.R") # helps with relative paths

# Read in the cleaned data ----

brexit_data <- read_rds(here("data_processed/Brexit.rds"))

# Main analysis ----

model <- lm(leave ~ turnout + income + noqual, data = brexit_data)

# Output the model summary ----

reg_table <- modelsummary(model, stars = TRUE, output = "latex")
write_rds(reg_table, here("outputs/regression_table.rds"))

Two additional tips are worth noting. First, the # SECTION ---- syntax creates collapsible sections and structured outlines in RStudio and the Positron IDE. This makes longer scripts significantly easier to navigate and encourages more intentional organization of code.

Second, managing working directories is a seemingly basic task that is frequently mishandled—especially in collaborative projects. In RStudio, the recommended approach is to work within an .Rproj file, which defines a project root and ensures that relative paths behave consistently across machines. However, in many academic settings (including political science), this practice is not systematically taught. As a result, it is still common to see replication files from top journals that begin with something like setwd("~/Path/To/Project"), often commented out with the expectation that collaborators will manually adjust it.

This approach is fragile. It assumes a specific directory structure on every machine and introduces hidden dependencies on local file paths. Code that depends on setwd() is difficult to port, share, or automate.

Positron improves this situation by automatically setting the working directory to the folder opened in the IDE, encouraging a project-level workflow by default. However, many users carry over the habit of opening individual scripts rather than entire project directories.

The here package provides a robust solution that avoids reliance on the working directory altogether. By anchoring a script to the project root using here::i_am() and constructing paths with here(), file references become explicit and portable. This ensures that scripts run consistently across machines, IDEs, and collaboration environments—regardless of local directory structures.

The outputs/ folder complements this approach by providing a dedicated location for all results generated by the code. This includes intermediate objects (e.g., fitted models) and final products (e.g., tables and figures). If intermediate artifacts become numerous, they can be stored in a separate objects/ folder. The accompanying .gitignore file ensures that these generated files are not versioned, reinforcing the principle that results should be regenerated from code rather than preserved as static artifacts.

Virtual environments and dependencies (renv)

Many R users do not update their R or package versions regularly. In practice, the version in use is often determined by when the researcher first learned R—or when the machine was purchased, whichever is later. Deprecation warnings are politely ignored as long as the code continues to run.

The problem only becomes visible when code that worked perfectly on your old machine suddenly fails on a new machine—or worse, on a collaborator’s machine. At that point, dependency management stops being an abstract concern and becomes a very practical one.

A common solution to this problem is the use of virtual environments, which capture the exact package versions used in a project—a concept long established in the Python ecosystem. In R, the renv package provides a convenient way to create and manage project-specific libraries. Package versions remain fixed within the project, independent of updates to the global R installation or differences across collaborators’ machines.

The state of the environment is recorded in the renv.lock file, which is committed to Git. This file serves as a snapshot of the project’s dependency graph at a given point in time, including package versions and their sources. As a result, the same software environment can be reproduced on any machine with a simple call to renv::restore().

Three key components related to renv appear in the template structure:

renv/
.Rprofile
renv.lock

The renv/ folder contains the project-specific package library. Most of its contents are not tracked in Git, since the environment can be regenerated from the information stored in renv.lock. Additionally, some packages include compiled C or C++ code that is platform-specific, meaning those installed binaries are not portable across operating systems.

The .Rprofile file includes a line that automatically activates the renv environment when the project is loaded, ensuring that the correct package versions are used without manual setup.

Automatic activation, however, only works when the project is opened as a whole—for example, by opening the .Rproj file or the project folder in Positron. If individual scripts are opened in isolation, the working directory will not be set to the project root at startup, and the project-level .Rprofile will not be executed. This is yet another reason to adopt a project-level workflow rather than treating scripts as standalone files.

Quarto: manuscripts and presentations

The most visible stage of the research lifecycle is the dissemination of results—through manuscripts, presentations, and other public outputs. For quantitatively oriented researchers (which, if you have read this far, likely includes you), producing well-formatted documents that do justice to your carefully constructed tables and figures is essential.

Quarto is an open-source scientific and technical publishing system built on Pandoc. I use it for all of my manuscript writing and presentation slides, and I recommend it for researchers working in R, Python, or Julia. The template presented here is deliberately centered around Quarto, and in what follows I will briefly explain the reasoning behind that choice.

Why Quarto?

What makes Quarto stand out is its ability to run R, Python, or Julia code directly within a document, seamlessly integrating analysis and writing. Tables, figures, and results can therefore be generated and updated automatically as the underlying code changes, ensuring that the manuscript always reflects the current state of the analysis.

Quarto is not a replacement for output formats such as HTML, Microsoft Word, LaTeX, or Typst. Rather, it acts as a unifying layer that can render the same .qmd source file into multiple formats simultaneously. This flexibility allows format decisions to be postponed and adapted to collaborators, institutions, or journal requirements.

Beyond manuscripts, Quarto also supports presentation formats and full websites. Learning a single tool therefore enables the production of academic papers, conference slides, and project or personal websites within a consistent workflow.

While these capabilities may sound familiar to experienced R Markdown users, I would still argue that Quarto is worth trying—even for those already comfortable with R Markdown—for four main reasons:

1. Language-agnostic support: Quarto is designed to work seamlessly with multiple programming languages (R, Python, Julia). A document can be executed using the native engine for each language—for example, a Python-only document runs through a Jupyter kernel without requiring R¹, which is more friendly to non-R users.

2. Native support for extended features: Quarto includes built-in support for cross-referencing, citations, and advanced formatting without requiring additional packages or complex configurations. In contrast, R Markdown often relies on extensions such as bookdown to achieve similar functionality, which introduces additional dependencies. In practice, many students are taught only the basic R Markdown setup and may not be aware of these extensions. Quarto provides these features out of the box.

3. More scannable code cell/chunk options and syntax: R Markdown users may be familiar with setting document-wide execution options inside a setup chunk using knitr::opts_chunk$set(...), and specifying chunk options inline in a comma-separated format. While functional, this approach can become difficult to scan and maintain in larger documents.

   ```{r}
   knitr::opts_chunk$set(echo = FALSE, message = FALSE, warning = FALSE, error = FALSE)
   ```
   
   ```{r barplot, fig.cap='Bar plot for y by x.', fig.height=3, fig.width=5}
   # code for the bar plot
   ```

   In Quarto, document-level execution options are defined declaratively in YAML, while cell-level options use a multi-line, command-style syntax. This makes both levels of configuration easier to scan, and typically easier to review in diffs and modify.

   ```yaml
   execute:
     echo: false
     warning: false
     error: false
     message: false
   ```
   
   ```{r}
   #| label: fig-barplot
   #| fig-cap: 'Bar plot for y by x.'
   #| fig-height: 3
   #| fig-width: 5
   # code for the bar plot
   ```

4. Centralized documentation: While R Markdown benefits from a large community and extensive online resources, its documentation is distributed across multiple sites². Quarto, by contrast, maintains a single, comprehensive documentation portal that covers core usage and advanced features in one place, making it easier to navigate and learn systematically.

Quarto project structure

The remaining components of the template are primarily related to Quarto. Below, I break down the key elements and explain the reasoning behind their organization.

_extensions
.quarto (created by Quarto; not tracked in Git)
extras/
manuscript/
├── manuscript.pdf (not tracked in Git)
└── manuscript.qmd
slides/
├── slides.pdf (not tracked in Git)
└── slides.qmd
_quarto.yml

Configuration file

The _quarto.yml file serves two main purposes.

First, it defines project-level execution behavior. In particular, the following setting ensures that code is executed relative to the project root, regardless of where individual .qmd files are located:

project:
  execute-dir: project

This provides an additional layer of protection for resolving relative paths correctly. Used together with the here package, it helps ensure that file paths behave consistently across different machines and execution contexts.

Second, _quarto.yml centralizes shared configuration options so they do not need to be repeated in each individual .qmd file. This reduces duplication, minimizes the risk of inconsistencies, and improves readability across documents.

In the template, shared options include the bibliography file and citation style:

bibliography: extras/references.bib
csl: extras/apa.csl

Because these settings are defined at the project level, they apply automatically to both the manuscript and presentation slides.

Extras: bibliography and citation styles

As mentioned above, the extras/ folder contains the bibliography file (references.bib) and citation style file (apa.csl). These are referenced in the _quarto.yml configuration file, which means they are automatically available to all .qmd files in the project without needing to specify them individually.

extras/
├── references.bib
└── apa.csl

Ideally, if you have other supplementary materials that are not part of the core code or data but are still relevant to the project (e.g., codebooks), they could also be stored in this folder. However, I have kept it focused on bibliography-related files for simplicity.

Bibliography file

The references.bib file is a standard BibTeX/BibLaTeX bibliography file familiar to LaTeX users. It contains structured reference entries, including fields such as author, title, journal, year, and other publication metadata.

Entries can be exported from reference managers such as Zotero or Mendeley, or generated directly within Quarto using the Visual Editor. Crucially, the bibliography file is kept separate from both the manuscript and the citation style. This separation allows the same reference database to be shared across manuscripts and slides, while making it easy to change formatting styles without modifying the source content.

Citation style file

The apa.csl file is a Citation Style Language (CSL) file that defines how citations and bibliography entries are formatted. It acts as a translation layer between the structured data in references.bib and the rendered output format.

In this template, the CSL file specifies APA style, which is common in the social sciences. However, switching styles is straightforward: replace the apa.csl file with another CSL file (e.g., Chicago, MLA, or a journal-specific style) and update the reference in _quarto.yml. No changes to the manuscript text are required.

To find a CSL file for a specific discipline or journal, you can browse the CSL Style Repository , which contains thousands of maintained styles.

Manuscript

Dissemination of research findings is the ultimate goal of the research process, and the manuscript remains its primary vehicle.

manuscript/
├── manuscript.pdf (not tracked in Git)
└── manuscript.qmd
_extensions/
└── kv9898/
    └── orcid/

The manuscript/ folder contains the Quarto source file (manuscript.qmd) and the compiled PDF output (manuscript.pdf). As with other generated artifacts, the PDF is excluded from version control because it can always be regenerated from the source document.

The example manuscript.qmd provides a minimal template illustrating a typical academic structure, including numbered sections, figures, tables, cross-references, and citations. The template is intentionally simple but can be extended with additional formatting and structural elements as needed.

While Quarto’s built-in PDF format supports core elements such as title, authors, date, and abstract, more specialized academic requirements—such as detailed affiliation formatting, ORCID display, keywords, custom headers, or journal-style front matter—often require additional customization.

To address this, I created a custom extension that builds on the default PDF format and adds features commonly required in academic manuscripts. The extension is included in the template under _extensions/kv9898/orcid/.

Packaging the manuscript format as a Quarto extension ensures that formatting logic is versioned and shared alongside the project, rather than maintained as ad hoc local tweaks. The extension is activated via the YAML front matter of manuscript.qmd:

format:
  orcid-pdf:

The resulting PDF output more closely resembles a conventional academic paper, with structured author information, affiliations, ORCID identifiers, and keywords clearly presented. Because the template is implemented as a Quarto extension, it remains portable and reusable across projects, and can be further modified, particularly by those comfortable with LaTeX, to accommodate journal-specific formatting requirements.

Slides

Researchers often need to present their findings at conferences, seminars, or in teaching settings. Quarto supports multiple presentation formats, including Reveal.js, Beamer, and PowerPoint. In this template, I use Beamer to produce PDF slides, which are widely accepted in academic contexts and easy to share or print.

slides/
├── slides.pdf (not tracked in Git)
└── slides.qmd

The slides/ folder contains the Quarto source file (slides.qmd) and the compiled PDF output (slides.pdf). As with the manuscript, the PDF is treated as a generated artifact and excluded from version control.

The resulting slides are indistinguishable from those produced using a traditional LaTeX Beamer workflow. The key difference is that all content—including code, tables, and figures—can be generated directly from the same analytical pipeline.

In this example, I deliberately reuse the same output objects (e.g., tables and figures) in both the manuscript and the slides. This guarantees consistency across formats and eliminates the risk of discrepancies between what appears in the paper and what is presented publicly.

The example slides also demonstrate practical considerations such as resizing tables and figures to fit slide layouts. They reference the same shared bibliography file used in the manuscript, ensuring consistent citation formatting across outputs. Although only a single reference is included in the example, the template is configured to allow references to span multiple frames, accommodating a realistic bibliography.

README file

README.md

Following both GitHub conventions and academic best practice, every project should include a README.md file. This file serves as the primary entry point for others who wish to understand, reproduce, or build upon the research.

In the template, the README provides:

• A brief project description
• Instructions for setting up the environment
• Steps to reproduce the analysis, manuscript, and slides

Additional placeholders are included for information such as the machine model and operating system used during development. While not always necessary, this metadata can be helpful when troubleshooting platform-specific issues, particularly for projects involving compiled dependencies. The README also notes the approximate time required to run the full analysis and render outputs, which helps set realistic expectations for replication.

Notably, the template does not include a LICENSE file by default. This is intentional. The appropriate license for academic code and data depends on disciplinary norms, institutional policies, journal requirements, and the researcher’s intended level of openness. Common choices include MIT or GPL licenses for code, and Creative Commons licenses for data. In some cases, more restrictive or custom licenses may be appropriate. Researchers should select a license deliberately, ensuring it aligns with their sharing goals and complies with relevant policies.

GitHub as infrastructure — not just hosting

Once a project is structured clearly and pushed to GitHub, it becomes more than a collection of files. It becomes infrastructure.

A well-organized repository makes collaboration dramatically smoother. Issues can serve as lightweight meeting minutes, evolving naturally into task lists. They can be assigned to specific contributors, grouped into milestones, and tracked over time. Pull requests and branching strategies help keep the main branch stable while allowing experimentation and iterative refinement. Code reviews become part of the workflow rather than an afterthought.

These practices, borrowed from software development, translate surprisingly well into academic collaboration. Instead of emailing attachments back and forth, collaborators work against a shared, versioned source of truth.

A clear project structure also makes modern AI tools significantly more useful. When your data, scripts, outputs, and manuscripts are logically organized, AI assistants in VS Code, Positron, or GitHub can reason about your project more effectively. They can trace how tables were generated, suggest improvements to analysis code, help refine writing based on the underlying results, or flag inconsistencies between figures and text. In other words, organization enables context — and context is what makes AI assistance meaningful rather than superficial.

There are also practical benefits. Once your work is version-controlled and backed up remotely, you no longer fear data loss due to a failed hard drive, a stolen laptop, or accidental overwrites. The repository itself becomes a durable record of the project’s evolution.

Perhaps most importantly, a well-structured project reduces the asymmetry of knowledge among collaborators. Instead of each co-author being familiar with only one portion of the workflow, everyone can develop a holistic understanding of how the project fits together — from raw data to final manuscript. This makes feedback more constructive, collaboration more efficient, and the research process more transparent.

Reproducibility, then, is not merely about satisfying journal requirements. It is about building research projects that are resilient, collaborative, and adaptable — projects that scale not only across machines, but across people.

Conclusion

At its core, none of the tools discussed here—Git, renv, Quarto, or GitHub—are revolutionary on their own. What matters is how they are combined into a coherent project structure. Once that structure becomes habitual, reproducibility stops being an afterthought and becomes the default.

Adopting this workflow does not require perfect foresight or advanced technical expertise. It simply requires deciding, from the outset, that clarity, versioning, and regeneration will guide the project. The payoff is substantial: fewer replication headaches, smoother collaboration, better integration with modern tooling, and greater confidence in the durability of your work.

In the long run, a well-structured project is not just easier to reproduce—it is easier to think with.

Welcome to the first edition of our Positron newsletter! Here, we will share highlights from our latest release, tips on how to be more productive with Positron, and useful resources.

We just returned from an in-person onsite in beautiful Monterey, California. During the trip, we got a chance to meet (some of us for the first time), touch grass and sand, and brainstorm ways we can improve to build better products for you.

Let’s get into the updates.

Key Product Updates

The April 2026 release of Positron brings significant improvements across:

• Positron Server for Academic Use via JupyterHub
• AI enhancements : Next Steps in Jupyter Notebooks, Agent Skills, and Azure AI Foundry Support
• Telemetry updates
• R improvements : Addins, Debugging, and more
• Data Explorer Performance Improvement
• Windows ARM in GA
• What’s Coming Next : Inline Outputs, Packages Pane, and Posit Assistant

Here’s a look at the key features that shipped with the April 2026 release.

Positron Server for Academic Use via JupyterHub

What we built: Academic institutions can now offer Positron Server to their students at no cost through JupyterHub ( blog post ). If your institution already runs JupyterHub, you can add Positron as a launcher option alongside JupyterLab, with no additional infrastructure required. Students simply log in and select Positron from the launcher, getting the full Positron experience including rich Python and R support, the extension marketplace, and (optionally) Positron Assistant.

Why this matters: This removes the barrier for students and educators who want to use Positron in a classroom setting. No local installs, no configuration headaches — just a familiar JupyterHub login with Positron ready to go.

Get started: Review the eligibility criteria and send an email to academic-licenses@posit.co to request a free teaching license.

AI Next Steps in the Native Jupyter Notebook Editor

What we built: AI Next Steps uses the Positron Assistant to analyze your current cell output and suggest a logical next step in a “ghost cell” at the bottom of your notebook. If you just loaded a CSV, it might suggest data cleaning steps or a visualization, without you needing to open a chat pane or write a prompt. Suggestions stay aligned with the notebook’s live kernel state, updating as your code and outputs change.

Why this matters: The design came out of interviews with data scientists who kept telling us the same thing: switching to a chat pane mid-analysis breaks their concentration. AI Next Steps sits at the bottom of your notebook and updates as your outputs change. You just run a cell, and if there’s a logical next step, it surfaces, with no prompt required.

Get started: Enable the feature by setting positron.assistant.notebook.ghostCellSuggestions.enabled to true in your settings. When you run a cell, look for the ghost cell suggestion at the bottom of the notebook, accept, reject, or hide it.

Agent Skills in Positron Assistant

What we built: Agent skills — reusable, structured capabilities that extend what agents can do in agent.md files — are now integrated into Positron ( #11753 ). Skills let agents execute multi-step workflows like “profile this dataset and suggest cleaning steps” or “run this test suite and summarize failures,” so you define a task once and reuse it across sessions and projects.

Why this matters: Skills make agents composable building blocks rather than one-off chat interactions. Instead of re-explaining a complex workflow every time, you codify it as a skill that any team member can use.

Get started: Open the chat gear icon and select Skills, or run Chat: Configure Skills from the Command Palette.

Positron Assistant Now Supports Microsoft Foundry as a Provider

What we built: Positron Assistant now supports Microsoft Foundry as a model provider ( #8583 ) with API key-based access via a custom base URL.

Why this matters: If your team runs on Azure and uses LLMs through Foundry, you can now use Positron Assistant with them.

Get Started: In Positron Assistant’s provider settings, set positron.assistant.provider.msFoundry.enable to true to select Microsoft Foundry as a provider. You can authenticate with an API key and your Foundry endpoint URL.

Telemetry Update: Anonymous Session Identifiers

What we changed: Positron now generates an anonymous, random session identifier to help us understand usage patterns like session frequency and retention across releases. This identifier contains no personal information, account data, or workspace content; it’s a cryptographically random UUID that cannot be linked to any other identifiers, including the identifier that VS Code uses for telemetry.

Why we’re doing this: As a free, source available project, we don’t have traditional product analytics. Understanding whether people come back, how often they use Positron, and whether releases improve or regress the experience helps us prioritize the right work to build a better experience for you.

You can opt out by updating your settings outlined here , or you can reset the anonymous identifier with the command Preferences: Reset Anonymous Telemetry ID. If you’ve opted out of product updates, no session identifier is generated or sent.

RStudio Addins Support

What we built: Positron now supports running RStudio addins from R packages. If a package registers an addin (like styler, reprex, clipr, or shinyuieditor), you can run it directly from Positron ( #1313 ).

Why this matters: This was one of our most upvoted issues this release (25 👍). Many R users rely on addins as part of their daily workflow for code formatting, generating reproducible examples, or launching Shiny tools.

Get started: Open the Command Palette (Ctrl-Shift-P (windows), Ctrl-Shift-P (linux), Command-Shift-P (mac)) and search for Run RStudio Addin. You’ll see a quick pick with all available addins from your installed packages.

R Debugger & Workflow Improvements

What we built: The R debugger received a suite of improvements this release. In addition to conditional breakpoints, hit count breakpoints, and log breakpoints ( #12360 ), the debugger now supports error and warning breakpoints ( #11797 ), the ability to pause R at any time ( #11799 ), Watch Pane support ( #1765 ), and synchronization between the Console and Variables pane with the selected call stack frame ( #3078 and #12131 ).

Why this matters: Advanced debugging in R has traditionally meant scattering if (...) browser() calls through your code or setting options(error = recover) by hand. These new features put Positron’s R debugger on par with what you’d expect from any modern language:

• Conditional, hit count, and log breakpoints let you control exactly when breakpoints fire and print diagnostic info, all without touching your source code.
• Error and warning breakpoints drop you into the debugger the moment an error or warning is emitted, so you can inspect the state that caused it.
• Pause R at any time. If R is stuck in a long computation or an infinite loop, you can drop into the debugger mid-execution, look around, and resume by clicking Continue.
• Watch Pane lets you track expressions across debug steps. Prefix an expression with /print to see R’s printed output (hover to get full output) instead of a structured variable.
• Synchronization with the call stack. Click any frame in the Call Stack view and the Console, completions, and Variables pane all switch to that frame’s environment. The Console synchronization is like recover(), but built into the IDE.

Get started: Set a breakpoint in any R file, then right-click it and choose Edit Breakpoint. Select “Expression” to add a condition (e.g., i > 100), “Hit Count” to break after N hits, or “Log Message” to print a message without pausing. For error and warning breakpoints, open the Breakpoints pane and enable them there. To pause R while code is running, use the command Debug: Pause or check the Interrupt breakpoint option in the Breakpoints pane. While debugging, add expressions in the Watch section of the debug sidebar and click on frames in the Call Stack to navigate environments.

Data Explorer: Faster with Multiple DataFrames

What we built: We fixed two long-standing performance issues in the Data Explorer. Background Data Explorer tabs no longer trigger backend recomputation, and the summary panel no longer recalculates summary statistics for large DataFrames on every cell execution ( #4279 and #2795 ).

Why this matters: If you work with multiple DataFrames open, you may have noticed lag as Positron recomputed statistics for tabs you weren’t even looking at. That’s gone now.

Get started: Nothing to configure. When you open multiple DataFrames in the Data Explorer and switch between them, you should notice snappier performance, especially with large datasets.

Windows ARM Is Generally Available

What we built: We started creating experimental builds for Windows ARM several months ago, and our early users have had good experiences with them. This release, we promoted the Windows ARM builds from experimental to stable and they are now available through all standard installation channels ( #12207 ).

Why this matters: ARM-based devices are increasingly common for Windows users, whether you’re a student or a professional. GA support means these users get the same Positron experience, including Quarto with R and Python support, without needing workarounds or experimental builds. Do be aware that the Windows ARM build bundles the non-ARM version of Quarto, which runs under emulation.

Get started: Install Positron on your ARM-based Windows device through standard installation channels .

View all issues in the 2026.04.0 Release milestone .

What’s Coming Next

We are currently building the following features and we’d love your feedback. Please share on GitHub . These early alpha features with some rough edges are available for testing by enabling their respective settings.

Inline Outputs for Quarto and R Markdown Files

This was the second most upvoted issue we have ever, ever had! We just completed an initial run to allow displaying inline outputs within Quarto and R Markdown files ( #5640 ), and it is available for early testing. Note that this experimental version, while it does get the basics into Positron, does not have support for many popular RStudio features. You can opt in to the experimental feature using the positron.quarto.inlineOutput.enabled setting.

Packages Pane for Managing Environments

We are currently building out a new Packages pane that will allow you to install, update, and uninstall packages without leaving your workspace or needing to use the terminal ( #11214 ). We’d love to hear your feedback on this discussion thread .

Events and Resources

Explore Positron’s Video Walkthroughs on YouTube

We hosted a walkthrough of exploring GitHub data in a Jupyter Notebook and converting this into an interactive Shiny app with AI. Catch up on the recording or explore more Positron videos .

Registration for posit::conf(2026) Is Now Open!

Registration is officially open for posit::conf(2026)! Join the global data community in Houston or tune in online from September 14–16. Register today!

How We Chose a Python Type Checker

Ever wondered about the decision making process behind how we chose which Python type checker to bundle in Positron? Check out Austin Dickey’s blog post walking through his research and decision making process.

Community Affirmations

Thank you all for your support, ideas and engagement. We’re building Positron in the open because the best ideas come from the people using it. If there’s a feature you’d love to see, open an issue or upvote an existing one, it genuinely shapes what we work on next.

Have a great April!

Positron Team

Students can launch Positron the same way they would open JupyterLab or a notebook. Just select it from the JupyterHub launcher and start working.

Once launched, Positron provides the full IDE experience, including:

• Rich Python and R support
• Access to the OpenVSX extension marketplace
• Built in data viewer and variables explorer
• Integrated help pane, debugger, version control and other features to help students level up when they’re ready

How it works

Positron Server is designed to integrate directly with existing JupyterHub deployments. It’s compatible with JupyterHub environments running JupyterLab 4 and Python 3.9+.

It’s installed via the jupyter-positron-server Python package , built on Jupyter Server Proxy. If you’ve configured similar services before, setup will feel familiar. This is not a standalone desktop install. Rather, it lets you bring Positron into an existing JupyterHub setup.

Who can use it?

This offering is available to academic institutions using Positron for teaching. Under a free license, institutions can provide access to enrolled students, course participants, or staff involved in the delivery or receipt of educational programming.

Full eligibility details are available in the Positron Education License Rider .

Getting started

Hosting Positron for teaching purposes requires a free license key. To get set up:

1. Review the eligibility criteria in the Positron Education License Rider .
2. Email academic-licenses@posit.co to request a teaching license.
3. Once your license is confirmed, follow the jupyter-positron-server documentation to complete setup in your JupyterHub environment.

Quarto 2 is a full rewrite of the Quarto CLI, written from the ground up in Rust to better support your existing use cases, and enable a number of new, exciting use cases. Most importantly, Quarto 2 will include a built-in collaborative editor, and we plan on adding support for collaborative writing in Posit’s commercial products such as Posit Cloud, Connect, and Workbench. With that said, the design of those integrations is still taking shape.

It is also very early in the project. If you interact with the Quarto project solely as a user of the tool, nothing in your workflow will change, and you should proceed as if you didn’t know about our plans for Quarto 2. We don’t expect to have a public release of Quarto 2 for at least 6 months. In addition, we will continue to develop and maintain parallel versions until Quarto 2 is a suitable replacement for users of Quarto 1.

Just like Quarto 1, Quarto 2 is open source and MIT licensed. The GitHub repository for Quarto 2 is currently quarto-dev/q2 .

Why Quarto 2?

There are some fundamental pain points in Quarto 1 that can’t be solved incrementally. The goal of Quarto 2 is not to change how you currently work with Quarto; instead, we’ve arrived at a point where incremental improvements do not provide the value you deserve given our team size and constraints. These are some of the things we want to do in Quarto 2:

• A new Markdown parser enables tighter integration with editors for the entire rendering pipeline. We know that good error messages, autocompletion, and YAML validation are some of your favorite features in Quarto 1. Quarto has about 1,000 different YAML configuration options, and we know how important it is to be able to provide good error messages. We want to extend this same idea to everything in your Quarto project: Markdown syntax errors, Lua filter errors, broken links, etc. Whenever possible, these should be flagged in your editor of choice.

• A fundamental solution for long-standing performance problems. Quarto 1 is built by integrating a number of tools that work very well in isolation, but aren’t designed to be performant when used together. A full rewrite of the Quarto core functionality in a single programming language will enable us to provide much better performance than before.

• A collaborative editor. Quarto 2 will ship with a collaborative editor designed to work directly on the web as well as on the command-line. Keeping in the tradition and ethos of the Quarto project, this will include a robust open-source foundation based on automerge , as well as a commercial solution for hosted project management. This follows the relationship between Quarto 1 and its integration with other Posit commercial offerings.

• A visual editor that works well alongside a source editor. The visual editor we ship in RStudio, VS Code, and Positron works well if everyone working on the document is using the visual editor. On the other hand, if you choose the visual editor, but your colleague chooses the source editor, then you’ll find that the experience is full of sharp edges. Quarto 2 is built from the ground up to support bidirectional editing workflows. A small change in your document using the visual editor shouldn’t cause a large change in the .qmd file that is disruptive for your colleagues using a source editor.

• Support for Quarto 1 projects. We aim for Quarto 2 to be backwards compatible with Quarto 1. Concretely, we’re aiming to incorporate our Quarto 1 test suite directly into Quarto 2’s project, including support for Pandoc and its output formats that our community depends on. Your existing extensions and projects should just work in Quarto 2. Early on, there will be gaps, and Quarto 2 will initially be a better fit for new projects.

What happens to Quarto 1 development?

It’s not going anywhere, and will be in active development for at least the next year. We’ll still provide bugfixes, and accept pull requests.

Current status

The development is happening in a separate GitHub repository . Feel free to look around! However, this code base isn’t ready for public consumption, and is very much in flux: that means we’re not going to spend a lot of time answering architectural questions about it until things have settled, and all discussion of Quarto should remain in our current discussion forum and issue tracker.

There are big, interesting changes in the Quarto 2 architecture, and they deserve a longer exposition. We are working on those documents right now, and will share them with you in the next few weeks. Stay tuned!

We’re pleased to announce that Shiny for Python v1.6 is now available on PyPI !

Install it now with pip install -U shiny.

This release has two big additions: toolbar components for building compact, modern UIs, and OpenTelemetry support for understanding how your apps behave in production. A full list of changes is available in the CHANGELOG .

Toolbars

Toolbars are a new set of compact components designed to fit controls into tight spaces — card headers and footers, input labels, and text areas. They’re perfect for dashboards that are running out of room, or for AI chat interfaces where you want to add controls without cluttering the layout.

The core components are:

Each input also has a corresponding ui.update_toolbar_input_*() function for updating it dynamically.

Toolbars in card headers and footers

The most common use case is placing a toolbar in a card header to attach controls directly to a card’s content:

from faicons import icon_svg
from shiny.express import input, render, ui

with ui.card(full_screen=True):
    with ui.card_header():
        "Header"
        with ui.toolbar(align="right"):
            ui.toolbar_input_button(
                id="action1",
                label="Refresh",
                icon=icon_svg("arrows-rotate"),
            )
            ui.toolbar_divider()
            ui.toolbar_input_select(
                id="options",
                label="Filter",
                choices=["ABC", "CDE", "EFG"],
            )

    @render.text
    def toolbar_status():
        return f"Button clicks: {input.action1()}, Selected: {input.options()}"

Toolbars in input labels

You can also pass a toolbar as an input’s label to add an info button for additional information or provide quick actions, like resetting an input value.

from faicons import icon_svg
from shiny.express import ui

with ui.card():
    ui.card_header("Data Settings")
    ui.input_slider(
        "threshold",
        label=ui.toolbar(
            ui.toolbar_input_button(
                "threshold_info",
                label="About this setting",
                icon=icon_svg("circle-info"),
                tooltip="Standard deviations from the mean before a value is flagged as an outlier.",
            ),
            "Outlier threshold",
            align="left",
        ),
        min=1,
        max=5,
        value=2,
        step=0.5,
    )
    ui.input_numeric(
        "sample_size",
        label=ui.toolbar(
            ui.toolbar_input_button(
                "sample_info",
                label="About this setting",
                icon=icon_svg("circle-info"),
                tooltip="Number of observations to draw from the dataset for each analysis run.",
            ),
            "Sample size",
            align="left",
        ),
        value=100,
        min=10,
        max=1000,
        step=10,
    )

Toolbars in text areas

The input_submit_textarea() component accepts a toolbar parameter directly, making it easy to add contextual controls for AI chat interfaces and message composers:

from faicons import icon_svg
from shiny import reactive
from shiny.express import input, render, ui

ui.page_opts(fillable=False)

messages = reactive.value([])

with ui.card(full_screen=True, height="250px"):
    ui.card_header("Message Composer")
    with ui.card_body():
        ui.input_submit_textarea(
            "message",
            label="Message",
            placeholder="Compose your message...",
            rows=4,
            toolbar=ui.toolbar(
                ui.toolbar_input_select(
                    "priority",
                    label="Priority",
                    choices=["Low", "Medium", "High"],
                    selected="Medium",
                    icon=icon_svg("flag"),
                ),
                ui.toolbar_divider(),
                ui.toolbar_input_button(
                    "attach",
                    label="Attach",
                    icon=icon_svg("paperclip"),
                ),
                align="right",
            ),
        )

with ui.card(full_screen=True, height="250px"):
    ui.card_header("Sent Messages")

    with ui.card_body():
        @render.ui
        def messages_output():
            msg_list = messages.get()
            if not msg_list:
                return ui.p("No messages sent yet.", style="color: #888;")

            return ui.div(
                *[
                    ui.p(
                        f"[{msg['priority']}] {msg['text']}",
                        style="margin: 4px 0;",
                    )
                    for msg in reversed(msg_list)
                ]
            )

@reactive.effect
@reactive.event(input.message)
def _():
    message_text = input.message()
    if message_text and message_text.strip():
        current_messages = list(messages.get())
        current_messages.append(
            {"text": message_text, "priority": input.priority()}
        )
        messages.set(current_messages)

Toolbars are available in py-shiny and forthcoming in bslib for R. For a complete walkthrough with full app examples, see the Toolbar component page .

OpenTelemetry

Starting with Shiny v1.6.0, OpenTelemetry support is built directly into the framework.

OpenTelemetry (OTel) is a vendor-neutral observability standard that lets you collect telemetry data — traces, logs, and metrics — and send it to any compatible backend. For Shiny apps, this means you can finally answer questions like:

• Why is my app slow for certain users?
• Which reactive expressions are taking the most time?
• How long does it take for outputs to render?
• What sequence of events occurs when a user interacts with my app?

Getting started

The fastest way to get started is with Pydantic Logfire , which provides zero-configuration OTel setup:

pip install logfire
logfire auth

Then set an environment variable to tell Shiny what level of tracing to collect:

export SHINY_OTEL_COLLECT=reactivity

That’s it — no changes to your app code required. Run your app and visit logfire.pydantic.dev to see traces.

OTel is great for GenAI apps

Shiny’s OTel integration pairs especially well with Generative AI applications. When a user reports that your chatbot feels slow, traces make it easy to pinpoint whether the delay is in the AI model request, streaming, tool execution, or a downstream reactive calculation.

The image below shows a trace from a weather forecast app powered by a Generative AI model. A single user session is captured in full detail:

What gets traced automatically

Shiny automatically creates spans for all of the following — no manual instrumentation needed:

• Session lifecycle: When sessions start and end, including HTTP request details
• Reactive updates: The entire cascade of reactive calculations triggered by an input change or a new output to be rendered
• Reactive expressions: Individual calculations such as @reactive.calc, @reactive.effect, @render.*, and other reactive constructs

Works with any OTel backend

Logfire is our recommended starting point, but Shiny’s OTel integration is fully vendor-neutral. You can send traces to Jaeger , Zipkin , Grafana Cloud , Langfuse , or any other OTLP-compatible backend.

For local debugging without a backend, install the OpenTelemetry SDK and use the console exporter:

pip install "shiny[otel]"

Full documentation — including custom spans, database instrumentation, and production considerations — is available in the OpenTelemetry guide .

In closing

We’re excited to bring you these new features in Shiny v1.6. As always, if you have questions or feedback, join us on Discord or open an issue on GitHub . Happy Shiny-ing!

The open-source Python type checker and language server ecosystem has exploded. Over the past year or two, four language server extensions have appeared, each with a different take on what Python type checking should look like. We evaluated each of them to decide which one to bundle with Positron to enhance the Python data science experience.

Background

The Language Server Protocol (LSP) is a cross-language, cross-IDE specification that allows different IDE extensions to contribute smart features like tab completions, hover info, and more. The four¹ Python extensions in this post are powered by type checkers, which are Python-specific tools that catch bugs in your code before runtime by guessing and checking the types of your variables. They do this by statically analyzing your code before you run it.

With AI tools writing more of your code, a good language server helps you read and navigate code you didn’t write. LLM-generated code also introduces bugs that type checkers catch before you run anything. For data scientists, who rely on code to be the reproducibility layer, and who can’t automate away human judgment, what matters is a tool that helps you understand and trust your code.

We did this evaluation in November 2025 but have refreshed the data in this post at the time of publish.

The contenders

Pyrefly is Meta’s successor to Pyre. It takes a fast, aggressive approach to type inference, being able to catch issues even in code with no type annotations. It reached beta status in November 2025.

ty is from Astral, the team behind uv and ruff. OpenAI announced its acquisition of Astral recently; Astral has stated that ty, ruff, and uv will remain open source and MIT-licensed. It’s the newest project, with a focus on speed and tight integration with the Astral toolchain. It reached beta status in December 2025 and follows a “gradual guarantee” philosophy (more on that below).

Basedpyright is a community fork of Microsoft’s Pyright type checker, with additional type-checking rules and LSP features baked in. It’s the most mature of the four and has the largest contributor base.

Zuban is from David Halter, the author of Jedi (the longtime Python autocompletion library). It aims for mypy compatibility and ships as a pip-installable tool.

What we tested

We tested each language server across several dimensions, roughly following the rubric we outlined publicly :

• Feature completeness: Completions, hover, go-to-definition, rename, code actions, diagnostics, inlay hints, call hierarchy
• Correctness: How well does the type checker handle real-world Python code?
• Performance: Startup time and time to first completion
• Ecosystem: License, community health, development velocity, production readiness

We tested inside Positron with a mix of data science and general Python code.

Feature completeness

Here are some screenshots of hovers, tab-completions, and diagnostics from each extension:

• Pyrefly
• ty
• Basedpyright
• Zuban

All four provide the core features you’d expect: completions, hover documentation, go-to-definition, semantic highlighting, and diagnostics. The differences show up in the details.

Pyrefly

Strong feature set. The hover documentation is the best of the four; Pyrefly renders it cleanly and sometimes includes hyperlinks to class definitions.

ty

Fast and clean, now in beta. The completion details can sometimes feel a little overwhelming, but can help when expanded.

Basedpyright

Handles type checking comprehensively well. The main friction point: it surfaces a lot of warnings out of the box. If you’re doing exploratory data science, a wall of type errors on your first pandas import can feel hostile. You can tune this down, but the defaults are oriented toward stricter use cases like package development.

Zuban

The least mature of the four so far. Installation requires a two-step process (pip install zuban, then configure the interpreter), and the analysis is tied to that specific Python installation on saved files only. Third-party library completions only work when stubs are available, not from installed packages. Symbol renaming once broke standard library code in our testing.

Type checking philosophy

The bigger difference between these tools isn’t features but how they think about type checking.

Gradual guarantee vs. aggressive inference

ty follows what’s called the gradual guarantee: removing a type annotation from correct code should never introduce a type error. The idea is that type checking should be additive. You opt in by adding types, and the checker only flags things it’s sure about.

The other extensions take the opposite approach. They always infer types from your code, even when you haven’t written any annotations. This means they can catch bugs in completely untyped code, but it also means they may flag code that runs perfectly fine.

For example:

my_list = [1, 2, 3]
my_list.append("foo")

# Pyrefly: bad-argument-type
# ty: <no error>
# Basedpyright: reportArgumentType
# Zuban: arg-type

Pyrefly infers my_list as list[int] and flags the append("foo") call as a type error. ty sees no annotations and stays silent. The code is dynamically typed and that’s fine.

If you’re doing exploratory data analysis and don’t want to annotate everything, ty’s restraint might be more comfortable. But if you’re writing a library and want to catch bugs early, Pyrefly’s aggressiveness is helpful. For example:

def process(data):
    return str(data)

process(42) + 1  # Raises a runtime AttributeError

# Pyrefly: unsupported-operation
# ty: <no error>
# Basedpyright: reportOperatorIssue
# Zuban: operator

Basedpyright and Zuban land somewhere in between, with Basedpyright leaning toward stricter checking and Zuban aiming for mypy compatibility. Each of these extensions has the ability to suppress certain diagnostics you actually see when typing if you wish.

For a deeper dive on this topic, Edward Li’s comparison of Pyrefly and ty and Rob Hand’s overview of future Python type checkers are both worth reading, though some bugs have been fixed since they were published.

Performance

We measured startup time (how long until the language server responds to an initialize request) and time to first completion (how long a textDocument/completion request takes after initialization) in a relatively small repository. We ran each measurement five times and averaged. As always, these results only represent our computer’s experimental setup.

ty was the fastest across the board. But the practical differences are small: a 3-second difference in startup happens once per session, and a 100ms difference in completions is imperceptible. All four are fast enough that differences are negligible for daily use.

Ecosystem health

We also looked at each project’s development velocity and community health metrics. A language server you rely on daily needs to keep up with Python’s evolution.

ty and Pyrefly are shipping fast. Both are on a weekly release cadence or higher with high issue throughput. ty’s issue volume is notable: 789 issues opened in 90 days reflects both heavy adoption and active bug reporting. Pyrefly is closing more issues than it’s opening, a good sign for a beta project.

Response times are quick. In a spot-check of recent issues, ty and Pyrefly both had first responses from core maintainers within minutes to hours. Basedpyright’s maintainer responds quickly too, though at a lower volume. Zuban’s maintainer often replies within an hour.

What we chose

We bundled Pyrefly as Positron’s default Python language server.

The deciding factors:

• Pyrefly’s clean design decisions felt like the best fit for Positron. The hover docs are rendered and hyperlinked, with sources for type inference. The type inference catches real bugs without requiring you to annotate everything. While it has the strictest type checking, this is configured to a moderate level by default.
• It has active development with strong backing. Meta has committed to making Pyrefly genuinely open-source and community-driven, with biweekly office hours and a public Discord. Development velocity is high.
• It is MIT licensed, which allows us to bundle it into Positron.

It wasn’t a runaway winner. Basedpyright is more mature and feature-complete. ty has a lot of long-term potential, especially for ruff users and fans of the gradual guarantee, and is closing feature gaps fast. But for the specific use case of “Python data science in an IDE,” Pyrefly had the best balance of features, UX, and readiness.

How to switch

This space is competitive and moving fast, and you shouldn’t feel locked in. Positron makes it straightforward to switch language servers:

1. Open the Extensions view (Ctrl-Shift-X (linux), Ctrl-Shift-X (windows), Command-Shift-X (mac)).
2. Search for and install the language server you want to try (e.g., basedpyright, ty, or zuban).
3. Disable Pyrefly: search for pyrefly in Extensions, click Disable.
4. Reload the window with the command Developer: Reload Window.

Or, if you want to keep Pyrefly installed but prevent it from auto-activating, you can use the extensions.allowed setting:

{
    "extensions.allowed": {
        "meta.pyrefly": false,
        "*": true
    }
}

What’s next

We started bundling Pyrefly in November and have been quite pleased with the results. It solved some longstanding user-requested issues (like better semantic highlighting) and feels snappier to users than our previous internal solution.

ty is adding features at an aggressive pace and will likely close its remaining gaps. OpenAI’s acquisition of Astral adds resources but also uncertainty; it’s unclear how it will affect ty’s priorities. Pyrefly continues to improve its type checking and performance (a recent release noted 20% faster PyTorch benchmarks ). Basedpyright tracks upstream Pyright closely and keeps shipping.

Both ty and Pyrefly have been receptive to PRs that improve the experience for Positron users, which suggests they care about working well across editors, not just VS Code. For example, both contribute hover, completions, and semantic highlighting in the Positron Console.

We’ll keep evaluating as these tools mature! Want to try Positron? Download it here .