Writing Python inside your Rust code — Part 4
Contents
In this final part of the series,
we’ll explore a trick to make the behaviour of a macro depend
on whether it’s used as a statement or as part of an expression.
Using that, we’ll make the python!{} macro
more flexible to allow saving, reusing, and inspecting Python variables.
Previously: Part 3
Note: If you’re here for the macro trick, but not very interested in Python, you can skip to that part directly.
The idea
There are a few things left that are still missing from our python!{} macro.
Most importantly, the possibility to read back data from Python into our Rust code.
As a first thought, it might make sense to try to use the same 'a syntax we used for data flowing in the opposite direction.
Maybe by making variables mut and letting Python change them?
let mut a = 5;
python! {
'a += 100
}
println!("a = {}", a);
But unfortunately, our macro is completely unaware of any of the surrounding code,
and cannot know that either a is mut nor that a will be used afterwards.
Alternatively, we could add a syntax for defining Rust variables from a python!{} block. Maybe something like:
python! {
'let a = 100
}
println!("a = {}", a);
But this idea also opens up a whole can of worms.
The 'let a might be only executed conditionally, or repeatedly, or maybe inside a function.
What to do in those cases is not at all obvious.
Additionally, type deduction is going to be tricky, if possible at all.
On top of it all, the braces and indentation of python!{..} make it look like a scope,
so variables defined inside being visible outside of it would be very unexpected.
It’s probably best if we mimic the behaviour of any other block: variables go in, but not out.
let a = 10;
if a > 5 {
let b = a + 1; // a is visible here.
..
}
// b is not visible (nor alive) here.
Eventually, the best idea might be to have a variant of the python!{} macro that evaluates to something
from which the data can be read.
Something like:
let vars = python_get! {
a = 123 + 456
};
let a_from_python: i32 = vars.get("a");
Another useful feature that’s still missing from python!{} is the ability to
spread code over multiple blocks, or ‘continue’ an execution context:
python! {
import some_library
blabla = 123
}
...
python! {
// This doesn't work. `some_library` and `blabla` don't exist here.
some_library.f(blabla)
}
This could be implemented by using the same global PyDict for every python!{} invocation,
but it’s probably best if the execution context is explicitly saved and re-used, if needed.
Saving the context is the exact same as saving the global variables, for which we already came up with a syntax:
let ctx = python_get! {
import some_library
blabla = 123
};
But for (re)using such a context, we still need something.
Maybe this:
python_with_context!{
[ctx]
some_library.f(blabla)
}
Or this:
ctx.run(python_in_context!{
some_library.f(blabla)
});
Either way, this gives yet another variant of the macro, which needs yet another name.
We could come up with three short but descriptive names, explain it properly in the documentation, and call it a day.
Or, we can be stubborn and try to find a trick to make the same macro work in all three cases:
// As a statement.
python! {
print(123)
}
// As an expression evaluating to a `Context`.
let ctx: Context = python! {
a = 123
};
// As an argument to `Context::run`.
ctx.run(python! {
print(a)
});
Context-dependent macro
Let’s go over some theory first.
‘Function-like’ macros, such as print!, vec!, and thread_local!, can be invoked using either (), [] or {}.
Although there are conventions (print!(), vec![], and thread_local!{}), each one can be invoked with any of these brackets.
fn main() {
println!["Hello, world!"];
println!{"{:?}", vec!(1, 2, 3)};
}
$ cargo r
Compiling scratchpad v0.1.0
Finished dev [unoptimized + debuginfo] target(s) in 0.21s
Running `target/debug/scratchpad`
Hello, world!
[1, 2, 3]
The macro doesn’t even know which bracket type you used.
There is however a subtle but important difference in how {} is handled.
Macros can expand to expressions (like 1 + 2), items (like struct A {}), or statements (like let a = 5;).
(And a few other things like types and patterns. But those aren’t relevant for us now.)
Important to note here is that simply adding a semicolon to an expression makes it a valid statement.
And since the last statement in a block can be an expression without semicolon,
just an expression by itself can be a valid statement too.
Similarly, statements are also a superset of items.
For example, an item like struct A {} is a valid statement.
Macro invocations using {} can be parsed as a statement or item without a semicolon.
With [] or (), they can only be parsed as such when followed by a semicolon.
thread_local! {
static A: i32 = 1;
} // No semicolon
thread_local!(
static B: i32 = 1;
); // Needs a semicolon.
Just like struct A {} needs no semicolon, but struct A(); needs one.
This leads to some interesting situations:
vec![1, 2, 3].len(); // ok
vec!{1, 2, 3}.len(); // error
(vec!{1, 2, 3}.len()); // ok
On the first line, vec![1, 2, 3] is an expression, so we can directly apply .len() to it.
The semicolon afterwards finishes the statement.
On the second line, vec!{1, 2, 3} is parsed as a full statement by itself, completed by the }, as it does not need a semicolon.
So, the .len() afterwards is parsed as the start of the next statement, which is invalid.
The third line is fine however, as the ( started an expression, such that vec!{1, 2, 3} can only be parsed as a (sub)expression.
The } couldn’t possibly end the statement, as there’s still a ( to close.
Expressions as statements
Unlike what you might be used to from a C preprocessor, macros in Rust are not expanded while parsing. The compiler will simply remember all the tokens within the brackets to pass to the macro later, and continues parsing as normal without knowing what the eventual expansion might look like. This is why it has to look at the type of brackets used to know if a semicolon should follow, because it doesn’t know yet what the macro will expand to, and whether that needs a semicolon.
Let’s take a look at this example:
macro_rules! a {
() => {
println!("hello") // No semicolon!
}
}
fn main() {
a!{}
a!{}
}
While parsing the definition of the a! macro,
the parser doesn’t know how it will be used later.
Maybe as an item, maybe as an expression, maybe something else.
So it will accept pretty much anything at this point.
While parsing the main function, the parser does not look at the definition and
does not know what a!{} might expand to.
It only sees two full statements, which is perfectly correct.
Only later, while expanding macros, it turns out a!{} expands to an expression (println!("hello")),
which will be used as a statement (remember that expressions are valid statements),
since the parser already decided that a!{} here is a full statement.
So, running this code will work just fine, even though the whole program doesn’t contain a single semicolon to separate the two printlns.
$ cargo r
Compiling scratchpad v0.1.0
Finished dev [unoptimized + debuginfo] target(s) in 0.13s
Running `target/debug/scratchpad`
hello
hello
This clearly shows that Rust macros aren’t just text substitution, but substitutions at a higher level, after parsing. Just substituting the text wouldn’t have worked:
fn main() {
println!("hello")
println!("hello")
}
$ cargo r
Compiling scratchpad v0.1.0
error: expected one of `.`, `;`, `?`, `}`, or an operator, found `println`
--> src/main.rs:3:5
|
2 | println!("hello")
| - expected one of `.`, `;`, `?`, `}`, or an operator
3 | println!("hello")
| ^^^^^^^ unexpected token
So we managed to use a macro to expand our code to something that can’t be written ‘by hand’:
a block (the body of main) consisting of two statements which are both expressions without semicolon.
An expression with semicolon as a statement means that the result of the expression is simply ignored.
This is only an error if the type was marked with #[must_use].
An expression without semicolon however, means that the result is not ignored, but used as the resulting value.
But what does that even mean for a statement that’s not the last one in a block?
What if this is not () like in our example, but, say, i32?
macro_rules! a {
() => {
5
}
}
fn five() -> i32 {
a!{} // error
a!{}
}
$ cargo b
Compiling scratchpad v0.1.0
error[E0308]: mismatched types
--> src/lib.rs:3:9
|
3 | 5
| ^- help: consider using a semicolon here
| |
| expected `()`, found integer
...
8 | a!{}
| ---- in this macro invocation
|
Aha, that’s not allowed!
Without a ;, the result is not explicitly ignored, so it must be ().
(Replacing the first a!{} by a!(); or a!{}; would make it compile again.)
Return type deduction
The last piece we need for our trick, is return type deduction.
Some expressions, like "123".parse() or x.into(), can evaluate to different types,
depending on type deduction.
The type can often be given explicitly, like "123".parse::<i32>(),
or somewhat less directly, like let a: i32 = "123".parse()?;,
or very indirectly, through something like let a = "123".parse()?; f(a);.
Making a function that could return any T (with no bounds) is pretty much impossible, other than by never returning.
So, usually, these type of functions are implemented through traits for specific types, such as FromStr or Into.
Let’s experiment a bit:
use std::any::type_name;
fn anything<T>() -> T {
// Print the name of the type.
println!("T = {}", type_name::<T>());
// Panic, because don't know how to make a T.
panic!()
}
fn main() {
anything()
}
When we run this, it displays T = () right before panicking.
The compiler successfully deduced that T must be () in this case,
because main returns nothing.
Similarly, it can deduce types from arguments, and many other things:
fn f(_: i32) {}
fn main() {
f(anything())
}
This will output T = i32, as expected.
If we go back to the previous example, and add a semicolon, we see deduction fails:
fn main() {
anything(); // error
}
The semicolon here is explicitly throwing the result of the expression away,
so it could’ve been anything.
Without an explicit type, rustc will not know which version of anything you want here,
as literally all of them (anything<()>, anything<i32>, etc.) would fit.
Now back to our trick to make a non-semicolon-terminated expression appear in the middle of a block.
That one must be a (), as we’ve seen.
Does that also work with type deduction?
macro_rules! a {
() => {
anything()
}
}
fn main() {
a!{}
a!{}
}
$ cargo r
Compiling scratchpad v0.1.0
Finished dev [unoptimized + debuginfo] target(s) in 0.14s
T = ()
...
Yup! Only () was allowed there, so anything<()> was called.
The trick
Now we have something very interesting:
fn f(_: i32) {}
fn main() {
a!{} // anything<()> is called
f(a!{}); // anything<i32> is called
let _: u8 = a!{}; // anything<u8> is called
}
The macro a! expands to a single call to anything.
But anything isn’t just a function, it’s a whole family of functions.
And most importantly, which function of the family is called depends on the context the macro is used in!
This is exactly what we need for our python! macro!
All that’s left is making anything do something more useful than panicking,
by having different implementations for different types:
trait Trick { fn trick() -> Self; }
impl Trick for () { fn trick() { println!("A statement."); } }
impl Trick for ArgToF { fn trick() -> Self { println!("Argument to f()."); ArgToF } }
impl Trick for u8 { fn trick() -> u8 { println!("Assignment to u8."); 0 } }
macro_rules! a { () => { Trick::trick() } } // No semicolon!
struct ArgToF;
fn f(_: ArgToF) {}
fn main() {
a!{}
f(a!{});
let _: u8 = a!{};
}
$ cargo r
Compiling scratchpad v0.1.0
Finished dev [unoptimized + debuginfo] target(s) in 0.13s
Running `target/debug/scratchpad`
A statement.
Argument to f().
Assignment to u8.
✨
Extending the python! macro
Now that we know how to make a macro’s behaviour a bit more flexible,
let’s get back to our python!{} macro and implement the idea we came up with.
Right now, our procedural macro generates a direct call to run_python,
giving it two arguments:
the Python bytecode,
and a function which will put the captured variables into a PyDict.
The only thing we need to change in our proc-macro crate,
is to replace the run_python call by a call to a function of a trait, which we’ll call InlinePython:
quote!(
inline_python::InlinePython::inline_python(
// <snip>
) // no semicolon!
)
.into()
All other changes, including the trait itself, go in the runtime crate inline_python.
We’ll start with the () case.
All we need to do, is replace the function
pub fn run_python(bytecode: &[u8], f: impl FnOnce(&PyDict)) {
...
}
by an implementation on a trait:
pub trait InlinePython {
fn inline_python(bytecode: &[u8], f: impl FnOnce(&PyDict)) -> Self;
}
impl InlinePython for () {
fn inline_python(bytecode: &[u8], f: impl FnOnce(&PyDict)) {
...
}
}
And everything should still work like before:
#![feature(proc_macro_hygiene)]
use inline_python::python;
fn main() {
let x = "World";
python! {
print("Hello")
print('x)
}
}
$ cargo r
Compiling inline-python-macros v0.1.0
Compiling inline-python v0.1.0
Compiling example v0.1.0
Finished dev [unoptimized + debuginfo] target(s) in 0.77s
Running `target/debug/example`
Hello
World
Okay, now we can start adding features.
The globals dictionary
Next, we make a type (let’s call it Context), that will hold the dictionary of global Python variables
after or between python!{} blocks.
pub struct Context {
globals: Py<PyDict>,
}
A Py<PyDict> is an alternative to &'p PyDict without its lifetime bound to the Python global interpreter lock (GIL).
Doing anything useful with it still requires the lock, but it can exist without holding the lock.
This makes things a bit easier to implement.
We also add a function to read a variable from it:
impl Context {
pub fn get<T: for<'p> FromPyObject<'p>>(&self, name: &str) -> T {
let gil = Python::acquire_gil();
let value = self.globals.as_ref(gil.python()).get_item(name).unwrap();
FromPyObject::extract(value).unwrap()
}
}
Now that signature looks a bit complicated.
The FromPyObject trait is parameterized by a lifetime, the Python GIL.
This means it could possibly produce a borrowed value,
given that we hold the lock long enough.
To keep thing simple, we’ll only focus now on types which don’t require the lock to be held,
such as i32 or String, which own their data.
But then what do we give as a lifetime?
The lock isn’t held outside this function, so we can’t possibly name this lifetime in the signature.
With for<'p> we can introduce a new lifetime, which tells the compiler that it should work for any lifetime 'p,
ruling out anything that would actually use the lifetime.
Okay, back to the code. We’ll move the implementation of inline_python for () to
the implementation for Context, and implement one using the other:
impl InlinePython for () {
fn inline_python(bytecode: &[u8], f: impl FnOnce(&PyDict)) {
// Ignore the resulting context.
Context::inline_python(bytecode, f);
}
}
impl InlinePython for Context {
fn inline_python(bytecode: &[u8], f: impl FnOnce(&PyDict)) -> Self {
// <snip>
// Create the globals dict. (Same as before.)
let globals = ...;
f(globals); // Put the captured variables in this dict.
// Run the Python code.
// <snip>
// Return the globals dict, which has now been modified.
Self {
globals: globals.into()
}
}
}
That should do the trick.
Let’s try!
#![feature(proc_macro_hygiene)]
use inline_python::{Context, python};
fn main() {
python! {
print("This still works")
}
let c: Context = python! {
a = 1 + 2 + 3
};
dbg!(c.get::<i32>("a"));
}
$ cargo r
Compiling inline-python v0.1.0
Compiling example v0.1.0
Finished dev [unoptimized + debuginfo] target(s) in 0.38s
Running `target/debug/example`
This still works
[src/main.rs:13] c.get::<i32>("a") = 6
Nice!
python!{} as an argument
The last one, context.run(python!{..}), is a little harder.
In this case, the InlinePython::inline_python implementation can’t just directly run the Python code,
because it doesn’t have access to the Context.
Instead, it’ll need to store the bytecode and globals-setting-function f into some object
that can be passed to context.run(), which will then run the Python code.
Because the exact type of the f (the closure generated by the procedural macro)
can’t be named, we’ve used impl FnOnce(&PyDict) so far.
Putting it into a struct means making this struct generic over this type.
pub struct PythonCode<F: FnOnce(&PyDict)> {
bytecode: &'static [u8],
set_globals: F,
}
But now we can’t implement the InlinePython trait for it,
because the F in PythonCode<F> depends on the generic type parameter of the function, not the trait:
impl InlinePython for PythonCode<...> {
fn inline_python(bytecode: &[u8], f: impl FnOnce(&PyDict)) -> Self {
PythonCode {
bytecode,
set_globals: f,
}
}
}
We can’t possibly fill in the ... here to make it work.
It’d need to refer to the type that’ll be defined on the line below it.
So, we’ll have to move the generic parameter from the function to the trait.
pub trait InlinePython<F: FnOnce(&PyDict)> {
fn inline_python(bytecode: &'static [u8], f: F) -> Self;
}
We also added a 'static lifetime to the bytecode,
since we are now storing it instead of using it directly.
Now we can implement it for our new struct:
impl<F: FnOnce(&PyDict)> InlinePython<F> for PythonCode<F> {
fn inline_python(bytecode: &'static [u8], f: F) -> Self {
PythonCode {
bytecode,
set_globals: f,
}
}
}
And finally implement Context::run:
impl Context {
pub fn run<F: FnOnce(&PyDict)>(&self, p: PythonCode<F>) {
// <snip>
let code = // <snip> (PyMarshal_ReadObjectFromString, same as before)
(p.set_globals)(self.globals.as_ref(py));
// Run the Python code.
// <snip> (PyEval_EvalCode, same as before)
}
}
And implement the first two implementations using Context::run, to avoid code duplication:
// Implementation of `python!{}`
impl<F: FnOnce(&PyDict)> InlinePython<F> for () {
fn inline_python(bytecode: &'static [u8], f: F) {
Context::inline_python(bytecode, f);
}
}
// Implementation of `let x: Context = python!{};`
impl<F: FnOnce(&PyDict)> InlinePython<F> for Context {
fn inline_python(bytecode: &'static [u8], f: F) -> Self {
let c = Context {
globals: Python::acquire_gil().python()
.import("__main__").unwrap()
.dict().copy().unwrap()
.into(),
};
c.run(PythonCode { bytecode, set_globals: f });
c
}
}
And we’re ready to try it out!
#![feature(proc_macro_hygiene)]
use inline_python::{Context, python};
fn main() {
python! {
print("This still works")
}
let c: Context = python! {
a = 1 + 2 + 3
};
c.run(python! {
print(a)
a += 100
});
dbg!(c.get::<i32>("a"));
}
Fingers crossed!
$ cargo r
Compiling inline-python v0.1.0
Compiling example v0.1.0
Finished dev [unoptimized + debuginfo] target(s) in 0.39s
Running `target/debug/example`
This still works
6
[src/main.rs:18] c.get::<i32>("a") = 106
🎉
What’s next
This was the final part in the series about writing Python inside Rust.
The macro described in this series (with a lot of improvements) is available
in the inline-python crate.
Go ahead and play with it, use it, report bugs,
and enjoy.
I’m not sure yet what the next blog post will be about, but I’m planning to write something about (in no particular order):
- How colors work, why we use RGB even though human eyes are most sensitive to yellow, green and purple, what gamma is, why magenta doesn’t occur in the rainbow, etc.
- How atomics and memory ordering work, and in which way CPU caches are and aren’t involved.
- How mutexes and condition variables work, how they are implemented on different platforms,
and why
std::sync::Mutexis inefficient. - Something about
mmapand sharing memory between processes. - Something about UTF-8 and WTF-8 and Rust’s string handling on UTF-16 platforms (Windows).
- Something about floating point numbers and their representation.
Let me know if any of these subjects stands out to you, or if you have other ideas. :)